2024-07-26T12:19:54.564 INFO:root:teuthology version: 0.0.1.dev379+g53ce146 2024-07-26T12:19:54.591 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-26T12:19:54.681 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818826 branch: wip-vshankar-testing-20240725.121331-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/fsx}} email: vshankar@redhat.com first_in_suite: false job_id: '7818826' kernel: client: branch: testing flavor: default kdb: 1 sha1: 3e9255e907100f88049f320ca33d977e29155d12 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240725.121331-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug cephsqlite: 20 mgr/snap_schedule/allow_m_granularity: true mgr/snap_schedule/dump_on_update: true mgr/snap_schedule/log_level: debug mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: global: ms die on skipped message: false mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request mgr-modules: - snap_schedule sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240725.121331-debug sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5532 sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 sleep_before_teardown: 0 subset: 1001/1024 suite: fs suite_branch: wip-vshankar-testing-20240725.121331-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCC+TT/kNc8dBslQWQQ23T0qREE6FrNe7fgB32vvXks9lJta8/ler6LMsky2aO99fXwLYtF9iAU9PiZVT+KxtPE= smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMwB/W+Xty8L+FAwqLypsbVUx/g03YxgTJW5CDvqb6f0I0xBNwENubEoQwTUKH0/u+gRlZnTQNM0g+D5Yd83/kU= smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLRWjPWyX9rPzxof4hF3c0XDhNUGCQkRCW/ZGZjTXITWoXKAfOQ0jBRZ8qLQBKwtvn3W+O/ut/YXu6hNLzN7ksc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 53ce1462e129f6eb4071986336534c740fdebd31 timestamp: 2024-07-26_04:18:59 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-07-26T12:19:54.681 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa; will attempt to use it 2024-07-26T12:19:54.682 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks 2024-07-26T12:19:54.682 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-07-26T12:19:54.684 INFO:teuthology.task.internal:Checking packages... 2024-07-26T12:19:54.704 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528' 2024-07-26T12:19:54.704 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-07-26T12:19:54.704 INFO:teuthology.packaging:ref: None 2024-07-26T12:19:54.704 INFO:teuthology.packaging:tag: None 2024-07-26T12:19:54.704 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240725.121331-debug 2024-07-26T12:19:54.704 INFO:teuthology.packaging:sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:19:54.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240725.121331-debug 2024-07-26T12:19:54.847 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5320.ge8f88bb1 2024-07-26T12:19:54.849 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-07-26T12:19:54.855 INFO:teuthology.task.internal:no buildpackages task found 2024-07-26T12:19:54.855 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-07-26T12:19:54.891 INFO:teuthology.task.internal:Saving configuration 2024-07-26T12:19:54.911 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-07-26T12:19:54.918 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-07-26T12:19:54.939 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818826', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 12:14:56.414954', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCC+TT/kNc8dBslQWQQ23T0qREE6FrNe7fgB32vvXks9lJta8/ler6LMsky2aO99fXwLYtF9iAU9PiZVT+KxtPE='} 2024-07-26T12:19:54.956 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818826', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 12:14:56.412866', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMwB/W+Xty8L+FAwqLypsbVUx/g03YxgTJW5CDvqb6f0I0xBNwENubEoQwTUKH0/u+gRlZnTQNM0g+D5Yd83/kU='} 2024-07-26T12:19:54.975 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818826', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 12:14:56.414044', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLRWjPWyX9rPzxof4hF3c0XDhNUGCQkRCW/ZGZjTXITWoXKAfOQ0jBRZ8qLQBKwtvn3W+O/ut/YXu6hNLzN7ksc='} 2024-07-26T12:19:54.976 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-07-26T12:19:54.981 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-07-26T12:19:54.982 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-07-26T12:19:54.982 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-07-26T12:19:54.982 INFO:teuthology.run_tasks:Running task console_log... 2024-07-26T12:19:55.060 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f641eab7ac0>, signals=[15]) 2024-07-26T12:19:55.060 INFO:teuthology.run_tasks:Running task internal.connect... 2024-07-26T12:19:55.067 INFO:teuthology.task.internal:Opening connections... 2024-07-26T12:19:55.067 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-07-26T12:19:55.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T12:19:55.143 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-07-26T12:19:55.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T12:19:55.215 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-07-26T12:19:55.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T12:19:55.291 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-07-26T12:19:55.298 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-07-26T12:19:55.316 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-07-26T12:19:55.316 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-07-26T12:19:55.373 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-07-26T12:19:55.373 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-07-26T12:19:55.373 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-07-26T12:19:55.373 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-07-26T12:19:55.373 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-07-26T12:19:55.373 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-07-26T12:19:55.373 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T12:19:55.374 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-07-26T12:19:55.374 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-07-26T12:19:55.374 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T12:19:55.374 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-07-26T12:19:55.374 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T12:19:55.374 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T12:19:55.374 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T12:19:55.375 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-07-26T12:19:55.399 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-07-26T12:19:55.417 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-07-26T12:19:55.417 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-07-26T12:19:55.475 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="CentOS Stream" 2024-07-26T12:19:55.475 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="9" 2024-07-26T12:19:55.475 INFO:teuthology.orchestra.run.smithi112.stdout:ID="centos" 2024-07-26T12:19:55.476 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="rhel fedora" 2024-07-26T12:19:55.476 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="9" 2024-07-26T12:19:55.476 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el9" 2024-07-26T12:19:55.476 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T12:19:55.476 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2024-07-26T12:19:55.476 INFO:teuthology.orchestra.run.smithi112.stdout:LOGO="fedora-logo-icon" 2024-07-26T12:19:55.477 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T12:19:55.477 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://centos.org/" 2024-07-26T12:19:55.477 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T12:19:55.477 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T12:19:55.477 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T12:19:55.478 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-07-26T12:19:55.499 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-07-26T12:19:55.517 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-07-26T12:19:55.518 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-07-26T12:19:55.574 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2024-07-26T12:19:55.575 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="9" 2024-07-26T12:19:55.575 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2024-07-26T12:19:55.575 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2024-07-26T12:19:55.575 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="9" 2024-07-26T12:19:55.575 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el9" 2024-07-26T12:19:55.575 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T12:19:55.575 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-07-26T12:19:55.575 INFO:teuthology.orchestra.run.smithi132.stdout:LOGO="fedora-logo-icon" 2024-07-26T12:19:55.575 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T12:19:55.576 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2024-07-26T12:19:55.576 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T12:19:55.576 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T12:19:55.576 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T12:19:55.576 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-07-26T12:19:55.599 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-07-26T12:19:55.610 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-07-26T12:19:55.632 INFO:teuthology.task.internal:Checking for old test directory... 2024-07-26T12:19:55.632 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-07-26T12:19:55.635 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-07-26T12:19:55.637 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-07-26T12:19:55.653 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-07-26T12:19:55.660 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-07-26T12:19:55.660 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-07-26T12:19:55.692 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-07-26T12:19:55.695 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-07-26T12:19:55.799 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-07-26T12:19:55.869 INFO:teuthology.run_tasks:Running task kernel... 2024-07-26T12:19:55.885 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'} 2024-07-26T12:19:55.885 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-07-26T12:19:55.885 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-07-26T12:19:55.886 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T12:19:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T12:19:55.902 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-07-26T12:19:56.092 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T12:19:56.092 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3e9255e907100f88049f320ca33d977e29155d12"... 2024-07-26T12:19:56.092 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-07-26T12:19:56.109 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-437.el9.x86_64 2024-07-26T12:19:56.109 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T12:19:56.110 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-07-26T12:19:56.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum upgrade -y linux-firmware 2024-07-26T12:19:59.259 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.2 MB 00:01 2024-07-26T12:20:02.337 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-07-26T12:20:06.709 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-07-26T12:20:08.202 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 18 kB 00:00 2024-07-26T12:20:09.014 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 32 MB/s | 22 MB 00:00 2024-07-26T12:20:14.270 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-07-26T12:20:16.406 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-07-26T12:20:16.417 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-07-26T12:20:16.417 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-07-26T12:20:16.417 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-07-26T12:20:16.417 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-07-26T12:20:16.417 INFO:teuthology.orchestra.run.smithi107.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-07-26T12:20:16.417 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:20:16.417 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-07-26T12:20:16.417 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-07-26T12:20:16.417 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 1 Package 2024-07-26T12:20:16.417 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:20:16.418 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 383 M 2024-07-26T12:20:16.418 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-07-26T12:20:29.736 INFO:teuthology.orchestra.run.smithi107.stdout:linux-firmware-20240716-145.el9.noarch.rpm 29 MB/s | 383 MB 00:13 2024-07-26T12:20:29.736 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-07-26T12:20:29.736 INFO:teuthology.orchestra.run.smithi107.stdout:Total 29 MB/s | 383 MB 00:13 2024-07-26T12:20:32.816 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-07-26T12:20:32.902 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-07-26T12:20:32.902 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-07-26T12:20:36.146 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-07-26T12:20:36.147 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-07-26T12:20:45.084 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-07-26T12:20:45.794 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T12:20:45.877 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T12:20:46.637 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T12:20:46.637 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T12:20:47.376 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T12:20:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:20:47.377 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-07-26T12:20:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: linux-firmware-20240716-145.el9.noarch 2024-07-26T12:20:47.377 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:20:47.377 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-07-26T12:20:47.690 INFO:teuthology.task.kernel:Downloading kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.0... 2024-07-26T12:20:47.691 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T12:20:47.841 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ 2024-07-26T12:20:47.841 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.10.0_g3e9255e90710-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ --input-file=- 2024-07-26T12:20:47.849 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi107.front.sepia.ceph.com to complete... 2024-07-26T12:20:54.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26 12:20:54 URL:https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/kernel-6.10.0_g3e9255e90710-1.x86_64.rpm [358399635/358399635] -> "/tmp/kernel.x86_64.rpm" [1] 2024-07-26T12:20:54.120 INFO:teuthology.orchestra.run.smithi107.stderr:FINISHED --2024-07-26 12:20:54-- 2024-07-26T12:20:54.120 INFO:teuthology.orchestra.run.smithi107.stderr:Total wall clock time: 6.2s 2024-07-26T12:20:54.120 INFO:teuthology.orchestra.run.smithi107.stderr:Downloaded: 1 files, 342M in 6.0s (57.2 MB/s) 2024-07-26T12:20:54.123 INFO:teuthology.task.kernel:Installing kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.0... 2024-07-26T12:20:54.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-07-26T12:20:56.168 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying... ######################################## 2024-07-26T12:20:56.228 INFO:teuthology.orchestra.run.smithi107.stdout:Preparing... ######################################## 2024-07-26T12:20:56.277 INFO:teuthology.orchestra.run.smithi107.stdout:Updating / installing... 2024-07-26T12:21:04.243 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-6.10.0_g3e9255e90710-1 ######################################## 2024-07-26T12:21:43.839 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-07-26T12:21:43.839 DEBUG:teuthology.orchestra.run.smithi107:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-07-26T12:21:43.921 INFO:teuthology.orchestra.run.smithi107.stdout:/boot/System.map-6.10.0-g3e9255e90710 2024-07-26T12:21:43.921 INFO:teuthology.orchestra.run.smithi107.stdout:/boot/config-6.10.0-g3e9255e90710 2024-07-26T12:21:43.921 INFO:teuthology.orchestra.run.smithi107.stdout:/boot/vmlinuz-6.10.0-g3e9255e90710 2024-07-26T12:21:43.921 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710 2024-07-26T12:21:43.921 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/System.map 2024-07-26T12:21:43.921 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/config 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32c-intel.ko 2024-07-26T12:21:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx2.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu 2024-07-26T12:21:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-amd.ko 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-intel.ko 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm.ko 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ansi_cprng.ko 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/anubis.ko 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/arc4.ko 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_memcpy.ko 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_pq.ko 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_raid6_recov.ko 2024-07-26T12:21:43.924 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_tx.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_xor.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/raid6test.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authenc.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authencesn.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blake2b_generic.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_common.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_generic.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/camellia_generic.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast5_generic.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast6_generic.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast_common.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ccm.ko 2024-07-26T12:21:43.925 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cmac.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crc32_generic.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_engine.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_user.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cts.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/des_generic.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/dh_generic.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecc.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecdh_generic.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/echainiv.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/essiv.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/fcrypt.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/khazad.ko 2024-07-26T12:21:43.926 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4hc.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/md4.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/michael_mic.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcbc.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcrypt.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/rmd160.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/seed.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/serpent_generic.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tcrypt.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tea.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_common.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_generic.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/vmac.ko 2024-07-26T12:21:43.927 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/wp512.ko 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xcbc.ko 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xor.ko 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xxhash_generic.ko 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_ipmi.ko 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_pad.ko 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/ec_sys.ko 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/platform_profile.ko 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbs.ko 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbshc.ko 2024-07-26T12:21:43.928 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/video.ko 2024-07-26T12:21:43.929 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata 2024-07-26T12:21:43.929 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/acard-ahci.ko 2024-07-26T12:21:43.929 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ahci_platform.ko 2024-07-26T12:21:43.929 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ata_generic.ko 2024-07-26T12:21:43.929 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/libahci_platform.ko 2024-07-26T12:21:43.929 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_acpi.ko 2024-07-26T12:21:43.929 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ali.ko 2024-07-26T12:21:43.929 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_amd.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_artop.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atiixp.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atp867x.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd640.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd64x.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cypress.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_efar.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt366.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt37x.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x2n.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x3.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it8213.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it821x.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_jmicron.ko 2024-07-26T12:21:43.930 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_marvell.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_mpiix.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_netcell.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ninja32.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87410.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87415.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_oldpiix.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_opti.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_optidma.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pcmcia.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc2027x.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_piccolo.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_rdc.ko 2024-07-26T12:21:43.931 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sch.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_serverworks.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sil680.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sis.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sl82c105.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_triflex.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_via.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pdc_adma.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_inic162x.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_mv.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_nv.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_promise.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_qstor.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil24.ko 2024-07-26T12:21:43.932 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sis.ko 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_svw.ko 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sx4.ko 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_uli.ko 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_via.ko 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_vsc.ko 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay/ks0108.ko 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-i2c.ko 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-mmio.ko 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma/bcma.ko 2024-07-26T12:21:43.933 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe/aoe.ko 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/brd.ko 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd/drbd.ko 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/floppy.ko 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/loop.ko 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/nbd.ko 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk/null_blk.ko 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/pktcdvd.ko 2024-07-26T12:21:43.934 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/rbd.ko 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/virtio_blk.ko 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkfront.ko 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram/zram.ko 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hangcheck-timer.ko 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/amd-rng.ko 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/intel-rng.ko 2024-07-26T12:21:43.935 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/via-rng.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/virtio-rng.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_si.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/lp.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave/mwave.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ppdev.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tlclk.ko 2024-07-26T12:21:43.936 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_atmel.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_infineon.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_nsc.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/uv_mmtimer.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/virtio_console.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/p4-clockmod.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-07-26T12:21:43.937 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/powernow-k8.ko 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/speedstep-lib.ko 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp.ko 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-07-26T12:21:43.938 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-aes.ko 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-sha.ko 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca/dca.ko 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw/dw_dmac.ko 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat 2024-07-26T12:21:43.939 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat/ioatdma.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/amd64_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/e752x_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/edac_mce_amd.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3000_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3200_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5100_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5400_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7300_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7core_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i82975x_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/ie31200_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/sb_edac.ko 2024-07-26T12:21:43.940 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/x38_edac.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/edd.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/iscsi_ibft.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-ich.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-viperboard.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-appleir.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-aureal.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-axff.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-dr.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elecom.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elo.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-emsff.ko 2024-07-26T12:21:43.941 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gaff.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gt683r.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gyration.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-kbd.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-mouse.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtekff.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-hyperv.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-icade.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-keytouch.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-kye.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lcpower.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-led.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lenovo.ko 2024-07-26T12:21:43.942 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-dj.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-multitouch.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-ortek.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-penmount.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-petalynx.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-picolcd.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-pl.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-plantronics.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-primax.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-rmi.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-arvo.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-common.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-isku.ko 2024-07-26T12:21:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kone.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-konepure.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-lua.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-pyra.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-ryos.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-savu.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-saitek.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-samsung.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sensor-hub.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sjoy.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sony.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-speedlink.ko 2024-07-26T12:21:43.944 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-steelseries.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sunplus.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-thrustmaster.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tivo.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tmff.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-topseed.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-twinhan.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-uclogic.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-waltop.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-wiimote.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-xinmo.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zpff.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zydacron.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/uhid.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/wacom.ko 2024-07-26T12:21:43.945 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_balloon.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_utils.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_vmbus.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru3.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/acpi_power_meter.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7414.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7418.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adc128d818.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1025.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1026.ko 2024-07-26T12:21:43.946 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1029.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1031.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm9240.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ads7828.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7410.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7411.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7462.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7470.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7475.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7x10.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/amc6821.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/applesmc.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asb100.ko 2024-07-26T12:21:43.947 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asc7621.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asus_atk0110.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/atxp1.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/coretemp.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dme1737.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds1621.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds620.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc1403.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc6w201.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71805f.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71882fg.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f75375s.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fam15h_power.ko 2024-07-26T12:21:43.948 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fschmd.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g760a.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g762.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl518sm.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl520sm.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/hwmon-vid.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5500_temp.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5k_amb.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmaem.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmpex.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina209.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina2xx.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/it87.ko 2024-07-26T12:21:43.949 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k10temp.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k8temp.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lineage-pem.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm63.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm73.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm75.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm77.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm78.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm80.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm83.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm85.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm87.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm90.ko 2024-07-26T12:21:43.950 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm92.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm93.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95234.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95241.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95245.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc2945.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4151.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4215.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4222.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4245.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4260.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4261.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max16065.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1619.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1668.ko 2024-07-26T12:21:43.951 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max197.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6639.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6650.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6697.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/mcp3021.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6683.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775-core.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct7802.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87360.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87427.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pcf8591.ko 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus 2024-07-26T12:21:43.952 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max16064.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max34440.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max8688.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/powr1220.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5627.ko 2024-07-26T12:21:43.953 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5636.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch56xx-common.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht15.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht21.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/shtc1.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sis5595.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47b397.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m1.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m192.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/thmc50.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp102.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp103.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp401.ko 2024-07-26T12:21:43.954 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp421.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via-cputemp.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via686a.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt1211.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt8231.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627ehf.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627hf.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83781d.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83791d.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83792d.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83793.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83795.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l785ts.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l786ng.ko 2024-07-26T12:21:43.955 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-i801.ko 2024-07-26T12:21:43.956 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-isch.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-parport.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-via.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-07-26T12:21:43.957 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-dev.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-mux.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-smbus.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-stub.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_cm.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_core.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_umad.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/iw_cm.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_cm.ko 2024-07-26T12:21:43.958 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-07-26T12:21:43.959 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/fm801-gp.ko 2024-07-26T12:21:43.960 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/gameport.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/joydev.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard/gpio_keys.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/appletouch.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/bcm5974.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/cyapatp.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/elan_i2c.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/sermouse.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_usb.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/vsxxxaa.ko 2024-07-26T12:21:43.961 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4/rmi_core.ko 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/altera_ps2.ko 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/arc_ps2.ko 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/serio_raw.ko 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/sparse-keymap.ko 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-blinkm.ko 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lm3530.ko 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lp3944.ko 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-ss4200.ko 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger 2024-07-26T12:21:43.962 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache/bcache.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-bio-prison.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache-smq.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-crypt.ko 2024-07-26T12:21:43.963 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-delay.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-flakey.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-log-userspace.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-multipath.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-queue-length.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-raid.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-round-robin.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-service-time.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-switch.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-thin-pool.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-verity.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid0.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid1.ko 2024-07-26T12:21:43.964 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid10.ko 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid456.ko 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptbase.ko 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptctl.ko 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptfc.ko 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptlan.ko 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptsas.ko 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptscsih.ko 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptspi.ko 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_ich.ko 2024-07-26T12:21:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_sch.ko 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/mfd-core.ko 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/sm501.ko 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/viperboard.ko 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/vx855.ko 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/wl1273-core.ko 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds9802als.ko 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds990x.ko 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/bh1770glc.ko 2024-07-26T12:21:43.966 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710/cb710.ko 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/at24.ko 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/max6875.ko 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/enclosure.ko 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/hpilo.ko 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29003.ko 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29020.ko 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei 2024-07-26T12:21:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-me.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-txe.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru/gru.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xp.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpc.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_7xx1.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_core.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tsl2550.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_balloon.ko 2024-07-26T12:21:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding/bonding.ko 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53/b53_common.ko 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/bcm-sf2.ko 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6060.ko 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dummy.ko 2024-07-26T12:21:43.969 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/eql.ko 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/8390.ko 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-07-26T12:21:43.970 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere/et131x.ko 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-07-26T12:21:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade 2024-07-26T12:21:43.972 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-07-26T12:21:43.973 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dnet.ko 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex 2024-07-26T12:21:43.974 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ethoc.ko 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/fealnx.ko 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e100.ko 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf 2024-07-26T12:21:43.975 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/jme.ko 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell 2024-07-26T12:21:43.976 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/skge.ko 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-07-26T12:21:43.977 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen 2024-07-26T12:21:43.978 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/atp.ko 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc 2024-07-26T12:21:43.979 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis190.ko 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis900.ko 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-07-26T12:21:43.980 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/cassini.ko 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/niu.ko 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sungem.ko 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti/tlan.ko 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet 2024-07-26T12:21:43.981 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/geneve.ko 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154/fakelb.ko 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ifb.ko 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan/ipvlan.ko 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvlan.ko 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvtap.ko 2024-07-26T12:21:43.982 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio.ko 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-regmap.ko 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mii.ko 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/net_failover.ko 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/netconsole.ko 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/nlmon.ko 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs-lynx.ko 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy 2024-07-26T12:21:43.983 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/amd.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm7xxx.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm87xx.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/broadcom.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/cicada.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/davicom.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/dp83640.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/et1011c.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/icplus.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/lxt.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/marvell.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/micrel.ko 2024-07-26T12:21:43.984 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/national.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/phylink.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/qsemi.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/realtek.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/smsc.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/ste10Xp.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/vitesse.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/bsd_comp.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_async.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_deflate.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_generic.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_mppe.ko 2024-07-26T12:21:43.985 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_synctty.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppoe.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppox.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pptp.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slhc.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slip.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/sungem_phy.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tap.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_activebackup.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_broadcast.ko 2024-07-26T12:21:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_random.ko 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tun.ko 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/veth.ko 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/virtio_net.ko 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan/vxlan.ko 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback/xen-netback.ko 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netfront.ko 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb 2024-07-26T12:21:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb/ntb.ko 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme-core.ko 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme.ko 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport.ko 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_cs.ko 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_pc.ko 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_serial.ko 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug 2024-07-26T12:21:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie/aer_inject.ko 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/xen-pcifront.ko 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/i82092.ko 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/pd6729.ko 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/yenta_socket.ko 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acer-wmi.ko 2024-07-26T12:21:43.989 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acerhdf.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/apple-gmux.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-laptop.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-wmi.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/classmate-laptop.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/hdaps.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel_ips.ko 2024-07-26T12:21:43.990 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/msi-wmi.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/mxm-wmi.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-laptop.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-q10.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/topstar-laptop.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_haps.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi-bmof.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_common.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_msr.ko 2024-07-26T12:21:43.991 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-gpio.ko 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-ldisc.ko 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps_parport.ko 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/pps_core.ko 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp.ko 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp_kvm.ko 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/fmpm.ko 2024-07-26T12:21:43.992 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-bq32k.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1286.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1307.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1374.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1511.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1553.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1672.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1742.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds2404.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds3232.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-em3027.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-fm3130.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl12022.ko 2024-07-26T12:21:43.993 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl1208.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m41t80.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t35.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t59.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-max6900.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-msm6242.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf2127.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf85063.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8523.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8563.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8583.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rp5c01.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rs5c372.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-07-26T12:21:43.994 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8025.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8581.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-x1205.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-9xxx.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-sas.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-xxxx.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/BusLogic.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/a100u2w.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid/aacraid.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/advansys.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-07-26T12:21:43.995 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/am53c974.ko 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/atp870u.ko 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa/bfa.ko 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ch.ko 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor 2024-07-26T12:21:43.996 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor/csiostor.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dc395x.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dmx3191d.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r/esas2r.ko 2024-07-26T12:21:43.997 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esp_scsi.ko 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/fcoe.ko 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic/fnic.ko 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hpsa.ko 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hptiop.ko 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hv_storvsc.ko 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/initio.ko 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ipr.ko 2024-07-26T12:21:43.998 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ips.ko 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci/isci.ko 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_tcp.ko 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc/libfc.ko 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi.ko 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi_tcp.ko 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas/libsas.ko 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc/lpfc.ko 2024-07-26T12:21:43.999 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid.ko 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas/mvsas.ko 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvumi.ko 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pmcraid.ko 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla1280.ko 2024-07-26T12:21:44.000 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/raid_class.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_debug.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_fc.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_sas.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_spi.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_srp.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ses.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/st.ko 2024-07-26T12:21:44.001 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/stex.ko 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/virtio_scsi.ko 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/vmw_pvscsi.ko 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/wd719x.ko 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb/ssb.ko 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback/tcm_loop.ko 2024-07-26T12:21:44.002 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_file.ko 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_iblock.ko 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_mod.ko 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_pscsi.ko 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_user.ko 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-07-26T12:21:44.003 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-07-26T12:21:44.004 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_gsm.ko 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_hdlc.ko 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/nozomi.ko 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/arc_uart.ko 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm 2024-07-26T12:21:44.005 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm/jsm.ko 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core/ufshcd-core.ko 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio.ko 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_aec.ko 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_cif.ko 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_pci_generic.ko 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_sercos3.ko 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb 2024-07-26T12:21:44.006 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-acm.ko 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-wdm.ko 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usblp.ko 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usbtmc.ko 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host/sl811-hcd.ko 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/mdc800.ko 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/microtek.ko 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/adutux.ko 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/appledisplay.ko 2024-07-26T12:21:44.007 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi26.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi62.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ezusb.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/idmouse.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/iowarrior.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/isight_firmware.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ldusb.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/legousbtower.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/trancevibrator.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usb3503.ko 2024-07-26T12:21:44.008 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usblcd.ko 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usbsevseg.ko 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/uss720.ko 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/yurex.ko 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy/phy-generic.ko 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/aircable.ko 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ark3116.ko 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/belkin_sa.ko 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ch341.ko 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cp210x.ko 2024-07-26T12:21:44.009 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cyberjack.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cypress_m8.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/digi_acceleport.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/empeg.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ftdi_sio.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/garmin_gps.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_edgeport.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_ti.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipaq.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipw.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ir-usb.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan_pda.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kl5kusb105.ko 2024-07-26T12:21:44.010 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kobil_sct.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mct_u232.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7720.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7840.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/navman.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/omninet.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/opticon.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/option.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/oti6858.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/pl2303.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcaux.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcserial.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/quatech2.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/safe_serial.ko 2024-07-26T12:21:44.011 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/sierra.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/spcp8x5.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ssu100.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/symbolserial.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_debug.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_wwan.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/visor.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/whiteheat.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/xsens_mt.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/uas.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-alauda.ko 2024-07-26T12:21:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-cypress.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-datafab.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-freecom.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-isd200.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-karma.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-onetouch.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-realtek.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr09.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr55.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-usbat.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/usb-storage.ko 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip 2024-07-26T12:21:44.013 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-core.ko 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-host.ko 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci.ko 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio.ko 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost.ko 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_iotlb.ko 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_net.ko 2024-07-26T12:21:44.014 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_scsi.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/apple_bl.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/lcd.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/platform_lcd.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_balloon.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_mmio.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_ring.ko 2024-07-26T12:21:44.015 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2482.ko 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2490.ko 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-07-26T12:21:44.016 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_smem.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_therm.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/wire.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim1535_wdt.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim7101_wdt.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/f71808e_wdt.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/hpwdt.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/i6300esb.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_wdt.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ib700wdt.ko 2024-07-26T12:21:44.017 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ibmasr.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it8712f_wdt.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it87_wdt.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/machzwd.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/nv_tco.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_pci.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_usb.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sch311x_wdt.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/softdog.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sp5100_tco.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/via_wdt.ko 2024-07-26T12:21:44.018 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83877f_wdt.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83977f_wdt.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/wdt_pci.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/xen_wdt.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-acpi-processor.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-evtchn.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntalloc.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntdev.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-privcmd.ko 2024-07-26T12:21:44.019 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs/xenfs.ko 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p/9p.ko 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs/affs.ko 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs/befs.ko 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/binfmt_misc.ko 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs/btrfs.ko 2024-07-26T12:21:44.020 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles/cachefiles.ko 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph/ceph.ko 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda/coda.ko 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs/cramfs.ko 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm/dlm.ko 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs/ecryptfs.ko 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs/f2fs.ko 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat 2024-07-26T12:21:44.021 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/fat.ko 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/msdos.ko 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/vfat.ko 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/cuse.ko 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/fuse.ko 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2/gfs2.ko 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs/hfs.ko 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus/hfsplus.ko 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs 2024-07-26T12:21:44.022 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs/isofs.ko 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs/jfs.ko 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd/lockd.ko 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix/minix.ko 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs/netfs.ko 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-07-26T12:21:44.023 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfs.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv3.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv4.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/grace.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/nfs_acl.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd/nfsd.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2/nilfs2.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-celtic.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-centeuro.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-croatian.ko 2024-07-26T12:21:44.024 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-cyrillic.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-gaelic.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-greek.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-iceland.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-inuit.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-roman.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-romanian.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-turkish.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1250.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1251.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1255.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp737.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp775.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp850.ko 2024-07-26T12:21:44.025 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp852.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp855.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp857.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp860.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp861.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp862.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp863.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp864.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp865.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp866.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp869.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp874.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp932.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp936.ko 2024-07-26T12:21:44.026 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp949.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp950.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_euc-jp.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-1.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-13.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-14.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-15.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-2.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-3.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-4.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-5.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-6.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-7.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-9.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-r.ko 2024-07-26T12:21:44.027 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-ru.ko 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-u.ko 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_ucs2_utils.ko 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_utf8.ko 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2.ko 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-07-26T12:21:44.028 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs/overlay.ko 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore/ramoops.ko 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs/reiserfs.ko 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs/romfs.ko 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client/cifs.ko 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common 2024-07-26T12:21:44.029 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_arc4.ko 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_md4.ko 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs/squashfs.ko 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv/sysv.ko 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf/udf.ko 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs/ufs.ko 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs/xfs.ko 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel 2024-07-26T12:21:44.030 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu/rcutorture.ko 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/torture.ko 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/asn1_encoder.ko 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc-itu-t.ko 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc8.ko 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libaescfb.ko 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libarc4.ko 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libdes.ko 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim 2024-07-26T12:21:44.031 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim/dimlib.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/libcrc32c.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lru_cache.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4_compress.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4hc_compress.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math/cordic.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6/raid6_pq.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon/reed_solomon.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_bm.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_fsm.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_kmp.ko 2024-07-26T12:21:44.032 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd/zstd_compress.ko 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm/hwpoison-inject.ko 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/6lowpan.ko 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_dest.ko 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_fragment.ko 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_hop.ko 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_ipv6.ko 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_mobility.ko 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_routing.ko 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_udp.ko 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802 2024-07-26T12:21:44.033 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/garp.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/mrp.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/p8022.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/psnap.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/stp.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q/8021q.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_fd.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_rdma.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_virtio.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv/batman-adv.ko 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge 2024-07-26T12:21:44.034 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/br_netfilter.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/bridge.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_among.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arp.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_limit.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_log.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-07-26T12:21:44.035 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_snat.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_stp.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtables.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph/libceph.ko 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core 2024-07-26T12:21:44.036 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/failover.ko 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/pktgen.ko 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp.ko 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_diag.ko 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv4.ko 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv6.ko 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver/dns_resolver.ko 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/dsa_core.ko 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_brcm.ko 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_dsa.ko 2024-07-26T12:21:44.037 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_none.ko 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_trailer.ko 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154.ko 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154_socket.ko 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ah4.ko 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/esp4.ko 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/fou.ko 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/gre.ko 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/inet_diag.ko 2024-07-26T12:21:44.038 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_gre.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_tunnel.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_vti.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipcomp.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipip.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arp_tables.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-07-26T12:21:44.039 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_security.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-07-26T12:21:44.040 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_bic.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_dctcp.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_diag.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_highspeed.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_htcp.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_hybla.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_illinois.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_lp.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_scalable.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_vegas.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_veno.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_westwood.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_yeah.ko 2024-07-26T12:21:44.041 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tunnel4.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_diag.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_tunnel.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/xfrm4_tunnel.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ah6.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/esp6.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/fou6.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_tunnel.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_vti.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ipcomp6.ko 2024-07-26T12:21:44.042 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-07-26T12:21:44.043 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/sit.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/tunnel6.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/xfrm6_tunnel.ko 2024-07-26T12:21:44.044 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key/af_key.ko 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_core.ko 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_debugfs.ko 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_eth.ko 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip.ko 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip6.ko 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_netlink.ko 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ppp.ko 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc/llc.ko 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154/mac802154.ko 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls 2024-07-26T12:21:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls/mpls_gso.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-07-26T12:21:44.046 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-07-26T12:21:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conncount.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_h323.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_irc.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-07-26T12:21:44.048 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sane.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sip.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_dup_netdev.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table_inet.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_log_syslog.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_amanda.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_ftp.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_irc.ko 2024-07-26T12:21:44.049 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_sip.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_tftp.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_synproxy_core.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_tables.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_acct.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_log.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_osf.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_queue.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_chain_nat.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_compat.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_ct.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_dup_netdev.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fib.ko 2024-07-26T12:21:44.050 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fwd_netdev.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_hash.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_limit.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_log.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_masq.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_nat.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_numgen.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_queue.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_quota.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_redir.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject_inet.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_socket.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_tunnel.ko 2024-07-26T12:21:44.051 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_AUDIT.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CHECKSUM.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CLASSIFY.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CT.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_DSCP.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HL.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HMARK.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_IDLETIMER.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LED.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LOG.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_MASQUERADE.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NETMAP.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFLOG.ko 2024-07-26T12:21:44.052 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFQUEUE.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_RATEEST.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_REDIRECT.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_SECMARK.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPMSS.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TEE.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TPROXY.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TRACE.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_addrtype.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_bpf.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cgroup.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cluster.ko 2024-07-26T12:21:44.053 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_comment.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connbytes.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlabel.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlimit.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connmark.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_conntrack.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cpu.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dccp.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_devgroup.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dscp.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ecn.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_esp.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hashlimit.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_helper.ko 2024-07-26T12:21:44.054 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hl.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipcomp.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_iprange.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipvs.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_l2tp.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_length.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_limit.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mac.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mark.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_multiport.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nat.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nfacct.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_osf.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_owner.ko 2024-07-26T12:21:44.055 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_physdev.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_pkttype.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_policy.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_quota.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_rateest.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_realm.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_recent.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_sctp.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_set.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_socket.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_state.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_statistic.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_string.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_tcpmss.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_time.ko 2024-07-26T12:21:44.056 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_u32.ko 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink/netlink_diag.ko 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh/nsh.ko 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/openvswitch.ko 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-geneve.ko 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-gre.ko 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-vxlan.ko 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet/af_packet_diag.ko 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds.ko 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_rdma.ko 2024-07-26T12:21:44.057 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_tcp.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_csum.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_gact.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_mirred.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_nat.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_pedit.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_police.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_simple.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_skbedit.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_vlan.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_basic.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_bpf.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_flow.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_fw.ko 2024-07-26T12:21:44.058 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_route.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_u32.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_cmp.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_ipset.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_meta.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_nbyte.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_text.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_u32.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_choke.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_codel.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_drr.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_fq.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_gred.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hfsc.ko 2024-07-26T12:21:44.059 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hhf.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_htb.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_ingress.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio_lib.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_multiq.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_netem.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_pie.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_plug.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_prio.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_qfq.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_red.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfb.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfq.ko 2024-07-26T12:21:44.060 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_tbf.ko 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_teql.ko 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp.ko 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp_diag.ko 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/sunrpc.ko 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc.ko 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc_diag.ko 2024-07-26T12:21:44.061 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix/unix_diag.ko 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock.ko 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_diag.ko 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_loopback.ko 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm/xfrm_ipcomp.ko 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys 2024-07-26T12:21:44.062 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys/trusted.ko 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias.bin 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.alias.bin 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.bin 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.modinfo 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep.bin 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.devname 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.order 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.softdep 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols.bin 2024-07-26T12:21:44.063 INFO:teuthology.orchestra.run.smithi107.stdout:/lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T12:21:44.067 DEBUG:teuthology.task.kernel:get_image_version: 6.10.0-g3e9255e90710 2024-07-26T12:21:44.067 DEBUG:teuthology.orchestra.run.smithi107:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-07-26T12:21:44.090 INFO:teuthology.orchestra.run.smithi107.stdout:postinstall scriptlet (using /bin/sh): 2024-07-26T12:21:44.090 INFO:teuthology.orchestra.run.smithi107.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T12:21:44.090 INFO:teuthology.orchestra.run.smithi107.stdout: /usr/bin/kernel-install add 6.10.0-g3e9255e90710 /lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout:fi 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout:for file in vmlinuz System.map config; do 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout: if ! cmp --silent "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710"; then 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout: cp "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710" 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout: fi 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout:done 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout:if [ -d "/lib/modules/6.10.0-g3e9255e90710/dtb" ] && \ 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout: ! diff -rq "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" >/dev/null 2>&1; then 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout: rm -rf "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout: cp -r "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout:fi 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout:if [ ! -e "/lib/modules/6.10.0-g3e9255e90710/modules.dep" ]; then 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout: /usr/sbin/depmod 6.10.0-g3e9255e90710 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout:fi 2024-07-26T12:21:44.091 INFO:teuthology.orchestra.run.smithi107.stdout:preuninstall scriptlet (using /bin/sh): 2024-07-26T12:21:44.092 INFO:teuthology.orchestra.run.smithi107.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T12:21:44.092 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-install remove 6.10.0-g3e9255e90710 2024-07-26T12:21:44.092 INFO:teuthology.orchestra.run.smithi107.stdout:fi 2024-07-26T12:21:44.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 68.el9 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8264254 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T12:21:44.165 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-07-26T12:21:44.166 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:21:44.167 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T12:21:44.167 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T12:21:44.167 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 6.10.0-g3e9255e90710 2024-07-26T12:21:44.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T12:21:44.717 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-07-26T12:21:46.063 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T12:21:46.093 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-07-26T12:21:46.096 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-07-26T12:21:46.110 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.KFjqIb4XIL 2024-07-26T12:21:46.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KFjqIb4XIL 2024-07-26T12:21:46.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.KFjqIb4XIL 2024-07-26T12:21:46.312 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.KFjqIb4XIL is 6KB 2024-07-26T12:21:46.362 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.KFjqIb4XIL 2024-07-26T12:21:46.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T12:21:46.437 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T12:21:46.438 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T12:21:46.438 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T12:21:46.438 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710.conf 2024-07-26T12:21:46.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710 2024-07-26T12:21:46.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-07-26T12:22:16.598 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-07-26T12:22:16.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-07-26T12:22:16.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T12:22:34.975 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-07-26T12:22:43.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-07-26T12:22:43.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T12:22:47.038 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-07-26T12:22:59.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-07-26T12:22:59.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T12:23:17.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-07-26T12:23:32.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-07-26T12:23:32.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T12:23:32.894 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-07-26T12:23:33.461 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-07-26T12:23:33.461 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.10.0-g3e9255e90710"... 2024-07-26T12:23:33.462 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-07-26T12:23:33.499 INFO:teuthology.orchestra.run.smithi107.stdout:6.10.0-g3e9255e90710 2024-07-26T12:23:33.500 DEBUG:teuthology.task.kernel:current kernel version is 6.10.0-g3e9255e90710 vs 6.10.0-g3e9255e90710 2024-07-26T12:23:33.500 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T12:23:33.500 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T12:23:34.501 DEBUG:teuthology.parallel:result is None 2024-07-26T12:23:34.501 INFO:teuthology.run_tasks:Running task internal.base... 2024-07-26T12:23:34.512 INFO:teuthology.task.internal:Creating test directory... 2024-07-26T12:23:34.512 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T12:23:34.517 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T12:23:34.520 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T12:23:34.559 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-07-26T12:23:34.565 INFO:teuthology.run_tasks:Running task internal.archive... 2024-07-26T12:23:34.572 INFO:teuthology.task.internal:Creating archive directory... 2024-07-26T12:23:34.572 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T12:23:34.603 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T12:23:34.605 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T12:23:34.710 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-07-26T12:23:34.717 INFO:teuthology.task.internal:Enabling coredump saving... 2024-07-26T12:23:34.717 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T12:23:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T12:23:34.734 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T12:23:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T12:23:34.751 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T12:23:34.755 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T12:23:34.777 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T12:23:34.827 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T12:23:34.830 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T12:23:34.843 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T12:23:34.848 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T12:23:34.882 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T12:23:34.946 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T12:23:34.950 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-07-26T12:23:34.957 INFO:teuthology.task.internal:Configuring sudo... 2024-07-26T12:23:34.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T12:23:34.961 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T12:23:34.963 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T12:23:35.023 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-07-26T12:23:35.032 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-07-26T12:23:35.032 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T12:23:35.036 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T12:23:35.039 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T12:23:35.074 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T12:23:35.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T12:23:35.284 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T12:23:35.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T12:23:35.419 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:23:35.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T12:23:35.546 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T12:23:35.593 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T12:23:35.644 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T12:23:35.729 DEBUG:teuthology.orchestra.run.smithi112:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T12:23:35.769 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:23:35.769 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T12:23:35.834 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T12:23:35.881 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T12:23:35.931 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T12:23:36.015 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T12:23:36.056 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:23:36.056 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T12:23:36.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-07-26T12:23:36.126 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-07-26T12:23:36.129 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-07-26T12:23:36.167 INFO:teuthology.orchestra.run.smithi112.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T12:23:36.200 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T12:23:36.227 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T12:23:36.688 INFO:teuthology.run_tasks:Running task internal.timer... 2024-07-26T12:23:36.696 INFO:teuthology.task.internal:Starting timer... 2024-07-26T12:23:36.696 INFO:teuthology.run_tasks:Running task pcp... 2024-07-26T12:23:36.738 INFO:teuthology.run_tasks:Running task selinux... 2024-07-26T12:23:36.747 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-07-26T12:23:36.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-07-26T12:23:36.862 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-07-26T12:23:36.866 DEBUG:teuthology.orchestra.run.smithi112:> sudo service auditd rotate 2024-07-26T12:23:36.916 INFO:teuthology.orchestra.run.smithi112.stdout:Rotating logs: 2024-07-26T12:23:36.918 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-07-26T12:23:36.992 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-07-26T12:23:36.994 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-07-26T12:23:36.994 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-07-26T12:23:37.043 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-07-26T12:23:37.044 DEBUG:teuthology.orchestra.run.smithi112:> /usr/sbin/getenforce 2024-07-26T12:23:37.068 INFO:teuthology.orchestra.run.smithi112.stdout:Permissive 2024-07-26T12:23:37.068 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-07-26T12:23:37.093 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-07-26T12:23:37.093 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi112.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi132.front.sepia.ceph.com': 'permissive'} 2024-07-26T12:23:37.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T12:23:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T12:23:37.158 DEBUG:teuthology.orchestra.run.smithi112:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T12:23:37.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T12:23:37.184 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T12:23:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T12:23:37.210 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-07-26T12:23:37.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-07-26T12:23:37.321 DEBUG:teuthology.orchestra.run.smithi112:> sudo /usr/sbin/setenforce permissive 2024-07-26T12:23:37.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-07-26T12:23:37.377 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-07-26T12:23:37.388 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-07-26T12:23:37.391 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-07-26T12:23:37.495 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-07-26T12:23:37.506 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-07-26T12:23:37.507 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi112.front.sepia.ceph.com,smithi132.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-07-26T12:30:02.172 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi132.front.sepia.ceph.com')] 2024-07-26T12:30:02.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-07-26T12:30:02.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T12:30:02.284 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-07-26T12:30:02.430 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-07-26T12:30:02.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-07-26T12:30:02.431 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T12:30:02.504 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-07-26T12:30:02.582 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-07-26T12:30:02.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-07-26T12:30:02.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T12:30:02.676 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-07-26T12:30:02.759 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-07-26T12:30:02.759 INFO:teuthology.run_tasks:Running task clock... 2024-07-26T12:30:02.770 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-07-26T12:30:02.770 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T12:30:02.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T12:30:02.777 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T12:30:02.777 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T12:30:02.780 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T12:30:02.780 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T12:30:02.808 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T12:30:02.821 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T12:30:02.828 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T12:30:02.843 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T12:30:02.845 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: ntpd: command not found 2024-07-26T12:30:02.856 INFO:teuthology.orchestra.run.smithi112.stdout:506 Cannot talk to daemon 2024-07-26T12:30:02.862 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T12:30:02.866 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-07-26T12:30:02.869 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T12:30:02.877 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-07-26T12:30:02.882 INFO:teuthology.orchestra.run.smithi112.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T12:30:02.890 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T12:30:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T12:30:02.903 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T12:30:02.936 INFO:teuthology.orchestra.run.smithi112.stderr:bash: line 1: ntpq: command not found 2024-07-26T12:30:02.940 INFO:teuthology.orchestra.run.smithi112.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T12:30:02.940 INFO:teuthology.orchestra.run.smithi112.stdout:=============================================================================== 2024-07-26T12:30:02.940 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:02.940 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:02.940 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:02.940 INFO:teuthology.orchestra.run.smithi112.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:02.955 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-07-26T12:30:02.958 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T12:30:02.959 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-07-26T12:30:02.959 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:02.959 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:02.959 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:02.959 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:02.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-07-26T12:30:03.000 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-07-26T12:30:03.041 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T12:30:03.082 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T12:30:03.212 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-07-26T12:30:03.219 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T12:30:03.219 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-07-26T12:30:03.220 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:03.220 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:03.220 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:03.220 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T12:30:03.221 INFO:teuthology.run_tasks:Running task install... 2024-07-26T12:30:03.229 DEBUG:teuthology.task.install:project ceph 2024-07-26T12:30:03.229 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'}} 2024-07-26T12:30:03.229 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'} 2024-07-26T12:30:03.229 INFO:teuthology.task.install:Using flavor: default 2024-07-26T12:30:03.235 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-07-26T12:30:03.235 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-07-26T12:30:03.235 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T12:30:03.236 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:30:03.238 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T12:30:03.238 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:30:03.239 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T12:30:03.239 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:30:03.383 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T12:30:03.383 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T12:30:03.391 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T12:30:03.391 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T12:30:03.394 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T12:30:03.395 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T12:30:03.496 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T12:30:03.496 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:30:03.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T12:30:03.515 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T12:30:03.516 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:30:03.516 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T12:30:03.518 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T12:30:03.518 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:30:03.518 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T12:30:03.544 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T12:30:03.544 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T12:30:03.545 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T12:30:03.545 DEBUG:teuthology.orchestra.run.smithi112:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T12:30:03.585 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T12:30:03.585 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T12:30:03.610 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T12:30:03.611 DEBUG:teuthology.orchestra.run.smithi112:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T12:30:03.682 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T12:30:03.685 DEBUG:teuthology.orchestra.run.smithi112:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T12:30:03.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T12:30:03.745 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-07-26T12:30:03.747 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-07-26T12:30:03.748 INFO:teuthology.orchestra.run.smithi112.stdout:check_obsoletes = 1 2024-07-26T12:30:03.750 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum clean all 2024-07-26T12:30:03.816 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T12:30:03.892 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-07-26T12:30:03.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-07-26T12:30:03.970 INFO:teuthology.orchestra.run.smithi132.stdout:45 files removed 2024-07-26T12:30:03.974 INFO:teuthology.orchestra.run.smithi112.stdout:45 files removed 2024-07-26T12:30:03.997 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T12:30:04.000 DEBUG:teuthology.orchestra.run.smithi112:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T12:30:04.211 INFO:teuthology.orchestra.run.smithi107.stdout:45 files removed 2024-07-26T12:30:04.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T12:30:04.601 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 243 kB/s | 83 kB 00:00 2024-07-26T12:30:04.617 INFO:teuthology.orchestra.run.smithi112.stdout:ceph packages for x86_64 234 kB/s | 83 kB 00:00 2024-07-26T12:30:04.969 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 74 kB/s | 24 kB 00:00 2024-07-26T12:30:05.001 INFO:teuthology.orchestra.run.smithi112.stdout:ceph noarch packages 70 kB/s | 24 kB 00:00 2024-07-26T12:30:05.103 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 197 kB/s | 83 kB 00:00 2024-07-26T12:30:05.286 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-07-26T12:30:05.335 INFO:teuthology.orchestra.run.smithi112.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-07-26T12:30:05.578 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 63 kB/s | 24 kB 00:00 2024-07-26T12:30:06.037 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-07-26T12:30:06.478 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.2 MB 00:01 2024-07-26T12:30:08.285 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.2 MB 00:02 2024-07-26T12:30:08.994 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.2 MB 00:02 2024-07-26T12:30:09.737 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 20 MB 00:02 2024-07-26T12:30:11.328 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - AppStream 10 MB/s | 20 MB 00:01 2024-07-26T12:30:12.852 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 20 MB 00:02 2024-07-26T12:30:14.846 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.5 MB 00:01 2024-07-26T12:30:16.262 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.5 MB 00:01 2024-07-26T12:30:16.280 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 18 kB 00:00 2024-07-26T12:30:17.501 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 20 MB/s | 22 MB 00:01 2024-07-26T12:30:17.888 INFO:teuthology.orchestra.run.smithi112.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 18 kB 00:00 2024-07-26T12:30:18.953 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.5 MB 00:02 2024-07-26T12:30:19.109 INFO:teuthology.orchestra.run.smithi112.stdout:Extra Packages for Enterprise Linux 20 MB/s | 22 MB 00:01 2024-07-26T12:30:21.232 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 18 kB 00:00 2024-07-26T12:30:22.236 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-07-26T12:30:22.841 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-07-26T12:30:24.423 INFO:teuthology.orchestra.run.smithi112.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-07-26T12:30:25.103 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T12:30:25.103 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T12:30:25.115 INFO:teuthology.orchestra.run.smithi132.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T12:30:25.115 INFO:teuthology.orchestra.run.smithi132.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T12:30:25.116 INFO:teuthology.orchestra.run.smithi132.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T12:30:25.117 INFO:teuthology.orchestra.run.smithi132.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T12:30:25.119 INFO:teuthology.orchestra.run.smithi132.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T12:30:25.121 INFO:teuthology.orchestra.run.smithi132.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T12:30:25.122 INFO:teuthology.orchestra.run.smithi132.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T12:30:25.186 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-07-26T12:30:25.210 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-07-26T12:30:25.210 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-07-26T12:30:25.210 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-07-26T12:30:25.210 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-07-26T12:30:25.210 INFO:teuthology.orchestra.run.smithi132.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T12:30:25.210 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T12:30:25.210 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T12:30:25.210 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T12:30:25.210 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T12:30:25.211 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T12:30:25.212 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T12:30:25.213 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T12:30:25.214 INFO:teuthology.orchestra.run.smithi132.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T12:30:25.215 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T12:30:25.216 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T12:30:25.217 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T12:30:25.218 INFO:teuthology.orchestra.run.smithi132.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T12:30:25.219 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T12:30:25.220 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T12:30:25.221 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T12:30:25.222 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2024-07-26T12:30:25.223 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T12:30:25.224 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:30:25.224 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-07-26T12:30:25.224 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================= 2024-07-26T12:30:25.224 INFO:teuthology.orchestra.run.smithi132.stdout:Install 148 Packages 2024-07-26T12:30:25.224 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 12 Packages 2024-07-26T12:30:25.224 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:30:25.224 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 233 M 2024-07-26T12:30:25.224 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-07-26T12:30:25.659 INFO:teuthology.orchestra.run.smithi132.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-07-26T12:30:25.834 INFO:teuthology.orchestra.run.smithi132.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 4.9 MB/s | 872 kB 00:00 2024-07-26T12:30:25.893 INFO:teuthology.orchestra.run.smithi132.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 2.5 MB/s | 148 kB 00:00 2024-07-26T12:30:26.052 INFO:teuthology.orchestra.run.smithi132.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 9.6 MB/s | 5.5 MB 00:00 2024-07-26T12:30:26.102 INFO:teuthology.orchestra.run.smithi132.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 11 MB/s | 2.3 MB 00:00 2024-07-26T12:30:26.177 INFO:teuthology.orchestra.run.smithi132.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 12 MB/s | 1.5 MB 00:00 2024-07-26T12:30:26.486 INFO:teuthology.orchestra.run.smithi132.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 13 MB/s | 4.9 MB 00:00 2024-07-26T12:30:26.666 INFO:teuthology.orchestra.run.smithi112.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T12:30:26.666 INFO:teuthology.orchestra.run.smithi112.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T12:30:26.676 INFO:teuthology.orchestra.run.smithi112.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T12:30:26.677 INFO:teuthology.orchestra.run.smithi112.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T12:30:26.677 INFO:teuthology.orchestra.run.smithi112.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T12:30:26.678 INFO:teuthology.orchestra.run.smithi112.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T12:30:26.680 INFO:teuthology.orchestra.run.smithi112.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T12:30:26.682 INFO:teuthology.orchestra.run.smithi112.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T12:30:26.683 INFO:teuthology.orchestra.run.smithi112.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T12:30:26.745 INFO:teuthology.orchestra.run.smithi112.stdout:Dependencies resolved. 2024-07-26T12:30:26.769 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-07-26T12:30:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: Package Arch Version Repository Size 2024-07-26T12:30:26.769 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-07-26T12:30:26.769 INFO:teuthology.orchestra.run.smithi112.stdout:Installing: 2024-07-26T12:30:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T12:30:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T12:30:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T12:30:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T12:30:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T12:30:26.769 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T12:30:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T12:30:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T12:30:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T12:30:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T12:30:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T12:30:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T12:30:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T12:30:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T12:30:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T12:30:26.770 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T12:30:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T12:30:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T12:30:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T12:30:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T12:30:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T12:30:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T12:30:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T12:30:26.771 INFO:teuthology.orchestra.run.smithi112.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T12:30:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T12:30:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T12:30:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T12:30:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T12:30:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T12:30:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T12:30:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T12:30:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T12:30:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T12:30:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T12:30:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T12:30:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T12:30:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T12:30:26.773 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrading: 2024-07-26T12:30:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T12:30:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T12:30:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T12:30:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T12:30:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T12:30:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T12:30:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T12:30:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T12:30:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T12:30:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T12:30:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T12:30:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T12:30:26.775 INFO:teuthology.orchestra.run.smithi112.stdout:Installing dependencies: 2024-07-26T12:30:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T12:30:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T12:30:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T12:30:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T12:30:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T12:30:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T12:30:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T12:30:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T12:30:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T12:30:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T12:30:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T12:30:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T12:30:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T12:30:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T12:30:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T12:30:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T12:30:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T12:30:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T12:30:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T12:30:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T12:30:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T12:30:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T12:30:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T12:30:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T12:30:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T12:30:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T12:30:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T12:30:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T12:30:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T12:30:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T12:30:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T12:30:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T12:30:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T12:30:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T12:30:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T12:30:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T12:30:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T12:30:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T12:30:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T12:30:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T12:30:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T12:30:26.781 INFO:teuthology.orchestra.run.smithi112.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T12:30:26.781 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T12:30:26.781 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T12:30:26.781 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T12:30:26.781 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T12:30:26.781 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T12:30:26.781 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T12:30:26.781 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T12:30:26.782 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T12:30:26.782 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T12:30:26.782 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T12:30:26.782 INFO:teuthology.orchestra.run.smithi112.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T12:30:26.782 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T12:30:26.782 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T12:30:26.782 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T12:30:26.783 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T12:30:26.783 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T12:30:26.783 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T12:30:26.783 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T12:30:26.783 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T12:30:26.783 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T12:30:26.783 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T12:30:26.783 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T12:30:26.784 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T12:30:26.784 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T12:30:26.784 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T12:30:26.784 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T12:30:26.784 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T12:30:26.784 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T12:30:26.784 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T12:30:26.784 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T12:30:26.784 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T12:30:26.785 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T12:30:26.786 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T12:30:26.787 INFO:teuthology.orchestra.run.smithi112.stdout:Installing weak dependencies: 2024-07-26T12:30:26.788 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T12:30:26.788 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:30:26.788 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction Summary 2024-07-26T12:30:26.788 INFO:teuthology.orchestra.run.smithi112.stdout:======================================================================================= 2024-07-26T12:30:26.788 INFO:teuthology.orchestra.run.smithi112.stdout:Install 148 Packages 2024-07-26T12:30:26.788 INFO:teuthology.orchestra.run.smithi112.stdout:Upgrade 12 Packages 2024-07-26T12:30:26.788 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:30:26.789 INFO:teuthology.orchestra.run.smithi112.stdout:Total download size: 233 M 2024-07-26T12:30:26.789 INFO:teuthology.orchestra.run.smithi112.stdout:Downloading Packages: 2024-07-26T12:30:26.844 INFO:teuthology.orchestra.run.smithi132.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 15 MB/s | 20 MB 00:01 2024-07-26T12:30:26.886 INFO:teuthology.orchestra.run.smithi132.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 608 kB/s | 25 kB 00:00 2024-07-26T12:30:27.178 INFO:teuthology.orchestra.run.smithi132.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 15 MB/s | 10 MB 00:00 2024-07-26T12:30:27.374 INFO:teuthology.orchestra.run.smithi112.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-07-26T12:30:27.378 INFO:teuthology.orchestra.run.smithi132.stdout:(11/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 15 MB/s | 18 MB 00:01 2024-07-26T12:30:27.403 INFO:teuthology.orchestra.run.smithi132.stdout:(12/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 1.0 MB/s | 225 kB 00:00 2024-07-26T12:30:27.420 INFO:teuthology.orchestra.run.smithi132.stdout:(13/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 649 kB/s | 27 kB 00:00 2024-07-26T12:30:27.437 INFO:teuthology.orchestra.run.smithi132.stdout:(14/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 944 kB/s | 32 kB 00:00 2024-07-26T12:30:27.488 INFO:teuthology.orchestra.run.smithi132.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 765 kB 00:00 2024-07-26T12:30:27.513 INFO:teuthology.orchestra.run.smithi132.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.1 MB/s | 158 kB 00:00 2024-07-26T12:30:27.555 INFO:teuthology.orchestra.run.smithi132.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.9 MB/s | 127 kB 00:00 2024-07-26T12:30:27.597 INFO:teuthology.orchestra.run.smithi132.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 5.9 MB/s | 503 kB 00:00 2024-07-26T12:30:27.697 INFO:teuthology.orchestra.run.smithi132.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 450 kB/s | 45 kB 00:00 2024-07-26T12:30:27.749 INFO:teuthology.orchestra.run.smithi112.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 2.3 MB/s | 872 kB 00:00 2024-07-26T12:30:27.815 INFO:teuthology.orchestra.run.smithi132.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 1.2 MB/s | 138 kB 00:00 2024-07-26T12:30:27.850 INFO:teuthology.orchestra.run.smithi112.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 1.4 MB/s | 148 kB 00:00 2024-07-26T12:30:28.065 INFO:teuthology.orchestra.run.smithi132.stdout:(21/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 10 MB/s | 5.3 MB 00:00 2024-07-26T12:30:28.090 INFO:teuthology.orchestra.run.smithi132.stdout:(22/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 682 kB/s | 188 kB 00:00 2024-07-26T12:30:28.116 INFO:teuthology.orchestra.run.smithi132.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 7.2 MB/s | 372 kB 00:00 2024-07-26T12:30:28.141 INFO:teuthology.orchestra.run.smithi132.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 6.9 MB/s | 355 kB 00:00 2024-07-26T12:30:28.141 INFO:teuthology.orchestra.run.smithi112.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 5.8 MB/s | 5.5 MB 00:00 2024-07-26T12:30:28.166 INFO:teuthology.orchestra.run.smithi132.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 116 kB 00:00 2024-07-26T12:30:28.191 INFO:teuthology.orchestra.run.smithi132.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.7 MB/s | 86 kB 00:00 2024-07-26T12:30:28.275 INFO:teuthology.orchestra.run.smithi132.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 2.0 MB/s | 170 kB 00:00 2024-07-26T12:30:28.284 INFO:teuthology.orchestra.run.smithi112.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 5.2 MB/s | 2.3 MB 00:00 2024-07-26T12:30:28.350 INFO:teuthology.orchestra.run.smithi112.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 7.3 MB/s | 1.5 MB 00:00 2024-07-26T12:30:28.359 INFO:teuthology.orchestra.run.smithi132.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 412 kB/s | 34 kB 00:00 2024-07-26T12:30:28.426 INFO:teuthology.orchestra.run.smithi132.stdout:(29/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 12 MB/s | 3.1 MB 00:00 2024-07-26T12:30:28.451 INFO:teuthology.orchestra.run.smithi132.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 1.6 MB/s | 149 kB 00:00 2024-07-26T12:30:28.513 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-07-26T12:30:29.093 INFO:teuthology.orchestra.run.smithi132.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-07-26T12:30:29.151 INFO:teuthology.orchestra.run.smithi132.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 4.8 MB/s | 284 kB 00:00 2024-07-26T12:30:29.210 INFO:teuthology.orchestra.run.smithi132.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 846 kB/s | 49 kB 00:00 2024-07-26T12:30:29.267 INFO:teuthology.orchestra.run.smithi112.stdout:(7/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 9.6 MB/s | 20 MB 00:02 2024-07-26T12:30:29.277 INFO:teuthology.orchestra.run.smithi132.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 247 kB/s | 16 kB 00:00 2024-07-26T12:30:29.369 INFO:teuthology.orchestra.run.smithi132.stdout:(35/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 2.9 MB/s | 274 kB 00:00 2024-07-26T12:30:29.518 INFO:teuthology.orchestra.run.smithi112.stdout:(8/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 4.0 MB/s | 4.9 MB 00:01 2024-07-26T12:30:29.528 INFO:teuthology.orchestra.run.smithi132.stdout:(36/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 13 MB/s | 14 MB 00:01 2024-07-26T12:30:29.561 INFO:teuthology.orchestra.run.smithi132.stdout:(37/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 3.9 MB/s | 771 kB 00:00 2024-07-26T12:30:29.576 INFO:teuthology.orchestra.run.smithi112.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 433 kB/s | 25 kB 00:00 2024-07-26T12:30:29.586 INFO:teuthology.orchestra.run.smithi132.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 313 kB/s | 18 kB 00:00 2024-07-26T12:30:29.868 INFO:teuthology.orchestra.run.smithi112.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 17 MB/s | 10 MB 00:00 2024-07-26T12:30:29.968 INFO:teuthology.orchestra.run.smithi112.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 2.2 MB/s | 225 kB 00:00 2024-07-26T12:30:29.979 INFO:teuthology.orchestra.run.smithi132.stdout:(39/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 14 MB/s | 44 MB 00:03 2024-07-26T12:30:30.010 INFO:teuthology.orchestra.run.smithi112.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 651 kB/s | 27 kB 00:00 2024-07-26T12:30:30.060 INFO:teuthology.orchestra.run.smithi112.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 637 kB/s | 32 kB 00:00 2024-07-26T12:30:30.245 INFO:teuthology.orchestra.run.smithi112.stdout:(14/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 9.4 MB/s | 18 MB 00:01 2024-07-26T12:30:30.278 INFO:teuthology.orchestra.run.smithi112.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 3.4 MB/s | 765 kB 00:00 2024-07-26T12:30:30.303 INFO:teuthology.orchestra.run.smithi112.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.6 MB/s | 158 kB 00:00 2024-07-26T12:30:30.329 INFO:teuthology.orchestra.run.smithi112.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 2.5 MB/s | 127 kB 00:00 2024-07-26T12:30:30.362 INFO:teuthology.orchestra.run.smithi112.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 8.4 MB/s | 503 kB 00:00 2024-07-26T12:30:30.396 INFO:teuthology.orchestra.run.smithi132.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 96 kB/s | 80 kB 00:00 2024-07-26T12:30:30.404 INFO:teuthology.orchestra.run.smithi112.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 1.1 MB/s | 45 kB 00:00 2024-07-26T12:30:30.455 INFO:teuthology.orchestra.run.smithi112.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.7 MB/s | 138 kB 00:00 2024-07-26T12:30:30.513 INFO:teuthology.orchestra.run.smithi112.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.2 MB/s | 188 kB 00:00 2024-07-26T12:30:30.605 INFO:teuthology.orchestra.run.smithi112.stdout:(22/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 19 MB/s | 5.3 MB 00:00 2024-07-26T12:30:30.621 INFO:teuthology.orchestra.run.smithi132.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 332 kB/s | 195 kB 00:00 2024-07-26T12:30:30.630 INFO:teuthology.orchestra.run.smithi112.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 3.1 MB/s | 372 kB 00:00 2024-07-26T12:30:30.655 INFO:teuthology.orchestra.run.smithi132.stdout:(42/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 980 kB/s | 253 kB 00:00 2024-07-26T12:30:30.655 INFO:teuthology.orchestra.run.smithi112.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 6.9 MB/s | 355 kB 00:00 2024-07-26T12:30:30.681 INFO:teuthology.orchestra.run.smithi112.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 2.3 MB/s | 116 kB 00:00 2024-07-26T12:30:30.697 INFO:teuthology.orchestra.run.smithi132.stdout:(43/160): libgfortran-11.4.1-3.el9.x86_64.rpm 724 kB/s | 803 kB 00:01 2024-07-26T12:30:30.706 INFO:teuthology.orchestra.run.smithi112.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.7 MB/s | 86 kB 00:00 2024-07-26T12:30:30.758 INFO:teuthology.orchestra.run.smithi112.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 3.3 MB/s | 170 kB 00:00 2024-07-26T12:30:30.807 INFO:teuthology.orchestra.run.smithi112.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 690 kB/s | 34 kB 00:00 2024-07-26T12:30:30.857 INFO:teuthology.orchestra.run.smithi112.stdout:(29/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 2.9 MB/s | 149 kB 00:00 2024-07-26T12:30:30.916 INFO:teuthology.orchestra.run.smithi112.stdout:(30/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 13 MB/s | 3.1 MB 00:00 2024-07-26T12:30:30.922 INFO:teuthology.orchestra.run.smithi132.stdout:(44/160): python3-cryptography-36.0.1-4.el9.x86 4.1 MB/s | 1.2 MB 00:00 2024-07-26T12:30:30.947 INFO:teuthology.orchestra.run.smithi132.stdout:(45/160): python3-ply-3.11-14.el9.noarch.rpm 365 kB/s | 106 kB 00:00 2024-07-26T12:30:30.972 INFO:teuthology.orchestra.run.smithi132.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 491 kB/s | 135 kB 00:00 2024-07-26T12:30:31.023 INFO:teuthology.orchestra.run.smithi132.stdout:(47/160): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-07-26T12:30:31.081 INFO:teuthology.orchestra.run.smithi132.stdout:(48/160): python3-urllib3-1.26.5-6.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-07-26T12:30:31.107 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T12:30:31.108 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T12:30:31.128 INFO:teuthology.orchestra.run.smithi107.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T12:30:31.129 INFO:teuthology.orchestra.run.smithi107.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T12:30:31.130 INFO:teuthology.orchestra.run.smithi107.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T12:30:31.132 INFO:teuthology.orchestra.run.smithi107.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T12:30:31.134 INFO:teuthology.orchestra.run.smithi107.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T12:30:31.138 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T12:30:31.139 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T12:30:31.211 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-07-26T12:30:31.228 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-07-26T12:30:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-07-26T12:30:31.228 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-07-26T12:30:31.228 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-07-26T12:30:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T12:30:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T12:30:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T12:30:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T12:30:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T12:30:31.228 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T12:30:31.229 INFO:teuthology.orchestra.run.smithi107.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T12:30:31.230 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T12:30:31.231 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T12:30:31.232 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T12:30:31.233 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T12:30:31.234 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T12:30:31.235 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T12:30:31.236 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T12:30:31.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T12:30:31.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T12:30:31.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T12:30:31.240 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T12:30:31.241 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-07-26T12:30:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T12:30:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:30:31.241 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-07-26T12:30:31.241 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-07-26T12:30:31.241 INFO:teuthology.orchestra.run.smithi107.stdout:Install 148 Packages 2024-07-26T12:30:31.241 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 12 Packages 2024-07-26T12:30:31.241 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:30:31.242 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 233 M 2024-07-26T12:30:31.242 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-07-26T12:30:31.416 INFO:teuthology.orchestra.run.smithi112.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-07-26T12:30:31.506 INFO:teuthology.orchestra.run.smithi132.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 251 kB/s | 107 kB 00:00 2024-07-26T12:30:31.558 INFO:teuthology.orchestra.run.smithi112.stdout:(32/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 20 MB/s | 14 MB 00:00 2024-07-26T12:30:31.583 INFO:teuthology.orchestra.run.smithi112.stdout:(33/160): ceph-mgr-modules-core-19.0.0-5320.ge8 1.7 MB/s | 284 kB 00:00 2024-07-26T12:30:31.607 INFO:teuthology.orchestra.run.smithi132.stdout:(50/160): elfutils-libelf-devel-0.191-4.el9.x86 217 kB/s | 22 kB 00:00 2024-07-26T12:30:31.608 INFO:teuthology.orchestra.run.smithi112.stdout:(34/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 987 kB/s | 49 kB 00:00 2024-07-26T12:30:31.634 INFO:teuthology.orchestra.run.smithi112.stdout:(35/160): ceph-prometheus-alerts-19.0.0-5320.ge 329 kB/s | 16 kB 00:00 2024-07-26T12:30:31.640 INFO:teuthology.orchestra.run.smithi132.stdout:(51/160): automake-1.16.2-8.el9.noarch.rpm 1.0 MB/s | 689 kB 00:00 2024-07-26T12:30:31.659 INFO:teuthology.orchestra.run.smithi112.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 5.3 MB/s | 274 kB 00:00 2024-07-26T12:30:31.709 INFO:teuthology.orchestra.run.smithi112.stdout:(37/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 366 kB/s | 18 kB 00:00 2024-07-26T12:30:31.743 INFO:teuthology.orchestra.run.smithi112.stdout:(38/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 6.9 MB/s | 771 kB 00:00 2024-07-26T12:30:31.749 INFO:teuthology.orchestra.run.smithi132.stdout:(52/160): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-07-26T12:30:31.866 INFO:teuthology.orchestra.run.smithi132.stdout:(53/160): bison-3.7.4-5.el9.x86_64.rpm 1.1 MB/s | 943 kB 00:00 2024-07-26T12:30:31.950 INFO:teuthology.orchestra.run.smithi132.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 916 kB/s | 313 kB 00:00 2024-07-26T12:30:31.975 INFO:teuthology.orchestra.run.smithi132.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 168 kB/s | 18 kB 00:00 2024-07-26T12:30:32.050 INFO:teuthology.orchestra.run.smithi132.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-07-26T12:30:32.072 INFO:teuthology.orchestra.run.smithi107.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-07-26T12:30:32.084 INFO:teuthology.orchestra.run.smithi132.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-07-26T12:30:32.151 INFO:teuthology.orchestra.run.smithi132.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 452 kB/s | 45 kB 00:00 2024-07-26T12:30:32.160 INFO:teuthology.orchestra.run.smithi112.stdout:(39/160): fuse-2.9.9-16.el9.x86_64.rpm 177 kB/s | 80 kB 00:00 2024-07-26T12:30:32.226 INFO:teuthology.orchestra.run.smithi132.stdout:(59/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.3 MB/s | 3.0 MB 00:00 2024-07-26T12:30:32.273 INFO:teuthology.orchestra.run.smithi107.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 4.3 MB/s | 872 kB 00:00 2024-07-26T12:30:32.276 INFO:teuthology.orchestra.run.smithi132.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-07-26T12:30:32.327 INFO:teuthology.orchestra.run.smithi132.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 253 kB/s | 25 kB 00:00 2024-07-26T12:30:32.339 INFO:teuthology.orchestra.run.smithi107.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 2.2 MB/s | 148 kB 00:00 2024-07-26T12:30:32.377 INFO:teuthology.orchestra.run.smithi132.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 473 kB/s | 47 kB 00:00 2024-07-26T12:30:32.427 INFO:teuthology.orchestra.run.smithi112.stdout:(40/160): libquadmath-11.4.1-3.el9.x86_64.rpm 733 kB/s | 195 kB 00:00 2024-07-26T12:30:32.435 INFO:teuthology.orchestra.run.smithi132.stdout:(63/160): libtool-2.4.6-46.el9.x86_64.rpm 2.0 MB/s | 581 kB 00:00 2024-07-26T12:30:32.461 INFO:teuthology.orchestra.run.smithi132.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-07-26T12:30:32.486 INFO:teuthology.orchestra.run.smithi132.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-07-26T12:30:32.494 INFO:teuthology.orchestra.run.smithi112.stdout:(41/160): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-07-26T12:30:32.506 INFO:teuthology.orchestra.run.smithi107.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 8.0 MB/s | 5.5 MB 00:00 2024-07-26T12:30:32.537 INFO:teuthology.orchestra.run.smithi132.stdout:(66/160): openblas-0.3.21-2.el9.x86_64.rpm 350 kB/s | 35 kB 00:00 2024-07-26T12:30:32.590 INFO:teuthology.orchestra.run.smithi107.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 9.1 MB/s | 2.3 MB 00:00 2024-07-26T12:30:32.619 INFO:teuthology.orchestra.run.smithi112.stdout:(42/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-07-26T12:30:32.665 INFO:teuthology.orchestra.run.smithi107.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 9.6 MB/s | 1.5 MB 00:00 2024-07-26T12:30:32.728 INFO:teuthology.orchestra.run.smithi112.stdout:(43/160): python3-ply-3.11-14.el9.noarch.rpm 984 kB/s | 106 kB 00:00 2024-07-26T12:30:32.803 INFO:teuthology.orchestra.run.smithi112.stdout:(44/160): python3-cryptography-36.0.1-4.el9.x86 4.0 MB/s | 1.2 MB 00:00 2024-07-26T12:30:32.820 INFO:teuthology.orchestra.run.smithi132.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-07-26T12:30:32.829 INFO:teuthology.orchestra.run.smithi112.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-07-26T12:30:32.912 INFO:teuthology.orchestra.run.smithi112.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-07-26T12:30:32.938 INFO:teuthology.orchestra.run.smithi112.stdout:(47/160): python3-urllib3-1.26.5-6.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-07-26T12:30:32.941 INFO:teuthology.orchestra.run.smithi107.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 14 MB/s | 4.9 MB 00:00 2024-07-26T12:30:32.970 INFO:teuthology.orchestra.run.smithi132.stdout:(68/160): openblas-openmp-0.3.21-2.el9.x86_64.r 9.3 MB/s | 4.7 MB 00:00 2024-07-26T12:30:33.013 INFO:teuthology.orchestra.run.smithi132.stdout:(69/160): postgresql-contrib-13.14-1.el9.x86_64 4.6 MB/s | 911 kB 00:00 2024-07-26T12:30:33.079 INFO:teuthology.orchestra.run.smithi132.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-07-26T12:30:33.305 INFO:teuthology.orchestra.run.smithi112.stdout:(48/160): automake-1.16.2-8.el9.noarch.rpm 1.7 MB/s | 689 kB 00:00 2024-07-26T12:30:33.324 INFO:teuthology.orchestra.run.smithi107.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 13 MB/s | 20 MB 00:01 2024-07-26T12:30:33.374 INFO:teuthology.orchestra.run.smithi107.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 515 kB/s | 25 kB 00:00 2024-07-26T12:30:33.405 INFO:teuthology.orchestra.run.smithi112.stdout:(49/160): bison-3.7.4-5.el9.x86_64.rpm 2.0 MB/s | 943 kB 00:00 2024-07-26T12:30:33.421 INFO:teuthology.orchestra.run.smithi132.stdout:(71/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-07-26T12:30:33.564 INFO:teuthology.orchestra.run.smithi132.stdout:(72/160): protobuf-3.14.0-13.el9.x86_64.rpm 2.1 MB/s | 1.0 MB 00:00 2024-07-26T12:30:33.605 INFO:teuthology.orchestra.run.smithi112.stdout:(50/160): boost-program-options-1.75.0-8.el9.x8 356 kB/s | 107 kB 00:00 2024-07-26T12:30:33.630 INFO:teuthology.orchestra.run.smithi112.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 96 kB/s | 22 kB 00:00 2024-07-26T12:30:33.647 INFO:teuthology.orchestra.run.smithi132.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 389 kB/s | 87 kB 00:00 2024-07-26T12:30:33.739 INFO:teuthology.orchestra.run.smithi132.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 668 kB/s | 61 kB 00:00 2024-07-26T12:30:33.750 INFO:teuthology.orchestra.run.smithi107.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 13 MB/s | 10 MB 00:00 2024-07-26T12:30:33.881 INFO:teuthology.orchestra.run.smithi112.stdout:(52/160): flexiblas-3.0.4-8.el9.x86_64.rpm 131 kB/s | 33 kB 00:00 2024-07-26T12:30:33.983 INFO:teuthology.orchestra.run.smithi107.stdout:(11/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 13 MB/s | 18 MB 00:01 2024-07-26T12:30:33.989 INFO:teuthology.orchestra.run.smithi112.stdout:(53/160): flex-2.6.4-9.el9.x86_64.rpm 817 kB/s | 313 kB 00:00 2024-07-26T12:30:34.017 INFO:teuthology.orchestra.run.smithi107.stdout:(12/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 847 kB/s | 225 kB 00:00 2024-07-26T12:30:34.042 INFO:teuthology.orchestra.run.smithi107.stdout:(13/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 468 kB/s | 27 kB 00:00 2024-07-26T12:30:34.068 INFO:teuthology.orchestra.run.smithi107.stdout:(14/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 643 kB/s | 32 kB 00:00 2024-07-26T12:30:34.089 INFO:teuthology.orchestra.run.smithi132.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 697 kB/s | 244 kB 00:00 2024-07-26T12:30:34.117 INFO:teuthology.orchestra.run.smithi107.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 10 MB/s | 765 kB 00:00 2024-07-26T12:30:34.151 INFO:teuthology.orchestra.run.smithi107.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 1.9 MB/s | 158 kB 00:00 2024-07-26T12:30:34.185 INFO:teuthology.orchestra.run.smithi107.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.9 MB/s | 127 kB 00:00 2024-07-26T12:30:34.235 INFO:teuthology.orchestra.run.smithi107.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 5.9 MB/s | 503 kB 00:00 2024-07-26T12:30:34.302 INFO:teuthology.orchestra.run.smithi107.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 687 kB/s | 45 kB 00:00 2024-07-26T12:30:34.369 INFO:teuthology.orchestra.run.smithi107.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.1 MB/s | 138 kB 00:00 2024-07-26T12:30:34.428 INFO:teuthology.orchestra.run.smithi107.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.2 MB/s | 188 kB 00:00 2024-07-26T12:30:34.431 INFO:teuthology.orchestra.run.smithi132.stdout:(76/160): python3-jinja2-2.11.3-6.el9.noarch.rp 726 kB/s | 248 kB 00:00 2024-07-26T12:30:34.544 INFO:teuthology.orchestra.run.smithi132.stdout:(77/160): python3-jsonpatch-1.21-16.el9.noarch. 227 kB/s | 26 kB 00:00 2024-07-26T12:30:34.545 INFO:teuthology.orchestra.run.smithi107.stdout:(22/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 15 MB/s | 5.3 MB 00:00 2024-07-26T12:30:34.548 INFO:teuthology.orchestra.run.smithi112.stdout:(54/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 8.9 MB/s | 44 MB 00:04 2024-07-26T12:30:34.573 INFO:teuthology.orchestra.run.smithi112.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 31 kB/s | 18 kB 00:00 2024-07-26T12:30:34.587 INFO:teuthology.orchestra.run.smithi107.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 2.3 MB/s | 372 kB 00:00 2024-07-26T12:30:34.629 INFO:teuthology.orchestra.run.smithi107.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 4.2 MB/s | 355 kB 00:00 2024-07-26T12:30:34.651 INFO:teuthology.orchestra.run.smithi132.stdout:(78/160): python3-jsonpointer-2.0-4.el9.noarch. 180 kB/s | 19 kB 00:00 2024-07-26T12:30:34.662 INFO:teuthology.orchestra.run.smithi107.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.5 MB/s | 116 kB 00:00 2024-07-26T12:30:34.688 INFO:teuthology.orchestra.run.smithi107.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.5 MB/s | 86 kB 00:00 2024-07-26T12:30:34.747 INFO:teuthology.orchestra.run.smithi107.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 2.9 MB/s | 170 kB 00:00 2024-07-26T12:30:34.749 INFO:teuthology.orchestra.run.smithi112.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 56 kB/s | 11 kB 00:00 2024-07-26T12:30:34.789 INFO:teuthology.orchestra.run.smithi107.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 837 kB/s | 34 kB 00:00 2024-07-26T12:30:34.824 INFO:teuthology.orchestra.run.smithi112.stdout:(57/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.2 MB/s | 3.0 MB 00:00 2024-07-26T12:30:34.889 INFO:teuthology.orchestra.run.smithi107.stdout:(29/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 14 MB/s | 3.1 MB 00:00 2024-07-26T12:30:34.918 INFO:teuthology.orchestra.run.smithi132.stdout:(79/160): python3-mako-1.1.4-6.el9.noarch.rpm 646 kB/s | 172 kB 00:00 2024-07-26T12:30:34.948 INFO:teuthology.orchestra.run.smithi107.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 947 kB/s | 149 kB 00:00 2024-07-26T12:30:34.949 INFO:teuthology.orchestra.run.smithi112.stdout:(58/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 427 kB/s | 160 kB 00:00 2024-07-26T12:30:35.027 INFO:teuthology.orchestra.run.smithi132.stdout:(80/160): python3-markupsafe-1.1.1-12.el9.x86_6 321 kB/s | 35 kB 00:00 2024-07-26T12:30:35.066 INFO:teuthology.orchestra.run.smithi112.stdout:(59/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 143 kB/s | 45 kB 00:00 2024-07-26T12:30:35.291 INFO:teuthology.orchestra.run.smithi112.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 1.5 MB/s | 662 kB 00:00 2024-07-26T12:30:35.317 INFO:teuthology.orchestra.run.smithi112.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 102 kB/s | 25 kB 00:00 2024-07-26T12:30:35.350 INFO:teuthology.orchestra.run.smithi112.stdout:(62/160): libtool-2.4.6-46.el9.x86_64.rpm 1.4 MB/s | 581 kB 00:00 2024-07-26T12:30:35.431 INFO:teuthology.orchestra.run.smithi107.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-07-26T12:30:35.507 INFO:teuthology.orchestra.run.smithi107.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 3.8 MB/s | 284 kB 00:00 2024-07-26T12:30:35.565 INFO:teuthology.orchestra.run.smithi107.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 859 kB/s | 49 kB 00:00 2024-07-26T12:30:35.592 INFO:teuthology.orchestra.run.smithi112.stdout:(63/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 158 kB/s | 47 kB 00:00 2024-07-26T12:30:35.615 INFO:teuthology.orchestra.run.smithi107.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 333 kB/s | 16 kB 00:00 2024-07-26T12:30:35.743 INFO:teuthology.orchestra.run.smithi112.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 687 kB/s | 292 kB 00:00 2024-07-26T12:30:35.768 INFO:teuthology.orchestra.run.smithi112.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 511 kB/s | 188 kB 00:00 2024-07-26T12:30:35.808 INFO:teuthology.orchestra.run.smithi107.stdout:(35/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 15 MB/s | 14 MB 00:00 2024-07-26T12:30:35.849 INFO:teuthology.orchestra.run.smithi107.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 1.2 MB/s | 274 kB 00:00 2024-07-26T12:30:35.868 INFO:teuthology.orchestra.run.smithi112.stdout:(66/160): openblas-0.3.21-2.el9.x86_64.rpm 128 kB/s | 35 kB 00:00 2024-07-26T12:30:35.908 INFO:teuthology.orchestra.run.smithi107.stdout:(37/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 7.6 MB/s | 771 kB 00:00 2024-07-26T12:30:35.933 INFO:teuthology.orchestra.run.smithi107.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 220 kB/s | 18 kB 00:00 2024-07-26T12:30:36.327 INFO:teuthology.orchestra.run.smithi112.stdout:(67/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2024-07-26T12:30:36.368 INFO:teuthology.orchestra.run.smithi112.stdout:(68/160): postgresql-13.14-1.el9.x86_64.rpm 3.2 MB/s | 1.6 MB 00:00 2024-07-26T12:30:36.452 INFO:teuthology.orchestra.run.smithi112.stdout:(69/160): openblas-openmp-0.3.21-2.el9.x86_64.r 6.7 MB/s | 4.7 MB 00:00 2024-07-26T12:30:36.700 INFO:teuthology.orchestra.run.smithi107.stdout:(39/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 13 MB/s | 44 MB 00:03 2024-07-26T12:30:36.736 INFO:teuthology.orchestra.run.smithi112.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 388 kB/s | 133 kB 00:00 2024-07-26T12:30:36.778 INFO:teuthology.orchestra.run.smithi112.stdout:(71/160): postgresql-contrib-13.14-1.el9.x86_64 2.0 MB/s | 911 kB 00:00 2024-07-26T12:30:37.178 INFO:teuthology.orchestra.run.smithi112.stdout:(72/160): postgresql-server-13.14-1.el9.x86_64. 8.2 MB/s | 5.8 MB 00:00 2024-07-26T12:30:37.203 INFO:teuthology.orchestra.run.smithi112.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 218 kB/s | 87 kB 00:00 2024-07-26T12:30:37.279 INFO:teuthology.orchestra.run.smithi112.stdout:(74/160): protobuf-3.14.0-13.el9.x86_64.rpm 1.9 MB/s | 1.0 MB 00:00 2024-07-26T12:30:37.324 INFO:teuthology.orchestra.run.smithi107.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 56 kB/s | 80 kB 00:01 2024-07-26T12:30:37.529 INFO:teuthology.orchestra.run.smithi112.stdout:(75/160): python3-colorama-0.4.6-3.el9.noarch.r 188 kB/s | 61 kB 00:00 2024-07-26T12:30:37.576 INFO:teuthology.orchestra.run.smithi107.stdout:(41/160): libgfortran-11.4.1-3.el9.x86_64.rpm 489 kB/s | 803 kB 00:01 2024-07-26T12:30:37.679 INFO:teuthology.orchestra.run.smithi112.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 649 kB/s | 244 kB 00:00 2024-07-26T12:30:37.826 INFO:teuthology.orchestra.run.smithi107.stdout:(42/160): libquadmath-11.4.1-3.el9.x86_64.rpm 174 kB/s | 195 kB 00:01 2024-07-26T12:30:37.830 INFO:teuthology.orchestra.run.smithi112.stdout:(77/160): python3-babel-2.9.1-2.el9.noarch.rpm 9.2 MB/s | 6.0 MB 00:00 2024-07-26T12:30:37.930 INFO:teuthology.orchestra.run.smithi112.stdout:(78/160): python3-jinja2-2.11.3-6.el9.noarch.rp 620 kB/s | 248 kB 00:00 2024-07-26T12:30:37.964 INFO:teuthology.orchestra.run.smithi112.stdout:(79/160): python3-jsonpatch-1.21-16.el9.noarch. 90 kB/s | 26 kB 00:00 2024-07-26T12:30:38.089 INFO:teuthology.orchestra.run.smithi112.stdout:(80/160): python3-jsonpointer-2.0-4.el9.noarch. 74 kB/s | 19 kB 00:00 2024-07-26T12:30:38.160 INFO:teuthology.orchestra.run.smithi132.stdout:(81/160): python3-babel-2.9.1-2.el9.noarch.rpm 1.3 MB/s | 6.0 MB 00:04 2024-07-26T12:30:38.231 INFO:teuthology.orchestra.run.smithi112.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 131 kB/s | 35 kB 00:00 2024-07-26T12:30:38.256 INFO:teuthology.orchestra.run.smithi112.stdout:(82/160): python3-mako-1.1.4-6.el9.noarch.rpm 529 kB/s | 172 kB 00:00 2024-07-26T12:30:38.285 INFO:teuthology.orchestra.run.smithi107.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 265 kB/s | 253 kB 00:00 2024-07-26T12:30:38.543 INFO:teuthology.orchestra.run.smithi132.stdout:(82/160): postgresql-server-13.14-1.el9.x86_64. 1.0 MB/s | 5.8 MB 00:05 2024-07-26T12:30:38.568 INFO:teuthology.orchestra.run.smithi132.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 614 kB/s | 250 kB 00:00 2024-07-26T12:30:38.606 INFO:teuthology.orchestra.run.smithi112.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 668 kB/s | 250 kB 00:00 2024-07-26T12:30:38.631 INFO:teuthology.orchestra.run.smithi112.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 592 kB/s | 222 kB 00:00 2024-07-26T12:30:38.651 INFO:teuthology.orchestra.run.smithi107.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 129 kB/s | 106 kB 00:00 2024-07-26T12:30:38.707 INFO:teuthology.orchestra.run.smithi112.stdout:(85/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.3 MB/s | 5.1 MB 00:00 2024-07-26T12:30:38.752 INFO:teuthology.orchestra.run.smithi132.stdout:(84/160): python3-packaging-20.9-5.el9.noarch.r 421 kB/s | 77 kB 00:00 2024-07-26T12:30:38.835 INFO:teuthology.orchestra.run.smithi107.stdout:(45/160): python3-cryptography-36.0.1-4.el9.x86 1.0 MB/s | 1.2 MB 00:01 2024-07-26T12:30:38.860 INFO:teuthology.orchestra.run.smithi132.stdout:(85/160): python3-prettytable-0.7.2-27.el9.noar 390 kB/s | 42 kB 00:00 2024-07-26T12:30:38.911 INFO:teuthology.orchestra.run.smithi132.stdout:(86/160): python3-oauthlib-3.1.1-5.el9.noarch.r 605 kB/s | 222 kB 00:00 2024-07-26T12:30:38.924 INFO:teuthology.orchestra.run.smithi112.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 243 kB/s | 77 kB 00:00 2024-07-26T12:30:38.965 INFO:teuthology.orchestra.run.smithi112.stdout:(87/160): python3-prettytable-0.7.2-27.el9.noar 127 kB/s | 42 kB 00:00 2024-07-26T12:30:39.116 INFO:teuthology.orchestra.run.smithi112.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 748 kB/s | 269 kB 00:00 2024-07-26T12:30:39.135 INFO:teuthology.orchestra.run.smithi107.stdout:(46/160): python3-pycparser-2.20-6.el9.noarch.r 159 kB/s | 135 kB 00:00 2024-07-26T12:30:39.170 INFO:teuthology.orchestra.run.smithi132.stdout:(87/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 615 kB/s | 159 kB 00:00 2024-07-26T12:30:39.283 INFO:teuthology.orchestra.run.smithi112.stdout:(89/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 443 kB/s | 159 kB 00:00 2024-07-26T12:30:39.303 INFO:teuthology.orchestra.run.smithi132.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 609 kB/s | 269 kB 00:00 2024-07-26T12:30:39.375 INFO:teuthology.orchestra.run.smithi112.stdout:(90/160): python3-pytz-2021.1-5.el9.noarch.rpm 197 kB/s | 51 kB 00:00 2024-07-26T12:30:39.400 INFO:teuthology.orchestra.run.smithi112.stdout:(91/160): python3-pyasn1-modules-0.4.8-6.el9.no 680 kB/s | 279 kB 00:00 2024-07-26T12:30:39.412 INFO:teuthology.orchestra.run.smithi132.stdout:(89/160): python3-pytz-2021.1-5.el9.noarch.rpm 470 kB/s | 51 kB 00:00 2024-07-26T12:30:39.519 INFO:teuthology.orchestra.run.smithi107.stdout:(47/160): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 689 kB 00:00 2024-07-26T12:30:39.520 INFO:teuthology.orchestra.run.smithi132.stdout:(90/160): python3-requests-oauthlib-1.3.0-12.el 496 kB/s | 54 kB 00:00 2024-07-26T12:30:39.575 INFO:teuthology.orchestra.run.smithi112.stdout:(92/160): python3-requests-oauthlib-1.3.0-12.el 184 kB/s | 54 kB 00:00 2024-07-26T12:30:39.577 INFO:teuthology.orchestra.run.smithi107.stdout:(48/160): python3-requests-2.25.1-8.el9.noarch. 135 kB/s | 125 kB 00:00 2024-07-26T12:30:39.595 INFO:teuthology.orchestra.run.smithi132.stdout:(91/160): python3-pyasn1-modules-0.4.8-6.el9.no 654 kB/s | 279 kB 00:00 2024-07-26T12:30:39.684 INFO:teuthology.orchestra.run.smithi112.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 147 kB/s | 42 kB 00:00 2024-07-26T12:30:39.729 INFO:teuthology.orchestra.run.smithi132.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 312 kB/s | 42 kB 00:00 2024-07-26T12:30:39.753 INFO:teuthology.orchestra.run.smithi107.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 616 kB/s | 107 kB 00:00 2024-07-26T12:30:39.795 INFO:teuthology.orchestra.run.smithi107.stdout:(50/160): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 943 kB 00:00 2024-07-26T12:30:39.820 INFO:teuthology.orchestra.run.smithi107.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 328 kB/s | 22 kB 00:00 2024-07-26T12:30:39.826 INFO:teuthology.orchestra.run.smithi112.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 176 kB/s | 44 kB 00:00 2024-07-26T12:30:39.829 INFO:teuthology.orchestra.run.smithi132.stdout:(93/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 442 kB/s | 44 kB 00:00 2024-07-26T12:30:39.854 INFO:teuthology.orchestra.run.smithi107.stdout:(52/160): python3-urllib3-1.26.5-6.el9.noarch.r 211 kB/s | 215 kB 00:01 2024-07-26T12:30:39.879 INFO:teuthology.orchestra.run.smithi107.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 584 kB/s | 33 kB 00:00 2024-07-26T12:30:39.938 INFO:teuthology.orchestra.run.smithi107.stdout:(54/160): flexiblas-openblas-openmp-3.0.4-8.el9 316 kB/s | 18 kB 00:00 2024-07-26T12:30:39.971 INFO:teuthology.orchestra.run.smithi107.stdout:(55/160): flex-2.6.4-9.el9.x86_64.rpm 1.7 MB/s | 313 kB 00:00 2024-07-26T12:30:39.988 INFO:teuthology.orchestra.run.smithi107.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 225 kB/s | 11 kB 00:00 2024-07-26T12:30:40.035 INFO:teuthology.orchestra.run.smithi112.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 632 kB/s | 221 kB 00:00 2024-07-26T12:30:40.039 INFO:teuthology.orchestra.run.smithi107.stdout:(57/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 916 kB/s | 45 kB 00:00 2024-07-26T12:30:40.060 INFO:teuthology.orchestra.run.smithi112.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 154 kB/s | 36 kB 00:00 2024-07-26T12:30:40.089 INFO:teuthology.orchestra.run.smithi107.stdout:(58/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 160 kB 00:00 2024-07-26T12:30:40.180 INFO:teuthology.orchestra.run.smithi132.stdout:(94/160): qatlib-24.02.0-1.el9.x86_64.rpm 630 kB/s | 221 kB 00:00 2024-07-26T12:30:40.197 INFO:teuthology.orchestra.run.smithi107.stdout:(59/160): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-07-26T12:30:40.256 INFO:teuthology.orchestra.run.smithi107.stdout:(60/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 440 kB/s | 25 kB 00:00 2024-07-26T12:30:40.288 INFO:teuthology.orchestra.run.smithi132.stdout:(95/160): qatlib-service-24.02.0-1.el9.x86_64.r 332 kB/s | 36 kB 00:00 2024-07-26T12:30:40.306 INFO:teuthology.orchestra.run.smithi107.stdout:(61/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 963 kB/s | 47 kB 00:00 2024-07-26T12:30:40.360 INFO:teuthology.orchestra.run.smithi112.stdout:(97/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-07-26T12:30:40.373 INFO:teuthology.orchestra.run.smithi107.stdout:(62/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-07-26T12:30:40.386 INFO:teuthology.orchestra.run.smithi112.stdout:(98/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 135 kB/s | 47 kB 00:00 2024-07-26T12:30:40.397 INFO:teuthology.orchestra.run.smithi132.stdout:(96/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 438 kB/s | 47 kB 00:00 2024-07-26T12:30:40.441 INFO:teuthology.orchestra.run.smithi107.stdout:(63/160): libtool-2.4.6-46.el9.x86_64.rpm 1.6 MB/s | 581 kB 00:00 2024-07-26T12:30:40.474 INFO:teuthology.orchestra.run.smithi107.stdout:(64/160): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-07-26T12:30:40.499 INFO:teuthology.orchestra.run.smithi107.stdout:(65/160): openblas-0.3.21-2.el9.x86_64.rpm 610 kB/s | 35 kB 00:00 2024-07-26T12:30:40.511 INFO:teuthology.orchestra.run.smithi112.stdout:(99/160): socat-1.7.4.1-6.el9.x86_64.rpm 675 kB/s | 304 kB 00:00 2024-07-26T12:30:40.611 INFO:teuthology.orchestra.run.smithi112.stdout:(100/160): uuid-1.6.2-55.el9.x86_64.rpm 231 kB/s | 58 kB 00:00 2024-07-26T12:30:40.703 INFO:teuthology.orchestra.run.smithi112.stdout:(101/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 301 kB/s | 96 kB 00:00 2024-07-26T12:30:40.756 INFO:teuthology.orchestra.run.smithi132.stdout:(97/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 911 kB/s | 5.1 MB 00:05 2024-07-26T12:30:40.795 INFO:teuthology.orchestra.run.smithi112.stdout:(102/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 225 kB/s | 64 kB 00:00 2024-07-26T12:30:40.856 INFO:teuthology.orchestra.run.smithi132.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 663 kB/s | 304 kB 00:00 2024-07-26T12:30:40.870 INFO:teuthology.orchestra.run.smithi112.stdout:(103/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 173 kB/s | 45 kB 00:00 2024-07-26T12:30:40.881 INFO:teuthology.orchestra.run.smithi132.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 462 kB/s | 58 kB 00:00 2024-07-26T12:30:40.916 INFO:teuthology.orchestra.run.smithi107.stdout:(66/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.8 MB/s | 3.0 MB 00:01 2024-07-26T12:30:40.970 INFO:teuthology.orchestra.run.smithi112.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 5.4 MB/s | 548 kB 00:00 2024-07-26T12:30:40.981 INFO:teuthology.orchestra.run.smithi132.stdout:(100/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 637 kB/s | 64 kB 00:00 2024-07-26T12:30:41.004 INFO:teuthology.orchestra.run.smithi112.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 9.1 MB/s | 309 kB 00:00 2024-07-26T12:30:41.029 INFO:teuthology.orchestra.run.smithi112.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 854 kB/s | 19 kB 00:00 2024-07-26T12:30:41.040 INFO:teuthology.orchestra.run.smithi132.stdout:(101/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 520 kB/s | 96 kB 00:00 2024-07-26T12:30:41.082 INFO:teuthology.orchestra.run.smithi132.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 447 kB/s | 45 kB 00:00 2024-07-26T12:30:41.088 INFO:teuthology.orchestra.run.smithi112.stdout:(107/160): lua-devel-5.4.4-4.el9.x86_64.rpm 58 kB/s | 22 kB 00:00 2024-07-26T12:30:41.116 INFO:teuthology.orchestra.run.smithi132.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-07-26T12:30:41.147 INFO:teuthology.orchestra.run.smithi112.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-07-26T12:30:41.180 INFO:teuthology.orchestra.run.smithi112.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-07-26T12:30:41.206 INFO:teuthology.orchestra.run.smithi112.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-07-26T12:30:41.208 INFO:teuthology.orchestra.run.smithi132.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 5.8 MB/s | 548 kB 00:00 2024-07-26T12:30:41.231 INFO:teuthology.orchestra.run.smithi112.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 5.9 MB/s | 151 kB 00:00 2024-07-26T12:30:41.250 INFO:teuthology.orchestra.run.smithi132.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 7.3 MB/s | 309 kB 00:00 2024-07-26T12:30:41.275 INFO:teuthology.orchestra.run.smithi132.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 782 kB/s | 19 kB 00:00 2024-07-26T12:30:41.306 INFO:teuthology.orchestra.run.smithi112.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-07-26T12:30:41.317 INFO:teuthology.orchestra.run.smithi132.stdout:(107/160): protobuf-compiler-3.14.0-13.el9.x86_ 3.6 MB/s | 863 kB 00:00 2024-07-26T12:30:41.381 INFO:teuthology.orchestra.run.smithi112.stdout:(113/160): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-07-26T12:30:41.384 INFO:teuthology.orchestra.run.smithi132.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-07-26T12:30:41.407 INFO:teuthology.orchestra.run.smithi112.stdout:(114/160): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-07-26T12:30:41.409 INFO:teuthology.orchestra.run.smithi132.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-07-26T12:30:41.432 INFO:teuthology.orchestra.run.smithi112.stdout:(115/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 863 kB/s | 43 kB 00:00 2024-07-26T12:30:41.443 INFO:teuthology.orchestra.run.smithi132.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-07-26T12:30:41.457 INFO:teuthology.orchestra.run.smithi112.stdout:(116/160): python3-cachetools-4.2.4-1.el9.noarc 641 kB/s | 32 kB 00:00 2024-07-26T12:30:41.476 INFO:teuthology.orchestra.run.smithi132.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-07-26T12:30:41.482 INFO:teuthology.orchestra.run.smithi112.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-07-26T12:30:41.508 INFO:teuthology.orchestra.run.smithi112.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-07-26T12:30:41.542 INFO:teuthology.orchestra.run.smithi112.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-07-26T12:30:41.567 INFO:teuthology.orchestra.run.smithi112.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-07-26T12:30:41.568 INFO:teuthology.orchestra.run.smithi132.stdout:(112/160): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-07-26T12:30:41.592 INFO:teuthology.orchestra.run.smithi112.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 4.2 MB/s | 216 kB 00:00 2024-07-26T12:30:41.593 INFO:teuthology.orchestra.run.smithi132.stdout:(113/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-07-26T12:30:41.617 INFO:teuthology.orchestra.run.smithi112.stdout:(122/160): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-07-26T12:30:41.627 INFO:teuthology.orchestra.run.smithi132.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-07-26T12:30:41.653 INFO:teuthology.orchestra.run.smithi132.stdout:(115/160): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-07-26T12:30:41.668 INFO:teuthology.orchestra.run.smithi112.stdout:(123/160): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-07-26T12:30:41.678 INFO:teuthology.orchestra.run.smithi132.stdout:(116/160): python3-cachetools-4.2.4-1.el9.noarc 638 kB/s | 32 kB 00:00 2024-07-26T12:30:41.701 INFO:teuthology.orchestra.run.smithi112.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 574 kB/s | 18 kB 00:00 2024-07-26T12:30:41.703 INFO:teuthology.orchestra.run.smithi132.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-07-26T12:30:41.726 INFO:teuthology.orchestra.run.smithi112.stdout:(125/160): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-07-26T12:30:41.728 INFO:teuthology.orchestra.run.smithi132.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-07-26T12:30:41.752 INFO:teuthology.orchestra.run.smithi112.stdout:(126/160): python3-jaraco-collections-3.0.0-8.e 457 kB/s | 23 kB 00:00 2024-07-26T12:30:41.754 INFO:teuthology.orchestra.run.smithi132.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-07-26T12:30:41.775 INFO:teuthology.orchestra.run.smithi107.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-07-26T12:30:41.777 INFO:teuthology.orchestra.run.smithi112.stdout:(127/160): python3-jaraco-functools-3.5.0-2.el9 384 kB/s | 19 kB 00:00 2024-07-26T12:30:41.787 INFO:teuthology.orchestra.run.smithi132.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-07-26T12:30:41.803 INFO:teuthology.orchestra.run.smithi112.stdout:(128/160): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-07-26T12:30:41.812 INFO:teuthology.orchestra.run.smithi132.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 3.6 MB/s | 216 kB 00:00 2024-07-26T12:30:41.828 INFO:teuthology.orchestra.run.smithi112.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T12:30:41.846 INFO:teuthology.orchestra.run.smithi132.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-07-26T12:30:41.853 INFO:teuthology.orchestra.run.smithi112.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-07-26T12:30:41.871 INFO:teuthology.orchestra.run.smithi132.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 427 kB/s | 11 kB 00:00 2024-07-26T12:30:41.895 INFO:teuthology.orchestra.run.smithi112.stdout:(131/160): python3-grpcio-1.46.7-10.el9.x86_64. 6.2 MB/s | 2.0 MB 00:00 2024-07-26T12:30:41.897 INFO:teuthology.orchestra.run.smithi132.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 711 kB/s | 18 kB 00:00 2024-07-26T12:30:41.920 INFO:teuthology.orchestra.run.smithi112.stdout:(132/160): python3-natsort-7.1.1-5.el9.noarch.r 861 kB/s | 58 kB 00:00 2024-07-26T12:30:41.922 INFO:teuthology.orchestra.run.smithi132.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 935 kB/s | 23 kB 00:00 2024-07-26T12:30:41.939 INFO:teuthology.orchestra.run.smithi132.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 1.1 MB/s | 19 kB 00:00 2024-07-26T12:30:41.954 INFO:teuthology.orchestra.run.smithi112.stdout:(133/160): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-07-26T12:30:41.956 INFO:teuthology.orchestra.run.smithi132.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 1.2 MB/s | 20 kB 00:00 2024-07-26T12:30:41.971 INFO:teuthology.orchestra.run.smithi112.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-07-26T12:30:41.997 INFO:teuthology.orchestra.run.smithi112.stdout:(135/160): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-07-26T12:30:41.998 INFO:teuthology.orchestra.run.smithi132.stdout:(128/160): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-07-26T12:30:42.015 INFO:teuthology.orchestra.run.smithi132.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 2.7 MB/s | 46 kB 00:00 2024-07-26T12:30:42.022 INFO:teuthology.orchestra.run.smithi112.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-07-26T12:30:42.039 INFO:teuthology.orchestra.run.smithi112.stdout:(137/160): python3-pyperclip-1.8.2-2.el9.noarch 420 kB/s | 28 kB 00:00 2024-07-26T12:30:42.048 INFO:teuthology.orchestra.run.smithi132.stdout:(130/160): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-07-26T12:30:42.056 INFO:teuthology.orchestra.run.smithi112.stdout:(138/160): python3-repoze-lru-0.7-16.el9.noarch 522 kB/s | 31 kB 00:00 2024-07-26T12:30:42.065 INFO:teuthology.orchestra.run.smithi132.stdout:(131/160): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-07-26T12:30:42.073 INFO:teuthology.orchestra.run.smithi112.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-07-26T12:30:42.089 INFO:teuthology.orchestra.run.smithi112.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-07-26T12:30:42.091 INFO:teuthology.orchestra.run.smithi132.stdout:(132/160): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-07-26T12:30:42.106 INFO:teuthology.orchestra.run.smithi112.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-07-26T12:30:42.117 INFO:teuthology.orchestra.run.smithi107.stdout:(68/160): openblas-openmp-0.3.21-2.el9.x86_64.r 2.9 MB/s | 4.7 MB 00:01 2024-07-26T12:30:42.124 INFO:teuthology.orchestra.run.smithi132.stdout:(133/160): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-07-26T12:30:42.132 INFO:teuthology.orchestra.run.smithi112.stdout:(142/160): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-07-26T12:30:42.157 INFO:teuthology.orchestra.run.smithi112.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-07-26T12:30:42.158 INFO:teuthology.orchestra.run.smithi132.stdout:(134/160): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-07-26T12:30:42.182 INFO:teuthology.orchestra.run.smithi112.stdout:(144/160): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-07-26T12:30:42.183 INFO:teuthology.orchestra.run.smithi132.stdout:(135/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-07-26T12:30:42.208 INFO:teuthology.orchestra.run.smithi112.stdout:(145/160): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-07-26T12:30:42.209 INFO:teuthology.orchestra.run.smithi132.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-07-26T12:30:42.225 INFO:teuthology.orchestra.run.smithi107.stdout:(69/160): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-07-26T12:30:42.233 INFO:teuthology.orchestra.run.smithi112.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-07-26T12:30:42.234 INFO:teuthology.orchestra.run.smithi132.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-07-26T12:30:42.258 INFO:teuthology.orchestra.run.smithi112.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-07-26T12:30:42.259 INFO:teuthology.orchestra.run.smithi132.stdout:(138/160): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-07-26T12:30:42.284 INFO:teuthology.orchestra.run.smithi132.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-07-26T12:30:42.301 INFO:teuthology.orchestra.run.smithi107.stdout:(70/160): postgresql-contrib-13.14-1.el9.x86_64 1.7 MB/s | 911 kB 00:00 2024-07-26T12:30:42.310 INFO:teuthology.orchestra.run.smithi132.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-07-26T12:30:42.317 INFO:teuthology.orchestra.run.smithi112.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-07-26T12:30:42.335 INFO:teuthology.orchestra.run.smithi132.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-07-26T12:30:42.360 INFO:teuthology.orchestra.run.smithi132.stdout:(142/160): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-07-26T12:30:42.386 INFO:teuthology.orchestra.run.smithi132.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-07-26T12:30:42.401 INFO:teuthology.orchestra.run.smithi107.stdout:(71/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 2.3 MB/s | 4.4 MB 00:01 2024-07-26T12:30:42.411 INFO:teuthology.orchestra.run.smithi132.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 806 kB/s | 20 kB 00:00 2024-07-26T12:30:42.436 INFO:teuthology.orchestra.run.smithi132.stdout:(145/160): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-07-26T12:30:42.462 INFO:teuthology.orchestra.run.smithi132.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-07-26T12:30:42.485 INFO:teuthology.orchestra.run.smithi107.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-07-26T12:30:42.520 INFO:teuthology.orchestra.run.smithi132.stdout:(147/160): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-07-26T12:30:42.584 INFO:teuthology.orchestra.run.smithi112.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 9.6 MB/s | 3.4 MB 00:00 2024-07-26T12:30:42.634 INFO:teuthology.orchestra.run.smithi112.stdout:(150/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 8.4 MB/s | 3.1 MB 00:00 2024-07-26T12:30:42.766 INFO:teuthology.orchestra.run.smithi132.stdout:(148/160): librados2-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 3.4 MB 00:00 2024-07-26T12:30:42.821 INFO:teuthology.orchestra.run.smithi132.stdout:(149/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 10 MB/s | 3.1 MB 00:00 2024-07-26T12:30:42.876 INFO:teuthology.orchestra.run.smithi107.stdout:(73/160): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-07-26T12:30:42.901 INFO:teuthology.orchestra.run.smithi112.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 153 kB/s | 41 kB 00:00 2024-07-26T12:30:42.935 INFO:teuthology.orchestra.run.smithi107.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.0 MB/s | 61 kB 00:00 2024-07-26T12:30:42.971 INFO:teuthology.orchestra.run.smithi132.stdout:(150/160): NetworkManager-1.48.2-2.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-07-26T12:30:43.030 INFO:teuthology.orchestra.run.smithi132.stdout:(151/160): NetworkManager-libnm-1.48.2-2.el9.x8 8.6 MB/s | 1.8 MB 00:00 2024-07-26T12:30:43.071 INFO:teuthology.orchestra.run.smithi132.stdout:(152/160): NetworkManager-team-1.48.2-2.el9.x86 416 kB/s | 41 kB 00:00 2024-07-26T12:30:43.085 INFO:teuthology.orchestra.run.smithi107.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-07-26T12:30:43.138 INFO:teuthology.orchestra.run.smithi132.stdout:(153/160): NetworkManager-tui-1.48.2-2.el9.x86_ 2.2 MB/s | 247 kB 00:00 2024-07-26T12:30:43.172 INFO:teuthology.orchestra.run.smithi132.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 366 kB/s | 37 kB 00:00 2024-07-26T12:30:43.236 INFO:teuthology.orchestra.run.smithi107.stdout:(76/160): python3-jinja2-2.11.3-6.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-07-26T12:30:43.239 INFO:teuthology.orchestra.run.smithi132.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 2.0 MB/s | 207 kB 00:00 2024-07-26T12:30:43.273 INFO:teuthology.orchestra.run.smithi132.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 2.5 MB/s | 260 kB 00:00 2024-07-26T12:30:43.294 INFO:teuthology.orchestra.run.smithi107.stdout:(77/160): python3-jsonpatch-1.21-16.el9.noarch. 446 kB/s | 26 kB 00:00 2024-07-26T12:30:43.326 INFO:teuthology.orchestra.run.smithi112.stdout:(152/160): NetworkManager-tui-1.48.2-2.el9.x86_ 582 kB/s | 247 kB 00:00 2024-07-26T12:30:43.353 INFO:teuthology.orchestra.run.smithi107.stdout:(78/160): python3-jsonpointer-2.0-4.el9.noarch. 334 kB/s | 19 kB 00:00 2024-07-26T12:30:43.364 INFO:teuthology.orchestra.run.smithi132.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 11 MB/s | 1.4 MB 00:00 2024-07-26T12:30:43.432 INFO:teuthology.orchestra.run.smithi132.stdout:(158/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-07-26T12:30:43.435 INFO:teuthology.orchestra.run.smithi112.stdout:(153/160): elfutils-debuginfod-client-0.191-4.e 339 kB/s | 37 kB 00:00 2024-07-26T12:30:43.478 INFO:teuthology.orchestra.run.smithi107.stdout:(79/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-07-26T12:30:43.490 INFO:teuthology.orchestra.run.smithi132.stdout:(159/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 8.8 MB/s | 1.1 MB 00:00 2024-07-26T12:30:43.543 INFO:teuthology.orchestra.run.smithi112.stdout:(154/160): elfutils-libelf-0.191-4.el9.x86_64.r 1.9 MB/s | 207 kB 00:00 2024-07-26T12:30:43.545 INFO:teuthology.orchestra.run.smithi107.stdout:(80/160): python3-markupsafe-1.1.1-12.el9.x86_6 528 kB/s | 35 kB 00:00 2024-07-26T12:30:43.586 INFO:teuthology.orchestra.run.smithi112.stdout:(155/160): NetworkManager-1.48.2-2.el9.x86_64.r 1.8 MB/s | 2.3 MB 00:01 2024-07-26T12:30:43.654 INFO:teuthology.orchestra.run.smithi112.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 2.3 MB/s | 260 kB 00:00 2024-07-26T12:30:43.720 INFO:teuthology.orchestra.run.smithi112.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 10 MB/s | 1.4 MB 00:00 2024-07-26T12:30:43.786 INFO:teuthology.orchestra.run.smithi112.stdout:(158/160): NetworkManager-libnm-1.48.2-2.el9.x8 1.6 MB/s | 1.8 MB 00:01 2024-07-26T12:30:43.837 INFO:teuthology.orchestra.run.smithi112.stdout:(159/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 9.3 MB/s | 1.1 MB 00:00 2024-07-26T12:30:44.053 INFO:teuthology.orchestra.run.smithi112.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 6.0 MB/s | 2.4 MB 00:00 2024-07-26T12:30:44.062 INFO:teuthology.orchestra.run.smithi112.stdout:-------------------------------------------------------------------------------- 2024-07-26T12:30:44.062 INFO:teuthology.orchestra.run.smithi112.stdout:Total 13 MB/s | 233 MB 00:17 2024-07-26T12:30:45.179 INFO:teuthology.orchestra.run.smithi107.stdout:(81/160): postgresql-server-13.14-1.el9.x86_64. 2.0 MB/s | 5.8 MB 00:02 2024-07-26T12:30:45.314 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction check 2024-07-26T12:30:45.329 INFO:teuthology.orchestra.run.smithi107.stdout:(82/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-07-26T12:30:45.510 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction check succeeded. 2024-07-26T12:30:45.510 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction test 2024-07-26T12:30:45.563 INFO:teuthology.orchestra.run.smithi107.stdout:(83/160): python3-babel-2.9.1-2.el9.noarch.rpm 1.9 MB/s | 6.0 MB 00:03 2024-07-26T12:30:45.588 INFO:teuthology.orchestra.run.smithi107.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 862 kB/s | 222 kB 00:00 2024-07-26T12:30:45.696 INFO:teuthology.orchestra.run.smithi107.stdout:(85/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.4 MB/s | 5.1 MB 00:02 2024-07-26T12:30:45.722 INFO:teuthology.orchestra.run.smithi107.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 488 kB/s | 77 kB 00:00 2024-07-26T12:30:45.747 INFO:teuthology.orchestra.run.smithi107.stdout:(87/160): python3-prettytable-0.7.2-27.el9.noar 267 kB/s | 42 kB 00:00 2024-07-26T12:30:45.781 INFO:teuthology.orchestra.run.smithi107.stdout:(88/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-07-26T12:30:45.831 INFO:teuthology.orchestra.run.smithi107.stdout:(89/160): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-07-26T12:30:45.890 INFO:teuthology.orchestra.run.smithi107.stdout:(90/160): python3-requests-oauthlib-1.3.0-12.el 927 kB/s | 54 kB 00:00 2024-07-26T12:30:45.923 INFO:teuthology.orchestra.run.smithi107.stdout:(91/160): python3-protobuf-3.14.0-13.el9.noarch 1.2 MB/s | 269 kB 00:00 2024-07-26T12:30:45.957 INFO:teuthology.orchestra.run.smithi107.stdout:(92/160): python3-pyasn1-modules-0.4.8-6.el9.no 1.3 MB/s | 279 kB 00:00 2024-07-26T12:30:45.974 INFO:teuthology.orchestra.run.smithi107.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 848 kB/s | 42 kB 00:00 2024-07-26T12:30:46.016 INFO:teuthology.orchestra.run.smithi107.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 768 kB/s | 44 kB 00:00 2024-07-26T12:30:46.074 INFO:teuthology.orchestra.run.smithi107.stdout:(95/160): qatlib-service-24.02.0-1.el9.x86_64.r 625 kB/s | 36 kB 00:00 2024-07-26T12:30:46.133 INFO:teuthology.orchestra.run.smithi107.stdout:(96/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 824 kB/s | 47 kB 00:00 2024-07-26T12:30:46.158 INFO:teuthology.orchestra.run.smithi107.stdout:(97/160): qatlib-24.02.0-1.el9.x86_64.rpm 1.2 MB/s | 221 kB 00:00 2024-07-26T12:30:46.217 INFO:teuthology.orchestra.run.smithi107.stdout:(98/160): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-07-26T12:30:46.309 INFO:teuthology.orchestra.run.smithi107.stdout:(99/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rpm 1.0 MB/s | 96 kB 00:00 2024-07-26T12:30:46.343 INFO:teuthology.orchestra.run.smithi107.stdout:(100/160): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-07-26T12:30:46.568 INFO:teuthology.orchestra.run.smithi107.stdout:(101/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-07-26T12:30:46.593 INFO:teuthology.orchestra.run.smithi107.stdout:(102/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 224 kB/s | 64 kB 00:00 2024-07-26T12:30:46.618 INFO:teuthology.orchestra.run.smithi107.stdout:(103/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 163 kB/s | 45 kB 00:00 2024-07-26T12:30:46.777 INFO:teuthology.orchestra.run.smithi107.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 3.4 MB/s | 548 kB 00:00 2024-07-26T12:30:46.819 INFO:teuthology.orchestra.run.smithi107.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 7.3 MB/s | 309 kB 00:00 2024-07-26T12:30:46.845 INFO:teuthology.orchestra.run.smithi107.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 787 kB/s | 19 kB 00:00 2024-07-26T12:30:46.878 INFO:teuthology.orchestra.run.smithi107.stdout:(107/160): lua-devel-5.4.4-4.el9.x86_64.rpm 72 kB/s | 22 kB 00:00 2024-07-26T12:30:46.987 INFO:teuthology.orchestra.run.smithi107.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 242 kB/s | 26 kB 00:00 2024-07-26T12:30:47.020 INFO:teuthology.orchestra.run.smithi107.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-07-26T12:30:47.054 INFO:teuthology.orchestra.run.smithi107.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-07-26T12:30:47.087 INFO:teuthology.orchestra.run.smithi107.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 4.6 MB/s | 151 kB 00:00 2024-07-26T12:30:47.163 INFO:teuthology.orchestra.run.smithi107.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-07-26T12:30:47.222 INFO:teuthology.orchestra.run.smithi107.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 9.3 MB/s | 548 kB 00:00 2024-07-26T12:30:47.255 INFO:teuthology.orchestra.run.smithi107.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-07-26T12:30:47.297 INFO:teuthology.orchestra.run.smithi107.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 784 kB/s | 32 kB 00:00 2024-07-26T12:30:47.323 INFO:teuthology.orchestra.run.smithi107.stdout:(116/160): python3-certifi-2023.05.07-4.el9.noa 584 kB/s | 14 kB 00:00 2024-07-26T12:30:47.356 INFO:teuthology.orchestra.run.smithi107.stdout:(117/160): python3-cheroot-8.6.0-4.el9.noarch.r 5.2 MB/s | 172 kB 00:00 2024-07-26T12:30:47.366 INFO:teuthology.orchestra.run.smithi112.stdout:Transaction test succeeded. 2024-07-26T12:30:47.366 INFO:teuthology.orchestra.run.smithi112.stdout:Running transaction 2024-07-26T12:30:47.407 INFO:teuthology.orchestra.run.smithi107.stdout:(118/160): python3-cherrypy-18.6.1-2.el9.noarch 7.1 MB/s | 358 kB 00:00 2024-07-26T12:30:47.449 INFO:teuthology.orchestra.run.smithi107.stdout:(119/160): protobuf-compiler-3.14.0-13.el9.x86_ 1.0 MB/s | 863 kB 00:00 2024-07-26T12:30:47.549 INFO:teuthology.orchestra.run.smithi107.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-07-26T12:30:47.649 INFO:teuthology.orchestra.run.smithi107.stdout:(121/160): libarrow-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-07-26T12:30:47.683 INFO:teuthology.orchestra.run.smithi107.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-07-26T12:30:47.733 INFO:teuthology.orchestra.run.smithi107.stdout:(123/160): python3-google-auth-2.32.0-1.el9.noa 760 kB/s | 216 kB 00:00 2024-07-26T12:30:47.759 INFO:teuthology.orchestra.run.smithi107.stdout:(124/160): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-07-26T12:30:47.784 INFO:teuthology.orchestra.run.smithi107.stdout:(125/160): python3-jaraco-classes-3.2.1-5.el9.n 357 kB/s | 18 kB 00:00 2024-07-26T12:30:47.817 INFO:teuthology.orchestra.run.smithi107.stdout:(126/160): python3-jaraco-collections-3.0.0-8.e 400 kB/s | 23 kB 00:00 2024-07-26T12:30:47.834 INFO:teuthology.orchestra.run.smithi107.stdout:(127/160): python3-jaraco-functools-3.5.0-2.el9 393 kB/s | 19 kB 00:00 2024-07-26T12:30:47.851 INFO:teuthology.orchestra.run.smithi107.stdout:(128/160): python3-jaraco-text-3.2.0-6.el9.noar 600 kB/s | 20 kB 00:00 2024-07-26T12:30:47.877 INFO:teuthology.orchestra.run.smithi107.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.9 MB/s | 46 kB 00:00 2024-07-26T12:30:47.910 INFO:teuthology.orchestra.run.smithi107.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.4 MB/s | 79 kB 00:00 2024-07-26T12:30:47.952 INFO:teuthology.orchestra.run.smithi107.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-07-26T12:30:48.002 INFO:teuthology.orchestra.run.smithi107.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 5.4 MB/s | 272 kB 00:00 2024-07-26T12:30:48.020 INFO:teuthology.orchestra.run.smithi107.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 1.0 MB/s | 16 kB 00:00 2024-07-26T12:30:48.053 INFO:teuthology.orchestra.run.smithi107.stdout:(134/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.7 MB/s | 90 kB 00:00 2024-07-26T12:30:48.070 INFO:teuthology.orchestra.run.smithi107.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.7 MB/s | 28 kB 00:00 2024-07-26T12:30:48.088 INFO:teuthology.orchestra.run.smithi107.stdout:(136/160): python3-repoze-lru-0.7-16.el9.noarch 1.9 MB/s | 31 kB 00:00 2024-07-26T12:30:48.138 INFO:teuthology.orchestra.run.smithi107.stdout:(137/160): python3-kubernetes-26.1.0-2.el9.noar 3.4 MB/s | 1.0 MB 00:00 2024-07-26T12:30:48.163 INFO:teuthology.orchestra.run.smithi107.stdout:(138/160): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-07-26T12:30:48.189 INFO:teuthology.orchestra.run.smithi107.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-07-26T12:30:48.213 INFO:teuthology.orchestra.run.smithi107.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 724 kB/s | 36 kB 00:00 2024-07-26T12:30:48.247 INFO:teuthology.orchestra.run.smithi107.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-07-26T12:30:48.280 INFO:teuthology.orchestra.run.smithi107.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 2.7 MB/s | 90 kB 00:00 2024-07-26T12:30:48.314 INFO:teuthology.orchestra.run.smithi107.stdout:(143/160): python3-webob-1.8.7-6.el9.noarch.rpm 2.3 MB/s | 230 kB 00:00 2024-07-26T12:30:48.340 INFO:teuthology.orchestra.run.smithi107.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 824 kB/s | 20 kB 00:00 2024-07-26T12:30:48.415 INFO:teuthology.orchestra.run.smithi107.stdout:(145/160): python3-grpcio-1.46.7-10.el9.x86_64. 2.4 MB/s | 2.0 MB 00:00 2024-07-26T12:30:48.440 INFO:teuthology.orchestra.run.smithi107.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-07-26T12:30:48.474 INFO:teuthology.orchestra.run.smithi107.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 2.2 MB/s | 427 kB 00:00 2024-07-26T12:30:48.558 INFO:teuthology.orchestra.run.smithi107.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-07-26T12:30:48.741 INFO:teuthology.orchestra.run.smithi107.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 3.4 MB 00:00 2024-07-26T12:30:48.825 INFO:teuthology.orchestra.run.smithi107.stdout:(150/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 9.0 MB/s | 3.1 MB 00:00 2024-07-26T12:30:49.683 INFO:teuthology.orchestra.run.smithi107.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 52 kB/s | 41 kB 00:00 2024-07-26T12:30:50.092 INFO:teuthology.orchestra.run.smithi107.stdout:(152/160): NetworkManager-1.48.2-2.el9.x86_64.r 1.5 MB/s | 2.3 MB 00:01 2024-07-26T12:30:50.384 INFO:teuthology.orchestra.run.smithi107.stdout:(153/160): NetworkManager-libnm-1.48.2-2.el9.x8 1.1 MB/s | 1.8 MB 00:01 2024-07-26T12:30:50.926 INFO:teuthology.orchestra.run.smithi107.stdout:(154/160): NetworkManager-tui-1.48.2-2.el9.x86_ 199 kB/s | 247 kB 00:01 2024-07-26T12:30:50.942 INFO:teuthology.orchestra.run.smithi107.stdout:(155/160): elfutils-debuginfod-client-0.191-4.e 43 kB/s | 37 kB 00:00 2024-07-26T12:30:51.318 INFO:teuthology.orchestra.run.smithi107.stdout:(156/160): elfutils-libelf-0.191-4.el9.x86_64.r 222 kB/s | 207 kB 00:00 2024-07-26T12:30:51.860 INFO:teuthology.orchestra.run.smithi107.stdout:(157/160): elfutils-libs-0.191-4.el9.x86_64.rpm 278 kB/s | 260 kB 00:00 2024-07-26T12:30:52.135 INFO:teuthology.orchestra.run.smithi107.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 1.1 MB/s | 1.4 MB 00:01 2024-07-26T12:30:52.610 INFO:teuthology.orchestra.run.smithi107.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 1.9 MB/s | 2.4 MB 00:01 2024-07-26T12:30:53.002 INFO:teuthology.orchestra.run.smithi107.stdout:(160/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 971 kB/s | 1.1 MB 00:01 2024-07-26T12:30:53.015 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-07-26T12:30:53.015 INFO:teuthology.orchestra.run.smithi107.stdout:Total 11 MB/s | 233 MB 00:21 2024-07-26T12:30:53.683 INFO:teuthology.orchestra.run.smithi112.stdout: Preparing : 1/1 2024-07-26T12:30:53.810 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T12:30:54.055 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T12:30:54.119 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T12:30:54.159 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T12:30:54.185 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T12:30:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T12:30:54.570 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T12:30:54.696 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T12:30:54.821 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T12:30:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T12:30:54.949 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T12:30:54.976 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T12:30:55.009 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T12:30:55.209 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T12:30:55.257 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T12:30:55.286 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T12:30:55.390 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T12:30:55.420 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T12:30:55.466 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T12:30:55.488 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T12:30:55.513 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T12:30:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T12:30:55.823 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T12:30:55.992 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T12:30:56.138 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T12:30:56.198 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T12:30:56.292 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T12:30:56.366 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T12:30:56.383 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-07-26T12:30:56.563 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T12:30:56.674 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-07-26T12:30:56.674 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-07-26T12:30:56.674 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T12:30:56.703 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T12:30:56.755 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T12:30:56.837 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T12:30:56.882 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T12:30:56.922 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T12:30:56.966 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T12:30:56.998 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T12:30:57.026 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T12:30:57.087 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T12:30:57.141 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T12:30:57.171 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T12:30:57.206 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T12:30:57.282 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T12:30:57.317 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T12:30:57.362 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T12:30:57.389 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T12:30:57.458 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T12:30:57.504 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T12:30:57.578 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T12:30:57.711 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T12:30:57.765 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T12:30:58.287 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T12:30:58.361 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T12:30:58.398 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T12:30:58.434 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T12:30:58.483 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T12:30:58.521 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T12:30:58.563 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T12:30:58.725 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T12:30:58.791 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T12:30:58.881 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T12:30:58.946 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T12:30:59.019 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T12:30:59.050 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T12:30:59.170 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T12:30:59.243 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T12:30:59.278 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T12:30:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T12:30:59.329 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-07-26T12:30:59.329 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-07-26T12:30:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T12:30:59.498 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T12:30:59.532 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T12:30:59.573 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T12:30:59.680 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T12:30:59.715 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T12:30:59.755 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T12:30:59.781 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T12:30:59.805 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T12:30:59.881 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T12:31:00.007 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T12:31:00.046 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T12:31:00.074 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T12:31:00.187 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T12:31:00.373 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T12:31:00.554 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T12:31:00.584 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T12:31:00.619 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T12:31:00.652 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T12:31:00.693 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T12:31:00.719 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T12:31:00.806 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T12:31:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T12:31:00.881 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T12:31:00.984 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T12:31:01.031 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T12:31:01.589 INFO:teuthology.orchestra.run.smithi132.stdout:(160/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 754 kB/s | 16 MB 00:22 2024-07-26T12:31:01.595 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-07-26T12:31:01.595 INFO:teuthology.orchestra.run.smithi132.stdout:Total 6.4 MB/s | 233 MB 00:36 2024-07-26T12:31:01.746 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T12:31:01.796 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T12:31:01.851 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T12:31:01.952 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T12:31:02.002 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T12:31:02.047 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T12:31:02.135 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T12:31:02.174 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T12:31:02.351 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T12:31:02.388 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T12:31:02.467 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T12:31:02.855 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-07-26T12:31:03.050 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-07-26T12:31:03.050 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-07-26T12:31:03.110 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T12:31:03.220 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T12:31:04.447 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T12:31:04.527 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T12:31:04.554 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T12:31:04.587 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T12:31:04.922 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T12:31:04.962 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-07-26T12:31:04.962 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-07-26T12:31:04.967 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T12:31:05.046 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T12:31:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T12:31:05.107 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T12:31:05.607 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T12:31:05.636 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T12:31:05.692 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T12:31:05.721 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T12:31:06.287 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-07-26T12:31:06.375 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T12:31:06.456 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T12:31:06.559 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T12:31:06.610 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T12:31:06.643 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T12:31:06.749 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T12:31:07.183 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T12:31:07.316 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T12:31:07.513 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T12:31:07.542 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T12:31:07.671 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T12:31:07.707 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T12:31:07.783 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T12:31:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T12:31:08.036 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T12:31:08.073 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T12:31:08.136 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T12:31:08.163 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T12:31:08.316 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T12:31:08.348 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T12:31:08.435 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T12:31:08.465 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T12:31:08.532 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T12:31:08.598 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T12:31:08.639 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T12:31:08.735 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T12:31:08.735 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T12:31:08.735 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T12:31:08.736 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:08.765 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T12:31:08.808 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T12:31:08.931 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T12:31:09.281 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T12:31:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T12:31:09.545 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T12:31:09.735 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T12:31:09.835 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T12:31:10.147 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T12:31:10.356 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T12:31:10.390 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T12:31:10.436 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T12:31:10.540 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T12:31:10.600 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T12:31:10.646 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T12:31:10.708 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T12:31:10.757 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T12:31:10.798 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T12:31:10.936 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T12:31:11.003 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T12:31:11.042 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T12:31:11.095 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T12:31:11.207 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T12:31:11.245 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T12:31:11.296 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-07-26T12:31:11.308 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T12:31:11.352 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T12:31:11.367 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T12:31:11.406 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T12:31:11.459 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T12:31:11.472 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T12:31:11.493 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T12:31:11.519 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T12:31:11.529 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T12:31:11.562 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T12:31:11.629 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T12:31:11.807 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T12:31:11.875 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T12:31:11.980 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T12:31:12.153 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T12:31:12.278 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T12:31:12.304 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T12:31:12.399 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T12:31:12.425 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T12:31:12.458 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T12:31:12.538 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T12:31:12.632 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T12:31:12.650 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T12:31:12.666 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T12:31:12.698 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T12:31:12.704 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T12:31:12.726 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T12:31:12.759 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T12:31:12.806 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T12:31:12.814 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T12:31:12.844 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T12:31:12.880 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T12:31:12.889 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T12:31:12.912 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T12:31:12.944 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T12:31:13.101 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T12:31:13.112 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T12:31:13.197 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T12:31:13.208 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T12:31:13.328 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T12:31:13.390 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T12:31:13.402 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T12:31:13.491 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T12:31:13.530 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T12:31:13.569 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T12:31:13.570 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T12:31:13.621 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T12:31:13.635 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T12:31:13.677 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T12:31:13.712 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T12:31:13.739 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T12:31:13.789 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T12:31:13.795 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T12:31:13.994 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T12:31:13.998 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T12:31:14.046 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T12:31:14.097 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T12:31:14.112 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T12:31:14.143 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T12:31:14.178 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T12:31:14.229 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T12:31:14.252 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T12:31:14.272 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T12:31:14.297 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T12:31:14.337 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T12:31:14.342 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T12:31:14.377 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T12:31:14.380 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T12:31:14.410 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T12:31:14.421 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T12:31:14.450 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T12:31:14.518 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T12:31:14.571 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T12:31:14.595 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T12:31:14.602 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T12:31:14.638 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T12:31:14.714 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T12:31:14.748 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T12:31:14.777 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T12:31:14.794 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T12:31:14.812 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T12:31:14.824 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T12:31:14.861 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T12:31:14.881 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T12:31:14.904 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T12:31:14.926 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T12:31:14.966 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T12:31:15.003 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T12:31:15.019 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T12:31:15.060 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T12:31:15.106 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T12:31:15.142 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T12:31:15.156 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T12:31:15.196 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T12:31:15.232 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T12:31:15.289 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T12:31:15.434 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T12:31:15.500 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T12:31:15.545 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T12:31:15.700 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T12:31:15.714 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T12:31:15.758 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T12:31:15.792 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T12:31:15.822 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T12:31:15.857 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T12:31:15.906 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T12:31:15.944 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T12:31:15.986 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T12:31:16.130 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T12:31:16.181 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T12:31:16.263 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T12:31:16.328 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T12:31:16.400 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T12:31:16.432 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T12:31:16.468 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T12:31:16.515 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T12:31:16.551 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T12:31:16.586 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T12:31:16.628 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T12:31:16.733 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T12:31:16.765 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T12:31:16.790 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T12:31:16.805 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T12:31:16.823 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T12:31:16.846 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T12:31:16.943 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T12:31:16.948 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T12:31:16.979 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T12:31:17.006 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T12:31:17.019 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T12:31:17.045 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T12:31:17.061 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T12:31:17.070 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T12:31:17.143 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T12:31:17.170 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T12:31:17.217 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T12:31:17.271 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T12:31:17.302 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T12:31:17.330 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T12:31:17.389 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T12:31:17.429 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T12:31:17.446 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T12:31:17.462 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T12:31:17.472 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T12:31:17.490 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T12:31:17.525 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T12:31:17.570 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T12:31:17.583 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T12:31:17.736 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T12:31:17.879 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T12:31:18.040 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T12:31:18.082 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T12:31:18.113 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T12:31:18.215 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T12:31:18.271 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T12:31:18.405 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T12:31:18.545 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T12:31:18.998 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T12:31:19.043 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T12:31:19.077 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T12:31:19.178 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T12:31:19.233 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T12:31:19.278 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T12:31:19.365 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T12:31:19.405 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T12:31:19.583 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T12:31:19.611 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T12:31:19.689 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T12:31:20.157 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T12:31:20.277 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T12:31:20.315 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T12:31:20.339 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T12:31:20.349 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T12:31:20.451 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T12:31:20.469 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T12:31:20.469 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /sys 2024-07-26T12:31:20.469 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /proc 2024-07-26T12:31:20.469 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /mnt 2024-07-26T12:31:20.470 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /var/tmp 2024-07-26T12:31:20.470 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /home 2024-07-26T12:31:20.470 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /root 2024-07-26T12:31:20.470 INFO:teuthology.orchestra.run.smithi112.stdout:skipping the directory /tmp 2024-07-26T12:31:20.470 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:20.644 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T12:31:20.661 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T12:31:20.661 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:20.661 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T12:31:20.661 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:20.742 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T12:31:20.789 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T12:31:20.914 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T12:31:20.946 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T12:31:21.001 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T12:31:21.560 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T12:31:21.600 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T12:31:21.685 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T12:31:21.701 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T12:31:21.727 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T12:31:21.773 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T12:31:21.801 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T12:31:21.834 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T12:31:22.163 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T12:31:22.214 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T12:31:22.272 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T12:31:22.298 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T12:31:22.298 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:22.298 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T12:31:22.299 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T12:31:22.299 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T12:31:22.299 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:22.301 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T12:31:22.337 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T12:31:22.373 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T12:31:22.523 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T12:31:22.546 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T12:31:22.546 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:22.546 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T12:31:22.546 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T12:31:22.547 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T12:31:22.547 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:22.864 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T12:31:22.896 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T12:31:22.978 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T12:31:23.006 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T12:31:23.007 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T12:31:23.030 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T12:31:23.030 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:23.030 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T12:31:23.030 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T12:31:23.030 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T12:31:23.031 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:23.145 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T12:31:23.194 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T12:31:23.249 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T12:31:23.435 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T12:31:23.491 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T12:31:23.578 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T12:31:23.675 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T12:31:23.825 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T12:31:23.866 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T12:31:23.906 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T12:31:23.942 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T12:31:24.350 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T12:31:24.390 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T12:31:24.394 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T12:31:24.434 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T12:31:24.464 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T12:31:24.528 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T12:31:24.564 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T12:31:25.055 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T12:31:25.102 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T12:31:25.262 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T12:31:25.262 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T12:31:25.262 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T12:31:25.263 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:25.267 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T12:31:25.302 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T12:31:25.303 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T12:31:25.870 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T12:31:25.907 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T12:31:25.988 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T12:31:25.988 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T12:31:25.988 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T12:31:25.988 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:26.016 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T12:31:27.709 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T12:31:27.761 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T12:31:28.722 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T12:31:28.753 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T12:31:28.864 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T12:31:29.037 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T12:31:29.072 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T12:31:29.095 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T12:31:29.095 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:29.095 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T12:31:29.095 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T12:31:29.095 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T12:31:29.096 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:29.148 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T12:31:29.159 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T12:31:29.258 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T12:31:29.312 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T12:31:30.266 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T12:31:30.292 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T12:31:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T12:31:30.314 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:30.314 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T12:31:30.314 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T12:31:30.315 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T12:31:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:30.361 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T12:31:30.382 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T12:31:30.382 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:30.382 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T12:31:30.382 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:30.426 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T12:31:30.449 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T12:31:30.449 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:30.449 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T12:31:30.450 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T12:31:30.450 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T12:31:30.450 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:30.769 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T12:31:30.792 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T12:31:30.792 INFO:teuthology.orchestra.run.smithi112.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:30.792 INFO:teuthology.orchestra.run.smithi112.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T12:31:30.792 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T12:31:30.792 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T12:31:30.792 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:35.139 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T12:31:35.189 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T12:31:35.254 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T12:31:35.293 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T12:31:35.321 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T12:31:35.502 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T12:31:35.555 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T12:31:35.587 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T12:31:35.619 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T12:31:35.649 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T12:31:35.695 INFO:teuthology.orchestra.run.smithi112.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T12:31:36.268 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T12:31:36.311 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T12:31:36.331 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T12:31:37.178 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T12:31:37.282 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T12:31:37.324 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T12:31:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T12:31:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T12:31:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T12:31:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T12:31:37.538 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T12:31:37.538 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T12:31:37.622 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T12:31:37.655 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T12:31:37.655 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T12:31:37.740 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T12:31:37.765 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T12:31:37.770 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T12:31:37.770 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T12:31:37.889 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T12:31:37.889 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-07-26T12:31:37.890 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-07-26T12:31:37.890 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-07-26T12:31:37.890 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-07-26T12:31:37.890 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-07-26T12:31:37.890 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /root 2024-07-26T12:31:37.890 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-07-26T12:31:37.890 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:37.898 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T12:31:37.898 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-07-26T12:31:37.898 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-07-26T12:31:37.898 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-07-26T12:31:37.898 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-07-26T12:31:37.898 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-07-26T12:31:37.898 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-07-26T12:31:37.898 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-07-26T12:31:37.898 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:37.959 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T12:31:37.959 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T12:31:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T12:31:38.057 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T12:31:38.061 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T12:31:38.073 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T12:31:38.074 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:38.074 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T12:31:38.074 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:38.087 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T12:31:38.135 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T12:31:38.151 INFO:teuthology.orchestra.run.smithi112.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T12:31:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T12:31:38.186 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:38.186 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T12:31:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:39.669 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T12:31:39.695 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T12:31:39.695 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:39.695 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T12:31:39.695 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T12:31:39.695 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T12:31:39.695 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:39.911 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T12:31:39.919 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T12:31:39.942 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T12:31:39.942 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:39.942 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T12:31:39.942 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T12:31:39.942 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T12:31:39.942 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:39.986 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T12:31:39.987 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:39.987 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T12:31:39.987 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T12:31:39.987 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T12:31:39.987 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:40.236 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T12:31:40.311 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T12:31:40.311 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:40.311 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T12:31:40.311 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T12:31:40.311 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T12:31:40.311 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:40.403 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T12:31:40.426 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T12:31:40.426 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:40.426 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T12:31:40.426 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T12:31:40.426 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T12:31:40.426 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:40.533 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T12:31:40.581 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T12:31:40.637 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T12:31:40.820 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T12:31:40.821 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T12:31:40.871 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T12:31:40.890 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T12:31:40.890 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:40.890 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T12:31:40.891 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T12:31:40.891 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T12:31:40.891 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:40.948 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T12:31:41.031 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T12:31:41.060 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T12:31:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T12:31:41.167 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T12:31:41.199 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T12:31:41.246 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T12:31:41.277 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T12:31:41.314 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T12:31:41.398 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T12:31:41.459 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T12:31:41.545 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T12:31:41.655 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T12:31:41.742 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T12:31:41.771 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T12:31:41.803 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T12:31:41.822 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T12:31:41.867 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T12:31:41.874 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T12:31:41.894 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T12:31:41.915 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T12:31:41.949 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T12:31:42.543 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T12:31:42.577 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T12:31:42.632 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T12:31:42.745 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T12:31:42.779 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T12:31:44.510 INFO:teuthology.orchestra.run.smithi112.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T12:31:44.511 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T12:31:44.512 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T12:31:44.513 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T12:31:44.513 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T12:31:44.513 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T12:31:44.513 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T12:31:44.526 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T12:31:44.526 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T12:31:44.526 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T12:31:44.526 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T12:31:44.526 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T12:31:44.526 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T12:31:44.526 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T12:31:44.526 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T12:31:44.526 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T12:31:44.527 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T12:31:44.528 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T12:31:44.529 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T12:31:44.530 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T12:31:44.531 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T12:31:44.532 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T12:31:44.533 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T12:31:44.534 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T12:31:44.535 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T12:31:44.536 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T12:31:44.537 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T12:31:44.537 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T12:31:45.050 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T12:31:45.109 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T12:31:46.073 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T12:31:46.100 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T12:31:46.213 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T12:31:46.376 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T12:31:46.411 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T12:31:46.435 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T12:31:46.435 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:46.435 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T12:31:46.435 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T12:31:46.435 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T12:31:46.435 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:46.487 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T12:31:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T12:31:46.498 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T12:31:46.570 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T12:31:46.598 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T12:31:46.652 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T12:31:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T12:31:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:47.351 INFO:teuthology.orchestra.run.smithi112.stdout:Upgraded: 2024-07-26T12:31:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T12:31:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T12:31:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T12:31:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T12:31:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T12:31:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T12:31:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout:Installed: 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T12:31:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T12:31:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T12:31:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T12:31:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T12:31:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T12:31:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T12:31:47.361 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T12:31:47.362 INFO:teuthology.orchestra.run.smithi112.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T12:31:47.365 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.365 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.365 INFO:teuthology.orchestra.run.smithi112.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:31:47.365 INFO:teuthology.orchestra.run.smithi112.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T12:31:47.365 INFO:teuthology.orchestra.run.smithi112.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T12:31:47.365 INFO:teuthology.orchestra.run.smithi112.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T12:31:47.365 INFO:teuthology.orchestra.run.smithi112.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T12:31:47.365 INFO:teuthology.orchestra.run.smithi112.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T12:31:47.365 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T12:31:47.366 INFO:teuthology.orchestra.run.smithi112.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T12:31:47.366 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-07-26T12:31:47.366 INFO:teuthology.orchestra.run.smithi112.stdout:Complete! 2024-07-26T12:31:47.604 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T12:31:47.631 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T12:31:47.654 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T12:31:47.655 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:47.655 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T12:31:47.655 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T12:31:47.655 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T12:31:47.655 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:47.662 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T12:31:47.698 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T12:31:47.719 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T12:31:47.721 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T12:31:47.721 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:47.721 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T12:31:47.721 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:47.766 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T12:31:47.771 DEBUG:teuthology.parallel:result is None 2024-07-26T12:31:47.789 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T12:31:47.789 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:47.789 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T12:31:47.789 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T12:31:47.789 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T12:31:47.789 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:47.887 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T12:31:48.074 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T12:31:48.105 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T12:31:48.116 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T12:31:48.139 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T12:31:48.139 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:48.140 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T12:31:48.140 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T12:31:48.140 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T12:31:48.140 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:31:48.170 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T12:31:48.171 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:48.171 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T12:31:48.171 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T12:31:48.171 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T12:31:48.171 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:48.239 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T12:31:48.276 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T12:31:48.392 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T12:31:48.467 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T12:31:49.520 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T12:31:49.550 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T12:31:49.621 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T12:31:49.621 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:49.621 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T12:31:49.621 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T12:31:49.621 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T12:31:49.621 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:49.668 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T12:31:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T12:31:49.763 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:49.763 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T12:31:49.763 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:49.835 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T12:31:49.937 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T12:31:49.937 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:49.937 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T12:31:49.937 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T12:31:49.937 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T12:31:49.937 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:50.336 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T12:31:50.415 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T12:31:50.415 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T12:31:50.416 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T12:31:50.416 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T12:31:50.416 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T12:31:50.416 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:31:52.554 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T12:31:52.595 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T12:31:52.635 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T12:31:52.668 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T12:31:52.699 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T12:31:52.881 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T12:31:52.921 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T12:31:52.954 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T12:31:52.995 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T12:31:53.024 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T12:31:53.062 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T12:31:53.628 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T12:31:53.666 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T12:31:53.686 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T12:31:54.517 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T12:31:54.607 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T12:31:54.648 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T12:31:54.729 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T12:31:54.769 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T12:31:54.794 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T12:31:54.819 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T12:31:54.845 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T12:31:54.846 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T12:31:54.930 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T12:31:54.955 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T12:31:54.955 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T12:31:55.041 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T12:31:55.066 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T12:31:55.071 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T12:31:55.071 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T12:31:55.259 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T12:31:55.259 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T12:31:55.352 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T12:31:55.377 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T12:31:55.403 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T12:31:55.466 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T12:31:55.513 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T12:31:55.584 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T12:31:55.649 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T12:31:55.688 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T12:31:55.727 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T12:31:56.050 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T12:31:56.109 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T12:31:56.133 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T12:31:56.173 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T12:31:56.220 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T12:31:56.299 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T12:31:57.129 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T12:31:57.179 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T12:31:57.248 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T12:31:58.602 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T12:31:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T12:31:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T12:31:58.903 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T12:31:58.961 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T12:31:58.999 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T12:31:59.035 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T12:31:59.078 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T12:31:59.078 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T12:31:59.217 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T12:31:59.267 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T12:31:59.267 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T12:31:59.406 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T12:31:59.444 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T12:31:59.478 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T12:31:59.479 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T12:31:59.824 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T12:31:59.825 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T12:31:59.969 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T12:32:00.014 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T12:32:00.052 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T12:32:00.183 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T12:32:00.428 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T12:32:00.428 INFO:teuthology.orchestra.run.smithi107.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-07-26T12:32:00.428 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:32:01.890 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T12:32:01.890 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T12:32:01.890 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T12:32:01.890 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T12:32:01.890 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T12:32:01.890 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T12:32:01.890 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T12:32:01.890 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T12:32:01.890 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T12:32:01.891 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T12:32:01.892 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T12:32:01.906 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T12:32:01.906 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T12:32:01.906 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T12:32:01.906 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T12:32:01.906 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T12:32:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T12:32:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T12:32:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T12:32:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T12:32:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T12:32:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T12:32:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T12:32:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T12:32:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T12:32:01.907 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T12:32:01.908 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T12:32:01.908 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T12:32:01.908 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T12:32:01.908 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T12:32:01.908 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T12:32:01.908 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T12:32:01.908 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T12:32:01.908 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T12:32:01.908 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T12:32:01.908 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T12:32:01.909 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T12:32:01.910 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T12:32:01.910 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T12:32:01.910 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T12:32:01.910 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T12:32:01.910 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T12:32:01.910 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T12:32:01.910 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T12:32:01.910 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T12:32:01.910 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T12:32:01.910 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T12:32:01.911 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T12:32:01.911 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T12:32:01.914 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T12:32:01.914 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T12:32:01.914 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T12:32:01.914 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T12:32:01.914 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T12:32:01.914 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T12:32:01.914 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T12:32:01.914 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T12:32:01.915 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T12:32:01.916 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T12:32:01.917 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T12:32:01.918 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T12:32:01.919 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T12:32:01.920 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T12:32:01.921 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T12:32:01.921 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T12:32:01.921 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T12:32:01.921 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T12:32:01.921 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T12:32:04.668 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T12:32:04.669 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:32:04.669 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-07-26T12:32:04.669 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T12:32:04.669 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T12:32:04.669 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T12:32:04.669 INFO:teuthology.orchestra.run.smithi132.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T12:32:04.669 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T12:32:04.669 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T12:32:04.669 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.670 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.671 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T12:32:04.672 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T12:32:04.673 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T12:32:04.674 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T12:32:04.676 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T12:32:04.677 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T12:32:04.678 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T12:32:04.679 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:04.680 INFO:teuthology.orchestra.run.smithi132.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T12:32:04.681 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T12:32:04.681 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T12:32:04.681 INFO:teuthology.orchestra.run.smithi132.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T12:32:04.681 INFO:teuthology.orchestra.run.smithi132.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T12:32:04.681 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T12:32:04.681 INFO:teuthology.orchestra.run.smithi132.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T12:32:04.681 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:32:04.681 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-07-26T12:32:05.090 DEBUG:teuthology.parallel:result is None 2024-07-26T12:32:06.858 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T12:32:06.858 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T12:32:06.858 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T12:32:06.858 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T12:32:06.858 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T12:32:06.858 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T12:32:06.858 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T12:32:06.858 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T12:32:06.859 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T12:32:06.860 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T12:32:06.860 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T12:32:06.860 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T12:32:06.860 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T12:32:06.860 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T12:32:06.873 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T12:32:06.873 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T12:32:06.873 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T12:32:06.873 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T12:32:06.873 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T12:32:06.873 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T12:32:06.874 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T12:32:06.875 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T12:32:06.876 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T12:32:06.877 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T12:32:06.878 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T12:32:06.879 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T12:32:06.880 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T12:32:06.881 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T12:32:06.882 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T12:32:06.883 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T12:32:06.884 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T12:32:06.884 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T12:32:06.884 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T12:32:06.884 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T12:32:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T12:32:09.153 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:32:09.154 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-07-26T12:32:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T12:32:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T12:32:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T12:32:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T12:32:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T12:32:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T12:32:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T12:32:09.154 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T12:32:09.157 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T12:32:09.158 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T12:32:09.159 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T12:32:09.160 INFO:teuthology.orchestra.run.smithi107.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T12:32:09.161 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T12:32:09.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T12:32:09.163 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T12:32:09.164 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.165 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T12:32:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T12:32:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T12:32:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T12:32:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T12:32:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T12:32:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T12:32:09.166 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T12:32:09.167 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:32:09.167 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-07-26T12:32:09.840 DEBUG:teuthology.parallel:result is None 2024-07-26T12:32:09.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:32:09.992 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T12:32:10.048 INFO:teuthology.orchestra.run.smithi107.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T12:32:10.049 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T12:32:10.049 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T12:32:10.051 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:32:10.199 DEBUG:teuthology.orchestra.run.smithi112:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T12:32:10.225 INFO:teuthology.orchestra.run.smithi112.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T12:32:10.225 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T12:32:10.225 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T12:32:10.227 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:32:10.370 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T12:32:10.396 INFO:teuthology.orchestra.run.smithi132.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T12:32:10.396 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T12:32:10.397 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T12:32:10.398 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-07-26T12:32:10.399 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:32:10.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T12:32:10.466 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:32:10.466 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T12:32:10.497 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:32:10.497 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T12:32:10.527 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-07-26T12:32:10.528 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:32:10.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T12:32:10.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T12:32:10.691 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:32:10.691 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T12:32:10.719 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T12:32:10.786 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:32:10.786 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T12:32:10.814 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T12:32:10.880 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-07-26T12:32:10.881 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:32:10.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T12:32:10.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T12:32:11.048 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:32:11.048 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T12:32:11.072 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T12:32:11.135 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:32:11.135 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T12:32:11.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T12:32:11.221 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-07-26T12:32:11.221 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:32:11.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T12:32:11.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T12:32:11.379 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:32:11.379 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T12:32:11.403 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T12:32:11.466 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:32:11.466 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T12:32:11.491 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T12:32:11.555 INFO:teuthology.run_tasks:Running task cephadm... 2024-07-26T12:32:11.651 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug cephsqlite': 20, 'mgr/snap_schedule/allow_m_granularity': True, 'mgr/snap_schedule/dump_on_update': True, 'mgr/snap_schedule/log_level': 'debug'}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'ms die on skipped message': False}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'mgr-modules': ['snap_schedule'], 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'subvols': {'create': 2}} 2024-07-26T12:32:11.651 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:32:11.652 INFO:tasks.cephadm:Cluster fsid is 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:32:11.652 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-07-26T12:32:11.652 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.107', 'mon.b': '172.21.15.112', 'mon.c': '172.21.15.132'} 2024-07-26T12:32:11.652 INFO:tasks.cephadm:First mon is mon.a on smithi107 2024-07-26T12:32:11.652 INFO:tasks.cephadm:First mgr is x 2024-07-26T12:32:11.653 INFO:tasks.cephadm:Normalizing hostnames... 2024-07-26T12:32:11.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2024-07-26T12:32:11.724 DEBUG:teuthology.orchestra.run.smithi112:> sudo hostname $(hostname -s) 2024-07-26T12:32:11.748 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname $(hostname -s) 2024-07-26T12:32:11.773 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-07-26T12:32:11.773 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:32:11.920 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 16:18:49.953482', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['arm64', 'x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.2.8+braggi08', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 15:10:42.298781', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-07-26T12:32:12.065 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240725.121331-debug, sha1 e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:32:12.067 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T12:32:12.067 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T12:32:12.067 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T12:32:12.490 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 12:32 /home/ubuntu/cephtest/cephadm 2024-07-26T12:32:12.490 DEBUG:teuthology.orchestra.run.smithi112:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T12:32:12.866 INFO:teuthology.orchestra.run.smithi112.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 12:32 /home/ubuntu/cephtest/cephadm 2024-07-26T12:32:12.866 DEBUG:teuthology.orchestra.run.smithi132:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T12:32:13.244 INFO:teuthology.orchestra.run.smithi132.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 12:32 /home/ubuntu/cephtest/cephadm 2024-07-26T12:32:13.244 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T12:32:13.304 DEBUG:teuthology.orchestra.run.smithi112:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T12:32:13.322 DEBUG:teuthology.orchestra.run.smithi132:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T12:32:13.357 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 on all hosts... 2024-07-26T12:32:13.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T12:32:13.364 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T12:32:13.366 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T12:32:14.263 INFO:teuthology.orchestra.run.smithi112.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T12:32:14.299 INFO:teuthology.orchestra.run.smithi132.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T12:32:15.080 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T12:32:47.567 INFO:teuthology.orchestra.run.smithi132.stdout:{ 2024-07-26T12:32:47.568 INFO:teuthology.orchestra.run.smithi132.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T12:32:47.568 INFO:teuthology.orchestra.run.smithi132.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T12:32:47.568 INFO:teuthology.orchestra.run.smithi132.stdout: "repo_digests": [ 2024-07-26T12:32:47.568 INFO:teuthology.orchestra.run.smithi132.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T12:32:47.568 INFO:teuthology.orchestra.run.smithi132.stdout: ] 2024-07-26T12:32:47.568 INFO:teuthology.orchestra.run.smithi132.stdout:} 2024-07-26T12:32:48.188 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-07-26T12:32:48.188 INFO:teuthology.orchestra.run.smithi112.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T12:32:48.188 INFO:teuthology.orchestra.run.smithi112.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T12:32:48.188 INFO:teuthology.orchestra.run.smithi112.stdout: "repo_digests": [ 2024-07-26T12:32:48.189 INFO:teuthology.orchestra.run.smithi112.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T12:32:48.189 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2024-07-26T12:32:48.189 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-07-26T12:32:57.179 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:32:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T12:32:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T12:32:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: "repo_digests": [ 2024-07-26T12:32:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T12:32:57.180 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-07-26T12:32:57.180 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:32:57.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2024-07-26T12:32:57.305 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph 2024-07-26T12:32:57.336 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph 2024-07-26T12:32:57.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2024-07-26T12:32:57.450 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /etc/ceph 2024-07-26T12:32:57.479 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /etc/ceph 2024-07-26T12:32:57.507 INFO:tasks.cephadm:Writing seed config... 2024-07-26T12:32:57.508 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-07-26T12:32:57.508 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-07-26T12:32:57.508 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-07-26T12:32:57.508 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-07-26T12:32:57.508 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-07-26T12:32:57.508 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-07-26T12:32:57.508 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-07-26T12:32:57.509 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-07-26T12:32:57.510 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-07-26T12:32:57.510 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:32:57.510 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-07-26T12:32:57.567 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 14de5e40-4b4b-11ef-bcb7-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-07-26T12:32:57.568 DEBUG:teuthology.orchestra.run.smithi107:mon.a> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mon.a.service 2024-07-26T12:32:57.574 DEBUG:teuthology.orchestra.run.smithi107:mgr.x> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mgr.x.service 2024-07-26T12:32:57.620 INFO:tasks.cephadm:Bootstrapping... 2024-07-26T12:32:57.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 -v bootstrap --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.107 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-07-26T12:32:57.909 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-07-26T12:32:57.909 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', '-v', 'bootstrap', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.107', '--skip-admin-label'] 2024-07-26T12:32:57.909 INFO:teuthology.orchestra.run.smithi107.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-07-26T12:32:57.910 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying podman|docker is present... 2024-07-26T12:32:57.968 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 5.1.2 2024-07-26T12:32:57.968 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying lvm2 is present... 2024-07-26T12:32:57.968 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying time synchronization is in place... 2024-07-26T12:32:57.987 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T12:32:57.987 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T12:32:58.004 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T12:32:58.005 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-07-26T12:32:58.025 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout enabled 2024-07-26T12:32:58.043 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout active 2024-07-26T12:32:58.043 INFO:teuthology.orchestra.run.smithi107.stdout:Unit chronyd.service is enabled and running 2024-07-26T12:32:58.043 INFO:teuthology.orchestra.run.smithi107.stdout:Repeating the final host check... 2024-07-26T12:32:58.100 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 5.1.2 2024-07-26T12:32:58.101 INFO:teuthology.orchestra.run.smithi107.stdout:podman (/bin/podman) version 5.1.2 is present 2024-07-26T12:32:58.101 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl is present 2024-07-26T12:32:58.101 INFO:teuthology.orchestra.run.smithi107.stdout:lvcreate is present 2024-07-26T12:32:58.117 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T12:32:58.117 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T12:32:58.133 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T12:32:58.133 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-07-26T12:32:58.150 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout enabled 2024-07-26T12:32:58.167 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout active 2024-07-26T12:32:58.167 INFO:teuthology.orchestra.run.smithi107.stdout:Unit chronyd.service is enabled and running 2024-07-26T12:32:58.167 INFO:teuthology.orchestra.run.smithi107.stdout:Host looks OK 2024-07-26T12:32:58.167 INFO:teuthology.orchestra.run.smithi107.stdout:Cluster fsid: 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:32:58.167 INFO:teuthology.orchestra.run.smithi107.stdout:Acquiring lock 139953242846016 on /run/cephadm/14de5e40-4b4b-11ef-bcb7-c7b262605968.lock 2024-07-26T12:32:58.168 INFO:teuthology.orchestra.run.smithi107.stdout:Lock 139953242846016 acquired on /run/cephadm/14de5e40-4b4b-11ef-bcb7-c7b262605968.lock 2024-07-26T12:32:58.168 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying IP 172.21.15.107 port 3300 ... 2024-07-26T12:32:58.168 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying IP 172.21.15.107 port 6789 ... 2024-07-26T12:32:58.168 INFO:teuthology.orchestra.run.smithi107.stdout:Base mon IP(s) is [172.21.15.107:3300, 172.21.15.107:6789], mon addrv is [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-07-26T12:32:58.178 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.107 metric 100 2024-07-26T12:32:58.178 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.107 metric 100 2024-07-26T12:32:58.186 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-07-26T12:32:58.186 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-07-26T12:32:58.186 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2024-07-26T12:32:58.194 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-07-26T12:32:58.194 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-07-26T12:32:58.194 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T12:32:58.194 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-07-26T12:32:58.195 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2024-07-26T12:32:58.195 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T12:32:58.195 INFO:teuthology.orchestra.run.smithi107.stdout:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-07-26T12:32:58.196 INFO:teuthology.orchestra.run.smithi107.stdout:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-07-26T12:32:58.196 INFO:teuthology.orchestra.run.smithi107.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-07-26T12:32:58.196 INFO:teuthology.orchestra.run.smithi107.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-07-26T12:32:58.198 INFO:teuthology.orchestra.run.smithi107.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T12:32:58.912 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T12:32:58.912 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T12:32:58.912 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Getting image source signatures 2024-07-26T12:32:58.912 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying blob sha256:27b44fafaf0d8ddf6712bf077b0e069fd1632e81c2016bcdf9194ad64e762303 2024-07-26T12:32:58.912 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying blob sha256:43735fbce9214f061a6894341d7822563f5aafe16fc7532189f4415cff6d2d91 2024-07-26T12:32:58.913 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying config sha256:ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T12:32:58.913 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Writing manifest to image destination 2024-07-26T12:33:00.213 INFO:teuthology.orchestra.run.smithi107.stdout:ceph: stdout ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:33:00.214 INFO:teuthology.orchestra.run.smithi107.stdout:Ceph version: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:33:00.214 INFO:teuthology.orchestra.run.smithi107.stdout:Extracting ceph user uid/gid from container image... 2024-07-26T12:33:01.367 INFO:teuthology.orchestra.run.smithi107.stdout:stat: stdout 167 167 2024-07-26T12:33:01.367 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial keys... 2024-07-26T12:33:02.482 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQB9l6Nmrr0KMxAAOw+TSfONIuwUOsSKtkLiRQ== 2024-07-26T12:33:03.611 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQB/l6NmHe13ARAA2mqj8xN505ynRkdKniOTzA== 2024-07-26T12:33:04.799 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQCAl6NmJVKLCBAAV4CIHtnFEsyjT+pgbDLh8Q== 2024-07-26T12:33:04.801 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial monmap... 2024-07-26T12:33:05.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T12:33:05.937 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-07-26T12:33:05.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:05.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T12:33:05.938 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool for a [v2:172.21.15.107:3300,v1:172.21.15.107:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T12:33:05.938 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = quincy 2024-07-26T12:33:05.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: set fsid to 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:05.938 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T12:33:05.938 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:33:05.938 INFO:teuthology.orchestra.run.smithi107.stdout:Creating mon... 2024-07-26T12:33:07.200 INFO:teuthology.orchestra.run.smithi107.stdout:create mon.a on 2024-07-26T12:33:07.584 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-07-26T12:33:07.822 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-07-26T12:33:08.115 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968.target → /etc/systemd/system/ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968.target. 2024-07-26T12:33:08.115 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968.target → /etc/systemd/system/ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968.target. 2024-07-26T12:33:08.501 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mon.a 2024-07-26T12:33:08.501 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to reset failed state of unit ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mon.a.service: Unit ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mon.a.service not loaded. 2024-07-26T12:33:08.777 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968.target.wants/ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mon.a.service → /etc/systemd/system/ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@.service. 2024-07-26T12:33:09.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:09 smithi107 podman[32276]: 2024-07-26 12:33:09.176777621 +0000 UTC m=+0.150782633 container create e320f4cb17d81d89e58afd54b7582b6cc96f8aa335d294168c4e1c0eaf8b78f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:33:09.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:09 smithi107 podman[32276]: 2024-07-26 12:33:09.438653752 +0000 UTC m=+0.412658775 container init e320f4cb17d81d89e58afd54b7582b6cc96f8aa335d294168c4e1c0eaf8b78f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-26T12:33:09.545 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T12:33:09.545 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-07-26T12:33:09.567 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T12:33:09.568 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-07-26T12:33:09.568 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-07-26T12:33:09.568 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T12:33:09.570 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mon to start... 2024-07-26T12:33:09.570 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mon... 2024-07-26T12:33:09.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:09 smithi107 podman[32276]: 2024-07-26 12:33:09.449605929 +0000 UTC m=+0.423610933 container start e320f4cb17d81d89e58afd54b7582b6cc96f8aa335d294168c4e1c0eaf8b78f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-07-26T12:33:09.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:09 smithi107 bash[32276]: e320f4cb17d81d89e58afd54b7582b6cc96f8aa335d294168c4e1c0eaf8b78f2 2024-07-26T12:33:09.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:09 smithi107 systemd[1]: Started Ceph mon.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:33:09.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:09 smithi107 ceph-mon[32295]: mkfs 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:09.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:09 smithi107 ceph-mon[32295]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T12:33:10.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T12:33:10.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: monmap epoch 1 2024-07-26T12:33:10.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:10.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: last_changed 2024-07-26T12:33:05.278670+0000 2024-07-26T12:33:10.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: created 2024-07-26T12:33:05.278670+0000 2024-07-26T12:33:10.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: min_mon_release 19 (squid) 2024-07-26T12:33:10.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: election_strategy: 1 2024-07-26T12:33:10.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-07-26T12:33:10.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: fsmap 2024-07-26T12:33:10.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T12:33:10.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: mgrmap e1: no daemons active 2024-07-26T12:33:10.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:10 smithi107 ceph-mon[32295]: from='client.? 172.21.15.107:0/3785668499' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-07-26T12:33:10.877 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout cluster: 2024-07-26T12:33:10.878 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout id: 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:10.878 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-07-26T12:33:10.878 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:10.878 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout services: 2024-07-26T12:33:10.879 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.702637s) 2024-07-26T12:33:10.879 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-07-26T12:33:10.879 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-07-26T12:33:10.879 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:10.879 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout data: 2024-07-26T12:33:10.879 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-07-26T12:33:10.879 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-07-26T12:33:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-07-26T12:33:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout pgs: 2024-07-26T12:33:10.880 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:10.881 INFO:teuthology.orchestra.run.smithi107.stdout:mon is available 2024-07-26T12:33:10.881 INFO:teuthology.orchestra.run.smithi107.stdout:Assimilating anything we can from ceph.conf... 2024-07-26T12:33:11.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:11 smithi107 ceph-mon[32295]: from='client.? 172.21.15.107:0/1891167130' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T12:33:11.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:11 smithi107 ceph-mon[32295]: from='client.? 172.21.15.107:0/1891167130' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-07-26T12:33:12.196 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:12.197 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T12:33:12.197 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout fsid = 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:12.197 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T12:33:12.198 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-07-26T12:33:12.198 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T12:33:12.198 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T12:33:12.198 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T12:33:12.198 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T12:33:12.199 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:12.199 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T12:33:12.199 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T12:33:12.200 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:12.200 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T12:33:12.200 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T12:33:12.200 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T12:33:12.200 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T12:33:12.200 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T12:33:12.200 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T12:33:12.200 INFO:teuthology.orchestra.run.smithi107.stdout:Generating new minimal ceph.conf... 2024-07-26T12:33:13.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:12 smithi107 ceph-mon[32295]: from='client.? 172.21.15.107:0/2364133764' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:13.468 INFO:teuthology.orchestra.run.smithi107.stdout:Restarting the monitor... 2024-07-26T12:33:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:13 smithi107 systemd[1]: Stopping Ceph mon.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:33:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a[32291]: 2024-07-26T12:33:13.960+0000 7efd8e800640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-07-26T12:33:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a[32291]: 2024-07-26T12:33:13.960+0000 7efd8e800640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-07-26T12:33:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:14 smithi107 podman[32536]: 2024-07-26 12:33:14.166430469 +0000 UTC m=+0.328198988 container died e320f4cb17d81d89e58afd54b7582b6cc96f8aa335d294168c4e1c0eaf8b78f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T12:33:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:14 smithi107 podman[32536]: 2024-07-26 12:33:14.592641172 +0000 UTC m=+0.754409688 container cleanup e320f4cb17d81d89e58afd54b7582b6cc96f8aa335d294168c4e1c0eaf8b78f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-07-26T12:33:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:14 smithi107 bash[32536]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a 2024-07-26T12:33:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:14 smithi107 podman[32550]: 2024-07-26 12:33:14.851960399 +0000 UTC m=+0.669356068 container remove e320f4cb17d81d89e58afd54b7582b6cc96f8aa335d294168c4e1c0eaf8b78f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-07-26T12:33:15.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:15 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mon.a.service: Deactivated successfully. 2024-07-26T12:33:15.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:15 smithi107 systemd[1]: Stopped Ceph mon.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:33:15.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:15 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mon.a.service: Consumed 2.063s CPU time. 2024-07-26T12:33:15.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:15 smithi107 systemd[1]: Starting Ceph mon.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:33:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:15 smithi107 podman[32641]: 2024-07-26 12:33:15.638996096 +0000 UTC m=+0.033072675 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:33:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:20 smithi107 podman[32641]: 2024-07-26 12:33:20.45590321 +0000 UTC m=+4.849979824 container create 4bc7dea60603896a79f03fb279d3c182c51f9e5d5cb53bfac5f19e2119c3b4b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True) 2024-07-26T12:33:26.135 INFO:teuthology.orchestra.run.smithi107.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-07-26T12:33:26.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 podman[32641]: 2024-07-26 12:33:26.031091949 +0000 UTC m=+10.425168580 container init 4bc7dea60603896a79f03fb279d3c182c51f9e5d5cb53bfac5f19e2119c3b4b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-26T12:33:26.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 podman[32641]: 2024-07-26 12:33:26.042045691 +0000 UTC m=+10.436122308 container start 4bc7dea60603896a79f03fb279d3c182c51f9e5d5cb53bfac5f19e2119c3b4b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-a, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:33:26.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 bash[32641]: 4bc7dea60603896a79f03fb279d3c182c51f9e5d5cb53bfac5f19e2119c3b4b4 2024-07-26T12:33:26.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 systemd[1]: Started Ceph mon.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:33:26.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T12:33:26.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T12:33:26.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: pidfile_write: ignore empty --pid-file 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: load: jerasure load: lrc 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: RocksDB version: 7.9.2 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Git sha 0 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: DB SUMMARY 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: DB Session ID: ATWVQ5YR6WG661K6ZRRJ 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: CURRENT file: CURRENT 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 93905 ; 2024-07-26T12:33:26.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.error_if_exists: 0 2024-07-26T12:33:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.create_if_missing: 0 2024-07-26T12:33:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.paranoid_checks: 1 2024-07-26T12:33:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T12:33:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T12:33:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T12:33:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.env: 0x56320de51280 2024-07-26T12:33:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T12:33:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.info_log: 0x56323bd7b060 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.statistics: (nil) 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.use_fsync: 0 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_log_file_size: 0 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.allow_fallocate: 1 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T12:33:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.use_direct_reads: 0 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.db_log_dir: 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.wal_dir: 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T12:33:26.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.write_buffer_manager: 0x56323bd7f5e0 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.unordered_write: 0 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T12:33:26.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.row_cache: None 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.wal_filter: None 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.two_write_queues: 0 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.wal_compression: 0 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.atomic_flush: 0 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T12:33:26.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.log_readahead_size: 0 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_background_jobs: 2 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_background_compactions: -1 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_subcompactions: 1 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T12:33:26.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_open_files: -1 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_background_flushes: -1 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Compression algorithms supported: 2024-07-26T12:33:26.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: kZSTD supported: 0 2024-07-26T12:33:26.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: kXpressCompression supported: 0 2024-07-26T12:33:26.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: kBZip2Compression supported: 0 2024-07-26T12:33:26.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T12:33:26.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: kLZ4Compression supported: 1 2024-07-26T12:33:26.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: kZlibCompression supported: 1 2024-07-26T12:33:26.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T12:33:26.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: kSnappyCompression supported: 1 2024-07-26T12:33:26.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T12:33:26.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T12:33:26.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-07-26T12:33:26.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T12:33:26.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.merge_operator: 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_filter: None 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56323bd7ac00) 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks: 1 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_top_level_index_and_filter: 1 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_type: 0 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_index_type: 0 2024-07-26T12:33:26.301 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_shortening: 1 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: checksum: 4 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: no_block_cache: 0 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache: 0x563237d3f610 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_name: BinnedLRUCache 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_options: 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: capacity : 536870912 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_shard_bits : 4 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: strict_capacity_limit : 0 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: high_pri_pool_ratio: 0.000 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_compressed: (nil) 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: persistent_cache: (nil) 2024-07-26T12:33:26.302 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size: 4096 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size_deviation: 10 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_restart_interval: 16 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_block_restart_interval: 1 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: metadata_block_size: 4096 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: partition_filters: 0 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: use_delta_encoding: 1 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: filter_policy: bloomfilter 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: whole_key_filtering: 1 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: verify_compression: 0 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: read_amp_bytes_per_bit: 0 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: format_version: 5 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_index_compression: 1 2024-07-26T12:33:26.303 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_align: 0 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout: max_auto_readahead_size: 262144 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout: prepopulate_block_cache: 0 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout: initial_auto_readahead_size: 8192 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compression: NoCompression 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.num_levels: 7 2024-07-26T12:33:26.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T12:33:26.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T12:33:26.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T12:33:26.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T12:33:26.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.inplace_update_support: 0 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.bloom_locality: 0 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.max_successive_merges: 0 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.ttl: 2592000 2024-07-26T12:33:26.309 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.enable_blob_files: false 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.min_blob_size: 0 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T12:33:26.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: bdd60a1e-5d28-48ec-b75c-67dab7d76893 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721997206161966, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721997206163586, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90105, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 278, "table_properties": {"data_size": 88175, "index_size": 250, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11355, "raw_average_key_size": 47, "raw_value_size": 81822, "raw_average_value_size": 345, "num_data_blocks": 11, "num_entries": 237, "num_filter_entries": 237, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721997206, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bdd60a1e-5d28-48ec-b75c-67dab7d76893", "db_session_id": "ATWVQ5YR6WG661K6ZRRJ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721997206163780, "job": 1, "event": "recovery_finished"} 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56323bd9ea80 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: DB pointer 0x56323cb3e000 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T12:33:26.311 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** DB Stats ** 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: L0 2/0 89.85 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T12:33:26.312 INFO:journalctl@ceph.mon.a.smithi107.stdout: Sum 2/0 89.85 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 84.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 84.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T12:33:26.313 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative compaction: 0.00 GB write, 12.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval compaction: 0.00 GB write, 12.30 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout: Block cache BinnedLRUCache@0x563237d3f610#2 capacity: 512.00 MB usage: 1.28 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: starting mon.a rank 0 at public addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] at bind addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: mon.a@-1(???) e1 preinit fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: mon.a@-1(???).mds e1 new map 2024-07-26T12:33:26.314 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: mon.a@-1(???).mds e1 print_map 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout: e1 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout: btime 2024-07-26T12:33:09:598123+0000 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout: legacy client fscid: -1 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout: No filesystems configured 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-07-26T12:33:26.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T12:33:26.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: monmap epoch 1 2024-07-26T12:33:26.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:26.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: last_changed 2024-07-26T12:33:05.278670+0000 2024-07-26T12:33:26.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: created 2024-07-26T12:33:05.278670+0000 2024-07-26T12:33:26.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: min_mon_release 19 (squid) 2024-07-26T12:33:26.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: election_strategy: 1 2024-07-26T12:33:26.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-07-26T12:33:26.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: fsmap 2024-07-26T12:33:26.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T12:33:26.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:26 smithi107 ceph-mon[32659]: mgrmap e1: no daemons active 2024-07-26T12:33:27.537 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote config to /etc/ceph/ceph.conf 2024-07-26T12:33:27.560 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-07-26T12:33:27.560 INFO:teuthology.orchestra.run.smithi107.stdout:Creating mgr... 2024-07-26T12:33:27.562 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying port 0.0.0.0:9283 ... 2024-07-26T12:33:27.563 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying port 0.0.0.0:8765 ... 2024-07-26T12:33:27.954 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mgr.x 2024-07-26T12:33:27.954 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to reset failed state of unit ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mgr.x.service: Unit ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mgr.x.service not loaded. 2024-07-26T12:33:28.215 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968.target.wants/ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@.service. 2024-07-26T12:33:28.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:27 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/249567095' entity='client.admin' 2024-07-26T12:33:28.542 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:28 smithi107 systemd[1]: Starting Ceph mgr.x for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:33:28.859 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:28 smithi107 podman[32882]: 2024-07-26 12:33:28.504107859 +0000 UTC m=+0.032141366 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:33:28.860 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:28 smithi107 podman[32882]: 2024-07-26 12:33:28.654092299 +0000 UTC m=+0.182125836 container create eccaf212cc6453441868f9a1d70c62e1e22a46257d43e29472bd24084febe009 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724) 2024-07-26T12:33:28.860 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:28 smithi107 podman[32882]: 2024-07-26 12:33:28.857627828 +0000 UTC m=+0.385661385 container init eccaf212cc6453441868f9a1d70c62e1e22a46257d43e29472bd24084febe009 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:33:28.987 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T12:33:28.988 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-07-26T12:33:29.002 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T12:33:29.002 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-07-26T12:33:29.002 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-07-26T12:33:29.002 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T12:33:29.020 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T12:33:29.020 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-07-26T12:33:29.035 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T12:33:29.035 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-07-26T12:33:29.036 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-07-26T12:33:29.036 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-07-26T12:33:29.036 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr to start... 2024-07-26T12:33:29.036 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr... 2024-07-26T12:33:29.184 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:28 smithi107 podman[32882]: 2024-07-26 12:33:28.867621697 +0000 UTC m=+0.395655238 container start eccaf212cc6453441868f9a1d70c62e1e22a46257d43e29472bd24084febe009 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:33:29.184 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:28 smithi107 bash[32882]: eccaf212cc6453441868f9a1d70c62e1e22a46257d43e29472bd24084febe009 2024-07-26T12:33:29.184 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:28 smithi107 systemd[1]: Started Ceph mgr.x for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:33:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:29 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/803100949' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T12:33:30.467 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:30.467 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-07-26T12:33:30.467 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:33:30.467 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T12:33:30.467 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T12:33:30.467 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T12:33:30.467 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T12:33:30.468 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T12:33:30.469 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T12:33:09:598123+0000", 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T12:33:30.470 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T12:33:09.599176+0000", 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T12:33:30.471 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:30.472 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T12:33:30.472 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-07-26T12:33:30.472 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (1/15)... 2024-07-26T12:33:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:33 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3115695825' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T12:33:33.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:33.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-07-26T12:33:33.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:33:33.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T12:33:33.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T12:33:33.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T12:33:33.907 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:33.908 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T12:33:33.909 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T12:33:33.909 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:33.909 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T12:33:33.909 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:33.909 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T12:33:33.909 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T12:33:33.909 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T12:33:33.909 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T12:33:33.909 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T12:33:33.909 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T12:33:33.912 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T12:33:09:598123+0000", 2024-07-26T12:33:33.913 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T12:33:33.914 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:33.915 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T12:33:09.599176+0000", 2024-07-26T12:33:33.915 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T12:33:33.915 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:33.915 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T12:33:33.915 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-07-26T12:33:33.915 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (2/15)... 2024-07-26T12:33:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: Activating manager daemon x 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: mgrmap e2: x(active, starting, since 0.0127281s) 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: from='mgr.14100 172.21.15.107:0/4133188765' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: from='mgr.14100 172.21.15.107:0/4133188765' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: from='mgr.14100 172.21.15.107:0/4133188765' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: from='mgr.14100 172.21.15.107:0/4133188765' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: from='mgr.14100 172.21.15.107:0/4133188765' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: Manager daemon x is now available 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: from='mgr.14100 172.21.15.107:0/4133188765' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: from='mgr.14100 172.21.15.107:0/4133188765' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: from='mgr.14100 172.21.15.107:0/4133188765' entity='mgr.x' 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: from='mgr.14100 172.21.15.107:0/4133188765' entity='mgr.x' 2024-07-26T12:33:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:35 smithi107 ceph-mon[32659]: from='mgr.14100 172.21.15.107:0/4133188765' entity='mgr.x' 2024-07-26T12:33:37.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:37 smithi107 ceph-mon[32659]: mgrmap e3: x(active, since 1.02125s) 2024-07-26T12:33:37.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:37 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3629915143' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T12:33:37.563 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:37.564 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-07-26T12:33:37.564 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:33:37.564 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T12:33:37.564 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T12:33:37.564 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T12:33:37.564 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T12:33:37.564 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:37.564 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T12:33:37.564 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T12:33:37.565 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-07-26T12:33:37.565 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-07-26T12:33:37.565 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T12:33:37.565 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T12:33:37.565 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-07-26T12:33:37.565 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T12:33:37.566 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T12:33:37.567 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T12:33:09:598123+0000", 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T12:33:37.568 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T12:33:09.599176+0000", 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-07-26T12:33:37.569 INFO:teuthology.orchestra.run.smithi107.stdout:mgr is available 2024-07-26T12:33:38.193 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:38 smithi107 ceph-mon[32659]: mgrmap e4: x(active, since 2s) 2024-07-26T12:33:39.139 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:39 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1606851533' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T12:33:39.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:39.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T12:33:39.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout fsid = 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:39.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T12:33:39.150 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T12:33:39.151 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T12:33:39.152 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T12:33:39.152 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling cephadm module... 2024-07-26T12:33:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:40 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1139311638' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-07-26T12:33:40.292 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:40 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: ignoring --setuser ceph since I am not root 2024-07-26T12:33:40.292 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:40 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: ignoring --setgroup ceph since I am not root 2024-07-26T12:33:41.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:41 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1139311638' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-07-26T12:33:41.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:41 smithi107 ceph-mon[32659]: mgrmap e5: x(active, since 5s) 2024-07-26T12:33:42.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:42 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3339220795' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T12:33:42.242 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-07-26T12:33:42.243 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-07-26T12:33:42.243 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T12:33:42.243 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T12:33:42.243 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T12:33:42.243 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-07-26T12:33:42.243 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for the mgr to restart... 2024-07-26T12:33:42.243 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr epoch 5... 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: Active manager daemon x restarted 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: Activating manager daemon x 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: osdmap e2: 0 total, 0 up, 0 in 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: mgrmap e6: x(active, starting, since 0.00595934s) 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: Manager daemon x is now available 2024-07-26T12:33:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:33:46.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:33:46.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:46 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:47.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:47 smithi107 ceph-mon[32659]: Found migration_current of "None". Setting to last migration. 2024-07-26T12:33:47.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:47 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:47.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:47 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T12:33:47.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:47 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T12:33:47.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:47 smithi107 ceph-mon[32659]: mgrmap e7: x(active, since 1.01758s) 2024-07-26T12:33:47.860 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-07-26T12:33:47.861 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-07-26T12:33:47.861 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T12:33:47.861 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-07-26T12:33:47.861 INFO:teuthology.orchestra.run.smithi107.stdout:mgr epoch 5 is available 2024-07-26T12:33:47.862 INFO:teuthology.orchestra.run.smithi107.stdout:Setting orchestrator backend to cephadm... 2024-07-26T12:33:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-mon[32659]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T12:33:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-mon[32659]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T12:33:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:33:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:33:49.015 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:33:48.643+0000 7f0d4ac00640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:12:33:48] ENGINE Error in HTTPServer.serve 2024-07-26T12:33:49.015 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: Traceback (most recent call last): 2024-07-26T12:33:49.015 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T12:33:49.015 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self._connections.run(self.expiration_interval) 2024-07-26T12:33:49.015 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T12:33:49.015 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self._run(expiration_interval) 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: s = self.context.wrap_socket( 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: return self.sslsocket_class._create( 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self.do_handshake() 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self._sslobj.do_handshake() 2024-07-26T12:33:49.016 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T12:33:49.017 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:33:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:49 smithi107 ceph-mon[32659]: [26/Jul/2024:12:33:48] ENGINE Bus STARTING 2024-07-26T12:33:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:49 smithi107 ceph-mon[32659]: mgrmap e8: x(active, since 2s) 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:49 smithi107 ceph-mon[32659]: [26/Jul/2024:12:33:48] ENGINE Serving on http://172.21.15.107:8765 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:49 smithi107 ceph-mon[32659]: [26/Jul/2024:12:33:48] ENGINE Serving on https://172.21.15.107:7150 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:49 smithi107 ceph-mon[32659]: [26/Jul/2024:12:33:48] ENGINE Bus STARTED 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:49 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:49 smithi107 ceph-mon[32659]: [26/Jul/2024:12:33:48] ENGINE Error in HTTPServer.serve 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout: Traceback (most recent call last): 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._run(expiration_interval) 2024-07-26T12:33:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: s = self.context.wrap_socket( 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: return self.sslsocket_class._create( 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: self.do_handshake() 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._sslobj.do_handshake() 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T12:33:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:49 smithi107 ceph-mon[32659]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:33:49.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:49 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:33:49.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:49 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:50.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:50 smithi107 ceph-mon[32659]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:33:50.726 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout value unchanged 2024-07-26T12:33:50.726 INFO:teuthology.orchestra.run.smithi107.stdout:Generating ssh key... 2024-07-26T12:33:52.043 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: Generating public/private rsa key pair. 2024-07-26T12:33:52.043 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: Your identification has been saved in /tmp/tmp0mxhh982/key 2024-07-26T12:33:52.043 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: Your public key has been saved in /tmp/tmp0mxhh982/key.pub 2024-07-26T12:33:52.043 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: The key fingerprint is: 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: SHA256:BMsUB5NcX9dl2oCeyl1bicaFvyCyoV0XF579kZPVsY8 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: The key's randomart image is: 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: +---[RSA 3072]----+ 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: | .*=o ..++O| 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: | oo= . ..+.BX| 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: | o . ....BO+| 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: | . o oo*.+*| 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: | S.=o+.Eo+| 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: | . oo . .. | 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: | | 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: | | 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: | | 2024-07-26T12:33:52.044 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:33:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: +----[SHA256]-----+ 2024-07-26T12:33:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:52 smithi107 ceph-mon[32659]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:33:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:52 smithi107 ceph-mon[32659]: Generating ssh key... 2024-07-26T12:33:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:52 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:33:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:52 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:33:53.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbixzQLRVlJzxYQfsr1mk+CYip/L1l8vAI93dSC+x6jThv+pm7SQzzfTcR4eLfE4WJRgJAXXRA7YpcxsBUJLo0DGo2Nw+SF+IOFZnv9pVCaJoS037AsJT9WJyLRKxYMU/uBKlcrygwhWT3KwzC/1AaYwA7W6MafY/9EMeex/uiVM1B2oXFJk6MHpl27EJMSwMDao1vGwkRckUqoGZXmC4OwyB3tcBfjDvC9OGS8zZE8FlSvsr5WkAt8TgTG4NmetW7lBGd/tUgLh16IhHcSf245qS2N/Cdm5vE/8JmBhICy95rYrPI1Dfw4n+yxg3yH6MvF5tssEfhb+PaR6fPz6QR/5uzYnmdpmOSxFMBYY70An+PgdAROLcuOcRYCGdQUTL091D1cJXEWqY8Ez2rjjk6vpCQwhrijXzuYUereQaBI5Ig76sggXoMsdAkijE7y0oE6xCIAlyVY/a3M/PifHWF+I/1MF0w9EGxwWPgKFrNHK8f9bTZIznKjtN+WNn7G5k= ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:33:53.995 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-07-26T12:33:53.995 INFO:teuthology.orchestra.run.smithi107.stdout:Adding key to root@localhost authorized_keys... 2024-07-26T12:33:53.997 INFO:teuthology.orchestra.run.smithi107.stdout:Adding host smithi107... 2024-07-26T12:33:54.526 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:54 smithi107 ceph-mon[32659]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:33:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:55 smithi107 ceph-mon[32659]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:33:57.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:33:57 smithi107 ceph-mon[32659]: Deploying cephadm binary to smithi107 2024-07-26T12:34:00.232 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Added host 'smithi107' with addr '172.21.15.107' 2024-07-26T12:34:00.232 INFO:teuthology.orchestra.run.smithi107.stdout:Deploying unmanaged mon service... 2024-07-26T12:34:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:00 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:34:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:00 smithi107 ceph-mon[32659]: Added host smithi107 2024-07-26T12:34:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:00 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:01.780 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-07-26T12:34:01.781 INFO:teuthology.orchestra.run.smithi107.stdout:Deploying unmanaged mgr service... 2024-07-26T12:34:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:02 smithi107 ceph-mon[32659]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:34:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:02 smithi107 ceph-mon[32659]: Saving service mon spec with placement count:5 2024-07-26T12:34:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:02 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:34:03.550 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-07-26T12:34:04.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:03 smithi107 ceph-mon[32659]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:34:04.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:03 smithi107 ceph-mon[32659]: Saving service mgr spec with placement count:2 2024-07-26T12:34:04.117 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:03 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:34:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:05 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:34:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:05 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/62920440' entity='client.admin' 2024-07-26T12:34:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:05 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:34:06.847 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling the dashboard module... 2024-07-26T12:34:07.495 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:07 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/4118411816' entity='client.admin' 2024-07-26T12:34:07.495 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:07 smithi107 ceph-mon[32659]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:08.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:08 smithi107 ceph-mon[32659]: from='mgr.14118 172.21.15.107:0/2382332308' entity='mgr.x' 2024-07-26T12:34:08.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:08 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2858497849' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-07-26T12:34:08.884 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:08 smithi107 ceph-mon[32659]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:08.884 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:08 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: ignoring --setuser ceph since I am not root 2024-07-26T12:34:08.884 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:08 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: ignoring --setgroup ceph since I am not root 2024-07-26T12:34:09.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:09 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2858497849' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-07-26T12:34:09.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:09 smithi107 ceph-mon[32659]: mgrmap e9: x(active, since 22s) 2024-07-26T12:34:10.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:10 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2402709117' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T12:34:10.749 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-07-26T12:34:10.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-07-26T12:34:10.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T12:34:10.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T12:34:10.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T12:34:10.750 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-07-26T12:34:10.751 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for the mgr to restart... 2024-07-26T12:34:10.751 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr epoch 9... 2024-07-26T12:34:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: Active manager daemon x restarted 2024-07-26T12:34:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: Activating manager daemon x 2024-07-26T12:34:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: osdmap e3: 0 total, 0 up, 0 in 2024-07-26T12:34:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: mgrmap e10: x(active, starting, since 0.00662145s) 2024-07-26T12:34:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:34:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T12:34:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T12:34:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T12:34:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T12:34:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: Manager daemon x is now available 2024-07-26T12:34:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:14 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:15.937 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T12:34:15.937 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T12:34:15.937 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-mon[32659]: mgrmap e11: x(active, since 1.01022s) 2024-07-26T12:34:16.188 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:34:15.935+0000 7f3720600640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:12:34:15] ENGINE Error in HTTPServer.serve 2024-07-26T12:34:16.189 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: Traceback (most recent call last): 2024-07-26T12:34:16.189 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T12:34:16.189 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self._connections.run(self.expiration_interval) 2024-07-26T12:34:16.189 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T12:34:16.189 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self._run(expiration_interval) 2024-07-26T12:34:16.190 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T12:34:16.190 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T12:34:16.190 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T12:34:16.190 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T12:34:16.190 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T12:34:16.191 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: s = self.context.wrap_socket( 2024-07-26T12:34:16.191 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T12:34:16.191 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: return self.sslsocket_class._create( 2024-07-26T12:34:16.191 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T12:34:16.191 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self.do_handshake() 2024-07-26T12:34:16.191 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T12:34:16.192 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self._sslobj.do_handshake() 2024-07-26T12:34:16.192 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T12:34:16.192 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:34:15 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:34:16.591 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-07-26T12:34:16.591 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-07-26T12:34:16.591 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T12:34:16.591 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-07-26T12:34:16.591 INFO:teuthology.orchestra.run.smithi107.stdout:mgr epoch 9 is available 2024-07-26T12:34:16.591 INFO:teuthology.orchestra.run.smithi107.stdout:Generating a dashboard self-signed certificate... 2024-07-26T12:34:16.926 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:16 smithi107 ceph-mon[32659]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T12:34:16.926 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:16 smithi107 ceph-mon[32659]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T12:34:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:17 smithi107 ceph-mon[32659]: [26/Jul/2024:12:34:15] ENGINE Bus STARTING 2024-07-26T12:34:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:17 smithi107 ceph-mon[32659]: [26/Jul/2024:12:34:15] ENGINE Serving on https://172.21.15.107:7150 2024-07-26T12:34:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:17 smithi107 ceph-mon[32659]: [26/Jul/2024:12:34:15] ENGINE Error in HTTPServer.serve 2024-07-26T12:34:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout: Traceback (most recent call last): 2024-07-26T12:34:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T12:34:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-07-26T12:34:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._run(expiration_interval) 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: s = self.context.wrap_socket( 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: return self.sslsocket_class._create( 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: self.do_handshake() 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T12:34:18.044 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._sslobj.do_handshake() 2024-07-26T12:34:18.045 INFO:journalctl@ceph.mon.a.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T12:34:18.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:17 smithi107 ceph-mon[32659]: [26/Jul/2024:12:34:16] ENGINE Serving on http://172.21.15.107:8765 2024-07-26T12:34:18.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:17 smithi107 ceph-mon[32659]: [26/Jul/2024:12:34:16] ENGINE Bus STARTED 2024-07-26T12:34:18.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:17 smithi107 ceph-mon[32659]: mgrmap e12: x(active, since 2s) 2024-07-26T12:34:18.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:17 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:18.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:17 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:18.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:17 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:18.341 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-07-26T12:34:18.342 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial admin user... 2024-07-26T12:34:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:18 smithi107 ceph-mon[32659]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:34:20.179 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$DH9dytjhUcIsDN33RdfixeeFPs2tkBXvM6hZ7U3HcXdJlTPMU5X..", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1721997259, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-07-26T12:34:20.179 INFO:teuthology.orchestra.run.smithi107.stdout:Fetching dashboard port number... 2024-07-26T12:34:20.465 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:20 smithi107 ceph-mon[32659]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:34:20.465 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:20 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:20.465 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:20 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:21 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1135240627' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-07-26T12:34:22.129 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 8443 2024-07-26T12:34:22.159 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T12:34:22.159 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-07-26T12:34:22.182 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T12:34:22.182 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-07-26T12:34:22.182 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-07-26T12:34:22.182 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-07-26T12:34:22.186 INFO:teuthology.orchestra.run.smithi107.stdout:Ceph Dashboard is now available at: 2024-07-26T12:34:22.186 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:22.186 INFO:teuthology.orchestra.run.smithi107.stdout: URL: https://smithi107.front.sepia.ceph.com:8443/ 2024-07-26T12:34:22.187 INFO:teuthology.orchestra.run.smithi107.stdout: User: admin 2024-07-26T12:34:22.187 INFO:teuthology.orchestra.run.smithi107.stdout: Password: gkns5fh24v 2024-07-26T12:34:22.187 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:22.187 INFO:teuthology.orchestra.run.smithi107.stdout:Saving cluster configuration to /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config directory 2024-07-26T12:34:22.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:22 smithi107 ceph-mon[32659]: mgrmap e13: x(active, since 6s) 2024-07-26T12:34:24.175 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-07-26T12:34:24.175 INFO:teuthology.orchestra.run.smithi107.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-07-26T12:34:24.175 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:24.175 INFO:teuthology.orchestra.run.smithi107.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-07-26T12:34:24.175 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:24.175 INFO:teuthology.orchestra.run.smithi107.stdout:Or, if you are only running a single cluster on this host: 2024-07-26T12:34:24.175 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:24.176 INFO:teuthology.orchestra.run.smithi107.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-07-26T12:34:24.176 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:24.176 INFO:teuthology.orchestra.run.smithi107.stdout:Please consider enabling telemetry to help improve Ceph: 2024-07-26T12:34:24.176 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:24.176 INFO:teuthology.orchestra.run.smithi107.stdout: ceph telemetry on 2024-07-26T12:34:24.176 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:24.176 INFO:teuthology.orchestra.run.smithi107.stdout:For more information see: 2024-07-26T12:34:24.176 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:24.177 INFO:teuthology.orchestra.run.smithi107.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-07-26T12:34:24.177 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:24.177 INFO:teuthology.orchestra.run.smithi107.stdout:Bootstrap complete. 2024-07-26T12:34:24.252 INFO:tasks.cephadm:Fetching config... 2024-07-26T12:34:24.253 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:34:24.253 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-07-26T12:34:24.303 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-07-26T12:34:24.304 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:34:24.304 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-07-26T12:34:24.348 INFO:tasks.cephadm:Fetching mon keyring... 2024-07-26T12:34:24.348 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:34:24.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/keyring of=/dev/stdout 2024-07-26T12:34:24.451 INFO:tasks.cephadm:Fetching pub ssh key... 2024-07-26T12:34:24.451 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:34:24.451 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-07-26T12:34:24.550 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-07-26T12:34:24.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbixzQLRVlJzxYQfsr1mk+CYip/L1l8vAI93dSC+x6jThv+pm7SQzzfTcR4eLfE4WJRgJAXXRA7YpcxsBUJLo0DGo2Nw+SF+IOFZnv9pVCaJoS037AsJT9WJyLRKxYMU/uBKlcrygwhWT3KwzC/1AaYwA7W6MafY/9EMeex/uiVM1B2oXFJk6MHpl27EJMSwMDao1vGwkRckUqoGZXmC4OwyB3tcBfjDvC9OGS8zZE8FlSvsr5WkAt8TgTG4NmetW7lBGd/tUgLh16IhHcSf245qS2N/Cdm5vE/8JmBhICy95rYrPI1Dfw4n+yxg3yH6MvF5tssEfhb+PaR6fPz6QR/5uzYnmdpmOSxFMBYY70An+PgdAROLcuOcRYCGdQUTL091D1cJXEWqY8Ez2rjjk6vpCQwhrijXzuYUereQaBI5Ig76sggXoMsdAkijE7y0oE6xCIAlyVY/a3M/PifHWF+I/1MF0w9EGxwWPgKFrNHK8f9bTZIznKjtN+WNn7G5k= ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T12:34:24.660 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbixzQLRVlJzxYQfsr1mk+CYip/L1l8vAI93dSC+x6jThv+pm7SQzzfTcR4eLfE4WJRgJAXXRA7YpcxsBUJLo0DGo2Nw+SF+IOFZnv9pVCaJoS037AsJT9WJyLRKxYMU/uBKlcrygwhWT3KwzC/1AaYwA7W6MafY/9EMeex/uiVM1B2oXFJk6MHpl27EJMSwMDao1vGwkRckUqoGZXmC4OwyB3tcBfjDvC9OGS8zZE8FlSvsr5WkAt8TgTG4NmetW7lBGd/tUgLh16IhHcSf245qS2N/Cdm5vE/8JmBhICy95rYrPI1Dfw4n+yxg3yH6MvF5tssEfhb+PaR6fPz6QR/5uzYnmdpmOSxFMBYY70An+PgdAROLcuOcRYCGdQUTL091D1cJXEWqY8Ez2rjjk6vpCQwhrijXzuYUereQaBI5Ig76sggXoMsdAkijE7y0oE6xCIAlyVY/a3M/PifHWF+I/1MF0w9EGxwWPgKFrNHK8f9bTZIznKjtN+WNn7G5k= ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:34:24.689 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbixzQLRVlJzxYQfsr1mk+CYip/L1l8vAI93dSC+x6jThv+pm7SQzzfTcR4eLfE4WJRgJAXXRA7YpcxsBUJLo0DGo2Nw+SF+IOFZnv9pVCaJoS037AsJT9WJyLRKxYMU/uBKlcrygwhWT3KwzC/1AaYwA7W6MafY/9EMeex/uiVM1B2oXFJk6MHpl27EJMSwMDao1vGwkRckUqoGZXmC4OwyB3tcBfjDvC9OGS8zZE8FlSvsr5WkAt8TgTG4NmetW7lBGd/tUgLh16IhHcSf245qS2N/Cdm5vE/8JmBhICy95rYrPI1Dfw4n+yxg3yH6MvF5tssEfhb+PaR6fPz6QR/5uzYnmdpmOSxFMBYY70An+PgdAROLcuOcRYCGdQUTL091D1cJXEWqY8Ez2rjjk6vpCQwhrijXzuYUereQaBI5Ig76sggXoMsdAkijE7y0oE6xCIAlyVY/a3M/PifHWF+I/1MF0w9EGxwWPgKFrNHK8f9bTZIznKjtN+WNn7G5k= ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T12:34:24.732 INFO:teuthology.orchestra.run.smithi112.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbixzQLRVlJzxYQfsr1mk+CYip/L1l8vAI93dSC+x6jThv+pm7SQzzfTcR4eLfE4WJRgJAXXRA7YpcxsBUJLo0DGo2Nw+SF+IOFZnv9pVCaJoS037AsJT9WJyLRKxYMU/uBKlcrygwhWT3KwzC/1AaYwA7W6MafY/9EMeex/uiVM1B2oXFJk6MHpl27EJMSwMDao1vGwkRckUqoGZXmC4OwyB3tcBfjDvC9OGS8zZE8FlSvsr5WkAt8TgTG4NmetW7lBGd/tUgLh16IhHcSf245qS2N/Cdm5vE/8JmBhICy95rYrPI1Dfw4n+yxg3yH6MvF5tssEfhb+PaR6fPz6QR/5uzYnmdpmOSxFMBYY70An+PgdAROLcuOcRYCGdQUTL091D1cJXEWqY8Ez2rjjk6vpCQwhrijXzuYUereQaBI5Ig76sggXoMsdAkijE7y0oE6xCIAlyVY/a3M/PifHWF+I/1MF0w9EGxwWPgKFrNHK8f9bTZIznKjtN+WNn7G5k= ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:34:24.743 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbixzQLRVlJzxYQfsr1mk+CYip/L1l8vAI93dSC+x6jThv+pm7SQzzfTcR4eLfE4WJRgJAXXRA7YpcxsBUJLo0DGo2Nw+SF+IOFZnv9pVCaJoS037AsJT9WJyLRKxYMU/uBKlcrygwhWT3KwzC/1AaYwA7W6MafY/9EMeex/uiVM1B2oXFJk6MHpl27EJMSwMDao1vGwkRckUqoGZXmC4OwyB3tcBfjDvC9OGS8zZE8FlSvsr5WkAt8TgTG4NmetW7lBGd/tUgLh16IhHcSf245qS2N/Cdm5vE/8JmBhICy95rYrPI1Dfw4n+yxg3yH6MvF5tssEfhb+PaR6fPz6QR/5uzYnmdpmOSxFMBYY70An+PgdAROLcuOcRYCGdQUTL091D1cJXEWqY8Ez2rjjk6vpCQwhrijXzuYUereQaBI5Ig76sggXoMsdAkijE7y0oE6xCIAlyVY/a3M/PifHWF+I/1MF0w9EGxwWPgKFrNHK8f9bTZIznKjtN+WNn7G5k= ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T12:34:24.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:24 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2143336283' entity='client.admin' 2024-07-26T12:34:24.783 INFO:teuthology.orchestra.run.smithi132.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbixzQLRVlJzxYQfsr1mk+CYip/L1l8vAI93dSC+x6jThv+pm7SQzzfTcR4eLfE4WJRgJAXXRA7YpcxsBUJLo0DGo2Nw+SF+IOFZnv9pVCaJoS037AsJT9WJyLRKxYMU/uBKlcrygwhWT3KwzC/1AaYwA7W6MafY/9EMeex/uiVM1B2oXFJk6MHpl27EJMSwMDao1vGwkRckUqoGZXmC4OwyB3tcBfjDvC9OGS8zZE8FlSvsr5WkAt8TgTG4NmetW7lBGd/tUgLh16IhHcSf245qS2N/Cdm5vE/8JmBhICy95rYrPI1Dfw4n+yxg3yH6MvF5tssEfhb+PaR6fPz6QR/5uzYnmdpmOSxFMBYY70An+PgdAROLcuOcRYCGdQUTL091D1cJXEWqY8Ez2rjjk6vpCQwhrijXzuYUereQaBI5Ig76sggXoMsdAkijE7y0oE6xCIAlyVY/a3M/PifHWF+I/1MF0w9EGxwWPgKFrNHK8f9bTZIznKjtN+WNn7G5k= ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:34:24.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-07-26T12:34:25.221 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:34:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:28 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:28 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:28 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:34:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:28 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:28 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:28 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:28 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/673478977' entity='client.admin' 2024-07-26T12:34:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:28 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:28 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:28.684 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-07-26T12:34:28.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-07-26T12:34:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:34:32.065 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi112 2024-07-26T12:34:32.065 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:34:32.066 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.conf 2024-07-26T12:34:32.084 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:34:32.084 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:34:32.142 INFO:tasks.cephadm:Adding host smithi112 to orchestrator... 2024-07-26T12:34:32.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch host add smithi112 2024-07-26T12:34:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:32 smithi107 ceph-mon[32659]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:34:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:32 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:32 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:32 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:32 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:32 smithi107 ceph-mon[32659]: Updating smithi107:/etc/ceph/ceph.conf 2024-07-26T12:34:32.565 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:34:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:33 smithi107 ceph-mon[32659]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:34:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:33 smithi107 ceph-mon[32659]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:34:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:34 smithi107 ceph-mon[32659]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:34:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:34 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:34 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:34 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:35 smithi107 ceph-mon[32659]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:36 smithi107 ceph-mon[32659]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi112", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:34:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:37 smithi107 ceph-mon[32659]: Deploying cephadm binary to smithi112 2024-07-26T12:34:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:37 smithi107 ceph-mon[32659]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:38.521 INFO:teuthology.orchestra.run.smithi107.stdout:Added host 'smithi112' with addr '172.21.15.112' 2024-07-26T12:34:39.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T12:34:39.721 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:34:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:39 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:39 smithi107 ceph-mon[32659]: Added host smithi112 2024-07-26T12:34:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:39 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:39 smithi107 ceph-mon[32659]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:39 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:39 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:41.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:40 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:41.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:40 smithi107 ceph-mon[32659]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:42.044 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:42.044 INFO:teuthology.orchestra.run.smithi107.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.112", "hostname": "smithi112", "labels": [], "status": ""}] 2024-07-26T12:34:42.753 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi132 2024-07-26T12:34:42.754 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:34:42.754 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.conf 2024-07-26T12:34:42.772 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:34:42.772 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:34:42.829 INFO:tasks.cephadm:Adding host smithi132 to orchestrator... 2024-07-26T12:34:42.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch host add smithi132 2024-07-26T12:34:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:34:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:43 smithi107 ceph-mon[32659]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:34:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:43 smithi107 ceph-mon[32659]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:43 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:43 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:43 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:43 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:43 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:34:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:43 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:43 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:43 smithi107 ceph-mon[32659]: Updating smithi112:/etc/ceph/ceph.conf 2024-07-26T12:34:45.000 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:44 smithi107 ceph-mon[32659]: Updating smithi112:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:34:45.000 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:44 smithi107 ceph-mon[32659]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:34:45.001 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:44 smithi107 ceph-mon[32659]: Updating smithi112:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:34:45.001 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:44 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:45.001 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:44 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:45.001 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:44 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:45 smithi107 ceph-mon[32659]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:46 smithi107 ceph-mon[32659]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:34:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:47 smithi107 ceph-mon[32659]: Deploying cephadm binary to smithi132 2024-07-26T12:34:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:47 smithi107 ceph-mon[32659]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:49.149 INFO:teuthology.orchestra.run.smithi107.stdout:Added host 'smithi132' with addr '172.21.15.132' 2024-07-26T12:34:49.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T12:34:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:49 smithi107 ceph-mon[32659]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:49 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:49 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:49 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:50.258 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:34:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:50 smithi107 ceph-mon[32659]: Added host smithi132 2024-07-26T12:34:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:50 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:50 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:51.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:51 smithi107 ceph-mon[32659]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:34:52.416 INFO:teuthology.orchestra.run.smithi107.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.112", "hostname": "smithi112", "labels": [], "status": ""}, {"addr": "172.21.15.132", "hostname": "smithi132", "labels": [], "status": ""}] 2024-07-26T12:34:53.155 INFO:tasks.cephadm:Setting crush tunables to default 2024-07-26T12:34:53.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd crush tunables default 2024-07-26T12:34:53.610 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:34:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:53 smithi107 ceph-mon[32659]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:34:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:53 smithi107 ceph-mon[32659]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:53 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:53 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:53 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:53 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:53 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:34:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:53 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:53 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:54 smithi107 ceph-mon[32659]: Updating smithi132:/etc/ceph/ceph.conf 2024-07-26T12:34:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:54 smithi107 ceph-mon[32659]: Updating smithi132:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:34:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:54 smithi107 ceph-mon[32659]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:34:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:54 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:54 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:54 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:34:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:55 smithi107 ceph-mon[32659]: Updating smithi132:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:34:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:55 smithi107 ceph-mon[32659]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:56.789 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2024-07-26T12:34:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:56 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/800085700' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T12:34:57.482 INFO:tasks.cephadm:Adding mon.a on smithi107 2024-07-26T12:34:57.482 INFO:tasks.cephadm:Adding mon.b on smithi112 2024-07-26T12:34:57.482 INFO:tasks.cephadm:Adding mon.c on smithi132 2024-07-26T12:34:57.482 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch apply mon '3;smithi107:172.21.15.107=a;smithi112:172.21.15.112=b;smithi132:172.21.15.132=c' 2024-07-26T12:34:57.733 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:34:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:57 smithi107 ceph-mon[32659]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:57 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/800085700' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T12:34:58.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:57 smithi107 ceph-mon[32659]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T12:34:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:34:58 smithi107 ceph-mon[32659]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:34:59.383 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mon update... 2024-07-26T12:34:59.947 DEBUG:teuthology.orchestra.run.smithi112:mon.b> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mon.b.service 2024-07-26T12:34:59.950 DEBUG:teuthology.orchestra.run.smithi132:mon.c> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mon.c.service 2024-07-26T12:34:59.953 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T12:34:59.953 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T12:35:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:00 smithi107 ceph-mon[32659]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi112:172.21.15.112=b;smithi132:172.21.15.132=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:35:00.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:00 smithi107 ceph-mon[32659]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi112:172.21.15.112=b;smithi132:172.21.15.132=c;count:3 2024-07-26T12:35:00.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:00 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:00.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:00 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:00.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:00 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:00.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:00 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:00.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:00 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:00.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:00 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:00.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:00 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:00.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:00 smithi107 ceph-mon[32659]: Deploying daemon mon.c on smithi132 2024-07-26T12:35:01.057 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:01 smithi107 ceph-mon[32659]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:03 smithi132 systemd[1]: Starting Ceph mon.c for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:35:04.012 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:35:04.012 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":1,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","modified":"2024-07-26T12:33:05.278670Z","created":"2024-07-26T12:33:05.278670Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-07-26T12:35:04.012 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 1 2024-07-26T12:35:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:03 smithi107 ceph-mon[32659]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:03 smithi132 podman[37544]: 2024-07-26 12:35:03.958047349 +0000 UTC m=+0.097678645 container create b4799022bd95dd7681f20f15c893d49052d83693fc78df7f60cec7812ef768cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-07-26T12:35:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:03 smithi132 podman[37544]: 2024-07-26 12:35:03.877802752 +0000 UTC m=+0.017434050 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:35:04.856 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 podman[37544]: 2024-07-26 12:35:04.781821023 +0000 UTC m=+0.921452311 container init b4799022bd95dd7681f20f15c893d49052d83693fc78df7f60cec7812ef768cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-c, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T12:35:04.856 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 podman[37544]: 2024-07-26 12:35:04.785500268 +0000 UTC m=+0.925131569 container start b4799022bd95dd7681f20f15c893d49052d83693fc78df7f60cec7812ef768cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mon-c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-07-26T12:35:04.856 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T12:35:04.856 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pidfile_write: ignore empty --pid-file 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: load: jerasure load: lrc 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: RocksDB version: 7.9.2 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Git sha 0 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: DB SUMMARY 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: DB Session ID: 3K65H37GYAJXRRNIUEZP 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: CURRENT file: CURRENT 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-07-26T12:35:04.857 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.error_if_exists: 0 2024-07-26T12:35:04.858 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.create_if_missing: 0 2024-07-26T12:35:04.858 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.paranoid_checks: 1 2024-07-26T12:35:04.858 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T12:35:04.858 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T12:35:04.858 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T12:35:04.858 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.env: 0x55d9c382d280 2024-07-26T12:35:04.858 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T12:35:04.858 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.info_log: 0x55d9c8407080 2024-07-26T12:35:04.858 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T12:35:04.859 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.statistics: (nil) 2024-07-26T12:35:04.859 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.use_fsync: 0 2024-07-26T12:35:04.859 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_log_file_size: 0 2024-07-26T12:35:04.859 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T12:35:04.859 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.allow_fallocate: 1 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.use_direct_reads: 0 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.db_log_dir: 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.wal_dir: 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T12:35:04.860 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.write_buffer_manager: 0x55d9c840b5e0 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T12:35:04.861 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.unordered_write: 0 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.row_cache: None 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.wal_filter: None 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T12:35:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.two_write_queues: 0 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.wal_compression: 0 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.atomic_flush: 0 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.log_readahead_size: 0 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T12:35:04.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_background_jobs: 2 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_background_compactions: -1 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_subcompactions: 1 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T12:35:04.864 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_open_files: -1 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_background_flushes: -1 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Compression algorithms supported: 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: kZSTD supported: 0 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: kXpressCompression supported: 0 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: kBZip2Compression supported: 0 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T12:35:04.865 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: kLZ4Compression supported: 1 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: kZlibCompression supported: 1 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: kSnappyCompression supported: 1 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.merge_operator: 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_filter: None 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T12:35:04.866 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d9c8406c80) 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: cache_index_and_filter_blocks: 1 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: pin_top_level_index_and_filter: 1 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: index_type: 0 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: data_block_index_type: 0 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: index_shortening: 1 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: checksum: 4 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: no_block_cache: 0 2024-07-26T12:35:04.867 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_cache: 0x55d9c43cb610 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_cache_name: BinnedLRUCache 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_cache_options: 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: capacity : 536870912 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: num_shard_bits : 4 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: strict_capacity_limit : 0 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: high_pri_pool_ratio: 0.000 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_cache_compressed: (nil) 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: persistent_cache: (nil) 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_size: 4096 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_size_deviation: 10 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_restart_interval: 16 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: index_block_restart_interval: 1 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: metadata_block_size: 4096 2024-07-26T12:35:04.868 INFO:journalctl@ceph.mon.c.smithi132.stdout: partition_filters: 0 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: use_delta_encoding: 1 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: filter_policy: bloomfilter 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: whole_key_filtering: 1 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: verify_compression: 0 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: read_amp_bytes_per_bit: 0 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: format_version: 5 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: enable_index_compression: 1 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: block_align: 0 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: max_auto_readahead_size: 262144 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: prepopulate_block_cache: 0 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: initial_auto_readahead_size: 8192 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T12:35:04.869 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compression: NoCompression 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.num_levels: 7 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T12:35:04.870 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T12:35:04.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T12:35:04.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T12:35:04.873 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.inplace_update_support: 0 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.bloom_locality: 0 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.max_successive_merges: 0 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T12:35:04.874 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T12:35:04.875 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T12:35:04.875 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T12:35:04.875 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.ttl: 2592000 2024-07-26T12:35:04.875 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T12:35:04.875 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.enable_blob_files: false 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.min_blob_size: 0 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T12:35:04.876 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 461c677e-8cf1-4f91-838b-93b243e22a93 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721997304824170, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721997304824611, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721997304, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "461c677e-8cf1-4f91-838b-93b243e22a93", "db_session_id": "3K65H37GYAJXRRNIUEZP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721997304824715, "job": 1, "event": "recovery_finished"} 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d9c8428a80 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: DB pointer 0x55d9c91c8000 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T12:35:04.877 INFO:journalctl@ceph.mon.c.smithi132.stdout: ** DB Stats ** 2024-07-26T12:35:04.878 INFO:journalctl@ceph.mon.c.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T12:35:04.878 INFO:journalctl@ceph.mon.c.smithi132.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T12:35:04.878 INFO:journalctl@ceph.mon.c.smithi132.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T12:35:04.878 INFO:journalctl@ceph.mon.c.smithi132.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T12:35:04.878 INFO:journalctl@ceph.mon.c.smithi132.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T12:35:04.878 INFO:journalctl@ceph.mon.c.smithi132.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T12:35:04.878 INFO:journalctl@ceph.mon.c.smithi132.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T12:35:04.878 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: ** Compaction Stats [default] ** 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: ** Compaction Stats [default] ** 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-07-26T12:35:04.879 INFO:journalctl@ceph.mon.c.smithi132.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: Block cache BinnedLRUCache@0x55d9c43cb610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-07-26T12:35:04.880 INFO:journalctl@ceph.mon.c.smithi132.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: using public_addr v2:172.21.15.132:0/0 -> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: starting mon.c rank -1 at public addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] at bind addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(???) e0 preinit fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).mds e1 new map 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).mds e1 print_map 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout: e1 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout: btime 2024-07-26T12:33:09:598123+0000 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout: legacy client fscid: -1 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout: 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout: No filesystems configured 2024-07-26T12:35:04.881 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 bash[37544]: b4799022bd95dd7681f20f15c893d49052d83693fc78df7f60cec7812ef768cd 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.882 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Updating smithi112:/etc/ceph/ceph.conf 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Updating smithi112:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Updating smithi112:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.883 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Deploying cephadm binary to smithi132 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Added host smithi132 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Updating smithi132:/etc/ceph/ceph.conf 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Updating smithi132:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Updating smithi132:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:04.885 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/800085700' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/800085700' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi112:172.21.15.112=b;smithi132:172.21.15.132=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi112:172.21.15.112=b;smithi132:172.21.15.132=c;count:3 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:04.886 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:04.887 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:04.887 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: Deploying daemon mon.c on smithi132 2024-07-26T12:35:04.887 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.887 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:04.887 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: from='client.? 172.21.15.132:0/3845833312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T12:35:04.887 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 ceph-mon[37580]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-07-26T12:35:04.887 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:04 smithi132 systemd[1]: Started Ceph mon.c for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:35:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:04 smithi107 ceph-mon[32659]: from='client.? 172.21.15.132:0/3845833312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T12:35:05.634 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T12:35:05.634 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T12:35:05.885 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:35:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:35:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: mon.a calling monitor election 2024-07-26T12:35:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: mon.c calling monitor election 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: monmap epoch 2 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: last_changed 2024-07-26T12:35:04.863322+0000 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: created 2024-07-26T12:33:05.278670+0000 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: min_mon_release 19 (squid) 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: election_strategy: 1 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: fsmap 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: mgrmap e13: x(active, since 55s) 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: overall HEALTH_OK 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:10.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:09 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:35:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: mon.a calling monitor election 2024-07-26T12:35:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: mon.c calling monitor election 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: monmap epoch 2 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: last_changed 2024-07-26T12:35:04.863322+0000 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: created 2024-07-26T12:33:05.278670+0000 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: min_mon_release 19 (squid) 2024-07-26T12:35:10.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: election_strategy: 1 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: fsmap 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: mgrmap e13: x(active, since 55s) 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: overall HEALTH_OK 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:10.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:09 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:10.640 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:35:10.641 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":2,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","modified":"2024-07-26T12:35:04.863322Z","created":"2024-07-26T12:33:05.278670Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T12:35:10.641 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 2 2024-07-26T12:35:11.218 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:10 smithi132 ceph-mon[37580]: Deploying daemon mon.b on smithi112 2024-07-26T12:35:11.218 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:10 smithi132 ceph-mon[37580]: from='client.? 172.21.15.132:0/1659609340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T12:35:11.218 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:10 smithi132 ceph-mon[37580]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:11.218 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:10 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:10 smithi107 ceph-mon[32659]: Deploying daemon mon.b on smithi112 2024-07-26T12:35:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:10 smithi107 ceph-mon[32659]: from='client.? 172.21.15.132:0/1659609340' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T12:35:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:10 smithi107 ceph-mon[32659]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:10 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:12.245 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T12:35:12.279 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T12:35:12.293 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:11 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:35:11.864+0000 7f3760c00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-07-26T12:35:12.536 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:35:13.757 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:13 smithi132 ceph-mon[37580]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:13 smithi107 ceph-mon[32659]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:13 smithi112 systemd[1]: Started Ceph mon.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:35:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: mon.c calling monitor election 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: mon.a calling monitor election 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: monmap epoch 3 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: last_changed 2024-07-26T12:35:13.843847+0000 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: created 2024-07-26T12:33:05.278670+0000 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: min_mon_release 19 (squid) 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: election_strategy: 1 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: fsmap 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: mgrmap e13: x(active, since 64s) 2024-07-26T12:35:19.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: overall HEALTH_OK 2024-07-26T12:35:19.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:19.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:19.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:19.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:18 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: mon.c calling monitor election 2024-07-26T12:35:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:35:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: mon.a calling monitor election 2024-07-26T12:35:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: monmap epoch 3 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: last_changed 2024-07-26T12:35:13.843847+0000 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: created 2024-07-26T12:33:05.278670+0000 2024-07-26T12:35:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: min_mon_release 19 (squid) 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: election_strategy: 1 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: fsmap 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: mgrmap e13: x(active, since 64s) 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: overall HEALTH_OK 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:19.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:18 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:19 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:19 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:19 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:20.198 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-07-26T12:35:20.198 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":3,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","modified":"2024-07-26T12:35:13.843847Z","created":"2024-07-26T12:33:05.278670Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:3300","nonce":0},{"type":"v1","addr":"172.21.15.112:6789","nonce":0}]},"addr":"172.21.15.112:6789/0","public_addr":"172.21.15.112:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T12:35:20.198 INFO:teuthology.orchestra.run.smithi132.stderr:dumped monmap epoch 3 2024-07-26T12:35:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:19 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:19 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:19 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:20.927 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-07-26T12:35:20.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config generate-minimal-conf 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: mon.c calling monitor election 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: mon.a calling monitor election 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: monmap epoch 3 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: last_changed 2024-07-26T12:35:13.843847+0000 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: created 2024-07-26T12:33:05.278670+0000 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: min_mon_release 19 (squid) 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: election_strategy: 1 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: fsmap 2024-07-26T12:35:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T12:35:21.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: mgrmap e13: x(active, since 64s) 2024-07-26T12:35:21.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: overall HEALTH_OK 2024-07-26T12:35:21.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:21.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:21.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:21.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:21.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:21.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:21.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:20 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:21.341 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: mon.b calling monitor election 2024-07-26T12:35:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: mon.b calling monitor election 2024-07-26T12:35:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: mon.c calling monitor election 2024-07-26T12:35:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: mon.a calling monitor election 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: monmap epoch 3 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: last_changed 2024-07-26T12:35:13.843847+0000 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: created 2024-07-26T12:33:05.278670+0000 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: min_mon_release 19 (squid) 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: election_strategy: 1 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: fsmap 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T12:35:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: mgrmap e13: x(active, since 66s) 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: overall HEALTH_OK 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T12:35:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:22.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:21 smithi132 ceph-mon[37580]: Reconfiguring daemon mon.a on smithi107 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: mon.b calling monitor election 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: mon.b calling monitor election 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: mon.c calling monitor election 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: mon.a calling monitor election 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: monmap epoch 3 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: last_changed 2024-07-26T12:35:13.843847+0000 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: created 2024-07-26T12:33:05.278670+0000 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: min_mon_release 19 (squid) 2024-07-26T12:35:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: election_strategy: 1 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: fsmap 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: mgrmap e13: x(active, since 66s) 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: overall HEALTH_OK 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T12:35:22.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:22.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T12:35:22.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:22.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:21 smithi112 ceph-mon[37501]: Reconfiguring daemon mon.a on smithi107 2024-07-26T12:35:22.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: mon.b calling monitor election 2024-07-26T12:35:22.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: mon.b calling monitor election 2024-07-26T12:35:22.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: mon.c calling monitor election 2024-07-26T12:35:22.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: mon.a calling monitor election 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: monmap epoch 3 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: last_changed 2024-07-26T12:35:13.843847+0000 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: created 2024-07-26T12:33:05.278670+0000 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: min_mon_release 19 (squid) 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: election_strategy: 1 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: 1: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.c 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: 2: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.b 2024-07-26T12:35:22.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: fsmap 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: mgrmap e13: x(active, since 66s) 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: overall HEALTH_OK 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:22.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T12:35:22.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:22.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T12:35:22.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:22.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:21 smithi107 ceph-mon[32659]: Reconfiguring daemon mon.a on smithi107 2024-07-26T12:35:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:22 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:22 smithi132 ceph-mon[37580]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:23.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:22 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:23.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:22 smithi107 ceph-mon[32659]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:23.073 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:22 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:35:22.844+0000 7f3760c00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-07-26T12:35:23.186 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:22 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:23.186 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:22 smithi112 ceph-mon[37501]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:23 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/4061362005' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:24.162 INFO:teuthology.orchestra.run.smithi107.stdout:# minimal ceph.conf for 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:24.163 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-07-26T12:35:24.163 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:35:24.163 INFO:teuthology.orchestra.run.smithi107.stdout: mon_host = [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] 2024-07-26T12:35:24.163 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:23 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/4061362005' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:23 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/4061362005' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:24.918 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-07-26T12:35:24.918 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:35:24.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T12:35:25.004 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:35:25.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:25.132 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:35:25.133 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T12:35:25.160 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:35:25.160 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:25.224 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:35:25.224 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T12:35:25.255 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:35:25.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:25.322 INFO:tasks.cephadm:enabling module snap_schedule 2024-07-26T12:35:25.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- sudo ceph --cluster ceph mgr module enable snap_schedule 2024-07-26T12:35:25.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:25 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:25.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:25 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:25.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:25 smithi107 ceph-mon[32659]: Reconfiguring mon.b (monmap changed)... 2024-07-26T12:35:25.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:25 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:25.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:25 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T12:35:25.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:25 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:25.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:25 smithi107 ceph-mon[32659]: Reconfiguring daemon mon.b on smithi112 2024-07-26T12:35:25.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:25 smithi107 ceph-mon[32659]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:25.418 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:25 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:25.419 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:25 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:25.419 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:25 smithi112 ceph-mon[37501]: Reconfiguring mon.b (monmap changed)... 2024-07-26T12:35:25.419 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:25 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:25.419 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:25 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T12:35:25.419 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:25 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:25.419 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:25 smithi112 ceph-mon[37501]: Reconfiguring daemon mon.b on smithi112 2024-07-26T12:35:25.419 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:25 smithi112 ceph-mon[37501]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:25 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:25 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:25 smithi132 ceph-mon[37580]: Reconfiguring mon.b (monmap changed)... 2024-07-26T12:35:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:25 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:25 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T12:35:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:25 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:25 smithi132 ceph-mon[37580]: Reconfiguring daemon mon.b on smithi112 2024-07-26T12:35:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:25 smithi132 ceph-mon[37580]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:25.753 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:26.733 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:26 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:26.734 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:26 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:26.734 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:26 smithi107 ceph-mon[32659]: Reconfiguring mon.c (monmap changed)... 2024-07-26T12:35:26.734 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:26 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:26.734 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:26 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T12:35:26.734 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:26 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:26.734 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:26 smithi107 ceph-mon[32659]: Reconfiguring daemon mon.c on smithi132 2024-07-26T12:35:26.740 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:26 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:26.740 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:26 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:26.740 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:26 smithi132 ceph-mon[37580]: Reconfiguring mon.c (monmap changed)... 2024-07-26T12:35:26.740 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:26 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:26.740 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:26 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T12:35:26.740 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:26 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:26.740 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:26 smithi132 ceph-mon[37580]: Reconfiguring daemon mon.c on smithi132 2024-07-26T12:35:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:26 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:26 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:26 smithi112 ceph-mon[37501]: Reconfiguring mon.c (monmap changed)... 2024-07-26T12:35:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:26 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T12:35:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:26 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T12:35:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:26 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:26 smithi112 ceph-mon[37501]: Reconfiguring daemon mon.c on smithi132 2024-07-26T12:35:27.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:27 smithi107 ceph-mon[32659]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:27.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:27 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:27.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:27 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:27.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:27 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:27.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:27 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:27.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:27 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:27.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:27 smithi107 ceph-mon[32659]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:27 smithi132 ceph-mon[37580]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:27 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:27 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:27 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:27 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:27 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:27 smithi132 ceph-mon[37580]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:27 smithi112 ceph-mon[37501]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:27 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:27 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:27 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:27 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:27 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:27 smithi112 ceph-mon[37501]: from='mgr.14150 172.21.15.107:0/1930004008' entity='mgr.x' 2024-07-26T12:35:28.599 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:28 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3102945008' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T12:35:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:28 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3102945008' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T12:35:28.792 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:28 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: ignoring --setuser ceph since I am not root 2024-07-26T12:35:28.792 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:28 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: ignoring --setgroup ceph since I am not root 2024-07-26T12:35:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:28 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3102945008' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T12:35:29.278 INFO:tasks.cephadm:Adding mgr.x on smithi107 2024-07-26T12:35:29.278 INFO:tasks.cephadm:Adding mgr.y on smithi112 2024-07-26T12:35:29.279 INFO:tasks.cephadm:Adding mgr.z on smithi132 2024-07-26T12:35:29.279 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch apply mgr '3;smithi107=x;smithi112=y;smithi132=z' 2024-07-26T12:35:29.524 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:35:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:29 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3102945008' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T12:35:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:29 smithi132 ceph-mon[37580]: mgrmap e14: x(active, since 73s) 2024-07-26T12:35:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:29 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3102945008' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T12:35:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:29 smithi107 ceph-mon[32659]: mgrmap e14: x(active, since 73s) 2024-07-26T12:35:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:29 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3102945008' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T12:35:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:29 smithi112 ceph-mon[37501]: mgrmap e14: x(active, since 73s) 2024-07-26T12:35:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: Active manager daemon x restarted 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: Activating manager daemon x 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: mgrmap e15: x(active, starting, since 0.013609s) 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: Manager daemon x is now available 2024-07-26T12:35:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:34.801 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: Active manager daemon x restarted 2024-07-26T12:35:34.801 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: Activating manager daemon x 2024-07-26T12:35:34.801 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T12:35:34.801 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: mgrmap e15: x(active, starting, since 0.013609s) 2024-07-26T12:35:34.801 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:35:34.801 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:34.801 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:34.801 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T12:35:34.801 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T12:35:34.802 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T12:35:34.802 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T12:35:34.802 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: Manager daemon x is now available 2024-07-26T12:35:34.802 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:34.807 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: Active manager daemon x restarted 2024-07-26T12:35:34.807 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: Activating manager daemon x 2024-07-26T12:35:34.807 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T12:35:34.807 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: mgrmap e15: x(active, starting, since 0.013609s) 2024-07-26T12:35:34.807 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:35:34.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:35:34.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:35:34.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T12:35:34.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T12:35:34.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T12:35:34.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T12:35:34.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: Manager daemon x is now available 2024-07-26T12:35:34.808 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:35.520 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mgr update... 2024-07-26T12:35:35.796 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T12:35:35.797 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T12:35:35.797 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T12:35:35.797 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T12:35:35.797 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:35 smithi112 ceph-mon[37501]: mgrmap e16: x(active, since 1.01734s) 2024-07-26T12:35:35.797 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T12:35:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T12:35:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T12:35:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T12:35:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:35 smithi132 ceph-mon[37580]: mgrmap e16: x(active, since 1.01734s) 2024-07-26T12:35:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:35.929 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T12:35:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T12:35:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T12:35:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T12:35:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:35 smithi107 ceph-mon[32659]: mgrmap e16: x(active, since 1.01734s) 2024-07-26T12:35:35.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.331 DEBUG:teuthology.orchestra.run.smithi112:mgr.y> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mgr.y.service 2024-07-26T12:35:36.333 DEBUG:teuthology.orchestra.run.smithi132:mgr.z> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mgr.z.service 2024-07-26T12:35:36.336 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-07-26T12:35:36.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set client client_mount_timeout 600 2024-07-26T12:35:36.343 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-07-26T12:35:36.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set client debug_client 20 2024-07-26T12:35:36.390 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-07-26T12:35:36.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set client debug_ms 1 2024-07-26T12:35:36.436 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-07-26T12:35:36.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set client rados_mon_op_timeout 900 2024-07-26T12:35:36.483 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-07-26T12:35:36.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set client rados_osd_op_timeout 900 2024-07-26T12:35:36.531 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-07-26T12:35:36.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds debug_mds 20 2024-07-26T12:35:36.536 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:35:36.272+0000 7f1f68c00640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:12:35:36] ENGINE Error in HTTPServer.serve 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: Traceback (most recent call last): 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self._connections.run(self.expiration_interval) 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self._run(expiration_interval) 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: s = self.context.wrap_socket( 2024-07-26T12:35:36.537 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T12:35:36.538 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: return self.sslsocket_class._create( 2024-07-26T12:35:36.538 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T12:35:36.538 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self.do_handshake() 2024-07-26T12:35:36.538 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T12:35:36.538 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: self._sslobj.do_handshake() 2024-07-26T12:35:36.538 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T12:35:36.538 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:35:36.540 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-07-26T12:35:36.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds debug_mds_balancer 20 2024-07-26T12:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:36 smithi132 ceph-mon[37580]: Saving service mgr spec with placement smithi107=x;smithi112=y;smithi132=z;count:3 2024-07-26T12:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.589 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-07-26T12:35:36.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds debug_ms 2 2024-07-26T12:35:36.644 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-07-26T12:35:36.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-07-26T12:35:36.697 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-07-26T12:35:36.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_merge_size 5 2024-07-26T12:35:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:36 smithi112 ceph-mon[37501]: Saving service mgr spec with placement smithi107=x;smithi112=y;smithi132=z;count:3 2024-07-26T12:35:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.753 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-07-26T12:35:36.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_split_bits 3 2024-07-26T12:35:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-mon[32659]: Saving service mgr spec with placement smithi107=x;smithi112=y;smithi132=z;count:3 2024-07-26T12:35:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:36.802 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-07-26T12:35:36.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_split_size 100 2024-07-26T12:35:36.863 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-07-26T12:35:36.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds mds_debug_frag True 2024-07-26T12:35:36.925 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-07-26T12:35:36.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds mds_debug_scatterstat True 2024-07-26T12:35:36.976 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-07-26T12:35:36.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds mds_op_complaint_time 180 2024-07-26T12:35:37.035 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-07-26T12:35:37.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds mds_verify_scatter True 2024-07-26T12:35:37.086 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-07-26T12:35:37.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds osd_op_complaint_time 180 2024-07-26T12:35:37.139 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-07-26T12:35:37.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds rados_mon_op_timeout 900 2024-07-26T12:35:37.198 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-07-26T12:35:37.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mds rados_osd_op_timeout 900 2024-07-26T12:35:37.267 INFO:tasks.cephadm:setting config [mgr] debug_cephsqlite = 20 2024-07-26T12:35:37.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mgr debug_cephsqlite 20 2024-07-26T12:35:37.269 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:37.318 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/allow_m_granularity = True 2024-07-26T12:35:37.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-07-26T12:35:37.338 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:37.353 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/dump_on_update = True 2024-07-26T12:35:37.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-07-26T12:35:37.410 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/log_level = debug 2024-07-26T12:35:37.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/log_level debug 2024-07-26T12:35:37.466 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-07-26T12:35:37.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mon mon_op_complaint_time 120 2024-07-26T12:35:37.527 INFO:tasks.cephadm:setting config [mon] mon_warn_on_pool_no_app = False 2024-07-26T12:35:37.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set mon mon_warn_on_pool_no_app False 2024-07-26T12:35:37.586 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-07-26T12:35:37.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config set osd osd_op_complaint_time 180 2024-07-26T12:35:37.599 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:37.614 DEBUG:tasks.cephadm:set 26 configs 2024-07-26T12:35:37.614 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi107') 2024-07-26T12:35:37.757 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:37.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: [26/Jul/2024:12:35:36] ENGINE Bus STARTING 2024-07-26T12:35:37.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: [26/Jul/2024:12:35:36] ENGINE Serving on http://172.21.15.107:8765 2024-07-26T12:35:37.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: [26/Jul/2024:12:35:36] ENGINE Serving on https://172.21.15.107:7150 2024-07-26T12:35:37.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: [26/Jul/2024:12:35:36] ENGINE Bus STARTED 2024-07-26T12:35:37.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: [26/Jul/2024:12:35:36] ENGINE Error in HTTPServer.serve 2024-07-26T12:35:37.875 INFO:journalctl@ceph.mon.a.smithi107.stdout: Traceback (most recent call last): 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._run(expiration_interval) 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: s = self.context.wrap_socket( 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: return self.sslsocket_class._create( 2024-07-26T12:35:37.876 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T12:35:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout: self.do_handshake() 2024-07-26T12:35:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T12:35:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._sslobj.do_handshake() 2024-07-26T12:35:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T12:35:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:37.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: mgrmap e17: x(active, since 2s) 2024-07-26T12:35:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:37.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: [26/Jul/2024:12:35:36] ENGINE Bus STARTING 2024-07-26T12:35:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: [26/Jul/2024:12:35:36] ENGINE Serving on http://172.21.15.107:8765 2024-07-26T12:35:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: [26/Jul/2024:12:35:36] ENGINE Serving on https://172.21.15.107:7150 2024-07-26T12:35:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: [26/Jul/2024:12:35:36] ENGINE Bus STARTED 2024-07-26T12:35:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: [26/Jul/2024:12:35:36] ENGINE Error in HTTPServer.serve 2024-07-26T12:35:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout: Traceback (most recent call last): 2024-07-26T12:35:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T12:35:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout: self._connections.run(self.expiration_interval) 2024-07-26T12:35:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T12:35:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout: self._run(expiration_interval) 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: s = self.context.wrap_socket( 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: return self.sslsocket_class._create( 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: self.do_handshake() 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: self._sslobj.do_handshake() 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:37.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:37.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: mgrmap e17: x(active, since 2s) 2024-07-26T12:35:37.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:37.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:37.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: [26/Jul/2024:12:35:36] ENGINE Bus STARTING 2024-07-26T12:35:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: [26/Jul/2024:12:35:36] ENGINE Serving on http://172.21.15.107:8765 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: [26/Jul/2024:12:35:36] ENGINE Serving on https://172.21.15.107:7150 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: [26/Jul/2024:12:35:36] ENGINE Bus STARTED 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: [26/Jul/2024:12:35:36] ENGINE Error in HTTPServer.serve 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout: Traceback (most recent call last): 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout: self._connections.run(self.expiration_interval) 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout: self._run(expiration_interval) 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T12:35:38.061 INFO:journalctl@ceph.mon.c.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout: s = self.context.wrap_socket( 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout: return self.sslsocket_class._create( 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout: self.do_handshake() 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout: self._sslobj.do_handshake() 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T12:35:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:38.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:38.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:38.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:38.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:38.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi112", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:38.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: mgrmap e17: x(active, since 2s) 2024-07-26T12:35:38.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:38.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:38.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:38.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi132", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:39.052 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:40.978 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: Updating smithi107:/etc/ceph/ceph.conf 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: Updating smithi112:/etc/ceph/ceph.conf 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: Updating smithi132:/etc/ceph/ceph.conf 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: Updating smithi132:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: Updating smithi112:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:40.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:40.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:40.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:40.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: Updating smithi132:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:40.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: Updating smithi112:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:40.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:40.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:40.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:40.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: Updating smithi107:/etc/ceph/ceph.conf 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: Updating smithi112:/etc/ceph/ceph.conf 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: Updating smithi132:/etc/ceph/ceph.conf 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: Updating smithi132:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: Updating smithi112:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:41.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:41.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:41.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: Updating smithi132:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:41.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: Updating smithi112:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:41.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:35:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: Updating smithi107:/etc/ceph/ceph.conf 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: Updating smithi112:/etc/ceph/ceph.conf 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: Updating smithi132:/etc/ceph/ceph.conf 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: Updating smithi132:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: Updating smithi112:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: Updating smithi132:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: Updating smithi112:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: Updating smithi132:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: Updating smithi112:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:41.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:41.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:41 smithi107 ceph-mon[32659]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:41 smithi107 ceph-mon[32659]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:41 smithi132 ceph-mon[37580]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:41 smithi132 ceph-mon[37580]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:41 smithi112 ceph-mon[37501]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-07-26T12:35:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:41 smithi112 ceph-mon[37501]: Updating smithi107:/var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T12:35:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:43.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:43.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:43.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:43.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:43.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T12:35:43.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T12:35:43.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:43.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:43 smithi107 ceph-mon[32659]: Deploying daemon mgr.z on smithi132 2024-07-26T12:35:43.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:43 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:35:43.465 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:43.478 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T12:35:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T12:35:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:43 smithi132 ceph-mon[37580]: Deploying daemon mgr.z on smithi132 2024-07-26T12:35:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:43 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:35:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T12:35:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T12:35:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:43 smithi112 ceph-mon[37501]: Deploying daemon mgr.z on smithi132 2024-07-26T12:35:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:43 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:35:44.310 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Jul 26 12:35:44 smithi132 systemd[1]: Starting Ceph mgr.z for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:35:44.603 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Jul 26 12:35:44 smithi132 podman[40402]: 2024-07-26 12:35:44.226777129 +0000 UTC m=+0.017066622 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:35:44.603 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Jul 26 12:35:44 smithi132 podman[40402]: 2024-07-26 12:35:44.338825409 +0000 UTC m=+0.129114906 container create 4231fdaf7e41043ba952c9ebd9393421a356c68df6b88809f2ac9d83671dcc33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-z, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724) 2024-07-26T12:35:44.603 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Jul 26 12:35:44 smithi132 podman[40402]: 2024-07-26 12:35:44.566217271 +0000 UTC m=+0.356506764 container init 4231fdaf7e41043ba952c9ebd9393421a356c68df6b88809f2ac9d83671dcc33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-z, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-26T12:35:44.603 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Jul 26 12:35:44 smithi132 podman[40402]: 2024-07-26 12:35:44.568978305 +0000 UTC m=+0.359267799 container start 4231fdaf7e41043ba952c9ebd9393421a356c68df6b88809f2ac9d83671dcc33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-z, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:35:45.060 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Jul 26 12:35:44 smithi132 ceph-mgr[40421]: -- 172.21.15.132:0/1871278806 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c3735910e0 con 0x55c378330480 2024-07-26T12:35:45.060 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Jul 26 12:35:44 smithi132 bash[40402]: 4231fdaf7e41043ba952c9ebd9393421a356c68df6b88809f2ac9d83671dcc33 2024-07-26T12:35:45.060 INFO:journalctl@ceph.mgr.z.smithi132.stdout:Jul 26 12:35:44 smithi132 systemd[1]: Started Ceph mgr.z for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:35:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:45.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:45.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:45.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:45.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:45.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T12:35:45.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T12:35:45.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:45.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:45 smithi107 ceph-mon[32659]: Deploying daemon mgr.y on smithi112 2024-07-26T12:35:45.920 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:45.920 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:45.920 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:45.920 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:45.920 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:45.921 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T12:35:45.921 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T12:35:45.921 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:45.921 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:45 smithi112 ceph-mon[37501]: Deploying daemon mgr.y on smithi112 2024-07-26T12:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T12:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T12:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:45 smithi132 ceph-mon[37580]: Deploying daemon mgr.y on smithi112 2024-07-26T12:35:46.450 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Jul 26 12:35:46 smithi112 systemd[1]: Starting Ceph mgr.y for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:35:46.833 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Jul 26 12:35:46 smithi112 podman[39861]: 2024-07-26 12:35:46.465812578 +0000 UTC m=+0.017193875 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:35:46.833 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Jul 26 12:35:46 smithi112 podman[39861]: 2024-07-26 12:35:46.563550188 +0000 UTC m=+0.114931473 container create 5958d5a6199589a4678f62bc52250ae9702895f51d9658dea37de6c5824e519d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-y, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-07-26T12:35:46.833 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Jul 26 12:35:46 smithi112 podman[39861]: 2024-07-26 12:35:46.741100841 +0000 UTC m=+0.292482127 container init 5958d5a6199589a4678f62bc52250ae9702895f51d9658dea37de6c5824e519d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-y, io.buildah.version=1.36.0, ceph=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-07-26T12:35:46.834 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Jul 26 12:35:46 smithi112 podman[39861]: 2024-07-26 12:35:46.744003283 +0000 UTC m=+0.295384570 container start 5958d5a6199589a4678f62bc52250ae9702895f51d9658dea37de6c5824e519d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:35:46.834 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Jul 26 12:35:46 smithi112 ceph-mgr[39880]: -- 172.21.15.112:0/2987427377 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ea3cd8d0e0 con 0x55ea41b2c000 2024-07-26T12:35:46.834 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Jul 26 12:35:46 smithi112 bash[39861]: 5958d5a6199589a4678f62bc52250ae9702895f51d9658dea37de6c5824e519d 2024-07-26T12:35:47.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.105+0000 7f6dcbe00640 1 -- 172.21.15.107:0/2770320397 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6dac03edb0 msgr2=0x7f6dac0412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.105+0000 7f6dcbe00640 1 --2- 172.21.15.107:0/2770320397 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6dac03edb0 0x7f6dac0412c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6db802a860 tx=0x7f6db8005d30 comp rx=0 tx=0).stop 2024-07-26T12:35:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.105+0000 7f6dcbe00640 1 -- 172.21.15.107:0/2770320397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6dc41057e0 msgr2=0x7f6dc419beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.105+0000 7f6dcbe00640 1 --2- 172.21.15.107:0/2770320397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6dc41057e0 0x7f6dc419beb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6db000ea60 tx=0x7f6db000ef20 comp rx=0 tx=0).stop 2024-07-26T12:35:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.105+0000 7f6dcbe00640 1 -- 172.21.15.107:0/2770320397 shutdown_connections 2024-07-26T12:35:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.105+0000 7f6dcbe00640 1 --2- 172.21.15.107:0/2770320397 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6dac03edb0 0x7f6dac0412c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.105+0000 7f6dcbe00640 1 --2- 172.21.15.107:0/2770320397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6dc410ba70 0x7f6dc41a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.105+0000 7f6dcbe00640 1 --2- 172.21.15.107:0/2770320397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6dc410b120 0x7f6dc419c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.105+0000 7f6dcbe00640 1 --2- 172.21.15.107:0/2770320397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6dc41057e0 0x7f6dc419beb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:47.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.105+0000 7f6dcbe00640 1 -- 172.21.15.107:0/2770320397 >> 172.21.15.107:0/2770320397 conn(0x7f6dc4101210 msgr2=0x7f6dc4103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:35:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.106+0000 7f6dcbe00640 1 -- 172.21.15.107:0/2770320397 shutdown_connections 2024-07-26T12:35:47.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.106+0000 7f6dcbe00640 1 -- 172.21.15.107:0/2770320397 wait complete. 2024-07-26T12:35:47.165 INFO:journalctl@ceph.mgr.y.smithi112.stdout:Jul 26 12:35:46 smithi112 systemd[1]: Started Ceph mgr.y for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:35:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:47 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:35:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:47 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:35:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:47 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:35:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:47 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:35:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:47 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:35:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:47.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:47 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:35:47.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.827+0000 7f7b57800640 1 -- 172.21.15.107:0/3919944649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b501057b0 msgr2=0x7f7b50105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:47.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.827+0000 7f7b57800640 1 --2- 172.21.15.107:0/3919944649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b501057b0 0x7f7b50105bc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009900 tx=0x7f7b4402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:35:47.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.827+0000 7f7b57800640 1 -- 172.21.15.107:0/3919944649 shutdown_connections 2024-07-26T12:35:47.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.827+0000 7f7b57800640 1 --2- 172.21.15.107:0/3919944649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5010ba40 0x7f7b50112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.827+0000 7f7b57800640 1 --2- 172.21.15.107:0/3919944649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b5010b0f0 0x7f7b5010b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.827+0000 7f7b57800640 1 --2- 172.21.15.107:0/3919944649 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b501057b0 0x7f7b50105bc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.827+0000 7f7b57800640 1 -- 172.21.15.107:0/3919944649 >> 172.21.15.107:0/3919944649 conn(0x7f7b50101240 msgr2=0x7f7b50103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.827+0000 7f7b57800640 1 -- 172.21.15.107:0/3919944649 shutdown_connections 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.827+0000 7f7b57800640 1 -- 172.21.15.107:0/3919944649 wait complete. 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.828+0000 7f7b57800640 1 Processor -- start 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.828+0000 7f7b57800640 1 -- start start 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b57800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b501057b0 0x7f7b5019bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b57800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5010b0f0 0x7f7b5019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b57800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b5010ba40 0x7f7b501a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b57800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b50115480 con 0x7f7b5010b0f0 2024-07-26T12:35:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b57800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7b50115300 con 0x7f7b5010ba40 2024-07-26T12:35:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b57800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7b50115600 con 0x7f7b501057b0 2024-07-26T12:35:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b55800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b5010ba40 0x7f7b501a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b55800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b5010ba40 0x7f7b501a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:50638/0 (socket says 172.21.15.107:50638) 2024-07-26T12:35:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b55800640 1 -- 172.21.15.107:0/3064625746 learned_addr learned my addr 172.21.15.107:0/3064625746 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:35:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b4fe00640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5010b0f0 0x7f7b5019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b54e00640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b501057b0 0x7f7b5019bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b55800640 1 -- 172.21.15.107:0/3064625746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b501057b0 msgr2=0x7f7b5019bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.829+0000 7f7b55800640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b501057b0 0x7f7b5019bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.830+0000 7f7b55800640 1 -- 172.21.15.107:0/3064625746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5010b0f0 msgr2=0x7f7b5019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:47.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.830+0000 7f7b55800640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5010b0f0 0x7f7b5019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.830+0000 7f7b55800640 1 -- 172.21.15.107:0/3064625746 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b501a1ee0 con 0x7f7b5010ba40 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.830+0000 7f7b55800640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b5010ba40 0x7f7b501a17b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800eb10 tx=0x7f7b4800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.830+0000 7f7b4d600640 1 -- 172.21.15.107:0/3064625746 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48004b20 con 0x7f7b5010ba40 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.830+0000 7f7b4d600640 1 -- 172.21.15.107:0/3064625746 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(24 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f7b48004cc0 con 0x7f7b5010ba40 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.830+0000 7f7b4d600640 1 -- 172.21.15.107:0/3064625746 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b48010690 con 0x7f7b5010ba40 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.830+0000 7f7b57800640 1 -- 172.21.15.107:0/3064625746 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b501a21d0 con 0x7f7b5010ba40 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.830+0000 7f7b57800640 1 -- 172.21.15.107:0/3064625746 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b501a8930 con 0x7f7b5010ba40 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.831+0000 7f7b57800640 1 -- 172.21.15.107:0/3064625746 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b50105bc0 con 0x7f7b5010ba40 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.831+0000 7f7b4d600640 1 -- 172.21.15.107:0/3064625746 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 17) ==== 50765+0+0 (secure 0 0 0) 0x7f7b48005020 con 0x7f7b5010ba40 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.832+0000 7f7b4d600640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b3803edb0 0x7f7b380412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.832+0000 7f7b4d600640 1 -- 172.21.15.107:0/3064625746 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f7b48053a10 con 0x7f7b5010ba40 2024-07-26T12:35:47.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.832+0000 7f7b54e00640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b3803edb0 0x7f7b380412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:47.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.833+0000 7f7b54e00640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b3803edb0 0x7f7b380412c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009230 tx=0x7f7b440058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:35:47.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:47.841+0000 7f7b4d600640 1 -- 172.21.15.107:0/3064625746 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7b480059f0 con 0x7f7b5010ba40 2024-07-26T12:35:48.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.040+0000 7f7b57800640 1 -- 172.21.15.107:0/3064625746 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f7b5002e0c0 con 0x7f7b5010ba40 2024-07-26T12:35:48.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.048+0000 7f7b4d600640 1 -- 172.21.15.107:0/3064625746 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v13)=0 v13) ==== 108+0+0 (secure 0 0 0) 0x7f7b4800a250 con 0x7f7b5010ba40 2024-07-26T12:35:48.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 -- 172.21.15.107:0/3064625746 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b3803edb0 msgr2=0x7f7b380412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:48.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b3803edb0 0x7f7b380412c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7b44009230 tx=0x7f7b440058d0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 -- 172.21.15.107:0/3064625746 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b5010ba40 msgr2=0x7f7b501a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:48.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b5010ba40 0x7f7b501a17b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800eb10 tx=0x7f7b4800c2d0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 -- 172.21.15.107:0/3064625746 shutdown_connections 2024-07-26T12:35:48.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b3803edb0 0x7f7b380412c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b5010ba40 0x7f7b501a17b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b5010b0f0 0x7f7b5019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 --2- 172.21.15.107:0/3064625746 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b501057b0 0x7f7b5019bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 -- 172.21.15.107:0/3064625746 >> 172.21.15.107:0/3064625746 conn(0x7f7b50101240 msgr2=0x7f7b50103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:35:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 -- 172.21.15.107:0/3064625746 shutdown_connections 2024-07-26T12:35:48.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.052+0000 7f7b57800640 1 -- 172.21.15.107:0/3064625746 wait complete. 2024-07-26T12:35:48.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.153+0000 7f4b3c200640 1 -- 172.21.15.107:0/3091677150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b3410ba50 msgr2=0x7f4b34112280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:48.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.153+0000 7f4b3c200640 1 --2- 172.21.15.107:0/3091677150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b3410ba50 0x7f4b34112280 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c009ca0 tx=0x7f4b2c02a8c0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.153+0000 7f4b3c200640 1 -- 172.21.15.107:0/3091677150 shutdown_connections 2024-07-26T12:35:48.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.153+0000 7f4b3c200640 1 --2- 172.21.15.107:0/3091677150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b3410ba50 0x7f4b34112280 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.153+0000 7f4b3c200640 1 --2- 172.21.15.107:0/3091677150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b3410b100 0x7f4b3410b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.153+0000 7f4b3c200640 1 --2- 172.21.15.107:0/3091677150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b341057c0 0x7f4b34105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.153+0000 7f4b3c200640 1 -- 172.21.15.107:0/3091677150 >> 172.21.15.107:0/3091677150 conn(0x7f4b34101290 msgr2=0x7f4b341036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:35:48.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.153+0000 7f4b3c200640 1 -- 172.21.15.107:0/3091677150 shutdown_connections 2024-07-26T12:35:48.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.153+0000 7f4b3c200640 1 -- 172.21.15.107:0/3091677150 wait complete. 2024-07-26T12:35:48.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.154+0000 7f4b3c200640 1 Processor -- start 2024-07-26T12:35:48.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.154+0000 7f4b3c200640 1 -- start start 2024-07-26T12:35:48.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b3c200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b341057c0 0x7f4b3419beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:48.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b3c200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b3410b100 0x7f4b3419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b3c200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b3410ba50 0x7f4b341a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b3c200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4b341154a0 con 0x7f4b3410b100 2024-07-26T12:35:48.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b3c200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4b34115320 con 0x7f4b341057c0 2024-07-26T12:35:48.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b3c200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4b34115620 con 0x7f4b3410ba50 2024-07-26T12:35:48.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b38e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b3410b100 0x7f4b3419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:48.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b38e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b3410b100 0x7f4b3419c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44194/0 (socket says 172.21.15.107:44194) 2024-07-26T12:35:48.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b38e00640 1 -- 172.21.15.107:0/2243604514 learned_addr learned my addr 172.21.15.107:0/2243604514 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:35:48.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b3a200640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b3410ba50 0x7f4b341a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:48.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.155+0000 7f4b39800640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b341057c0 0x7f4b3419beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:48.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b38e00640 1 -- 172.21.15.107:0/2243604514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b3410ba50 msgr2=0x7f4b341a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:48.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b38e00640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b3410ba50 0x7f4b341a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b38e00640 1 -- 172.21.15.107:0/2243604514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b341057c0 msgr2=0x7f4b3419beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:48.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b38e00640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b341057c0 0x7f4b3419beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b38e00640 1 -- 172.21.15.107:0/2243604514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b341a1e20 con 0x7f4b3410b100 2024-07-26T12:35:48.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b3a200640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b3410ba50 0x7f4b341a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:35:48.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b38e00640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b3410b100 0x7f4b3419c450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000e9c0 tx=0x7f4b2000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:35:48.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b32000640 1 -- 172.21.15.107:0/2243604514 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2000ccc0 con 0x7f4b3410b100 2024-07-26T12:35:48.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b32000640 1 -- 172.21.15.107:0/2243604514 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(24 keys) ==== 1039+0+0 (secure 0 0 0) 0x7f4b20005490 con 0x7f4b3410b100 2024-07-26T12:35:48.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b3c200640 1 -- 172.21.15.107:0/2243604514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b341a2110 con 0x7f4b3410b100 2024-07-26T12:35:48.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b32000640 1 -- 172.21.15.107:0/2243604514 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20010430 con 0x7f4b3410b100 2024-07-26T12:35:48.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.156+0000 7f4b3c200640 1 -- 172.21.15.107:0/2243604514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b341a8870 con 0x7f4b3410b100 2024-07-26T12:35:48.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.157+0000 7f4b3c200640 1 -- 172.21.15.107:0/2243604514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b34105bd0 con 0x7f4b3410b100 2024-07-26T12:35:48.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.158+0000 7f4b32000640 1 -- 172.21.15.107:0/2243604514 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 50765+0+0 (secure 0 0 0) 0x7f4b200105d0 con 0x7f4b3410b100 2024-07-26T12:35:48.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.159+0000 7f4b32000640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b1803edb0 0x7f4b180412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:48.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.159+0000 7f4b32000640 1 -- 172.21.15.107:0/2243604514 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f4b20053870 con 0x7f4b3410b100 2024-07-26T12:35:48.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.159+0000 7f4b39800640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b1803edb0 0x7f4b180412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:48.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.160+0000 7f4b39800640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b1803edb0 0x7f4b180412c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009bd0 tx=0x7f4b24005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:35:48.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.164+0000 7f4b32000640 1 -- 172.21.15.107:0/2243604514 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4b20016330 con 0x7f4b3410b100 2024-07-26T12:35:48.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.383+0000 7f4b3c200640 1 -- 172.21.15.107:0/2243604514 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f4b3402e0c0 con 0x7f4b3410b100 2024-07-26T12:35:48.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.388+0000 7f4b32000640 1 -- 172.21.15.107:0/2243604514 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7f4b2000a250 con 0x7f4b3410b100 2024-07-26T12:35:48.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.389+0000 7f4b32000640 1 -- 172.21.15.107:0/2243604514 <== mon.0 v2:172.21.15.107:3300/0 8 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4b2001f6e0 con 0x7f4b3410b100 2024-07-26T12:35:48.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 -- 172.21.15.107:0/2243604514 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b1803edb0 msgr2=0x7f4b180412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:48.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b1803edb0 0x7f4b180412c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4b24009bd0 tx=0x7f4b24005f90 comp rx=0 tx=0).stop 2024-07-26T12:35:48.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 -- 172.21.15.107:0/2243604514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b3410b100 msgr2=0x7f4b3419c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:48.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b3410b100 0x7f4b3419c450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000e9c0 tx=0x7f4b2000ee80 comp rx=0 tx=0).stop 2024-07-26T12:35:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 -- 172.21.15.107:0/2243604514 shutdown_connections 2024-07-26T12:35:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b1803edb0 0x7f4b180412c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b3410ba50 0x7f4b341a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b3410b100 0x7f4b3419c450 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 --2- 172.21.15.107:0/2243604514 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b341057c0 0x7f4b3419beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 -- 172.21.15.107:0/2243604514 >> 172.21.15.107:0/2243604514 conn(0x7f4b34101290 msgr2=0x7f4b34109770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:35:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 -- 172.21.15.107:0/2243604514 shutdown_connections 2024-07-26T12:35:48.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:48.391+0000 7f4b3c200640 1 -- 172.21.15.107:0/2243604514 wait complete. 2024-07-26T12:35:48.746 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:48.749 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:48.749 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:48.750 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:35:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:49.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2243604514' entity='client.admin' 2024-07-26T12:35:49.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T12:35:49.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:49.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:49.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T12:35:49.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:49.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:49 smithi132 ceph-mon[37580]: Reconfiguring daemon mgr.x on smithi107 2024-07-26T12:35:49.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:35:49.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2243604514' entity='client.admin' 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:49.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:49 smithi107 ceph-mon[32659]: Reconfiguring daemon mgr.x on smithi107 2024-07-26T12:35:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:35:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2243604514' entity='client.admin' 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:49 smithi112 ceph-mon[37501]: Reconfiguring daemon mgr.x on smithi107 2024-07-26T12:35:49.913 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:50 smithi107 ceph-mon[32659]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T12:35:50.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:50 smithi107 ceph-mon[32659]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T12:35:50.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:50 smithi107 ceph-mon[32659]: Standby manager daemon z started 2024-07-26T12:35:50.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:50 smithi107 ceph-mon[32659]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T12:35:50.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:50.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:50 smithi107 ceph-mon[32659]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T12:35:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:50 smithi132 ceph-mon[37580]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T12:35:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:50 smithi132 ceph-mon[37580]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T12:35:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:50 smithi132 ceph-mon[37580]: Standby manager daemon z started 2024-07-26T12:35:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:50 smithi132 ceph-mon[37580]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T12:35:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:50 smithi132 ceph-mon[37580]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T12:35:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:50 smithi112 ceph-mon[37501]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T12:35:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:50 smithi112 ceph-mon[37501]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T12:35:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:50 smithi112 ceph-mon[37501]: Standby manager daemon z started 2024-07-26T12:35:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:50 smithi112 ceph-mon[37501]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T12:35:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:35:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:50 smithi112 ceph-mon[37501]: from='mgr.? 172.21.15.132:0/3416077715' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T12:35:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:51 smithi107 ceph-mon[32659]: mgrmap e18: x(active, since 15s), standbys: z 2024-07-26T12:35:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T12:35:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:51 smithi112 ceph-mon[37501]: mgrmap e18: x(active, since 15s), standbys: z 2024-07-26T12:35:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T12:35:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:51 smithi132 ceph-mon[37580]: mgrmap e18: x(active, since 15s), standbys: z 2024-07-26T12:35:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T12:35:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:52 smithi107 ceph-mon[32659]: Standby manager daemon y started 2024-07-26T12:35:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:52 smithi107 ceph-mon[32659]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T12:35:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:52 smithi107 ceph-mon[32659]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T12:35:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:52 smithi107 ceph-mon[32659]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T12:35:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:52 smithi107 ceph-mon[32659]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T12:35:52.299 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi107') 2024-07-26T12:35:52.299 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi107') 2024-07-26T12:35:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:52 smithi112 ceph-mon[37501]: Standby manager daemon y started 2024-07-26T12:35:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:52 smithi112 ceph-mon[37501]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T12:35:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:52 smithi112 ceph-mon[37501]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T12:35:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:52 smithi112 ceph-mon[37501]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T12:35:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:52 smithi112 ceph-mon[37501]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T12:35:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:52 smithi132 ceph-mon[37580]: Standby manager daemon y started 2024-07-26T12:35:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:52 smithi132 ceph-mon[37580]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T12:35:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:52 smithi132 ceph-mon[37580]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T12:35:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:52 smithi132 ceph-mon[37580]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T12:35:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:52 smithi132 ceph-mon[37580]: from='mgr.? 172.21.15.112:0/1428257307' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T12:35:52.706 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:53 smithi112 ceph-mon[37501]: mgrmap e19: x(active, since 17s), standbys: z, y 2024-07-26T12:35:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T12:35:53.467 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:53 smithi107 ceph-mon[32659]: mgrmap e19: x(active, since 17s), standbys: z, y 2024-07-26T12:35:53.467 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T12:35:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:53 smithi132 ceph-mon[37580]: mgrmap e19: x(active, since 17s), standbys: z, y 2024-07-26T12:35:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T12:35:53.636 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:53.768 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:53.769 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:53.769 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:53.769 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:53.853 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:53.863 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:53.878 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:53.883 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:53.963 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:54.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.081+0000 7fad3c200640 1 -- 172.21.15.107:0/3622914518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34075ad0 msgr2=0x7fad3410f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:54.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.081+0000 7fad3c200640 1 --2- 172.21.15.107:0/3622914518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34075ad0 0x7fad3410f630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fad20009900 tx=0x7fad2002a410 comp rx=0 tx=0).stop 2024-07-26T12:35:54.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.081+0000 7fad3c200640 1 -- 172.21.15.107:0/3622914518 shutdown_connections 2024-07-26T12:35:54.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.081+0000 7fad3c200640 1 --2- 172.21.15.107:0/3622914518 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad3410fb70 0x7fad34111f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:54.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.081+0000 7fad3c200640 1 --2- 172.21.15.107:0/3622914518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34075ad0 0x7fad3410f630 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:54.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.081+0000 7fad3c200640 1 --2- 172.21.15.107:0/3622914518 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad34075180 0x7fad34075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:54.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.081+0000 7fad3c200640 1 -- 172.21.15.107:0/3622914518 >> 172.21.15.107:0/3622914518 conn(0x7fad340ff000 msgr2=0x7fad34101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:35:54.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.081+0000 7fad3c200640 1 -- 172.21.15.107:0/3622914518 shutdown_connections 2024-07-26T12:35:54.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.081+0000 7fad3c200640 1 -- 172.21.15.107:0/3622914518 wait complete. 2024-07-26T12:35:54.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.082+0000 7fad3c200640 1 Processor -- start 2024-07-26T12:35:54.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.082+0000 7fad3c200640 1 -- start start 2024-07-26T12:35:54.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad3c200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34075180 0x7fad3419c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad3c200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34075ad0 0x7fad3419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad3c200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad3410fb70 0x7fad341a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad3c200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad341155b0 con 0x7fad34075ad0 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad3c200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fad34115430 con 0x7fad34075180 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad3c200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fad34115730 con 0x7fad3410fb70 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad38e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34075ad0 0x7fad3419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad38e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34075ad0 0x7fad3419c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49402/0 (socket says 172.21.15.107:49402) 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad38e00640 1 -- 172.21.15.107:0/744736520 learned_addr learned my addr 172.21.15.107:0/744736520 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad39800640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34075180 0x7fad3419c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad3a200640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad3410fb70 0x7fad341a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad39800640 1 -- 172.21.15.107:0/744736520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad3410fb70 msgr2=0x7fad341a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.083+0000 7fad39800640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad3410fb70 0x7fad341a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.084+0000 7fad39800640 1 -- 172.21.15.107:0/744736520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34075ad0 msgr2=0x7fad3419c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:54.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.084+0000 7fad39800640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34075ad0 0x7fad3419c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:54.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.084+0000 7fad39800640 1 -- 172.21.15.107:0/744736520 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad341a1fb0 con 0x7fad34075180 2024-07-26T12:35:54.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.084+0000 7fad38e00640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34075ad0 0x7fad3419c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:35:54.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.084+0000 7fad39800640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34075180 0x7fad3419c040 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fad2800e9f0 tx=0x7fad2800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:35:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.084+0000 7fad32000640 1 -- 172.21.15.107:0/744736520 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2800ccc0 con 0x7fad34075180 2024-07-26T12:35:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.084+0000 7fad32000640 1 -- 172.21.15.107:0/744736520 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fad28005020 con 0x7fad34075180 2024-07-26T12:35:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.084+0000 7fad3c200640 1 -- 172.21.15.107:0/744736520 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad341a22a0 con 0x7fad34075180 2024-07-26T12:35:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.084+0000 7fad32000640 1 -- 172.21.15.107:0/744736520 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2800ce60 con 0x7fad34075180 2024-07-26T12:35:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.084+0000 7fad3c200640 1 -- 172.21.15.107:0/744736520 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad341a8a00 con 0x7fad34075180 2024-07-26T12:35:54.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.085+0000 7fad3c200640 1 -- 172.21.15.107:0/744736520 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad34075590 con 0x7fad34075180 2024-07-26T12:35:54.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.088+0000 7fad32000640 1 -- 172.21.15.107:0/744736520 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fad28004a60 con 0x7fad34075180 2024-07-26T12:35:54.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.089+0000 7fad32000640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad1c0b3c10 0x7fad1c0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:35:54.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.089+0000 7fad32000640 1 -- 172.21.15.107:0/744736520 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fad280e1fa0 con 0x7fad34075180 2024-07-26T12:35:54.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.089+0000 7fad38e00640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad1c0b3c10 0x7fad1c0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:35:54.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.090+0000 7fad38e00640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad1c0b3c10 0x7fad1c0b6120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fad20005790 tx=0x7fad20005e60 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:35:54.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.093+0000 7fad32000640 1 -- 172.21.15.107:0/744736520 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fad280051d0 con 0x7fad34075180 2024-07-26T12:35:54.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.320+0000 7fad3c200640 1 -- 172.21.15.107:0/744736520 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7fad341a8cc0 con 0x7fad34075180 2024-07-26T12:35:54.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.327+0000 7fad32000640 1 -- 172.21.15.107:0/744736520 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v15)=0 v15) ==== 117+0+0 (secure 0 0 0) 0x7fad280ab1f0 con 0x7fad34075180 2024-07-26T12:35:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.330+0000 7fad3c200640 1 -- 172.21.15.107:0/744736520 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad1c0b3c10 msgr2=0x7fad1c0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.330+0000 7fad3c200640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad1c0b3c10 0x7fad1c0b6120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fad20005790 tx=0x7fad20005e60 comp rx=0 tx=0).stop 2024-07-26T12:35:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.330+0000 7fad3c200640 1 -- 172.21.15.107:0/744736520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34075180 msgr2=0x7fad3419c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:35:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.330+0000 7fad3c200640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34075180 0x7fad3419c040 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fad2800e9f0 tx=0x7fad2800eeb0 comp rx=0 tx=0).stop 2024-07-26T12:35:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.331+0000 7fad3c200640 1 -- 172.21.15.107:0/744736520 shutdown_connections 2024-07-26T12:35:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.331+0000 7fad3c200640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad1c0b3c10 0x7fad1c0b6120 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.331+0000 7fad3c200640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad3410fb70 0x7fad341a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.331+0000 7fad3c200640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34075ad0 0x7fad3419c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.331+0000 7fad3c200640 1 --2- 172.21.15.107:0/744736520 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34075180 0x7fad3419c040 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:35:54.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.331+0000 7fad3c200640 1 -- 172.21.15.107:0/744736520 >> 172.21.15.107:0/744736520 conn(0x7fad340ff000 msgr2=0x7fad34109750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:35:54.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.331+0000 7fad3c200640 1 -- 172.21.15.107:0/744736520 shutdown_connections 2024-07-26T12:35:54.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:35:54.331+0000 7fad3c200640 1 -- 172.21.15.107:0/744736520 wait complete. 2024-07-26T12:35:55.455 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:55 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:35:55.455 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:55 smithi107 ceph-mon[32659]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:55 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:35:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:55 smithi112 ceph-mon[37501]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:55 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:35:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:55 smithi132 ceph-mon[37580]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:56.470 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi107') 2024-07-26T12:35:56.471 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi107') 2024-07-26T12:35:57.884 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:57 smithi132 ceph-mon[37580]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:57 smithi112 ceph-mon[37501]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:57 smithi107 ceph-mon[32659]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:58.764 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:35:59.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:35:59 smithi107 ceph-mon[32659]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:35:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:35:59 smithi112 ceph-mon[37501]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:35:59 smithi132 ceph-mon[37580]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:01 smithi107 ceph-mon[32659]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:01 smithi112 ceph-mon[37501]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:01 smithi132 ceph-mon[37580]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:03.693 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:03 smithi107 ceph-mon[32659]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:03 smithi112 ceph-mon[37501]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:03 smithi132 ceph-mon[37580]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:05.763 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:05 smithi107 ceph-mon[32659]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:05 smithi112 ceph-mon[37501]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:05 smithi132 ceph-mon[37580]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.156+0000 7f965d400640 1 -- 172.21.15.107:0/3603710737 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f965810a910 msgr2=0x7f9658115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.156+0000 7f965d400640 1 --2- 172.21.15.107:0/3603710737 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f965810a910 0x7f9658115a00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9640009860 tx=0x7f964002a340 comp rx=0 tx=0).stop 2024-07-26T12:36:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.156+0000 7f965d400640 1 -- 172.21.15.107:0/3603710737 shutdown_connections 2024-07-26T12:36:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.156+0000 7f965d400640 1 --2- 172.21.15.107:0/3603710737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9658116070 0x7f9658118580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.156+0000 7f965d400640 1 --2- 172.21.15.107:0/3603710737 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f965810a910 0x7f9658115a00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:06.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.156+0000 7f965d400640 1 --2- 172.21.15.107:0/3603710737 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9658109fc0 0x7f965810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:06.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.157+0000 7f965d400640 1 -- 172.21.15.107:0/3603710737 >> 172.21.15.107:0/3603710737 conn(0x7f9658070d20 msgr2=0x7f9658073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:06.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.157+0000 7f965d400640 1 -- 172.21.15.107:0/3603710737 shutdown_connections 2024-07-26T12:36:06.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.157+0000 7f965d400640 1 -- 172.21.15.107:0/3603710737 wait complete. 2024-07-26T12:36:06.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.158+0000 7f965d400640 1 Processor -- start 2024-07-26T12:36:06.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.158+0000 7f965d400640 1 -- start start 2024-07-26T12:36:06.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f965d400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9658109fc0 0x7f96581a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:06.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f965d400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f965810a910 0x7f96581a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:06.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f965d400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9658116070 0x7f96581aa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:06.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f965d400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f965811bbd0 con 0x7f9658116070 2024-07-26T12:36:06.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f965d400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f965811ba50 con 0x7f9658109fc0 2024-07-26T12:36:06.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f965d400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f965811bd50 con 0x7f965810a910 2024-07-26T12:36:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f9656800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9658109fc0 0x7f96581a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f9656800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9658109fc0 0x7f96581a4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:35472/0 (socket says 172.21.15.107:35472) 2024-07-26T12:36:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f9656800640 1 -- 172.21.15.107:0/990680998 learned_addr learned my addr 172.21.15.107:0/990680998 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f9657200640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9658116070 0x7f96581aa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:06.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f964de00640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f965810a910 0x7f96581a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f9656800640 1 -- 172.21.15.107:0/990680998 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f965810a910 msgr2=0x7f96581a4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f9656800640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f965810a910 0x7f96581a4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f9656800640 1 -- 172.21.15.107:0/990680998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9658116070 msgr2=0x7f96581aa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f9656800640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9658116070 0x7f96581aa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.159+0000 7f9656800640 1 -- 172.21.15.107:0/990680998 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96581aa7f0 con 0x7f9658109fc0 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.160+0000 7f9656800640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9658109fc0 0x7f96581a4880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f964800e9c0 tx=0x7f964800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.160+0000 7f964fe00640 1 -- 172.21.15.107:0/990680998 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964800ccc0 con 0x7f9658109fc0 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.160+0000 7f964fe00640 1 -- 172.21.15.107:0/990680998 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f96480054b0 con 0x7f9658109fc0 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.160+0000 7f965d400640 1 -- 172.21.15.107:0/990680998 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96581aaae0 con 0x7f9658109fc0 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.160+0000 7f964fe00640 1 -- 172.21.15.107:0/990680998 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648010430 con 0x7f9658109fc0 2024-07-26T12:36:06.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.160+0000 7f965d400640 1 -- 172.21.15.107:0/990680998 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96581b1240 con 0x7f9658109fc0 2024-07-26T12:36:06.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.162+0000 7f965d400640 1 -- 172.21.15.107:0/990680998 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f965810a3d0 con 0x7f9658109fc0 2024-07-26T12:36:06.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.164+0000 7f964fe00640 1 -- 172.21.15.107:0/990680998 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f96480046c0 con 0x7f9658109fc0 2024-07-26T12:36:06.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.165+0000 7f964fe00640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f963c0b3b90 0x7f963c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:06.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.165+0000 7f964fe00640 1 -- 172.21.15.107:0/990680998 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f96480e24b0 con 0x7f9658109fc0 2024-07-26T12:36:06.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.165+0000 7f964de00640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f963c0b3b90 0x7f963c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:06.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.166+0000 7f964de00640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f963c0b3b90 0x7f963c0b60a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9640035040 tx=0x7f9640000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:06.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.169+0000 7f964fe00640 1 -- 172.21.15.107:0/990680998 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f96480ab660 con 0x7f9658109fc0 2024-07-26T12:36:06.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.371+0000 7f965d400640 1 -- 172.21.15.107:0/990680998 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f96581b1530 con 0x7f9658109fc0 2024-07-26T12:36:06.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.376+0000 7f964fe00640 1 -- 172.21.15.107:0/990680998 <== mon.2 v2:172.21.15.112:3300/0 7 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96480aaa70 con 0x7f9658109fc0 2024-07-26T12:36:06.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.379+0000 7f964fe00640 1 -- 172.21.15.107:0/990680998 <== mon.2 v2:172.21.15.112:3300/0 8 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v16)=0 v16) ==== 123+0+0 (secure 0 0 0) 0x7f96480b4010 con 0x7f9658109fc0 2024-07-26T12:36:06.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 -- 172.21.15.107:0/990680998 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f963c0b3b90 msgr2=0x7f963c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:06.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f963c0b3b90 0x7f963c0b60a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9640035040 tx=0x7f9640000bc0 comp rx=0 tx=0).stop 2024-07-26T12:36:06.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 -- 172.21.15.107:0/990680998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9658109fc0 msgr2=0x7f96581a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:06.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9658109fc0 0x7f96581a4880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f964800e9c0 tx=0x7f964800ee80 comp rx=0 tx=0).stop 2024-07-26T12:36:06.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 -- 172.21.15.107:0/990680998 shutdown_connections 2024-07-26T12:36:06.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f963c0b3b90 0x7f963c0b60a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:06.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9658116070 0x7f96581aa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:06.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f965810a910 0x7f96581a4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:06.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 --2- 172.21.15.107:0/990680998 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9658109fc0 0x7f96581a4880 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:06.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 -- 172.21.15.107:0/990680998 >> 172.21.15.107:0/990680998 conn(0x7f9658070d20 msgr2=0x7f9658071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:06.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 -- 172.21.15.107:0/990680998 shutdown_connections 2024-07-26T12:36:06.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:06.383+0000 7f965d400640 1 -- 172.21.15.107:0/990680998 wait complete. 2024-07-26T12:36:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:07 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:36:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:07 smithi107 ceph-mon[32659]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:07 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:36:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:07 smithi112 ceph-mon[37501]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:07 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:36:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:07 smithi132 ceph-mon[37580]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:08.557 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:36:09.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.110+0000 7faafcc00640 1 -- 172.21.15.107:0/3399425384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8109670 msgr2=0x7faaf810fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.110+0000 7faafcc00640 1 --2- 172.21.15.107:0/3399425384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8109670 0x7faaf810fea0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009ca0 tx=0x7faaf002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.110+0000 7faafcc00640 1 -- 172.21.15.107:0/3399425384 shutdown_connections 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.110+0000 7faafcc00640 1 --2- 172.21.15.107:0/3399425384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf8109670 0x7faaf810fea0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.110+0000 7faafcc00640 1 --2- 172.21.15.107:0/3399425384 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faaf8108d20 0x7faaf8109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.110+0000 7faafcc00640 1 --2- 172.21.15.107:0/3399425384 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faaf81033e0 0x7faaf81037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.110+0000 7faafcc00640 1 -- 172.21.15.107:0/3399425384 >> 172.21.15.107:0/3399425384 conn(0x7faaf80fee70 msgr2=0x7faaf81012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.111+0000 7faafcc00640 1 -- 172.21.15.107:0/3399425384 shutdown_connections 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.111+0000 7faafcc00640 1 -- 172.21.15.107:0/3399425384 wait complete. 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.112+0000 7faafcc00640 1 Processor -- start 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.112+0000 7faafcc00640 1 -- start start 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.113+0000 7faafcc00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf81033e0 0x7faaf819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:09.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.113+0000 7faafcc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faaf8108d20 0x7faaf819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.113+0000 7faafcc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faaf8109670 0x7faaf81a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.113+0000 7faafcc00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8113390 con 0x7faaf81033e0 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.113+0000 7faafcc00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8113210 con 0x7faaf8108d20 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.113+0000 7faafcc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8113510 con 0x7faaf8109670 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.113+0000 7faaf7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faaf8109670 0x7faaf81a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.113+0000 7faaf7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf81033e0 0x7faaf819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.113+0000 7faaf7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faaf8109670 0x7faaf81a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:34194/0 (socket says 172.21.15.107:34194) 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.113+0000 7faaf7e00640 1 -- 172.21.15.107:0/1144336631 learned_addr learned my addr 172.21.15.107:0/1144336631 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.114+0000 7faaf7400640 1 -- 172.21.15.107:0/1144336631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faaf8109670 msgr2=0x7faaf81a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.114+0000 7faaf7400640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faaf8109670 0x7faaf81a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.114+0000 7faaf7400640 1 -- 172.21.15.107:0/1144336631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faaf8108d20 msgr2=0x7faaf819c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:09.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.114+0000 7faaf7400640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faaf8108d20 0x7faaf819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.114+0000 7faaf7400640 1 -- 172.21.15.107:0/1144336631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf81a1e90 con 0x7faaf81033e0 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.115+0000 7faaf7400640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf81033e0 0x7faaf819bf20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faae800e9f0 tx=0x7faae800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.115+0000 7faaf4c00640 1 -- 172.21.15.107:0/1144336631 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae800ccc0 con 0x7faaf81033e0 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.115+0000 7faaf4c00640 1 -- 172.21.15.107:0/1144336631 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faae8005020 con 0x7faaf81033e0 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.115+0000 7faaf4c00640 1 -- 172.21.15.107:0/1144336631 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae80049e0 con 0x7faaf81033e0 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.115+0000 7faafcc00640 1 -- 172.21.15.107:0/1144336631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf81a2180 con 0x7faaf81033e0 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.115+0000 7faafcc00640 1 -- 172.21.15.107:0/1144336631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf81a88e0 con 0x7faaf81033e0 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.117+0000 7faafcc00640 1 -- 172.21.15.107:0/1144336631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaf81037f0 con 0x7faaf81033e0 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.119+0000 7faaf4c00640 1 -- 172.21.15.107:0/1144336631 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7faae8004b80 con 0x7faaf81033e0 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.121+0000 7faaf4c00640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faadc0b3c10 0x7faadc0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.121+0000 7faaf4c00640 1 -- 172.21.15.107:0/1144336631 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7faae80e2090 con 0x7faaf81033e0 2024-07-26T12:36:09.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.122+0000 7faaefe00640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faadc0b3c10 0x7faadc0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:09.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.124+0000 7faaefe00640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faadc0b3c10 0x7faadc0b6120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faae00058c0 tx=0x7faae0005850 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:09.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.126+0000 7faaf4c00640 1 -- 172.21.15.107:0/1144336631 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7faae80ab2d0 con 0x7faaf81033e0 2024-07-26T12:36:09.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.349+0000 7faafcc00640 1 -- 172.21.15.107:0/1144336631 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7faaf81a8c70 con 0x7faaf81033e0 2024-07-26T12:36:09.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.356+0000 7faaf4c00640 1 -- 172.21.15.107:0/1144336631 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v17)=0 v17) ==== 115+0+0 (secure 0 0 0) 0x7faae80aabd0 con 0x7faaf81033e0 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.358+0000 7faafcc00640 1 -- 172.21.15.107:0/1144336631 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faadc0b3c10 msgr2=0x7faadc0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.358+0000 7faafcc00640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faadc0b3c10 0x7faadc0b6120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faae00058c0 tx=0x7faae0005850 comp rx=0 tx=0).stop 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.358+0000 7faafcc00640 1 -- 172.21.15.107:0/1144336631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf81033e0 msgr2=0x7faaf819bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.358+0000 7faafcc00640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf81033e0 0x7faaf819bf20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faae800e9f0 tx=0x7faae800eeb0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.359+0000 7faafcc00640 1 -- 172.21.15.107:0/1144336631 shutdown_connections 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.359+0000 7faafcc00640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faadc0b3c10 0x7faadc0b6120 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.359+0000 7faafcc00640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faaf8109670 0x7faaf81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.359+0000 7faafcc00640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faaf8108d20 0x7faaf819c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.359+0000 7faafcc00640 1 --2- 172.21.15.107:0/1144336631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faaf81033e0 0x7faaf819bf20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.359+0000 7faafcc00640 1 -- 172.21.15.107:0/1144336631 >> 172.21.15.107:0/1144336631 conn(0x7faaf80fee70 msgr2=0x7faaf8106280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.359+0000 7faafcc00640 1 -- 172.21.15.107:0/1144336631 shutdown_connections 2024-07-26T12:36:09.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:09.359+0000 7faafcc00640 1 -- 172.21.15.107:0/1144336631 wait complete. 2024-07-26T12:36:09.906 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:09 smithi107 ceph-mon[32659]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:09.906 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:09.906 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:09.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:09.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:09.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:09.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:09.907 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:09 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1144336631' entity='client.admin' 2024-07-26T12:36:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:09 smithi112 ceph-mon[37501]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:09 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1144336631' entity='client.admin' 2024-07-26T12:36:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:09 smithi132 ceph-mon[37580]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:10.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:10.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:10.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:10.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:09 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1144336631' entity='client.admin' 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.352+0000 7fab46400640 1 -- 172.21.15.107:0/2879575105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab401017f0 msgr2=0x7fab40101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.352+0000 7fab46400640 1 --2- 172.21.15.107:0/2879575105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab401017f0 0x7fab40101c00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fab2c009900 tx=0x7fab2c02a410 comp rx=0 tx=0).stop 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.353+0000 7fab46400640 1 -- 172.21.15.107:0/2879575105 shutdown_connections 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.353+0000 7fab46400640 1 --2- 172.21.15.107:0/2879575105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab40102310 0x7fab40111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.353+0000 7fab46400640 1 --2- 172.21.15.107:0/2879575105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab401017f0 0x7fab40101c00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.353+0000 7fab46400640 1 --2- 172.21.15.107:0/2879575105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab401040e0 0x7fab401044f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.353+0000 7fab46400640 1 -- 172.21.15.107:0/2879575105 >> 172.21.15.107:0/2879575105 conn(0x7fab400799c0 msgr2=0x7fab40079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.353+0000 7fab46400640 1 -- 172.21.15.107:0/2879575105 shutdown_connections 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.353+0000 7fab46400640 1 -- 172.21.15.107:0/2879575105 wait complete. 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.354+0000 7fab46400640 1 Processor -- start 2024-07-26T12:36:10.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.354+0000 7fab46400640 1 -- start start 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.354+0000 7fab46400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab401017f0 0x7fab4019c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.354+0000 7fab46400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40102310 0x7fab4019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.354+0000 7fab46400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab401040e0 0x7fab401a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.354+0000 7fab46400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fab40115700 con 0x7fab401040e0 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.354+0000 7fab46400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab40115580 con 0x7fab40102310 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.354+0000 7fab46400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab40115880 con 0x7fab401017f0 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab401040e0 0x7fab401a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab401017f0 0x7fab4019c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab401040e0 0x7fab401a1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58874/0 (socket says 172.21.15.107:58874) 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3fe00640 1 -- 172.21.15.107:0/440378948 learned_addr learned my addr 172.21.15.107:0/440378948 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3ea00640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40102310 0x7fab4019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:10.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3ea00640 1 -- 172.21.15.107:0/440378948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab401017f0 msgr2=0x7fab4019c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3ea00640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab401017f0 0x7fab4019c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3ea00640 1 -- 172.21.15.107:0/440378948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab401040e0 msgr2=0x7fab401a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3ea00640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab401040e0 0x7fab401a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3ea00640 1 -- 172.21.15.107:0/440378948 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab401a20a0 con 0x7fab40102310 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3fe00640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab401040e0 0x7fab401a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.355+0000 7fab3ea00640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40102310 0x7fab4019c6d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fab2c02a930 tx=0x7fab2c031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.356+0000 7fab37e00640 1 -- 172.21.15.107:0/440378948 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c03f070 con 0x7fab40102310 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.356+0000 7fab37e00640 1 -- 172.21.15.107:0/440378948 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab2c004490 con 0x7fab40102310 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.356+0000 7fab46400640 1 -- 172.21.15.107:0/440378948 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab401a2330 con 0x7fab40102310 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.356+0000 7fab37e00640 1 -- 172.21.15.107:0/440378948 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab2c035070 con 0x7fab40102310 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.356+0000 7fab46400640 1 -- 172.21.15.107:0/440378948 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab401a8af0 con 0x7fab40102310 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.357+0000 7fab46400640 1 -- 172.21.15.107:0/440378948 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab40102770 con 0x7fab40102310 2024-07-26T12:36:10.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.359+0000 7fab37e00640 1 -- 172.21.15.107:0/440378948 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fab2c03b460 con 0x7fab40102310 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.361+0000 7fab37e00640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab240b3c10 0x7fab240b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.361+0000 7fab37e00640 1 -- 172.21.15.107:0/440378948 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fab2c1026c0 con 0x7fab40102310 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.361+0000 7fab3f400640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab240b3c10 0x7fab240b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.362+0000 7fab3f400640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab240b3c10 0x7fab240b6120 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fab30009bd0 tx=0x7fab30005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.363+0000 7fab37e00640 1 -- 172.21.15.107:0/440378948 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fab2c0cb220 con 0x7fab40102310 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.572+0000 7fab46400640 1 -- 172.21.15.107:0/440378948 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7fab40101c00 con 0x7fab40102310 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.580+0000 7fab37e00640 1 -- 172.21.15.107:0/440378948 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v18)=0 v18) ==== 119+0+0 (secure 0 0 0) 0x7fab40101c00 con 0x7fab40102310 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 -- 172.21.15.107:0/440378948 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab240b3c10 msgr2=0x7fab240b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab240b3c10 0x7fab240b6120 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fab30009bd0 tx=0x7fab30005f90 comp rx=0 tx=0).stop 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 -- 172.21.15.107:0/440378948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40102310 msgr2=0x7fab4019c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40102310 0x7fab4019c6d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fab2c02a930 tx=0x7fab2c031a20 comp rx=0 tx=0).stop 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 -- 172.21.15.107:0/440378948 shutdown_connections 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab240b3c10 0x7fab240b6120 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab401040e0 0x7fab401a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:10.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab40102310 0x7fab4019c6d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:10.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 --2- 172.21.15.107:0/440378948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab401017f0 0x7fab4019c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:10.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 -- 172.21.15.107:0/440378948 >> 172.21.15.107:0/440378948 conn(0x7fab400799c0 msgr2=0x7fab40100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:10.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 -- 172.21.15.107:0/440378948 shutdown_connections 2024-07-26T12:36:10.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.584+0000 7fab46400640 1 -- 172.21.15.107:0/440378948 wait complete. 2024-07-26T12:36:11.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.569+0000 7fa276800640 1 -- 172.21.15.107:0/1292478358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701055e0 msgr2=0x7fa2701059f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.569+0000 7fa276800640 1 --2- 172.21.15.107:0/1292478358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701055e0 0x7fa2701059f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa264009900 tx=0x7fa26402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.570+0000 7fa276800640 1 -- 172.21.15.107:0/1292478358 shutdown_connections 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.570+0000 7fa276800640 1 --2- 172.21.15.107:0/1292478358 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa27010b870 0x7fa2701120a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.570+0000 7fa276800640 1 --2- 172.21.15.107:0/1292478358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa27010af20 0x7fa27010b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.570+0000 7fa276800640 1 --2- 172.21.15.107:0/1292478358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701055e0 0x7fa2701059f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.570+0000 7fa276800640 1 -- 172.21.15.107:0/1292478358 >> 172.21.15.107:0/1292478358 conn(0x7fa2701010b0 msgr2=0x7fa270103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.570+0000 7fa276800640 1 -- 172.21.15.107:0/1292478358 shutdown_connections 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.570+0000 7fa276800640 1 -- 172.21.15.107:0/1292478358 wait complete. 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.571+0000 7fa276800640 1 Processor -- start 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.572+0000 7fa276800640 1 -- start start 2024-07-26T12:36:11.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.572+0000 7fa276800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701055e0 0x7fa27019bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.572+0000 7fa276800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa27010af20 0x7fa27019c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.572+0000 7fa276800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa27010b870 0x7fa2701a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.572+0000 7fa276800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa270115520 con 0x7fa2701055e0 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.572+0000 7fa276800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa2701153a0 con 0x7fa27010b870 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.572+0000 7fa276800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa2701156a0 con 0x7fa27010af20 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.572+0000 7fa275400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701055e0 0x7fa27019bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa275400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701055e0 0x7fa27019bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58894/0 (socket says 172.21.15.107:58894) 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa275400640 1 -- 172.21.15.107:0/3324783565 learned_addr learned my addr 172.21.15.107:0/3324783565 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa275e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa27010b870 0x7fa2701a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa274a00640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa27010af20 0x7fa27019c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa275400640 1 -- 172.21.15.107:0/3324783565 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa27010af20 msgr2=0x7fa27019c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa275400640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa27010af20 0x7fa27019c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa275400640 1 -- 172.21.15.107:0/3324783565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa27010b870 msgr2=0x7fa2701a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa275400640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa27010b870 0x7fa2701a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa275400640 1 -- 172.21.15.107:0/3324783565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2701a1f00 con 0x7fa2701055e0 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa275e00640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa27010b870 0x7fa2701a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.573+0000 7fa275400640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701055e0 0x7fa27019bf90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa264005940 tx=0x7fa2640042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.574+0000 7fa26e000640 1 -- 172.21.15.107:0/3324783565 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26403f030 con 0x7fa2701055e0 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.574+0000 7fa26e000640 1 -- 172.21.15.107:0/3324783565 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa264003cb0 con 0x7fa2701055e0 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.574+0000 7fa276800640 1 -- 172.21.15.107:0/3324783565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2701a2190 con 0x7fa2701055e0 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.574+0000 7fa26e000640 1 -- 172.21.15.107:0/3324783565 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264035070 con 0x7fa2701055e0 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.574+0000 7fa276800640 1 -- 172.21.15.107:0/3324783565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2701a8950 con 0x7fa2701055e0 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.575+0000 7fa276800640 1 -- 172.21.15.107:0/3324783565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2701059f0 con 0x7fa2701055e0 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.578+0000 7fa26e000640 1 -- 172.21.15.107:0/3324783565 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fa264042030 con 0x7fa2701055e0 2024-07-26T12:36:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.579+0000 7fa26e000640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa2580b3b90 0x7fa2580b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.579+0000 7fa26e000640 1 -- 172.21.15.107:0/3324783565 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa2641028b0 con 0x7fa2701055e0 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.580+0000 7fa274a00640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa2580b3b90 0x7fa2580b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.580+0000 7fa274a00640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa2580b3b90 0x7fa2580b60a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa27010ab90 tx=0x7fa25c00b390 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.585+0000 7fa26e000640 1 -- 172.21.15.107:0/3324783565 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa2640cb410 con 0x7fa2701055e0 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.795+0000 7fa276800640 1 -- 172.21.15.107:0/3324783565 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fa27019d0c0 con 0x7fa2701055e0 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.804+0000 7fa26e000640 1 -- 172.21.15.107:0/3324783565 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v19)=0 v19) ==== 106+0+0 (secure 0 0 0) 0x7fa2640cf960 con 0x7fa2701055e0 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.807+0000 7fa276800640 1 -- 172.21.15.107:0/3324783565 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa2580b3b90 msgr2=0x7fa2580b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.807+0000 7fa276800640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa2580b3b90 0x7fa2580b60a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa27010ab90 tx=0x7fa25c00b390 comp rx=0 tx=0).stop 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.807+0000 7fa276800640 1 -- 172.21.15.107:0/3324783565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701055e0 msgr2=0x7fa27019bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.807+0000 7fa276800640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701055e0 0x7fa27019bf90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa264005940 tx=0x7fa2640042f0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.808+0000 7fa276800640 1 -- 172.21.15.107:0/3324783565 shutdown_connections 2024-07-26T12:36:11.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.808+0000 7fa276800640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa2580b3b90 0x7fa2580b60a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.808+0000 7fa276800640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa27010b870 0x7fa2701a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.808+0000 7fa276800640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa27010af20 0x7fa27019c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.808+0000 7fa276800640 1 --2- 172.21.15.107:0/3324783565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2701055e0 0x7fa27019bf90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.808+0000 7fa276800640 1 -- 172.21.15.107:0/3324783565 >> 172.21.15.107:0/3324783565 conn(0x7fa2701010b0 msgr2=0x7fa2701034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.808+0000 7fa276800640 1 -- 172.21.15.107:0/3324783565 shutdown_connections 2024-07-26T12:36:11.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:10.808+0000 7fa276800640 1 -- 172.21.15.107:0/3324783565 wait complete. 2024-07-26T12:36:11.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:11 smithi107 ceph-mon[32659]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:11.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:11 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:36:11.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:11 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3324783565' entity='client.admin' 2024-07-26T12:36:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:11 smithi112 ceph-mon[37501]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:11 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:36:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:11 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3324783565' entity='client.admin' 2024-07-26T12:36:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:11 smithi132 ceph-mon[37580]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:11 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:36:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:11 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3324783565' entity='client.admin' 2024-07-26T12:36:12.450 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi107') 2024-07-26T12:36:12.451 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi107') 2024-07-26T12:36:12.451 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi107') 2024-07-26T12:36:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.194+0000 7fc1b2c00640 1 -- 172.21.15.107:0/1818432933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1ac112050 msgr2=0x7fc1ac118910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.194+0000 7fc1b2c00640 1 --2- 172.21.15.107:0/1818432933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1ac112050 0x7fc1ac118910 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4009ca0 tx=0x7fc1a402a8c0 comp rx=0 tx=0).stop 2024-07-26T12:36:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.194+0000 7fc1b2c00640 1 -- 172.21.15.107:0/1818432933 shutdown_connections 2024-07-26T12:36:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.194+0000 7fc1b2c00640 1 --2- 172.21.15.107:0/1818432933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1ac112050 0x7fc1ac118910 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.194+0000 7fc1b2c00640 1 --2- 172.21.15.107:0/1818432933 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1ac111700 0x7fc1ac111b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.194+0000 7fc1b2c00640 1 --2- 172.21.15.107:0/1818432933 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ac10a250 0x7fc1ac10a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.194+0000 7fc1b2c00640 1 -- 172.21.15.107:0/1818432933 >> 172.21.15.107:0/1818432933 conn(0x7fc1ac074110 msgr2=0x7fc1ac074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.195+0000 7fc1b2c00640 1 -- 172.21.15.107:0/1818432933 shutdown_connections 2024-07-26T12:36:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.195+0000 7fc1b2c00640 1 -- 172.21.15.107:0/1818432933 wait complete. 2024-07-26T12:36:13.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.196+0000 7fc1b2c00640 1 Processor -- start 2024-07-26T12:36:13.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1b2c00640 1 -- start start 2024-07-26T12:36:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1b2c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1ac10a250 0x7fc1ac10b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1b2c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ac111700 0x7fc1ac10b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1b2c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1ac112050 0x7fc1ac10bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1b2c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ac11bd00 con 0x7fc1ac112050 2024-07-26T12:36:13.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1b2c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ac11bb80 con 0x7fc1ac111700 2024-07-26T12:36:13.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1b2c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ac11be80 con 0x7fc1ac10a250 2024-07-26T12:36:13.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1abe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1ac10a250 0x7fc1ac10b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:13.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1abe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1ac10a250 0x7fc1ac10b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:34258/0 (socket says 172.21.15.107:34258) 2024-07-26T12:36:13.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1abe00640 1 -- 172.21.15.107:0/2067400343 learned_addr learned my addr 172.21.15.107:0/2067400343 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:13.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.197+0000 7fc1ab400640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ac111700 0x7fc1ac10b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:13.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.198+0000 7fc1b0c00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1ac112050 0x7fc1ac10bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:13.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.198+0000 7fc1abe00640 1 -- 172.21.15.107:0/2067400343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ac111700 msgr2=0x7fc1ac10b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:13.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.198+0000 7fc1abe00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ac111700 0x7fc1ac10b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:13.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.198+0000 7fc1abe00640 1 -- 172.21.15.107:0/2067400343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1ac112050 msgr2=0x7fc1ac10bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:13.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.198+0000 7fc1abe00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1ac112050 0x7fc1ac10bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:13.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.198+0000 7fc1abe00640 1 -- 172.21.15.107:0/2067400343 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1ac1b0990 con 0x7fc1ac10a250 2024-07-26T12:36:13.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.198+0000 7fc1abe00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1ac10a250 0x7fc1ac10b1f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00d950 tx=0x7fc19c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:13.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.199+0000 7fc1a8c00640 1 -- 172.21.15.107:0/2067400343 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c017030 con 0x7fc1ac10a250 2024-07-26T12:36:13.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.199+0000 7fc1a8c00640 1 -- 172.21.15.107:0/2067400343 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc19c004020 con 0x7fc1ac10a250 2024-07-26T12:36:13.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.199+0000 7fc1a8c00640 1 -- 172.21.15.107:0/2067400343 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0047a0 con 0x7fc1ac10a250 2024-07-26T12:36:13.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.199+0000 7fc1b2c00640 1 -- 172.21.15.107:0/2067400343 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1ac1b0c80 con 0x7fc1ac10a250 2024-07-26T12:36:13.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.199+0000 7fc1b2c00640 1 -- 172.21.15.107:0/2067400343 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1ac1b1170 con 0x7fc1ac10a250 2024-07-26T12:36:13.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.200+0000 7fc1b2c00640 1 -- 172.21.15.107:0/2067400343 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1ac10a660 con 0x7fc1ac10a250 2024-07-26T12:36:13.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.202+0000 7fc1a8c00640 1 -- 172.21.15.107:0/2067400343 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fc19c009a30 con 0x7fc1ac10a250 2024-07-26T12:36:13.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.204+0000 7fc1a8c00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1900b3c10 0x7fc1900b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:13.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.204+0000 7fc1a8c00640 1 -- 172.21.15.107:0/2067400343 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fc19c0e1cb0 con 0x7fc1ac10a250 2024-07-26T12:36:13.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.204+0000 7fc1ab400640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1900b3c10 0x7fc1900b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:13.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.205+0000 7fc1ab400640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1900b3c10 0x7fc1900b6120 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac111d50 tx=0x7fc19800a250 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:13.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.205+0000 7fc1a8c00640 1 -- 172.21.15.107:0/2067400343 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc19c0aaee0 con 0x7fc1ac10a250 2024-07-26T12:36:13.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.407+0000 7fc1b2c00640 1 -- 172.21.15.107:0/2067400343 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=debug_cephsqlite}] v 0) -- 0x7fc1ac10c4f0 con 0x7fc1ac10a250 2024-07-26T12:36:13.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.417+0000 7fc1a8c00640 1 -- 172.21.15.107:0/2067400343 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_cephsqlite}]=0 v20)=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7fc19c0aa7e0 con 0x7fc1ac10a250 2024-07-26T12:36:13.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.421+0000 7fc1b2c00640 1 -- 172.21.15.107:0/2067400343 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1900b3c10 msgr2=0x7fc1900b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.421+0000 7fc1b2c00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1900b3c10 0x7fc1900b6120 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac111d50 tx=0x7fc19800a250 comp rx=0 tx=0).stop 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.421+0000 7fc1b2c00640 1 -- 172.21.15.107:0/2067400343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1ac10a250 msgr2=0x7fc1ac10b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.421+0000 7fc1b2c00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1ac10a250 0x7fc1ac10b1f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00d950 tx=0x7fc19c00de10 comp rx=0 tx=0).stop 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.421+0000 7fc1b2c00640 1 -- 172.21.15.107:0/2067400343 shutdown_connections 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.421+0000 7fc1b2c00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1900b3c10 0x7fc1900b6120 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.421+0000 7fc1b2c00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1ac112050 0x7fc1ac10bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.421+0000 7fc1b2c00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1ac111700 0x7fc1ac10b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.421+0000 7fc1b2c00640 1 --2- 172.21.15.107:0/2067400343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1ac10a250 0x7fc1ac10b1f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.421+0000 7fc1b2c00640 1 -- 172.21.15.107:0/2067400343 >> 172.21.15.107:0/2067400343 conn(0x7fc1ac074110 msgr2=0x7fc1ac10f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.422+0000 7fc1b2c00640 1 -- 172.21.15.107:0/2067400343 shutdown_connections 2024-07-26T12:36:13.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:13.422+0000 7fc1b2c00640 1 -- 172.21.15.107:0/2067400343 wait complete. 2024-07-26T12:36:13.665 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:13 smithi107 ceph-mon[32659]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:13.665 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:13 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:36:13.665 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:13.665 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:13.665 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:13.665 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:13 smithi112 ceph-mon[37501]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:13 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:36:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:13 smithi132 ceph-mon[37580]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:13 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:36:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:14.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.189+0000 7f9680000640 1 -- 172.21.15.107:0/28143044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9678076db0 msgr2=0x7f96780751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:14.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.189+0000 7f9680000640 1 --2- 172.21.15.107:0/28143044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9678076db0 0x7f96780751e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f966c009900 tx=0x7f966c02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:36:14.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.189+0000 7f9680000640 1 -- 172.21.15.107:0/28143044 shutdown_connections 2024-07-26T12:36:14.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.189+0000 7f9680000640 1 --2- 172.21.15.107:0/28143044 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9678113e60 0x7f9678116330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:14.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.189+0000 7f9680000640 1 --2- 172.21.15.107:0/28143044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9678075800 0x7f9678075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:14.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.189+0000 7f9680000640 1 --2- 172.21.15.107:0/28143044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9678076db0 0x7f96780751e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:14.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.189+0000 7f9680000640 1 -- 172.21.15.107:0/28143044 >> 172.21.15.107:0/28143044 conn(0x7f967807b280 msgr2=0x7f967807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:14.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.189+0000 7f9680000640 1 -- 172.21.15.107:0/28143044 shutdown_connections 2024-07-26T12:36:14.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.189+0000 7f9680000640 1 -- 172.21.15.107:0/28143044 wait complete. 2024-07-26T12:36:14.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.191+0000 7f9680000640 1 Processor -- start 2024-07-26T12:36:14.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.191+0000 7f9680000640 1 -- start start 2024-07-26T12:36:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f9680000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9678075800 0x7f96781a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f9680000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9678076db0 0x7f96781a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f9680000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9678113e60 0x7f96781a5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f9680000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9678119af0 con 0x7f9678113e60 2024-07-26T12:36:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f9680000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9678119970 con 0x7f9678076db0 2024-07-26T12:36:14.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f9680000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9678119c70 con 0x7f9678075800 2024-07-26T12:36:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f967d600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9678075800 0x7f96781a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f967e000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9678113e60 0x7f96781a5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f967d600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9678075800 0x7f96781a0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:50924/0 (socket says 172.21.15.107:50924) 2024-07-26T12:36:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f967d600640 1 -- 172.21.15.107:0/2889456010 learned_addr learned my addr 172.21.15.107:0/2889456010 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.192+0000 7f967cc00640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9678076db0 0x7f96781a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.193+0000 7f967e000640 1 -- 172.21.15.107:0/2889456010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9678075800 msgr2=0x7f96781a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.193+0000 7f967e000640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9678075800 0x7f96781a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:14.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.193+0000 7f967e000640 1 -- 172.21.15.107:0/2889456010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9678076db0 msgr2=0x7f96781a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.193+0000 7f967e000640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9678076db0 0x7f96781a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.193+0000 7f967e000640 1 -- 172.21.15.107:0/2889456010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96781a6410 con 0x7f9678113e60 2024-07-26T12:36:14.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.193+0000 7f967e000640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9678113e60 0x7f96781a5da0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f967000cab0 tx=0x7f967000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.194+0000 7f9676000640 1 -- 172.21.15.107:0/2889456010 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967000ae20 con 0x7f9678113e60 2024-07-26T12:36:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.194+0000 7f9676000640 1 -- 172.21.15.107:0/2889456010 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f967000f900 con 0x7f9678113e60 2024-07-26T12:36:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.194+0000 7f9680000640 1 -- 172.21.15.107:0/2889456010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96781a6700 con 0x7f9678113e60 2024-07-26T12:36:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.194+0000 7f9676000640 1 -- 172.21.15.107:0/2889456010 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96700189f0 con 0x7f9678113e60 2024-07-26T12:36:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.194+0000 7f9680000640 1 -- 172.21.15.107:0/2889456010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96781acf20 con 0x7f9678113e60 2024-07-26T12:36:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.195+0000 7f9680000640 1 -- 172.21.15.107:0/2889456010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96580052c0 con 0x7f9678113e60 2024-07-26T12:36:14.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.200+0000 7f9676000640 1 -- 172.21.15.107:0/2889456010 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f9670018b90 con 0x7f9678113e60 2024-07-26T12:36:14.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.204+0000 7f9676000640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96600b3cc0 0x7f96600b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:14.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.205+0000 7f967d600640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96600b3cc0 0x7f96600b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:14.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.205+0000 7f9676000640 1 -- 172.21.15.107:0/2889456010 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f96700e2570 con 0x7f9678113e60 2024-07-26T12:36:14.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.205+0000 7f9676000640 1 -- 172.21.15.107:0/2889456010 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9670014030 con 0x7f9678113e60 2024-07-26T12:36:14.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.206+0000 7f967d600640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96600b3cc0 0x7f96600b61d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f966c000c00 tx=0x7f966c031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:14.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.433+0000 7f9680000640 1 -- 172.21.15.107:0/2889456010 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f96580055b0 con 0x7f9678113e60 2024-07-26T12:36:14.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.442+0000 7f9676000640 1 -- 172.21.15.107:0/2889456010 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v21)=0 v21) ==== 122+0+0 (secure 0 0 0) 0x7f96700ab7a0 con 0x7f9678113e60 2024-07-26T12:36:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.445+0000 7f9680000640 1 -- 172.21.15.107:0/2889456010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96600b3cc0 msgr2=0x7f96600b61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.445+0000 7f9680000640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96600b3cc0 0x7f96600b61d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f966c000c00 tx=0x7f966c031400 comp rx=0 tx=0).stop 2024-07-26T12:36:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.445+0000 7f9680000640 1 -- 172.21.15.107:0/2889456010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9678113e60 msgr2=0x7f96781a5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:14.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.445+0000 7f9680000640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9678113e60 0x7f96781a5da0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f967000cab0 tx=0x7f967000cf70 comp rx=0 tx=0).stop 2024-07-26T12:36:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.446+0000 7f9680000640 1 -- 172.21.15.107:0/2889456010 shutdown_connections 2024-07-26T12:36:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.446+0000 7f9680000640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96600b3cc0 0x7f96600b61d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.446+0000 7f9680000640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9678113e60 0x7f96781a5da0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.446+0000 7f9680000640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9678076db0 0x7f96781a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.446+0000 7f9680000640 1 --2- 172.21.15.107:0/2889456010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9678075800 0x7f96781a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.446+0000 7f9680000640 1 -- 172.21.15.107:0/2889456010 >> 172.21.15.107:0/2889456010 conn(0x7f967807b280 msgr2=0x7f9678112b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.446+0000 7f9680000640 1 -- 172.21.15.107:0/2889456010 shutdown_connections 2024-07-26T12:36:14.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:14.446+0000 7f9680000640 1 -- 172.21.15.107:0/2889456010 wait complete. 2024-07-26T12:36:15.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.048+0000 7f28a2000640 1 -- 172.21.15.107:0/627868196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289c10fb00 msgr2=0x7f289c111f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:15.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.048+0000 7f28a2000640 1 --2- 172.21.15.107:0/627868196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289c10fb00 0x7f289c111f40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2894009900 tx=0x7f289402a410 comp rx=0 tx=0).stop 2024-07-26T12:36:15.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.048+0000 7f28a2000640 1 -- 172.21.15.107:0/627868196 shutdown_connections 2024-07-26T12:36:15.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.048+0000 7f28a2000640 1 --2- 172.21.15.107:0/627868196 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289c10fb00 0x7f289c111f40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.048+0000 7f28a2000640 1 --2- 172.21.15.107:0/627868196 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f289c075b30 0x7f289c10f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.048+0000 7f28a2000640 1 --2- 172.21.15.107:0/627868196 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f289c0751e0 0x7f289c0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.048+0000 7f28a2000640 1 -- 172.21.15.107:0/627868196 >> 172.21.15.107:0/627868196 conn(0x7f289c0ff010 msgr2=0x7f289c101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:15.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.048+0000 7f28a2000640 1 -- 172.21.15.107:0/627868196 shutdown_connections 2024-07-26T12:36:15.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.048+0000 7f28a2000640 1 -- 172.21.15.107:0/627868196 wait complete. 2024-07-26T12:36:15.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.050+0000 7f28a2000640 1 Processor -- start 2024-07-26T12:36:15.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.050+0000 7f28a2000640 1 -- start start 2024-07-26T12:36:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.050+0000 7f28a2000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289c0751e0 0x7f289c19c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.050+0000 7f28a2000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f289c075b30 0x7f289c19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.050+0000 7f28a2000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f289c10fb00 0x7f289c1a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.050+0000 7f28a2000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f289c115700 con 0x7f289c0751e0 2024-07-26T12:36:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.050+0000 7f28a2000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f289c115580 con 0x7f289c075b30 2024-07-26T12:36:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.050+0000 7f28a2000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f289c115880 con 0x7f289c10fb00 2024-07-26T12:36:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289c0751e0 0x7f289c19c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289c0751e0 0x7f289c19c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54262/0 (socket says 172.21.15.107:54262) 2024-07-26T12:36:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289b400640 1 -- 172.21.15.107:0/378704285 learned_addr learned my addr 172.21.15.107:0/378704285 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:15.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289aa00640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f289c075b30 0x7f289c19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:15.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289b400640 1 -- 172.21.15.107:0/378704285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f289c10fb00 msgr2=0x7f289c1a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:15.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289b400640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f289c10fb00 0x7f289c1a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289b400640 1 -- 172.21.15.107:0/378704285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f289c075b30 msgr2=0x7f289c19c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:15.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289b400640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f289c075b30 0x7f289c19c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289b400640 1 -- 172.21.15.107:0/378704285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f289c1a2000 con 0x7f289c0751e0 2024-07-26T12:36:15.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289aa00640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f289c075b30 0x7f289c19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:15.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.051+0000 7f289b400640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289c0751e0 0x7f289c19c090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f288c00ebc0 tx=0x7f288c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:15.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.052+0000 7f2893e00640 1 -- 172.21.15.107:0/378704285 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c010630 con 0x7f289c0751e0 2024-07-26T12:36:15.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.052+0000 7f2893e00640 1 -- 172.21.15.107:0/378704285 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f288c010c80 con 0x7f289c0751e0 2024-07-26T12:36:15.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.052+0000 7f2893e00640 1 -- 172.21.15.107:0/378704285 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288c018b60 con 0x7f289c0751e0 2024-07-26T12:36:15.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.052+0000 7f28a2000640 1 -- 172.21.15.107:0/378704285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f289c1a22f0 con 0x7f289c0751e0 2024-07-26T12:36:15.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.052+0000 7f28a2000640 1 -- 172.21.15.107:0/378704285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f289c1a8a50 con 0x7f289c0751e0 2024-07-26T12:36:15.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.053+0000 7f28a2000640 1 -- 172.21.15.107:0/378704285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f289c0755f0 con 0x7f289c0751e0 2024-07-26T12:36:15.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.056+0000 7f2893e00640 1 -- 172.21.15.107:0/378704285 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f288c018d00 con 0x7f289c0751e0 2024-07-26T12:36:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.058+0000 7f2893e00640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f28800b3c90 0x7f28800b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.058+0000 7f2893e00640 1 -- 172.21.15.107:0/378704285 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f288c0e2b70 con 0x7f289c0751e0 2024-07-26T12:36:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.058+0000 7f289aa00640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f28800b3c90 0x7f28800b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:15.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.059+0000 7f289aa00640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f28800b3c90 0x7f28800b61a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f289c111d90 tx=0x7f2884005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:15.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.060+0000 7f2893e00640 1 -- 172.21.15.107:0/378704285 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f288c0abd20 con 0x7f289c0751e0 2024-07-26T12:36:15.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.270+0000 7f28a2000640 1 -- 172.21.15.107:0/378704285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f289c19d1c0 con 0x7f289c0751e0 2024-07-26T12:36:15.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.279+0000 7f2893e00640 1 -- 172.21.15.107:0/378704285 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v22)=0 v22) ==== 120+0+0 (secure 0 0 0) 0x7f288c0ab620 con 0x7f289c0751e0 2024-07-26T12:36:15.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.284+0000 7f28a2000640 1 -- 172.21.15.107:0/378704285 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f28800b3c90 msgr2=0x7f28800b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:15.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.284+0000 7f28a2000640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f28800b3c90 0x7f28800b61a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f289c111d90 tx=0x7f2884005f90 comp rx=0 tx=0).stop 2024-07-26T12:36:15.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.284+0000 7f28a2000640 1 -- 172.21.15.107:0/378704285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289c0751e0 msgr2=0x7f289c19c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:15.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.284+0000 7f28a2000640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289c0751e0 0x7f289c19c090 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f288c00ebc0 tx=0x7f288c00c6a0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.284+0000 7f28a2000640 1 -- 172.21.15.107:0/378704285 shutdown_connections 2024-07-26T12:36:15.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.284+0000 7f28a2000640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f28800b3c90 0x7f28800b61a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.284+0000 7f28a2000640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f289c10fb00 0x7f289c1a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.284+0000 7f28a2000640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f289c075b30 0x7f289c19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.284+0000 7f28a2000640 1 --2- 172.21.15.107:0/378704285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f289c0751e0 0x7f289c19c090 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.284+0000 7f28a2000640 1 -- 172.21.15.107:0/378704285 >> 172.21.15.107:0/378704285 conn(0x7f289c0ff010 msgr2=0x7f289c109610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:15.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.285+0000 7f28a2000640 1 -- 172.21.15.107:0/378704285 shutdown_connections 2024-07-26T12:36:15.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.285+0000 7f28a2000640 1 -- 172.21.15.107:0/378704285 wait complete. 2024-07-26T12:36:15.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:15 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2889456010' entity='client.admin' 2024-07-26T12:36:15.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:15 smithi107 ceph-mon[32659]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:15.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:15 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/378704285' entity='client.admin' 2024-07-26T12:36:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:15 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2889456010' entity='client.admin' 2024-07-26T12:36:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:15 smithi112 ceph-mon[37501]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:15 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/378704285' entity='client.admin' 2024-07-26T12:36:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:15 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2889456010' entity='client.admin' 2024-07-26T12:36:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:15 smithi132 ceph-mon[37580]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:15 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/378704285' entity='client.admin' 2024-07-26T12:36:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.888+0000 7f46d1400640 1 -- 172.21.15.107:0/2301115075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46cc1033e0 msgr2=0x7f46cc1037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.888+0000 7f46d1400640 1 --2- 172.21.15.107:0/2301115075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46cc1033e0 0x7f46cc1037f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f46bc009900 tx=0x7f46bc02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.888+0000 7f46d1400640 1 -- 172.21.15.107:0/2301115075 shutdown_connections 2024-07-26T12:36:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.888+0000 7f46d1400640 1 --2- 172.21.15.107:0/2301115075 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46cc109670 0x7f46cc10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.888+0000 7f46d1400640 1 --2- 172.21.15.107:0/2301115075 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc108d20 0x7f46cc109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.888+0000 7f46d1400640 1 --2- 172.21.15.107:0/2301115075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46cc1033e0 0x7f46cc1037f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.888+0000 7f46d1400640 1 -- 172.21.15.107:0/2301115075 >> 172.21.15.107:0/2301115075 conn(0x7f46cc0fee70 msgr2=0x7f46cc1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.888+0000 7f46d1400640 1 -- 172.21.15.107:0/2301115075 shutdown_connections 2024-07-26T12:36:15.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.889+0000 7f46d1400640 1 -- 172.21.15.107:0/2301115075 wait complete. 2024-07-26T12:36:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.890+0000 7f46d1400640 1 Processor -- start 2024-07-26T12:36:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.890+0000 7f46d1400640 1 -- start start 2024-07-26T12:36:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.890+0000 7f46d1400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46cc1033e0 0x7f46cc199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.890+0000 7f46d1400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46cc108d20 0x7f46cc19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.890+0000 7f46d1400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc109670 0x7f46cc19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.890+0000 7f46d1400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc113340 con 0x7f46cc108d20 2024-07-26T12:36:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.890+0000 7f46d1400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc1131c0 con 0x7f46cc109670 2024-07-26T12:36:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.890+0000 7f46d1400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc1134c0 con 0x7f46cc1033e0 2024-07-26T12:36:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46cb400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46cc108d20 0x7f46cc19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46cb400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46cc108d20 0x7f46cc19a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54276/0 (socket says 172.21.15.107:54276) 2024-07-26T12:36:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46cb400640 1 -- 172.21.15.107:0/685354852 learned_addr learned my addr 172.21.15.107:0/685354852 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46d0a00640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc109670 0x7f46cc19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46cbe00640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46cc1033e0 0x7f46cc199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46cb400640 1 -- 172.21.15.107:0/685354852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46cc1033e0 msgr2=0x7f46cc199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46cb400640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46cc1033e0 0x7f46cc199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46cb400640 1 -- 172.21.15.107:0/685354852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc109670 msgr2=0x7f46cc19f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:15.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46cb400640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc109670 0x7f46cc19f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:15.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46cb400640 1 -- 172.21.15.107:0/685354852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc19fd00 con 0x7f46cc108d20 2024-07-26T12:36:15.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46d0a00640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc109670 0x7f46cc19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.891+0000 7f46cb400640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46cc108d20 0x7f46cc19a330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f46b800e970 tx=0x7f46b800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.892+0000 7f46c8c00640 1 -- 172.21.15.107:0/685354852 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b800cd80 con 0x7f46cc108d20 2024-07-26T12:36:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.892+0000 7f46c8c00640 1 -- 172.21.15.107:0/685354852 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46b80035f0 con 0x7f46cc108d20 2024-07-26T12:36:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.892+0000 7f46c8c00640 1 -- 172.21.15.107:0/685354852 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8010690 con 0x7f46cc108d20 2024-07-26T12:36:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.892+0000 7f46d1400640 1 -- 172.21.15.107:0/685354852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc19fff0 con 0x7f46cc108d20 2024-07-26T12:36:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.892+0000 7f46d1400640 1 -- 172.21.15.107:0/685354852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc1a6750 con 0x7f46cc108d20 2024-07-26T12:36:15.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.893+0000 7f46d1400640 1 -- 172.21.15.107:0/685354852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46cc1037f0 con 0x7f46cc108d20 2024-07-26T12:36:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.896+0000 7f46c8c00640 1 -- 172.21.15.107:0/685354852 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f46b8003aa0 con 0x7f46cc108d20 2024-07-26T12:36:15.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.898+0000 7f46c8c00640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f46b00b3c10 0x7f46b00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:15.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.899+0000 7f46c8c00640 1 -- 172.21.15.107:0/685354852 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f46b80e2280 con 0x7f46cc108d20 2024-07-26T12:36:15.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.899+0000 7f46cbe00640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f46b00b3c10 0x7f46b00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:15.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.899+0000 7f46cbe00640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f46b00b3c10 0x7f46b00b6120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f46bc031440 tx=0x7f46bc009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:15.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:15.903+0000 7f46c8c00640 1 -- 172.21.15.107:0/685354852 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f46b80ab430 con 0x7f46cc108d20 2024-07-26T12:36:16.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.091+0000 7f46d1400640 1 -- 172.21.15.107:0/685354852 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f46cc02e0c0 con 0x7f46cc108d20 2024-07-26T12:36:16.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.100+0000 7f46c8c00640 1 -- 172.21.15.107:0/685354852 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v23)=0 v23) ==== 116+0+0 (secure 0 0 0) 0x7f46b80aad30 con 0x7f46cc108d20 2024-07-26T12:36:16.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.104+0000 7f46d1400640 1 -- 172.21.15.107:0/685354852 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f46b00b3c10 msgr2=0x7f46b00b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:16.105 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.104+0000 7f46d1400640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f46b00b3c10 0x7f46b00b6120 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f46bc031440 tx=0x7f46bc009560 comp rx=0 tx=0).stop 2024-07-26T12:36:16.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.104+0000 7f46d1400640 1 -- 172.21.15.107:0/685354852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46cc108d20 msgr2=0x7f46cc19a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:16.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.104+0000 7f46d1400640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46cc108d20 0x7f46cc19a330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f46b800e970 tx=0x7f46b800ee30 comp rx=0 tx=0).stop 2024-07-26T12:36:16.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.105+0000 7f46d1400640 1 -- 172.21.15.107:0/685354852 shutdown_connections 2024-07-26T12:36:16.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.105+0000 7f46d1400640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f46b00b3c10 0x7f46b00b6120 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.105+0000 7f46d1400640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f46cc109670 0x7f46cc19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.105+0000 7f46d1400640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46cc108d20 0x7f46cc19a330 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.105+0000 7f46d1400640 1 --2- 172.21.15.107:0/685354852 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46cc1033e0 0x7f46cc199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.105+0000 7f46d1400640 1 -- 172.21.15.107:0/685354852 >> 172.21.15.107:0/685354852 conn(0x7f46cc0fee70 msgr2=0x7f46cc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.105+0000 7f46d1400640 1 -- 172.21.15.107:0/685354852 shutdown_connections 2024-07-26T12:36:16.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:16.105+0000 7f46d1400640 1 -- 172.21.15.107:0/685354852 wait complete. 2024-07-26T12:36:17.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:17 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/685354852' entity='client.admin' 2024-07-26T12:36:17.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:17 smithi107 ceph-mon[32659]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:17.358 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi107') 2024-07-26T12:36:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:17 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/685354852' entity='client.admin' 2024-07-26T12:36:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:17 smithi112 ceph-mon[37501]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:17 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/685354852' entity='client.admin' 2024-07-26T12:36:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:17 smithi132 ceph-mon[37580]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.603+0000 7ff625200640 1 -- 172.21.15.107:0/2255574448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff620075b30 msgr2=0x7ff62010f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:17.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.603+0000 7ff625200640 1 --2- 172.21.15.107:0/2255574448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff620075b30 0x7ff62010f400 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009900 tx=0x7ff60c02a410 comp rx=0 tx=0).stop 2024-07-26T12:36:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.604+0000 7ff625200640 1 -- 172.21.15.107:0/2255574448 shutdown_connections 2024-07-26T12:36:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.604+0000 7ff625200640 1 --2- 172.21.15.107:0/2255574448 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff62010f940 0x7ff620111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.604+0000 7ff625200640 1 --2- 172.21.15.107:0/2255574448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff620075b30 0x7ff62010f400 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.604+0000 7ff625200640 1 --2- 172.21.15.107:0/2255574448 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff6200751e0 0x7ff6200755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.604+0000 7ff625200640 1 -- 172.21.15.107:0/2255574448 >> 172.21.15.107:0/2255574448 conn(0x7ff6200fee70 msgr2=0x7ff6201012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.604+0000 7ff625200640 1 -- 172.21.15.107:0/2255574448 shutdown_connections 2024-07-26T12:36:17.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.604+0000 7ff625200640 1 -- 172.21.15.107:0/2255574448 wait complete. 2024-07-26T12:36:17.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.606+0000 7ff625200640 1 Processor -- start 2024-07-26T12:36:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.606+0000 7ff625200640 1 -- start start 2024-07-26T12:36:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.606+0000 7ff625200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6200751e0 0x7ff62019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:17.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.606+0000 7ff625200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff620075b30 0x7ff62019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:17.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.606+0000 7ff625200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff62010f940 0x7ff6201a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:17.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.606+0000 7ff625200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff620115530 con 0x7ff6200751e0 2024-07-26T12:36:17.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.606+0000 7ff625200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff6201153b0 con 0x7ff620075b30 2024-07-26T12:36:17.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.606+0000 7ff625200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff6201156b0 con 0x7ff62010f940 2024-07-26T12:36:17.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.607+0000 7ff61ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff620075b30 0x7ff62019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:17.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.607+0000 7ff61f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6200751e0 0x7ff62019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:17.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.607+0000 7ff61ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff620075b30 0x7ff62019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:34736/0 (socket says 172.21.15.107:34736) 2024-07-26T12:36:17.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.607+0000 7ff61ea00640 1 -- 172.21.15.107:0/4208790929 learned_addr learned my addr 172.21.15.107:0/4208790929 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:17.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.607+0000 7ff61fe00640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff62010f940 0x7ff6201a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:17.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.607+0000 7ff61ea00640 1 -- 172.21.15.107:0/4208790929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff62010f940 msgr2=0x7ff6201a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:17.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.607+0000 7ff61ea00640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff62010f940 0x7ff6201a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:17.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.607+0000 7ff61ea00640 1 -- 172.21.15.107:0/4208790929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6200751e0 msgr2=0x7ff62019bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:17.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.607+0000 7ff61ea00640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6200751e0 0x7ff62019bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:17.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.607+0000 7ff61ea00640 1 -- 172.21.15.107:0/4208790929 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6201a1ed0 con 0x7ff620075b30 2024-07-26T12:36:17.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.608+0000 7ff61ea00640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff620075b30 0x7ff62019c500 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff60c005ab0 tx=0x7ff60c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:17.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.608+0000 7ff617e00640 1 -- 172.21.15.107:0/4208790929 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c03f070 con 0x7ff620075b30 2024-07-26T12:36:17.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.608+0000 7ff617e00640 1 -- 172.21.15.107:0/4208790929 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff60c031b50 con 0x7ff620075b30 2024-07-26T12:36:17.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.608+0000 7ff617e00640 1 -- 172.21.15.107:0/4208790929 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60c035030 con 0x7ff620075b30 2024-07-26T12:36:17.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.608+0000 7ff625200640 1 -- 172.21.15.107:0/4208790929 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6201a2160 con 0x7ff620075b30 2024-07-26T12:36:17.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.609+0000 7ff625200640 1 -- 172.21.15.107:0/4208790929 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff620102f90 con 0x7ff620075b30 2024-07-26T12:36:17.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.610+0000 7ff625200640 1 -- 172.21.15.107:0/4208790929 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6200766f0 con 0x7ff620075b30 2024-07-26T12:36:17.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.613+0000 7ff617e00640 1 -- 172.21.15.107:0/4208790929 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7ff60c042070 con 0x7ff620075b30 2024-07-26T12:36:17.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.614+0000 7ff617e00640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff6040b3c10 0x7ff6040b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:17.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.615+0000 7ff617e00640 1 -- 172.21.15.107:0/4208790929 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff60c102e70 con 0x7ff620075b30 2024-07-26T12:36:17.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.615+0000 7ff61f400640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff6040b3c10 0x7ff6040b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:17.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.616+0000 7ff617e00640 1 -- 172.21.15.107:0/4208790929 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff60c049030 con 0x7ff620075b30 2024-07-26T12:36:17.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.616+0000 7ff61f400640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff6040b3c10 0x7ff6040b6120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff610005d00 tx=0x7ff61000b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:17.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.826+0000 7ff625200640 1 -- 172.21.15.107:0/4208790929 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7ff6200755f0 con 0x7ff620075b30 2024-07-26T12:36:17.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.837+0000 7ff617e00640 1 -- 172.21.15.107:0/4208790929 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v24)=0 v24) ==== 129+0+0 (secure 0 0 0) 0x7ff60c0ece20 con 0x7ff620075b30 2024-07-26T12:36:17.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.840+0000 7ff625200640 1 -- 172.21.15.107:0/4208790929 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff6040b3c10 msgr2=0x7ff6040b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:17.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.840+0000 7ff625200640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff6040b3c10 0x7ff6040b6120 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff610005d00 tx=0x7ff61000b410 comp rx=0 tx=0).stop 2024-07-26T12:36:17.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.840+0000 7ff625200640 1 -- 172.21.15.107:0/4208790929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff620075b30 msgr2=0x7ff62019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:17.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.840+0000 7ff625200640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff620075b30 0x7ff62019c500 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff60c005ab0 tx=0x7ff60c003b90 comp rx=0 tx=0).stop 2024-07-26T12:36:17.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.841+0000 7ff625200640 1 -- 172.21.15.107:0/4208790929 shutdown_connections 2024-07-26T12:36:17.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.841+0000 7ff625200640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff6040b3c10 0x7ff6040b6120 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:17.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.841+0000 7ff625200640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff62010f940 0x7ff6201a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:17.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.841+0000 7ff625200640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff620075b30 0x7ff62019c500 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:17.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.841+0000 7ff625200640 1 --2- 172.21.15.107:0/4208790929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6200751e0 0x7ff62019bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:17.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.841+0000 7ff625200640 1 -- 172.21.15.107:0/4208790929 >> 172.21.15.107:0/4208790929 conn(0x7ff6200fee70 msgr2=0x7ff6201094c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:17.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.841+0000 7ff625200640 1 -- 172.21.15.107:0/4208790929 shutdown_connections 2024-07-26T12:36:17.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:17.841+0000 7ff625200640 1 -- 172.21.15.107:0/4208790929 wait complete. 2024-07-26T12:36:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:18 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:36:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:18 smithi107 ceph-mon[32659]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:18 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:36:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:18 smithi112 ceph-mon[37501]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:18 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:36:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:18 smithi132 ceph-mon[37580]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.430+0000 7fc1e0000640 1 -- 172.21.15.107:0/3220830124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d810ba70 msgr2=0x7fc1d81122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.430+0000 7fc1e0000640 1 --2- 172.21.15.107:0/3220830124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d810ba70 0x7fc1d81122a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0009ca0 tx=0x7fc1d002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:36:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.431+0000 7fc1e0000640 1 -- 172.21.15.107:0/3220830124 shutdown_connections 2024-07-26T12:36:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.431+0000 7fc1e0000640 1 --2- 172.21.15.107:0/3220830124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d810ba70 0x7fc1d81122a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.431+0000 7fc1e0000640 1 --2- 172.21.15.107:0/3220830124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1d810b120 0x7fc1d810b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.431+0000 7fc1e0000640 1 --2- 172.21.15.107:0/3220830124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1d81057e0 0x7fc1d8105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.431+0000 7fc1e0000640 1 -- 172.21.15.107:0/3220830124 >> 172.21.15.107:0/3220830124 conn(0x7fc1d81011f0 msgr2=0x7fc1d8103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.431+0000 7fc1e0000640 1 -- 172.21.15.107:0/3220830124 shutdown_connections 2024-07-26T12:36:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.431+0000 7fc1e0000640 1 -- 172.21.15.107:0/3220830124 wait complete. 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.432+0000 7fc1e0000640 1 Processor -- start 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.432+0000 7fc1e0000640 1 -- start start 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.432+0000 7fc1e0000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1d81057e0 0x7fc1d819c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.432+0000 7fc1e0000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1d810b120 0x7fc1d819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.432+0000 7fc1e0000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d810ba70 0x7fc1d81a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.432+0000 7fc1e0000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d8115790 con 0x7fc1d810ba70 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.432+0000 7fc1e0000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d8115610 con 0x7fc1d810b120 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.432+0000 7fc1e0000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d8115910 con 0x7fc1d81057e0 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1dcc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1d810b120 0x7fc1d819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1de000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d810ba70 0x7fc1d81a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1dcc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1d810b120 0x7fc1d819c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:34768/0 (socket says 172.21.15.107:34768) 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1dcc00640 1 -- 172.21.15.107:0/2840953150 learned_addr learned my addr 172.21.15.107:0/2840953150 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:19.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1dd600640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1d81057e0 0x7fc1d819c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1dcc00640 1 -- 172.21.15.107:0/2840953150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1d81057e0 msgr2=0x7fc1d819c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1dcc00640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1d81057e0 0x7fc1d819c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1dcc00640 1 -- 172.21.15.107:0/2840953150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d810ba70 msgr2=0x7fc1d81a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1dcc00640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d810ba70 0x7fc1d81a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1dcc00640 1 -- 172.21.15.107:0/2840953150 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d81a2110 con 0x7fc1d810b120 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.433+0000 7fc1de000640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d810ba70 0x7fc1d81a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.434+0000 7fc1dcc00640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1d810b120 0x7fc1d819c740 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400e9c0 tx=0x7fc1c400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.434+0000 7fc1d6000640 1 -- 172.21.15.107:0/2840953150 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c400ccc0 con 0x7fc1d810b120 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.434+0000 7fc1e0000640 1 -- 172.21.15.107:0/2840953150 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d81a2400 con 0x7fc1d810b120 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.434+0000 7fc1e0000640 1 -- 172.21.15.107:0/2840953150 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d81a8b60 con 0x7fc1d810b120 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.434+0000 7fc1d6000640 1 -- 172.21.15.107:0/2840953150 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1c40054d0 con 0x7fc1d810b120 2024-07-26T12:36:19.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.434+0000 7fc1d6000640 1 -- 172.21.15.107:0/2840953150 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4010430 con 0x7fc1d810b120 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.435+0000 7fc1e0000640 1 -- 172.21.15.107:0/2840953150 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1b80052c0 con 0x7fc1d810b120 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.438+0000 7fc1d6000640 1 -- 172.21.15.107:0/2840953150 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fc1c40046c0 con 0x7fc1d810b120 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.440+0000 7fc1d6000640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1c00b3c10 0x7fc1c00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.440+0000 7fc1d6000640 1 -- 172.21.15.107:0/2840953150 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fc1c40e2220 con 0x7fc1d810b120 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.440+0000 7fc1dd600640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1c00b3c10 0x7fc1c00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.440+0000 7fc1dd600640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1c00b3c10 0x7fc1c00b6120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009bd0 tx=0x7fc1cc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.443+0000 7fc1d6000640 1 -- 172.21.15.107:0/2840953150 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc1c40ab3d0 con 0x7fc1d810b120 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.656+0000 7fc1e0000640 1 -- 172.21.15.107:0/2840953150 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fc1b80055b0 con 0x7fc1d810b120 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.663+0000 7fc1d6000640 1 -- 172.21.15.107:0/2840953150 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v25)=0 v25) ==== 121+0+0 (secure 0 0 0) 0x7fc1c40aacd0 con 0x7fc1d810b120 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.665+0000 7fc1e0000640 1 -- 172.21.15.107:0/2840953150 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1c00b3c10 msgr2=0x7fc1c00b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.665+0000 7fc1e0000640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1c00b3c10 0x7fc1c00b6120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc1cc009bd0 tx=0x7fc1cc005f90 comp rx=0 tx=0).stop 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.666+0000 7fc1e0000640 1 -- 172.21.15.107:0/2840953150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1d810b120 msgr2=0x7fc1d819c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.666+0000 7fc1e0000640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1d810b120 0x7fc1d819c740 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400e9c0 tx=0x7fc1c400ee80 comp rx=0 tx=0).stop 2024-07-26T12:36:19.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.666+0000 7fc1e0000640 1 -- 172.21.15.107:0/2840953150 shutdown_connections 2024-07-26T12:36:19.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.666+0000 7fc1e0000640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1c00b3c10 0x7fc1c00b6120 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:19.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.666+0000 7fc1e0000640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d810ba70 0x7fc1d81a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:19.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.666+0000 7fc1e0000640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc1d810b120 0x7fc1d819c740 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:19.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.666+0000 7fc1e0000640 1 --2- 172.21.15.107:0/2840953150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1d81057e0 0x7fc1d819c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:19.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.666+0000 7fc1e0000640 1 -- 172.21.15.107:0/2840953150 >> 172.21.15.107:0/2840953150 conn(0x7fc1d81011f0 msgr2=0x7fc1d810a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:19.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.666+0000 7fc1e0000640 1 -- 172.21.15.107:0/2840953150 shutdown_connections 2024-07-26T12:36:19.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:19.666+0000 7fc1e0000640 1 -- 172.21.15.107:0/2840953150 wait complete. 2024-07-26T12:36:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.567+0000 7f5259a00640 1 -- 172.21.15.107:0/627125139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5254104b50 msgr2=0x7f525410d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.567+0000 7f5259a00640 1 --2- 172.21.15.107:0/627125139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5254104b50 0x7f525410d0a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5240009900 tx=0x7f524002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.567+0000 7f5259a00640 1 -- 172.21.15.107:0/627125139 shutdown_connections 2024-07-26T12:36:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.567+0000 7f5259a00640 1 --2- 172.21.15.107:0/627125139 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f525410d660 0x7f525410fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.567+0000 7f5259a00640 1 --2- 172.21.15.107:0/627125139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5254104b50 0x7f525410d0a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.567+0000 7f5259a00640 1 --2- 172.21.15.107:0/627125139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5254104200 0x7f5254104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.567+0000 7f5259a00640 1 -- 172.21.15.107:0/627125139 >> 172.21.15.107:0/627125139 conn(0x7f52540fdd00 msgr2=0x7f5254100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.567+0000 7f5259a00640 1 -- 172.21.15.107:0/627125139 shutdown_connections 2024-07-26T12:36:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.567+0000 7f5259a00640 1 -- 172.21.15.107:0/627125139 wait complete. 2024-07-26T12:36:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.568+0000 7f5259a00640 1 Processor -- start 2024-07-26T12:36:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.568+0000 7f5259a00640 1 -- start start 2024-07-26T12:36:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5259a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5254104200 0x7f5254199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5259a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5254104b50 0x7f525419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5259a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f525410d660 0x7f525419f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5259a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f52541132e0 con 0x7f5254104b50 2024-07-26T12:36:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5259a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5254113160 con 0x7f525410d660 2024-07-26T12:36:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5259a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5254113460 con 0x7f5254104200 2024-07-26T12:36:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5253400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5254104b50 0x7f525419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5253e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5254104200 0x7f5254199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5253400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5254104b50 0x7f525419a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54348/0 (socket says 172.21.15.107:54348) 2024-07-26T12:36:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5253400640 1 -- 172.21.15.107:0/428400126 learned_addr learned my addr 172.21.15.107:0/428400126 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.569+0000 7f5259000640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f525410d660 0x7f525419f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.570+0000 7f5253e00640 1 -- 172.21.15.107:0/428400126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f525410d660 msgr2=0x7f525419f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.570+0000 7f5253e00640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f525410d660 0x7f525419f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.570+0000 7f5253e00640 1 -- 172.21.15.107:0/428400126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5254104b50 msgr2=0x7f525419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:20.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.570+0000 7f5253e00640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5254104b50 0x7f525419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.570+0000 7f5253e00640 1 -- 172.21.15.107:0/428400126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525419fc40 con 0x7f5254104200 2024-07-26T12:36:20.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.570+0000 7f5253e00640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5254104200 0x7f5254199cd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5244005fd0 tx=0x7f524400dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:20.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.571+0000 7f5250c00640 1 -- 172.21.15.107:0/428400126 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5244009c30 con 0x7f5254104200 2024-07-26T12:36:20.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.571+0000 7f5250c00640 1 -- 172.21.15.107:0/428400126 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5244009dd0 con 0x7f5254104200 2024-07-26T12:36:20.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.571+0000 7f5250c00640 1 -- 172.21.15.107:0/428400126 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5244010460 con 0x7f5254104200 2024-07-26T12:36:20.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.571+0000 7f5259a00640 1 -- 172.21.15.107:0/428400126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525419ff30 con 0x7f5254104200 2024-07-26T12:36:20.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.571+0000 7f5259a00640 1 -- 172.21.15.107:0/428400126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52541a6690 con 0x7f5254104200 2024-07-26T12:36:20.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.572+0000 7f5259a00640 1 -- 172.21.15.107:0/428400126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5254104610 con 0x7f5254104200 2024-07-26T12:36:20.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.574+0000 7f5250c00640 1 -- 172.21.15.107:0/428400126 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f5244005030 con 0x7f5254104200 2024-07-26T12:36:20.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.576+0000 7f5250c00640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f52380b3c10 0x7f52380b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:20.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.576+0000 7f5250c00640 1 -- 172.21.15.107:0/428400126 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f52440e1f50 con 0x7f5254104200 2024-07-26T12:36:20.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.576+0000 7f5253400640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f52380b3c10 0x7f52380b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.577+0000 7f5253400640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f52380b3c10 0x7f52380b6120 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5240006010 tx=0x7f5240005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:20.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.578+0000 7f5250c00640 1 -- 172.21.15.107:0/428400126 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5244017020 con 0x7f5254104200 2024-07-26T12:36:20.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.783+0000 7f5259a00640 1 -- 172.21.15.107:0/428400126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f52541a6980 con 0x7f5254104200 2024-07-26T12:36:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:20 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:36:20.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.795+0000 7f5250c00640 1 -- 172.21.15.107:0/428400126 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v26)=0 v26) ==== 121+0+0 (secure 0 0 0) 0x7f52440aab30 con 0x7f5254104200 2024-07-26T12:36:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.798+0000 7f5259a00640 1 -- 172.21.15.107:0/428400126 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f52380b3c10 msgr2=0x7f52380b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.798+0000 7f5259a00640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f52380b3c10 0x7f52380b6120 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5240006010 tx=0x7f5240005a20 comp rx=0 tx=0).stop 2024-07-26T12:36:20.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.798+0000 7f5259a00640 1 -- 172.21.15.107:0/428400126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5254104200 msgr2=0x7f5254199cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.798+0000 7f5259a00640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5254104200 0x7f5254199cd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5244005fd0 tx=0x7f524400dde0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.798+0000 7f5259a00640 1 -- 172.21.15.107:0/428400126 shutdown_connections 2024-07-26T12:36:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.798+0000 7f5259a00640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f52380b3c10 0x7f52380b6120 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.798+0000 7f5259a00640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f525410d660 0x7f525419f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.798+0000 7f5259a00640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5254104b50 0x7f525419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.798+0000 7f5259a00640 1 --2- 172.21.15.107:0/428400126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5254104200 0x7f5254199cd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.798+0000 7f5259a00640 1 -- 172.21.15.107:0/428400126 >> 172.21.15.107:0/428400126 conn(0x7f52540fdd00 msgr2=0x7f5254107160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.799+0000 7f5259a00640 1 -- 172.21.15.107:0/428400126 shutdown_connections 2024-07-26T12:36:20.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:20.799+0000 7f5259a00640 1 -- 172.21.15.107:0/428400126 wait complete. 2024-07-26T12:36:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:20 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:36:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:20 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:36:21.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.023+0000 7f0b9ea00640 1 -- 172.21.15.107:0/701308531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b9810f330 msgr2=0x7f0b9810f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.023+0000 7f0b9ea00640 1 --2- 172.21.15.107:0/701308531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b9810f330 0x7f0b9810f740 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009900 tx=0x7f0b8402a410 comp rx=0 tx=0).stop 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.023+0000 7f0b9ea00640 1 -- 172.21.15.107:0/701308531 shutdown_connections 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.023+0000 7f0b9ea00640 1 --2- 172.21.15.107:0/701308531 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b9810fc80 0x7f0b981164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.023+0000 7f0b9ea00640 1 --2- 172.21.15.107:0/701308531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b9810f330 0x7f0b9810f740 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.023+0000 7f0b9ea00640 1 --2- 172.21.15.107:0/701308531 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b981099f0 0x7f0b98109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.023+0000 7f0b9ea00640 1 -- 172.21.15.107:0/701308531 >> 172.21.15.107:0/701308531 conn(0x7f0b98078ff0 msgr2=0x7f0b98107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.023+0000 7f0b9ea00640 1 -- 172.21.15.107:0/701308531 shutdown_connections 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.023+0000 7f0b9ea00640 1 -- 172.21.15.107:0/701308531 wait complete. 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.025+0000 7f0b9ea00640 1 Processor -- start 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.025+0000 7f0b9ea00640 1 -- start start 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.025+0000 7f0b9ea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b981099f0 0x7f0b981a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.025+0000 7f0b9ea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b9810f330 0x7f0b981a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:21.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.025+0000 7f0b9ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b9810fc80 0x7f0b981a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.025+0000 7f0b9ea00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0b98119940 con 0x7f0b981099f0 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.025+0000 7f0b9ea00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0b981197c0 con 0x7f0b9810fc80 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.025+0000 7f0b9ea00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0b98119ac0 con 0x7f0b9810f330 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9d600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b981099f0 0x7f0b981a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9d600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b981099f0 0x7f0b981a0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54368/0 (socket says 172.21.15.107:54368) 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9d600640 1 -- 172.21.15.107:0/2079466897 learned_addr learned my addr 172.21.15.107:0/2079466897 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9e000640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b9810fc80 0x7f0b981a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9cc00640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b9810f330 0x7f0b981a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9d600640 1 -- 172.21.15.107:0/2079466897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b9810f330 msgr2=0x7f0b981a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9d600640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b9810f330 0x7f0b981a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9d600640 1 -- 172.21.15.107:0/2079466897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b9810fc80 msgr2=0x7f0b981a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9d600640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b9810fc80 0x7f0b981a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:21.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9d600640 1 -- 172.21.15.107:0/2079466897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b981a62a0 con 0x7f0b981099f0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.026+0000 7f0b9e000640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b9810fc80 0x7f0b981a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.027+0000 7f0b9d600640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b981099f0 0x7f0b981a0330 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00d950 tx=0x7f0b8c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.027+0000 7f0b96000640 1 -- 172.21.15.107:0/2079466897 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c017030 con 0x7f0b981099f0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.027+0000 7f0b96000640 1 -- 172.21.15.107:0/2079466897 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b8c004020 con 0x7f0b981099f0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.027+0000 7f0b96000640 1 -- 172.21.15.107:0/2079466897 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c0047a0 con 0x7f0b981099f0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.027+0000 7f0b9ea00640 1 -- 172.21.15.107:0/2079466897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b981a6590 con 0x7f0b981099f0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.027+0000 7f0b9ea00640 1 -- 172.21.15.107:0/2079466897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b981accf0 con 0x7f0b981099f0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.028+0000 7f0b9ea00640 1 -- 172.21.15.107:0/2079466897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b98109e00 con 0x7f0b981099f0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.031+0000 7f0b96000640 1 -- 172.21.15.107:0/2079466897 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f0b8c009a30 con 0x7f0b981099f0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.032+0000 7f0b96000640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0b800b3b90 0x7f0b800b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.033+0000 7f0b96000640 1 -- 172.21.15.107:0/2079466897 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f0b8c0e1cf0 con 0x7f0b981099f0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.033+0000 7f0b9cc00640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0b800b3b90 0x7f0b800b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.033+0000 7f0b9cc00640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0b800b3b90 0x7f0b800b60a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009230 tx=0x7f0b84031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:21.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.034+0000 7f0b96000640 1 -- 172.21.15.107:0/2079466897 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0b8c0aaf40 con 0x7f0b981099f0 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.233+0000 7f0b9ea00640 1 -- 172.21.15.107:0/2079466897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7f0b9802e0c0 con 0x7f0b981099f0 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.239+0000 7f0b96000640 1 -- 172.21.15.107:0/2079466897 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v27)=0 v27) ==== 118+0+0 (secure 0 0 0) 0x7f0b8c0aa840 con 0x7f0b981099f0 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.242+0000 7f0b9ea00640 1 -- 172.21.15.107:0/2079466897 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0b800b3b90 msgr2=0x7f0b800b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.242+0000 7f0b9ea00640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0b800b3b90 0x7f0b800b60a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0b84009230 tx=0x7f0b84031400 comp rx=0 tx=0).stop 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.242+0000 7f0b9ea00640 1 -- 172.21.15.107:0/2079466897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b981099f0 msgr2=0x7f0b981a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.242+0000 7f0b9ea00640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b981099f0 0x7f0b981a0330 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00d950 tx=0x7f0b8c00de10 comp rx=0 tx=0).stop 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.242+0000 7f0b9ea00640 1 -- 172.21.15.107:0/2079466897 shutdown_connections 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.242+0000 7f0b9ea00640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0b800b3b90 0x7f0b800b60a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.242+0000 7f0b9ea00640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0b9810fc80 0x7f0b981a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.242+0000 7f0b9ea00640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b9810f330 0x7f0b981a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.242+0000 7f0b9ea00640 1 --2- 172.21.15.107:0/2079466897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b981099f0 0x7f0b981a0330 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.242+0000 7f0b9ea00640 1 -- 172.21.15.107:0/2079466897 >> 172.21.15.107:0/2079466897 conn(0x7f0b98078ff0 msgr2=0x7f0b98107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:21.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.243+0000 7f0b9ea00640 1 -- 172.21.15.107:0/2079466897 shutdown_connections 2024-07-26T12:36:21.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:21.243+0000 7f0b9ea00640 1 -- 172.21.15.107:0/2079466897 wait complete. 2024-07-26T12:36:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:21 smithi112 ceph-mon[37501]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:21 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:36:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:21 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2079466897' entity='client.admin' 2024-07-26T12:36:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:21 smithi107 ceph-mon[32659]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:21 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:36:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:21 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2079466897' entity='client.admin' 2024-07-26T12:36:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:21 smithi132 ceph-mon[37580]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:21 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:36:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:21 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2079466897' entity='client.admin' 2024-07-26T12:36:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:23 smithi112 ceph-mon[37501]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:23 smithi107 ceph-mon[32659]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:23 smithi132 ceph-mon[37580]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:24.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.734+0000 7f4cc4e00640 1 -- 172.21.15.107:0/2106912857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cc01033c0 msgr2=0x7f4cc01037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:24.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.734+0000 7f4cc4e00640 1 --2- 172.21.15.107:0/2106912857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cc01033c0 0x7f4cc01037d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009900 tx=0x7f4cb002a410 comp rx=0 tx=0).stop 2024-07-26T12:36:24.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.734+0000 7f4cc4e00640 1 -- 172.21.15.107:0/2106912857 shutdown_connections 2024-07-26T12:36:24.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.734+0000 7f4cc4e00640 1 --2- 172.21.15.107:0/2106912857 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cc0109650 0x7f4cc010fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:24.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.734+0000 7f4cc4e00640 1 --2- 172.21.15.107:0/2106912857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cc0108d00 0x7f4cc0109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:24.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.734+0000 7f4cc4e00640 1 --2- 172.21.15.107:0/2106912857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cc01033c0 0x7f4cc01037d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:24.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.734+0000 7f4cc4e00640 1 -- 172.21.15.107:0/2106912857 >> 172.21.15.107:0/2106912857 conn(0x7f4cc00fee10 msgr2=0x7f4cc0101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:24.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.734+0000 7f4cc4e00640 1 -- 172.21.15.107:0/2106912857 shutdown_connections 2024-07-26T12:36:24.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.734+0000 7f4cc4e00640 1 -- 172.21.15.107:0/2106912857 wait complete. 2024-07-26T12:36:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.736+0000 7f4cc4e00640 1 Processor -- start 2024-07-26T12:36:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.736+0000 7f4cc4e00640 1 -- start start 2024-07-26T12:36:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.736+0000 7f4cc4e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cc01033c0 0x7f4cc0199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.736+0000 7f4cc4e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cc0108d00 0x7f4cc019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.736+0000 7f4cc4e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cc0109650 0x7f4cc019f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.736+0000 7f4cc4e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc0113370 con 0x7f4cc01033c0 2024-07-26T12:36:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.736+0000 7f4cc4e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc01131f0 con 0x7f4cc0108d00 2024-07-26T12:36:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.736+0000 7f4cc4e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc01134f0 con 0x7f4cc0109650 2024-07-26T12:36:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.737+0000 7f4cbf400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cc01033c0 0x7f4cc0199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.737+0000 7f4cb6a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cc0108d00 0x7f4cc019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:24.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.737+0000 7f4cbf400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cc01033c0 0x7f4cc0199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54914/0 (socket says 172.21.15.107:54914) 2024-07-26T12:36:24.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.737+0000 7f4cbf400640 1 -- 172.21.15.107:0/3520009872 learned_addr learned my addr 172.21.15.107:0/3520009872 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:24.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.737+0000 7f4cbfe00640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cc0109650 0x7f4cc019f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.737+0000 7f4cbf400640 1 -- 172.21.15.107:0/3520009872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cc0109650 msgr2=0x7f4cc019f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.737+0000 7f4cbf400640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cc0109650 0x7f4cc019f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.737+0000 7f4cbf400640 1 -- 172.21.15.107:0/3520009872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cc0108d00 msgr2=0x7f4cc019a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.737+0000 7f4cbf400640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cc0108d00 0x7f4cc019a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.737+0000 7f4cbf400640 1 -- 172.21.15.107:0/3520009872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc019fcf0 con 0x7f4cc01033c0 2024-07-26T12:36:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.738+0000 7f4cbf400640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cc01033c0 0x7f4cc0199d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0005ab0 tx=0x7f4cb0003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.738+0000 7f4cbcc00640 1 -- 172.21.15.107:0/3520009872 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb003f070 con 0x7f4cc01033c0 2024-07-26T12:36:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.738+0000 7f4cbcc00640 1 -- 172.21.15.107:0/3520009872 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cb0031d60 con 0x7f4cc01033c0 2024-07-26T12:36:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.738+0000 7f4cbcc00640 1 -- 172.21.15.107:0/3520009872 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0035030 con 0x7f4cc01033c0 2024-07-26T12:36:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.738+0000 7f4cc4e00640 1 -- 172.21.15.107:0/3520009872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc019ff80 con 0x7f4cc01033c0 2024-07-26T12:36:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.738+0000 7f4cc4e00640 1 -- 172.21.15.107:0/3520009872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc01a6740 con 0x7f4cc01033c0 2024-07-26T12:36:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.740+0000 7f4cc4e00640 1 -- 172.21.15.107:0/3520009872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cc01037d0 con 0x7f4cc01033c0 2024-07-26T12:36:24.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.744+0000 7f4cbcc00640 1 -- 172.21.15.107:0/3520009872 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f4cb0042070 con 0x7f4cc01033c0 2024-07-26T12:36:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.745+0000 7f4cbcc00640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4ca40b3c10 0x7f4ca40b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.745+0000 7f4cbcc00640 1 -- 172.21.15.107:0/3520009872 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f4cb0102e70 con 0x7f4cc01033c0 2024-07-26T12:36:24.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.745+0000 7f4cb6a00640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4ca40b3c10 0x7f4ca40b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.746+0000 7f4cb6a00640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4ca40b3c10 0x7f4ca40b6120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0108760 tx=0x7f4ca8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:24.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.747+0000 7f4cbcc00640 1 -- 172.21.15.107:0/3520009872 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4cb00cb9d0 con 0x7f4cc01033c0 2024-07-26T12:36:24.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.962+0000 7f4cc4e00640 1 -- 172.21.15.107:0/3520009872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f4cc002e0c0 con 0x7f4cc01033c0 2024-07-26T12:36:24.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.972+0000 7f4cbcc00640 1 -- 172.21.15.107:0/3520009872 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v28)=0 v28) ==== 127+0+0 (secure 0 0 0) 0x7f4cb00ece20 con 0x7f4cc01033c0 2024-07-26T12:36:24.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.974+0000 7f4cc4e00640 1 -- 172.21.15.107:0/3520009872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4ca40b3c10 msgr2=0x7f4ca40b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:24.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.974+0000 7f4cc4e00640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4ca40b3c10 0x7f4ca40b6120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0108760 tx=0x7f4ca8005f90 comp rx=0 tx=0).stop 2024-07-26T12:36:24.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.974+0000 7f4cc4e00640 1 -- 172.21.15.107:0/3520009872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cc01033c0 msgr2=0x7f4cc0199d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:24.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.975+0000 7f4cc4e00640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cc01033c0 0x7f4cc0199d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0005ab0 tx=0x7f4cb0003c80 comp rx=0 tx=0).stop 2024-07-26T12:36:24.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.975+0000 7f4cc4e00640 1 -- 172.21.15.107:0/3520009872 shutdown_connections 2024-07-26T12:36:24.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.975+0000 7f4cc4e00640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4ca40b3c10 0x7f4ca40b6120 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:24.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.975+0000 7f4cc4e00640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4cc0109650 0x7f4cc019f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:24.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.975+0000 7f4cc4e00640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4cc0108d00 0x7f4cc019a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:24.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.975+0000 7f4cc4e00640 1 --2- 172.21.15.107:0/3520009872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cc01033c0 0x7f4cc0199d80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:24.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.975+0000 7f4cc4e00640 1 -- 172.21.15.107:0/3520009872 >> 172.21.15.107:0/3520009872 conn(0x7f4cc00fee10 msgr2=0x7f4cc0101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:24.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.975+0000 7f4cc4e00640 1 -- 172.21.15.107:0/3520009872 shutdown_connections 2024-07-26T12:36:24.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:24.975+0000 7f4cc4e00640 1 -- 172.21.15.107:0/3520009872 wait complete. 2024-07-26T12:36:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:25 smithi107 ceph-mon[32659]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:25 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3520009872' entity='client.admin' 2024-07-26T12:36:25.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.802+0000 7f7c2f200640 1 -- 172.21.15.107:0/1444883806 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c300751e0 msgr2=0x7f7c300755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:25.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.802+0000 7f7c2f200640 1 --2- 172.21.15.107:0/1444883806 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c300751e0 0x7f7c300755f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7c20009860 tx=0x7f7c2002a340 comp rx=0 tx=0).stop 2024-07-26T12:36:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.802+0000 7f7c2f200640 1 -- 172.21.15.107:0/1444883806 shutdown_connections 2024-07-26T12:36:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.802+0000 7f7c2f200640 1 --2- 172.21.15.107:0/1444883806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c3010f940 0x7f7c30111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.802+0000 7f7c2f200640 1 --2- 172.21.15.107:0/1444883806 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c30075b30 0x7f7c3010f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.802+0000 7f7c2f200640 1 --2- 172.21.15.107:0/1444883806 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c300751e0 0x7f7c300755f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.802+0000 7f7c2f200640 1 -- 172.21.15.107:0/1444883806 >> 172.21.15.107:0/1444883806 conn(0x7f7c300fee70 msgr2=0x7f7c301012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.802+0000 7f7c2f200640 1 -- 172.21.15.107:0/1444883806 shutdown_connections 2024-07-26T12:36:25.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.802+0000 7f7c2f200640 1 -- 172.21.15.107:0/1444883806 wait complete. 2024-07-26T12:36:25.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.803+0000 7f7c2f200640 1 Processor -- start 2024-07-26T12:36:25.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.803+0000 7f7c2f200640 1 -- start start 2024-07-26T12:36:25.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2f200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c300751e0 0x7f7c3019bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:25.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2f200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c30075b30 0x7f7c3019c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:25.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2f200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c3010f940 0x7f7c301a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2f200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7c301154f0 con 0x7f7c3010f940 2024-07-26T12:36:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2f200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7c30115370 con 0x7f7c30075b30 2024-07-26T12:36:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2f200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7c30115670 con 0x7f7c300751e0 2024-07-26T12:36:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2e800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c3010f940 0x7f7c301a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2d400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c30075b30 0x7f7c3019c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2de00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c300751e0 0x7f7c3019bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2e800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c3010f940 0x7f7c301a17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54928/0 (socket says 172.21.15.107:54928) 2024-07-26T12:36:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.804+0000 7f7c2e800640 1 -- 172.21.15.107:0/173412341 learned_addr learned my addr 172.21.15.107:0/173412341 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c2e800640 1 -- 172.21.15.107:0/173412341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c300751e0 msgr2=0x7f7c3019bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:25.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c2e800640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c300751e0 0x7f7c3019bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c2e800640 1 -- 172.21.15.107:0/173412341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c30075b30 msgr2=0x7f7c3019c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c2e800640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c30075b30 0x7f7c3019c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c2e800640 1 -- 172.21.15.107:0/173412341 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c301a1ef0 con 0x7f7c3010f940 2024-07-26T12:36:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c2de00640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c300751e0 0x7f7c3019bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c2d400640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c30075b30 0x7f7c3019c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c2e800640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c3010f940 0x7f7c301a17c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800eb10 tx=0x7f7c2800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c26a00640 1 -- 172.21.15.107:0/173412341 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28004b20 con 0x7f7c3010f940 2024-07-26T12:36:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c2f200640 1 -- 172.21.15.107:0/173412341 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c301a21e0 con 0x7f7c3010f940 2024-07-26T12:36:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c26a00640 1 -- 172.21.15.107:0/173412341 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c28004cc0 con 0x7f7c3010f940 2024-07-26T12:36:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.805+0000 7f7c26a00640 1 -- 172.21.15.107:0/173412341 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28010670 con 0x7f7c3010f940 2024-07-26T12:36:25.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.806+0000 7f7c2f200640 1 -- 172.21.15.107:0/173412341 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c301a8940 con 0x7f7c3010f940 2024-07-26T12:36:25.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.807+0000 7f7c2f200640 1 -- 172.21.15.107:0/173412341 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c300755f0 con 0x7f7c3010f940 2024-07-26T12:36:25.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.809+0000 7f7c26a00640 1 -- 172.21.15.107:0/173412341 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f7c28005020 con 0x7f7c3010f940 2024-07-26T12:36:25.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.810+0000 7f7c26a00640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7c140b3b90 0x7f7c140b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.811+0000 7f7c26a00640 1 -- 172.21.15.107:0/173412341 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f7c280e2570 con 0x7f7c3010f940 2024-07-26T12:36:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.811+0000 7f7c2de00640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7c140b3b90 0x7f7c140b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:25.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.812+0000 7f7c2de00640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7c140b3b90 0x7f7c140b60a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7c2002a930 tx=0x7f7c200058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:25.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:25.814+0000 7f7c26a00640 1 -- 172.21.15.107:0/173412341 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7c280ab720 con 0x7f7c3010f940 2024-07-26T12:36:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:25 smithi112 ceph-mon[37501]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:25 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3520009872' entity='client.admin' 2024-07-26T12:36:26.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.017+0000 7f7c2f200640 1 -- 172.21.15.107:0/173412341 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f7c301a8cd0 con 0x7f7c3010f940 2024-07-26T12:36:26.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.025+0000 7f7c26a00640 1 -- 172.21.15.107:0/173412341 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v29)=0 v29) ==== 121+0+0 (secure 0 0 0) 0x7f7c280ab020 con 0x7f7c3010f940 2024-07-26T12:36:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 -- 172.21.15.107:0/173412341 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7c140b3b90 msgr2=0x7f7c140b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7c140b3b90 0x7f7c140b60a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7c2002a930 tx=0x7f7c200058d0 comp rx=0 tx=0).stop 2024-07-26T12:36:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 -- 172.21.15.107:0/173412341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c3010f940 msgr2=0x7f7c301a17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c3010f940 0x7f7c301a17c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800eb10 tx=0x7f7c2800c2d0 comp rx=0 tx=0).stop 2024-07-26T12:36:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 -- 172.21.15.107:0/173412341 shutdown_connections 2024-07-26T12:36:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7c140b3b90 0x7f7c140b60a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:26.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c3010f940 0x7f7c301a17c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:26.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7c30075b30 0x7f7c3019c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:26.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 --2- 172.21.15.107:0/173412341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7c300751e0 0x7f7c3019bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:26.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 -- 172.21.15.107:0/173412341 >> 172.21.15.107:0/173412341 conn(0x7f7c300fee70 msgr2=0x7f7c30100a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:26.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.028+0000 7f7c2f200640 1 -- 172.21.15.107:0/173412341 shutdown_connections 2024-07-26T12:36:26.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:26.029+0000 7f7c2f200640 1 -- 172.21.15.107:0/173412341 wait complete. 2024-07-26T12:36:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:25 smithi132 ceph-mon[37580]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:25 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3520009872' entity='client.admin' 2024-07-26T12:36:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:27 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/173412341' entity='client.admin' 2024-07-26T12:36:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:27 smithi107 ceph-mon[32659]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:27 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/173412341' entity='client.admin' 2024-07-26T12:36:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:27 smithi132 ceph-mon[37580]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:27 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/173412341' entity='client.admin' 2024-07-26T12:36:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:27 smithi112 ceph-mon[37501]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:27.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.702+0000 7f3515800640 1 -- 172.21.15.107:0/1764856821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35101099f0 msgr2=0x7f3510109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:27.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.702+0000 7f3515800640 1 --2- 172.21.15.107:0/1764856821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35101099f0 0x7f3510109e00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3500009900 tx=0x7f350002a410 comp rx=0 tx=0).stop 2024-07-26T12:36:27.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.702+0000 7f3515800640 1 -- 172.21.15.107:0/1764856821 shutdown_connections 2024-07-26T12:36:27.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.702+0000 7f3515800640 1 --2- 172.21.15.107:0/1764856821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f351010fc80 0x7f35101164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:27.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.702+0000 7f3515800640 1 --2- 172.21.15.107:0/1764856821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351010f330 0x7f351010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:27.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.702+0000 7f3515800640 1 --2- 172.21.15.107:0/1764856821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35101099f0 0x7f3510109e00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:27.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.702+0000 7f3515800640 1 -- 172.21.15.107:0/1764856821 >> 172.21.15.107:0/1764856821 conn(0x7f3510078ff0 msgr2=0x7f3510107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:27.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.702+0000 7f3515800640 1 -- 172.21.15.107:0/1764856821 shutdown_connections 2024-07-26T12:36:27.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.702+0000 7f3515800640 1 -- 172.21.15.107:0/1764856821 wait complete. 2024-07-26T12:36:27.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.703+0000 7f3515800640 1 Processor -- start 2024-07-26T12:36:27.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f3515800640 1 -- start start 2024-07-26T12:36:27.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f3515800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35101099f0 0x7f35101a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:27.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f3515800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351010f330 0x7f35101a0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:27.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f3515800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351010fc80 0x7f35101a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:27.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f3515800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3510119940 con 0x7f351010f330 2024-07-26T12:36:27.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f3515800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35101197c0 con 0x7f351010fc80 2024-07-26T12:36:27.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f3515800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3510119ac0 con 0x7f35101099f0 2024-07-26T12:36:27.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f350f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351010f330 0x7f35101a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:27.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f350fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35101099f0 0x7f35101a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:27.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f350f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351010f330 0x7f35101a0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54948/0 (socket says 172.21.15.107:54948) 2024-07-26T12:36:27.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.704+0000 7f350f400640 1 -- 172.21.15.107:0/2907459921 learned_addr learned my addr 172.21.15.107:0/2907459921 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:27.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.705+0000 7f3514e00640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351010fc80 0x7f35101a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:27.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.705+0000 7f350fe00640 1 -- 172.21.15.107:0/2907459921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351010fc80 msgr2=0x7f35101a5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:27.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.705+0000 7f350fe00640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351010fc80 0x7f35101a5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:27.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.705+0000 7f350fe00640 1 -- 172.21.15.107:0/2907459921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351010f330 msgr2=0x7f35101a0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:27.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.705+0000 7f350fe00640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351010f330 0x7f35101a0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:27.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.705+0000 7f350fe00640 1 -- 172.21.15.107:0/2907459921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35101a6360 con 0x7f35101099f0 2024-07-26T12:36:27.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.705+0000 7f350f400640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351010f330 0x7f35101a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:27.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.705+0000 7f350fe00640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35101099f0 0x7f35101a03f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f350002a930 tx=0x7f3500031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:27.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.706+0000 7f350cc00640 1 -- 172.21.15.107:0/2907459921 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350003f030 con 0x7f35101099f0 2024-07-26T12:36:27.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.706+0000 7f350cc00640 1 -- 172.21.15.107:0/2907459921 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3500004490 con 0x7f35101099f0 2024-07-26T12:36:27.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.706+0000 7f350cc00640 1 -- 172.21.15.107:0/2907459921 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3500035070 con 0x7f35101099f0 2024-07-26T12:36:27.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.706+0000 7f3515800640 1 -- 172.21.15.107:0/2907459921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35101a65f0 con 0x7f35101099f0 2024-07-26T12:36:27.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.706+0000 7f3515800640 1 -- 172.21.15.107:0/2907459921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35101acdb0 con 0x7f35101099f0 2024-07-26T12:36:27.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.707+0000 7f3515800640 1 -- 172.21.15.107:0/2907459921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34f00052c0 con 0x7f35101099f0 2024-07-26T12:36:27.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.710+0000 7f350cc00640 1 -- 172.21.15.107:0/2907459921 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f3500042030 con 0x7f35101099f0 2024-07-26T12:36:27.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.711+0000 7f350cc00640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34f40b3c10 0x7f34f40b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:27.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.711+0000 7f350cc00640 1 -- 172.21.15.107:0/2907459921 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f3500102340 con 0x7f35101099f0 2024-07-26T12:36:27.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.711+0000 7f350f400640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34f40b3c10 0x7f34f40b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:27.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.712+0000 7f350f400640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34f40b3c10 0x7f34f40b6120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f351010fad0 tx=0x7f34fc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:27.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.713+0000 7f350cc00640 1 -- 172.21.15.107:0/2907459921 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f350003b460 con 0x7f35101099f0 2024-07-26T12:36:27.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.924+0000 7f3515800640 1 -- 172.21.15.107:0/2907459921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f34f00055b0 con 0x7f35101099f0 2024-07-26T12:36:27.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.931+0000 7f350cc00640 1 -- 172.21.15.107:0/2907459921 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) ==== 138+0+0 (secure 0 0 0) 0x7f35000caea0 con 0x7f35101099f0 2024-07-26T12:36:27.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 -- 172.21.15.107:0/2907459921 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34f40b3c10 msgr2=0x7f34f40b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:27.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34f40b3c10 0x7f34f40b6120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f351010fad0 tx=0x7f34fc005f90 comp rx=0 tx=0).stop 2024-07-26T12:36:27.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 -- 172.21.15.107:0/2907459921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35101099f0 msgr2=0x7f35101a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35101099f0 0x7f35101a03f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f350002a930 tx=0x7f3500031a20 comp rx=0 tx=0).stop 2024-07-26T12:36:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 -- 172.21.15.107:0/2907459921 shutdown_connections 2024-07-26T12:36:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34f40b3c10 0x7f34f40b6120 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f351010fc80 0x7f35101a5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f351010f330 0x7f35101a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 --2- 172.21.15.107:0/2907459921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35101099f0 0x7f35101a03f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 -- 172.21.15.107:0/2907459921 >> 172.21.15.107:0/2907459921 conn(0x7f3510078ff0 msgr2=0x7f3510107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 -- 172.21.15.107:0/2907459921 shutdown_connections 2024-07-26T12:36:27.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:27.935+0000 7f3515800640 1 -- 172.21.15.107:0/2907459921 wait complete. 2024-07-26T12:36:28.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.325+0000 7f2492800640 1 -- 172.21.15.107:0/3819518386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f248c109650 msgr2=0x7f248c10fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.325+0000 7f2492800640 1 --2- 172.21.15.107:0/3819518386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f248c109650 0x7f248c10fe80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2484009ca0 tx=0x7f248402a8c0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.325+0000 7f2492800640 1 -- 172.21.15.107:0/3819518386 shutdown_connections 2024-07-26T12:36:28.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.325+0000 7f2492800640 1 --2- 172.21.15.107:0/3819518386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f248c109650 0x7f248c10fe80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.325+0000 7f2492800640 1 --2- 172.21.15.107:0/3819518386 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f248c108d00 0x7f248c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.325+0000 7f2492800640 1 --2- 172.21.15.107:0/3819518386 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f248c1033c0 0x7f248c1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.325+0000 7f2492800640 1 -- 172.21.15.107:0/3819518386 >> 172.21.15.107:0/3819518386 conn(0x7f248c0fee10 msgr2=0x7f248c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:28.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.325+0000 7f2492800640 1 -- 172.21.15.107:0/3819518386 shutdown_connections 2024-07-26T12:36:28.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.325+0000 7f2492800640 1 -- 172.21.15.107:0/3819518386 wait complete. 2024-07-26T12:36:28.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.327+0000 7f2492800640 1 Processor -- start 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.327+0000 7f2492800640 1 -- start start 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.327+0000 7f2492800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f248c1033c0 0x7f248c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.327+0000 7f2492800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f248c108d00 0x7f248c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.327+0000 7f2492800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f248c109650 0x7f248c19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.327+0000 7f2492800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f248c113350 con 0x7f248c108d00 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.327+0000 7f2492800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1131d0 con 0x7f248c1033c0 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.327+0000 7f2492800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1134d0 con 0x7f248c109650 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2490a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f248c108d00 0x7f248c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2491400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f248c1033c0 0x7f248c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2490a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f248c108d00 0x7f248c19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54956/0 (socket says 172.21.15.107:54956) 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2490a00640 1 -- 172.21.15.107:0/3204194864 learned_addr learned my addr 172.21.15.107:0/3204194864 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:28.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2491e00640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f248c109650 0x7f248c19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2490a00640 1 -- 172.21.15.107:0/3204194864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f248c109650 msgr2=0x7f248c19f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2490a00640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f248c109650 0x7f248c19f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2490a00640 1 -- 172.21.15.107:0/3204194864 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f248c1033c0 msgr2=0x7f248c199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2490a00640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f248c1033c0 0x7f248c199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2490a00640 1 -- 172.21.15.107:0/3204194864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248c19fcb0 con 0x7f248c108d00 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2491400640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f248c1033c0 0x7f248c199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.328+0000 7f2491e00640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f248c109650 0x7f248c19f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.329+0000 7f2490a00640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f248c108d00 0x7f248c19a2e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f247800e9f0 tx=0x7f247800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.329+0000 7f248a000640 1 -- 172.21.15.107:0/3204194864 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247800ccc0 con 0x7f248c108d00 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.329+0000 7f248a000640 1 -- 172.21.15.107:0/3204194864 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24780108e0 con 0x7f248c108d00 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.329+0000 7f2492800640 1 -- 172.21.15.107:0/3204194864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f248c19ffa0 con 0x7f248c108d00 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.329+0000 7f248a000640 1 -- 172.21.15.107:0/3204194864 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478018860 con 0x7f248c108d00 2024-07-26T12:36:28.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.329+0000 7f2492800640 1 -- 172.21.15.107:0/3204194864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f248c1a6700 con 0x7f248c108d00 2024-07-26T12:36:28.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.330+0000 7f2492800640 1 -- 172.21.15.107:0/3204194864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f248c1037d0 con 0x7f248c108d00 2024-07-26T12:36:28.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.335+0000 7f248a000640 1 -- 172.21.15.107:0/3204194864 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f2478010430 con 0x7f248c108d00 2024-07-26T12:36:28.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.336+0000 7f248a000640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24740b3c10 0x7f24740b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.337+0000 7f248a000640 1 -- 172.21.15.107:0/3204194864 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f24780e27e0 con 0x7f248c108d00 2024-07-26T12:36:28.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.337+0000 7f2491400640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24740b3c10 0x7f24740b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.338+0000 7f2491400640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24740b3c10 0x7f24740b6120 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2480009fc0 tx=0x7f2480005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:28.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.339+0000 7f248a000640 1 -- 172.21.15.107:0/3204194864 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f24780ab340 con 0x7f248c108d00 2024-07-26T12:36:28.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.545+0000 7f2492800640 1 -- 172.21.15.107:0/3204194864 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7f248c19ae70 con 0x7f248c108d00 2024-07-26T12:36:28.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.552+0000 7f248a000640 1 -- 172.21.15.107:0/3204194864 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v31)=0 v31) ==== 116+0+0 (secure 0 0 0) 0x7f248c19ae70 con 0x7f248c108d00 2024-07-26T12:36:28.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 -- 172.21.15.107:0/3204194864 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24740b3c10 msgr2=0x7f24740b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24740b3c10 0x7f24740b6120 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2480009fc0 tx=0x7f2480005f90 comp rx=0 tx=0).stop 2024-07-26T12:36:28.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 -- 172.21.15.107:0/3204194864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f248c108d00 msgr2=0x7f248c19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f248c108d00 0x7f248c19a2e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f247800e9f0 tx=0x7f247800eeb0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 -- 172.21.15.107:0/3204194864 shutdown_connections 2024-07-26T12:36:28.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24740b3c10 0x7f24740b6120 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f248c109650 0x7f248c19f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f248c108d00 0x7f248c19a2e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 --2- 172.21.15.107:0/3204194864 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f248c1033c0 0x7f248c199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 -- 172.21.15.107:0/3204194864 >> 172.21.15.107:0/3204194864 conn(0x7f248c0fee10 msgr2=0x7f248c107320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:28.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 -- 172.21.15.107:0/3204194864 shutdown_connections 2024-07-26T12:36:28.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.555+0000 7f2492800640 1 -- 172.21.15.107:0/3204194864 wait complete. 2024-07-26T12:36:28.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.744+0000 7f41a6200640 1 -- 172.21.15.107:0/1117753320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41a0075ad0 msgr2=0x7f41a010f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.744+0000 7f41a6200640 1 --2- 172.21.15.107:0/1117753320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41a0075ad0 0x7f41a010f610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f418c009860 tx=0x7f418c02a340 comp rx=0 tx=0).stop 2024-07-26T12:36:28.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.744+0000 7f41a6200640 1 -- 172.21.15.107:0/1117753320 shutdown_connections 2024-07-26T12:36:28.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.744+0000 7f41a6200640 1 --2- 172.21.15.107:0/1117753320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41a010fb50 0x7f41a0111f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.744+0000 7f41a6200640 1 --2- 172.21.15.107:0/1117753320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41a0075ad0 0x7f41a010f610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.744+0000 7f41a6200640 1 --2- 172.21.15.107:0/1117753320 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41a0075180 0x7f41a0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.744+0000 7f41a6200640 1 -- 172.21.15.107:0/1117753320 >> 172.21.15.107:0/1117753320 conn(0x7f41a007b280 msgr2=0x7f41a007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:28.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.744+0000 7f41a6200640 1 -- 172.21.15.107:0/1117753320 shutdown_connections 2024-07-26T12:36:28.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.745+0000 7f41a6200640 1 -- 172.21.15.107:0/1117753320 wait complete. 2024-07-26T12:36:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.746+0000 7f41a6200640 1 Processor -- start 2024-07-26T12:36:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.746+0000 7f41a6200640 1 -- start start 2024-07-26T12:36:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.746+0000 7f41a6200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41a0075180 0x7f41a019c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.746+0000 7f41a6200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41a0075ad0 0x7f41a019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f41a6200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41a010fb50 0x7f41a01a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f41a6200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f41a0115670 con 0x7f41a0075180 2024-07-26T12:36:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f41a6200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f41a01154f0 con 0x7f41a0075ad0 2024-07-26T12:36:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f41a6200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f41a01157f0 con 0x7f41a010fb50 2024-07-26T12:36:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f419f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41a0075180 0x7f41a019c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f419f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41a0075180 0x7f41a019c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:54972/0 (socket says 172.21.15.107:54972) 2024-07-26T12:36:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f419f400640 1 -- 172.21.15.107:0/165373390 learned_addr learned my addr 172.21.15.107:0/165373390 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f419ea00640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41a0075ad0 0x7f41a019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f419fe00640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41a010fb50 0x7f41a01a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f419ea00640 1 -- 172.21.15.107:0/165373390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41a010fb50 msgr2=0x7f41a01a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f419ea00640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41a010fb50 0x7f41a01a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f419ea00640 1 -- 172.21.15.107:0/165373390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41a0075180 msgr2=0x7f41a019c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.747+0000 7f419ea00640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41a0075180 0x7f41a019c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.748+0000 7f419ea00640 1 -- 172.21.15.107:0/165373390 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a01a20f0 con 0x7f41a0075ad0 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.748+0000 7f419f400640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41a0075180 0x7f41a019c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.748+0000 7f419ea00640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41a0075ad0 0x7f41a019c720 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f418c009780 tx=0x7f418c032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.748+0000 7f4197e00640 1 -- 172.21.15.107:0/165373390 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c037040 con 0x7f41a0075ad0 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.748+0000 7f4197e00640 1 -- 172.21.15.107:0/165373390 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f418c030c10 con 0x7f41a0075ad0 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.748+0000 7f4197e00640 1 -- 172.21.15.107:0/165373390 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c0035f0 con 0x7f41a0075ad0 2024-07-26T12:36:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.748+0000 7f41a6200640 1 -- 172.21.15.107:0/165373390 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a01a2380 con 0x7f41a0075ad0 2024-07-26T12:36:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.748+0000 7f41a6200640 1 -- 172.21.15.107:0/165373390 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a01a27d0 con 0x7f41a0075ad0 2024-07-26T12:36:28.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.750+0000 7f41a6200640 1 -- 172.21.15.107:0/165373390 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41a0075590 con 0x7f41a0075ad0 2024-07-26T12:36:28.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.752+0000 7f4197e00640 1 -- 172.21.15.107:0/165373390 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f418c043020 con 0x7f41a0075ad0 2024-07-26T12:36:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.753+0000 7f4197e00640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f41840b3b90 0x7f41840b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.754+0000 7f4197e00640 1 -- 172.21.15.107:0/165373390 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f418c101100 con 0x7f41a0075ad0 2024-07-26T12:36:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.754+0000 7f419f400640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f41840b3b90 0x7f41840b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.754+0000 7f419f400640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f41840b3b90 0x7f41840b60a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4190011550 tx=0x7f4190005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:28.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.758+0000 7f4197e00640 1 -- 172.21.15.107:0/165373390 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f418c0ca2b0 con 0x7f41a0075ad0 2024-07-26T12:36:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.928+0000 7f4482a00640 1 -- 172.21.15.107:0/2308839991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f447c1033e0 msgr2=0x7f447c1037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.928+0000 7f4482a00640 1 --2- 172.21.15.107:0/2308839991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f447c1033e0 0x7f447c1037f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4470009900 tx=0x7f447002a410 comp rx=0 tx=0).stop 2024-07-26T12:36:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.928+0000 7f4482a00640 1 -- 172.21.15.107:0/2308839991 shutdown_connections 2024-07-26T12:36:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.928+0000 7f4482a00640 1 --2- 172.21.15.107:0/2308839991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f447c109670 0x7f447c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.928+0000 7f4482a00640 1 --2- 172.21.15.107:0/2308839991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f447c108d20 0x7f447c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.928+0000 7f4482a00640 1 --2- 172.21.15.107:0/2308839991 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f447c1033e0 0x7f447c1037f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.928+0000 7f4482a00640 1 -- 172.21.15.107:0/2308839991 >> 172.21.15.107:0/2308839991 conn(0x7f447c0fee70 msgr2=0x7f447c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:28.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.928+0000 7f4482a00640 1 -- 172.21.15.107:0/2308839991 shutdown_connections 2024-07-26T12:36:28.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.928+0000 7f4482a00640 1 -- 172.21.15.107:0/2308839991 wait complete. 2024-07-26T12:36:28.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.929+0000 7f4482a00640 1 Processor -- start 2024-07-26T12:36:28.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.930+0000 7f4482a00640 1 -- start start 2024-07-26T12:36:28.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.930+0000 7f4482a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f447c1033e0 0x7f447c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.930+0000 7f4482a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f447c108d20 0x7f447c198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.930+0000 7f4482a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f447c109670 0x7f447c1a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.930+0000 7f4482a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f447c113320 con 0x7f447c1033e0 2024-07-26T12:36:28.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.930+0000 7f4482a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1131a0 con 0x7f447c108d20 2024-07-26T12:36:28.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.930+0000 7f4482a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f447c1134a0 con 0x7f447c109670 2024-07-26T12:36:28.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4480c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f447c108d20 0x7f447c198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4481600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f447c1033e0 0x7f447c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4480c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f447c108d20 0x7f447c198d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:50002/0 (socket says 172.21.15.107:50002) 2024-07-26T12:36:28.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4480c00640 1 -- 172.21.15.107:0/2114571470 learned_addr learned my addr 172.21.15.107:0/2114571470 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:28.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4482000640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f447c109670 0x7f447c1a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4480c00640 1 -- 172.21.15.107:0/2114571470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f447c109670 msgr2=0x7f447c1a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4480c00640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f447c109670 0x7f447c1a3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4480c00640 1 -- 172.21.15.107:0/2114571470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f447c1033e0 msgr2=0x7f447c1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4480c00640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f447c1033e0 0x7f447c1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4480c00640 1 -- 172.21.15.107:0/2114571470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447c1a3db0 con 0x7f447c108d20 2024-07-26T12:36:28.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.931+0000 7f4481600640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f447c1033e0 0x7f447c1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:36:28.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.932+0000 7f4480c00640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f447c108d20 0x7f447c198d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f446800e9f0 tx=0x7f446800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:28.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.933+0000 7f447a000640 1 -- 172.21.15.107:0/2114571470 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446800ccc0 con 0x7f447c108d20 2024-07-26T12:36:28.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.933+0000 7f447a000640 1 -- 172.21.15.107:0/2114571470 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44680108e0 con 0x7f447c108d20 2024-07-26T12:36:28.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.933+0000 7f447a000640 1 -- 172.21.15.107:0/2114571470 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4468018860 con 0x7f447c108d20 2024-07-26T12:36:28.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.933+0000 7f4482a00640 1 -- 172.21.15.107:0/2114571470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f447c1a40a0 con 0x7f447c108d20 2024-07-26T12:36:28.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.933+0000 7f4482a00640 1 -- 172.21.15.107:0/2114571470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f447c1a4590 con 0x7f447c108d20 2024-07-26T12:36:28.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.934+0000 7f4482a00640 1 -- 172.21.15.107:0/2114571470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f447c104660 con 0x7f447c108d20 2024-07-26T12:36:28.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.936+0000 7f447a000640 1 -- 172.21.15.107:0/2114571470 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f4468010430 con 0x7f447c108d20 2024-07-26T12:36:28.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.938+0000 7f447a000640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3c10 0x7f44640b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:28.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.939+0000 7f447a000640 1 -- 172.21.15.107:0/2114571470 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f44680e2370 con 0x7f447c108d20 2024-07-26T12:36:28.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.939+0000 7f4481600640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3c10 0x7f44640b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:28.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.939+0000 7f4481600640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3c10 0x7f44640b6120 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4470000c00 tx=0x7f4470009680 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.942+0000 7f447a000640 1 -- 172.21.15.107:0/2114571470 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f44680ab520 con 0x7f447c108d20 2024-07-26T12:36:28.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.963+0000 7f41a6200640 1 -- 172.21.15.107:0/165373390 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f41a002e0c0 con 0x7f41a0075ad0 2024-07-26T12:36:28.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.970+0000 7f4197e00640 1 -- 172.21.15.107:0/165373390 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v32)=0 v32) ==== 120+0+0 (secure 0 0 0) 0x7f418c030dc0 con 0x7f41a0075ad0 2024-07-26T12:36:28.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 -- 172.21.15.107:0/165373390 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f41840b3b90 msgr2=0x7f41840b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f41840b3b90 0x7f41840b60a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4190011550 tx=0x7f4190005f90 comp rx=0 tx=0).stop 2024-07-26T12:36:28.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 -- 172.21.15.107:0/165373390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41a0075ad0 msgr2=0x7f41a019c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:28.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41a0075ad0 0x7f41a019c720 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f418c009780 tx=0x7f418c032d50 comp rx=0 tx=0).stop 2024-07-26T12:36:28.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 -- 172.21.15.107:0/165373390 shutdown_connections 2024-07-26T12:36:28.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f41840b3b90 0x7f41840b60a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41a010fb50 0x7f41a01a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f41a0075ad0 0x7f41a019c720 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 --2- 172.21.15.107:0/165373390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41a0075180 0x7f41a019c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:28.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 -- 172.21.15.107:0/165373390 >> 172.21.15.107:0/165373390 conn(0x7f41a007b280 msgr2=0x7f41a0101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:28.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 -- 172.21.15.107:0/165373390 shutdown_connections 2024-07-26T12:36:28.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:28.973+0000 7f41a6200640 1 -- 172.21.15.107:0/165373390 wait complete. 2024-07-26T12:36:29.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.138+0000 7f4482a00640 1 -- 172.21.15.107:0/2114571470 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f447c102110 con 0x7f447c108d20 2024-07-26T12:36:29.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:28 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:36:29.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:29.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:29.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:29.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:29.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:28 smithi107 ceph-mon[32659]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:29.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:28 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3204194864' entity='client.admin' 2024-07-26T12:36:29.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.146+0000 7f447a000640 1 -- 172.21.15.107:0/2114571470 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v33)=0 v33) ==== 121+0+0 (secure 0 0 0) 0x7f44680aae20 con 0x7f447c108d20 2024-07-26T12:36:29.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 -- 172.21.15.107:0/2114571470 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3c10 msgr2=0x7f44640b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:29.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3c10 0x7f44640b6120 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4470000c00 tx=0x7f4470009680 comp rx=0 tx=0).stop 2024-07-26T12:36:29.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 -- 172.21.15.107:0/2114571470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f447c108d20 msgr2=0x7f447c198d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:29.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f447c108d20 0x7f447c198d30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f446800e9f0 tx=0x7f446800eeb0 comp rx=0 tx=0).stop 2024-07-26T12:36:29.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 -- 172.21.15.107:0/2114571470 shutdown_connections 2024-07-26T12:36:29.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3c10 0x7f44640b6120 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:29.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f447c109670 0x7f447c1a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:29.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f447c108d20 0x7f447c198d30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:29.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 --2- 172.21.15.107:0/2114571470 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f447c1033e0 0x7f447c1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:29.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 -- 172.21.15.107:0/2114571470 >> 172.21.15.107:0/2114571470 conn(0x7f447c0fee70 msgr2=0x7f447c0ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:29.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 -- 172.21.15.107:0/2114571470 shutdown_connections 2024-07-26T12:36:29.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:29.149+0000 7f4482a00640 1 -- 172.21.15.107:0/2114571470 wait complete. 2024-07-26T12:36:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:28 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:36:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:28 smithi112 ceph-mon[37501]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:28 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3204194864' entity='client.admin' 2024-07-26T12:36:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:28 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:36:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:28 smithi132 ceph-mon[37580]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:28 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3204194864' entity='client.admin' 2024-07-26T12:36:29.375 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi107') 2024-07-26T12:36:29.375 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi107') 2024-07-26T12:36:30.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:29 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:36:30.078 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:29 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:36:30.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.285+0000 7f6d82600640 1 -- 172.21.15.107:0/3069764343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d7c109670 msgr2=0x7f6d7c10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:30.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.285+0000 7f6d82600640 1 --2- 172.21.15.107:0/3069764343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d7c109670 0x7f6d7c10fea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6d74009be0 tx=0x7f6d7402a6d0 comp rx=0 tx=0).stop 2024-07-26T12:36:30.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.285+0000 7f6d82600640 1 -- 172.21.15.107:0/3069764343 shutdown_connections 2024-07-26T12:36:30.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.285+0000 7f6d82600640 1 --2- 172.21.15.107:0/3069764343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d7c109670 0x7f6d7c10fea0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.285+0000 7f6d82600640 1 --2- 172.21.15.107:0/3069764343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d7c108d20 0x7f6d7c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.285+0000 7f6d82600640 1 --2- 172.21.15.107:0/3069764343 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c1033e0 0x7f6d7c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.285+0000 7f6d82600640 1 -- 172.21.15.107:0/3069764343 >> 172.21.15.107:0/3069764343 conn(0x7f6d7c0fee30 msgr2=0x7f6d7c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.285+0000 7f6d82600640 1 -- 172.21.15.107:0/3069764343 shutdown_connections 2024-07-26T12:36:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.285+0000 7f6d82600640 1 -- 172.21.15.107:0/3069764343 wait complete. 2024-07-26T12:36:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.286+0000 7f6d82600640 1 Processor -- start 2024-07-26T12:36:30.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.287+0000 7f6d82600640 1 -- start start 2024-07-26T12:36:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.287+0000 7f6d82600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d7c1033e0 0x7f6d7c19beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.287+0000 7f6d82600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c108d20 0x7f6d7c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.287+0000 7f6d82600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d7c109670 0x7f6d7c1a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.287+0000 7f6d82600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c113340 con 0x7f6d7c109670 2024-07-26T12:36:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.287+0000 7f6d82600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c1131c0 con 0x7f6d7c108d20 2024-07-26T12:36:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.287+0000 7f6d82600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c1134c0 con 0x7f6d7c1033e0 2024-07-26T12:36:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.287+0000 7f6d81c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d7c109670 0x7f6d7c1a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.287+0000 7f6d81c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d7c109670 0x7f6d7c1a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55014/0 (socket says 172.21.15.107:55014) 2024-07-26T12:36:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.287+0000 7f6d81c00640 1 -- 172.21.15.107:0/2504185492 learned_addr learned my addr 172.21.15.107:0/2504185492 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.288+0000 7f6d81200640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d7c1033e0 0x7f6d7c19beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:30.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.288+0000 7f6d7be00640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c108d20 0x7f6d7c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.288+0000 7f6d81c00640 1 -- 172.21.15.107:0/2504185492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d7c1033e0 msgr2=0x7f6d7c19beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.288+0000 7f6d81c00640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d7c1033e0 0x7f6d7c19beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.288+0000 7f6d81c00640 1 -- 172.21.15.107:0/2504185492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c108d20 msgr2=0x7f6d7c19c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.288+0000 7f6d81c00640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c108d20 0x7f6d7c19c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.288+0000 7f6d81c00640 1 -- 172.21.15.107:0/2504185492 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7c1a1e20 con 0x7f6d7c109670 2024-07-26T12:36:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.288+0000 7f6d81200640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d7c1033e0 0x7f6d7c19beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.288+0000 7f6d7be00640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c108d20 0x7f6d7c19c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:30.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.288+0000 7f6d81c00640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d7c109670 0x7f6d7c1a16f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6d74009b30 tx=0x7f6d74003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.289+0000 7f6d79600640 1 -- 172.21.15.107:0/2504185492 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d74040020 con 0x7f6d7c109670 2024-07-26T12:36:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.289+0000 7f6d79600640 1 -- 172.21.15.107:0/2504185492 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d74030970 con 0x7f6d7c109670 2024-07-26T12:36:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.289+0000 7f6d79600640 1 -- 172.21.15.107:0/2504185492 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d74036040 con 0x7f6d7c109670 2024-07-26T12:36:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.289+0000 7f6d82600640 1 -- 172.21.15.107:0/2504185492 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d7c1a20b0 con 0x7f6d7c109670 2024-07-26T12:36:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.289+0000 7f6d82600640 1 -- 172.21.15.107:0/2504185492 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d7c1a8870 con 0x7f6d7c109670 2024-07-26T12:36:30.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.290+0000 7f6d82600640 1 -- 172.21.15.107:0/2504185492 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d7c1037f0 con 0x7f6d7c109670 2024-07-26T12:36:30.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.293+0000 7f6d79600640 1 -- 172.21.15.107:0/2504185492 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f6d74043020 con 0x7f6d7c109670 2024-07-26T12:36:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.295+0000 7f6d79600640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6d640b3c90 0x7f6d640b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.295+0000 7f6d79600640 1 -- 172.21.15.107:0/2504185492 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f6d741030f0 con 0x7f6d7c109670 2024-07-26T12:36:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.295+0000 7f6d81200640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6d640b3c90 0x7f6d640b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:30.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.296+0000 7f6d81200640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6d640b3c90 0x7f6d640b61a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6d68009fc0 tx=0x7f6d68005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:30.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.296+0000 7f6d79600640 1 -- 172.21.15.107:0/2504185492 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6d7404a020 con 0x7f6d7c109670 2024-07-26T12:36:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:29 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:36:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:29 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:36:30.437 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:36:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:29 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:36:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:29 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:36:30.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.500+0000 7f6d82600640 1 -- 172.21.15.107:0/2504185492 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f6d7c1a8a40 con 0x7f6d7c109670 2024-07-26T12:36:30.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.506+0000 7f6d79600640 1 -- 172.21.15.107:0/2504185492 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v34)=0 v34) ==== 125+0+0 (secure 0 0 0) 0x7f6d740cc2a0 con 0x7f6d7c109670 2024-07-26T12:36:30.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 -- 172.21.15.107:0/2504185492 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6d640b3c90 msgr2=0x7f6d640b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6d640b3c90 0x7f6d640b61a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6d68009fc0 tx=0x7f6d68005f90 comp rx=0 tx=0).stop 2024-07-26T12:36:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 -- 172.21.15.107:0/2504185492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d7c109670 msgr2=0x7f6d7c1a16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d7c109670 0x7f6d7c1a16f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6d74009b30 tx=0x7f6d74003c10 comp rx=0 tx=0).stop 2024-07-26T12:36:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 -- 172.21.15.107:0/2504185492 shutdown_connections 2024-07-26T12:36:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6d640b3c90 0x7f6d640b61a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d7c109670 0x7f6d7c1a16f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6d7c108d20 0x7f6d7c19c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 --2- 172.21.15.107:0/2504185492 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6d7c1033e0 0x7f6d7c19beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:30.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 -- 172.21.15.107:0/2504185492 >> 172.21.15.107:0/2504185492 conn(0x7f6d7c0fee30 msgr2=0x7f6d7c1062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:30.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.509+0000 7f6d82600640 1 -- 172.21.15.107:0/2504185492 shutdown_connections 2024-07-26T12:36:30.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:30.510+0000 7f6d82600640 1 -- 172.21.15.107:0/2504185492 wait complete. 2024-07-26T12:36:31.343 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi107') 2024-07-26T12:36:31.344 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi107') 2024-07-26T12:36:31.344 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi107') 2024-07-26T12:36:31.344 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi107') 2024-07-26T12:36:31.344 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi107') 2024-07-26T12:36:31.344 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi107') 2024-07-26T12:36:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:31 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2504185492' entity='client.admin' 2024-07-26T12:36:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:31 smithi107 ceph-mon[32659]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:31 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2504185492' entity='client.admin' 2024-07-26T12:36:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:31 smithi132 ceph-mon[37580]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:31 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2504185492' entity='client.admin' 2024-07-26T12:36:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:31 smithi112 ceph-mon[37501]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:32.521 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi107') 2024-07-26T12:36:32.521 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi107') 2024-07-26T12:36:32.521 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'debug_cephsqlite', '20'], check_status=True, hostname='smithi107') 2024-07-26T12:36:32.521 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/allow_m_granularity', 'True'], check_status=True, hostname='smithi107') 2024-07-26T12:36:32.521 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/dump_on_update', 'True'], check_status=True, hostname='smithi107') 2024-07-26T12:36:33.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:33 smithi107 ceph-mon[32659]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:33 smithi112 ceph-mon[37501]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:33 smithi132 ceph-mon[37580]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:34.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.907+0000 7f447f200640 1 -- 172.21.15.107:0/1561868655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4480109650 msgr2=0x7f448010fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:34.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.907+0000 7f447f200640 1 --2- 172.21.15.107:0/1561868655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4480109650 0x7f448010fe80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4478009900 tx=0x7f447802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:36:34.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.908+0000 7f447f200640 1 -- 172.21.15.107:0/1561868655 shutdown_connections 2024-07-26T12:36:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.908+0000 7f447f200640 1 --2- 172.21.15.107:0/1561868655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4480109650 0x7f448010fe80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.908+0000 7f447f200640 1 --2- 172.21.15.107:0/1561868655 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4480108d00 0x7f4480109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.908+0000 7f447f200640 1 --2- 172.21.15.107:0/1561868655 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f44801033c0 0x7f44801037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.908+0000 7f447f200640 1 -- 172.21.15.107:0/1561868655 >> 172.21.15.107:0/1561868655 conn(0x7f44800fee10 msgr2=0x7f4480101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.908+0000 7f447f200640 1 -- 172.21.15.107:0/1561868655 shutdown_connections 2024-07-26T12:36:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.908+0000 7f447f200640 1 -- 172.21.15.107:0/1561868655 wait complete. 2024-07-26T12:36:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.909+0000 7f447f200640 1 Processor -- start 2024-07-26T12:36:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.909+0000 7f447f200640 1 -- start start 2024-07-26T12:36:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.909+0000 7f447f200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44801033c0 0x7f448019bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.909+0000 7f447f200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4480108d00 0x7f448019c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.909+0000 7f447f200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4480109650 0x7f44801a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.909+0000 7f447f200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4480113370 con 0x7f4480109650 2024-07-26T12:36:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.909+0000 7f447f200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f44801131f0 con 0x7f44801033c0 2024-07-26T12:36:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.909+0000 7f447f200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f44801134f0 con 0x7f4480108d00 2024-07-26T12:36:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.910+0000 7f447de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44801033c0 0x7f448019bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.910+0000 7f447de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44801033c0 0x7f448019bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44456/0 (socket says 172.21.15.107:44456) 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.910+0000 7f447de00640 1 -- 172.21.15.107:0/997997945 learned_addr learned my addr 172.21.15.107:0/997997945 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.910+0000 7f4475400640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4480108d00 0x7f448019c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.910+0000 7f447e800640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4480109650 0x7f44801a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.910+0000 7f447de00640 1 -- 172.21.15.107:0/997997945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4480108d00 msgr2=0x7f448019c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.910+0000 7f447de00640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4480108d00 0x7f448019c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.910+0000 7f447de00640 1 -- 172.21.15.107:0/997997945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4480109650 msgr2=0x7f44801a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.910+0000 7f447de00640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4480109650 0x7f44801a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.910+0000 7f447de00640 1 -- 172.21.15.107:0/997997945 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44801a1e30 con 0x7f44801033c0 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.911+0000 7f447de00640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44801033c0 0x7f448019bec0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f447000eb90 tx=0x7f447000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.911+0000 7f4477400640 1 -- 172.21.15.107:0/997997945 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4470004d20 con 0x7f44801033c0 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.911+0000 7f4477400640 1 -- 172.21.15.107:0/997997945 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f447000ee90 con 0x7f44801033c0 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.911+0000 7f447f200640 1 -- 172.21.15.107:0/997997945 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44801a2120 con 0x7f44801033c0 2024-07-26T12:36:34.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.911+0000 7f4477400640 1 -- 172.21.15.107:0/997997945 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4470010670 con 0x7f44801033c0 2024-07-26T12:36:34.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.911+0000 7f447f200640 1 -- 172.21.15.107:0/997997945 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44801a8880 con 0x7f44801033c0 2024-07-26T12:36:34.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.912+0000 7f447f200640 1 -- 172.21.15.107:0/997997945 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44801037d0 con 0x7f44801033c0 2024-07-26T12:36:34.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.914+0000 7f4477400640 1 -- 172.21.15.107:0/997997945 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f4470005020 con 0x7f44801033c0 2024-07-26T12:36:34.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.916+0000 7f4477400640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3b90 0x7f44640b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:34.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.916+0000 7f4477400640 1 -- 172.21.15.107:0/997997945 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f44700e26d0 con 0x7f44801033c0 2024-07-26T12:36:34.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.916+0000 7f4475400640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3b90 0x7f44640b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:34.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.917+0000 7f4475400640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3b90 0x7f44640b60a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4480109600 tx=0x7f4468005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:34.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:34.920+0000 7f4477400640 1 -- 172.21.15.107:0/997997945 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f44700ab880 con 0x7f44801033c0 2024-07-26T12:36:35.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.117+0000 7f447f200640 1 -- 172.21.15.107:0/997997945 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f44801a8ab0 con 0x7f44801033c0 2024-07-26T12:36:35.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.129+0000 7f4477400640 1 -- 172.21.15.107:0/997997945 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v35)=0 v35) ==== 133+0+0 (secure 0 0 0) 0x7f44700ab180 con 0x7f44801033c0 2024-07-26T12:36:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 -- 172.21.15.107:0/997997945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3b90 msgr2=0x7f44640b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3b90 0x7f44640b60a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4480109600 tx=0x7f4468005f90 comp rx=0 tx=0).stop 2024-07-26T12:36:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 -- 172.21.15.107:0/997997945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44801033c0 msgr2=0x7f448019bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44801033c0 0x7f448019bec0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f447000eb90 tx=0x7f447000c6a0 comp rx=0 tx=0).stop 2024-07-26T12:36:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 -- 172.21.15.107:0/997997945 shutdown_connections 2024-07-26T12:36:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f44640b3b90 0x7f44640b60a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4480109650 0x7f44801a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4480108d00 0x7f448019c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 --2- 172.21.15.107:0/997997945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f44801033c0 0x7f448019bec0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 -- 172.21.15.107:0/997997945 >> 172.21.15.107:0/997997945 conn(0x7f44800fee10 msgr2=0x7f4480106290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 -- 172.21.15.107:0/997997945 shutdown_connections 2024-07-26T12:36:35.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:35.133+0000 7f447f200640 1 -- 172.21.15.107:0/997997945 wait complete. 2024-07-26T12:36:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:35 smithi107 ceph-mon[32659]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:35 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' 2024-07-26T12:36:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:35.879 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/log_level', 'debug'], check_status=True, hostname='smithi107') 2024-07-26T12:36:35.879 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi107') 2024-07-26T12:36:35.879 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi107') 2024-07-26T12:36:35.879 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '14de5e40-4b4b-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi107') 2024-07-26T12:36:35.880 INFO:tasks.cephadm:Deploying OSDs... 2024-07-26T12:36:35.880 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:36:35.880 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T12:36:35.936 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T12:36:35.936 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-07-26T12:36:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:35 smithi112 ceph-mon[37501]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:35 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-07-26T12:36:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:35.986 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T12:36:35.986 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:35.986 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-07-26T12:36:35.986 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:35.987 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:35.987 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-07-26 12:34:25.563018723 +0000 2024-07-26T12:36:35.987 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-07-26 12:29:12.423026516 +0000 2024-07-26T12:36:35.987 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-07-26 12:29:12.423026516 +0000 2024-07-26T12:36:35.987 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-07-26 12:29:12.423026516 +0000 2024-07-26T12:36:35.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T12:36:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:35 smithi132 ceph-mon[37580]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:35 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' 2024-07-26T12:36:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:36:36.061 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-07-26T12:36:36.061 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-07-26T12:36:36.061 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000362013 s, 1.4 MB/s 2024-07-26T12:36:36.063 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T12:36:36.116 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-07-26T12:36:36.164 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T12:36:36.165 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:36.165 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-07-26T12:36:36.165 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:36.165 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:36.165 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-07-26 12:34:25.641017482 +0000 2024-07-26T12:36:36.165 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-07-26 12:29:13.002017327 +0000 2024-07-26T12:36:36.165 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-07-26 12:29:13.002017327 +0000 2024-07-26T12:36:36.165 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-07-26 12:29:13.002017327 +0000 2024-07-26T12:36:36.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T12:36:36.235 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-07-26T12:36:36.235 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-07-26T12:36:36.235 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000390334 s, 1.3 MB/s 2024-07-26T12:36:36.238 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T12:36:36.291 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-07-26T12:36:36.340 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T12:36:36.340 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:36.341 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-07-26T12:36:36.341 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:36.341 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:36.341 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-07-26 12:34:25.712016352 +0000 2024-07-26T12:36:36.341 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-07-26 12:29:13.607007726 +0000 2024-07-26T12:36:36.341 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-07-26 12:29:13.607007726 +0000 2024-07-26T12:36:36.341 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-07-26 12:29:13.607007726 +0000 2024-07-26T12:36:36.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T12:36:36.418 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-07-26T12:36:36.418 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-07-26T12:36:36.418 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00051846 s, 988 kB/s 2024-07-26T12:36:36.422 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T12:36:36.474 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-07-26T12:36:36.521 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T12:36:36.522 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:36.522 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-07-26T12:36:36.522 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:36.522 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:36.522 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-07-26 12:34:25.769015445 +0000 2024-07-26T12:36:36.522 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-07-26 12:29:14.189998474 +0000 2024-07-26T12:36:36.522 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-07-26 12:29:14.189998474 +0000 2024-07-26T12:36:36.522 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-07-26 12:29:14.189998474 +0000 2024-07-26T12:36:36.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T12:36:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-07-26T12:36:36.602 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-07-26T12:36:36.602 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000349029 s, 1.5 MB/s 2024-07-26T12:36:36.605 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T12:36:36.657 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-07-26T12:36:36.657 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T12:36:36.676 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T12:36:36.676 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-07-26T12:36:36.734 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T12:36:36.734 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:36.734 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-07-26T12:36:36.734 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:36.734 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:36.734 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-07-26 12:35:46.289745376 +0000 2024-07-26T12:36:36.734 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-07-26 12:27:39.312283457 +0000 2024-07-26T12:36:36.734 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-07-26 12:27:39.312283457 +0000 2024-07-26T12:36:36.734 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-07-26 12:27:39.312283457 +0000 2024-07-26T12:36:36.735 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T12:36:36.804 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-07-26T12:36:36.804 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-07-26T12:36:36.804 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000291519 s, 1.8 MB/s 2024-07-26T12:36:36.805 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T12:36:36.864 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-07-26T12:36:36.920 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T12:36:36.921 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:36.921 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-07-26T12:36:36.921 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:36.921 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:36.921 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-07-26 12:35:46.289745376 +0000 2024-07-26T12:36:36.921 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-07-26 12:27:39.642274825 +0000 2024-07-26T12:36:36.921 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-07-26 12:27:39.642274825 +0000 2024-07-26T12:36:36.921 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-07-26 12:27:39.642274825 +0000 2024-07-26T12:36:36.921 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T12:36:36.986 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-07-26T12:36:36.987 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-07-26T12:36:36.987 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000207564 s, 2.5 MB/s 2024-07-26T12:36:36.987 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T12:36:37.043 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-07-26T12:36:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T12:36:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:37.099 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-07-26T12:36:37.099 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:37.099 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:37.099 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-07-26 12:35:46.290745350 +0000 2024-07-26T12:36:37.100 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-07-26 12:27:39.963266427 +0000 2024-07-26T12:36:37.100 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-07-26 12:27:39.963266427 +0000 2024-07-26T12:36:37.100 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-07-26 12:27:39.963266427 +0000 2024-07-26T12:36:37.100 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T12:36:37.162 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-07-26T12:36:37.162 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-07-26T12:36:37.162 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000290349 s, 1.8 MB/s 2024-07-26T12:36:37.163 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T12:36:37.219 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-07-26T12:36:37.274 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T12:36:37.274 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:37.274 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-07-26T12:36:37.275 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:37.275 INFO:teuthology.orchestra.run.smithi112.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:37.275 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-07-26 12:35:46.290745350 +0000 2024-07-26T12:36:37.275 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-07-26 12:27:40.291257847 +0000 2024-07-26T12:36:37.275 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-07-26 12:27:40.291257847 +0000 2024-07-26T12:36:37.275 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: 2024-07-26 12:27:40.291257847 +0000 2024-07-26T12:36:37.275 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T12:36:37.336 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-07-26T12:36:37.337 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-07-26T12:36:37.337 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000212105 s, 2.4 MB/s 2024-07-26T12:36:37.338 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T12:36:37.394 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-07-26T12:36:37.394 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T12:36:37.413 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T12:36:37.413 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-07-26T12:36:37.470 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T12:36:37.470 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:37.470 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-07-26T12:36:37.470 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:37.470 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:37.470 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-07-26 12:35:44.050374368 +0000 2024-07-26T12:36:37.471 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-07-26 12:27:38.796816462 +0000 2024-07-26T12:36:37.471 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-07-26 12:27:38.796816462 +0000 2024-07-26T12:36:37.471 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-07-26 12:27:38.796816462 +0000 2024-07-26T12:36:37.471 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T12:36:37.538 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-07-26T12:36:37.538 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-07-26T12:36:37.538 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000206045 s, 2.5 MB/s 2024-07-26T12:36:37.539 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T12:36:37.596 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-07-26T12:36:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T12:36:37.652 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:37.652 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-07-26T12:36:37.652 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:37.652 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:37.652 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-07-26 12:35:44.050374368 +0000 2024-07-26T12:36:37.653 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-07-26 12:27:39.142807528 +0000 2024-07-26T12:36:37.653 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-07-26 12:27:39.142807528 +0000 2024-07-26T12:36:37.653 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-07-26 12:27:39.142807528 +0000 2024-07-26T12:36:37.653 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T12:36:37.716 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-07-26T12:36:37.716 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-07-26T12:36:37.716 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000188763 s, 2.7 MB/s 2024-07-26T12:36:37.717 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T12:36:37.773 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-07-26T12:36:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:37 smithi132 ceph-mon[37580]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:37.824 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T12:36:37.824 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:37.825 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-07-26T12:36:37.825 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:37.825 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:37.825 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-07-26 12:35:44.051374343 +0000 2024-07-26T12:36:37.825 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-07-26 12:27:39.458799370 +0000 2024-07-26T12:36:37.825 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-07-26 12:27:39.458799370 +0000 2024-07-26T12:36:37.825 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-07-26 12:27:39.458799370 +0000 2024-07-26T12:36:37.825 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T12:36:37.887 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-07-26T12:36:37.887 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-07-26T12:36:37.887 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000215238 s, 2.4 MB/s 2024-07-26T12:36:37.889 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T12:36:37.944 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-07-26T12:36:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:37 smithi112 ceph-mon[37501]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:37.999 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T12:36:38.000 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T12:36:38.000 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-07-26T12:36:38.000 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:36:38.000 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T12:36:38.000 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-07-26 12:35:44.051374343 +0000 2024-07-26T12:36:38.000 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-07-26 12:27:39.784790952 +0000 2024-07-26T12:36:38.000 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-07-26 12:27:39.784790952 +0000 2024-07-26T12:36:38.000 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-07-26 12:27:39.784790952 +0000 2024-07-26T12:36:38.000 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T12:36:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:37 smithi107 ceph-mon[32659]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:38.062 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-07-26T12:36:38.062 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-07-26T12:36:38.062 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000198904 s, 2.6 MB/s 2024-07-26T12:36:38.063 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T12:36:38.119 INFO:tasks.cephadm:Deploying osd.0 on smithi107 with /dev/vg_nvme/lv_4... 2024-07-26T12:36:38.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T12:36:38.561 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:36:39.728 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:39 smithi107 ceph-mon[32659]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:39 smithi112 ceph-mon[37501]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:39 smithi132 ceph-mon[37580]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:40.837 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T12:36:41.016 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T12:36:41.061 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-07-26T12:36:41.061 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-07-26T12:36:41.061 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0351697 s, 298 MB/s 2024-07-26T12:36:41.061 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-07-26T12:36:41.722 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:41 smithi107 ceph-mon[32659]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:41.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_4 2024-07-26T12:36:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:41 smithi112 ceph-mon[37501]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:41 smithi132 ceph-mon[37580]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:42.261 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:36:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:43 smithi107 ceph-mon[32659]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:43 smithi112 ceph-mon[37501]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:43 smithi132 ceph-mon[37580]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:44.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.306+0000 7f1f84c00640 1 -- 172.21.15.107:0/995277353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80107e00 msgr2=0x7f1f80108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.306+0000 7f1f84c00640 1 --2- 172.21.15.107:0/995277353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80107e00 0x7f1f80108210 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009900 tx=0x7f1f6802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:36:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.306+0000 7f1f84c00640 1 -- 172.21.15.107:0/995277353 shutdown_connections 2024-07-26T12:36:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.306+0000 7f1f84c00640 1 --2- 172.21.15.107:0/995277353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f80108750 0x7f1f8010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.306+0000 7f1f84c00640 1 --2- 172.21.15.107:0/995277353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80107e00 0x7f1f80108210 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.306+0000 7f1f84c00640 1 --2- 172.21.15.107:0/995277353 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f80102510 0x7f1f80102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.306+0000 7f1f84c00640 1 -- 172.21.15.107:0/995277353 >> 172.21.15.107:0/995277353 conn(0x7f1f80079190 msgr2=0x7f1f800795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:36:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.306+0000 7f1f84c00640 1 -- 172.21.15.107:0/995277353 shutdown_connections 2024-07-26T12:36:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.306+0000 7f1f84c00640 1 -- 172.21.15.107:0/995277353 wait complete. 2024-07-26T12:36:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.307+0000 7f1f84c00640 1 Processor -- start 2024-07-26T12:36:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.308+0000 7f1f84c00640 1 -- start start 2024-07-26T12:36:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.308+0000 7f1f84c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80102510 0x7f1f80104b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.308+0000 7f1f84c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f80107e00 0x7f1f80105080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:44.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.308+0000 7f1f84c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f80108750 0x7f1f801055c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.308+0000 7f1f84c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1f801123a0 con 0x7f1f80102510 2024-07-26T12:36:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.308+0000 7f1f84c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80112220 con 0x7f1f80108750 2024-07-26T12:36:44.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.308+0000 7f1f84c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1f80112520 con 0x7f1f80107e00 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7e800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f80108750 0x7f1f801055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7d400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f80107e00 0x7f1f80105080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7e800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f80108750 0x7f1f801055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:56262/0 (socket says 172.21.15.107:56262) 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7e800640 1 -- 172.21.15.107:0/3645885136 learned_addr learned my addr 172.21.15.107:0/3645885136 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7de00640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80102510 0x7f1f80104b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7d400640 1 -- 172.21.15.107:0/3645885136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f80108750 msgr2=0x7f1f801055c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7d400640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f80108750 0x7f1f801055c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7d400640 1 -- 172.21.15.107:0/3645885136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80102510 msgr2=0x7f1f80104b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7d400640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80102510 0x7f1f80104b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7d400640 1 -- 172.21.15.107:0/3645885136 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f801a8150 con 0x7f1f80107e00 2024-07-26T12:36:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7e800640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f80108750 0x7f1f801055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:36:44.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.309+0000 7f1f7d400640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f80107e00 0x7f1f80105080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1f6802a910 tx=0x7f1f68031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:44.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.310+0000 7f1f76a00640 1 -- 172.21.15.107:0/3645885136 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6803f070 con 0x7f1f80107e00 2024-07-26T12:36:44.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.310+0000 7f1f76a00640 1 -- 172.21.15.107:0/3645885136 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f68004490 con 0x7f1f80107e00 2024-07-26T12:36:44.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.310+0000 7f1f84c00640 1 -- 172.21.15.107:0/3645885136 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f801a83e0 con 0x7f1f80107e00 2024-07-26T12:36:44.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.310+0000 7f1f76a00640 1 -- 172.21.15.107:0/3645885136 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f68035070 con 0x7f1f80107e00 2024-07-26T12:36:44.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.310+0000 7f1f84c00640 1 -- 172.21.15.107:0/3645885136 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f801a8910 con 0x7f1f80107e00 2024-07-26T12:36:44.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.311+0000 7f1f84c00640 1 -- 172.21.15.107:0/3645885136 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f80102920 con 0x7f1f80107e00 2024-07-26T12:36:44.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.313+0000 7f1f76a00640 1 -- 172.21.15.107:0/3645885136 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f1f6803b460 con 0x7f1f80107e00 2024-07-26T12:36:44.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.315+0000 7f1f76a00640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1f640b3c10 0x7f1f640b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:36:44.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.315+0000 7f1f76a00640 1 -- 172.21.15.107:0/3645885136 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f1f68102760 con 0x7f1f80107e00 2024-07-26T12:36:44.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.315+0000 7f1f7de00640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1f640b3c10 0x7f1f640b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:36:44.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.316+0000 7f1f7de00640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1f640b3c10 0x7f1f640b6120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f80106590 tx=0x7f1f70005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:36:44.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.317+0000 7f1f76a00640 1 -- 172.21.15.107:0/3645885136 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1f680cb2c0 con 0x7f1f80107e00 2024-07-26T12:36:44.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:36:44.523+0000 7f1f84c00640 1 -- 172.21.15.107:0/3645885136 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1f80063230 con 0x7f1f640b3c10 2024-07-26T12:36:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:36:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:36:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:36:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:36:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:36:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:36:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:45 smithi112 ceph-mon[37501]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:45 smithi112 ceph-mon[37501]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:36:46.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:45 smithi107 ceph-mon[32659]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:46.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:45 smithi107 ceph-mon[32659]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:36:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:45 smithi132 ceph-mon[37580]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:45 smithi132 ceph-mon[37580]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:36:47.729 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:47 smithi107 ceph-mon[32659]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:47 smithi112 ceph-mon[37501]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:47 smithi132 ceph-mon[37580]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:48 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2049098095' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f413e43-6b6b-4596-ab6e-c75cdf9c7c96"}]: dispatch 2024-07-26T12:36:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:48 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f413e43-6b6b-4596-ab6e-c75cdf9c7c96"}]: dispatch 2024-07-26T12:36:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:48 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f413e43-6b6b-4596-ab6e-c75cdf9c7c96"}]': finished 2024-07-26T12:36:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:48 smithi107 ceph-mon[32659]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T12:36:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:48 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:36:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:48 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2049098095' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f413e43-6b6b-4596-ab6e-c75cdf9c7c96"}]: dispatch 2024-07-26T12:36:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:48 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f413e43-6b6b-4596-ab6e-c75cdf9c7c96"}]: dispatch 2024-07-26T12:36:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:48 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f413e43-6b6b-4596-ab6e-c75cdf9c7c96"}]': finished 2024-07-26T12:36:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:48 smithi112 ceph-mon[37501]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T12:36:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:48 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:36:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:48 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2049098095' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f413e43-6b6b-4596-ab6e-c75cdf9c7c96"}]: dispatch 2024-07-26T12:36:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:48 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f413e43-6b6b-4596-ab6e-c75cdf9c7c96"}]: dispatch 2024-07-26T12:36:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:48 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f413e43-6b6b-4596-ab6e-c75cdf9c7c96"}]': finished 2024-07-26T12:36:49.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:48 smithi132 ceph-mon[37580]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T12:36:49.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:48 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:36:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:49 smithi112 ceph-mon[37501]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:49 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/616810981' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:36:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:49 smithi107 ceph-mon[32659]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:49 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/616810981' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:36:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:49 smithi132 ceph-mon[37580]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:49 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/616810981' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:36:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:51 smithi112 ceph-mon[37501]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:51 smithi107 ceph-mon[32659]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:51 smithi132 ceph-mon[37580]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:53 smithi112 ceph-mon[37501]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:54.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:53 smithi107 ceph-mon[32659]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:53 smithi132 ceph-mon[37580]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:55 smithi112 ceph-mon[37501]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:56.032 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:55 smithi107 ceph-mon[32659]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:55 smithi132 ceph-mon[37580]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:57.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:57 smithi107 ceph-mon[32659]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:57.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T12:36:57.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:57 smithi112 ceph-mon[37501]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T12:36:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:57 smithi132 ceph-mon[37580]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T12:36:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:58.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:58 smithi107 ceph-mon[32659]: Deploying daemon osd.0 on smithi107 2024-07-26T12:36:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:58 smithi112 ceph-mon[37501]: Deploying daemon osd.0 on smithi107 2024-07-26T12:36:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:58 smithi132 ceph-mon[37580]: Deploying daemon osd.0 on smithi107 2024-07-26T12:36:59.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:36:59 smithi107 ceph-mon[32659]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:36:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:36:59 smithi112 ceph-mon[37501]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:36:59 smithi132 ceph-mon[37580]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:01.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:01 smithi107 ceph-mon[32659]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:01 smithi112 ceph-mon[37501]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:01 smithi132 ceph-mon[37580]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:03 smithi112 ceph-mon[37501]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:03 smithi107 ceph-mon[32659]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:03 smithi132 ceph-mon[37580]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:05.895 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:05 smithi107 ceph-mon[32659]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:05.895 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:05.895 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:05.895 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:05 smithi112 ceph-mon[37501]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:05 smithi132 ceph-mon[37580]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:07 smithi112 ceph-mon[37501]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:08.015 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:07 smithi107 ceph-mon[32659]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:07 smithi132 ceph-mon[37580]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:08.993 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:08.993 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:08.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:08.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:08.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:09.926 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:09 smithi107 ceph-mon[32659]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:09.926 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:09 smithi107 ceph-mon[32659]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T12:37:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:09 smithi112 ceph-mon[37501]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:09 smithi112 ceph-mon[37501]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T12:37:10.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.027+0000 7f1f76a00640 1 -- 172.21.15.107:0/3645885136 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1f80063230 con 0x7f1f640b3c10 2024-07-26T12:37:10.071 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 0 on host 'smithi107' 2024-07-26T12:37:10.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.032+0000 7f1f84c00640 1 -- 172.21.15.107:0/3645885136 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1f640b3c10 msgr2=0x7f1f640b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:10.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.032+0000 7f1f84c00640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1f640b3c10 0x7f1f640b6120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f80106590 tx=0x7f1f70005f90 comp rx=0 tx=0).stop 2024-07-26T12:37:10.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.032+0000 7f1f84c00640 1 -- 172.21.15.107:0/3645885136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f80107e00 msgr2=0x7f1f80105080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:10.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.032+0000 7f1f84c00640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f80107e00 0x7f1f80105080 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1f6802a910 tx=0x7f1f68031a20 comp rx=0 tx=0).stop 2024-07-26T12:37:10.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.032+0000 7f1f84c00640 1 -- 172.21.15.107:0/3645885136 shutdown_connections 2024-07-26T12:37:10.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.032+0000 7f1f84c00640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1f640b3c10 0x7f1f640b6120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:10.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.032+0000 7f1f84c00640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1f80108750 0x7f1f801055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:10.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.032+0000 7f1f84c00640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1f80107e00 0x7f1f80105080 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:10.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.032+0000 7f1f84c00640 1 --2- 172.21.15.107:0/3645885136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f80102510 0x7f1f80104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:10.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.032+0000 7f1f84c00640 1 -- 172.21.15.107:0/3645885136 >> 172.21.15.107:0/3645885136 conn(0x7f1f80079190 msgr2=0x7f1f8006b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:37:10.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.033+0000 7f1f84c00640 1 -- 172.21.15.107:0/3645885136 shutdown_connections 2024-07-26T12:37:10.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:37:10.033+0000 7f1f84c00640 1 -- 172.21.15.107:0/3645885136 wait complete. 2024-07-26T12:37:10.073 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:09 smithi132 ceph-mon[37580]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:10.074 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:09 smithi132 ceph-mon[37580]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T12:37:10.787 DEBUG:teuthology.orchestra.run.smithi107:osd.0> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.0.service 2024-07-26T12:37:10.792 INFO:tasks.cephadm:Deploying osd.1 on smithi112 with /dev/vg_nvme/lv_4... 2024-07-26T12:37:10.792 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T12:37:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:10 smithi112 ceph-mon[37501]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T12:37:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:10 smithi112 ceph-mon[37501]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T12:37:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:10 smithi112 ceph-mon[37501]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:37:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:10 smithi112 ceph-mon[37501]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:37:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:10 smithi112 ceph-mon[37501]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T12:37:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:10 smithi107 ceph-mon[32659]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T12:37:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:10 smithi107 ceph-mon[32659]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T12:37:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:10 smithi107 ceph-mon[32659]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:37:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:10 smithi107 ceph-mon[32659]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:37:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:10 smithi107 ceph-mon[32659]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T12:37:11.046 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T12:37:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:10 smithi132 ceph-mon[37580]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T12:37:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:10 smithi132 ceph-mon[37580]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T12:37:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:10 smithi132 ceph-mon[37580]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:37:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:10 smithi132 ceph-mon[37580]: from='osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:37:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:11.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:10 smithi132 ceph-mon[37580]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T12:37:11.988 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:11 smithi112 ceph-mon[37501]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:11.988 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:11 smithi107 ceph-mon[32659]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:11 smithi132 ceph-mon[37580]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:12.577 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T12:37:12.661 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T12:37:12.674 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-07-26T12:37:12.675 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-07-26T12:37:12.675 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115297 s, 909 MB/s 2024-07-26T12:37:12.675 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-07-26T12:37:12.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:12 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:37:12.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:12 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:37:12.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:12.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:12 smithi107 ceph-mon[32659]: osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] boot 2024-07-26T12:37:12.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:12 smithi107 ceph-mon[32659]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T12:37:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:12 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:37:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:12 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:37:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:12 smithi112 ceph-mon[37501]: osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] boot 2024-07-26T12:37:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:12 smithi112 ceph-mon[37501]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T12:37:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:12 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:37:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:12 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:37:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T12:37:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:12 smithi132 ceph-mon[37580]: osd.0 [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] boot 2024-07-26T12:37:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:12 smithi132 ceph-mon[37580]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T12:37:13.231 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_4 2024-07-26T12:37:13.484 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T12:37:13.770 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:13 smithi112 ceph-mon[37501]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:13.771 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:13 smithi112 ceph-mon[37501]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T12:37:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:13 smithi107 ceph-mon[32659]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:13 smithi107 ceph-mon[32659]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T12:37:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:13 smithi132 ceph-mon[37580]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T12:37:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:13 smithi132 ceph-mon[37580]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T12:37:14.903 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:14 smithi112 ceph-mon[37501]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:14.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.918+0000 7f8bdf04e640 1 -- 172.21.15.112:0/1079963608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bd81035b0 msgr2=0x7f8bd81039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:14.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.918+0000 7f8bdf04e640 1 --2- 172.21.15.112:0/1079963608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bd81035b0 0x7f8bd81039c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009900 tx=0x7f8bc802a410 comp rx=0 tx=0).stop 2024-07-26T12:37:14.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.918+0000 7f8bdf04e640 1 -- 172.21.15.112:0/1079963608 shutdown_connections 2024-07-26T12:37:14.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.918+0000 7f8bdf04e640 1 --2- 172.21.15.112:0/1079963608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd8109840 0x7f8bd8110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:14.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.918+0000 7f8bdf04e640 1 --2- 172.21.15.112:0/1079963608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bd8108ef0 0x7f8bd8109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:14.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.918+0000 7f8bdf04e640 1 --2- 172.21.15.112:0/1079963608 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bd81035b0 0x7f8bd81039c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:14.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.918+0000 7f8bdf04e640 1 -- 172.21.15.112:0/1079963608 >> 172.21.15.112:0/1079963608 conn(0x7f8bd80ff000 msgr2=0x7f8bd8101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:37:14.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.918+0000 7f8bdf04e640 1 -- 172.21.15.112:0/1079963608 shutdown_connections 2024-07-26T12:37:14.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.918+0000 7f8bdf04e640 1 -- 172.21.15.112:0/1079963608 wait complete. 2024-07-26T12:37:14.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdf04e640 1 Processor -- start 2024-07-26T12:37:14.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdf04e640 1 -- start start 2024-07-26T12:37:14.922 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdf04e640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bd81035b0 0x7f8bd8199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdf04e640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bd8108ef0 0x7f8bd819a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdf04e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd8109840 0x7f8bd819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdf04e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8113580 con 0x7f8bd8109840 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdf04e640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8113400 con 0x7f8bd81035b0 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdf04e640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8113700 con 0x7f8bd8108ef0 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdcdea640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bd81035b0 0x7f8bd8199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdcdea640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bd81035b0 0x7f8bd8199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:38870/0 (socket says 172.21.15.112:38870) 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bdcdea640 1 -- 172.21.15.112:0/389344515 learned_addr learned my addr 172.21.15.112:0/389344515 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.919+0000 7f8bd7fff640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bd8108ef0 0x7f8bd819a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bdcdea640 1 -- 172.21.15.112:0/389344515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bd8108ef0 msgr2=0x7f8bd819a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bdcdea640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bd8108ef0 0x7f8bd819a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bdcdea640 1 -- 172.21.15.112:0/389344515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd8109840 msgr2=0x7f8bd819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:14.923 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bdcdea640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd8109840 0x7f8bd819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bdcdea640 1 -- 172.21.15.112:0/389344515 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd819fee0 con 0x7f8bd81035b0 2024-07-26T12:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bdcdea640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bd81035b0 0x7f8bd8199f70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8005940 tx=0x7f8bc80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bd5ffb640 1 -- 172.21.15.112:0/389344515 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc803f030 con 0x7f8bd81035b0 2024-07-26T12:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bd5ffb640 1 -- 172.21.15.112:0/389344515 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bc8003cb0 con 0x7f8bd81035b0 2024-07-26T12:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bdf04e640 1 -- 172.21.15.112:0/389344515 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd81a0170 con 0x7f8bd81035b0 2024-07-26T12:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bd5ffb640 1 -- 172.21.15.112:0/389344515 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8035070 con 0x7f8bd81035b0 2024-07-26T12:37:14.924 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.920+0000 7f8bdf04e640 1 -- 172.21.15.112:0/389344515 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd81a6930 con 0x7f8bd81035b0 2024-07-26T12:37:14.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.921+0000 7f8bdf04e640 1 -- 172.21.15.112:0/389344515 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd81039c0 con 0x7f8bd81035b0 2024-07-26T12:37:14.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.922+0000 7f8bd5ffb640 1 -- 172.21.15.112:0/389344515 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f8bc8042030 con 0x7f8bd81035b0 2024-07-26T12:37:14.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.923+0000 7f8bd5ffb640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8bc00b3c10 0x7f8bc00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:37:14.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.923+0000 7f8bd5ffb640 1 -- 172.21.15.112:0/389344515 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1982+0+0 (secure 0 0 0) 0x7f8bc8102d00 con 0x7f8bd81035b0 2024-07-26T12:37:14.925 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.924+0000 7f8bd7fff640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8bc00b3c10 0x7f8bc00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:14.926 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.925+0000 7f8bd7fff640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8bc00b3c10 0x7f8bc00b6120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8108a60 tx=0x7f8bcc00b390 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:37:14.928 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:14.926+0000 7f8bd5ffb640 1 -- 172.21.15.112:0/389344515 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8bc80cbcf0 con 0x7f8bd81035b0 2024-07-26T12:37:15.034 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:14 smithi107 ceph-mon[32659]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:14 smithi132 ceph-mon[37580]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:15.111 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:15.109+0000 7f8bdf04e640 1 -- 172.21.15.112:0/389344515 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8bd8063230 con 0x7f8bc00b3c10 2024-07-26T12:37:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi107 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 17089M 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:16.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:16.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi107 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 17089M 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:16.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi107 2024-07-26T12:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 17089M 2024-07-26T12:37:16.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:16.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:16.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:16.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:17 smithi112 ceph-mon[37501]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:17 smithi107 ceph-mon[32659]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:17 smithi132 ceph-mon[37580]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:18 smithi112 ceph-mon[37501]: from='client.? 172.21.15.112:0/3578382452' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ac0320-aac0-4810-ad34-6798956f1969"}]: dispatch 2024-07-26T12:37:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:18 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ac0320-aac0-4810-ad34-6798956f1969"}]: dispatch 2024-07-26T12:37:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:18 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44ac0320-aac0-4810-ad34-6798956f1969"}]': finished 2024-07-26T12:37:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:18 smithi112 ceph-mon[37501]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T12:37:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:18 smithi112 ceph-mon[37501]: from='client.? 172.21.15.112:0/3224549502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:37:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:18 smithi107 ceph-mon[32659]: from='client.? 172.21.15.112:0/3578382452' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ac0320-aac0-4810-ad34-6798956f1969"}]: dispatch 2024-07-26T12:37:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:18 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ac0320-aac0-4810-ad34-6798956f1969"}]: dispatch 2024-07-26T12:37:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:18 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44ac0320-aac0-4810-ad34-6798956f1969"}]': finished 2024-07-26T12:37:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:18.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:18 smithi107 ceph-mon[32659]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T12:37:18.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:18 smithi107 ceph-mon[32659]: from='client.? 172.21.15.112:0/3224549502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:37:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:18 smithi132 ceph-mon[37580]: from='client.? 172.21.15.112:0/3578382452' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ac0320-aac0-4810-ad34-6798956f1969"}]: dispatch 2024-07-26T12:37:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:18 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44ac0320-aac0-4810-ad34-6798956f1969"}]: dispatch 2024-07-26T12:37:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:18 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44ac0320-aac0-4810-ad34-6798956f1969"}]': finished 2024-07-26T12:37:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:18 smithi132 ceph-mon[37580]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T12:37:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:18 smithi132 ceph-mon[37580]: from='client.? 172.21.15.112:0/3224549502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:37:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:19 smithi112 ceph-mon[37501]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:19 smithi107 ceph-mon[32659]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:19 smithi132 ceph-mon[37580]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:21 smithi112 ceph-mon[37501]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:21 smithi107 ceph-mon[32659]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:21 smithi132 ceph-mon[37580]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:23.780 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:23 smithi112 ceph-mon[37501]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:23 smithi107 ceph-mon[32659]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:23 smithi132 ceph-mon[37580]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:24.793 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T12:37:24.793 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T12:37:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T12:37:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:25.688 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:25 smithi112 ceph-mon[37501]: Deploying daemon osd.1 on smithi112 2024-07-26T12:37:25.688 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:25 smithi112 ceph-mon[37501]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:25 smithi107 ceph-mon[32659]: Deploying daemon osd.1 on smithi112 2024-07-26T12:37:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:25 smithi107 ceph-mon[32659]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:25 smithi132 ceph-mon[37580]: Deploying daemon osd.1 on smithi112 2024-07-26T12:37:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:25 smithi132 ceph-mon[37580]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:27.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:27 smithi132 ceph-mon[37580]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:27 smithi112 ceph-mon[37501]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:27 smithi107 ceph-mon[32659]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:29 smithi107 ceph-mon[32659]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:29 smithi112 ceph-mon[37501]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:29 smithi132 ceph-mon[37580]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:31.921 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:31 smithi112 ceph-mon[37501]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:31.922 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:31.922 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:31.922 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:31.922 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:31.922 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:31 smithi107 ceph-mon[32659]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:31 smithi132 ceph-mon[37580]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:32.268 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.266+0000 7f8bd5ffb640 1 -- 172.21.15.112:0/389344515 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8bd8063230 con 0x7f8bc00b3c10 2024-07-26T12:37:32.268 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 1 on host 'smithi112' 2024-07-26T12:37:32.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 -- 172.21.15.112:0/389344515 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8bc00b3c10 msgr2=0x7f8bc00b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:32.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8bc00b3c10 0x7f8bc00b6120 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8108a60 tx=0x7f8bcc00b390 comp rx=0 tx=0).stop 2024-07-26T12:37:32.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 -- 172.21.15.112:0/389344515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bd81035b0 msgr2=0x7f8bd8199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:32.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bd81035b0 0x7f8bd8199f70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8005940 tx=0x7f8bc80042f0 comp rx=0 tx=0).stop 2024-07-26T12:37:32.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 -- 172.21.15.112:0/389344515 shutdown_connections 2024-07-26T12:37:32.270 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8bc00b3c10 0x7f8bc00b6120 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:32.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8bd8109840 0x7f8bd819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:32.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8bd8108ef0 0x7f8bd819a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:32.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 --2- 172.21.15.112:0/389344515 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8bd81035b0 0x7f8bd8199f70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:32.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 -- 172.21.15.112:0/389344515 >> 172.21.15.112:0/389344515 conn(0x7f8bd80ff000 msgr2=0x7f8bd8101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:37:32.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 -- 172.21.15.112:0/389344515 shutdown_connections 2024-07-26T12:37:32.271 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:37:32.268+0000 7f8bdf04e640 1 -- 172.21.15.112:0/389344515 wait complete. 2024-07-26T12:37:32.829 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:32.829 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:32.829 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:32.920 DEBUG:teuthology.orchestra.run.smithi112:osd.1> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.1.service 2024-07-26T12:37:32.922 INFO:tasks.cephadm:Deploying osd.2 on smithi132 with /dev/vg_nvme/lv_4... 2024-07-26T12:37:32.923 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T12:37:32.948 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:32.948 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:32.948 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:33.177 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:37:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:33 smithi132 ceph-mon[37580]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:33 smithi132 ceph-mon[37580]: from='osd.1 [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T12:37:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:33 smithi132 ceph-mon[37580]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T12:37:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:33 smithi112 ceph-mon[37501]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:33 smithi112 ceph-mon[37501]: from='osd.1 [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T12:37:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:33 smithi112 ceph-mon[37501]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T12:37:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:33 smithi107 ceph-mon[32659]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:33 smithi107 ceph-mon[32659]: from='osd.1 [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T12:37:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:33 smithi107 ceph-mon[32659]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T12:37:34.776 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T12:37:34.776 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:34 smithi132 ceph-mon[37580]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T12:37:34.776 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:34.777 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:34 smithi132 ceph-mon[37580]: from='osd.1 [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:37:34.777 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:34 smithi132 ceph-mon[37580]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T12:37:34.777 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:34 smithi132 ceph-mon[37580]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:37:34.863 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T12:37:34.876 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-07-26T12:37:34.876 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-07-26T12:37:34.876 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114943 s, 912 MB/s 2024-07-26T12:37:34.876 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-07-26T12:37:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:34 smithi112 ceph-mon[37501]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T12:37:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:34 smithi112 ceph-mon[37501]: from='osd.1 [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:37:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:34 smithi112 ceph-mon[37501]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T12:37:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:34 smithi112 ceph-mon[37501]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:37:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:34 smithi107 ceph-mon[32659]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T12:37:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:34 smithi107 ceph-mon[32659]: from='osd.1 [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:37:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:34 smithi107 ceph-mon[32659]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T12:37:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:34 smithi107 ceph-mon[32659]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:37:35.519 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_4 2024-07-26T12:37:35.640 INFO:journalctl@ceph.osd.1.smithi112.stdout:Jul 26 12:37:35 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-1[43358]: 2024-07-26T12:37:35.426+0000 7f66512fd640 -1 osd.1 0 waiting for initial osdmap 2024-07-26T12:37:35.640 INFO:journalctl@ceph.osd.1.smithi112.stdout:Jul 26 12:37:35 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-1[43358]: 2024-07-26T12:37:35.430+0000 7f66490f3640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T12:37:35.814 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:37:35.938 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:35 smithi132 ceph-mon[37580]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:35.938 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:35 smithi132 ceph-mon[37580]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:37:35.938 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:35.938 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:35 smithi132 ceph-mon[37580]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T12:37:35.938 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:35.939 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:35 smithi132 ceph-mon[37580]: from='osd.1 ' entity='osd.1' 2024-07-26T12:37:35.939 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:35 smithi112 ceph-mon[37501]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:35 smithi112 ceph-mon[37501]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:37:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:35 smithi112 ceph-mon[37501]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T12:37:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:35 smithi112 ceph-mon[37501]: from='osd.1 ' entity='osd.1' 2024-07-26T12:37:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:35 smithi107 ceph-mon[32659]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:35 smithi107 ceph-mon[32659]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:37:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:35 smithi107 ceph-mon[32659]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T12:37:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:35 smithi107 ceph-mon[32659]: from='osd.1 ' entity='osd.1' 2024-07-26T12:37:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: osd.1 [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] boot 2024-07-26T12:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:36 smithi112 ceph-mon[37501]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T12:37:37.009 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:37:37.009 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:37:37.009 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:37.009 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:37.009 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:37.009 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:37.009 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:37.009 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:37.009 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:37.010 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:37.010 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:37.010 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: osd.1 [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] boot 2024-07-26T12:37:37.010 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:36 smithi132 ceph-mon[37580]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T12:37:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: osd.1 [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] boot 2024-07-26T12:37:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:36 smithi107 ceph-mon[32659]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T12:37:37.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.349+0000 7f143f84b640 1 -- 172.21.15.132:0/2233250099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1438109820 msgr2=0x7f1438110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:37.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.349+0000 7f143f84b640 1 --2- 172.21.15.132:0/2233250099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1438109820 0x7f1438110050 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1430009ca0 tx=0x7f143002ecd0 comp rx=0 tx=0).stop 2024-07-26T12:37:37.358 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.349+0000 7f143f84b640 1 -- 172.21.15.132:0/2233250099 shutdown_connections 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.349+0000 7f143f84b640 1 --2- 172.21.15.132:0/2233250099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1438109820 0x7f1438110050 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.349+0000 7f143f84b640 1 --2- 172.21.15.132:0/2233250099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1438108ed0 0x7f14381092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.349+0000 7f143f84b640 1 --2- 172.21.15.132:0/2233250099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1438103590 0x7f14381039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.349+0000 7f143f84b640 1 -- 172.21.15.132:0/2233250099 >> 172.21.15.132:0/2233250099 conn(0x7f143807b280 msgr2=0x7f143807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.349+0000 7f143f84b640 1 -- 172.21.15.132:0/2233250099 shutdown_connections 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.349+0000 7f143f84b640 1 -- 172.21.15.132:0/2233250099 wait complete. 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143f84b640 1 Processor -- start 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143f84b640 1 -- start start 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143f84b640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1438103590 0x7f1438197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143f84b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1438108ed0 0x7f1438198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143f84b640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1438109820 0x7f143819d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:37:37.359 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143f84b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1438111300 con 0x7f1438108ed0 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143f84b640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1438111180 con 0x7f1438103590 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143f84b640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1438111480 con 0x7f1438109820 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143dde8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1438109820 0x7f143819d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143dde8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1438109820 0x7f143819d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59432/0 (socket says 172.21.15.132:59432) 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.350+0000 7f143dde8640 1 -- 172.21.15.132:0/2808366581 learned_addr learned my addr 172.21.15.132:0/2808366581 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f143d5e7640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1438103590 0x7f1438197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f143dde8640 1 -- 172.21.15.132:0/2808366581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1438103590 msgr2=0x7f1438197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f143dde8640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1438103590 0x7f1438197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f143dde8640 1 -- 172.21.15.132:0/2808366581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1438108ed0 msgr2=0x7f1438198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f143dde8640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1438108ed0 0x7f1438198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:37.360 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f143dde8640 1 -- 172.21.15.132:0/2808366581 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f143819dc20 con 0x7f1438109820 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f143dde8640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1438109820 0x7f143819d4f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1430009dc0 tx=0x7f1430003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f142e7fc640 1 -- 172.21.15.132:0/2808366581 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143003c040 con 0x7f1438109820 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f142e7fc640 1 -- 172.21.15.132:0/2808366581 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1430047070 con 0x7f1438109820 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f143f84b640 1 -- 172.21.15.132:0/2808366581 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f143819deb0 con 0x7f1438109820 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f142e7fc640 1 -- 172.21.15.132:0/2808366581 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1430037960 con 0x7f1438109820 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.351+0000 7f143f84b640 1 -- 172.21.15.132:0/2808366581 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14381a4670 con 0x7f1438109820 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.352+0000 7f143f84b640 1 -- 172.21.15.132:0/2808366581 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14381039a0 con 0x7f1438109820 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.355+0000 7f142e7fc640 1 -- 172.21.15.132:0/2808366581 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f143002f9d0 con 0x7f1438109820 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.355+0000 7f142e7fc640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14240b3c10 0x7f14240b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.355+0000 7f142e7fc640 1 -- 172.21.15.132:0/2808366581 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2524+0+0 (secure 0 0 0) 0x7f1430107c80 con 0x7f1438109820 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.356+0000 7f143d5e7640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14240b3c10 0x7f14240b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:37:37.361 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.357+0000 7f143d5e7640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14240b3c10 0x7f14240b6120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1434009990 tx=0x7f1434009110 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:37:37.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.359+0000 7f142e7fc640 1 -- 172.21.15.132:0/2808366581 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f14300d0a50 con 0x7f1438109820 2024-07-26T12:37:37.546 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:37.545+0000 7f143f84b640 1 -- 172.21.15.132:0/2808366581 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1438063230 con 0x7f14240b3c10 2024-07-26T12:37:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:37 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi112 2024-07-26T12:37:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:37 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 17088M 2024-07-26T12:37:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:37 smithi132 ceph-mon[37580]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:37:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:37:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:37 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi112 2024-07-26T12:37:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:37 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 17088M 2024-07-26T12:37:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:37 smithi112 ceph-mon[37501]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:37:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:37:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:37 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi112 2024-07-26T12:37:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:37 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 17088M 2024-07-26T12:37:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:37 smithi107 ceph-mon[32659]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-07-26T12:37:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:37:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:37:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:38.890 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:38 smithi132 ceph-mon[37580]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:37:38.890 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:38 smithi132 ceph-mon[37580]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T12:37:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:38 smithi112 ceph-mon[37501]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:37:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:38 smithi112 ceph-mon[37501]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T12:37:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:38 smithi107 ceph-mon[32659]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:37:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:38 smithi107 ceph-mon[32659]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T12:37:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:39 smithi132 ceph-mon[37580]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:39 smithi112 ceph-mon[37501]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:39 smithi107 ceph-mon[32659]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:40 smithi132 ceph-mon[37580]: from='client.? 172.21.15.132:0/1946887363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "538267b7-07cf-4be0-95bf-bf9e8a4005f3"}]: dispatch 2024-07-26T12:37:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:40 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "538267b7-07cf-4be0-95bf-bf9e8a4005f3"}]: dispatch 2024-07-26T12:37:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:40 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "538267b7-07cf-4be0-95bf-bf9e8a4005f3"}]': finished 2024-07-26T12:37:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:40 smithi132 ceph-mon[37580]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T12:37:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:40 smithi132 ceph-mon[37580]: from='client.? 172.21.15.132:0/1814161829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:37:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:40 smithi112 ceph-mon[37501]: from='client.? 172.21.15.132:0/1946887363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "538267b7-07cf-4be0-95bf-bf9e8a4005f3"}]: dispatch 2024-07-26T12:37:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:40 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "538267b7-07cf-4be0-95bf-bf9e8a4005f3"}]: dispatch 2024-07-26T12:37:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:40 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "538267b7-07cf-4be0-95bf-bf9e8a4005f3"}]': finished 2024-07-26T12:37:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:40 smithi112 ceph-mon[37501]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T12:37:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:40 smithi112 ceph-mon[37501]: from='client.? 172.21.15.132:0/1814161829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:37:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:40 smithi107 ceph-mon[32659]: from='client.? 172.21.15.132:0/1946887363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "538267b7-07cf-4be0-95bf-bf9e8a4005f3"}]: dispatch 2024-07-26T12:37:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:40 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "538267b7-07cf-4be0-95bf-bf9e8a4005f3"}]: dispatch 2024-07-26T12:37:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:40 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "538267b7-07cf-4be0-95bf-bf9e8a4005f3"}]': finished 2024-07-26T12:37:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:40 smithi107 ceph-mon[32659]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T12:37:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:40 smithi107 ceph-mon[32659]: from='client.? 172.21.15.132:0/1814161829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:37:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:41 smithi112 ceph-mon[37501]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:41 smithi107 ceph-mon[32659]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:41 smithi132 ceph-mon[37580]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:43 smithi112 ceph-mon[37501]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:44.006 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:43 smithi132 ceph-mon[37580]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:43 smithi107 ceph-mon[32659]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:45.934 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:45 smithi132 ceph-mon[37580]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:45 smithi107 ceph-mon[32659]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:45 smithi112 ceph-mon[37501]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:47.958 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:47 smithi132 ceph-mon[37580]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:47.958 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T12:37:47.959 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:47 smithi107 ceph-mon[32659]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T12:37:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:48.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:47 smithi112 ceph-mon[37501]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:48.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T12:37:48.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:48 smithi107 ceph-mon[32659]: Deploying daemon osd.2 on smithi132 2024-07-26T12:37:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:48 smithi132 ceph-mon[37580]: Deploying daemon osd.2 on smithi132 2024-07-26T12:37:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:48 smithi112 ceph-mon[37501]: Deploying daemon osd.2 on smithi132 2024-07-26T12:37:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:49 smithi132 ceph-mon[37580]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:49 smithi107 ceph-mon[32659]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:49 smithi112 ceph-mon[37501]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:51 smithi107 ceph-mon[32659]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:51 smithi132 ceph-mon[37580]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:51 smithi112 ceph-mon[37501]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:53.105 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:53.106 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:53.106 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:53 smithi132 ceph-mon[37580]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:53 smithi107 ceph-mon[32659]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:53 smithi112 ceph-mon[37501]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:55.134 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:55.135 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:55.135 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:55.135 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:55.135 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:55.135 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:54 smithi132 ceph-mon[37580]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:55.195 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.194+0000 7f142e7fc640 1 -- 172.21.15.132:0/2808366581 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1438063230 con 0x7f14240b3c10 2024-07-26T12:37:55.195 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 2 on host 'smithi132' 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 -- 172.21.15.132:0/2808366581 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14240b3c10 msgr2=0x7f14240b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14240b3c10 0x7f14240b6120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1434009990 tx=0x7f1434009110 comp rx=0 tx=0).stop 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 -- 172.21.15.132:0/2808366581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1438109820 msgr2=0x7f143819d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1438109820 0x7f143819d4f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1430009dc0 tx=0x7f1430003f90 comp rx=0 tx=0).stop 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 -- 172.21.15.132:0/2808366581 shutdown_connections 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14240b3c10 0x7f14240b6120 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1438109820 0x7f143819d4f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1438108ed0 0x7f1438198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 --2- 172.21.15.132:0/2808366581 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1438103590 0x7f1438197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 -- 172.21.15.132:0/2808366581 >> 172.21.15.132:0/2808366581 conn(0x7f143807b280 msgr2=0x7f1438106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:37:55.198 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 -- 172.21.15.132:0/2808366581 shutdown_connections 2024-07-26T12:37:55.199 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:37:55.196+0000 7f143f84b640 1 -- 172.21.15.132:0/2808366581 wait complete. 2024-07-26T12:37:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:54 smithi112 ceph-mon[37501]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:55.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:55.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:55.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:55.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:55.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:54 smithi107 ceph-mon[32659]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:55.848 DEBUG:teuthology.orchestra.run.smithi132:osd.2> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.2.service 2024-07-26T12:37:55.850 INFO:tasks.cephadm:Deploying osd.3 on smithi107 with /dev/vg_nvme/lv_3... 2024-07-26T12:37:55.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T12:37:56.083 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:56.083 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:56.084 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:56.084 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:55 smithi132 ceph-mon[37580]: from='osd.2 [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T12:37:56.084 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:55 smithi132 ceph-mon[37580]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T12:37:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:55 smithi112 ceph-mon[37501]: from='osd.2 [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T12:37:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:55 smithi112 ceph-mon[37501]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T12:37:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:55 smithi107 ceph-mon[32659]: from='osd.2 [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T12:37:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:55 smithi107 ceph-mon[32659]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T12:37:56.301 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:37:57.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:57 smithi107 ceph-mon[32659]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T12:37:57.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:57 smithi107 ceph-mon[32659]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T12:37:57.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:57.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:57 smithi107 ceph-mon[32659]: from='osd.2 [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:37:57.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:57 smithi107 ceph-mon[32659]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:37:57.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:57 smithi107 ceph-mon[32659]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:57 smithi132 ceph-mon[37580]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T12:37:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:57 smithi132 ceph-mon[37580]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T12:37:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:57 smithi132 ceph-mon[37580]: from='osd.2 [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:37:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:57 smithi132 ceph-mon[37580]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:37:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:57 smithi132 ceph-mon[37580]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:57.561 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jul 26 12:37:57 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-2[43982]: 2024-07-26T12:37:57.199+0000 7f819c1f9640 -1 osd.2 0 waiting for initial osdmap 2024-07-26T12:37:57.561 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jul 26 12:37:57 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-2[43982]: 2024-07-26T12:37:57.204+0000 7f8197ff7640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T12:37:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:57 smithi112 ceph-mon[37501]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T12:37:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:57 smithi112 ceph-mon[37501]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T12:37:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:57 smithi112 ceph-mon[37501]: from='osd.2 [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:37:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:57 smithi112 ceph-mon[37501]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:37:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:57 smithi112 ceph-mon[37501]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:58.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:58 smithi107 ceph-mon[32659]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:37:58.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:58.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:58 smithi107 ceph-mon[32659]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T12:37:58.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:58.544 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:58 smithi132 ceph-mon[37580]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:37:58.544 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:58.544 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:58 smithi132 ceph-mon[37580]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T12:37:58.544 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:58 smithi112 ceph-mon[37501]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:37:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:58 smithi112 ceph-mon[37501]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T12:37:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:58.752 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T12:37:58.907 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T12:37:58.943 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-07-26T12:37:58.943 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-07-26T12:37:58.944 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.030154 s, 348 MB/s 2024-07-26T12:37:58.944 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-07-26T12:37:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:37:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:37:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: osd.2 [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] boot 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi132 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 17088M 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:37:59.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:37:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: osd.2 [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] boot 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi132 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 17088M 2024-07-26T12:37:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:37:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:37:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:37:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: osd.2 [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] boot 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi132 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 17088M 2024-07-26T12:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:59.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:59.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:37:59.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:37:59.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:37:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:37:59.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_3 2024-07-26T12:38:00.224 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:38:00.889 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T12:38:00.889 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:00 smithi112 ceph-mon[37501]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T12:38:00.889 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:38:00.889 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:38:00.979 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T12:38:00.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:00 smithi132 ceph-mon[37580]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T12:38:00.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:38:00.980 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:38:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T12:38:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:00 smithi107 ceph-mon[32659]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T12:38:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:38:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T12:38:01.043 INFO:journalctl@ceph.osd.0.smithi107.stdout:Jul 26 12:38:00 smithi107 sudo[54646]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T12:38:01.043 INFO:journalctl@ceph.osd.0.smithi107.stdout:Jul 26 12:38:00 smithi107 sudo[54646]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.043 INFO:journalctl@ceph.osd.0.smithi107.stdout:Jul 26 12:38:00 smithi107 sudo[54646]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.044 INFO:journalctl@ceph.osd.0.smithi107.stdout:Jul 26 12:38:00 smithi107 sudo[54646]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.044 INFO:journalctl@ceph.osd.0.smithi107.stdout:Jul 26 12:38:00 smithi107 sudo[54655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T12:38:01.044 INFO:journalctl@ceph.osd.0.smithi107.stdout:Jul 26 12:38:00 smithi107 sudo[54655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.044 INFO:journalctl@ceph.osd.0.smithi107.stdout:Jul 26 12:38:00 smithi107 sudo[54655]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.044 INFO:journalctl@ceph.osd.0.smithi107.stdout:Jul 26 12:38:00 smithi107 sudo[54655]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.200 INFO:journalctl@ceph.osd.1.smithi112.stdout:Jul 26 12:38:00 smithi112 sudo[47347]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T12:38:01.200 INFO:journalctl@ceph.osd.1.smithi112.stdout:Jul 26 12:38:00 smithi112 sudo[47347]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.201 INFO:journalctl@ceph.osd.1.smithi112.stdout:Jul 26 12:38:00 smithi112 sudo[47347]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.201 INFO:journalctl@ceph.osd.1.smithi112.stdout:Jul 26 12:38:00 smithi112 sudo[47347]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.201 INFO:journalctl@ceph.osd.1.smithi112.stdout:Jul 26 12:38:00 smithi112 sudo[47350]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T12:38:01.201 INFO:journalctl@ceph.osd.1.smithi112.stdout:Jul 26 12:38:00 smithi112 sudo[47350]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.201 INFO:journalctl@ceph.osd.1.smithi112.stdout:Jul 26 12:38:00 smithi112 sudo[47350]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.201 INFO:journalctl@ceph.osd.1.smithi112.stdout:Jul 26 12:38:00 smithi112 sudo[47350]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.310 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jul 26 12:38:00 smithi132 sudo[48031]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T12:38:01.310 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jul 26 12:38:00 smithi132 sudo[48031]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.310 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jul 26 12:38:00 smithi132 sudo[48031]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.310 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jul 26 12:38:00 smithi132 sudo[48031]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.310 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48034]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T12:38:01.310 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48034]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.310 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48034]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.311 INFO:journalctl@ceph.osd.2.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48034]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48037]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T12:38:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48037]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48037]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48037]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48040]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T12:38:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48040]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48040]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 sudo[48040]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.406 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 sudo[54658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T12:38:01.406 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 sudo[54658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 sudo[54658]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 sudo[54658]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 sudo[54665]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T12:38:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 sudo[54665]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 sudo[54665]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 sudo[54665]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.641 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 sudo[47353]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T12:38:01.642 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 sudo[47353]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.642 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 sudo[47353]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.642 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 sudo[47353]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.642 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 sudo[47356]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T12:38:01.642 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 sudo[47356]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T12:38:01.642 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 sudo[47356]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T12:38:01.642 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 sudo[47356]: pam_unix(sudo:session): session closed for user root 2024-07-26T12:38:01.642 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:01.642 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:38:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:38:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T12:38:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T12:38:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:38:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:38:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T12:38:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:38:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:38:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:38:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:01 smithi107 ceph-mon[32659]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T12:38:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T12:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:38:01.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:38:01.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:38:01.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T12:38:01.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:38:01.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:38:01.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:38:01.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:01 smithi112 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T12:38:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:38:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T12:38:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:38:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T12:38:02.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T12:38:02.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T12:38:02.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T12:38:02.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:01 smithi132 ceph-mon[37580]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T12:38:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.411+0000 7fc9e1000640 1 -- 172.21.15.107:0/3515712980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9dc075b30 msgr2=0x7fc9dc10f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:02.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.411+0000 7fc9e1000640 1 --2- 172.21.15.107:0/3515712980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9dc075b30 0x7fc9dc10f400 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009900 tx=0x7fc9c802a400 comp rx=0 tx=0).stop 2024-07-26T12:38:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.411+0000 7fc9e1000640 1 -- 172.21.15.107:0/3515712980 shutdown_connections 2024-07-26T12:38:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.411+0000 7fc9e1000640 1 --2- 172.21.15.107:0/3515712980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9dc10f940 0x7fc9dc111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.411+0000 7fc9e1000640 1 --2- 172.21.15.107:0/3515712980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9dc075b30 0x7fc9dc10f400 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.411+0000 7fc9e1000640 1 --2- 172.21.15.107:0/3515712980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9dc0751e0 0x7fc9dc0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.411+0000 7fc9e1000640 1 -- 172.21.15.107:0/3515712980 >> 172.21.15.107:0/3515712980 conn(0x7fc9dc0fee70 msgr2=0x7fc9dc1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:38:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.412+0000 7fc9e1000640 1 -- 172.21.15.107:0/3515712980 shutdown_connections 2024-07-26T12:38:02.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.412+0000 7fc9e1000640 1 -- 172.21.15.107:0/3515712980 wait complete. 2024-07-26T12:38:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.413+0000 7fc9e1000640 1 Processor -- start 2024-07-26T12:38:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.413+0000 7fc9e1000640 1 -- start start 2024-07-26T12:38:02.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.414+0000 7fc9e1000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9dc0751e0 0x7fc9dc19bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.414+0000 7fc9e1000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9dc075b30 0x7fc9dc19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.414+0000 7fc9e1000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9dc10f940 0x7fc9dc1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.414+0000 7fc9e1000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc1154e0 con 0x7fc9dc10f940 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.414+0000 7fc9e1000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc115360 con 0x7fc9dc075b30 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.414+0000 7fc9e1000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc115660 con 0x7fc9dc0751e0 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.414+0000 7fc9db400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9dc0751e0 0x7fc9dc19bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.414+0000 7fc9db400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9dc0751e0 0x7fc9dc19bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:50502/0 (socket says 172.21.15.107:50502) 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.414+0000 7fc9db400640 1 -- 172.21.15.107:0/1950935757 learned_addr learned my addr 172.21.15.107:0/1950935757 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.414+0000 7fc9dbe00640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9dc10f940 0x7fc9dc1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9daa00640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9dc075b30 0x7fc9dc19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9db400640 1 -- 172.21.15.107:0/1950935757 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9dc075b30 msgr2=0x7fc9dc19c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:02.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9db400640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9dc075b30 0x7fc9dc19c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9db400640 1 -- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9dc10f940 msgr2=0x7fc9dc1a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9db400640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9dc10f940 0x7fc9dc1a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9db400640 1 -- 172.21.15.107:0/1950935757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9dc1a1ee0 con 0x7fc9dc0751e0 2024-07-26T12:38:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9dbe00640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9dc10f940 0x7fc9dc1a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:38:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9db400640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9dc0751e0 0x7fc9dc19bf70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00ea60 tx=0x7fc9cc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:38:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9d3e00640 1 -- 172.21.15.107:0/1950935757 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc00cde0 con 0x7fc9dc0751e0 2024-07-26T12:38:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9d3e00640 1 -- 172.21.15.107:0/1950935757 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9cc0035f0 con 0x7fc9dc0751e0 2024-07-26T12:38:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9e1000640 1 -- 172.21.15.107:0/1950935757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9dc1a21d0 con 0x7fc9dc0751e0 2024-07-26T12:38:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.415+0000 7fc9d3e00640 1 -- 172.21.15.107:0/1950935757 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc010690 con 0x7fc9dc0751e0 2024-07-26T12:38:02.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.416+0000 7fc9e1000640 1 -- 172.21.15.107:0/1950935757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9dc1a8930 con 0x7fc9dc0751e0 2024-07-26T12:38:02.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.417+0000 7fc9e1000640 1 -- 172.21.15.107:0/1950935757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9dc0755f0 con 0x7fc9dc0751e0 2024-07-26T12:38:02.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.419+0000 7fc9d3e00640 1 -- 172.21.15.107:0/1950935757 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fc9cc0037a0 con 0x7fc9dc0751e0 2024-07-26T12:38:02.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.421+0000 7fc9d3e00640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9c00b3c90 0x7fc9c00b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:02.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.421+0000 7fc9d3e00640 1 -- 172.21.15.107:0/1950935757 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 3452+0+0 (secure 0 0 0) 0x7fc9cc0e2c50 con 0x7fc9dc0751e0 2024-07-26T12:38:02.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.421+0000 7fc9daa00640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9c00b3c90 0x7fc9c00b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:02.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.422+0000 7fc9daa00640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9c00b3c90 0x7fc9c00b61a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000c00 tx=0x7fc9c8031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:38:02.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.425+0000 7fc9d3e00640 1 -- 172.21.15.107:0/1950935757 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc9cc0ab5e0 con 0x7fc9dc0751e0 2024-07-26T12:38:02.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.638+0000 7fc9e1000640 1 -- 172.21.15.107:0/1950935757 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc9dc109c30 con 0x7fc9c00b3c90 2024-07-26T12:38:02.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:02.661+0000 7fc9d3e00640 1 -- 172.21.15.107:0/1950935757 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fc9cc004000 con 0x7fc9dc0751e0 2024-07-26T12:38:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:02 smithi112 ceph-mon[37501]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T12:38:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:02 smithi107 ceph-mon[32659]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T12:38:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:02 smithi132 ceph-mon[37580]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T12:38:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:03 smithi112 ceph-mon[37501]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:03 smithi112 ceph-mon[37501]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T12:38:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:03 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:38:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:03 smithi112 ceph-mon[37501]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:38:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:38:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:03 smithi112 ceph-mon[37501]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T12:38:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:38:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:03 smithi107 ceph-mon[32659]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:03 smithi107 ceph-mon[32659]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T12:38:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:03 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:38:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:03 smithi107 ceph-mon[32659]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:38:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:38:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:03 smithi107 ceph-mon[32659]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T12:38:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:38:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:03 smithi132 ceph-mon[37580]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:03 smithi132 ceph-mon[37580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T12:38:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:03 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:38:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:03 smithi132 ceph-mon[37580]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:38:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:38:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:03 smithi132 ceph-mon[37580]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T12:38:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:38:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:05.937 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:05 smithi107 ceph-mon[32659]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:05 smithi112 ceph-mon[37501]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:05 smithi132 ceph-mon[37580]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:06 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3392528186' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "914bea43-b9ac-4bed-817e-6af4d1d0aa2e"}]: dispatch 2024-07-26T12:38:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:06 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3392528186' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "914bea43-b9ac-4bed-817e-6af4d1d0aa2e"}]': finished 2024-07-26T12:38:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:06 smithi112 ceph-mon[37501]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T12:38:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:06 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3392528186' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "914bea43-b9ac-4bed-817e-6af4d1d0aa2e"}]: dispatch 2024-07-26T12:38:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:06 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3392528186' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "914bea43-b9ac-4bed-817e-6af4d1d0aa2e"}]': finished 2024-07-26T12:38:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:06 smithi107 ceph-mon[32659]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T12:38:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:06 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3392528186' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "914bea43-b9ac-4bed-817e-6af4d1d0aa2e"}]: dispatch 2024-07-26T12:38:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:06 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3392528186' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "914bea43-b9ac-4bed-817e-6af4d1d0aa2e"}]': finished 2024-07-26T12:38:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:06 smithi132 ceph-mon[37580]: osdmap e23: 4 total, 3 up, 4 in 2024-07-26T12:38:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:07 smithi112 ceph-mon[37501]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:07 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/693689986' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:38:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:07 smithi107 ceph-mon[32659]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:07 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/693689986' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:38:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:07 smithi132 ceph-mon[37580]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:07 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/693689986' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:38:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:09 smithi112 ceph-mon[37501]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:09 smithi107 ceph-mon[32659]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:09 smithi132 ceph-mon[37580]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:11 smithi112 ceph-mon[37501]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:11 smithi107 ceph-mon[32659]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:11 smithi132 ceph-mon[37580]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:13 smithi112 ceph-mon[37501]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:14.000 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:13 smithi107 ceph-mon[32659]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:13 smithi132 ceph-mon[37580]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:15 smithi107 ceph-mon[32659]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T12:38:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:15 smithi112 ceph-mon[37501]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T12:38:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:15 smithi132 ceph-mon[37580]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T12:38:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:16 smithi112 ceph-mon[37501]: Deploying daemon osd.3 on smithi107 2024-07-26T12:38:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:16 smithi107 ceph-mon[32659]: Deploying daemon osd.3 on smithi107 2024-07-26T12:38:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:16 smithi132 ceph-mon[37580]: Deploying daemon osd.3 on smithi107 2024-07-26T12:38:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:17 smithi112 ceph-mon[37501]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:17 smithi107 ceph-mon[32659]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:17 smithi132 ceph-mon[37580]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:19 smithi112 ceph-mon[37501]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:19 smithi107 ceph-mon[32659]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:19 smithi132 ceph-mon[37580]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:21 smithi112 ceph-mon[37501]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:21 smithi107 ceph-mon[32659]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:21 smithi132 ceph-mon[37580]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:24.003 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:23 smithi107 ceph-mon[32659]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:23 smithi132 ceph-mon[37580]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:23 smithi112 ceph-mon[37501]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:24.928 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:24.928 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:24.928 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:25 smithi107 ceph-mon[32659]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:26.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:25 smithi132 ceph-mon[37580]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:25 smithi112 ceph-mon[37501]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:27 smithi107 ceph-mon[32659]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:27 smithi132 ceph-mon[37580]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:27 smithi112 ceph-mon[37501]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:28.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:28.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:28.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:28.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:28 smithi107 ceph-mon[32659]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T12:38:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:28 smithi132 ceph-mon[37580]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T12:38:29.109 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 3 on host 'smithi107' 2024-07-26T12:38:29.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.107+0000 7fc9d3e00640 1 -- 172.21.15.107:0/1950935757 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc9dc109c30 con 0x7fc9c00b3c90 2024-07-26T12:38:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.111+0000 7fc9e1000640 1 -- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9c00b3c90 msgr2=0x7fc9c00b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.111+0000 7fc9e1000640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9c00b3c90 0x7fc9c00b61a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000c00 tx=0x7fc9c8031400 comp rx=0 tx=0).stop 2024-07-26T12:38:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.112+0000 7fc9e1000640 1 -- 172.21.15.107:0/1950935757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9dc0751e0 msgr2=0x7fc9dc19bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:29.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.112+0000 7fc9e1000640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9dc0751e0 0x7fc9dc19bf70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00ea60 tx=0x7fc9cc00ef20 comp rx=0 tx=0).stop 2024-07-26T12:38:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.112+0000 7fc9e1000640 1 -- 172.21.15.107:0/1950935757 shutdown_connections 2024-07-26T12:38:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.112+0000 7fc9e1000640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9c00b3c90 0x7fc9c00b61a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.112+0000 7fc9e1000640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9dc10f940 0x7fc9dc1a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.112+0000 7fc9e1000640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc9dc075b30 0x7fc9dc19c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.112+0000 7fc9e1000640 1 --2- 172.21.15.107:0/1950935757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9dc0751e0 0x7fc9dc19bf70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.112+0000 7fc9e1000640 1 -- 172.21.15.107:0/1950935757 >> 172.21.15.107:0/1950935757 conn(0x7fc9dc0fee70 msgr2=0x7fc9dc109520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:38:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.112+0000 7fc9e1000640 1 -- 172.21.15.107:0/1950935757 shutdown_connections 2024-07-26T12:38:29.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:38:29.112+0000 7fc9e1000640 1 -- 172.21.15.107:0/1950935757 wait complete. 2024-07-26T12:38:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:28 smithi112 ceph-mon[37501]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T12:38:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:29 smithi107 ceph-mon[32659]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:29 smithi107 ceph-mon[32659]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T12:38:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:29 smithi107 ceph-mon[32659]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T12:38:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:29 smithi107 ceph-mon[32659]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:38:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:29 smithi132 ceph-mon[37580]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:29 smithi132 ceph-mon[37580]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T12:38:30.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:29 smithi132 ceph-mon[37580]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T12:38:30.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:29 smithi132 ceph-mon[37580]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:38:30.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:30.103 DEBUG:teuthology.orchestra.run.smithi107:osd.3> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.3.service 2024-07-26T12:38:30.111 INFO:tasks.cephadm:Deploying osd.4 on smithi112 with /dev/vg_nvme/lv_3... 2024-07-26T12:38:30.112 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T12:38:30.142 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:29 smithi112 ceph-mon[37501]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:30.142 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:30.142 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:30.142 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:30.142 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:29 smithi112 ceph-mon[37501]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T12:38:30.142 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:29 smithi112 ceph-mon[37501]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T12:38:30.143 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:29 smithi112 ceph-mon[37501]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:38:30.143 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:30.369 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T12:38:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:31 smithi107 ceph-mon[32659]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:38:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:31 smithi107 ceph-mon[32659]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T12:38:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:31 smithi107 ceph-mon[32659]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:31 smithi107 ceph-mon[32659]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' 2024-07-26T12:38:31.294 INFO:journalctl@ceph.osd.3.smithi107.stdout:Jul 26 12:38:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-3[56947]: 2024-07-26T12:38:30.997+0000 7f3d0e200640 -1 osd.3 0 waiting for initial osdmap 2024-07-26T12:38:31.294 INFO:journalctl@ceph.osd.3.smithi107.stdout:Jul 26 12:38:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-3[56947]: 2024-07-26T12:38:31.015+0000 7f3d04200640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T12:38:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:31 smithi112 ceph-mon[37501]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:38:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:31 smithi112 ceph-mon[37501]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T12:38:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:31 smithi112 ceph-mon[37501]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:31 smithi112 ceph-mon[37501]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' 2024-07-26T12:38:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:31 smithi132 ceph-mon[37580]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:38:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:31 smithi132 ceph-mon[37580]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T12:38:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:31 smithi132 ceph-mon[37580]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-07-26T12:38:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:31 smithi132 ceph-mon[37580]: from='osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989]' entity='osd.3' 2024-07-26T12:38:32.174 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T12:38:32.280 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T12:38:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:32 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:38:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:32 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:38:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:32 smithi107 ceph-mon[32659]: osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] boot 2024-07-26T12:38:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:32 smithi107 ceph-mon[32659]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T12:38:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:32 smithi107 ceph-mon[32659]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T12:38:32.293 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-07-26T12:38:32.293 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-07-26T12:38:32.294 INFO:teuthology.orchestra.run.smithi112.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115753 s, 906 MB/s 2024-07-26T12:38:32.294 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-07-26T12:38:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:32 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:38:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:32 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:38:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:32 smithi112 ceph-mon[37501]: osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] boot 2024-07-26T12:38:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:32 smithi112 ceph-mon[37501]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T12:38:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:32 smithi112 ceph-mon[37501]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T12:38:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:32 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:38:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:32 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:38:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T12:38:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:32 smithi132 ceph-mon[37580]: osd.3 [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] boot 2024-07-26T12:38:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:32 smithi132 ceph-mon[37580]: osdmap e26: 4 total, 4 up, 4 in 2024-07-26T12:38:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:32 smithi132 ceph-mon[37580]: osdmap e27: 4 total, 4 up, 4 in 2024-07-26T12:38:33.010 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_3 2024-07-26T12:38:33.306 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T12:38:33.536 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:33 smithi112 ceph-mon[37501]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T12:38:33.537 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:33 smithi112 ceph-mon[37501]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:33 smithi107 ceph-mon[32659]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T12:38:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:33 smithi107 ceph-mon[32659]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:33 smithi132 ceph-mon[37580]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T12:38:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:33 smithi132 ceph-mon[37580]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:34.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.811+0000 7f0a232a8640 1 -- 172.21.15.112:0/3498853387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a1c103590 msgr2=0x7f0a1c1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:34.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.811+0000 7f0a232a8640 1 --2- 172.21.15.112:0/3498853387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a1c103590 0x7f0a1c1039a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c009900 tx=0x7f0a0c02a410 comp rx=0 tx=0).stop 2024-07-26T12:38:34.815 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.811+0000 7f0a232a8640 1 -- 172.21.15.112:0/3498853387 shutdown_connections 2024-07-26T12:38:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.811+0000 7f0a232a8640 1 --2- 172.21.15.112:0/3498853387 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c109820 0x7f0a1c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.811+0000 7f0a232a8640 1 --2- 172.21.15.112:0/3498853387 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a1c108ed0 0x7f0a1c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.811+0000 7f0a232a8640 1 --2- 172.21.15.112:0/3498853387 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a1c103590 0x7f0a1c1039a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.811+0000 7f0a232a8640 1 -- 172.21.15.112:0/3498853387 >> 172.21.15.112:0/3498853387 conn(0x7f0a1c07b280 msgr2=0x7f0a1c07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:38:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.812+0000 7f0a232a8640 1 -- 172.21.15.112:0/3498853387 shutdown_connections 2024-07-26T12:38:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.812+0000 7f0a232a8640 1 -- 172.21.15.112:0/3498853387 wait complete. 2024-07-26T12:38:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.812+0000 7f0a232a8640 1 Processor -- start 2024-07-26T12:38:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.812+0000 7f0a232a8640 1 -- start start 2024-07-26T12:38:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.812+0000 7f0a232a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c103590 0x7f0a1c199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:34.816 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.812+0000 7f0a232a8640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a1c108ed0 0x7f0a1c19a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:34.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.812+0000 7f0a232a8640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a1c109820 0x7f0a1c19f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:34.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.812+0000 7f0a232a8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c113560 con 0x7f0a1c103590 2024-07-26T12:38:34.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.812+0000 7f0a232a8640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c1133e0 con 0x7f0a1c108ed0 2024-07-26T12:38:34.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.812+0000 7f0a232a8640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c1136e0 con 0x7f0a1c109820 2024-07-26T12:38:34.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a20843640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a1c108ed0 0x7f0a1c19a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:34.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a20843640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a1c108ed0 0x7f0a1c19a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:56086/0 (socket says 172.21.15.112:56086) 2024-07-26T12:38:34.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a20843640 1 -- 172.21.15.112:0/1139245060 learned_addr learned my addr 172.21.15.112:0/1139245060 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-26T12:38:34.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a21845640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a1c109820 0x7f0a1c19f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:34.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a20843640 1 -- 172.21.15.112:0/1139245060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a1c109820 msgr2=0x7f0a1c19f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:34.817 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a20843640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a1c109820 0x7f0a1c19f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:34.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a20843640 1 -- 172.21.15.112:0/1139245060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c103590 msgr2=0x7f0a1c199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:34.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a20843640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c103590 0x7f0a1c199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:34.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a20843640 1 -- 172.21.15.112:0/1139245060 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1c19fec0 con 0x7f0a1c108ed0 2024-07-26T12:38:34.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a20843640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a1c108ed0 0x7f0a1c19a4f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800d750 tx=0x7f0a1800dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:38:34.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a127fc640 1 -- 172.21.15.112:0/1139245060 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18017070 con 0x7f0a1c108ed0 2024-07-26T12:38:34.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a232a8640 1 -- 172.21.15.112:0/1139245060 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1c1a01b0 con 0x7f0a1c108ed0 2024-07-26T12:38:34.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a127fc640 1 -- 172.21.15.112:0/1139245060 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a180054d0 con 0x7f0a1c108ed0 2024-07-26T12:38:34.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a127fc640 1 -- 172.21.15.112:0/1139245060 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a18004c30 con 0x7f0a1c108ed0 2024-07-26T12:38:34.818 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.813+0000 7f0a232a8640 1 -- 172.21.15.112:0/1139245060 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a1c1a6910 con 0x7f0a1c108ed0 2024-07-26T12:38:34.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.814+0000 7f0a232a8640 1 -- 172.21.15.112:0/1139245060 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a1c1039a0 con 0x7f0a1c108ed0 2024-07-26T12:38:34.819 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.817+0000 7f0a127fc640 1 -- 172.21.15.112:0/1139245060 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f0a18004dd0 con 0x7f0a1c108ed0 2024-07-26T12:38:34.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.817+0000 7f0a127fc640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0a080b3ca0 0x7f0a080b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:34.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.818+0000 7f0a127fc640 1 -- 172.21.15.112:0/1139245060 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3884+0+0 (secure 0 0 0) 0x7f0a180e2ab0 con 0x7f0a1c108ed0 2024-07-26T12:38:34.820 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.818+0000 7f0a21044640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0a080b3ca0 0x7f0a080b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:34.821 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.819+0000 7f0a21044640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0a080b3ca0 0x7f0a080b61b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c02a930 tx=0x7f0a0c031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:38:34.822 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:34.821+0000 7f0a127fc640 1 -- 172.21.15.112:0/1139245060 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0a180ab1b0 con 0x7f0a1c108ed0 2024-07-26T12:38:35.006 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:35.004+0000 7f0a232a8640 1 -- 172.21.15.112:0/1139245060 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0a1c063230 con 0x7f0a080b3ca0 2024-07-26T12:38:35.692 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi107 2024-07-26T12:38:35.692 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.692 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.692 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.692 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.692 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.692 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.693 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 8544M 2024-07-26T12:38:35.693 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:35.693 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.693 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:35.693 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:35.693 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.693 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:38:35.693 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:38:35.693 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi107 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 8544M 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:35.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:35.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:38:35.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:38:35.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi107 2024-07-26T12:38:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:35.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 8544M 2024-07-26T12:38:35.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 516 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:35.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:35.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:35.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:35.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:38:35.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:38:35.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:36.646 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:36 smithi112 ceph-mon[37501]: from='client.24410 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:38:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:36 smithi132 ceph-mon[37580]: from='client.24410 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:38:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:36 smithi107 ceph-mon[32659]: from='client.24410 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:38:37.912 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:37 smithi112 ceph-mon[37501]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:37.913 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:37 smithi112 ceph-mon[37501]: from='client.? 172.21.15.112:0/655937127' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc83673b-3617-4245-bfe1-b61c9cbe2ab4"}]: dispatch 2024-07-26T12:38:37.913 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:37 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc83673b-3617-4245-bfe1-b61c9cbe2ab4"}]: dispatch 2024-07-26T12:38:37.913 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:37 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc83673b-3617-4245-bfe1-b61c9cbe2ab4"}]': finished 2024-07-26T12:38:37.913 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:37.913 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:37 smithi112 ceph-mon[37501]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T12:38:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:37 smithi107 ceph-mon[32659]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:37 smithi107 ceph-mon[32659]: from='client.? 172.21.15.112:0/655937127' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc83673b-3617-4245-bfe1-b61c9cbe2ab4"}]: dispatch 2024-07-26T12:38:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:37 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc83673b-3617-4245-bfe1-b61c9cbe2ab4"}]: dispatch 2024-07-26T12:38:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:37 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc83673b-3617-4245-bfe1-b61c9cbe2ab4"}]': finished 2024-07-26T12:38:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:37 smithi107 ceph-mon[32659]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T12:38:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:37 smithi132 ceph-mon[37580]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 121 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:37 smithi132 ceph-mon[37580]: from='client.? 172.21.15.112:0/655937127' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc83673b-3617-4245-bfe1-b61c9cbe2ab4"}]: dispatch 2024-07-26T12:38:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:37 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc83673b-3617-4245-bfe1-b61c9cbe2ab4"}]: dispatch 2024-07-26T12:38:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:37 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc83673b-3617-4245-bfe1-b61c9cbe2ab4"}]': finished 2024-07-26T12:38:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:37 smithi132 ceph-mon[37580]: osdmap e29: 5 total, 4 up, 5 in 2024-07-26T12:38:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:38 smithi112 ceph-mon[37501]: from='client.? 172.21.15.112:0/685117756' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:38:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:38 smithi107 ceph-mon[32659]: from='client.? 172.21.15.112:0/685117756' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:38:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:38 smithi132 ceph-mon[37580]: from='client.? 172.21.15.112:0/685117756' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:38:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:39 smithi112 ceph-mon[37501]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T12:38:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:39 smithi107 ceph-mon[32659]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T12:38:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:39 smithi132 ceph-mon[37580]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T12:38:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:41 smithi112 ceph-mon[37501]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-07-26T12:38:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:41 smithi107 ceph-mon[32659]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-07-26T12:38:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:41 smithi132 ceph-mon[37580]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-07-26T12:38:43.860 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:43 smithi112 ceph-mon[37501]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:38:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:43 smithi107 ceph-mon[32659]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:38:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:43 smithi132 ceph-mon[37580]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:38:44.906 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T12:38:44.906 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T12:38:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T12:38:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:45.921 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:45 smithi112 ceph-mon[37501]: Deploying daemon osd.4 on smithi112 2024-07-26T12:38:45.921 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:45 smithi112 ceph-mon[37501]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:38:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:45 smithi107 ceph-mon[32659]: Deploying daemon osd.4 on smithi112 2024-07-26T12:38:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:45 smithi107 ceph-mon[32659]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:38:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:45 smithi132 ceph-mon[37580]: Deploying daemon osd.4 on smithi112 2024-07-26T12:38:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:45 smithi132 ceph-mon[37580]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:38:47.920 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:47 smithi112 ceph-mon[37501]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:38:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:47 smithi107 ceph-mon[32659]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:38:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:47 smithi132 ceph-mon[37580]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:38:49.923 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:49 smithi112 ceph-mon[37501]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:49.924 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:49.924 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:49.924 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:49 smithi107 ceph-mon[32659]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:49 smithi132 ceph-mon[37580]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:51 smithi112 ceph-mon[37501]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:51 smithi107 ceph-mon[32659]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:51 smithi132 ceph-mon[37580]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:52.712 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.710+0000 7f0a127fc640 1 -- 172.21.15.112:0/1139245060 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0a1c063230 con 0x7f0a080b3ca0 2024-07-26T12:38:52.712 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 4 on host 'smithi112' 2024-07-26T12:38:52.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 -- 172.21.15.112:0/1139245060 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0a080b3ca0 msgr2=0x7f0a080b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:52.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0a080b3ca0 0x7f0a080b61b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c02a930 tx=0x7f0a0c031400 comp rx=0 tx=0).stop 2024-07-26T12:38:52.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 -- 172.21.15.112:0/1139245060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a1c108ed0 msgr2=0x7f0a1c19a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:52.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a1c108ed0 0x7f0a1c19a4f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0a1800d750 tx=0x7f0a1800dc10 comp rx=0 tx=0).stop 2024-07-26T12:38:52.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 -- 172.21.15.112:0/1139245060 shutdown_connections 2024-07-26T12:38:52.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0a080b3ca0 0x7f0a080b61b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:52.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0a1c109820 0x7f0a1c19f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:52.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0a1c108ed0 0x7f0a1c19a4f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:52.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 --2- 172.21.15.112:0/1139245060 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a1c103590 0x7f0a1c199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:52.715 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 -- 172.21.15.112:0/1139245060 >> 172.21.15.112:0/1139245060 conn(0x7f0a1c07b280 msgr2=0x7f0a1c106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:38:52.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 -- 172.21.15.112:0/1139245060 shutdown_connections 2024-07-26T12:38:52.716 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:38:52.713+0000 7f0a232a8640 1 -- 172.21.15.112:0/1139245060 wait complete. 2024-07-26T12:38:53.381 DEBUG:teuthology.orchestra.run.smithi112:osd.4> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.4.service 2024-07-26T12:38:53.384 INFO:tasks.cephadm:Deploying osd.5 on smithi132 with /dev/vg_nvme/lv_3... 2024-07-26T12:38:53.384 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T12:38:53.644 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:38:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:53 smithi112 ceph-mon[37501]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:53 smithi112 ceph-mon[37501]: from='osd.4 [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T12:38:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:53 smithi112 ceph-mon[37501]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T12:38:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.976 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:53 smithi132 ceph-mon[37580]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:53.976 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.976 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.976 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:53 smithi132 ceph-mon[37580]: from='osd.4 [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T12:38:53.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:53 smithi132 ceph-mon[37580]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T12:38:53.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:53.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:53 smithi107 ceph-mon[32659]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:53 smithi107 ceph-mon[32659]: from='osd.4 [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T12:38:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:53 smithi107 ceph-mon[32659]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T12:38:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:54.989 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:54 smithi112 ceph-mon[37501]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T12:38:54.989 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:54.989 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:54 smithi112 ceph-mon[37501]: from='osd.4 [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:38:54.990 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:54 smithi112 ceph-mon[37501]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T12:38:54.990 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:54 smithi112 ceph-mon[37501]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:38:54.990 INFO:journalctl@ceph.osd.4.smithi112.stdout:Jul 26 12:38:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-4[50124]: 2024-07-26T12:38:54.719+0000 7f57bbe66640 -1 osd.4 0 waiting for initial osdmap 2024-07-26T12:38:54.990 INFO:journalctl@ceph.osd.4.smithi112.stdout:Jul 26 12:38:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-4[50124]: 2024-07-26T12:38:54.725+0000 7f57b7c64640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T12:38:54.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:54 smithi132 ceph-mon[37580]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T12:38:54.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:54.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:54 smithi132 ceph-mon[37580]: from='osd.4 [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:38:54.991 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:54 smithi132 ceph-mon[37580]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T12:38:54.992 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:54 smithi132 ceph-mon[37580]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:38:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:54 smithi107 ceph-mon[32659]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T12:38:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:54 smithi107 ceph-mon[32659]: from='osd.4 [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:38:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:54 smithi107 ceph-mon[32659]: osdmap e30: 5 total, 4 up, 5 in 2024-07-26T12:38:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:54 smithi107 ceph-mon[32659]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:38:55.353 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T12:38:55.441 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T12:38:55.454 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-07-26T12:38:55.454 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-07-26T12:38:55.454 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113775 s, 922 MB/s 2024-07-26T12:38:55.454 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-07-26T12:38:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:55 smithi107 ceph-mon[32659]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:55 smithi107 ceph-mon[32659]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:38:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:55 smithi107 ceph-mon[32659]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T12:38:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:55 smithi132 ceph-mon[37580]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:55 smithi132 ceph-mon[37580]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:38:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:55 smithi132 ceph-mon[37580]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T12:38:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:56.108 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:55 smithi112 ceph-mon[37501]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:56.108 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:55 smithi112 ceph-mon[37501]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:38:56.109 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:56.109 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:55 smithi112 ceph-mon[37501]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T12:38:56.109 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:56.167 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_3 2024-07-26T12:38:56.425 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:38:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:38:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:38:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: osd.4 [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] boot 2024-07-26T12:38:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T12:38:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:57.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:38:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:38:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: osd.4 [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] boot 2024-07-26T12:38:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T12:38:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:38:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:38:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T12:38:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: osd.4 [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] boot 2024-07-26T12:38:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: osdmap e32: 5 total, 5 up, 5 in 2024-07-26T12:38:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:57.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:38:58.033 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:57 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi112 2024-07-26T12:38:58.033 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:57 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 8544M 2024-07-26T12:38:58.034 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:57 smithi132 ceph-mon[37580]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:58.034 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:57 smithi132 ceph-mon[37580]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T12:38:58.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.032+0000 7f47d14c7640 1 -- 172.21.15.132:0/90800430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc102490 msgr2=0x7f47cc1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:58.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.032+0000 7f47d14c7640 1 --2- 172.21.15.132:0/90800430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc102490 0x7f47cc1028a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f47bc009900 tx=0x7f47bc02a410 comp rx=0 tx=0).stop 2024-07-26T12:38:58.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.032+0000 7f47d14c7640 1 -- 172.21.15.132:0/90800430 shutdown_connections 2024-07-26T12:38:58.042 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.032+0000 7f47d14c7640 1 --2- 172.21.15.132:0/90800430 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47cc108720 0x7f47cc10ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.032+0000 7f47d14c7640 1 --2- 172.21.15.132:0/90800430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47cc107dd0 0x7f47cc1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.032+0000 7f47d14c7640 1 --2- 172.21.15.132:0/90800430 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc102490 0x7f47cc1028a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.032+0000 7f47d14c7640 1 -- 172.21.15.132:0/90800430 >> 172.21.15.132:0/90800430 conn(0x7f47cc079190 msgr2=0x7f47cc0795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.032+0000 7f47d14c7640 1 -- 172.21.15.132:0/90800430 shutdown_connections 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.032+0000 7f47d14c7640 1 -- 172.21.15.132:0/90800430 wait complete. 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.032+0000 7f47d14c7640 1 Processor -- start 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47d14c7640 1 -- start start 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47d14c7640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47cc102490 0x7f47cc19aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47d14c7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47cc107dd0 0x7f47cc19b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47d14c7640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc108720 0x7f47cc19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47d14c7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f47cc1124a0 con 0x7f47cc107dd0 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47d14c7640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f47cc112320 con 0x7f47cc102490 2024-07-26T12:38:58.043 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47d14c7640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f47cc112620 con 0x7f47cc108720 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47cb7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc108720 0x7f47cc19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47cb7fe640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc108720 0x7f47cc19b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:34056/0 (socket says 172.21.15.132:34056) 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47cb7fe640 1 -- 172.21.15.132:0/784152939 learned_addr learned my addr 172.21.15.132:0/784152939 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47caffd640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47cc102490 0x7f47cc19aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47cb7fe640 1 -- 172.21.15.132:0/784152939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47cc102490 msgr2=0x7f47cc19aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47cb7fe640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47cc102490 0x7f47cc19aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47cb7fe640 1 -- 172.21.15.132:0/784152939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47cc107dd0 msgr2=0x7f47cc19b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47cb7fe640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47cc107dd0 0x7f47cc19b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.033+0000 7f47cb7fe640 1 -- 172.21.15.132:0/784152939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47cc1a61c0 con 0x7f47cc108720 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.034+0000 7f47cb7fe640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc108720 0x7f47cc19b930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f47c400b970 tx=0x7f47c400be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.034+0000 7f47b3fff640 1 -- 172.21.15.132:0/784152939 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c4019070 con 0x7f47cc108720 2024-07-26T12:38:58.044 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.034+0000 7f47b3fff640 1 -- 172.21.15.132:0/784152939 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47c4014440 con 0x7f47cc108720 2024-07-26T12:38:58.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.034+0000 7f47d14c7640 1 -- 172.21.15.132:0/784152939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47cc1a64b0 con 0x7f47cc108720 2024-07-26T12:38:58.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.034+0000 7f47b3fff640 1 -- 172.21.15.132:0/784152939 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c40075d0 con 0x7f47cc108720 2024-07-26T12:38:58.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.034+0000 7f47d14c7640 1 -- 172.21.15.132:0/784152939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47cc1a69a0 con 0x7f47cc108720 2024-07-26T12:38:58.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.035+0000 7f47d14c7640 1 -- 172.21.15.132:0/784152939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47cc103710 con 0x7f47cc108720 2024-07-26T12:38:58.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.036+0000 7f47b3fff640 1 -- 172.21.15.132:0/784152939 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f47c401c070 con 0x7f47cc108720 2024-07-26T12:38:58.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.037+0000 7f47b3fff640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47b40b3c20 0x7f47b40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:38:58.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.037+0000 7f47b3fff640 1 -- 172.21.15.132:0/784152939 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4316+0+0 (secure 0 0 0) 0x7f47c40e2260 con 0x7f47cc108720 2024-07-26T12:38:58.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.037+0000 7f47caffd640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47b40b3c20 0x7f47b40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:38:58.045 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.038+0000 7f47caffd640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47b40b3c20 0x7f47b40b6130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f47bc000c00 tx=0x7f47bc031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:38:58.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:57 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi112 2024-07-26T12:38:58.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:57 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 8544M 2024-07-26T12:38:58.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:57 smithi107 ceph-mon[32659]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:58.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:57 smithi107 ceph-mon[32659]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T12:38:58.046 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.042+0000 7f47b3fff640 1 -- 172.21.15.132:0/784152939 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f47c40aa200 con 0x7f47cc108720 2024-07-26T12:38:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:57 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi112 2024-07-26T12:38:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:57 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 8544M 2024-07-26T12:38:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:57 smithi112 ceph-mon[37501]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-07-26T12:38:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:57 smithi112 ceph-mon[37501]: osdmap e33: 5 total, 5 up, 5 in 2024-07-26T12:38:58.229 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:38:58.227+0000 7f47d14c7640 1 -- 172.21.15.132:0/784152939 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f47cc10a450 con 0x7f47b40b3c20 2024-07-26T12:38:58.945 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:38:58.945 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:38:58.945 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:38:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:38:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:38:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:38:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:59.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:59 smithi132 ceph-mon[37580]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:38:59.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:38:59 smithi132 ceph-mon[37580]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:59 smithi107 ceph-mon[32659]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:39:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:38:59 smithi107 ceph-mon[32659]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:59 smithi112 ceph-mon[37501]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:39:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:38:59 smithi112 ceph-mon[37501]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:00 smithi107 ceph-mon[32659]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:00 smithi107 ceph-mon[32659]: from='client.? 172.21.15.132:0/4185675965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d0394cf-c561-4a68-b674-92ea81d9ffe1"}]: dispatch 2024-07-26T12:39:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:00 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d0394cf-c561-4a68-b674-92ea81d9ffe1"}]: dispatch 2024-07-26T12:39:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:00 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d0394cf-c561-4a68-b674-92ea81d9ffe1"}]': finished 2024-07-26T12:39:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:00 smithi107 ceph-mon[32659]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T12:39:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:00 smithi132 ceph-mon[37580]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:00 smithi132 ceph-mon[37580]: from='client.? 172.21.15.132:0/4185675965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d0394cf-c561-4a68-b674-92ea81d9ffe1"}]: dispatch 2024-07-26T12:39:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:00 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d0394cf-c561-4a68-b674-92ea81d9ffe1"}]: dispatch 2024-07-26T12:39:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:00 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d0394cf-c561-4a68-b674-92ea81d9ffe1"}]': finished 2024-07-26T12:39:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:00 smithi132 ceph-mon[37580]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T12:39:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:00 smithi112 ceph-mon[37501]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:00 smithi112 ceph-mon[37501]: from='client.? 172.21.15.132:0/4185675965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d0394cf-c561-4a68-b674-92ea81d9ffe1"}]: dispatch 2024-07-26T12:39:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:00 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d0394cf-c561-4a68-b674-92ea81d9ffe1"}]: dispatch 2024-07-26T12:39:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:00 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d0394cf-c561-4a68-b674-92ea81d9ffe1"}]': finished 2024-07-26T12:39:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:00 smithi112 ceph-mon[37501]: osdmap e34: 6 total, 5 up, 6 in 2024-07-26T12:39:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:01 smithi107 ceph-mon[32659]: from='client.? 172.21.15.132:0/3423879574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:39:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:01 smithi132 ceph-mon[37580]: from='client.? 172.21.15.132:0/3423879574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:39:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:01 smithi112 ceph-mon[37501]: from='client.? 172.21.15.132:0/3423879574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:39:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:02 smithi107 ceph-mon[32659]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:02 smithi132 ceph-mon[37580]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:02 smithi112 ceph-mon[37501]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:05 smithi112 ceph-mon[37501]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:05.968 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:05 smithi132 ceph-mon[37580]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:05 smithi107 ceph-mon[32659]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:07 smithi112 ceph-mon[37501]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:07 smithi107 ceph-mon[32659]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:07 smithi132 ceph-mon[37580]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T12:39:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:08 smithi112 ceph-mon[37501]: Deploying daemon osd.5 on smithi132 2024-07-26T12:39:08.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T12:39:08.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:08.983 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:08 smithi132 ceph-mon[37580]: Deploying daemon osd.5 on smithi132 2024-07-26T12:39:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T12:39:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:08 smithi107 ceph-mon[32659]: Deploying daemon osd.5 on smithi132 2024-07-26T12:39:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:09 smithi112 ceph-mon[37501]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:10.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:09 smithi107 ceph-mon[32659]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:09 smithi132 ceph-mon[37580]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:11 smithi112 ceph-mon[37501]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:11.959 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:11 smithi132 ceph-mon[37580]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:11 smithi107 ceph-mon[32659]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:13 smithi132 ceph-mon[37580]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:13 smithi112 ceph-mon[37501]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:13 smithi107 ceph-mon[32659]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:15.888 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:15 smithi132 ceph-mon[37580]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:15.948 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.945+0000 7f47b3fff640 1 -- 172.21.15.132:0/784152939 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f47cc10a450 con 0x7f47b40b3c20 2024-07-26T12:39:15.948 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 5 on host 'smithi132' 2024-07-26T12:39:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:15 smithi112 ceph-mon[37501]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:15.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 -- 172.21.15.132:0/784152939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47b40b3c20 msgr2=0x7f47b40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:39:15.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47b40b3c20 0x7f47b40b6130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f47bc000c00 tx=0x7f47bc031400 comp rx=0 tx=0).stop 2024-07-26T12:39:15.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 -- 172.21.15.132:0/784152939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc108720 msgr2=0x7f47cc19b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:39:15.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc108720 0x7f47cc19b930 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f47c400b970 tx=0x7f47c400be30 comp rx=0 tx=0).stop 2024-07-26T12:39:15.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 -- 172.21.15.132:0/784152939 shutdown_connections 2024-07-26T12:39:15.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47b40b3c20 0x7f47b40b6130 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:15.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47cc108720 0x7f47cc19b930 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:15.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47cc107dd0 0x7f47cc19b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:15.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 --2- 172.21.15.132:0/784152939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47cc102490 0x7f47cc19aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:15.951 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 -- 172.21.15.132:0/784152939 >> 172.21.15.132:0/784152939 conn(0x7f47cc079190 msgr2=0x7f47cc106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:39:15.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 -- 172.21.15.132:0/784152939 shutdown_connections 2024-07-26T12:39:15.952 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:39:15.948+0000 7f47d14c7640 1 -- 172.21.15.132:0/784152939 wait complete. 2024-07-26T12:39:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:15 smithi107 ceph-mon[32659]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:16.921 DEBUG:teuthology.orchestra.run.smithi132:osd.5> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.5.service 2024-07-26T12:39:16.924 INFO:tasks.cephadm:Deploying osd.6 on smithi107 with /dev/vg_nvme/lv_2... 2024-07-26T12:39:16.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T12:39:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:16 smithi112 ceph-mon[37501]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:16 smithi112 ceph-mon[37501]: from='osd.5 [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T12:39:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:16 smithi112 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T12:39:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:16 smithi107 ceph-mon[32659]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:16 smithi107 ceph-mon[32659]: from='osd.5 [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T12:39:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:16 smithi107 ceph-mon[32659]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T12:39:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:16 smithi132 ceph-mon[37580]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:16 smithi132 ceph-mon[37580]: from='osd.5 [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T12:39:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:16 smithi132 ceph-mon[37580]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T12:39:17.380 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:39:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:17 smithi132 ceph-mon[37580]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T12:39:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:17 smithi132 ceph-mon[37580]: from='osd.5 [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:39:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:17 smithi132 ceph-mon[37580]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T12:39:18.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:17 smithi132 ceph-mon[37580]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:39:18.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:18.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:17 smithi107 ceph-mon[32659]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T12:39:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:17 smithi107 ceph-mon[32659]: from='osd.5 [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:39:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:17 smithi107 ceph-mon[32659]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T12:39:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:17 smithi107 ceph-mon[32659]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:39:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:17 smithi112 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T12:39:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:17 smithi112 ceph-mon[37501]: from='osd.5 [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:39:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:17 smithi112 ceph-mon[37501]: osdmap e35: 6 total, 5 up, 6 in 2024-07-26T12:39:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:17 smithi112 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:39:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:18.768 INFO:journalctl@ceph.osd.5.smithi132.stdout:Jul 26 12:39:18 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-5[50909]: 2024-07-26T12:39:18.673+0000 7f4c0643e640 -1 osd.5 0 waiting for initial osdmap 2024-07-26T12:39:18.768 INFO:journalctl@ceph.osd.5.smithi132.stdout:Jul 26 12:39:18 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-5[50909]: 2024-07-26T12:39:18.680+0000 7f4bfda1f640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T12:39:19.014 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:18 smithi107 ceph-mon[32659]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:39:19.014 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:19.014 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:18 smithi107 ceph-mon[32659]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T12:39:19.014 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:19.014 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:18 smithi107 ceph-mon[32659]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:19.014 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:18 smithi107 ceph-mon[32659]: from='osd.5 ' entity='osd.5' 2024-07-26T12:39:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:18 smithi132 ceph-mon[37580]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:39:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:18 smithi132 ceph-mon[37580]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T12:39:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:18 smithi132 ceph-mon[37580]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:18 smithi132 ceph-mon[37580]: from='osd.5 ' entity='osd.5' 2024-07-26T12:39:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:18 smithi112 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:39:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:18 smithi112 ceph-mon[37501]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T12:39:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:18 smithi112 ceph-mon[37501]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 447 GiB / 447 GiB avail 2024-07-26T12:39:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:18 smithi112 ceph-mon[37501]: from='osd.5 ' entity='osd.5' 2024-07-26T12:39:19.839 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T12:39:20.001 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T12:39:20.039 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-07-26T12:39:20.039 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-07-26T12:39:20.039 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.030503 s, 344 MB/s 2024-07-26T12:39:20.039 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-07-26T12:39:20.171 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:19 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:39:20.171 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:19 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:39:20.171 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:20.172 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:19 smithi132 ceph-mon[37580]: osd.5 [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] boot 2024-07-26T12:39:20.172 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:19 smithi132 ceph-mon[37580]: osdmap e37: 6 total, 6 up, 6 in 2024-07-26T12:39:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:19 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:39:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:19 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:39:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:19 smithi107 ceph-mon[32659]: osd.5 [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] boot 2024-07-26T12:39:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:19 smithi107 ceph-mon[32659]: osdmap e37: 6 total, 6 up, 6 in 2024-07-26T12:39:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:19 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:39:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:19 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:39:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T12:39:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:19 smithi112 ceph-mon[37501]: osd.5 [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] boot 2024-07-26T12:39:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:19 smithi112 ceph-mon[37501]: osdmap e37: 6 total, 6 up, 6 in 2024-07-26T12:39:20.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_2 2024-07-26T12:39:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T12:39:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi132 2024-07-26T12:39:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 8544M 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:20 smithi107 ceph-mon[32659]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T12:39:21.310 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:39:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T12:39:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi132 2024-07-26T12:39:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 8544M 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:21.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:20 smithi132 ceph-mon[37580]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T12:39:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: osdmap e38: 6 total, 6 up, 6 in 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi132 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 8544M 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:20 smithi112 ceph-mon[37501]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T12:39:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.492+0000 7f8783e00640 1 -- 172.21.15.107:0/4008591443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87841015f0 msgr2=0x7f8784101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:39:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.492+0000 7f8783e00640 1 --2- 172.21.15.107:0/4008591443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87841015f0 0x7f8784101a00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8774009900 tx=0x7f877402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:39:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.492+0000 7f8783e00640 1 -- 172.21.15.107:0/4008591443 shutdown_connections 2024-07-26T12:39:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.492+0000 7f8783e00640 1 --2- 172.21.15.107:0/4008591443 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8784107880 0x7f878410e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.492+0000 7f8783e00640 1 --2- 172.21.15.107:0/4008591443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784106f30 0x7f8784107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.492+0000 7f8783e00640 1 --2- 172.21.15.107:0/4008591443 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87841015f0 0x7f8784101a00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:23.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.492+0000 7f8783e00640 1 -- 172.21.15.107:0/4008591443 >> 172.21.15.107:0/4008591443 conn(0x7f87840780b0 msgr2=0x7f87840784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:39:23.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.492+0000 7f8783e00640 1 -- 172.21.15.107:0/4008591443 shutdown_connections 2024-07-26T12:39:23.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.493+0000 7f8783e00640 1 -- 172.21.15.107:0/4008591443 wait complete. 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.494+0000 7f8783e00640 1 Processor -- start 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.494+0000 7f8783e00640 1 -- start start 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.494+0000 7f8783e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87841015f0 0x7f87840fe200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.494+0000 7f8783e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784106f30 0x7f87840fe740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.494+0000 7f8783e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8784107880 0x7f87840fec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.494+0000 7f8783e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8784111150 con 0x7f8784106f30 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.494+0000 7f8783e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8784110fd0 con 0x7f8784107880 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.494+0000 7f8783e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f87841112d0 con 0x7f87841015f0 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.494+0000 7f877be00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784106f30 0x7f87840fe740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.495+0000 7f877be00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784106f30 0x7f87840fe740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50860/0 (socket says 172.21.15.107:50860) 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.495+0000 7f8783400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8784107880 0x7f87840fec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:23.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.495+0000 7f877be00640 1 -- 172.21.15.107:0/3847057176 learned_addr learned my addr 172.21.15.107:0/3847057176 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.495+0000 7f8782a00640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87841015f0 0x7f87840fe200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.495+0000 7f8783400640 1 -- 172.21.15.107:0/3847057176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87841015f0 msgr2=0x7f87840fe200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.495+0000 7f8783400640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87841015f0 0x7f87840fe200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.495+0000 7f8783400640 1 -- 172.21.15.107:0/3847057176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784106f30 msgr2=0x7f87840fe740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.495+0000 7f8783400640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784106f30 0x7f87840fe740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.495+0000 7f8783400640 1 -- 172.21.15.107:0/3847057176 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87841a7fe0 con 0x7f8784107880 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.495+0000 7f8783400640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8784107880 0x7f87840fec80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f877c009fd0 tx=0x7f877c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.496+0000 7f877b400640 1 -- 172.21.15.107:0/3847057176 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c00ce20 con 0x7f8784107880 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.496+0000 7f877b400640 1 -- 172.21.15.107:0/3847057176 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f877c005020 con 0x7f8784107880 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.496+0000 7f8783e00640 1 -- 172.21.15.107:0/3847057176 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87841a82d0 con 0x7f8784107880 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.496+0000 7f877b400640 1 -- 172.21.15.107:0/3847057176 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c010690 con 0x7f8784107880 2024-07-26T12:39:23.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.496+0000 7f8783e00640 1 -- 172.21.15.107:0/3847057176 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87841a8830 con 0x7f8784107880 2024-07-26T12:39:23.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.497+0000 7f8783e00640 1 -- 172.21.15.107:0/3847057176 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8784101a00 con 0x7f8784107880 2024-07-26T12:39:23.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.500+0000 7f877b400640 1 -- 172.21.15.107:0/3847057176 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f877c0054d0 con 0x7f8784107880 2024-07-26T12:39:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.501+0000 7f877b400640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87640b3ca0 0x7f87640b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:39:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.501+0000 7f877b400640 1 -- 172.21.15.107:0/3847057176 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4748+0+0 (secure 0 0 0) 0x7f877c0e3500 con 0x7f8784107880 2024-07-26T12:39:23.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.502+0000 7f8782a00640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87640b3ca0 0x7f87640b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.502+0000 7f8782a00640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87640b3ca0 0x7f87640b61b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8774005ab0 tx=0x7f8774005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:39:23.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.503+0000 7f877b400640 1 -- 172.21.15.107:0/3847057176 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f877c0ab8a0 con 0x7f8784107880 2024-07-26T12:39:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:23.715+0000 7f8783e00640 1 -- 172.21.15.107:0/3847057176 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8784063230 con 0x7f87640b3ca0 2024-07-26T12:39:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:23 smithi112 ceph-mon[37501]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-07-26T12:39:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:23 smithi107 ceph-mon[32659]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-07-26T12:39:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:23 smithi132 ceph-mon[37580]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-07-26T12:39:24.719 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:24 smithi107 ceph-mon[32659]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:39:24.719 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:39:24.720 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:39:24.720 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:24 smithi112 ceph-mon[37501]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:39:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:39:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:39:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:24 smithi132 ceph-mon[37580]: from='client.24455 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:39:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:39:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:39:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:25 smithi112 ceph-mon[37501]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T12:39:25.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:25 smithi107 ceph-mon[32659]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T12:39:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:25 smithi132 ceph-mon[37580]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T12:39:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:27 smithi112 ceph-mon[37501]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-07-26T12:39:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:27 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2667349026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17052119-63ed-49e0-91a7-1e888c82dee2"}]: dispatch 2024-07-26T12:39:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:27 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17052119-63ed-49e0-91a7-1e888c82dee2"}]: dispatch 2024-07-26T12:39:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:27 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17052119-63ed-49e0-91a7-1e888c82dee2"}]': finished 2024-07-26T12:39:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:27 smithi112 ceph-mon[37501]: osdmap e40: 7 total, 6 up, 7 in 2024-07-26T12:39:28.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:27 smithi107 ceph-mon[32659]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-07-26T12:39:28.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:27 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2667349026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17052119-63ed-49e0-91a7-1e888c82dee2"}]: dispatch 2024-07-26T12:39:28.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:27 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17052119-63ed-49e0-91a7-1e888c82dee2"}]: dispatch 2024-07-26T12:39:28.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:27 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17052119-63ed-49e0-91a7-1e888c82dee2"}]': finished 2024-07-26T12:39:28.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:28.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:27 smithi107 ceph-mon[32659]: osdmap e40: 7 total, 6 up, 7 in 2024-07-26T12:39:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:27 smithi132 ceph-mon[37580]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-07-26T12:39:28.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:27 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2667349026' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17052119-63ed-49e0-91a7-1e888c82dee2"}]: dispatch 2024-07-26T12:39:28.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:27 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17052119-63ed-49e0-91a7-1e888c82dee2"}]: dispatch 2024-07-26T12:39:28.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:27 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17052119-63ed-49e0-91a7-1e888c82dee2"}]': finished 2024-07-26T12:39:28.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:28.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:27 smithi132 ceph-mon[37580]: osdmap e40: 7 total, 6 up, 7 in 2024-07-26T12:39:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:28 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3142827410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:39:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:28 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3142827410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:39:29.058 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:28 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3142827410' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:39:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:29 smithi112 ceph-mon[37501]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T12:39:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:29 smithi107 ceph-mon[32659]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T12:39:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:29 smithi132 ceph-mon[37580]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T12:39:31.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:31 smithi107 ceph-mon[32659]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-07-26T12:39:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:31 smithi112 ceph-mon[37501]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-07-26T12:39:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:31 smithi132 ceph-mon[37580]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-07-26T12:39:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:33 smithi107 ceph-mon[32659]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:33 smithi112 ceph-mon[37501]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:33 smithi132 ceph-mon[37580]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:35 smithi112 ceph-mon[37501]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:35 smithi107 ceph-mon[32659]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:35 smithi132 ceph-mon[37580]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T12:39:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T12:39:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T12:39:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:37.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:37 smithi107 ceph-mon[32659]: Deploying daemon osd.6 on smithi107 2024-07-26T12:39:37.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:37 smithi107 ceph-mon[32659]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:37 smithi112 ceph-mon[37501]: Deploying daemon osd.6 on smithi107 2024-07-26T12:39:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:37 smithi112 ceph-mon[37501]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:37 smithi132 ceph-mon[37580]: Deploying daemon osd.6 on smithi107 2024-07-26T12:39:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:37 smithi132 ceph-mon[37580]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:39.947 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:39 smithi107 ceph-mon[32659]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:39 smithi112 ceph-mon[37501]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:39 smithi132 ceph-mon[37580]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:41 smithi107 ceph-mon[32659]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:41 smithi112 ceph-mon[37501]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:41 smithi132 ceph-mon[37580]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:43.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:43 smithi107 ceph-mon[32659]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:43 smithi112 ceph-mon[37501]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:43 smithi132 ceph-mon[37580]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:45 smithi112 ceph-mon[37501]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:45 smithi107 ceph-mon[32659]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:45 smithi132 ceph-mon[37580]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:47 smithi107 ceph-mon[32659]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:47 smithi132 ceph-mon[37580]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:47 smithi112 ceph-mon[37501]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:49 smithi107 ceph-mon[32659]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:49 smithi107 ceph-mon[32659]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T12:39:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:49 smithi132 ceph-mon[37580]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:49 smithi132 ceph-mon[37580]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T12:39:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:49 smithi112 ceph-mon[37501]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:49 smithi112 ceph-mon[37501]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T12:39:50.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.772+0000 7f877b400640 1 -- 172.21.15.107:0/3847057176 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8784063230 con 0x7f87640b3ca0 2024-07-26T12:39:50.775 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 6 on host 'smithi107' 2024-07-26T12:39:50.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.776+0000 7f8783e00640 1 -- 172.21.15.107:0/3847057176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87640b3ca0 msgr2=0x7f87640b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:39:50.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.776+0000 7f8783e00640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87640b3ca0 0x7f87640b61b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8774005ab0 tx=0x7f8774005a20 comp rx=0 tx=0).stop 2024-07-26T12:39:50.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.776+0000 7f8783e00640 1 -- 172.21.15.107:0/3847057176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8784107880 msgr2=0x7f87840fec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:39:50.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.776+0000 7f8783e00640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8784107880 0x7f87840fec80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f877c009fd0 tx=0x7f877c00ef10 comp rx=0 tx=0).stop 2024-07-26T12:39:50.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.777+0000 7f8783e00640 1 -- 172.21.15.107:0/3847057176 shutdown_connections 2024-07-26T12:39:50.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.777+0000 7f8783e00640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87640b3ca0 0x7f87640b61b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:50.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.777+0000 7f8783e00640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8784107880 0x7f87840fec80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:50.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.777+0000 7f8783e00640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784106f30 0x7f87840fe740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:50.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.777+0000 7f8783e00640 1 --2- 172.21.15.107:0/3847057176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87841015f0 0x7f87840fe200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:50.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.777+0000 7f8783e00640 1 -- 172.21.15.107:0/3847057176 >> 172.21.15.107:0/3847057176 conn(0x7f87840780b0 msgr2=0x7f87841051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:39:50.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.777+0000 7f8783e00640 1 -- 172.21.15.107:0/3847057176 shutdown_connections 2024-07-26T12:39:50.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:39:50.777+0000 7f8783e00640 1 -- 172.21.15.107:0/3847057176 wait complete. 2024-07-26T12:39:50.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:50.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:50.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:50.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:50 smithi107 ceph-mon[32659]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T12:39:50.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:50.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:50 smithi107 ceph-mon[32659]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T12:39:50.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:50 smithi107 ceph-mon[32659]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:39:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:50 smithi132 ceph-mon[37580]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T12:39:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:50 smithi132 ceph-mon[37580]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T12:39:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:50 smithi132 ceph-mon[37580]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:39:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:50 smithi112 ceph-mon[37501]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T12:39:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:50 smithi112 ceph-mon[37501]: osdmap e41: 7 total, 6 up, 7 in 2024-07-26T12:39:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:50 smithi112 ceph-mon[37501]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:39:51.568 DEBUG:teuthology.orchestra.run.smithi107:osd.6> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.6.service 2024-07-26T12:39:51.572 INFO:tasks.cephadm:Deploying osd.7 on smithi112 with /dev/vg_nvme/lv_2... 2024-07-26T12:39:51.573 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T12:39:51.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:51 smithi107 ceph-mon[32659]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:51.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:51.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:51.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:51.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:51 smithi107 ceph-mon[32659]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:39:51.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:51.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:51 smithi107 ceph-mon[32659]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T12:39:51.755 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:51.831 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T12:39:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:51 smithi112 ceph-mon[37501]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:51 smithi112 ceph-mon[37501]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:39:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:51 smithi112 ceph-mon[37501]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T12:39:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:51 smithi132 ceph-mon[37580]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:51 smithi132 ceph-mon[37580]: from='osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:39:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:51 smithi132 ceph-mon[37580]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T12:39:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:53.168 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:53 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:39:53.168 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:53 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:39:53.168 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:53.168 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:53 smithi112 ceph-mon[37501]: osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] boot 2024-07-26T12:39:53.168 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:53 smithi112 ceph-mon[37501]: osdmap e43: 7 total, 7 up, 7 in 2024-07-26T12:39:53.168 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:53 smithi112 ceph-mon[37501]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:53.187 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:53 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:39:53.187 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:53 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:39:53.187 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:53.187 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:53 smithi107 ceph-mon[32659]: osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] boot 2024-07-26T12:39:53.187 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:53 smithi107 ceph-mon[32659]: osdmap e43: 7 total, 7 up, 7 in 2024-07-26T12:39:53.187 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:53 smithi107 ceph-mon[32659]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:53 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:39:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:53 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:39:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T12:39:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:53 smithi132 ceph-mon[37580]: osd.6 [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] boot 2024-07-26T12:39:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:53 smithi132 ceph-mon[37580]: osdmap e43: 7 total, 7 up, 7 in 2024-07-26T12:39:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:53 smithi132 ceph-mon[37580]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-07-26T12:39:53.522 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T12:39:53.602 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T12:39:53.615 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-07-26T12:39:53.615 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-07-26T12:39:53.615 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113832 s, 921 MB/s 2024-07-26T12:39:53.615 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-07-26T12:39:54.277 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:54 smithi112 ceph-mon[37501]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T12:39:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:54 smithi107 ceph-mon[32659]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T12:39:54.307 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_2 2024-07-26T12:39:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:54 smithi132 ceph-mon[37580]: osdmap e44: 7 total, 7 up, 7 in 2024-07-26T12:39:54.562 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T12:39:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:55 smithi112 ceph-mon[37501]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:39:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:55 smithi107 ceph-mon[32659]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:39:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:55 smithi132 ceph-mon[37580]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:39:56.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.145+0000 7fb0602bd640 1 -- 172.21.15.112:0/798047124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb058102490 msgr2=0x7fb0581028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:39:56.146 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.145+0000 7fb0602bd640 1 --2- 172.21.15.112:0/798047124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb058102490 0x7fb0581028a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb054009900 tx=0x7fb05402a410 comp rx=0 tx=0).stop 2024-07-26T12:39:56.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.145+0000 7fb0602bd640 1 -- 172.21.15.112:0/798047124 shutdown_connections 2024-07-26T12:39:56.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.145+0000 7fb0602bd640 1 --2- 172.21.15.112:0/798047124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb058108720 0x7fb05810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:56.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.145+0000 7fb0602bd640 1 --2- 172.21.15.112:0/798047124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb058107dd0 0x7fb0581081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:56.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.145+0000 7fb0602bd640 1 --2- 172.21.15.112:0/798047124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb058102490 0x7fb0581028a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:56.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.145+0000 7fb0602bd640 1 -- 172.21.15.112:0/798047124 >> 172.21.15.112:0/798047124 conn(0x7fb058079160 msgr2=0x7fb058079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:39:56.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.145+0000 7fb0602bd640 1 -- 172.21.15.112:0/798047124 shutdown_connections 2024-07-26T12:39:56.147 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.145+0000 7fb0602bd640 1 -- 172.21.15.112:0/798047124 wait complete. 2024-07-26T12:39:56.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.146+0000 7fb0602bd640 1 Processor -- start 2024-07-26T12:39:56.149 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.146+0000 7fb0602bd640 1 -- start start 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.146+0000 7fb0602bd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb058102490 0x7fb058199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.146+0000 7fb0602bd640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb058107dd0 0x7fb05819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.146+0000 7fb0602bd640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb058108720 0x7fb05819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.146+0000 7fb0602bd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb058113510 con 0x7fb058102490 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb0602bd640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb058113390 con 0x7fb058107dd0 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb0602bd640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb058113690 con 0x7fb058108720 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05d858640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb058107dd0 0x7fb05819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05d858640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb058107dd0 0x7fb05819a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:33946/0 (socket says 172.21.15.112:33946) 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05d858640 1 -- 172.21.15.112:0/1028243140 learned_addr learned my addr 172.21.15.112:0/1028243140 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05e85a640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb058108720 0x7fb05819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05d858640 1 -- 172.21.15.112:0/1028243140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb058108720 msgr2=0x7fb05819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:39:56.150 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05e059640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb058102490 0x7fb058199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05d858640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb058108720 0x7fb05819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05d858640 1 -- 172.21.15.112:0/1028243140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb058102490 msgr2=0x7fb058199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05d858640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb058102490 0x7fb058199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05d858640 1 -- 172.21.15.112:0/1028243140 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05819fe90 con 0x7fb058107dd0 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05e85a640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb058108720 0x7fb05819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb05d858640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb058107dd0 0x7fb05819a4c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb04800d740 tx=0x7fb04800dc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb04f7fe640 1 -- 172.21.15.112:0/1028243140 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048017070 con 0x7fb058107dd0 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb0602bd640 1 -- 172.21.15.112:0/1028243140 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0581a0180 con 0x7fb058107dd0 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb04f7fe640 1 -- 172.21.15.112:0/1028243140 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb04800de50 con 0x7fb058107dd0 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb04f7fe640 1 -- 172.21.15.112:0/1028243140 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048004ce0 con 0x7fb058107dd0 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.147+0000 7fb0602bd640 1 -- 172.21.15.112:0/1028243140 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0581a68e0 con 0x7fb058107dd0 2024-07-26T12:39:56.151 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.149+0000 7fb0602bd640 1 -- 172.21.15.112:0/1028243140 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb058103710 con 0x7fb058107dd0 2024-07-26T12:39:56.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.150+0000 7fb04f7fe640 1 -- 172.21.15.112:0/1028243140 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fb048004e80 con 0x7fb058107dd0 2024-07-26T12:39:56.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.151+0000 7fb04f7fe640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb0440b3c20 0x7fb0440b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:39:56.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.151+0000 7fb04f7fe640 1 -- 172.21.15.112:0/1028243140 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5180+0+0 (secure 0 0 0) 0x7fb0480e2ec0 con 0x7fb058107dd0 2024-07-26T12:39:56.152 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.151+0000 7fb05e059640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb0440b3c20 0x7fb0440b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:39:56.153 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.152+0000 7fb05e059640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb0440b3c20 0x7fb0440b6130 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb054005f70 tx=0x7fb054000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:39:56.154 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.154+0000 7fb04f7fe640 1 -- 172.21.15.112:0/1028243140 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb0480ab150 con 0x7fb058107dd0 2024-07-26T12:39:56.339 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:39:56.338+0000 7fb0602bd640 1 -- 172.21.15.112:0/1028243140 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb058063230 con 0x7fb0440b3c20 2024-07-26T12:39:57.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi107 2024-07-26T12:39:57.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 5696M 2024-07-26T12:39:57.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:57.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:57.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='client.24482 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:39:57.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:39:57.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:39:57.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:57.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:57 smithi107 ceph-mon[32659]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:39:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi107 2024-07-26T12:39:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 5696M 2024-07-26T12:39:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:57.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:57.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='client.24482 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:39:57.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:39:57.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:39:57.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:57.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:57 smithi112 ceph-mon[37501]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:39:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi107 2024-07-26T12:39:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 5696M 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='client.24482 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:39:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:57.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:57 smithi132 ceph-mon[37580]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:39:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:59 smithi112 ceph-mon[37501]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:39:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:59 smithi112 ceph-mon[37501]: from='client.? 172.21.15.112:0/3259569074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53"}]: dispatch 2024-07-26T12:39:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:59 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53"}]: dispatch 2024-07-26T12:39:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:59 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53"}]': finished 2024-07-26T12:39:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:39:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:59 smithi112 ceph-mon[37501]: osdmap e45: 8 total, 7 up, 8 in 2024-07-26T12:39:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:39:59 smithi112 ceph-mon[37501]: from='client.? 172.21.15.112:0/3114208654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:40:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:59 smithi107 ceph-mon[32659]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:59 smithi107 ceph-mon[32659]: from='client.? 172.21.15.112:0/3259569074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53"}]: dispatch 2024-07-26T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:59 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53"}]: dispatch 2024-07-26T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:59 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53"}]': finished 2024-07-26T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:59 smithi107 ceph-mon[32659]: osdmap e45: 8 total, 7 up, 8 in 2024-07-26T12:40:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:39:59 smithi107 ceph-mon[32659]: from='client.? 172.21.15.112:0/3114208654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:40:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:59 smithi132 ceph-mon[37580]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:59 smithi132 ceph-mon[37580]: from='client.? 172.21.15.112:0/3259569074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53"}]: dispatch 2024-07-26T12:40:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:59 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53"}]: dispatch 2024-07-26T12:40:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:59 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53"}]': finished 2024-07-26T12:40:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:59 smithi132 ceph-mon[37580]: osdmap e45: 8 total, 7 up, 8 in 2024-07-26T12:40:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:39:59 smithi132 ceph-mon[37580]: from='client.? 172.21.15.112:0/3114208654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:40:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:00 smithi112 ceph-mon[37501]: overall HEALTH_OK 2024-07-26T12:40:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:00 smithi107 ceph-mon[32659]: overall HEALTH_OK 2024-07-26T12:40:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:00 smithi132 ceph-mon[37580]: overall HEALTH_OK 2024-07-26T12:40:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:01 smithi112 ceph-mon[37501]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:01 smithi107 ceph-mon[32659]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:01 smithi132 ceph-mon[37580]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 205 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:03.801 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:03 smithi112 ceph-mon[37501]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:03 smithi107 ceph-mon[32659]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:03 smithi132 ceph-mon[37580]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:05 smithi112 ceph-mon[37501]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:05 smithi107 ceph-mon[32659]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:05 smithi132 ceph-mon[37580]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T12:40:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:06 smithi112 ceph-mon[37501]: Deploying daemon osd.7 on smithi112 2024-07-26T12:40:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T12:40:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:06 smithi107 ceph-mon[32659]: Deploying daemon osd.7 on smithi112 2024-07-26T12:40:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T12:40:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:06 smithi132 ceph-mon[37580]: Deploying daemon osd.7 on smithi112 2024-07-26T12:40:07.731 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:07 smithi112 ceph-mon[37501]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:07 smithi107 ceph-mon[32659]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:07 smithi132 ceph-mon[37580]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:09 smithi112 ceph-mon[37501]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:10.021 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:09 smithi132 ceph-mon[37580]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:09 smithi107 ceph-mon[32659]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:11.805 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:11 smithi112 ceph-mon[37501]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:11.805 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:11.805 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:11.805 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:11 smithi107 ceph-mon[32659]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:11 smithi132 ceph-mon[37580]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:13 smithi112 ceph-mon[37501]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:13 smithi107 ceph-mon[32659]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:13 smithi132 ceph-mon[37580]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:14.508 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.506+0000 7fb04f7fe640 1 -- 172.21.15.112:0/1028243140 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb058063230 con 0x7fb0440b3c20 2024-07-26T12:40:14.508 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 7 on host 'smithi112' 2024-07-26T12:40:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 -- 172.21.15.112:0/1028243140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb0440b3c20 msgr2=0x7fb0440b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:40:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb0440b3c20 0x7fb0440b6130 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb054005f70 tx=0x7fb054000bc0 comp rx=0 tx=0).stop 2024-07-26T12:40:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 -- 172.21.15.112:0/1028243140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb058107dd0 msgr2=0x7fb05819a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:40:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb058107dd0 0x7fb05819a4c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb04800d740 tx=0x7fb04800dc00 comp rx=0 tx=0).stop 2024-07-26T12:40:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 -- 172.21.15.112:0/1028243140 shutdown_connections 2024-07-26T12:40:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb0440b3c20 0x7fb0440b6130 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb058108720 0x7fb05819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb058107dd0 0x7fb05819a4c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:14.510 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 --2- 172.21.15.112:0/1028243140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb058102490 0x7fb058199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:14.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 -- 172.21.15.112:0/1028243140 >> 172.21.15.112:0/1028243140 conn(0x7fb058079160 msgr2=0x7fb058100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:40:14.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 -- 172.21.15.112:0/1028243140 shutdown_connections 2024-07-26T12:40:14.511 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:40:14.508+0000 7fb0602bd640 1 -- 172.21.15.112:0/1028243140 wait complete. 2024-07-26T12:40:15.216 DEBUG:teuthology.orchestra.run.smithi112:osd.7> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.7.service 2024-07-26T12:40:15.219 INFO:tasks.cephadm:Deploying osd.8 on smithi132 with /dev/vg_nvme/lv_2... 2024-07-26T12:40:15.219 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T12:40:15.481 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:40:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:15 smithi132 ceph-mon[37580]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:15 smithi132 ceph-mon[37580]: from='osd.7 [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T12:40:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:15 smithi132 ceph-mon[37580]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T12:40:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.744 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.744 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.744 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:15 smithi112 ceph-mon[37501]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:15.745 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:15 smithi112 ceph-mon[37501]: from='osd.7 [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T12:40:15.745 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:15 smithi112 ceph-mon[37501]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T12:40:15.745 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.745 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:15 smithi107 ceph-mon[32659]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:15 smithi107 ceph-mon[32659]: from='osd.7 [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T12:40:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:15 smithi107 ceph-mon[32659]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T12:40:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:16 smithi132 ceph-mon[37580]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T12:40:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:16 smithi132 ceph-mon[37580]: from='osd.7 [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:40:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:16 smithi132 ceph-mon[37580]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T12:40:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:16 smithi132 ceph-mon[37580]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:40:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:16 smithi107 ceph-mon[32659]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T12:40:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:16 smithi107 ceph-mon[32659]: from='osd.7 [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:40:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:16 smithi107 ceph-mon[32659]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T12:40:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:16 smithi107 ceph-mon[32659]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:40:16.837 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:16 smithi112 ceph-mon[37501]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T12:40:16.837 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:16.837 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:16 smithi112 ceph-mon[37501]: from='osd.7 [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:40:16.838 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:16 smithi112 ceph-mon[37501]: osdmap e46: 8 total, 7 up, 8 in 2024-07-26T12:40:16.838 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:16 smithi112 ceph-mon[37501]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:40:17.293 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T12:40:17.382 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T12:40:17.395 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-07-26T12:40:17.395 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-07-26T12:40:17.395 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114599 s, 915 MB/s 2024-07-26T12:40:17.395 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-07-26T12:40:17.450 INFO:journalctl@ceph.osd.7.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-7[56618]: 2024-07-26T12:40:17.214+0000 7ff36f38f640 -1 osd.7 0 waiting for initial osdmap 2024-07-26T12:40:17.451 INFO:journalctl@ceph.osd.7.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-7[56618]: 2024-07-26T12:40:17.227+0000 7ff367185640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T12:40:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:17 smithi132 ceph-mon[37580]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:40:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:17 smithi132 ceph-mon[37580]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T12:40:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:17 smithi132 ceph-mon[37580]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:17.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:17 smithi132 ceph-mon[37580]: from='osd.7 ' entity='osd.7' 2024-07-26T12:40:17.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:17.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:17 smithi132 ceph-mon[37580]: osd.7 [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] boot 2024-07-26T12:40:17.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:17 smithi132 ceph-mon[37580]: osdmap e48: 8 total, 8 up, 8 in 2024-07-26T12:40:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-mon[37501]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:40:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-mon[37501]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T12:40:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-mon[37501]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-mon[37501]: from='osd.7 ' entity='osd.7' 2024-07-26T12:40:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-mon[37501]: osd.7 [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] boot 2024-07-26T12:40:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:17 smithi112 ceph-mon[37501]: osdmap e48: 8 total, 8 up, 8 in 2024-07-26T12:40:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:17 smithi107 ceph-mon[32659]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:40:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:17 smithi107 ceph-mon[32659]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T12:40:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:17 smithi107 ceph-mon[32659]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-07-26T12:40:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:17 smithi107 ceph-mon[32659]: from='osd.7 ' entity='osd.7' 2024-07-26T12:40:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T12:40:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:17 smithi107 ceph-mon[32659]: osd.7 [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] boot 2024-07-26T12:40:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:17 smithi107 ceph-mon[32659]: osdmap e48: 8 total, 8 up, 8 in 2024-07-26T12:40:18.079 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_2 2024-07-26T12:40:18.333 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:40:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:40:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:18 smithi112 ceph-mon[37501]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T12:40:18.803 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:18.804 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:18.805 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:18 smithi132 ceph-mon[37580]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T12:40:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:40:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:40:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:19.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:19.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:19.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:19.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:19.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:18 smithi107 ceph-mon[32659]: osdmap e49: 8 total, 8 up, 8 in 2024-07-26T12:40:19.737 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:19 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi112 2024-07-26T12:40:19.737 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:19 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 5696M 2024-07-26T12:40:19.737 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:19 smithi132 ceph-mon[37580]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:19.738 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:19.738 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:19.738 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:19.738 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:19 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi112 2024-07-26T12:40:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:19 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 5696M 2024-07-26T12:40:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:19 smithi112 ceph-mon[37501]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:19.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.951+0000 7fa17b144640 1 -- 172.21.15.132:0/2001319033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa174106670 msgr2=0x7fa1741100e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:40:19.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.951+0000 7fa17b144640 1 --2- 172.21.15.132:0/2001319033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa174106670 0x7fa1741100e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009cd0 tx=0x7fa16c02a910 comp rx=0 tx=0).stop 2024-07-26T12:40:19.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.951+0000 7fa17b144640 1 -- 172.21.15.132:0/2001319033 shutdown_connections 2024-07-26T12:40:19.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.951+0000 7fa17b144640 1 --2- 172.21.15.132:0/2001319033 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa174106670 0x7fa1741100e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.951+0000 7fa17b144640 1 --2- 172.21.15.132:0/2001319033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa174105d20 0x7fa174106130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.951+0000 7fa17b144640 1 --2- 172.21.15.132:0/2001319033 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa174101ad0 0x7fa174101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.951+0000 7fa17b144640 1 -- 172.21.15.132:0/2001319033 >> 172.21.15.132:0/2001319033 conn(0x7fa1740799c0 msgr2=0x7fa174079dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.951+0000 7fa17b144640 1 -- 172.21.15.132:0/2001319033 shutdown_connections 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.951+0000 7fa17b144640 1 -- 172.21.15.132:0/2001319033 wait complete. 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa17b144640 1 Processor -- start 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa17b144640 1 -- start start 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa17b144640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa174101ad0 0x7fa174199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa17b144640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa174105d20 0x7fa17419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa17b144640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa174106670 0x7fa17419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:40:19.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa17b144640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1741135a0 con 0x7fa174101ad0 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa17b144640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa174113420 con 0x7fa174106670 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa17b144640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa174113720 con 0x7fa174105d20 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa173fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa174105d20 0x7fa17419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa173fff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa174105d20 0x7fa17419a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:35290/0 (socket says 172.21.15.132:35290) 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa173fff640 1 -- 172.21.15.132:0/405797346 learned_addr learned my addr 172.21.15.132:0/405797346 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa1796e1640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa174106670 0x7fa17419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa173fff640 1 -- 172.21.15.132:0/405797346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa174106670 msgr2=0x7fa17419f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa173fff640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa174106670 0x7fa17419f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa173fff640 1 -- 172.21.15.132:0/405797346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa174101ad0 msgr2=0x7fa174199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa173fff640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa174101ad0 0x7fa174199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.952+0000 7fa173fff640 1 -- 172.21.15.132:0/405797346 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17419ff60 con 0x7fa174105d20 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.953+0000 7fa173fff640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa174105d20 0x7fa17419a590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa168006fd0 tx=0x7fa16800b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:40:19.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.953+0000 7fa171ffb640 1 -- 172.21.15.132:0/405797346 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa168011830 con 0x7fa174105d20 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.953+0000 7fa171ffb640 1 -- 172.21.15.132:0/405797346 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa168016070 con 0x7fa174105d20 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.953+0000 7fa17b144640 1 -- 172.21.15.132:0/405797346 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1741a0250 con 0x7fa174105d20 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.953+0000 7fa171ffb640 1 -- 172.21.15.132:0/405797346 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa168010e00 con 0x7fa174105d20 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.953+0000 7fa17b144640 1 -- 172.21.15.132:0/405797346 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1741a69b0 con 0x7fa174105d20 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.954+0000 7fa17b144640 1 -- 172.21.15.132:0/405797346 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa174101ee0 con 0x7fa174105d20 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.955+0000 7fa171ffb640 1 -- 172.21.15.132:0/405797346 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fa16801b460 con 0x7fa174105d20 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.956+0000 7fa171ffb640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa15c0b3ca0 0x7fa15c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.956+0000 7fa171ffb640 1 -- 172.21.15.132:0/405797346 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5612+0+0 (secure 0 0 0) 0x7fa1680e27c0 con 0x7fa174105d20 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.956+0000 7fa178ee0640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa15c0b3ca0 0x7fa15c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.957+0000 7fa178ee0640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa15c0b3ca0 0x7fa15c0b61b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa164005fd0 tx=0x7fa16400b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:40:19.964 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:19.960+0000 7fa171ffb640 1 -- 172.21.15.132:0/405797346 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa1680aa8a0 con 0x7fa174105d20 2024-07-26T12:40:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:19 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi112 2024-07-26T12:40:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:19 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 5696M 2024-07-26T12:40:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:19 smithi107 ceph-mon[32659]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:20.145 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:20.143+0000 7fa17b144640 1 -- 172.21.15.132:0/405797346 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa174063230 con 0x7fa15c0b3ca0 2024-07-26T12:40:20.948 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:20 smithi132 ceph-mon[37580]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T12:40:20.948 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:40:20.948 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:40:20.948 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:20 smithi107 ceph-mon[32659]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T12:40:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:40:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:40:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:20 smithi112 ceph-mon[37501]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T12:40:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:40:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:40:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:21 smithi107 ceph-mon[32659]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:40:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:21 smithi107 ceph-mon[32659]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:22.054 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:21 smithi132 ceph-mon[37580]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:40:22.054 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:21 smithi132 ceph-mon[37580]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:21 smithi112 ceph-mon[37501]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:40:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:21 smithi112 ceph-mon[37501]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 632 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:23 smithi107 ceph-mon[32659]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T12:40:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:23 smithi107 ceph-mon[32659]: from='client.? 172.21.15.132:0/521837421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3958e5-fe71-448b-a6de-6bcc423daca4"}]: dispatch 2024-07-26T12:40:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:23 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3958e5-fe71-448b-a6de-6bcc423daca4"}]: dispatch 2024-07-26T12:40:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:23 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d3958e5-fe71-448b-a6de-6bcc423daca4"}]': finished 2024-07-26T12:40:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:23 smithi107 ceph-mon[32659]: osdmap e51: 9 total, 8 up, 9 in 2024-07-26T12:40:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:23 smithi107 ceph-mon[32659]: from='client.? 172.21.15.132:0/122658539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:40:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:23 smithi132 ceph-mon[37580]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T12:40:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:23 smithi132 ceph-mon[37580]: from='client.? 172.21.15.132:0/521837421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3958e5-fe71-448b-a6de-6bcc423daca4"}]: dispatch 2024-07-26T12:40:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:23 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3958e5-fe71-448b-a6de-6bcc423daca4"}]: dispatch 2024-07-26T12:40:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:23 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d3958e5-fe71-448b-a6de-6bcc423daca4"}]': finished 2024-07-26T12:40:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:23 smithi132 ceph-mon[37580]: osdmap e51: 9 total, 8 up, 9 in 2024-07-26T12:40:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:23 smithi132 ceph-mon[37580]: from='client.? 172.21.15.132:0/122658539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:40:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:23 smithi112 ceph-mon[37501]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T12:40:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:23 smithi112 ceph-mon[37501]: from='client.? 172.21.15.132:0/521837421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3958e5-fe71-448b-a6de-6bcc423daca4"}]: dispatch 2024-07-26T12:40:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:23 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d3958e5-fe71-448b-a6de-6bcc423daca4"}]: dispatch 2024-07-26T12:40:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:23 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d3958e5-fe71-448b-a6de-6bcc423daca4"}]': finished 2024-07-26T12:40:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:23 smithi112 ceph-mon[37501]: osdmap e51: 9 total, 8 up, 9 in 2024-07-26T12:40:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:23 smithi112 ceph-mon[37501]: from='client.? 172.21.15.132:0/122658539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:40:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:25 smithi107 ceph-mon[32659]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-07-26T12:40:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:25 smithi132 ceph-mon[37580]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-07-26T12:40:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:25 smithi112 ceph-mon[37501]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-07-26T12:40:27.868 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:27 smithi132 ceph-mon[37580]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T12:40:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:27 smithi107 ceph-mon[32659]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T12:40:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:27 smithi112 ceph-mon[37501]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T12:40:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:28 smithi132 ceph-mon[37580]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-07-26T12:40:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:28 smithi107 ceph-mon[32659]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-07-26T12:40:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:28 smithi112 ceph-mon[37501]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-07-26T12:40:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T12:40:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T12:40:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T12:40:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:31.404 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:31 smithi132 ceph-mon[37580]: Deploying daemon osd.8 on smithi132 2024-07-26T12:40:31.404 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:31 smithi132 ceph-mon[37580]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:40:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:31 smithi112 ceph-mon[37501]: Deploying daemon osd.8 on smithi132 2024-07-26T12:40:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:31 smithi112 ceph-mon[37501]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:40:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:31 smithi107 ceph-mon[32659]: Deploying daemon osd.8 on smithi132 2024-07-26T12:40:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:31 smithi107 ceph-mon[32659]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:40:33.702 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:33 smithi132 ceph-mon[37580]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:33 smithi112 ceph-mon[37501]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:33 smithi107 ceph-mon[32659]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:35.917 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:35 smithi132 ceph-mon[37580]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:35.917 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:35.917 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:35.917 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:35 smithi112 ceph-mon[37501]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:35 smithi107 ceph-mon[32659]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:37.905 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:37 smithi132 ceph-mon[37580]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:37 smithi112 ceph-mon[37501]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:37 smithi107 ceph-mon[32659]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:39.109 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.108+0000 7fa171ffb640 1 -- 172.21.15.132:0/405797346 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa174063230 con 0x7fa15c0b3ca0 2024-07-26T12:40:39.110 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 8 on host 'smithi132' 2024-07-26T12:40:39.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.110+0000 7fa17b144640 1 -- 172.21.15.132:0/405797346 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa15c0b3ca0 msgr2=0x7fa15c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:40:39.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.110+0000 7fa17b144640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa15c0b3ca0 0x7fa15c0b61b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa164005fd0 tx=0x7fa16400b410 comp rx=0 tx=0).stop 2024-07-26T12:40:39.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.110+0000 7fa17b144640 1 -- 172.21.15.132:0/405797346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa174105d20 msgr2=0x7fa17419a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:40:39.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.110+0000 7fa17b144640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa174105d20 0x7fa17419a590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa168006fd0 tx=0x7fa16800b8e0 comp rx=0 tx=0).stop 2024-07-26T12:40:39.112 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.111+0000 7fa17b144640 1 -- 172.21.15.132:0/405797346 shutdown_connections 2024-07-26T12:40:39.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.111+0000 7fa17b144640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa15c0b3ca0 0x7fa15c0b61b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:39.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.111+0000 7fa17b144640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa174106670 0x7fa17419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:39.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.111+0000 7fa17b144640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa174105d20 0x7fa17419a590 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:39.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.111+0000 7fa17b144640 1 --2- 172.21.15.132:0/405797346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa174101ad0 0x7fa174199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:39.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.111+0000 7fa17b144640 1 -- 172.21.15.132:0/405797346 >> 172.21.15.132:0/405797346 conn(0x7fa1740799c0 msgr2=0x7fa174107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:40:39.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.111+0000 7fa17b144640 1 -- 172.21.15.132:0/405797346 shutdown_connections 2024-07-26T12:40:39.113 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:40:39.111+0000 7fa17b144640 1 -- 172.21.15.132:0/405797346 wait complete. 2024-07-26T12:40:39.507 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.507 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.508 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:39.508 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:39.508 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.508 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:39 smithi132 ceph-mon[37580]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:39.508 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:39.508 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.508 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:39.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:39.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:39 smithi107 ceph-mon[32659]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:39.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:39.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:39 smithi112 ceph-mon[37501]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:39.811 DEBUG:teuthology.orchestra.run.smithi132:osd.8> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.8.service 2024-07-26T12:40:39.814 INFO:tasks.cephadm:Deploying osd.9 on smithi107 with /dev/vg_nvme/lv_1... 2024-07-26T12:40:39.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T12:40:40.273 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:40:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:40 smithi107 ceph-mon[32659]: from='osd.8 [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T12:40:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:40 smithi107 ceph-mon[32659]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T12:40:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:40 smithi132 ceph-mon[37580]: from='osd.8 [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T12:40:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:40 smithi132 ceph-mon[37580]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T12:40:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:40 smithi112 ceph-mon[37501]: from='osd.8 [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T12:40:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:40 smithi112 ceph-mon[37501]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T12:40:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:41 smithi107 ceph-mon[32659]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T12:40:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:41 smithi107 ceph-mon[32659]: from='osd.8 [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:40:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:41 smithi107 ceph-mon[32659]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T12:40:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:41 smithi107 ceph-mon[32659]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:40:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:41 smithi107 ceph-mon[32659]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:41 smithi132 ceph-mon[37580]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T12:40:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:41 smithi132 ceph-mon[37580]: from='osd.8 [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:40:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:41 smithi132 ceph-mon[37580]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T12:40:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:41 smithi132 ceph-mon[37580]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:40:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:41 smithi132 ceph-mon[37580]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:41.561 INFO:journalctl@ceph.osd.8.smithi132.stdout:Jul 26 12:40:41 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-8[57876]: 2024-07-26T12:40:41.269+0000 7f2a4b390640 -1 osd.8 0 waiting for initial osdmap 2024-07-26T12:40:41.561 INFO:journalctl@ceph.osd.8.smithi132.stdout:Jul 26 12:40:41 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-8[57876]: 2024-07-26T12:40:41.282+0000 7f2a4718e640 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T12:40:41.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:41 smithi112 ceph-mon[37501]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T12:40:41.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:41.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:41 smithi112 ceph-mon[37501]: from='osd.8 [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:40:41.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:41 smithi112 ceph-mon[37501]: osdmap e52: 9 total, 8 up, 9 in 2024-07-26T12:40:41.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:41 smithi112 ceph-mon[37501]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:40:41.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:41 smithi112 ceph-mon[37501]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-07-26T12:40:42.512 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:42 smithi107 ceph-mon[32659]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:40:42.512 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:42.512 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:42 smithi107 ceph-mon[32659]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T12:40:42.512 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:42.512 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:42.512 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:42 smithi107 ceph-mon[32659]: osd.8 [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] boot 2024-07-26T12:40:42.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:42 smithi107 ceph-mon[32659]: osdmap e54: 9 total, 9 up, 9 in 2024-07-26T12:40:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:42 smithi132 ceph-mon[37580]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:40:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:42 smithi132 ceph-mon[37580]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T12:40:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:42 smithi132 ceph-mon[37580]: osd.8 [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] boot 2024-07-26T12:40:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:42 smithi132 ceph-mon[37580]: osdmap e54: 9 total, 9 up, 9 in 2024-07-26T12:40:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:42 smithi112 ceph-mon[37501]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:40:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:42 smithi112 ceph-mon[37501]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T12:40:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T12:40:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:42 smithi112 ceph-mon[37501]: osd.8 [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] boot 2024-07-26T12:40:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:42 smithi112 ceph-mon[37501]: osdmap e54: 9 total, 9 up, 9 in 2024-07-26T12:40:42.745 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T12:40:42.938 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T12:40:42.979 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-07-26T12:40:42.979 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-07-26T12:40:42.979 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0316872 s, 331 MB/s 2024-07-26T12:40:42.980 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-07-26T12:40:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi132 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 5696M 2024-07-26T12:40:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:40:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:40:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T12:40:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:40:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi132 2024-07-26T12:40:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 5696M 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:40:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: osdmap e55: 9 total, 9 up, 9 in 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi132 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 5696M 2024-07-26T12:40:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:43.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:43.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:40:43.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_1 2024-07-26T12:40:44.350 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:40:44.665 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:44 smithi107 ceph-mon[32659]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T12:40:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:44 smithi112 ceph-mon[37501]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T12:40:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:44 smithi132 ceph-mon[37580]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T12:40:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:45 smithi112 ceph-mon[37501]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:40:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:45 smithi107 ceph-mon[32659]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:40:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:45 smithi132 ceph-mon[37580]: pgmap v199: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:40:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.787+0000 7fb202000640 1 -- 172.21.15.107:0/1534574090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1fc10f330 msgr2=0x7fb1fc10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:40:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.787+0000 7fb202000640 1 --2- 172.21.15.107:0/1534574090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1fc10f330 0x7fb1fc10f740 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009860 tx=0x7fb1e802a340 comp rx=0 tx=0).stop 2024-07-26T12:40:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.787+0000 7fb202000640 1 -- 172.21.15.107:0/1534574090 shutdown_connections 2024-07-26T12:40:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.787+0000 7fb202000640 1 --2- 172.21.15.107:0/1534574090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc10fc80 0x7fb1fc1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.787+0000 7fb202000640 1 --2- 172.21.15.107:0/1534574090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1fc10f330 0x7fb1fc10f740 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.787+0000 7fb202000640 1 --2- 172.21.15.107:0/1534574090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc1099f0 0x7fb1fc109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.787+0000 7fb202000640 1 -- 172.21.15.107:0/1534574090 >> 172.21.15.107:0/1534574090 conn(0x7fb1fc078ff0 msgr2=0x7fb1fc107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:40:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.788+0000 7fb202000640 1 -- 172.21.15.107:0/1534574090 shutdown_connections 2024-07-26T12:40:46.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.788+0000 7fb202000640 1 -- 172.21.15.107:0/1534574090 wait complete. 2024-07-26T12:40:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.789+0000 7fb202000640 1 Processor -- start 2024-07-26T12:40:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.789+0000 7fb202000640 1 -- start start 2024-07-26T12:40:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.789+0000 7fb202000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1fc1099f0 0x7fb1fc1a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:40:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.789+0000 7fb202000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc10f330 0x7fb1fc1a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:40:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.789+0000 7fb202000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc10fc80 0x7fb1fc1a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:40:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.789+0000 7fb202000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc119940 con 0x7fb1fc10fc80 2024-07-26T12:40:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.789+0000 7fb202000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc1197c0 con 0x7fb1fc10f330 2024-07-26T12:40:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.789+0000 7fb202000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc119ac0 con 0x7fb1fc1099f0 2024-07-26T12:40:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.790+0000 7fb201600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc10fc80 0x7fb1fc1a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:46.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.790+0000 7fb201600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc10fc80 0x7fb1fc1a5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60032/0 (socket says 172.21.15.107:60032) 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.790+0000 7fb1fbe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc10f330 0x7fb1fc1a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.790+0000 7fb201600640 1 -- 172.21.15.107:0/275147218 learned_addr learned my addr 172.21.15.107:0/275147218 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.790+0000 7fb200c00640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1fc1099f0 0x7fb1fc1a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.790+0000 7fb1fbe00640 1 -- 172.21.15.107:0/275147218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1fc1099f0 msgr2=0x7fb1fc1a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.790+0000 7fb1fbe00640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1fc1099f0 0x7fb1fc1a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.790+0000 7fb1fbe00640 1 -- 172.21.15.107:0/275147218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc10fc80 msgr2=0x7fb1fc1a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.790+0000 7fb1fbe00640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc10fc80 0x7fb1fc1a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.790+0000 7fb1fbe00640 1 -- 172.21.15.107:0/275147218 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc1a62e0 con 0x7fb1fc10f330 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.791+0000 7fb1fbe00640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc10f330 0x7fb1fc1a0910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb1e802a860 tx=0x7fb1e8032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.791+0000 7fb1f9600640 1 -- 172.21.15.107:0/275147218 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8040020 con 0x7fb1fc10f330 2024-07-26T12:40:46.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.791+0000 7fb202000640 1 -- 172.21.15.107:0/275147218 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1fc1a6570 con 0x7fb1fc10f330 2024-07-26T12:40:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.791+0000 7fb1f9600640 1 -- 172.21.15.107:0/275147218 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1e8030e10 con 0x7fb1fc10f330 2024-07-26T12:40:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.791+0000 7fb1f9600640 1 -- 172.21.15.107:0/275147218 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8037040 con 0x7fb1fc10f330 2024-07-26T12:40:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.791+0000 7fb202000640 1 -- 172.21.15.107:0/275147218 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1fc1acd30 con 0x7fb1fc10f330 2024-07-26T12:40:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.792+0000 7fb202000640 1 -- 172.21.15.107:0/275147218 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1fc10ac70 con 0x7fb1fc10f330 2024-07-26T12:40:46.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.794+0000 7fb1f9600640 1 -- 172.21.15.107:0/275147218 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fb1e8043020 con 0x7fb1fc10f330 2024-07-26T12:40:46.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.797+0000 7fb1f9600640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1e40b3c20 0x7fb1e40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:40:46.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.797+0000 7fb1f9600640 1 -- 172.21.15.107:0/275147218 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6044+0+0 (secure 0 0 0) 0x7fb1e8102ff0 con 0x7fb1fc10f330 2024-07-26T12:40:46.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.797+0000 7fb200c00640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1e40b3c20 0x7fb1e40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:40:46.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.798+0000 7fb200c00640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1e40b3c20 0x7fb1e40b6130 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009bd0 tx=0x7fb1f0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:40:46.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:46.800+0000 7fb1f9600640 1 -- 172.21.15.107:0/275147218 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb1e804a030 con 0x7fb1fc10f330 2024-07-26T12:40:47.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:40:47.009+0000 7fb202000640 1 -- 172.21.15.107:0/275147218 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fb1fc063230 con 0x7fb1e40b3c20 2024-07-26T12:40:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:47 smithi107 ceph-mon[32659]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:40:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:40:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:40:47.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:47 smithi112 ceph-mon[37501]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:40:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:40:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:40:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:47 smithi132 ceph-mon[37580]: pgmap v200: 1 pgs: 1 peering; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:40:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:40:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:40:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:48 smithi112 ceph-mon[37501]: from='client.24521 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:40:48.992 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:48 smithi107 ceph-mon[32659]: from='client.24521 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:40:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:48 smithi132 ceph-mon[37580]: from='client.24521 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:40:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:49 smithi112 ceph-mon[37501]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T12:40:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:49 smithi107 ceph-mon[32659]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T12:40:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:49 smithi132 ceph-mon[37580]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T12:40:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:51 smithi112 ceph-mon[37501]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-07-26T12:40:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:51 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1870601482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d641864-0787-4a2b-ae73-e590c1b5c0d0"}]: dispatch 2024-07-26T12:40:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:51 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d641864-0787-4a2b-ae73-e590c1b5c0d0"}]: dispatch 2024-07-26T12:40:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:51 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d641864-0787-4a2b-ae73-e590c1b5c0d0"}]': finished 2024-07-26T12:40:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:40:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:51 smithi112 ceph-mon[37501]: osdmap e57: 10 total, 9 up, 10 in 2024-07-26T12:40:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:51 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2535078473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:40:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:51 smithi107 ceph-mon[32659]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-07-26T12:40:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:51 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1870601482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d641864-0787-4a2b-ae73-e590c1b5c0d0"}]: dispatch 2024-07-26T12:40:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:51 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d641864-0787-4a2b-ae73-e590c1b5c0d0"}]: dispatch 2024-07-26T12:40:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:51 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d641864-0787-4a2b-ae73-e590c1b5c0d0"}]': finished 2024-07-26T12:40:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:40:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:51 smithi107 ceph-mon[32659]: osdmap e57: 10 total, 9 up, 10 in 2024-07-26T12:40:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:51 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2535078473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:40:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:51 smithi132 ceph-mon[37580]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-07-26T12:40:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:51 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1870601482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d641864-0787-4a2b-ae73-e590c1b5c0d0"}]: dispatch 2024-07-26T12:40:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:51 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d641864-0787-4a2b-ae73-e590c1b5c0d0"}]: dispatch 2024-07-26T12:40:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:51 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d641864-0787-4a2b-ae73-e590c1b5c0d0"}]': finished 2024-07-26T12:40:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:40:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:51 smithi132 ceph-mon[37580]: osdmap e57: 10 total, 9 up, 10 in 2024-07-26T12:40:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:51 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2535078473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:40:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:53 smithi112 ceph-mon[37501]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T12:40:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:53 smithi107 ceph-mon[32659]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T12:40:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:53 smithi132 ceph-mon[37580]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-07-26T12:40:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:55 smithi112 ceph-mon[37501]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:40:56.010 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:55 smithi107 ceph-mon[32659]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:40:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:55 smithi132 ceph-mon[37580]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:40:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:57 smithi112 ceph-mon[37501]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:40:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:57 smithi107 ceph-mon[32659]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:40:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:57 smithi132 ceph-mon[37580]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T12:40:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:40:59 smithi107 ceph-mon[32659]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:40:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:40:59 smithi112 ceph-mon[37501]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:40:59 smithi132 ceph-mon[37580]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T12:41:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:00.958 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T12:41:00.958 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T12:41:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:01 smithi107 ceph-mon[32659]: Deploying daemon osd.9 on smithi107 2024-07-26T12:41:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:01 smithi107 ceph-mon[32659]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:01 smithi132 ceph-mon[37580]: Deploying daemon osd.9 on smithi107 2024-07-26T12:41:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:01 smithi132 ceph-mon[37580]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:01 smithi112 ceph-mon[37501]: Deploying daemon osd.9 on smithi107 2024-07-26T12:41:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:01 smithi112 ceph-mon[37501]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:03 smithi107 ceph-mon[32659]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:03 smithi132 ceph-mon[37580]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:03 smithi112 ceph-mon[37501]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:05 smithi107 ceph-mon[32659]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:05 smithi132 ceph-mon[37580]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:05 smithi112 ceph-mon[37501]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:07 smithi107 ceph-mon[32659]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:07 smithi132 ceph-mon[37580]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:07 smithi112 ceph-mon[37501]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:08.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:08.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:09 smithi107 ceph-mon[32659]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:09 smithi132 ceph-mon[37580]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:09 smithi112 ceph-mon[37501]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:11.850 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:11 smithi107 ceph-mon[32659]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:12.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:11 smithi112 ceph-mon[37501]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:11 smithi132 ceph-mon[37580]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:13 smithi107 ceph-mon[32659]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:13 smithi107 ceph-mon[32659]: from='osd.9 [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T12:41:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:13 smithi107 ceph-mon[32659]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T12:41:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:13 smithi112 ceph-mon[37501]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:13 smithi112 ceph-mon[37501]: from='osd.9 [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T12:41:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:13 smithi112 ceph-mon[37501]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T12:41:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:13 smithi132 ceph-mon[37580]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:13 smithi132 ceph-mon[37580]: from='osd.9 [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T12:41:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:13 smithi132 ceph-mon[37580]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T12:41:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:13.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:13.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.946+0000 7fb1f9600640 1 -- 172.21.15.107:0/275147218 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb1fc063230 con 0x7fb1e40b3c20 2024-07-26T12:41:13.948 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 9 on host 'smithi107' 2024-07-26T12:41:13.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.949+0000 7fb202000640 1 -- 172.21.15.107:0/275147218 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1e40b3c20 msgr2=0x7fb1e40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:41:13.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.949+0000 7fb202000640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1e40b3c20 0x7fb1e40b6130 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009bd0 tx=0x7fb1f0005f90 comp rx=0 tx=0).stop 2024-07-26T12:41:13.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.950+0000 7fb202000640 1 -- 172.21.15.107:0/275147218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc10f330 msgr2=0x7fb1fc1a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:41:13.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.950+0000 7fb202000640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc10f330 0x7fb1fc1a0910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb1e802a860 tx=0x7fb1e8032d50 comp rx=0 tx=0).stop 2024-07-26T12:41:13.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.950+0000 7fb202000640 1 -- 172.21.15.107:0/275147218 shutdown_connections 2024-07-26T12:41:13.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.950+0000 7fb202000640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1e40b3c20 0x7fb1e40b6130 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.950+0000 7fb202000640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc10fc80 0x7fb1fc1a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.950+0000 7fb202000640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb1fc10f330 0x7fb1fc1a0910 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.950+0000 7fb202000640 1 --2- 172.21.15.107:0/275147218 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb1fc1099f0 0x7fb1fc1a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.950+0000 7fb202000640 1 -- 172.21.15.107:0/275147218 >> 172.21.15.107:0/275147218 conn(0x7fb1fc078ff0 msgr2=0x7fb1fc0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:41:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.950+0000 7fb202000640 1 -- 172.21.15.107:0/275147218 shutdown_connections 2024-07-26T12:41:13.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:41:13.950+0000 7fb202000640 1 -- 172.21.15.107:0/275147218 wait complete. 2024-07-26T12:41:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:14 smithi107 ceph-mon[32659]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T12:41:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:14 smithi107 ceph-mon[32659]: from='osd.9 [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:41:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:14 smithi107 ceph-mon[32659]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T12:41:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:14 smithi107 ceph-mon[32659]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:41:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:14 smithi132 ceph-mon[37580]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T12:41:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:14 smithi132 ceph-mon[37580]: from='osd.9 [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:41:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:14 smithi132 ceph-mon[37580]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T12:41:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:14 smithi132 ceph-mon[37580]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:41:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:14 smithi112 ceph-mon[37501]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T12:41:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:14 smithi112 ceph-mon[37501]: from='osd.9 [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:41:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:14 smithi112 ceph-mon[37501]: osdmap e58: 10 total, 9 up, 10 in 2024-07-26T12:41:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:14 smithi112 ceph-mon[37501]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-07-26T12:41:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:14.785 DEBUG:teuthology.orchestra.run.smithi107:osd.9> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.9.service 2024-07-26T12:41:14.792 INFO:tasks.cephadm:Deploying osd.10 on smithi112 with /dev/vg_nvme/lv_1... 2024-07-26T12:41:14.792 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T12:41:15.051 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T12:41:15.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:15 smithi107 ceph-mon[32659]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:41:15.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:15.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:15 smithi107 ceph-mon[32659]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T12:41:15.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:15.332 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:15 smithi107 ceph-mon[32659]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:15 smithi112 ceph-mon[37501]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:41:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:15 smithi112 ceph-mon[37501]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T12:41:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:15 smithi112 ceph-mon[37501]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:15 smithi132 ceph-mon[37580]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-07-26T12:41:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:15 smithi132 ceph-mon[37580]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T12:41:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:15 smithi132 ceph-mon[37580]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-07-26T12:41:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:16 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:41:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:16 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:41:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:16 smithi107 ceph-mon[32659]: osd.9 [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] boot 2024-07-26T12:41:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:16 smithi107 ceph-mon[32659]: osdmap e60: 10 total, 10 up, 10 in 2024-07-26T12:41:16.545 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:16 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:41:16.545 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:16 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:41:16.545 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:16.545 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:16 smithi112 ceph-mon[37501]: osd.9 [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] boot 2024-07-26T12:41:16.545 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:16 smithi112 ceph-mon[37501]: osdmap e60: 10 total, 10 up, 10 in 2024-07-26T12:41:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:16 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:41:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:16 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:41:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T12:41:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:16 smithi132 ceph-mon[37580]: osd.9 [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] boot 2024-07-26T12:41:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:16 smithi132 ceph-mon[37580]: osdmap e60: 10 total, 10 up, 10 in 2024-07-26T12:41:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T12:41:16.977 INFO:teuthology.orchestra.run.smithi112.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T12:41:16.990 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10+0 records in 2024-07-26T12:41:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:10+0 records out 2024-07-26T12:41:16.991 INFO:teuthology.orchestra.run.smithi112.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116952 s, 897 MB/s 2024-07-26T12:41:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:--> Zapping successful for: 2024-07-26T12:41:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:17 smithi107 ceph-mon[32659]: osdmap e61: 10 total, 10 up, 10 in 2024-07-26T12:41:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:17 smithi107 ceph-mon[32659]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:17 smithi132 ceph-mon[37580]: osdmap e61: 10 total, 10 up, 10 in 2024-07-26T12:41:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:17 smithi132 ceph-mon[37580]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:17.641 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:17 smithi112 ceph-mon[37501]: osdmap e61: 10 total, 10 up, 10 in 2024-07-26T12:41:17.641 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:17 smithi112 ceph-mon[37501]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:17.675 DEBUG:teuthology.orchestra.run.smithi112:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi112:vg_nvme/lv_1 2024-07-26T12:41:17.931 INFO:teuthology.orchestra.run.smithi112.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T12:41:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.510+0000 7efd261af640 1 -- 172.21.15.112:0/972811238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd201024b0 msgr2=0x7efd201028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:41:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.510+0000 7efd261af640 1 --2- 172.21.15.112:0/972811238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd201024b0 0x7efd201028c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efd10009900 tx=0x7efd1002a410 comp rx=0 tx=0).stop 2024-07-26T12:41:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.510+0000 7efd261af640 1 -- 172.21.15.112:0/972811238 shutdown_connections 2024-07-26T12:41:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.510+0000 7efd261af640 1 --2- 172.21.15.112:0/972811238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20108740 0x7efd2010ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:19.519 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.510+0000 7efd261af640 1 --2- 172.21.15.112:0/972811238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd20107df0 0x7efd20108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.510+0000 7efd261af640 1 --2- 172.21.15.112:0/972811238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd201024b0 0x7efd201028c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.510+0000 7efd261af640 1 -- 172.21.15.112:0/972811238 >> 172.21.15.112:0/972811238 conn(0x7efd20079170 msgr2=0x7efd20079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.510+0000 7efd261af640 1 -- 172.21.15.112:0/972811238 shutdown_connections 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.510+0000 7efd261af640 1 -- 172.21.15.112:0/972811238 wait complete. 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd261af640 1 Processor -- start 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd261af640 1 -- start start 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd261af640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd201024b0 0x7efd20199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd261af640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20107df0 0x7efd2019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd261af640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd20108740 0x7efd2019f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd261af640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efd20112410 con 0x7efd201024b0 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd261af640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efd20112290 con 0x7efd20108740 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd261af640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efd20112590 con 0x7efd20107df0 2024-07-26T12:41:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd1ffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd20108740 0x7efd2019f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd1ffff640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd20108740 0x7efd2019f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.112:53288/0 (socket says 172.21.15.112:53288) 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.511+0000 7efd1ffff640 1 -- 172.21.15.112:0/790029031 learned_addr learned my addr 172.21.15.112:0/790029031 (peer_addr_for_me v2:172.21.15.112:0/0) 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1effd640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20107df0 0x7efd2019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1ffff640 1 -- 172.21.15.112:0/790029031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20107df0 msgr2=0x7efd2019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1ffff640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20107df0 0x7efd2019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1ffff640 1 -- 172.21.15.112:0/790029031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd201024b0 msgr2=0x7efd20199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1ffff640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd201024b0 0x7efd20199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1ffff640 1 -- 172.21.15.112:0/790029031 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2019fef0 con 0x7efd20108740 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1effd640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20107df0 0x7efd2019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1ffff640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd20108740 0x7efd2019f7c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd1800eaa0 tx=0x7efd1800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1cff9640 1 -- 172.21.15.112:0/790029031 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1800cdb0 con 0x7efd20108740 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1cff9640 1 -- 172.21.15.112:0/790029031 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd18005020 con 0x7efd20108740 2024-07-26T12:41:19.521 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd261af640 1 -- 172.21.15.112:0/790029031 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd201a01e0 con 0x7efd20108740 2024-07-26T12:41:19.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd1cff9640 1 -- 172.21.15.112:0/790029031 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd180116d0 con 0x7efd20108740 2024-07-26T12:41:19.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.512+0000 7efd261af640 1 -- 172.21.15.112:0/790029031 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd201a6940 con 0x7efd20108740 2024-07-26T12:41:19.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.513+0000 7efd261af640 1 -- 172.21.15.112:0/790029031 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd201028c0 con 0x7efd20108740 2024-07-26T12:41:19.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.515+0000 7efd1cff9640 1 -- 172.21.15.112:0/790029031 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7efd18004b20 con 0x7efd20108740 2024-07-26T12:41:19.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.515+0000 7efd1cff9640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efd080b3c20 0x7efd080b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:41:19.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.516+0000 7efd1cff9640 1 -- 172.21.15.112:0/790029031 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6476+0+0 (secure 0 0 0) 0x7efd180e37a0 con 0x7efd20108740 2024-07-26T12:41:19.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.516+0000 7efd1f7fe640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efd080b3c20 0x7efd080b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:19.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.517+0000 7efd1f7fe640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efd080b3c20 0x7efd080b6130 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd1002a930 tx=0x7efd10031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:41:19.522 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.520+0000 7efd1cff9640 1 -- 172.21.15.112:0/790029031 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7efd18016030 con 0x7efd20108740 2024-07-26T12:41:19.707 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:19.706+0000 7efd261af640 1 -- 172.21.15.112:0/790029031 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7efd20106ba0 con 0x7efd080b3c20 2024-07-26T12:41:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:19.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:19.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:20.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:20.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:20.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:20 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi107 2024-07-26T12:41:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:20 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 4272M 2024-07-26T12:41:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:20 smithi112 ceph-mon[37501]: from='client.24548 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:41:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:41:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:41:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:20 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi107 2024-07-26T12:41:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:20 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 4272M 2024-07-26T12:41:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:20 smithi107 ceph-mon[32659]: from='client.24548 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:41:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:41:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:41:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:20 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi107 2024-07-26T12:41:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:20 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 4272M 2024-07-26T12:41:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:20 smithi132 ceph-mon[37580]: from='client.24548 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi112:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:41:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:41:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:41:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:21.840 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:21 smithi112 ceph-mon[37501]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:21 smithi107 ceph-mon[32659]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:21 smithi132 ceph-mon[37580]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:22.809 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:22 smithi112 ceph-mon[37501]: from='client.? 172.21.15.112:0/956778900' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5cdd3a2-311a-4c36-8553-3e86dad57015"}]: dispatch 2024-07-26T12:41:22.809 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:22 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5cdd3a2-311a-4c36-8553-3e86dad57015"}]: dispatch 2024-07-26T12:41:22.809 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:22 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5cdd3a2-311a-4c36-8553-3e86dad57015"}]': finished 2024-07-26T12:41:22.809 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:22.809 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:22 smithi112 ceph-mon[37501]: osdmap e62: 11 total, 10 up, 11 in 2024-07-26T12:41:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:22 smithi107 ceph-mon[32659]: from='client.? 172.21.15.112:0/956778900' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5cdd3a2-311a-4c36-8553-3e86dad57015"}]: dispatch 2024-07-26T12:41:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:22 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5cdd3a2-311a-4c36-8553-3e86dad57015"}]: dispatch 2024-07-26T12:41:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:22 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5cdd3a2-311a-4c36-8553-3e86dad57015"}]': finished 2024-07-26T12:41:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:22 smithi107 ceph-mon[32659]: osdmap e62: 11 total, 10 up, 11 in 2024-07-26T12:41:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:22 smithi132 ceph-mon[37580]: from='client.? 172.21.15.112:0/956778900' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5cdd3a2-311a-4c36-8553-3e86dad57015"}]: dispatch 2024-07-26T12:41:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:22 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5cdd3a2-311a-4c36-8553-3e86dad57015"}]: dispatch 2024-07-26T12:41:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:22 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5cdd3a2-311a-4c36-8553-3e86dad57015"}]': finished 2024-07-26T12:41:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:22 smithi132 ceph-mon[37580]: osdmap e62: 11 total, 10 up, 11 in 2024-07-26T12:41:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:23 smithi112 ceph-mon[37501]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:23 smithi112 ceph-mon[37501]: from='client.? 172.21.15.112:0/2412684505' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:41:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:23 smithi107 ceph-mon[32659]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:23 smithi107 ceph-mon[32659]: from='client.? 172.21.15.112:0/2412684505' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:41:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:23 smithi132 ceph-mon[37580]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:23 smithi132 ceph-mon[37580]: from='client.? 172.21.15.112:0/2412684505' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:41:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:25 smithi112 ceph-mon[37501]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:25 smithi107 ceph-mon[32659]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:25 smithi132 ceph-mon[37580]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:27.877 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:27 smithi112 ceph-mon[37501]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:27.965 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:27 smithi107 ceph-mon[32659]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:27 smithi132 ceph-mon[37580]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:29.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:29 smithi112 ceph-mon[37501]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:29.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T12:41:29.940 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:29 smithi107 ceph-mon[32659]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T12:41:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:29 smithi132 ceph-mon[37580]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T12:41:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:30.906 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:30 smithi112 ceph-mon[37501]: Deploying daemon osd.10 on smithi112 2024-07-26T12:41:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:30 smithi107 ceph-mon[32659]: Deploying daemon osd.10 on smithi112 2024-07-26T12:41:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:30 smithi132 ceph-mon[37580]: Deploying daemon osd.10 on smithi112 2024-07-26T12:41:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:31 smithi112 ceph-mon[37501]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:31 smithi107 ceph-mon[32659]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:31 smithi132 ceph-mon[37580]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:33.904 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:33 smithi112 ceph-mon[37501]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:33 smithi107 ceph-mon[32659]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:34.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:33 smithi132 ceph-mon[37580]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:35.005 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:35.005 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:35.005 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:36.041 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:35 smithi112 ceph-mon[37501]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:35 smithi107 ceph-mon[32659]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:35 smithi132 ceph-mon[37580]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:37 smithi107 ceph-mon[32659]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:37 smithi132 ceph-mon[37580]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:38.154 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:37 smithi112 ceph-mon[37501]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:38.154 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:38.154 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:38.154 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:38.155 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:38.155 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:38.221 INFO:teuthology.orchestra.run.smithi112.stdout:Created osd(s) 10 on host 'smithi112' 2024-07-26T12:41:38.222 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.219+0000 7efd1cff9640 1 -- 172.21.15.112:0/790029031 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7efd20106ba0 con 0x7efd080b3c20 2024-07-26T12:41:38.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 -- 172.21.15.112:0/790029031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efd080b3c20 msgr2=0x7efd080b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:41:38.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efd080b3c20 0x7efd080b6130 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efd1002a930 tx=0x7efd10031400 comp rx=0 tx=0).stop 2024-07-26T12:41:38.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 -- 172.21.15.112:0/790029031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd20108740 msgr2=0x7efd2019f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:41:38.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd20108740 0x7efd2019f7c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd1800eaa0 tx=0x7efd1800ef60 comp rx=0 tx=0).stop 2024-07-26T12:41:38.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 -- 172.21.15.112:0/790029031 shutdown_connections 2024-07-26T12:41:38.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efd080b3c20 0x7efd080b6130 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:38.224 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efd20108740 0x7efd2019f7c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:38.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efd20107df0 0x7efd2019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:38.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 --2- 172.21.15.112:0/790029031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd201024b0 0x7efd20199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:38.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 -- 172.21.15.112:0/790029031 >> 172.21.15.112:0/790029031 conn(0x7efd20079170 msgr2=0x7efd20100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:41:38.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 -- 172.21.15.112:0/790029031 shutdown_connections 2024-07-26T12:41:38.225 INFO:teuthology.orchestra.run.smithi112.stderr:2024-07-26T12:41:38.222+0000 7efd261af640 1 -- 172.21.15.112:0/790029031 wait complete. 2024-07-26T12:41:38.887 DEBUG:teuthology.orchestra.run.smithi112:osd.10> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.10.service 2024-07-26T12:41:38.889 INFO:tasks.cephadm:Deploying osd.11 on smithi132 with /dev/vg_nvme/lv_1... 2024-07-26T12:41:38.889 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T12:41:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:38 smithi107 ceph-mon[32659]: from='osd.10 [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T12:41:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:38 smithi107 ceph-mon[32659]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T12:41:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:38 smithi132 ceph-mon[37580]: from='osd.10 [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T12:41:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:38 smithi132 ceph-mon[37580]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T12:41:39.090 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:39.091 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:39.091 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:39.091 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:38 smithi112 ceph-mon[37501]: from='osd.10 [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T12:41:39.091 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:38 smithi112 ceph-mon[37501]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T12:41:39.149 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:41:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:39 smithi107 ceph-mon[32659]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:39 smithi107 ceph-mon[32659]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T12:41:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:39 smithi107 ceph-mon[32659]: from='osd.10 [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:41:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:39 smithi107 ceph-mon[32659]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T12:41:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:39 smithi107 ceph-mon[32659]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:41:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:39 smithi132 ceph-mon[37580]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:39 smithi132 ceph-mon[37580]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T12:41:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:39 smithi132 ceph-mon[37580]: from='osd.10 [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:41:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:39 smithi132 ceph-mon[37580]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T12:41:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:39 smithi132 ceph-mon[37580]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:41:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:39 smithi112 ceph-mon[37501]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:39 smithi112 ceph-mon[37501]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T12:41:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:39 smithi112 ceph-mon[37501]: from='osd.10 [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:41:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:39 smithi112 ceph-mon[37501]: osdmap e63: 11 total, 10 up, 11 in 2024-07-26T12:41:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:39 smithi112 ceph-mon[37501]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]: dispatch 2024-07-26T12:41:41.000 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T12:41:41.085 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T12:41:41.097 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10+0 records in 2024-07-26T12:41:41.097 INFO:teuthology.orchestra.run.smithi132.stderr:10+0 records out 2024-07-26T12:41:41.097 INFO:teuthology.orchestra.run.smithi132.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113768 s, 922 MB/s 2024-07-26T12:41:41.097 INFO:teuthology.orchestra.run.smithi132.stderr:--> Zapping successful for: 2024-07-26T12:41:41.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:41 smithi112 ceph-mon[37501]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:41:41.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:41.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:41 smithi112 ceph-mon[37501]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T12:41:41.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:41.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:41 smithi112 ceph-mon[37501]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:41.278 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:41 smithi112 ceph-mon[37501]: from='osd.10 ' entity='osd.10' 2024-07-26T12:41:41.279 INFO:journalctl@ceph.osd.10.smithi112.stdout:Jul 26 12:41:40 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-10[63624]: 2024-07-26T12:41:40.990+0000 7fdab7cb8640 -1 osd.10 0 waiting for initial osdmap 2024-07-26T12:41:41.279 INFO:journalctl@ceph.osd.10.smithi112.stdout:Jul 26 12:41:41 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-10[63624]: 2024-07-26T12:41:41.003+0000 7fdaafaae640 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T12:41:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:41 smithi132 ceph-mon[37580]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:41:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:41 smithi132 ceph-mon[37580]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T12:41:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:41 smithi132 ceph-mon[37580]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:41 smithi132 ceph-mon[37580]: from='osd.10 ' entity='osd.10' 2024-07-26T12:41:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:41 smithi107 ceph-mon[32659]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi112", "root=default"]}]': finished 2024-07-26T12:41:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:41 smithi107 ceph-mon[32659]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T12:41:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:41 smithi107 ceph-mon[32659]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T12:41:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:41 smithi107 ceph-mon[32659]: from='osd.10 ' entity='osd.10' 2024-07-26T12:41:41.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi132:vg_nvme/lv_1 2024-07-26T12:41:42.038 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:41:42.488 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:41:42.488 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:41:42.488 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:42.488 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: osd.10 [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] boot 2024-07-26T12:41:42.488 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: osdmap e65: 11 total, 11 up, 11 in 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.489 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:42.490 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:42.490 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: osd.10 [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] boot 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: osdmap e65: 11 total, 11 up, 11 in 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:42.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:42.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: osd.10 [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] boot 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: osdmap e65: 11 total, 11 up, 11 in 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: osdmap e66: 11 total, 11 up, 11 in 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:43.466 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:43 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi112 2024-07-26T12:41:43.467 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:43 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 4272M 2024-07-26T12:41:43.467 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:43 smithi112 ceph-mon[37501]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T12:41:43.467 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:43 smithi112 ceph-mon[37501]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 170 KiB/s, 0 objects/s recovering 2024-07-26T12:41:43.470 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:43 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi112 2024-07-26T12:41:43.470 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:43 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 4272M 2024-07-26T12:41:43.470 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:43 smithi132 ceph-mon[37580]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T12:41:43.470 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:43 smithi132 ceph-mon[37580]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 170 KiB/s, 0 objects/s recovering 2024-07-26T12:41:43.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.684+0000 7f95820d0640 1 -- 172.21.15.132:0/2308756256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f957c108760 msgr2=0x7f957c10ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:41:43.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.684+0000 7f95820d0640 1 --2- 172.21.15.132:0/2308756256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f957c108760 0x7f957c10ef90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9574009ca0 tx=0x7f957402a8e0 comp rx=0 tx=0).stop 2024-07-26T12:41:43.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.684+0000 7f95820d0640 1 -- 172.21.15.132:0/2308756256 shutdown_connections 2024-07-26T12:41:43.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.684+0000 7f95820d0640 1 --2- 172.21.15.132:0/2308756256 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f957c108760 0x7f957c10ef90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:43.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.684+0000 7f95820d0640 1 --2- 172.21.15.132:0/2308756256 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f957c107e10 0x7f957c108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:43.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.684+0000 7f95820d0640 1 --2- 172.21.15.132:0/2308756256 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f957c1024d0 0x7f957c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:43.688 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.684+0000 7f95820d0640 1 -- 172.21.15.132:0/2308756256 >> 172.21.15.132:0/2308756256 conn(0x7f957c079180 msgr2=0x7f957c079590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.684+0000 7f95820d0640 1 -- 172.21.15.132:0/2308756256 shutdown_connections 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.684+0000 7f95820d0640 1 -- 172.21.15.132:0/2308756256 wait complete. 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f95820d0640 1 Processor -- start 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f95820d0640 1 -- start start 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f95820d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f957c1024d0 0x7f957c19acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f95820d0640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f957c107e10 0x7f957c19b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f95820d0640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f957c108760 0x7f957c19b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f95820d0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f957c112460 con 0x7f957c1024d0 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f95820d0640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f957c1122e0 con 0x7f957c107e10 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f95820d0640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f957c1125e0 con 0x7f957c108760 2024-07-26T12:41:43.689 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f957bfff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f957c108760 0x7f957c19b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f957bfff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f957c108760 0x7f957c19b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56920/0 (socket says 172.21.15.132:56920) 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f957bfff640 1 -- 172.21.15.132:0/276220939 learned_addr learned my addr 172.21.15.132:0/276220939 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.685+0000 7f9572ffd640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f957c107e10 0x7f957c19b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f957b7fe640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f957c1024d0 0x7f957c19acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f957bfff640 1 -- 172.21.15.132:0/276220939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f957c107e10 msgr2=0x7f957c19b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f957bfff640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f957c107e10 0x7f957c19b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f957bfff640 1 -- 172.21.15.132:0/276220939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f957c1024d0 msgr2=0x7f957c19acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f957bfff640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f957c1024d0 0x7f957c19acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f957bfff640 1 -- 172.21.15.132:0/276220939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957c1a5f80 con 0x7f957c108760 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f9572ffd640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f957c107e10 0x7f957c19b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f957bfff640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f957c108760 0x7f957c19b770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9574005cc0 tx=0x7f95740039f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:41:43.690 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f95797fa640 1 -- 172.21.15.132:0/276220939 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9574003de0 con 0x7f957c108760 2024-07-26T12:41:43.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f95820d0640 1 -- 172.21.15.132:0/276220939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957c1a6210 con 0x7f957c108760 2024-07-26T12:41:43.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f95797fa640 1 -- 172.21.15.132:0/276220939 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95740044f0 con 0x7f957c108760 2024-07-26T12:41:43.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f95797fa640 1 -- 172.21.15.132:0/276220939 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9574030cb0 con 0x7f957c108760 2024-07-26T12:41:43.691 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.686+0000 7f95820d0640 1 -- 172.21.15.132:0/276220939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f957c1a6750 con 0x7f957c108760 2024-07-26T12:41:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.688+0000 7f95820d0640 1 -- 172.21.15.132:0/276220939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f957c1028e0 con 0x7f957c108760 2024-07-26T12:41:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.689+0000 7f95797fa640 1 -- 172.21.15.132:0/276220939 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f9574004040 con 0x7f957c108760 2024-07-26T12:41:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.689+0000 7f95797fa640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f95680b3c50 0x7f95680b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:41:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.689+0000 7f95797fa640 1 -- 172.21.15.132:0/276220939 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6909+0+0 (secure 0 0 0) 0x7f9574104aa0 con 0x7f957c108760 2024-07-26T12:41:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.690+0000 7f957b7fe640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f95680b3c50 0x7f95680b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:41:43.692 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.691+0000 7f957b7fe640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f95680b3c50 0x7f95680b6160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f956c005fd0 tx=0x7f956c005c70 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:41:43.695 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.694+0000 7f95797fa640 1 -- 172.21.15.132:0/276220939 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f95740cc670 con 0x7f957c108760 2024-07-26T12:41:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:43 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi112 2024-07-26T12:41:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:43 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 4272M 2024-07-26T12:41:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:43 smithi107 ceph-mon[32659]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T12:41:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:43 smithi107 ceph-mon[32659]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 170 KiB/s, 0 objects/s recovering 2024-07-26T12:41:43.878 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:41:43.877+0000 7f95820d0640 1 -- 172.21.15.132:0/276220939 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f957c063230 con 0x7f95680b3c50 2024-07-26T12:41:44.615 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:41:44.616 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:41:44.616 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:41:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:41:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T12:41:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T12:41:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:45.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:45 smithi132 ceph-mon[37580]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:41:45.649 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:45 smithi132 ceph-mon[37580]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-07-26T12:41:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:45 smithi112 ceph-mon[37501]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:41:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:45 smithi112 ceph-mon[37501]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-07-26T12:41:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:45 smithi107 ceph-mon[32659]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi132:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:41:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:45 smithi107 ceph-mon[32659]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-07-26T12:41:46.755 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:46 smithi132 ceph-mon[37580]: from='client.? 172.21.15.132:0/4167121391' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d738c87c-2b11-4ba4-b38d-a58e2fd9a235"}]: dispatch 2024-07-26T12:41:46.756 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:46 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d738c87c-2b11-4ba4-b38d-a58e2fd9a235"}]: dispatch 2024-07-26T12:41:46.756 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:46 smithi132 ceph-mon[37580]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d738c87c-2b11-4ba4-b38d-a58e2fd9a235"}]': finished 2024-07-26T12:41:46.756 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:41:46.756 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:46 smithi132 ceph-mon[37580]: osdmap e68: 12 total, 11 up, 12 in 2024-07-26T12:41:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:46 smithi107 ceph-mon[32659]: from='client.? 172.21.15.132:0/4167121391' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d738c87c-2b11-4ba4-b38d-a58e2fd9a235"}]: dispatch 2024-07-26T12:41:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:46 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d738c87c-2b11-4ba4-b38d-a58e2fd9a235"}]: dispatch 2024-07-26T12:41:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:46 smithi107 ceph-mon[32659]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d738c87c-2b11-4ba4-b38d-a58e2fd9a235"}]': finished 2024-07-26T12:41:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:41:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:46 smithi107 ceph-mon[32659]: osdmap e68: 12 total, 11 up, 12 in 2024-07-26T12:41:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:46 smithi112 ceph-mon[37501]: from='client.? 172.21.15.132:0/4167121391' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d738c87c-2b11-4ba4-b38d-a58e2fd9a235"}]: dispatch 2024-07-26T12:41:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:46 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d738c87c-2b11-4ba4-b38d-a58e2fd9a235"}]: dispatch 2024-07-26T12:41:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:46 smithi112 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d738c87c-2b11-4ba4-b38d-a58e2fd9a235"}]': finished 2024-07-26T12:41:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:41:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:46 smithi112 ceph-mon[37501]: osdmap e68: 12 total, 11 up, 12 in 2024-07-26T12:41:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:47 smithi107 ceph-mon[32659]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-07-26T12:41:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:47 smithi107 ceph-mon[32659]: from='client.? 172.21.15.132:0/4127040802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:41:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:47 smithi132 ceph-mon[37580]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-07-26T12:41:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:47 smithi132 ceph-mon[37580]: from='client.? 172.21.15.132:0/4127040802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:41:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:47 smithi112 ceph-mon[37501]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-07-26T12:41:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:47 smithi112 ceph-mon[37501]: from='client.? 172.21.15.132:0/4127040802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T12:41:49.939 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:49 smithi132 ceph-mon[37580]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T12:41:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:49 smithi112 ceph-mon[37501]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T12:41:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:49 smithi107 ceph-mon[32659]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-07-26T12:41:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:51 smithi132 ceph-mon[37580]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-07-26T12:41:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:51 smithi112 ceph-mon[37501]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-07-26T12:41:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:51 smithi107 ceph-mon[32659]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-07-26T12:41:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:53 smithi112 ceph-mon[37501]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:41:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T12:41:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:53 smithi107 ceph-mon[32659]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:41:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T12:41:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:54.053 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:53 smithi132 ceph-mon[37580]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:41:54.053 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T12:41:54.053 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:54 smithi112 ceph-mon[37501]: Deploying daemon osd.11 on smithi132 2024-07-26T12:41:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:54 smithi107 ceph-mon[32659]: Deploying daemon osd.11 on smithi132 2024-07-26T12:41:55.053 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:54 smithi132 ceph-mon[37580]: Deploying daemon osd.11 on smithi132 2024-07-26T12:41:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:55 smithi112 ceph-mon[37501]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:41:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:55 smithi107 ceph-mon[32659]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:41:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:55 smithi132 ceph-mon[37580]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:41:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:57 smithi107 ceph-mon[32659]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:41:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:57 smithi132 ceph-mon[37580]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:41:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:57 smithi112 ceph-mon[37501]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:41:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:59 smithi132 ceph-mon[37580]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:41:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:41:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:41:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:59 smithi107 ceph-mon[32659]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:41:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:59 smithi112 ceph-mon[37501]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:41:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:01 smithi107 ceph-mon[32659]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:01 smithi132 ceph-mon[37580]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:01 smithi112 ceph-mon[37501]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:02.501 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.499+0000 7f95797fa640 1 -- 172.21.15.132:0/276220939 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f957c063230 con 0x7f95680b3c50 2024-07-26T12:42:02.501 INFO:teuthology.orchestra.run.smithi132.stdout:Created osd(s) 11 on host 'smithi132' 2024-07-26T12:42:02.503 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 -- 172.21.15.132:0/276220939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f95680b3c50 msgr2=0x7f95680b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f95680b3c50 0x7f95680b6160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f956c005fd0 tx=0x7f956c005c70 comp rx=0 tx=0).stop 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 -- 172.21.15.132:0/276220939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f957c108760 msgr2=0x7f957c19b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f957c108760 0x7f957c19b770 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9574005cc0 tx=0x7f95740039f0 comp rx=0 tx=0).stop 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 -- 172.21.15.132:0/276220939 shutdown_connections 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f95680b3c50 0x7f95680b6160 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f957c108760 0x7f957c19b770 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f957c107e10 0x7f957c19b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 --2- 172.21.15.132:0/276220939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f957c1024d0 0x7f957c19acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 -- 172.21.15.132:0/276220939 >> 172.21.15.132:0/276220939 conn(0x7f957c079180 msgr2=0x7f957c105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 -- 172.21.15.132:0/276220939 shutdown_connections 2024-07-26T12:42:02.504 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:02.502+0000 7f95820d0640 1 -- 172.21.15.132:0/276220939 wait complete. 2024-07-26T12:42:03.121 DEBUG:teuthology.orchestra.run.smithi132:osd.11> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@osd.11.service 2024-07-26T12:42:03.124 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-07-26T12:42:03.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd stat -f json 2024-07-26T12:42:03.597 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:03 smithi107 ceph-mon[32659]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:03.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:03 smithi107 ceph-mon[32659]: from='osd.11 [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T12:42:03.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:03 smithi107 ceph-mon[32659]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T12:42:03.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.769 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.769 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.769 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:03 smithi132 ceph-mon[37580]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:03.770 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:03 smithi132 ceph-mon[37580]: from='osd.11 [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T12:42:03.770 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:03 smithi132 ceph-mon[37580]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T12:42:03.770 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.770 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:03 smithi112 ceph-mon[37501]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:03 smithi112 ceph-mon[37501]: from='osd.11 [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T12:42:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:03 smithi112 ceph-mon[37501]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T12:42:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:04.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:04 smithi107 ceph-mon[32659]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T12:42:04.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:04.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:04 smithi107 ceph-mon[32659]: from='osd.11 [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:42:04.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:04 smithi107 ceph-mon[32659]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T12:42:04.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:04 smithi107 ceph-mon[32659]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:42:04.787 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:04 smithi132 ceph-mon[37580]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T12:42:04.787 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:04.788 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:04 smithi132 ceph-mon[37580]: from='osd.11 [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:42:04.788 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:04 smithi132 ceph-mon[37580]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T12:42:04.788 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:04 smithi132 ceph-mon[37580]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:42:04.788 INFO:journalctl@ceph.osd.11.smithi132.stdout:Jul 26 12:42:04 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-11[64894]: 2024-07-26T12:42:04.514+0000 7f9052b5a640 -1 osd.11 0 waiting for initial osdmap 2024-07-26T12:42:04.788 INFO:journalctl@ceph.osd.11.smithi132.stdout:Jul 26 12:42:04 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-osd-11[64894]: 2024-07-26T12:42:04.527+0000 7f904e958640 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T12:42:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:04 smithi112 ceph-mon[37501]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T12:42:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:04 smithi112 ceph-mon[37501]: from='osd.11 [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:42:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:04 smithi112 ceph-mon[37501]: osdmap e69: 12 total, 11 up, 12 in 2024-07-26T12:42:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:04 smithi112 ceph-mon[37501]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]: dispatch 2024-07-26T12:42:05.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:05 smithi107 ceph-mon[32659]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:42:05.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:05.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:05 smithi107 ceph-mon[32659]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T12:42:05.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:05 smithi107 ceph-mon[32659]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:05.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:05 smithi132 ceph-mon[37580]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:42:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:05 smithi132 ceph-mon[37580]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T12:42:05.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:05 smithi132 ceph-mon[37580]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:05.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:05 smithi112 ceph-mon[37501]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi132", "root=default"]}]': finished 2024-07-26T12:42:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:05 smithi112 ceph-mon[37501]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T12:42:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:05 smithi112 ceph-mon[37501]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-07-26T12:42:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:05.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.969+0000 7f20efe00640 1 -- 172.21.15.107:0/3528875129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f01055d0 msgr2=0x7f20f01059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:05.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.969+0000 7f20efe00640 1 --2- 172.21.15.107:0/3528875129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f01055d0 0x7f20f01059e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009900 tx=0x7f20e002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:42:05.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.969+0000 7f20efe00640 1 -- 172.21.15.107:0/3528875129 shutdown_connections 2024-07-26T12:42:05.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.969+0000 7f20efe00640 1 --2- 172.21.15.107:0/3528875129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f010b860 0x7f20f0112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:05.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.969+0000 7f20efe00640 1 --2- 172.21.15.107:0/3528875129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20f010af10 0x7f20f010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:05.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.969+0000 7f20efe00640 1 --2- 172.21.15.107:0/3528875129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f01055d0 0x7f20f01059e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:05.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.969+0000 7f20efe00640 1 -- 172.21.15.107:0/3528875129 >> 172.21.15.107:0/3528875129 conn(0x7f20f0101040 msgr2=0x7f20f0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:05.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.969+0000 7f20efe00640 1 -- 172.21.15.107:0/3528875129 shutdown_connections 2024-07-26T12:42:05.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.969+0000 7f20efe00640 1 -- 172.21.15.107:0/3528875129 wait complete. 2024-07-26T12:42:05.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.971+0000 7f20efe00640 1 Processor -- start 2024-07-26T12:42:05.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.971+0000 7f20efe00640 1 -- start start 2024-07-26T12:42:05.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.971+0000 7f20efe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f01055d0 0x7f20f019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:05.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.971+0000 7f20efe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20f010af10 0x7f20f019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:05.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.971+0000 7f20efe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f010b860 0x7f20f01a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:05.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.971+0000 7f20efe00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0115560 con 0x7f20f01055d0 2024-07-26T12:42:05.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.971+0000 7f20efe00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f20f01153e0 con 0x7f20f010af10 2024-07-26T12:42:05.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.971+0000 7f20efe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f20f01156e0 con 0x7f20f010b860 2024-07-26T12:42:05.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20ee000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20f010af10 0x7f20f019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:05.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20ee000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20f010af10 0x7f20f019c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57454/0 (socket says 172.21.15.107:57454) 2024-07-26T12:42:05.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20ef400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f010b860 0x7f20f01a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:05.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20ee000640 1 -- 172.21.15.107:0/3269384733 learned_addr learned my addr 172.21.15.107:0/3269384733 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:42:05.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20eea00640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f01055d0 0x7f20f019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:05.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20ef400640 1 -- 172.21.15.107:0/3269384733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20f010af10 msgr2=0x7f20f019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20ef400640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20f010af10 0x7f20f019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20ef400640 1 -- 172.21.15.107:0/3269384733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f01055d0 msgr2=0x7f20f019bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20ef400640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f01055d0 0x7f20f019bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20ef400640 1 -- 172.21.15.107:0/3269384733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f01a1ec0 con 0x7f20f010b860 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.972+0000 7f20ee000640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20f010af10 0x7f20f019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.973+0000 7f20ef400640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f010b860 0x7f20f01a1790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f20e8009fd0 tx=0x7f20e800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.973+0000 7f20e7400640 1 -- 172.21.15.107:0/3269384733 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e800ce20 con 0x7f20f010b860 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.973+0000 7f20e7400640 1 -- 172.21.15.107:0/3269384733 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20e8005020 con 0x7f20f010b860 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.973+0000 7f20efe00640 1 -- 172.21.15.107:0/3269384733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f01a21b0 con 0x7f20f010b860 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.973+0000 7f20e7400640 1 -- 172.21.15.107:0/3269384733 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e8010690 con 0x7f20f010b860 2024-07-26T12:42:05.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.973+0000 7f20efe00640 1 -- 172.21.15.107:0/3269384733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f01a8910 con 0x7f20f010b860 2024-07-26T12:42:05.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.974+0000 7f20efe00640 1 -- 172.21.15.107:0/3269384733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f01059e0 con 0x7f20f010b860 2024-07-26T12:42:05.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.976+0000 7f20e7400640 1 -- 172.21.15.107:0/3269384733 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f20e80054d0 con 0x7f20f010b860 2024-07-26T12:42:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.978+0000 7f20e7400640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f20d40b3ca0 0x7f20d40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.978+0000 7f20e7400640 1 -- 172.21.15.107:0/3269384733 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7342+0+0 (secure 0 0 0) 0x7f20e80e3f50 con 0x7f20f010b860 2024-07-26T12:42:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.978+0000 7f20eea00640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f20d40b3ca0 0x7f20d40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:05.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.979+0000 7f20eea00640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f20d40b3ca0 0x7f20d40b61b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f20e0005ab0 tx=0x7f20e0005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:05.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:05.982+0000 7f20e7400640 1 -- 172.21.15.107:0/3269384733 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f20e80ab8d0 con 0x7f20f010b860 2024-07-26T12:42:06.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.187+0000 7f20efe00640 1 -- 172.21.15.107:0/3269384733 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f20f010cfd0 con 0x7f20f010b860 2024-07-26T12:42:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.188+0000 7f20e7400640 1 -- 172.21.15.107:0/3269384733 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v71) ==== 74+0+133 (secure 0 0 0) 0x7f20e80ab1d0 con 0x7f20f010b860 2024-07-26T12:42:06.190 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:42:06.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 -- 172.21.15.107:0/3269384733 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f20d40b3ca0 msgr2=0x7f20d40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:06.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f20d40b3ca0 0x7f20d40b61b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f20e0005ab0 tx=0x7f20e0005a20 comp rx=0 tx=0).stop 2024-07-26T12:42:06.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 -- 172.21.15.107:0/3269384733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f010b860 msgr2=0x7f20f01a1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:06.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f010b860 0x7f20f01a1790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f20e8009fd0 tx=0x7f20e800ef10 comp rx=0 tx=0).stop 2024-07-26T12:42:06.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 -- 172.21.15.107:0/3269384733 shutdown_connections 2024-07-26T12:42:06.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f20d40b3ca0 0x7f20d40b61b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:06.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f20f010b860 0x7f20f01a1790 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:06.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f20f010af10 0x7f20f019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:06.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 --2- 172.21.15.107:0/3269384733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20f01055d0 0x7f20f019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:06.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 -- 172.21.15.107:0/3269384733 >> 172.21.15.107:0/3269384733 conn(0x7f20f0101040 msgr2=0x7f20f0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:06.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.191+0000 7f20efe00640 1 -- 172.21.15.107:0/3269384733 shutdown_connections 2024-07-26T12:42:06.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:06.192+0000 7f20efe00640 1 -- 172.21.15.107:0/3269384733 wait complete. 2024-07-26T12:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:06 smithi132 ceph-mon[37580]: purged_snaps scrub starts 2024-07-26T12:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:06 smithi132 ceph-mon[37580]: purged_snaps scrub ok 2024-07-26T12:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:06 smithi132 ceph-mon[37580]: osd.11 [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] boot 2024-07-26T12:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:06 smithi132 ceph-mon[37580]: osdmap e71: 12 total, 12 up, 12 in 2024-07-26T12:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:06 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3269384733' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T12:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:06 smithi112 ceph-mon[37501]: purged_snaps scrub starts 2024-07-26T12:42:06.978 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:06 smithi112 ceph-mon[37501]: purged_snaps scrub ok 2024-07-26T12:42:06.979 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:06.979 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:06 smithi112 ceph-mon[37501]: osd.11 [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] boot 2024-07-26T12:42:06.979 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:06 smithi112 ceph-mon[37501]: osdmap e71: 12 total, 12 up, 12 in 2024-07-26T12:42:06.979 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:06 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3269384733' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T12:42:06.979 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:06.979 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:06.979 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:06.979 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:06 smithi107 ceph-mon[32659]: purged_snaps scrub starts 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:06 smithi107 ceph-mon[32659]: purged_snaps scrub ok 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:06 smithi107 ceph-mon[32659]: osd.11 [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] boot 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:06 smithi107 ceph-mon[32659]: osdmap e71: 12 total, 12 up, 12 in 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:06 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3269384733' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:06.980 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.029 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":71,"num_osds":12,"num_up_osds":12,"osd_up_since":1721997725,"num_in_osds":12,"osd_in_since":1721997706,"num_remapped_pgs":0} 2024-07-26T12:42:07.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:42:07.487 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:42:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: Detected new or changed devices on smithi132 2024-07-26T12:42:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 4272M 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:07.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: Detected new or changed devices on smithi132 2024-07-26T12:42:07.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T12:42:07.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 4272M 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:07.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: Detected new or changed devices on smithi132 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: osdmap e72: 12 total, 12 up, 12 in 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 4272M 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:07.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:07.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:07.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:07.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:07.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.724+0000 7f2ab9a00640 1 -- 172.21.15.107:0/1885247439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab410f330 msgr2=0x7f2ab410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.724+0000 7f2ab9a00640 1 --2- 172.21.15.107:0/1885247439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab410f330 0x7f2ab410f740 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c009900 tx=0x7f2a9c02a410 comp rx=0 tx=0).stop 2024-07-26T12:42:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.724+0000 7f2ab9a00640 1 -- 172.21.15.107:0/1885247439 shutdown_connections 2024-07-26T12:42:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.724+0000 7f2ab9a00640 1 --2- 172.21.15.107:0/1885247439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ab410fc80 0x7f2ab41164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.724+0000 7f2ab9a00640 1 --2- 172.21.15.107:0/1885247439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab410f330 0x7f2ab410f740 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.724+0000 7f2ab9a00640 1 --2- 172.21.15.107:0/1885247439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ab41099f0 0x7f2ab4109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.724+0000 7f2ab9a00640 1 -- 172.21.15.107:0/1885247439 >> 172.21.15.107:0/1885247439 conn(0x7f2ab4078ff0 msgr2=0x7f2ab4107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.724+0000 7f2ab9a00640 1 -- 172.21.15.107:0/1885247439 shutdown_connections 2024-07-26T12:42:09.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.724+0000 7f2ab9a00640 1 -- 172.21.15.107:0/1885247439 wait complete. 2024-07-26T12:42:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.725+0000 7f2ab9a00640 1 Processor -- start 2024-07-26T12:42:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.726+0000 7f2ab9a00640 1 -- start start 2024-07-26T12:42:09.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.726+0000 7f2ab9a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab41099f0 0x7f2ab41a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.726+0000 7f2ab9a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ab410f330 0x7f2ab41a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.726+0000 7f2ab9a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ab410fc80 0x7f2ab41a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.726+0000 7f2ab9a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab4119990 con 0x7f2ab41099f0 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.726+0000 7f2ab9a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab4119810 con 0x7f2ab410f330 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.726+0000 7f2ab9a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab4119b10 con 0x7f2ab410fc80 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.726+0000 7f2ab3e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab41099f0 0x7f2ab41a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.726+0000 7f2ab3e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab41099f0 0x7f2ab41a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40250/0 (socket says 172.21.15.107:40250) 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.726+0000 7f2ab3e00640 1 -- 172.21.15.107:0/2114158446 learned_addr learned my addr 172.21.15.107:0/2114158446 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.727+0000 7f2ab9000640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ab410fc80 0x7f2ab41a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.727+0000 7f2aab400640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ab410f330 0x7f2ab41a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.727+0000 7f2ab3e00640 1 -- 172.21.15.107:0/2114158446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ab410fc80 msgr2=0x7f2ab41a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:09.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.727+0000 7f2ab3e00640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ab410fc80 0x7f2ab41a5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.727+0000 7f2ab3e00640 1 -- 172.21.15.107:0/2114158446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ab410f330 msgr2=0x7f2ab41a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.727+0000 7f2ab3e00640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ab410f330 0x7f2ab41a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.727+0000 7f2ab3e00640 1 -- 172.21.15.107:0/2114158446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab41a62d0 con 0x7f2ab41099f0 2024-07-26T12:42:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.727+0000 7f2ab3e00640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab41099f0 0x7f2ab41a0360 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400ea60 tx=0x7f2aa400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.727+0000 7f2ab1600640 1 -- 172.21.15.107:0/2114158446 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa400cde0 con 0x7f2ab41099f0 2024-07-26T12:42:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.728+0000 7f2ab1600640 1 -- 172.21.15.107:0/2114158446 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2aa40035f0 con 0x7f2ab41099f0 2024-07-26T12:42:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.728+0000 7f2ab1600640 1 -- 172.21.15.107:0/2114158446 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4010690 con 0x7f2ab41099f0 2024-07-26T12:42:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.728+0000 7f2ab9a00640 1 -- 172.21.15.107:0/2114158446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab41a65c0 con 0x7f2ab41099f0 2024-07-26T12:42:09.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.728+0000 7f2ab9a00640 1 -- 172.21.15.107:0/2114158446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab41acd20 con 0x7f2ab41099f0 2024-07-26T12:42:09.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.729+0000 7f2ab9a00640 1 -- 172.21.15.107:0/2114158446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab4109e00 con 0x7f2ab41099f0 2024-07-26T12:42:09.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.732+0000 7f2ab1600640 1 -- 172.21.15.107:0/2114158446 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f2aa40037a0 con 0x7f2ab41099f0 2024-07-26T12:42:09.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.734+0000 7f2ab1600640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2a980b3ca0 0x7f2a980b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:09.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.735+0000 7f2ab1600640 1 -- 172.21.15.107:0/2114158446 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f2aa40e3d10 con 0x7f2ab41099f0 2024-07-26T12:42:09.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.735+0000 7f2aab400640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2a980b3ca0 0x7f2a980b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:09.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.735+0000 7f2aab400640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2a980b3ca0 0x7f2a980b61b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c02aa00 tx=0x7f2a9c005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:09.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.737+0000 7f2ab1600640 1 -- 172.21.15.107:0/2114158446 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2aa4055600 con 0x7f2ab41099f0 2024-07-26T12:42:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:09 smithi107 ceph-mon[32659]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:09 smithi132 ceph-mon[37580]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:09.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.938+0000 7f2ab9a00640 1 -- 172.21.15.107:0/2114158446 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ab41a1490 con 0x7f2ab41099f0 2024-07-26T12:42:09.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.939+0000 7f2ab1600640 1 -- 172.21.15.107:0/2114158446 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18816 (secure 0 0 0) 0x7f2aa40ab690 con 0x7f2ab41099f0 2024-07-26T12:42:09.942 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:42:09.942 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":72,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:42:06.534450+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:42:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.942+0000 7f2ab9a00640 1 -- 172.21.15.107:0/2114158446 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2a980b3ca0 msgr2=0x7f2a980b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.942+0000 7f2ab9a00640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2a980b3ca0 0x7f2a980b61b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c02aa00 tx=0x7f2a9c005a20 comp rx=0 tx=0).stop 2024-07-26T12:42:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.942+0000 7f2ab9a00640 1 -- 172.21.15.107:0/2114158446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab41099f0 msgr2=0x7f2ab41a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:09.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.942+0000 7f2ab9a00640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab41099f0 0x7f2ab41a0360 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400ea60 tx=0x7f2aa400ef20 comp rx=0 tx=0).stop 2024-07-26T12:42:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.942+0000 7f2ab9a00640 1 -- 172.21.15.107:0/2114158446 shutdown_connections 2024-07-26T12:42:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.942+0000 7f2ab9a00640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2a980b3ca0 0x7f2a980b61b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.942+0000 7f2ab9a00640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2ab410fc80 0x7f2ab41a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.943+0000 7f2ab9a00640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2ab410f330 0x7f2ab41a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.943+0000 7f2ab9a00640 1 --2- 172.21.15.107:0/2114158446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab41099f0 0x7f2ab41a0360 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.943+0000 7f2ab9a00640 1 -- 172.21.15.107:0/2114158446 >> 172.21.15.107:0/2114158446 conn(0x7f2ab4078ff0 msgr2=0x7f2ab40797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.943+0000 7f2ab9a00640 1 -- 172.21.15.107:0/2114158446 shutdown_connections 2024-07-26T12:42:09.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:09.943+0000 7f2ab9a00640 1 -- 172.21.15.107:0/2114158446 wait complete. 2024-07-26T12:42:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:09 smithi112 ceph-mon[37501]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:10.851 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T12:37:58.631844+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T12:42:10.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-07-26T12:42:10.934 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:10 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2114158446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:42:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:10 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2114158446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:42:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:10 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2114158446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:42:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:42:11.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:11 smithi107 ceph-mon[32659]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:11 smithi112 ceph-mon[37501]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:11 smithi132 ceph-mon[37580]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:13.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.593+0000 7fc5cce00640 1 -- 172.21.15.107:0/704582942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5c810b860 msgr2=0x7fc5c8112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:13.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.593+0000 7fc5cce00640 1 --2- 172.21.15.107:0/704582942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5c810b860 0x7fc5c8112090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009ca0 tx=0x7fc5c002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.593+0000 7fc5cce00640 1 -- 172.21.15.107:0/704582942 shutdown_connections 2024-07-26T12:42:13.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.593+0000 7fc5cce00640 1 --2- 172.21.15.107:0/704582942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5c810b860 0x7fc5c8112090 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.593+0000 7fc5cce00640 1 --2- 172.21.15.107:0/704582942 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5c810af10 0x7fc5c810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.593+0000 7fc5cce00640 1 --2- 172.21.15.107:0/704582942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c81055d0 0x7fc5c81059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.593+0000 7fc5cce00640 1 -- 172.21.15.107:0/704582942 >> 172.21.15.107:0/704582942 conn(0x7fc5c8101040 msgr2=0x7fc5c8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:13.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.593+0000 7fc5cce00640 1 -- 172.21.15.107:0/704582942 shutdown_connections 2024-07-26T12:42:13.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.593+0000 7fc5cce00640 1 -- 172.21.15.107:0/704582942 wait complete. 2024-07-26T12:42:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.594+0000 7fc5cce00640 1 Processor -- start 2024-07-26T12:42:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.594+0000 7fc5cce00640 1 -- start start 2024-07-26T12:42:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5cce00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5c81055d0 0x7fc5c819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5cce00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c810af10 0x7fc5c819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:13.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5cce00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5c810b860 0x7fc5c81a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5cce00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c8115530 con 0x7fc5c81055d0 2024-07-26T12:42:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5cce00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c81153b0 con 0x7fc5c810b860 2024-07-26T12:42:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5cce00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c81156b0 con 0x7fc5c810af10 2024-07-26T12:42:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5c7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5c81055d0 0x7fc5c819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5c7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5c81055d0 0x7fc5c819bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40280/0 (socket says 172.21.15.107:40280) 2024-07-26T12:42:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5c7400640 1 -- 172.21.15.107:0/2844470723 learned_addr learned my addr 172.21.15.107:0/2844470723 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:42:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5bea00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c810af10 0x7fc5c819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:13.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.595+0000 7fc5c7e00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5c810b860 0x7fc5c81a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.596+0000 7fc5c7400640 1 -- 172.21.15.107:0/2844470723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c810af10 msgr2=0x7fc5c819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.596+0000 7fc5c7400640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c810af10 0x7fc5c819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.596+0000 7fc5c7400640 1 -- 172.21.15.107:0/2844470723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5c810b860 msgr2=0x7fc5c81a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.596+0000 7fc5c7400640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5c810b860 0x7fc5c81a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.596+0000 7fc5c7400640 1 -- 172.21.15.107:0/2844470723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c81a1eb0 con 0x7fc5c81055d0 2024-07-26T12:42:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.596+0000 7fc5c7400640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5c81055d0 0x7fc5c819bf40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800e9f0 tx=0x7fc5b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.596+0000 7fc5c4c00640 1 -- 172.21.15.107:0/2844470723 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800cd80 con 0x7fc5c81055d0 2024-07-26T12:42:13.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.596+0000 7fc5c4c00640 1 -- 172.21.15.107:0/2844470723 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5b80054d0 con 0x7fc5c81055d0 2024-07-26T12:42:13.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.596+0000 7fc5cce00640 1 -- 172.21.15.107:0/2844470723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c81a21a0 con 0x7fc5c81055d0 2024-07-26T12:42:13.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.597+0000 7fc5c4c00640 1 -- 172.21.15.107:0/2844470723 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8010690 con 0x7fc5c81055d0 2024-07-26T12:42:13.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.597+0000 7fc5cce00640 1 -- 172.21.15.107:0/2844470723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c81a8900 con 0x7fc5c81055d0 2024-07-26T12:42:13.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.598+0000 7fc5cce00640 1 -- 172.21.15.107:0/2844470723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c81059e0 con 0x7fc5c81055d0 2024-07-26T12:42:13.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.600+0000 7fc5c4c00640 1 -- 172.21.15.107:0/2844470723 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fc5b8004e90 con 0x7fc5c81055d0 2024-07-26T12:42:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.602+0000 7fc5c4c00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5ac0b3c20 0x7fc5ac0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:13.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.602+0000 7fc5c4c00640 1 -- 172.21.15.107:0/2844470723 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fc5b80e3d00 con 0x7fc5c81055d0 2024-07-26T12:42:13.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.602+0000 7fc5bea00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5ac0b3c20 0x7fc5ac0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:13.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.603+0000 7fc5bea00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5ac0b3c20 0x7fc5ac0b6130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8104740 tx=0x7fc5b0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:13.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.606+0000 7fc5c4c00640 1 -- 172.21.15.107:0/2844470723 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc5b80ab680 con 0x7fc5c81055d0 2024-07-26T12:42:13.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.807+0000 7fc5cce00640 1 -- 172.21.15.107:0/2844470723 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc5c802e0c0 con 0x7fc5c81055d0 2024-07-26T12:42:13.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.808+0000 7fc5c4c00640 1 -- 172.21.15.107:0/2844470723 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7fc5b80aaf80 con 0x7fc5c81055d0 2024-07-26T12:42:13.810 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-07-26T12:42:13.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.810+0000 7fc5cce00640 1 -- 172.21.15.107:0/2844470723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5ac0b3c20 msgr2=0x7fc5ac0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:13.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5ac0b3c20 0x7fc5ac0b6130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8104740 tx=0x7fc5b0005f90 comp rx=0 tx=0).stop 2024-07-26T12:42:13.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 -- 172.21.15.107:0/2844470723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5c81055d0 msgr2=0x7fc5c819bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:13.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5c81055d0 0x7fc5c819bf40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800e9f0 tx=0x7fc5b800edf0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 -- 172.21.15.107:0/2844470723 shutdown_connections 2024-07-26T12:42:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5ac0b3c20 0x7fc5ac0b6130 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5c810b860 0x7fc5c81a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5c810af10 0x7fc5c819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 --2- 172.21.15.107:0/2844470723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5c81055d0 0x7fc5c819bf40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 -- 172.21.15.107:0/2844470723 >> 172.21.15.107:0/2844470723 conn(0x7fc5c8101040 msgr2=0x7fc5c8108500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 -- 172.21.15.107:0/2844470723 shutdown_connections 2024-07-26T12:42:13.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:13.811+0000 7fc5cce00640 1 -- 172.21.15.107:0/2844470723 wait complete. 2024-07-26T12:42:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:13 smithi112 ceph-mon[37501]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:13 smithi107 ceph-mon[32659]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:13 smithi132 ceph-mon[37580]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:14.757 INFO:tasks.cephadm:Adding mds.a on smithi107 2024-07-26T12:42:14.757 INFO:tasks.cephadm:Adding mds.d on smithi107 2024-07-26T12:42:14.757 INFO:tasks.cephadm:Adding mds.g on smithi107 2024-07-26T12:42:14.757 INFO:tasks.cephadm:Adding mds.j on smithi107 2024-07-26T12:42:14.757 INFO:tasks.cephadm:Adding mds.b on smithi112 2024-07-26T12:42:14.757 INFO:tasks.cephadm:Adding mds.e on smithi112 2024-07-26T12:42:14.758 INFO:tasks.cephadm:Adding mds.h on smithi112 2024-07-26T12:42:14.758 INFO:tasks.cephadm:Adding mds.k on smithi112 2024-07-26T12:42:14.758 INFO:tasks.cephadm:Adding mds.c on smithi132 2024-07-26T12:42:14.758 INFO:tasks.cephadm:Adding mds.f on smithi132 2024-07-26T12:42:14.758 INFO:tasks.cephadm:Adding mds.i on smithi132 2024-07-26T12:42:14.758 INFO:tasks.cephadm:Adding mds.l on smithi132 2024-07-26T12:42:14.758 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph orch apply mds all '12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l' 2024-07-26T12:42:15.023 INFO:teuthology.orchestra.run.smithi132.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T12:42:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:14 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2844470723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T12:42:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:14 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2844470723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T12:42:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:14 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2844470723' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T12:42:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:15 smithi132 ceph-mon[37580]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:15 smithi107 ceph-mon[32659]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:15 smithi112 ceph-mon[37501]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:16.668 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.651+0000 7fd435b1a640 1 -- 172.21.15.132:0/674294427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd430108740 msgr2=0x7fd43010ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.651+0000 7fd435b1a640 1 --2- 172.21.15.132:0/674294427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd430108740 0x7fd43010ef70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd428009ca0 tx=0x7fd42802a8c0 comp rx=0 tx=0).stop 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.651+0000 7fd435b1a640 1 -- 172.21.15.132:0/674294427 shutdown_connections 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.651+0000 7fd435b1a640 1 --2- 172.21.15.132:0/674294427 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd430108740 0x7fd43010ef70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.651+0000 7fd435b1a640 1 --2- 172.21.15.132:0/674294427 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd430107df0 0x7fd430108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.651+0000 7fd435b1a640 1 --2- 172.21.15.132:0/674294427 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4301024b0 0x7fd4301028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.651+0000 7fd435b1a640 1 -- 172.21.15.132:0/674294427 >> 172.21.15.132:0/674294427 conn(0x7fd430079170 msgr2=0x7fd430079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.651+0000 7fd435b1a640 1 -- 172.21.15.132:0/674294427 shutdown_connections 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.651+0000 7fd435b1a640 1 -- 172.21.15.132:0/674294427 wait complete. 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.651+0000 7fd435b1a640 1 Processor -- start 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd435b1a640 1 -- start start 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd435b1a640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4301024b0 0x7fd430198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd435b1a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd430107df0 0x7fd4301993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd435b1a640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd430108740 0x7fd43019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd435b1a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd430112440 con 0x7fd430107df0 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd435b1a640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd4301122c0 con 0x7fd4301024b0 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd435b1a640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd4301125c0 con 0x7fd430108740 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd42ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd430108740 0x7fd43019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd42ffff640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd430108740 0x7fd43019e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:56928/0 (socket says 172.21.15.132:56928) 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd42ffff640 1 -- 172.21.15.132:0/2608703010 learned_addr learned my addr 172.21.15.132:0/2608703010 (peer_addr_for_me v2:172.21.15.132:0/0) 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd42f7fe640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4301024b0 0x7fd430198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd42ffff640 1 -- 172.21.15.132:0/2608703010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4301024b0 msgr2=0x7fd430198e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd42ffff640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4301024b0 0x7fd430198e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd42ffff640 1 -- 172.21.15.132:0/2608703010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd430107df0 msgr2=0x7fd4301993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd42ffff640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd430107df0 0x7fd4301993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:16.670 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.652+0000 7fd42ffff640 1 -- 172.21.15.132:0/2608703010 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43019eda0 con 0x7fd430108740 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.653+0000 7fd42ffff640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd430108740 0x7fd43019e670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd428005cc0 tx=0x7fd42802f380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.653+0000 7fd42cff9640 1 -- 172.21.15.132:0/2608703010 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd428003bf0 con 0x7fd430108740 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.653+0000 7fd42cff9640 1 -- 172.21.15.132:0/2608703010 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd428003d90 con 0x7fd430108740 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.653+0000 7fd435b1a640 1 -- 172.21.15.132:0/2608703010 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd43019f030 con 0x7fd430108740 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.653+0000 7fd42cff9640 1 -- 172.21.15.132:0/2608703010 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42802fb20 con 0x7fd430108740 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.653+0000 7fd435b1a640 1 -- 172.21.15.132:0/2608703010 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4301a57f0 con 0x7fd430108740 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.654+0000 7fd435b1a640 1 -- 172.21.15.132:0/2608703010 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd430103730 con 0x7fd430108740 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.655+0000 7fd42cff9640 1 -- 172.21.15.132:0/2608703010 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fd428004580 con 0x7fd430108740 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.656+0000 7fd42cff9640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd4180b3c20 0x7fd4180b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.656+0000 7fd42cff9640 1 -- 172.21.15.132:0/2608703010 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fd428104fb0 con 0x7fd430108740 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.657+0000 7fd42f7fe640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd4180b3c20 0x7fd4180b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.658+0000 7fd42f7fe640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd4180b3c20 0x7fd4180b6130 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd420009990 tx=0x7fd420009110 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:16.671 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.659+0000 7fd42cff9640 1 -- 172.21.15.132:0/2608703010 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd4280cc9d0 con 0x7fd430108740 2024-07-26T12:42:16.847 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.845+0000 7fd435b1a640 1 -- 172.21.15.132:0/2608703010 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l", "target": ["mon-mgr", ""]}) -- 0x7fd430063230 con 0x7fd4180b3c20 2024-07-26T12:42:16.864 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.863+0000 7fd42cff9640 1 -- 172.21.15.132:0/2608703010 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd430063230 con 0x7fd4180b3c20 2024-07-26T12:42:16.864 INFO:teuthology.orchestra.run.smithi132.stdout:Scheduled mds.all update... 2024-07-26T12:42:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 -- 172.21.15.132:0/2608703010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd4180b3c20 msgr2=0x7fd4180b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd4180b3c20 0x7fd4180b6130 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd420009990 tx=0x7fd420009110 comp rx=0 tx=0).stop 2024-07-26T12:42:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 -- 172.21.15.132:0/2608703010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd430108740 msgr2=0x7fd43019e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd430108740 0x7fd43019e670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd428005cc0 tx=0x7fd42802f380 comp rx=0 tx=0).stop 2024-07-26T12:42:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 -- 172.21.15.132:0/2608703010 shutdown_connections 2024-07-26T12:42:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd4180b3c20 0x7fd4180b6130 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd430108740 0x7fd43019e670 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd430107df0 0x7fd4301993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 --2- 172.21.15.132:0/2608703010 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd4301024b0 0x7fd430198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:16.867 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 -- 172.21.15.132:0/2608703010 >> 172.21.15.132:0/2608703010 conn(0x7fd430079170 msgr2=0x7fd430104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:16.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 -- 172.21.15.132:0/2608703010 shutdown_connections 2024-07-26T12:42:16.868 INFO:teuthology.orchestra.run.smithi132.stderr:2024-07-26T12:42:16.865+0000 7fd435b1a640 1 -- 172.21.15.132:0/2608703010 wait complete. 2024-07-26T12:42:17.583 DEBUG:teuthology.orchestra.run.smithi107:mds.a> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service 2024-07-26T12:42:17.592 DEBUG:teuthology.orchestra.run.smithi107:mds.d> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service 2024-07-26T12:42:17.639 DEBUG:teuthology.orchestra.run.smithi107:mds.g> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.g.service 2024-07-26T12:42:17.684 DEBUG:teuthology.orchestra.run.smithi107:mds.j> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.j.service 2024-07-26T12:42:17.732 DEBUG:teuthology.orchestra.run.smithi112:mds.b> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service 2024-07-26T12:42:17.735 DEBUG:teuthology.orchestra.run.smithi112:mds.e> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.e.service 2024-07-26T12:42:17.779 DEBUG:teuthology.orchestra.run.smithi112:mds.h> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service 2024-07-26T12:42:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:17 smithi132 ceph-mon[37580]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:17.821 DEBUG:teuthology.orchestra.run.smithi112:mds.k> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.k.service 2024-07-26T12:42:17.866 DEBUG:teuthology.orchestra.run.smithi132:mds.c> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.c.service 2024-07-26T12:42:17.869 DEBUG:teuthology.orchestra.run.smithi132:mds.f> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.f.service 2024-07-26T12:42:17.912 DEBUG:teuthology.orchestra.run.smithi132:mds.i> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.i.service 2024-07-26T12:42:17.955 DEBUG:teuthology.orchestra.run.smithi132:mds.l> sudo journalctl -f -n 0 -u ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.l.service 2024-07-26T12:42:17.998 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-07-26T12:42:17.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:42:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:17 smithi107 ceph-mon[32659]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:17 smithi112 ceph-mon[37501]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:18.454 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:42:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:18 smithi132 ceph-mon[37580]: from='client.24628 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:42:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:18 smithi132 ceph-mon[37580]: Saving service mds.all spec with placement smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l;count:12 2024-07-26T12:42:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:18.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:18.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:18.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:18.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:18.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:18.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:18.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:18 smithi132 ceph-mon[37580]: Deploying daemon mds.l on smithi132 2024-07-26T12:42:18.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:18 smithi107 ceph-mon[32659]: from='client.24628 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:42:18.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:18 smithi107 ceph-mon[32659]: Saving service mds.all spec with placement smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l;count:12 2024-07-26T12:42:18.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:18.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:18.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:18.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:18.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:18.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:18.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:18.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:18 smithi107 ceph-mon[32659]: Deploying daemon mds.l on smithi132 2024-07-26T12:42:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:18 smithi112 ceph-mon[37501]: from='client.24628 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:42:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:18 smithi112 ceph-mon[37501]: Saving service mds.all spec with placement smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l;count:12 2024-07-26T12:42:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:18 smithi112 ceph-mon[37501]: Deploying daemon mds.l on smithi132 2024-07-26T12:42:19.784 INFO:journalctl@ceph.mds.l.smithi132.stdout:Jul 26 12:42:19 smithi132 systemd[1]: Starting Ceph mds.l for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:42:20.024 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:19 smithi107 ceph-mon[32659]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:19 smithi132 ceph-mon[37580]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:20.060 INFO:journalctl@ceph.mds.l.smithi132.stdout:Jul 26 12:42:19 smithi132 podman[69677]: 2024-07-26 12:42:19.763458725 +0000 UTC m=+0.017102888 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:42:20.060 INFO:journalctl@ceph.mds.l.smithi132.stdout:Jul 26 12:42:19 smithi132 podman[69677]: 2024-07-26 12:42:19.87627551 +0000 UTC m=+0.129919663 container create f7e4d4a9d544b24c4e9e3fcfe61cfe57bdc1ac90a35678c243be86f9157593e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-l, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, CEPH_POINT_RELEASE=) 2024-07-26T12:42:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:19 smithi112 ceph-mon[37501]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:20.560 INFO:journalctl@ceph.mds.l.smithi132.stdout:Jul 26 12:42:20 smithi132 podman[69677]: 2024-07-26 12:42:20.129187531 +0000 UTC m=+0.382831684 container init f7e4d4a9d544b24c4e9e3fcfe61cfe57bdc1ac90a35678c243be86f9157593e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-l, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0) 2024-07-26T12:42:20.560 INFO:journalctl@ceph.mds.l.smithi132.stdout:Jul 26 12:42:20 smithi132 podman[69677]: 2024-07-26 12:42:20.132097773 +0000 UTC m=+0.385741936 container start f7e4d4a9d544b24c4e9e3fcfe61cfe57bdc1ac90a35678c243be86f9157593e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-l, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T12:42:20.560 INFO:journalctl@ceph.mds.l.smithi132.stdout:Jul 26 12:42:20 smithi132 ceph-mds[69697]: -- 172.21.15.132:0/1485466700 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557cbbd1f860 con 0x557cbfec1a80 2024-07-26T12:42:20.561 INFO:journalctl@ceph.mds.l.smithi132.stdout:Jul 26 12:42:20 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-l[69692]: starting mds.l at 2024-07-26T12:42:20.561 INFO:journalctl@ceph.mds.l.smithi132.stdout:Jul 26 12:42:20 smithi132 bash[69677]: f7e4d4a9d544b24c4e9e3fcfe61cfe57bdc1ac90a35678c243be86f9157593e8 2024-07-26T12:42:20.561 INFO:journalctl@ceph.mds.l.smithi132.stdout:Jul 26 12:42:20 smithi132 systemd[1]: Started Ceph mds.l for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.579+0000 7fa003e00640 1 -- 172.21.15.107:0/107304397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa00410f330 msgr2=0x7fa00410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.579+0000 7fa003e00640 1 --2- 172.21.15.107:0/107304397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa00410f330 0x7fa00410f740 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009860 tx=0x7f9fec02a340 comp rx=0 tx=0).stop 2024-07-26T12:42:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.579+0000 7fa003e00640 1 -- 172.21.15.107:0/107304397 shutdown_connections 2024-07-26T12:42:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.579+0000 7fa003e00640 1 --2- 172.21.15.107:0/107304397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa00410fc80 0x7fa0041164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:20.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.579+0000 7fa003e00640 1 --2- 172.21.15.107:0/107304397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa00410f330 0x7fa00410f740 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.579+0000 7fa003e00640 1 --2- 172.21.15.107:0/107304397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa0041099f0 0x7fa004109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.579+0000 7fa003e00640 1 -- 172.21.15.107:0/107304397 >> 172.21.15.107:0/107304397 conn(0x7fa004078ff0 msgr2=0x7fa004107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.580+0000 7fa003e00640 1 -- 172.21.15.107:0/107304397 shutdown_connections 2024-07-26T12:42:20.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.580+0000 7fa003e00640 1 -- 172.21.15.107:0/107304397 wait complete. 2024-07-26T12:42:20.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.581+0000 7fa003e00640 1 Processor -- start 2024-07-26T12:42:20.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.581+0000 7fa003e00640 1 -- start start 2024-07-26T12:42:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7fa003e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041099f0 0x7fa0041a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7fa003e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa00410f330 0x7fa0041a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7fa003e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa00410fc80 0x7fa0041a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7fa003e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa004119940 con 0x7fa0041099f0 2024-07-26T12:42:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7fa003e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa0041197c0 con 0x7fa00410f330 2024-07-26T12:42:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7fa003e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa004119ac0 con 0x7fa00410fc80 2024-07-26T12:42:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7f9ffa000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa00410f330 0x7fa0041a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7f9ffa000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa00410f330 0x7fa0041a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:35830/0 (socket says 172.21.15.107:35830) 2024-07-26T12:42:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7f9ffa000640 1 -- 172.21.15.107:0/2332884902 learned_addr learned my addr 172.21.15.107:0/2332884902 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:42:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7fa002a00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041099f0 0x7fa0041a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.582+0000 7fa003400640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa00410fc80 0x7fa0041a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.583+0000 7f9ffa000640 1 -- 172.21.15.107:0/2332884902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa00410fc80 msgr2=0x7fa0041a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.583+0000 7f9ffa000640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa00410fc80 0x7fa0041a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.583+0000 7f9ffa000640 1 -- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041099f0 msgr2=0x7fa0041a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.583+0000 7f9ffa000640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041099f0 0x7fa0041a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.583+0000 7f9ffa000640 1 -- 172.21.15.107:0/2332884902 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0041a62e0 con 0x7fa00410f330 2024-07-26T12:42:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.583+0000 7fa002a00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041099f0 0x7fa0041a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:42:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.583+0000 7f9ffa000640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa00410f330 0x7fa0041a0910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9fec035040 tx=0x7f9fec032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.583+0000 7f9ffbe00640 1 -- 172.21.15.107:0/2332884902 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec037040 con 0x7fa00410f330 2024-07-26T12:42:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.584+0000 7f9ffbe00640 1 -- 172.21.15.107:0/2332884902 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fec030c10 con 0x7fa00410f330 2024-07-26T12:42:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.584+0000 7f9ffbe00640 1 -- 172.21.15.107:0/2332884902 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec0035f0 con 0x7fa00410f330 2024-07-26T12:42:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.584+0000 7fa003e00640 1 -- 172.21.15.107:0/2332884902 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0041a6570 con 0x7fa00410f330 2024-07-26T12:42:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.584+0000 7fa003e00640 1 -- 172.21.15.107:0/2332884902 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0041acd30 con 0x7fa00410f330 2024-07-26T12:42:20.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.585+0000 7fa003e00640 1 -- 172.21.15.107:0/2332884902 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa004109e00 con 0x7fa00410f330 2024-07-26T12:42:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.587+0000 7f9ffbe00640 1 -- 172.21.15.107:0/2332884902 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f9fec043020 con 0x7fa00410f330 2024-07-26T12:42:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.589+0000 7f9ffbe00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9fe80b3ca0 0x7f9fe80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.589+0000 7f9ffbe00640 1 -- 172.21.15.107:0/2332884902 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f9fec102b80 con 0x7fa00410f330 2024-07-26T12:42:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.590+0000 7fa002a00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9fe80b3ca0 0x7f9fe80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:20.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.590+0000 7fa002a00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9fe80b3ca0 0x7f9fe80b61b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4005fd0 tx=0x7f9ff4009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:20.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.593+0000 7f9ffbe00640 1 -- 172.21.15.107:0/2332884902 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9fec04a060 con 0x7fa00410f330 2024-07-26T12:42:20.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.831+0000 7fa003e00640 1 -- 172.21.15.107:0/2332884902 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa00402e0c0 con 0x7fa00410f330 2024-07-26T12:42:20.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.832+0000 7f9ffbe00640 1 -- 172.21.15.107:0/2332884902 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1435 (secure 0 0 0) 0x7f9fec0ca500 con 0x7fa00410f330 2024-07-26T12:42:20.834 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:42:20.834 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":2,"btime":"2024-07-26T12:42:20:170386+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24634,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-07-26T12:42:20.835 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 2 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.835+0000 7fa003e00640 1 -- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9fe80b3ca0 msgr2=0x7f9fe80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.835+0000 7fa003e00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9fe80b3ca0 0x7f9fe80b61b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4005fd0 tx=0x7f9ff4009260 comp rx=0 tx=0).stop 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.835+0000 7fa003e00640 1 -- 172.21.15.107:0/2332884902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa00410f330 msgr2=0x7fa0041a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.835+0000 7fa003e00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa00410f330 0x7fa0041a0910 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9fec035040 tx=0x7f9fec032d50 comp rx=0 tx=0).stop 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.835+0000 7fa003e00640 1 -- 172.21.15.107:0/2332884902 shutdown_connections 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.835+0000 7fa003e00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9fe80b3ca0 0x7f9fe80b61b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.835+0000 7fa003e00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa00410fc80 0x7fa0041a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.835+0000 7fa003e00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa00410f330 0x7fa0041a0910 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.835+0000 7fa003e00640 1 --2- 172.21.15.107:0/2332884902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0041099f0 0x7fa0041a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.836+0000 7fa003e00640 1 -- 172.21.15.107:0/2332884902 >> 172.21.15.107:0/2332884902 conn(0x7fa004078ff0 msgr2=0x7fa004108010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.836+0000 7fa003e00640 1 -- 172.21.15.107:0/2332884902 shutdown_connections 2024-07-26T12:42:20.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:20.836+0000 7fa003e00640 1 -- 172.21.15.107:0/2332884902 wait complete. 2024-07-26T12:42:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T12:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.132:6832/277912155,v1:172.21.15.132:6833/277912155] up:boot 2024-07-26T12:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: fsmap 1 up:standby 2024-07-26T12:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: Deploying daemon mds.k on smithi112 2024-07-26T12:42:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:21 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2332884902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T12:42:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.132:6832/277912155,v1:172.21.15.132:6833/277912155] up:boot 2024-07-26T12:42:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: fsmap 1 up:standby 2024-07-26T12:42:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:21.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: Deploying daemon mds.k on smithi112 2024-07-26T12:42:21.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:21.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:21 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2332884902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T12:42:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.132:6832/277912155,v1:172.21.15.132:6833/277912155] up:boot 2024-07-26T12:42:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: fsmap 1 up:standby 2024-07-26T12:42:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: Deploying daemon mds.k on smithi112 2024-07-26T12:42:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:21 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2332884902' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:22.596 INFO:journalctl@ceph.mds.k.smithi112.stdout:Jul 26 12:42:22 smithi112 systemd[1]: Starting Ceph mds.k for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:42:22.876 INFO:journalctl@ceph.mds.k.smithi112.stdout:Jul 26 12:42:22 smithi112 podman[68190]: 2024-07-26 12:42:22.497849896 +0000 UTC m=+0.017148383 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:42:22.877 INFO:journalctl@ceph.mds.k.smithi112.stdout:Jul 26 12:42:22 smithi112 podman[68190]: 2024-07-26 12:42:22.613209748 +0000 UTC m=+0.132508234 container create fe3a57bd22b0510ca8da390bdc660314a0dcba99c1386d34a716b47d84618f0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-k, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T12:42:22.877 INFO:journalctl@ceph.mds.k.smithi112.stdout:Jul 26 12:42:22 smithi112 podman[68190]: 2024-07-26 12:42:22.84104744 +0000 UTC m=+0.360345928 container init fe3a57bd22b0510ca8da390bdc660314a0dcba99c1386d34a716b47d84618f0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-26T12:42:22.877 INFO:journalctl@ceph.mds.k.smithi112.stdout:Jul 26 12:42:22 smithi112 podman[68190]: 2024-07-26 12:42:22.843551352 +0000 UTC m=+0.362849852 container start fe3a57bd22b0510ca8da390bdc660314a0dcba99c1386d34a716b47d84618f0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-k, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-07-26T12:42:23.200 INFO:journalctl@ceph.mds.k.smithi112.stdout:Jul 26 12:42:22 smithi112 ceph-mds[68210]: -- 172.21.15.112:0/3443981100 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558754eff860 con 0x5587590a1600 2024-07-26T12:42:23.200 INFO:journalctl@ceph.mds.k.smithi112.stdout:Jul 26 12:42:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-k[68205]: starting mds.k at 2024-07-26T12:42:23.200 INFO:journalctl@ceph.mds.k.smithi112.stdout:Jul 26 12:42:22 smithi112 bash[68190]: fe3a57bd22b0510ca8da390bdc660314a0dcba99c1386d34a716b47d84618f0a 2024-07-26T12:42:23.200 INFO:journalctl@ceph.mds.k.smithi112.stdout:Jul 26 12:42:22 smithi112 systemd[1]: Started Ceph mds.k for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:23.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:42:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:23 smithi112 ceph-mon[37501]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:23.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:23 smithi107 ceph-mon[32659]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:23.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:23.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:23.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:23.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:23.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:23.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:23.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:23 smithi132 ceph-mon[37580]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:24.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:24 smithi107 ceph-mon[32659]: Deploying daemon mds.j on smithi107 2024-07-26T12:42:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:24 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6832/2931369116,v1:172.21.15.112:6833/2931369116] up:boot 2024-07-26T12:42:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:24 smithi107 ceph-mon[32659]: fsmap 2 up:standby 2024-07-26T12:42:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T12:42:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:24 smithi112 ceph-mon[37501]: Deploying daemon mds.j on smithi107 2024-07-26T12:42:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:24 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6832/2931369116,v1:172.21.15.112:6833/2931369116] up:boot 2024-07-26T12:42:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:24 smithi112 ceph-mon[37501]: fsmap 2 up:standby 2024-07-26T12:42:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T12:42:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:24 smithi132 ceph-mon[37580]: Deploying daemon mds.j on smithi107 2024-07-26T12:42:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:24 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6832/2931369116,v1:172.21.15.112:6833/2931369116] up:boot 2024-07-26T12:42:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:24 smithi132 ceph-mon[37580]: fsmap 2 up:standby 2024-07-26T12:42:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T12:42:25.069 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:42:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:25 smithi107 ceph-mon[32659]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:25 smithi132 ceph-mon[37580]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:26.179 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:25 smithi112 ceph-mon[37501]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:26.752 INFO:journalctl@ceph.mds.j.smithi107.stdout:Jul 26 12:42:26 smithi107 podman[75954]: 2024-07-26 12:42:26.601694672 +0000 UTC m=+0.032792663 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:42:26.752 INFO:journalctl@ceph.mds.j.smithi107.stdout:Jul 26 12:42:26 smithi107 podman[75954]: 2024-07-26 12:42:26.75009695 +0000 UTC m=+0.181194954 container create c73824f5bc58aa0e5dc0320a5c57688d438090d0f230e1b16cf7cf127be91a4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-j, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:42:27.319 INFO:journalctl@ceph.mds.j.smithi107.stdout:Jul 26 12:42:27 smithi107 podman[75954]: 2024-07-26 12:42:27.232047036 +0000 UTC m=+0.663145083 container init c73824f5bc58aa0e5dc0320a5c57688d438090d0f230e1b16cf7cf127be91a4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-07-26T12:42:27.319 INFO:journalctl@ceph.mds.j.smithi107.stdout:Jul 26 12:42:27 smithi107 podman[75954]: 2024-07-26 12:42:27.244041979 +0000 UTC m=+0.675139986 container start c73824f5bc58aa0e5dc0320a5c57688d438090d0f230e1b16cf7cf127be91a4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:42:27.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.522+0000 7fdf41400640 1 -- 172.21.15.107:0/920513254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3c107df0 msgr2=0x7fdf3c108200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:27.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.522+0000 7fdf41400640 1 --2- 172.21.15.107:0/920513254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3c107df0 0x7fdf3c108200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009900 tx=0x7fdf2802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.523+0000 7fdf41400640 1 -- 172.21.15.107:0/920513254 shutdown_connections 2024-07-26T12:42:27.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.523+0000 7fdf41400640 1 --2- 172.21.15.107:0/920513254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c108740 0x7fdf3c10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.523+0000 7fdf41400640 1 --2- 172.21.15.107:0/920513254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3c107df0 0x7fdf3c108200 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.523+0000 7fdf41400640 1 --2- 172.21.15.107:0/920513254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf3c1024b0 0x7fdf3c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.523+0000 7fdf41400640 1 -- 172.21.15.107:0/920513254 >> 172.21.15.107:0/920513254 conn(0x7fdf3c079170 msgr2=0x7fdf3c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:27.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.523+0000 7fdf41400640 1 -- 172.21.15.107:0/920513254 shutdown_connections 2024-07-26T12:42:27.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.523+0000 7fdf41400640 1 -- 172.21.15.107:0/920513254 wait complete. 2024-07-26T12:42:27.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.524+0000 7fdf41400640 1 Processor -- start 2024-07-26T12:42:27.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.524+0000 7fdf41400640 1 -- start start 2024-07-26T12:42:27.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf41400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c1024b0 0x7fdf3c19c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:27.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf41400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3c107df0 0x7fdf3c19c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:27.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf41400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf3c108740 0x7fdf3c1a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:27.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf41400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c113500 con 0x7fdf3c1024b0 2024-07-26T12:42:27.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf41400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c113380 con 0x7fdf3c108740 2024-07-26T12:42:27.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf41400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdf3c113680 con 0x7fdf3c107df0 2024-07-26T12:42:27.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf3a800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c1024b0 0x7fdf3c19c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:27.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf39e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3c107df0 0x7fdf3c19c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:27.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf39e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3c107df0 0x7fdf3c19c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:41412/0 (socket says 172.21.15.107:41412) 2024-07-26T12:42:27.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf3a800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c1024b0 0x7fdf3c19c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47834/0 (socket says 172.21.15.107:47834) 2024-07-26T12:42:27.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf39e00640 1 -- 172.21.15.107:0/3402110605 learned_addr learned my addr 172.21.15.107:0/3402110605 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:42:27.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.525+0000 7fdf3b200640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf3c108740 0x7fdf3c1a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:27.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.526+0000 7fdf3a800640 1 -- 172.21.15.107:0/3402110605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3c107df0 msgr2=0x7fdf3c19c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:27.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.526+0000 7fdf3a800640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3c107df0 0x7fdf3c19c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.526+0000 7fdf3a800640 1 -- 172.21.15.107:0/3402110605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf3c108740 msgr2=0x7fdf3c1a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:27.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.526+0000 7fdf3a800640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf3c108740 0x7fdf3c1a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.526+0000 7fdf3a800640 1 -- 172.21.15.107:0/3402110605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf3c1a2080 con 0x7fdf3c1024b0 2024-07-26T12:42:27.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.526+0000 7fdf3a800640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c1024b0 0x7fdf3c19c110 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00e9f0 tx=0x7fdf2c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:27.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.530+0000 7fdf33400640 1 -- 172.21.15.107:0/3402110605 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c00ccc0 con 0x7fdf3c1024b0 2024-07-26T12:42:27.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.530+0000 7fdf33400640 1 -- 172.21.15.107:0/3402110605 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf2c005020 con 0x7fdf3c1024b0 2024-07-26T12:42:27.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.530+0000 7fdf41400640 1 -- 172.21.15.107:0/3402110605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf3c1a2370 con 0x7fdf3c1024b0 2024-07-26T12:42:27.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.530+0000 7fdf33400640 1 -- 172.21.15.107:0/3402110605 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c004a00 con 0x7fdf3c1024b0 2024-07-26T12:42:27.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.530+0000 7fdf41400640 1 -- 172.21.15.107:0/3402110605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf3c1a8ad0 con 0x7fdf3c1024b0 2024-07-26T12:42:27.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.532+0000 7fdf41400640 1 -- 172.21.15.107:0/3402110605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf3c103730 con 0x7fdf3c1024b0 2024-07-26T12:42:27.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.534+0000 7fdf33400640 1 -- 172.21.15.107:0/3402110605 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fdf2c004ba0 con 0x7fdf3c1024b0 2024-07-26T12:42:27.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.537+0000 7fdf33400640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdf200b3ca0 0x7fdf200b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:27.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.537+0000 7fdf33400640 1 -- 172.21.15.107:0/3402110605 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fdf2c0e3830 con 0x7fdf3c1024b0 2024-07-26T12:42:27.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.537+0000 7fdf39e00640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdf200b3ca0 0x7fdf200b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:27.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.538+0000 7fdf39e00640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdf200b3ca0 0x7fdf200b61b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdf28000c00 tx=0x7fdf28005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:27.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.547+0000 7fdf33400640 1 -- 172.21.15.107:0/3402110605 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdf2c0ab1b0 con 0x7fdf3c1024b0 2024-07-26T12:42:27.711 INFO:journalctl@ceph.mds.j.smithi107.stdout:Jul 26 12:42:27 smithi107 ceph-mds[75997]: -- 172.21.15.107:0/3396955083 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55577da55860 con 0x555781bf7180 2024-07-26T12:42:27.711 INFO:journalctl@ceph.mds.j.smithi107.stdout:Jul 26 12:42:27 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-j[75987]: starting mds.j at 2024-07-26T12:42:27.711 INFO:journalctl@ceph.mds.j.smithi107.stdout:Jul 26 12:42:27 smithi107 bash[75954]: c73824f5bc58aa0e5dc0320a5c57688d438090d0f230e1b16cf7cf127be91a4b 2024-07-26T12:42:27.711 INFO:journalctl@ceph.mds.j.smithi107.stdout:Jul 26 12:42:27 smithi107 systemd[1]: Started Ceph mds.j for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:27.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.807+0000 7fdf41400640 1 -- 172.21.15.107:0/3402110605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf3c1a8d80 con 0x7fdf3c1024b0 2024-07-26T12:42:27.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.808+0000 7fdf33400640 1 -- 172.21.15.107:0/3402110605 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3091 (secure 0 0 0) 0x7fdf2c0aaab0 con 0x7fdf3c1024b0 2024-07-26T12:42:27.809 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:42:27.809 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":4,"btime":"2024-07-26T12:42:27:703258+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24632,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24634,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24652,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-07-26T12:42:27.810 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 4 2024-07-26T12:42:27.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 -- 172.21.15.107:0/3402110605 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdf200b3ca0 msgr2=0x7fdf200b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:27.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdf200b3ca0 0x7fdf200b61b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdf28000c00 tx=0x7fdf28005a20 comp rx=0 tx=0).stop 2024-07-26T12:42:27.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 -- 172.21.15.107:0/3402110605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c1024b0 msgr2=0x7fdf3c19c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:27.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c1024b0 0x7fdf3c19c110 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c00e9f0 tx=0x7fdf2c00eeb0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 -- 172.21.15.107:0/3402110605 shutdown_connections 2024-07-26T12:42:27.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdf200b3ca0 0x7fdf200b61b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdf3c108740 0x7fdf3c1a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf3c107df0 0x7fdf3c19c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 --2- 172.21.15.107:0/3402110605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf3c1024b0 0x7fdf3c19c110 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:27.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 -- 172.21.15.107:0/3402110605 >> 172.21.15.107:0/3402110605 conn(0x7fdf3c079170 msgr2=0x7fdf3c103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:27.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 -- 172.21.15.107:0/3402110605 shutdown_connections 2024-07-26T12:42:27.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:27.811+0000 7fdf41400640 1 -- 172.21.15.107:0/3402110605 wait complete. 2024-07-26T12:42:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:27 smithi107 ceph-mon[32659]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:27 smithi132 ceph-mon[37580]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:27 smithi112 ceph-mon[37501]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:28.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:28 smithi107 ceph-mon[32659]: Deploying daemon mds.d on smithi107 2024-07-26T12:42:28.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T12:42:28.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:28 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6834/3403405722,v1:172.21.15.107:6835/3403405722] up:boot 2024-07-26T12:42:28.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:28 smithi107 ceph-mon[32659]: fsmap 3 up:standby 2024-07-26T12:42:28.913 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:28 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3402110605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:28 smithi132 ceph-mon[37580]: Deploying daemon mds.d on smithi107 2024-07-26T12:42:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T12:42:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:28 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6834/3403405722,v1:172.21.15.107:6835/3403405722] up:boot 2024-07-26T12:42:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:28 smithi132 ceph-mon[37580]: fsmap 3 up:standby 2024-07-26T12:42:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:28 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3402110605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:28 smithi112 ceph-mon[37501]: Deploying daemon mds.d on smithi107 2024-07-26T12:42:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T12:42:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:28 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6834/3403405722,v1:172.21.15.107:6835/3403405722] up:boot 2024-07-26T12:42:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:28 smithi112 ceph-mon[37501]: fsmap 3 up:standby 2024-07-26T12:42:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:28 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3402110605' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:29 smithi107 ceph-mon[32659]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:29 smithi132 ceph-mon[37580]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:29 smithi112 ceph-mon[37501]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:30.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:42:31.259 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:42:30 smithi107 systemd[1]: Starting Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:42:31.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:42:31 smithi107 podman[76350]: 2024-07-26 12:42:31.162285098 +0000 UTC m=+0.035374661 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:42:31.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:42:31 smithi107 podman[76350]: 2024-07-26 12:42:31.276663806 +0000 UTC m=+0.149753404 container create dab1f478088964ff72b80cf18fa861f82b0792a723586768db672397707b0178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:42:31.766 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:42:31.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:31 smithi107 ceph-mon[32659]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:31.915 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:42:31 smithi107 podman[76350]: 2024-07-26 12:42:31.565245524 +0000 UTC m=+0.438335128 container init dab1f478088964ff72b80cf18fa861f82b0792a723586768db672397707b0178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-07-26T12:42:31.915 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:42:31 smithi107 podman[76350]: 2024-07-26 12:42:31.575437886 +0000 UTC m=+0.448527491 container start dab1f478088964ff72b80cf18fa861f82b0792a723586768db672397707b0178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-07-26T12:42:31.915 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:42:31 smithi107 bash[76350]: dab1f478088964ff72b80cf18fa861f82b0792a723586768db672397707b0178 2024-07-26T12:42:31.915 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:42:31 smithi107 ceph-mds[76384]: -- 172.21.15.107:0/1892433432 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d1bd5e5860 con 0x55d1c1787600 2024-07-26T12:42:31.915 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:42:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: starting mds.d at 2024-07-26T12:42:31.915 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:42:31 smithi107 systemd[1]: Started Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:31 smithi132 ceph-mon[37580]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:31 smithi112 ceph-mon[37501]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:42:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:32 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:boot 2024-07-26T12:42:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:32 smithi112 ceph-mon[37501]: fsmap 4 up:standby 2024-07-26T12:42:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:42:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:32 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:boot 2024-07-26T12:42:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:32 smithi107 ceph-mon[32659]: fsmap 4 up:standby 2024-07-26T12:42:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:42:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:32 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:boot 2024-07-26T12:42:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:32 smithi132 ceph-mon[37580]: fsmap 4 up:standby 2024-07-26T12:42:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:33.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:33.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:33 smithi112 ceph-mon[37501]: Deploying daemon mds.e on smithi112 2024-07-26T12:42:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:33 smithi112 ceph-mon[37501]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:33.951 INFO:journalctl@ceph.mds.e.smithi112.stdout:Jul 26 12:42:33 smithi112 systemd[1]: Starting Ceph mds.e for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:42:33.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.970+0000 7fd790600640 1 -- 172.21.15.107:0/3596718063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd78810fcb0 msgr2=0x7fd7881164e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:33.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.970+0000 7fd790600640 1 --2- 172.21.15.107:0/3596718063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd78810fcb0 0x7fd7881164e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd780009ca0 tx=0x7fd78002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:42:33.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.970+0000 7fd790600640 1 -- 172.21.15.107:0/3596718063 shutdown_connections 2024-07-26T12:42:33.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.970+0000 7fd790600640 1 --2- 172.21.15.107:0/3596718063 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd78810fcb0 0x7fd7881164e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:33.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.970+0000 7fd790600640 1 --2- 172.21.15.107:0/3596718063 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd78810f360 0x7fd78810f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:33.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.970+0000 7fd790600640 1 --2- 172.21.15.107:0/3596718063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd788109a70 0x7fd788109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:33.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.970+0000 7fd790600640 1 -- 172.21.15.107:0/3596718063 >> 172.21.15.107:0/3596718063 conn(0x7fd788079040 msgr2=0x7fd788107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:33.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.970+0000 7fd790600640 1 -- 172.21.15.107:0/3596718063 shutdown_connections 2024-07-26T12:42:33.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.970+0000 7fd790600640 1 -- 172.21.15.107:0/3596718063 wait complete. 2024-07-26T12:42:33.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.971+0000 7fd790600640 1 Processor -- start 2024-07-26T12:42:33.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.972+0000 7fd790600640 1 -- start start 2024-07-26T12:42:33.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.972+0000 7fd790600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd788109a70 0x7fd7881a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:33.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.972+0000 7fd790600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78810f360 0x7fd7881a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:33.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.972+0000 7fd790600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd78810fcb0 0x7fd7881a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:33.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.972+0000 7fd790600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd788119970 con 0x7fd78810f360 2024-07-26T12:42:33.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.972+0000 7fd790600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd7881197f0 con 0x7fd788109a70 2024-07-26T12:42:33.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.972+0000 7fd790600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd788119af0 con 0x7fd78810fcb0 2024-07-26T12:42:33.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.972+0000 7fd78dc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd788109a70 0x7fd7881a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:33.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.973+0000 7fd78dc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd788109a70 0x7fd7881a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:52072/0 (socket says 172.21.15.107:52072) 2024-07-26T12:42:33.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.973+0000 7fd78dc00640 1 -- 172.21.15.107:0/3850369985 learned_addr learned my addr 172.21.15.107:0/3850369985 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:42:33.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.973+0000 7fd78e600640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd78810fcb0 0x7fd7881a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:33.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.973+0000 7fd78d200640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78810f360 0x7fd7881a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:33.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.973+0000 7fd78dc00640 1 -- 172.21.15.107:0/3850369985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd78810fcb0 msgr2=0x7fd7881a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:33.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.973+0000 7fd78dc00640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd78810fcb0 0x7fd7881a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.973+0000 7fd78dc00640 1 -- 172.21.15.107:0/3850369985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78810f360 msgr2=0x7fd7881a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.973+0000 7fd78dc00640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78810f360 0x7fd7881a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.973+0000 7fd78dc00640 1 -- 172.21.15.107:0/3850369985 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7881a61f0 con 0x7fd788109a70 2024-07-26T12:42:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.973+0000 7fd78dc00640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd788109a70 0x7fd7881a0280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00ea60 tx=0x7fd77c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.974+0000 7fd786000640 1 -- 172.21.15.107:0/3850369985 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c00cde0 con 0x7fd788109a70 2024-07-26T12:42:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.974+0000 7fd790600640 1 -- 172.21.15.107:0/3850369985 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7881a64e0 con 0x7fd788109a70 2024-07-26T12:42:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.974+0000 7fd786000640 1 -- 172.21.15.107:0/3850369985 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd77c0035f0 con 0x7fd788109a70 2024-07-26T12:42:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.974+0000 7fd786000640 1 -- 172.21.15.107:0/3850369985 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c010690 con 0x7fd788109a70 2024-07-26T12:42:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.974+0000 7fd790600640 1 -- 172.21.15.107:0/3850369985 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7881acc40 con 0x7fd788109a70 2024-07-26T12:42:33.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.975+0000 7fd790600640 1 -- 172.21.15.107:0/3850369985 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78810aca0 con 0x7fd788109a70 2024-07-26T12:42:33.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.977+0000 7fd786000640 1 -- 172.21.15.107:0/3850369985 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fd77c0037a0 con 0x7fd788109a70 2024-07-26T12:42:33.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.979+0000 7fd786000640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd7700b3ca0 0x7fd7700b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:33.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.979+0000 7fd786000640 1 -- 172.21.15.107:0/3850369985 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fd77c0e3c10 con 0x7fd788109a70 2024-07-26T12:42:33.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.979+0000 7fd78d200640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd7700b3ca0 0x7fd7700b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:33.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.980+0000 7fd78d200640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd7700b3ca0 0x7fd7700b61b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd78810fa00 tx=0x7fd774005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:33.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:33.981+0000 7fd786000640 1 -- 172.21.15.107:0/3850369985 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd77c0ab590 con 0x7fd788109a70 2024-07-26T12:42:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:33 smithi107 ceph-mon[32659]: Deploying daemon mds.e on smithi112 2024-07-26T12:42:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:33 smithi107 ceph-mon[32659]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:33 smithi132 ceph-mon[37580]: Deploying daemon mds.e on smithi112 2024-07-26T12:42:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:33 smithi132 ceph-mon[37580]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:34.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.219+0000 7fd790600640 1 -- 172.21.15.107:0/3850369985 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd788109e80 con 0x7fd788109a70 2024-07-26T12:42:34.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.220+0000 7fd786000640 1 -- 172.21.15.107:0/3850369985 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3919 (secure 0 0 0) 0x7fd77c0aae90 con 0x7fd788109a70 2024-07-26T12:42:34.221 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:42:34.222 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":5,"btime":"2024-07-26T12:42:31:723338+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24632,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24634,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24652,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-07-26T12:42:34.222 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 5 2024-07-26T12:42:34.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 -- 172.21.15.107:0/3850369985 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd7700b3ca0 msgr2=0x7fd7700b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:34.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd7700b3ca0 0x7fd7700b61b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd78810fa00 tx=0x7fd774005f90 comp rx=0 tx=0).stop 2024-07-26T12:42:34.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 -- 172.21.15.107:0/3850369985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd788109a70 msgr2=0x7fd7881a0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:34.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd788109a70 0x7fd7881a0280 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00ea60 tx=0x7fd77c00ef20 comp rx=0 tx=0).stop 2024-07-26T12:42:34.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 -- 172.21.15.107:0/3850369985 shutdown_connections 2024-07-26T12:42:34.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd7700b3ca0 0x7fd7700b61b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:34.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd78810fcb0 0x7fd7881a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:34.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd78810f360 0x7fd7881a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:34.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 --2- 172.21.15.107:0/3850369985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd788109a70 0x7fd7881a0280 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:34.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 -- 172.21.15.107:0/3850369985 >> 172.21.15.107:0/3850369985 conn(0x7fd788079040 msgr2=0x7fd7880796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:34.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 -- 172.21.15.107:0/3850369985 shutdown_connections 2024-07-26T12:42:34.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:34.223+0000 7fd790600640 1 -- 172.21.15.107:0/3850369985 wait complete. 2024-07-26T12:42:34.327 INFO:journalctl@ceph.mds.e.smithi112.stdout:Jul 26 12:42:34 smithi112 podman[68503]: 2024-07-26 12:42:33.976943594 +0000 UTC m=+0.017591770 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:42:34.327 INFO:journalctl@ceph.mds.e.smithi112.stdout:Jul 26 12:42:34 smithi112 podman[68503]: 2024-07-26 12:42:34.080504993 +0000 UTC m=+0.121153201 container create cda99e4a70109ad82df14123055dbfef7e20b6e62d8724c4bd02525328763d07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux ) 2024-07-26T12:42:34.327 INFO:journalctl@ceph.mds.e.smithi112.stdout:Jul 26 12:42:34 smithi112 podman[68503]: 2024-07-26 12:42:34.291703141 +0000 UTC m=+0.332351308 container init cda99e4a70109ad82df14123055dbfef7e20b6e62d8724c4bd02525328763d07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724) 2024-07-26T12:42:34.327 INFO:journalctl@ceph.mds.e.smithi112.stdout:Jul 26 12:42:34 smithi112 podman[68503]: 2024-07-26 12:42:34.29421367 +0000 UTC m=+0.334861839 container start cda99e4a70109ad82df14123055dbfef7e20b6e62d8724c4bd02525328763d07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-e, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T12:42:34.700 INFO:journalctl@ceph.mds.e.smithi112.stdout:Jul 26 12:42:34 smithi112 ceph-mds[68522]: -- 172.21.15.112:0/134513049 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560c2b86d860 con 0x560c2fa0f600 2024-07-26T12:42:34.700 INFO:journalctl@ceph.mds.e.smithi112.stdout:Jul 26 12:42:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-e[68518]: starting mds.e at 2024-07-26T12:42:34.700 INFO:journalctl@ceph.mds.e.smithi112.stdout:Jul 26 12:42:34 smithi112 bash[68503]: cda99e4a70109ad82df14123055dbfef7e20b6e62d8724c4bd02525328763d07 2024-07-26T12:42:34.700 INFO:journalctl@ceph.mds.e.smithi112.stdout:Jul 26 12:42:34 smithi112 systemd[1]: Started Ceph mds.e for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:34.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:34 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3850369985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:34.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:34.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:34.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:34.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:34.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:34.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:34.937 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:34 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3850369985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:34 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3850369985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:35 smithi112 ceph-mon[37501]: Deploying daemon mds.f on smithi132 2024-07-26T12:42:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:35 smithi112 ceph-mon[37501]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T12:42:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:35 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6834/1038904649,v1:172.21.15.112:6835/1038904649] up:boot 2024-07-26T12:42:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:35 smithi112 ceph-mon[37501]: fsmap 5 up:standby 2024-07-26T12:42:36.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:35 smithi107 ceph-mon[32659]: Deploying daemon mds.f on smithi132 2024-07-26T12:42:36.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:35 smithi107 ceph-mon[32659]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:36.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T12:42:36.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:35 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6834/1038904649,v1:172.21.15.112:6835/1038904649] up:boot 2024-07-26T12:42:36.040 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:35 smithi107 ceph-mon[32659]: fsmap 5 up:standby 2024-07-26T12:42:36.056 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:35 smithi132 ceph-mon[37580]: Deploying daemon mds.f on smithi132 2024-07-26T12:42:36.056 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:35 smithi132 ceph-mon[37580]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:36.056 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T12:42:36.056 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:35 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6834/1038904649,v1:172.21.15.112:6835/1038904649] up:boot 2024-07-26T12:42:36.056 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:35 smithi132 ceph-mon[37580]: fsmap 5 up:standby 2024-07-26T12:42:36.719 INFO:journalctl@ceph.mds.f.smithi132.stdout:Jul 26 12:42:36 smithi132 systemd[1]: Starting Ceph mds.f for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:42:36.970 INFO:journalctl@ceph.mds.f.smithi132.stdout:Jul 26 12:42:36 smithi132 podman[69984]: 2024-07-26 12:42:36.621003904 +0000 UTC m=+0.016835694 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:42:36.970 INFO:journalctl@ceph.mds.f.smithi132.stdout:Jul 26 12:42:36 smithi132 podman[69984]: 2024-07-26 12:42:36.733420383 +0000 UTC m=+0.129252191 container create 065e99edd8b0bc924f45a81f730be63a9ad481a3f11b024b901b95bcf1ce0045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:42:37.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:42:37.310 INFO:journalctl@ceph.mds.f.smithi132.stdout:Jul 26 12:42:36 smithi132 podman[69984]: 2024-07-26 12:42:36.969638292 +0000 UTC m=+0.365470075 container init 065e99edd8b0bc924f45a81f730be63a9ad481a3f11b024b901b95bcf1ce0045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-07-26T12:42:37.310 INFO:journalctl@ceph.mds.f.smithi132.stdout:Jul 26 12:42:36 smithi132 podman[69984]: 2024-07-26 12:42:36.972382191 +0000 UTC m=+0.368213959 container start 065e99edd8b0bc924f45a81f730be63a9ad481a3f11b024b901b95bcf1ce0045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:42:37.310 INFO:journalctl@ceph.mds.f.smithi132.stdout:Jul 26 12:42:37 smithi132 ceph-mds[70003]: -- 172.21.15.132:0/1744226478 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5589403c9860 con 0x55894456b180 2024-07-26T12:42:37.310 INFO:journalctl@ceph.mds.f.smithi132.stdout:Jul 26 12:42:37 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-f[69999]: starting mds.f at 2024-07-26T12:42:37.310 INFO:journalctl@ceph.mds.f.smithi132.stdout:Jul 26 12:42:37 smithi132 bash[69984]: 065e99edd8b0bc924f45a81f730be63a9ad481a3f11b024b901b95bcf1ce0045 2024-07-26T12:42:37.310 INFO:journalctl@ceph.mds.f.smithi132.stdout:Jul 26 12:42:37 smithi132 systemd[1]: Started Ceph mds.f for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:37.519 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:42:37.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:37 smithi132 ceph-mon[37580]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:37.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:37.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:37.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:37.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:37.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:37.925 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:37.926 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:37.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:37 smithi107 ceph-mon[32659]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:37.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:37.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:37.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:37.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:37.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:37.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:37.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:37 smithi112 ceph-mon[37501]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:38.919 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:38 smithi132 ceph-mon[37580]: Deploying daemon mds.c on smithi132 2024-07-26T12:42:38.920 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T12:42:38.920 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:38 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] up:boot 2024-07-26T12:42:38.920 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:38 smithi132 ceph-mon[37580]: fsmap 6 up:standby 2024-07-26T12:42:38.920 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T12:42:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:38 smithi107 ceph-mon[32659]: Deploying daemon mds.c on smithi132 2024-07-26T12:42:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T12:42:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:38 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] up:boot 2024-07-26T12:42:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:38 smithi107 ceph-mon[32659]: fsmap 6 up:standby 2024-07-26T12:42:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T12:42:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:38 smithi112 ceph-mon[37501]: Deploying daemon mds.c on smithi132 2024-07-26T12:42:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T12:42:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:38 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] up:boot 2024-07-26T12:42:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:38 smithi112 ceph-mon[37501]: fsmap 6 up:standby 2024-07-26T12:42:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T12:42:39.310 INFO:journalctl@ceph.mds.c.smithi132.stdout:Jul 26 12:42:39 smithi132 systemd[1]: Starting Ceph mds.c for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:42:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.598+0000 7f479fe00640 1 -- 172.21.15.107:0/2189225540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a0104250 msgr2=0x7f47a0104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.598+0000 7f479fe00640 1 --2- 172.21.15.107:0/2189225540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a0104250 0x7f47a0104660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4788009900 tx=0x7f478802a410 comp rx=0 tx=0).stop 2024-07-26T12:42:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.599+0000 7f479fe00640 1 -- 172.21.15.107:0/2189225540 shutdown_connections 2024-07-26T12:42:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.599+0000 7f479fe00640 1 --2- 172.21.15.107:0/2189225540 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47a010d6b0 0x7f47a010fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.599+0000 7f479fe00640 1 --2- 172.21.15.107:0/2189225540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47a0104ba0 0x7f47a010d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.599+0000 7f479fe00640 1 --2- 172.21.15.107:0/2189225540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a0104250 0x7f47a0104660 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:39.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.599+0000 7f479fe00640 1 -- 172.21.15.107:0/2189225540 >> 172.21.15.107:0/2189225540 conn(0x7f47a00fddb0 msgr2=0x7f47a0100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:39.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.599+0000 7f479fe00640 1 -- 172.21.15.107:0/2189225540 shutdown_connections 2024-07-26T12:42:39.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.599+0000 7f479fe00640 1 -- 172.21.15.107:0/2189225540 wait complete. 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.600+0000 7f479fe00640 1 Processor -- start 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.600+0000 7f479fe00640 1 -- start start 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.600+0000 7f479fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47a0104250 0x7f47a0199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a0104ba0 0x7f47a019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47a010d6b0 0x7f47a019f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479fe00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f47a0113340 con 0x7f47a0104ba0 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479fe00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f47a01131c0 con 0x7f47a0104250 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479fe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f47a01134c0 con 0x7f47a010d6b0 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47a0104250 0x7f47a0199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479e000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a0104ba0 0x7f47a019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:39.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47a0104250 0x7f47a0199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:52082/0 (socket says 172.21.15.107:52082) 2024-07-26T12:42:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479ea00640 1 -- 172.21.15.107:0/3048702340 learned_addr learned my addr 172.21.15.107:0/3048702340 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:42:39.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479f400640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47a010d6b0 0x7f47a019f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479ea00640 1 -- 172.21.15.107:0/3048702340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47a010d6b0 msgr2=0x7f47a019f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479ea00640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47a010d6b0 0x7f47a019f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479ea00640 1 -- 172.21.15.107:0/3048702340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a0104ba0 msgr2=0x7f47a019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479ea00640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a0104ba0 0x7f47a019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.601+0000 7f479ea00640 1 -- 172.21.15.107:0/3048702340 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a019fcc0 con 0x7f47a0104250 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.602+0000 7f479ea00640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47a0104250 0x7f47a0199d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4788005940 tx=0x7f47880040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.602+0000 7f4797400640 1 -- 172.21.15.107:0/3048702340 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478803f030 con 0x7f47a0104250 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.602+0000 7f479fe00640 1 -- 172.21.15.107:0/3048702340 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a019ff50 con 0x7f47a0104250 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.602+0000 7f4797400640 1 -- 172.21.15.107:0/3048702340 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47880044c0 con 0x7f47a0104250 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.602+0000 7f4797400640 1 -- 172.21.15.107:0/3048702340 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788035070 con 0x7f47a0104250 2024-07-26T12:42:39.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.602+0000 7f479fe00640 1 -- 172.21.15.107:0/3048702340 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a01a6710 con 0x7f47a0104250 2024-07-26T12:42:39.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.604+0000 7f479fe00640 1 -- 172.21.15.107:0/3048702340 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a0104660 con 0x7f47a0104250 2024-07-26T12:42:39.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.606+0000 7f4797400640 1 -- 172.21.15.107:0/3048702340 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f4788042030 con 0x7f47a0104250 2024-07-26T12:42:39.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.607+0000 7f4797400640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47840b3c20 0x7f47840b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:39.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.608+0000 7f4797400640 1 -- 172.21.15.107:0/3048702340 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f4788103ee0 con 0x7f47a0104250 2024-07-26T12:42:39.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.608+0000 7f479e000640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47840b3c20 0x7f47840b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:39.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.608+0000 7f479e000640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47840b3c20 0x7f47840b6130 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f47a010fc00 tx=0x7f4790005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:39.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.609+0000 7f4797400640 1 -- 172.21.15.107:0/3048702340 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f47880cb210 con 0x7f47a0104250 2024-07-26T12:42:39.660 INFO:journalctl@ceph.mds.c.smithi132.stdout:Jul 26 12:42:39 smithi132 podman[70291]: 2024-07-26 12:42:39.256477171 +0000 UTC m=+0.017447875 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:42:39.660 INFO:journalctl@ceph.mds.c.smithi132.stdout:Jul 26 12:42:39 smithi132 podman[70291]: 2024-07-26 12:42:39.36502557 +0000 UTC m=+0.125996270 container create 1ee9887ee094ec35ab6d3cfe8b4063223b4e5097d079234cbc90cd3d22f0966c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724) 2024-07-26T12:42:39.660 INFO:journalctl@ceph.mds.c.smithi132.stdout:Jul 26 12:42:39 smithi132 podman[70291]: 2024-07-26 12:42:39.659589196 +0000 UTC m=+0.420559887 container init 1ee9887ee094ec35ab6d3cfe8b4063223b4e5097d079234cbc90cd3d22f0966c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240724, ceph=True) 2024-07-26T12:42:39.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.851+0000 7f479fe00640 1 -- 172.21.15.107:0/3048702340 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47a01a68e0 con 0x7f47a0104250 2024-07-26T12:42:39.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.852+0000 7f4797400640 1 -- 172.21.15.107:0/3048702340 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6400 (secure 0 0 0) 0x7f47880cf760 con 0x7f47a0104250 2024-07-26T12:42:39.853 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:42:39.854 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":8,"btime":"2024-07-26T12:42:39:791565+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24632,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24634,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24652,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24662,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-07-26T12:42:39.854 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 8 2024-07-26T12:42:39.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.855+0000 7f479fe00640 1 -- 172.21.15.107:0/3048702340 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47840b3c20 msgr2=0x7f47840b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:39.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.855+0000 7f479fe00640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47840b3c20 0x7f47840b6130 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f47a010fc00 tx=0x7f4790005f90 comp rx=0 tx=0).stop 2024-07-26T12:42:39.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.855+0000 7f479fe00640 1 -- 172.21.15.107:0/3048702340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47a0104250 msgr2=0x7f47a0199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:39.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.855+0000 7f479fe00640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47a0104250 0x7f47a0199d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4788005940 tx=0x7f47880040c0 comp rx=0 tx=0).stop 2024-07-26T12:42:39.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.855+0000 7f479fe00640 1 -- 172.21.15.107:0/3048702340 shutdown_connections 2024-07-26T12:42:39.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.855+0000 7f479fe00640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f47840b3c20 0x7f47840b6130 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:39.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.856+0000 7f479fe00640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47a010d6b0 0x7f47a019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:39.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.856+0000 7f479fe00640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47a0104ba0 0x7f47a019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:39.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.856+0000 7f479fe00640 1 --2- 172.21.15.107:0/3048702340 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f47a0104250 0x7f47a0199d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:39.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.856+0000 7f479fe00640 1 -- 172.21.15.107:0/3048702340 >> 172.21.15.107:0/3048702340 conn(0x7f47a00fddb0 msgr2=0x7f47a01071f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:39.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.856+0000 7f479fe00640 1 -- 172.21.15.107:0/3048702340 shutdown_connections 2024-07-26T12:42:39.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:39.856+0000 7f479fe00640 1 -- 172.21.15.107:0/3048702340 wait complete. 2024-07-26T12:42:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:39 smithi132 ceph-mon[37580]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:40.060 INFO:journalctl@ceph.mds.c.smithi132.stdout:Jul 26 12:42:39 smithi132 podman[70291]: 2024-07-26 12:42:39.662231989 +0000 UTC m=+0.423202681 container start 1ee9887ee094ec35ab6d3cfe8b4063223b4e5097d079234cbc90cd3d22f0966c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-07-26T12:42:40.060 INFO:journalctl@ceph.mds.c.smithi132.stdout:Jul 26 12:42:39 smithi132 ceph-mds[70310]: -- 172.21.15.132:0/3268701253 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cb2dd4b860 con 0x55cb31eed600 2024-07-26T12:42:40.060 INFO:journalctl@ceph.mds.c.smithi132.stdout:Jul 26 12:42:39 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-c[70306]: starting mds.c at 2024-07-26T12:42:40.061 INFO:journalctl@ceph.mds.c.smithi132.stdout:Jul 26 12:42:39 smithi132 bash[70291]: 1ee9887ee094ec35ab6d3cfe8b4063223b4e5097d079234cbc90cd3d22f0966c 2024-07-26T12:42:40.061 INFO:journalctl@ceph.mds.c.smithi132.stdout:Jul 26 12:42:39 smithi132 systemd[1]: Started Ceph mds.c for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:39 smithi112 ceph-mon[37501]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:40.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:39 smithi107 ceph-mon[32659]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T12:42:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] up:boot 2024-07-26T12:42:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: fsmap 7 up:standby 2024-07-26T12:42:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3048702340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:41.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:41.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:41.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:41.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: Deploying daemon mds.h on smithi112 2024-07-26T12:42:41.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:40 smithi107 ceph-mon[32659]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T12:42:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] up:boot 2024-07-26T12:42:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: fsmap 7 up:standby 2024-07-26T12:42:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3048702340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:41.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:41.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:41.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:41.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: Deploying daemon mds.h on smithi112 2024-07-26T12:42:41.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:40 smithi132 ceph-mon[37580]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] up:boot 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: fsmap 7 up:standby 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3048702340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: Deploying daemon mds.h on smithi112 2024-07-26T12:42:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:40 smithi112 ceph-mon[37501]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:42.108 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:42:41 smithi112 systemd[1]: Starting Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:42:42.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:42:42 smithi112 podman[68804]: 2024-07-26 12:42:42.01005574 +0000 UTC m=+0.016995893 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:42:42.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:42:42 smithi112 podman[68804]: 2024-07-26 12:42:42.136526656 +0000 UTC m=+0.143466800 container create 93b16afe7021e1f4efdf1baade876c8322e78fa93de44d75ba5f7f3bf4cb3a8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T12:42:42.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:42:42.700 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:42:42 smithi112 podman[68804]: 2024-07-26 12:42:42.359400981 +0000 UTC m=+0.366341125 container init 93b16afe7021e1f4efdf1baade876c8322e78fa93de44d75ba5f7f3bf4cb3a8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-07-26T12:42:42.701 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:42:42 smithi112 podman[68804]: 2024-07-26 12:42:42.362243298 +0000 UTC m=+0.369183443 container start 93b16afe7021e1f4efdf1baade876c8322e78fa93de44d75ba5f7f3bf4cb3a8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, ceph=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-07-26T12:42:42.701 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:42:42 smithi112 ceph-mds[68823]: -- 172.21.15.112:0/3365216251 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56108f301860 con 0x5610934a3a80 2024-07-26T12:42:42.701 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:42:42 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[68819]: starting mds.h at 2024-07-26T12:42:42.701 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:42:42 smithi112 bash[68804]: 93b16afe7021e1f4efdf1baade876c8322e78fa93de44d75ba5f7f3bf4cb3a8a 2024-07-26T12:42:42.701 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:42:42 smithi112 systemd[1]: Started Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:43.121 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:42:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/3739854241,v1:172.21.15.112:6837/3739854241] up:boot 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: fsmap 8 up:standby 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: Deploying daemon mds.i on smithi132 2024-07-26T12:42:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:43 smithi107 ceph-mon[32659]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:43.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:42:43.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/3739854241,v1:172.21.15.112:6837/3739854241] up:boot 2024-07-26T12:42:43.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: fsmap 8 up:standby 2024-07-26T12:42:43.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:43.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:43.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:43.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:43.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:43.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:43.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:43.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: Deploying daemon mds.i on smithi132 2024-07-26T12:42:43.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:43 smithi132 ceph-mon[37580]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:43.872 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:42:43.872 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/3739854241,v1:172.21.15.112:6837/3739854241] up:boot 2024-07-26T12:42:43.872 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: fsmap 8 up:standby 2024-07-26T12:42:43.872 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:43.872 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:43.872 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:43.872 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:43.873 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:43.873 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:43.873 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:43.873 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: Deploying daemon mds.i on smithi132 2024-07-26T12:42:43.873 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:43 smithi112 ceph-mon[37501]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:44.755 INFO:journalctl@ceph.mds.i.smithi132.stdout:Jul 26 12:42:44 smithi132 systemd[1]: Starting Ceph mds.i for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:42:45.025 INFO:journalctl@ceph.mds.i.smithi132.stdout:Jul 26 12:42:44 smithi132 podman[70596]: 2024-07-26 12:42:44.656230041 +0000 UTC m=+0.016693963 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:42:45.026 INFO:journalctl@ceph.mds.i.smithi132.stdout:Jul 26 12:42:44 smithi132 podman[70596]: 2024-07-26 12:42:44.764116198 +0000 UTC m=+0.124580092 container create 92dd05271193108f04e331ceb691208fa3e9fcbf463897c00c11c0f2871505b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-i, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-07-26T12:42:45.026 INFO:journalctl@ceph.mds.i.smithi132.stdout:Jul 26 12:42:45 smithi132 podman[70596]: 2024-07-26 12:42:45.025241872 +0000 UTC m=+0.385705781 container init 92dd05271193108f04e331ceb691208fa3e9fcbf463897c00c11c0f2871505b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:42:45.310 INFO:journalctl@ceph.mds.i.smithi132.stdout:Jul 26 12:42:45 smithi132 podman[70596]: 2024-07-26 12:42:45.02778786 +0000 UTC m=+0.388251760 container start 92dd05271193108f04e331ceb691208fa3e9fcbf463897c00c11c0f2871505b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-07-26T12:42:45.310 INFO:journalctl@ceph.mds.i.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-mds[70615]: -- 172.21.15.132:0/1719258880 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5647186eb860 con 0x56471c88d180 2024-07-26T12:42:45.310 INFO:journalctl@ceph.mds.i.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-i[70611]: starting mds.i at 2024-07-26T12:42:45.310 INFO:journalctl@ceph.mds.i.smithi132.stdout:Jul 26 12:42:45 smithi132 bash[70596]: 92dd05271193108f04e331ceb691208fa3e9fcbf463897c00c11c0f2871505b9 2024-07-26T12:42:45.310 INFO:journalctl@ceph.mds.i.smithi132.stdout:Jul 26 12:42:45 smithi132 systemd[1]: Started Ceph mds.i for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.380+0000 7f665f800640 1 -- 172.21.15.107:0/4232399459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6658102a10 msgr2=0x7f665810f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.380+0000 7f665f800640 1 --2- 172.21.15.107:0/4232399459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6658102a10 0x7f665810f730 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6644009930 tx=0x7f664402a4d0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.380+0000 7f665f800640 1 -- 172.21.15.107:0/4232399459 shutdown_connections 2024-07-26T12:42:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.380+0000 7f665f800640 1 --2- 172.21.15.107:0/4232399459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665810fc70 0x7f6658112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.380+0000 7f665f800640 1 --2- 172.21.15.107:0/4232399459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6658102a10 0x7f665810f730 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.380+0000 7f665f800640 1 --2- 172.21.15.107:0/4232399459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66581020c0 0x7f66581024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.380+0000 7f665f800640 1 -- 172.21.15.107:0/4232399459 >> 172.21.15.107:0/4232399459 conn(0x7f6658079150 msgr2=0x7f6658079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:45.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.380+0000 7f665f800640 1 -- 172.21.15.107:0/4232399459 shutdown_connections 2024-07-26T12:42:45.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.380+0000 7f665f800640 1 -- 172.21.15.107:0/4232399459 wait complete. 2024-07-26T12:42:45.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.381+0000 7f665f800640 1 Processor -- start 2024-07-26T12:42:45.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f665f800640 1 -- start start 2024-07-26T12:42:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f665f800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66581020c0 0x7f665819c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f665f800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6658102a10 0x7f665819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f665f800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665810fc70 0x7f66581a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f665f800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f66581156d0 con 0x7f665810fc70 2024-07-26T12:42:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f665f800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6658115550 con 0x7f66581020c0 2024-07-26T12:42:45.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f665f800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6658115850 con 0x7f6658102a10 2024-07-26T12:42:45.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f6657e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6658102a10 0x7f665819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:45.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f665d800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665810fc70 0x7f66581a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:45.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f6657e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6658102a10 0x7f665819c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33716/0 (socket says 172.21.15.107:33716) 2024-07-26T12:42:45.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f6657e00640 1 -- 172.21.15.107:0/3980056086 learned_addr learned my addr 172.21.15.107:0/3980056086 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:42:45.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.382+0000 7f665ce00640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66581020c0 0x7f665819c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.383+0000 7f665ce00640 1 -- 172.21.15.107:0/3980056086 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6658102a10 msgr2=0x7f665819c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.383+0000 7f665ce00640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6658102a10 0x7f665819c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.383+0000 7f665ce00640 1 -- 172.21.15.107:0/3980056086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665810fc70 msgr2=0x7f66581a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.383+0000 7f665ce00640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665810fc70 0x7f66581a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.383+0000 7f665ce00640 1 -- 172.21.15.107:0/3980056086 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66581a2110 con 0x7f66581020c0 2024-07-26T12:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.383+0000 7f665ce00640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66581020c0 0x7f665819c1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f664c00e9f0 tx=0x7f664c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.383+0000 7f6655600640 1 -- 172.21.15.107:0/3980056086 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c00ccc0 con 0x7f66581020c0 2024-07-26T12:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.384+0000 7f6655600640 1 -- 172.21.15.107:0/3980056086 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f664c0108e0 con 0x7f66581020c0 2024-07-26T12:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.384+0000 7f665f800640 1 -- 172.21.15.107:0/3980056086 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66581a2400 con 0x7f66581020c0 2024-07-26T12:42:45.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.384+0000 7f6655600640 1 -- 172.21.15.107:0/3980056086 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c018840 con 0x7f66581020c0 2024-07-26T12:42:45.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.384+0000 7f665f800640 1 -- 172.21.15.107:0/3980056086 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66581a8b60 con 0x7f66581020c0 2024-07-26T12:42:45.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.385+0000 7f665f800640 1 -- 172.21.15.107:0/3980056086 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66581024d0 con 0x7f66581020c0 2024-07-26T12:42:45.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.387+0000 7f6655600640 1 -- 172.21.15.107:0/3980056086 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f664c010430 con 0x7f66581020c0 2024-07-26T12:42:45.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.389+0000 7f6655600640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f66400b3ca0 0x7f66400b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:45.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.389+0000 7f6655600640 1 -- 172.21.15.107:0/3980056086 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f664c0e3a70 con 0x7f66581020c0 2024-07-26T12:42:45.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.389+0000 7f6657e00640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f66400b3ca0 0x7f66400b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:45.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.390+0000 7f6657e00640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f66400b3ca0 0x7f66400b61b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6644000c00 tx=0x7f6644005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:45.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.393+0000 7f6655600640 1 -- 172.21.15.107:0/3980056086 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f664c0ab490 con 0x7f66581020c0 2024-07-26T12:42:45.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.636+0000 7f665f800640 1 -- 172.21.15.107:0/3980056086 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66581a8ee0 con 0x7f66581020c0 2024-07-26T12:42:45.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.637+0000 7f6655600640 1 -- 172.21.15.107:0/3980056086 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7228 (secure 0 0 0) 0x7f664c0aad90 con 0x7f66581020c0 2024-07-26T12:42:45.638 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:42:45.638 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":9,"btime":"2024-07-26T12:42:42:451028+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24632,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24634,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24652,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24662,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24680,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-07-26T12:42:45.638 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 9 2024-07-26T12:42:45.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 -- 172.21.15.107:0/3980056086 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f66400b3ca0 msgr2=0x7f66400b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f66400b3ca0 0x7f66400b61b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6644000c00 tx=0x7f6644005a20 comp rx=0 tx=0).stop 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 -- 172.21.15.107:0/3980056086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66581020c0 msgr2=0x7f665819c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66581020c0 0x7f665819c1a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f664c00e9f0 tx=0x7f664c00edf0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 -- 172.21.15.107:0/3980056086 shutdown_connections 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f66400b3ca0 0x7f66400b61b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f665810fc70 0x7f66581a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6658102a10 0x7f665819c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 --2- 172.21.15.107:0/3980056086 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f66581020c0 0x7f665819c1a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 -- 172.21.15.107:0/3980056086 >> 172.21.15.107:0/3980056086 conn(0x7f6658079150 msgr2=0x7f6658109810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 -- 172.21.15.107:0/3980056086 shutdown_connections 2024-07-26T12:42:45.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:45.640+0000 7f665f800640 1 -- 172.21.15.107:0/3980056086 wait complete. 2024-07-26T12:42:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:45 smithi112 ceph-mon[37501]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:45 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3980056086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:46.032 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:45 smithi107 ceph-mon[32659]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:46.032 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:46.032 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:46.033 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:46.033 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:46.033 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:46.033 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:46.033 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:46.033 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:45 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3980056086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-mon[37580]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:45 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3980056086' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:46.897 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:46 smithi107 ceph-mon[32659]: Deploying daemon mds.g on smithi107 2024-07-26T12:42:46.897 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T12:42:46.898 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:46 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] up:boot 2024-07-26T12:42:46.898 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:46 smithi107 ceph-mon[32659]: fsmap 9 up:standby 2024-07-26T12:42:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:46 smithi132 ceph-mon[37580]: Deploying daemon mds.g on smithi107 2024-07-26T12:42:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T12:42:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:46 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] up:boot 2024-07-26T12:42:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:46 smithi132 ceph-mon[37580]: fsmap 9 up:standby 2024-07-26T12:42:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:46 smithi112 ceph-mon[37501]: Deploying daemon mds.g on smithi107 2024-07-26T12:42:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T12:42:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:46 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] up:boot 2024-07-26T12:42:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:46 smithi112 ceph-mon[37501]: fsmap 9 up:standby 2024-07-26T12:42:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:47 smithi107 ceph-mon[32659]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:47 smithi132 ceph-mon[37580]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:47 smithi112 ceph-mon[37501]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:48.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:42:48.982 INFO:journalctl@ceph.mds.g.smithi107.stdout:Jul 26 12:42:48 smithi107 podman[77104]: 2024-07-26 12:42:48.734947627 +0000 UTC m=+0.136096393 container create f40a547edc8e8ba20144ba2a1e29631a793fda3b5a5e6cdc4a20f6c897a775d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-g, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-07-26T12:42:48.982 INFO:journalctl@ceph.mds.g.smithi107.stdout:Jul 26 12:42:48 smithi107 podman[77104]: 2024-07-26 12:42:48.980355594 +0000 UTC m=+0.381504364 container init f40a547edc8e8ba20144ba2a1e29631a793fda3b5a5e6cdc4a20f6c897a775d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-07-26T12:42:49.163 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:42:49.292 INFO:journalctl@ceph.mds.g.smithi107.stdout:Jul 26 12:42:48 smithi107 podman[77104]: 2024-07-26 12:42:48.990940469 +0000 UTC m=+0.392089205 container start f40a547edc8e8ba20144ba2a1e29631a793fda3b5a5e6cdc4a20f6c897a775d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-g, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:42:49.292 INFO:journalctl@ceph.mds.g.smithi107.stdout:Jul 26 12:42:49 smithi107 bash[77104]: f40a547edc8e8ba20144ba2a1e29631a793fda3b5a5e6cdc4a20f6c897a775d8 2024-07-26T12:42:49.292 INFO:journalctl@ceph.mds.g.smithi107.stdout:Jul 26 12:42:49 smithi107 systemd[1]: Started Ceph mds.g for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:49.292 INFO:journalctl@ceph.mds.g.smithi107.stdout:Jul 26 12:42:49 smithi107 ceph-mds[77150]: -- 172.21.15.107:0/1282476653 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ebf485860 con 0x562ec3627180 2024-07-26T12:42:49.292 INFO:journalctl@ceph.mds.g.smithi107.stdout:Jul 26 12:42:49 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-g[77146]: starting mds.g at 2024-07-26T12:42:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:49 smithi107 ceph-mon[32659]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:49 smithi132 ceph-mon[37580]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:49 smithi112 ceph-mon[37501]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:50.993 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:50 smithi107 ceph-mon[32659]: Deploying daemon mds.a on smithi107 2024-07-26T12:42:50.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T12:42:50.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:50 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] up:boot 2024-07-26T12:42:50.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:50 smithi107 ceph-mon[32659]: fsmap 10 up:standby 2024-07-26T12:42:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:50 smithi132 ceph-mon[37580]: Deploying daemon mds.a on smithi107 2024-07-26T12:42:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T12:42:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:50 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] up:boot 2024-07-26T12:42:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:50 smithi132 ceph-mon[37580]: fsmap 10 up:standby 2024-07-26T12:42:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:50 smithi112 ceph-mon[37501]: Deploying daemon mds.a on smithi107 2024-07-26T12:42:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T12:42:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:50 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] up:boot 2024-07-26T12:42:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:50 smithi112 ceph-mon[37501]: fsmap 10 up:standby 2024-07-26T12:42:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.437+0000 7f344ba00640 1 -- 172.21.15.107:0/126020326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f344410d870 msgr2=0x7f344410fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.437+0000 7f344ba00640 1 --2- 172.21.15.107:0/126020326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f344410d870 0x7f344410fd40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f343c009900 tx=0x7f343c02a410 comp rx=0 tx=0).stop 2024-07-26T12:42:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.438+0000 7f344ba00640 1 -- 172.21.15.107:0/126020326 shutdown_connections 2024-07-26T12:42:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.438+0000 7f344ba00640 1 --2- 172.21.15.107:0/126020326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f344410d870 0x7f344410fd40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.438+0000 7f344ba00640 1 --2- 172.21.15.107:0/126020326 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3444104cf0 0x7f344410d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.438+0000 7f344ba00640 1 --2- 172.21.15.107:0/126020326 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34441042c0 0x7f34441046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.438+0000 7f344ba00640 1 -- 172.21.15.107:0/126020326 >> 172.21.15.107:0/126020326 conn(0x7f3444079160 msgr2=0x7f3444079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:51.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.438+0000 7f344ba00640 1 -- 172.21.15.107:0/126020326 shutdown_connections 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.438+0000 7f344ba00640 1 -- 172.21.15.107:0/126020326 wait complete. 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.439+0000 7f344ba00640 1 Processor -- start 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.439+0000 7f344ba00640 1 -- start start 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.439+0000 7f344ba00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34441042c0 0x7f344419c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f344ba00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3444104cf0 0x7f344419c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f344ba00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f344410d870 0x7f34441a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f344ba00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f34441134b0 con 0x7f3444104cf0 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f344ba00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3444113330 con 0x7f34441042c0 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f344ba00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3444113630 con 0x7f344410d870 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f3443e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3444104cf0 0x7f344419c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f3443e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3444104cf0 0x7f344419c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39504/0 (socket says 172.21.15.107:39504) 2024-07-26T12:42:51.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f3443e00640 1 -- 172.21.15.107:0/2689350708 learned_addr learned my addr 172.21.15.107:0/2689350708 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f3449000640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34441042c0 0x7f344419c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f3449a00640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f344410d870 0x7f34441a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f3443e00640 1 -- 172.21.15.107:0/2689350708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f344410d870 msgr2=0x7f34441a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f3443e00640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f344410d870 0x7f34441a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f3443e00640 1 -- 172.21.15.107:0/2689350708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34441042c0 msgr2=0x7f344419c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f3443e00640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34441042c0 0x7f344419c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.440+0000 7f3443e00640 1 -- 172.21.15.107:0/2689350708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34441a2030 con 0x7f3444104cf0 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.441+0000 7f3443e00640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3444104cf0 0x7f344419c660 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f342c00e9c0 tx=0x7f342c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.441+0000 7f3441600640 1 -- 172.21.15.107:0/2689350708 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c00cd80 con 0x7f3444104cf0 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.441+0000 7f344ba00640 1 -- 172.21.15.107:0/2689350708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34441a2320 con 0x7f3444104cf0 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.441+0000 7f3441600640 1 -- 172.21.15.107:0/2689350708 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f342c0054d0 con 0x7f3444104cf0 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.441+0000 7f3441600640 1 -- 172.21.15.107:0/2689350708 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c010430 con 0x7f3444104cf0 2024-07-26T12:42:51.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.441+0000 7f344ba00640 1 -- 172.21.15.107:0/2689350708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34441a8a80 con 0x7f3444104cf0 2024-07-26T12:42:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.442+0000 7f344ba00640 1 -- 172.21.15.107:0/2689350708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f344406bb70 con 0x7f3444104cf0 2024-07-26T12:42:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.445+0000 7f3441600640 1 -- 172.21.15.107:0/2689350708 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f342c004fb0 con 0x7f3444104cf0 2024-07-26T12:42:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.447+0000 7f3441600640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34280b3c20 0x7f34280b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.447+0000 7f3441600640 1 -- 172.21.15.107:0/2689350708 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f342c0e3ff0 con 0x7f3444104cf0 2024-07-26T12:42:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.447+0000 7f3449000640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34280b3c20 0x7f34280b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.448+0000 7f3449000640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34280b3c20 0x7f34280b6130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f344406c270 tx=0x7f3434009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:51.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.448+0000 7f3441600640 1 -- 172.21.15.107:0/2689350708 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f342c055600 con 0x7f3444104cf0 2024-07-26T12:42:51.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.687+0000 7f344ba00640 1 -- 172.21.15.107:0/2689350708 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34441046d0 con 0x7f3444104cf0 2024-07-26T12:42:51.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.688+0000 7f3441600640 1 -- 172.21.15.107:0/2689350708 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8884 (secure 0 0 0) 0x7f342c0ab990 con 0x7f3444104cf0 2024-07-26T12:42:51.690 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:42:51.691 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":11,"btime":"2024-07-26T12:42:49:707138+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24632,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24634,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24652,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24662,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24680,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24703,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24715,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-07-26T12:42:51.691 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 11 2024-07-26T12:42:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.690+0000 7f344ba00640 1 -- 172.21.15.107:0/2689350708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34280b3c20 msgr2=0x7f34280b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.690+0000 7f344ba00640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34280b3c20 0x7f34280b6130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f344406c270 tx=0x7f3434009500 comp rx=0 tx=0).stop 2024-07-26T12:42:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.691+0000 7f344ba00640 1 -- 172.21.15.107:0/2689350708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3444104cf0 msgr2=0x7f344419c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.691+0000 7f344ba00640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3444104cf0 0x7f344419c660 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f342c00e9c0 tx=0x7f342c00ee80 comp rx=0 tx=0).stop 2024-07-26T12:42:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.691+0000 7f344ba00640 1 -- 172.21.15.107:0/2689350708 shutdown_connections 2024-07-26T12:42:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.691+0000 7f344ba00640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34280b3c20 0x7f34280b6130 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.691+0000 7f344ba00640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f344410d870 0x7f34441a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.691+0000 7f344ba00640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3444104cf0 0x7f344419c660 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:51.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.691+0000 7f344ba00640 1 --2- 172.21.15.107:0/2689350708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34441042c0 0x7f344419c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:51.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.691+0000 7f344ba00640 1 -- 172.21.15.107:0/2689350708 >> 172.21.15.107:0/2689350708 conn(0x7f3444079160 msgr2=0x7f3444100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:51.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.691+0000 7f344ba00640 1 -- 172.21.15.107:0/2689350708 shutdown_connections 2024-07-26T12:42:51.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:51.691+0000 7f344ba00640 1 -- 172.21.15.107:0/2689350708 wait complete. 2024-07-26T12:42:51.897 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:51 smithi107 ceph-mon[32659]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:51.898 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:51 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2689350708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:51 smithi132 ceph-mon[37580]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:51 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2689350708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:51 smithi112 ceph-mon[37501]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:51 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2689350708' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:53.943 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:53 smithi107 ceph-mon[32659]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:53 smithi132 ceph-mon[37580]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:53 smithi112 ceph-mon[37501]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:54.265 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:42:53 smithi107 podman[77539]: 2024-07-26 12:42:53.976298795 +0000 UTC m=+0.165236879 container create a91deb8200a253abaae47b4ed76dc2ca433b31ed0dd4752d8119c7f3cfda8cd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-07-26T12:42:54.265 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:42:54 smithi107 podman[77539]: 2024-07-26 12:42:54.262968191 +0000 UTC m=+0.451906264 container init a91deb8200a253abaae47b4ed76dc2ca433b31ed0dd4752d8119c7f3cfda8cd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-07-26T12:42:54.542 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:42:54 smithi107 podman[77539]: 2024-07-26 12:42:54.272736286 +0000 UTC m=+0.461674386 container start a91deb8200a253abaae47b4ed76dc2ca433b31ed0dd4752d8119c7f3cfda8cd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True) 2024-07-26T12:42:54.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:42:54 smithi107 bash[77539]: a91deb8200a253abaae47b4ed76dc2ca433b31ed0dd4752d8119c7f3cfda8cd8 2024-07-26T12:42:54.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:42:54 smithi107 systemd[1]: Started Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:54.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:42:54 smithi107 ceph-mds[77558]: -- 172.21.15.107:0/3291378587 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5555e9e89860 con 0x5555ee02b600 2024-07-26T12:42:54.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:42:54 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: starting mds.a at 2024-07-26T12:42:55.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:42:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: Deploying daemon mds.b on smithi112 2024-07-26T12:42:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:42:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:boot 2024-07-26T12:42:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:55 smithi107 ceph-mon[32659]: fsmap 11 up:standby 2024-07-26T12:42:55.699 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:42:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:55.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:55.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:55.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: Deploying daemon mds.b on smithi112 2024-07-26T12:42:55.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:55.814 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:42:55.814 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:boot 2024-07-26T12:42:55.814 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:55 smithi132 ceph-mon[37580]: fsmap 11 up:standby 2024-07-26T12:42:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T12:42:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T12:42:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: Deploying daemon mds.b on smithi112 2024-07-26T12:42:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:42:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:boot 2024-07-26T12:42:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:55 smithi112 ceph-mon[37501]: fsmap 11 up:standby 2024-07-26T12:42:56.685 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:42:56 smithi112 systemd[1]: Starting Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:42:56.950 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:42:56 smithi112 podman[69118]: 2024-07-26 12:42:56.586281283 +0000 UTC m=+0.017236253 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:42:56.950 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:42:56 smithi112 podman[69118]: 2024-07-26 12:42:56.71125923 +0000 UTC m=+0.142214234 container create 79e81ecd91c8b7344c1b48f68400eb126e9209f3fc49cb85561891912b22dad9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:42:57.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:42:56 smithi112 podman[69118]: 2024-07-26 12:42:56.959065749 +0000 UTC m=+0.390020710 container init 79e81ecd91c8b7344c1b48f68400eb126e9209f3fc49cb85561891912b22dad9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-07-26T12:42:57.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:42:56 smithi112 podman[69118]: 2024-07-26 12:42:56.961576305 +0000 UTC m=+0.392531267 container start 79e81ecd91c8b7344c1b48f68400eb126e9209f3fc49cb85561891912b22dad9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, ceph=True) 2024-07-26T12:42:57.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:42:56 smithi112 ceph-mds[69137]: -- 172.21.15.112:0/2348828278 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eaadf99860 con 0x55eab213b600 2024-07-26T12:42:57.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:42:56 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: starting mds.b at 2024-07-26T12:42:57.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:42:57 smithi112 bash[69118]: 79e81ecd91c8b7344c1b48f68400eb126e9209f3fc49cb85561891912b22dad9 2024-07-26T12:42:57.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:42:57 smithi112 systemd[1]: Started Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:42:57.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.871+0000 7fbdc5800640 1 -- 172.21.15.107:0/3679764363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbdc010b2d0 msgr2=0x7fbdc010b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:57.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.871+0000 7fbdc5800640 1 --2- 172.21.15.107:0/3679764363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbdc010b2d0 0x7fbdc010b6e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbdac009900 tx=0x7fbdac02a410 comp rx=0 tx=0).stop 2024-07-26T12:42:57.874 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:57 smithi107 ceph-mon[32659]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:57.874 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:57.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:57.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:57.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:57.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:57.875 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:57.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.871+0000 7fbdc5800640 1 -- 172.21.15.107:0/3679764363 shutdown_connections 2024-07-26T12:42:57.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.871+0000 7fbdc5800640 1 --2- 172.21.15.107:0/3679764363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc010bc20 0x7fbdc0112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:57.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.871+0000 7fbdc5800640 1 --2- 172.21.15.107:0/3679764363 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbdc010b2d0 0x7fbdc010b6e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.871+0000 7fbdc5800640 1 --2- 172.21.15.107:0/3679764363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc0105890 0x7fbdc0105ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.871+0000 7fbdc5800640 1 -- 172.21.15.107:0/3679764363 >> 172.21.15.107:0/3679764363 conn(0x7fbdc0101340 msgr2=0x7fbdc0103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.871+0000 7fbdc5800640 1 -- 172.21.15.107:0/3679764363 shutdown_connections 2024-07-26T12:42:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.871+0000 7fbdc5800640 1 -- 172.21.15.107:0/3679764363 wait complete. 2024-07-26T12:42:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.872+0000 7fbdc5800640 1 Processor -- start 2024-07-26T12:42:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.873+0000 7fbdc5800640 1 -- start start 2024-07-26T12:42:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.873+0000 7fbdc5800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc0105890 0x7fbdc019c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.873+0000 7fbdc5800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbdc010b2d0 0x7fbdc019c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.873+0000 7fbdc5800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc010bc20 0x7fbdc01a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.873+0000 7fbdc5800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc0115840 con 0x7fbdc010bc20 2024-07-26T12:42:57.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.873+0000 7fbdc5800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc01156c0 con 0x7fbdc010b2d0 2024-07-26T12:42:57.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.873+0000 7fbdc5800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc01159c0 con 0x7fbdc0105890 2024-07-26T12:42:57.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbe000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbdc010b2d0 0x7fbdc019c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:57.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbe000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbdc010b2d0 0x7fbdc019c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:52844/0 (socket says 172.21.15.107:52844) 2024-07-26T12:42:57.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbe000640 1 -- 172.21.15.107:0/2172721048 learned_addr learned my addr 172.21.15.107:0/2172721048 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:42:57.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbea00640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc0105890 0x7fbdc019c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:57.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbf400640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc010bc20 0x7fbdc01a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:57.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbe000640 1 -- 172.21.15.107:0/2172721048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc0105890 msgr2=0x7fbdc019c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:57.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbe000640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc0105890 0x7fbdc019c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:57.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbe000640 1 -- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc010bc20 msgr2=0x7fbdc01a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:57.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbe000640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc010bc20 0x7fbdc01a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:57.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbe000640 1 -- 172.21.15.107:0/2172721048 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc01a2140 con 0x7fbdc010b2d0 2024-07-26T12:42:57.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbf400640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc010bc20 0x7fbdc01a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:42:57.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.874+0000 7fbdbe000640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbdc010b2d0 0x7fbdc019c770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbdac02a930 tx=0x7fbdac031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:57.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.875+0000 7fbdb7400640 1 -- 172.21.15.107:0/2172721048 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac03f030 con 0x7fbdc010b2d0 2024-07-26T12:42:57.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.875+0000 7fbdc5800640 1 -- 172.21.15.107:0/2172721048 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc01a23d0 con 0x7fbdc010b2d0 2024-07-26T12:42:57.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.875+0000 7fbdb7400640 1 -- 172.21.15.107:0/2172721048 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdac004490 con 0x7fbdc010b2d0 2024-07-26T12:42:57.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.875+0000 7fbdb7400640 1 -- 172.21.15.107:0/2172721048 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac035070 con 0x7fbdc010b2d0 2024-07-26T12:42:57.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.875+0000 7fbdc5800640 1 -- 172.21.15.107:0/2172721048 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc01a8b90 con 0x7fbdc010b2d0 2024-07-26T12:42:57.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.876+0000 7fbdc5800640 1 -- 172.21.15.107:0/2172721048 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdc0105ca0 con 0x7fbdc010b2d0 2024-07-26T12:42:57.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.879+0000 7fbdb7400640 1 -- 172.21.15.107:0/2172721048 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fbdac042030 con 0x7fbdc010b2d0 2024-07-26T12:42:57.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.881+0000 7fbdb7400640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbda40b3ca0 0x7fbda40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:57.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.881+0000 7fbdb7400640 1 -- 172.21.15.107:0/2172721048 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fbdac103ca0 con 0x7fbdc010b2d0 2024-07-26T12:42:57.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.881+0000 7fbdbea00640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbda40b3ca0 0x7fbda40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:57.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.882+0000 7fbdbea00640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbda40b3ca0 0x7fbda40b61b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009b60 tx=0x7fbdb0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:57.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:57.885+0000 7fbdb7400640 1 -- 172.21.15.107:0/2172721048 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbdac0cb660 con 0x7fbdc010b2d0 2024-07-26T12:42:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:57 smithi112 ceph-mon[37501]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:58.014 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:57 smithi132 ceph-mon[37580]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:58.014 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:58.015 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:58.015 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:58.015 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:58.015 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:58.015 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:58.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.140+0000 7fbdc5800640 1 -- 172.21.15.107:0/2172721048 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdc010d9f0 con 0x7fbdc010b2d0 2024-07-26T12:42:58.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.141+0000 7fbdb7400640 1 -- 172.21.15.107:0/2172721048 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10539 (secure 0 0 0) 0x7fbdac0caf60 con 0x7fbdc010b2d0 2024-07-26T12:42:58.144 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:42:58.144 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":13,"btime":"2024-07-26T12:42:57:691087+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24632,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24634,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24652,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24662,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24680,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24703,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24715,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24716,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-07-26T12:42:58.145 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 13 2024-07-26T12:42:58.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.143+0000 7fbdc5800640 1 -- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbda40b3ca0 msgr2=0x7fbda40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:58.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.143+0000 7fbdc5800640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbda40b3ca0 0x7fbda40b61b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009b60 tx=0x7fbdb0005f90 comp rx=0 tx=0).stop 2024-07-26T12:42:58.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.143+0000 7fbdc5800640 1 -- 172.21.15.107:0/2172721048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbdc010b2d0 msgr2=0x7fbdc019c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:42:58.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.143+0000 7fbdc5800640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbdc010b2d0 0x7fbdc019c770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbdac02a930 tx=0x7fbdac031a20 comp rx=0 tx=0).stop 2024-07-26T12:42:58.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.144+0000 7fbdc5800640 1 -- 172.21.15.107:0/2172721048 shutdown_connections 2024-07-26T12:42:58.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.144+0000 7fbdc5800640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbda40b3ca0 0x7fbda40b61b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:58.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.144+0000 7fbdc5800640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdc010bc20 0x7fbdc01a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:58.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.144+0000 7fbdc5800640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbdc010b2d0 0x7fbdc019c770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:58.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.144+0000 7fbdc5800640 1 --2- 172.21.15.107:0/2172721048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbdc0105890 0x7fbdc019c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:58.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.144+0000 7fbdc5800640 1 -- 172.21.15.107:0/2172721048 >> 172.21.15.107:0/2172721048 conn(0x7fbdc0101340 msgr2=0x7fbdc0103760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:42:58.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.144+0000 7fbdc5800640 1 -- 172.21.15.107:0/2172721048 shutdown_connections 2024-07-26T12:42:58.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:42:58.144+0000 7fbdc5800640 1 -- 172.21.15.107:0/2172721048 wait complete. 2024-07-26T12:42:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:42:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:58 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] up:boot 2024-07-26T12:42:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:58 smithi107 ceph-mon[32659]: fsmap 12 up:standby 2024-07-26T12:42:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:58 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2172721048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:42:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:58 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] up:boot 2024-07-26T12:42:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:58 smithi132 ceph-mon[37580]: fsmap 12 up:standby 2024-07-26T12:42:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:58 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2172721048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:42:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:58 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] up:boot 2024-07-26T12:42:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:58 smithi112 ceph-mon[37501]: fsmap 12 up:standby 2024-07-26T12:42:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:58 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2172721048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:42:59.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-07-26T12:42:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:59 smithi112 ceph-mon[37501]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:42:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:42:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:42:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.039 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:59 smithi132 ceph-mon[37580]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:00.039 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.039 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.039 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.040 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.040 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:42:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:59 smithi107 ceph-mon[32659]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:42:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:00.102 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:01 smithi107 ceph-mon[32659]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:01 smithi132 ceph-mon[37580]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:02.145 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:01 smithi112 ceph-mon[37501]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:02.146 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:02.146 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:02.146 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:02.146 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:02.146 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:02.146 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:02.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.290+0000 7fe4bac00640 1 -- 172.21.15.107:0/2699615930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4b41035a0 msgr2=0x7fe4b41039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:02.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.290+0000 7fe4bac00640 1 --2- 172.21.15.107:0/2699615930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4b41035a0 0x7fe4b41039b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009900 tx=0x7fe4a402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:43:02.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.290+0000 7fe4bac00640 1 -- 172.21.15.107:0/2699615930 shutdown_connections 2024-07-26T12:43:02.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.290+0000 7fe4bac00640 1 --2- 172.21.15.107:0/2699615930 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b4109830 0x7fe4b4110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:02.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.290+0000 7fe4bac00640 1 --2- 172.21.15.107:0/2699615930 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4b4108ee0 0x7fe4b41092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:02.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.290+0000 7fe4bac00640 1 --2- 172.21.15.107:0/2699615930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4b41035a0 0x7fe4b41039b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:02.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.290+0000 7fe4bac00640 1 -- 172.21.15.107:0/2699615930 >> 172.21.15.107:0/2699615930 conn(0x7fe4b40ff010 msgr2=0x7fe4b4101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:02.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.290+0000 7fe4bac00640 1 -- 172.21.15.107:0/2699615930 shutdown_connections 2024-07-26T12:43:02.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.290+0000 7fe4bac00640 1 -- 172.21.15.107:0/2699615930 wait complete. 2024-07-26T12:43:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.291+0000 7fe4bac00640 1 Processor -- start 2024-07-26T12:43:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.291+0000 7fe4bac00640 1 -- start start 2024-07-26T12:43:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4bac00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4b41035a0 0x7fe4b419c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4bac00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b4108ee0 0x7fe4b419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4bac00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4b4109830 0x7fe4b41a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4bac00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b4113550 con 0x7fe4b41035a0 2024-07-26T12:43:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4bac00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b41133d0 con 0x7fe4b4108ee0 2024-07-26T12:43:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4bac00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b41136d0 con 0x7fe4b4109830 2024-07-26T12:43:02.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4b41035a0 0x7fe4b419c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:02.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4b3400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b4108ee0 0x7fe4b419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:02.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4b41035a0 0x7fe4b419c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37702/0 (socket says 172.21.15.107:37702) 2024-07-26T12:43:02.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4b3400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b4108ee0 0x7fe4b419c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:52866/0 (socket says 172.21.15.107:52866) 2024-07-26T12:43:02.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4b3e00640 1 -- 172.21.15.107:0/646944511 learned_addr learned my addr 172.21.15.107:0/646944511 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:02.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.292+0000 7fe4b8c00640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4b4109830 0x7fe4b41a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.293+0000 7fe4b3400640 1 -- 172.21.15.107:0/646944511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4b4109830 msgr2=0x7fe4b41a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.293+0000 7fe4b3400640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4b4109830 0x7fe4b41a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.293+0000 7fe4b3400640 1 -- 172.21.15.107:0/646944511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4b41035a0 msgr2=0x7fe4b419c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.293+0000 7fe4b3400640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4b41035a0 0x7fe4b419c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.293+0000 7fe4b3400640 1 -- 172.21.15.107:0/646944511 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b41a20b0 con 0x7fe4b4108ee0 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.293+0000 7fe4b3400640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b4108ee0 0x7fe4b419c6e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe4b406b770 tx=0x7fe4a000ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.293+0000 7fe4b0c00640 1 -- 172.21.15.107:0/646944511 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a000ccc0 con 0x7fe4b4108ee0 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.293+0000 7fe4b0c00640 1 -- 172.21.15.107:0/646944511 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4a0009e40 con 0x7fe4b4108ee0 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.293+0000 7fe4bac00640 1 -- 172.21.15.107:0/646944511 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b41a23a0 con 0x7fe4b4108ee0 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.294+0000 7fe4b0c00640 1 -- 172.21.15.107:0/646944511 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0004a00 con 0x7fe4b4108ee0 2024-07-26T12:43:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.294+0000 7fe4bac00640 1 -- 172.21.15.107:0/646944511 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b41a8b00 con 0x7fe4b4108ee0 2024-07-26T12:43:02.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.295+0000 7fe4bac00640 1 -- 172.21.15.107:0/646944511 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b41039b0 con 0x7fe4b4108ee0 2024-07-26T12:43:02.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.297+0000 7fe4b0c00640 1 -- 172.21.15.107:0/646944511 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fe4a0005020 con 0x7fe4b4108ee0 2024-07-26T12:43:02.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.299+0000 7fe4b0c00640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4980b3c20 0x7fe4980b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:02.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.299+0000 7fe4b0c00640 1 -- 172.21.15.107:0/646944511 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7fe4a00e3a20 con 0x7fe4b4108ee0 2024-07-26T12:43:02.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.299+0000 7fe4b3e00640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4980b3c20 0x7fe4980b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:02.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.300+0000 7fe4b3e00640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4980b3c20 0x7fe4980b6130 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40096c0 tx=0x7fe4a4005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:02.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.300+0000 7fe4b0c00640 1 -- 172.21.15.107:0/646944511 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe4a00ab3a0 con 0x7fe4b4108ee0 2024-07-26T12:43:02.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.506+0000 7fe4bac00640 1 -- 172.21.15.107:0/646944511 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe4b41a8d60 con 0x7fe4b4108ee0 2024-07-26T12:43:02.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.506+0000 7fe4b0c00640 1 -- 172.21.15.107:0/646944511 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7fe4a00aaca0 con 0x7fe4b4108ee0 2024-07-26T12:43:02.509 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:43:02.509 INFO:teuthology.orchestra.run.smithi107.stdout:[ 2024-07-26T12:43:02.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-07-26T12:43:02.509 INFO:teuthology.orchestra.run.smithi107.stdout: "poolnum": 1, 2024-07-26T12:43:02.509 INFO:teuthology.orchestra.run.smithi107.stdout: "poolname": ".mgr" 2024-07-26T12:43:02.509 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-07-26T12:43:02.509 INFO:teuthology.orchestra.run.smithi107.stdout:] 2024-07-26T12:43:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 -- 172.21.15.107:0/646944511 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4980b3c20 msgr2=0x7fe4980b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4980b3c20 0x7fe4980b6130 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40096c0 tx=0x7fe4a4005a20 comp rx=0 tx=0).stop 2024-07-26T12:43:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 -- 172.21.15.107:0/646944511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b4108ee0 msgr2=0x7fe4b419c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b4108ee0 0x7fe4b419c6e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe4b406b770 tx=0x7fe4a000ed90 comp rx=0 tx=0).stop 2024-07-26T12:43:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 -- 172.21.15.107:0/646944511 shutdown_connections 2024-07-26T12:43:02.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4980b3c20 0x7fe4980b6130 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4b4109830 0x7fe4b41a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe4b4108ee0 0x7fe4b419c6e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 --2- 172.21.15.107:0/646944511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4b41035a0 0x7fe4b419c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 -- 172.21.15.107:0/646944511 >> 172.21.15.107:0/646944511 conn(0x7fe4b40ff010 msgr2=0x7fe4b41085e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.509+0000 7fe4bac00640 1 -- 172.21.15.107:0/646944511 shutdown_connections 2024-07-26T12:43:02.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:02.510+0000 7fe4bac00640 1 -- 172.21.15.107:0/646944511 wait complete. 2024-07-26T12:43:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:02 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/646944511' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T12:43:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:02 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/646944511' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T12:43:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:02 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/646944511' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T12:43:03.734 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:03 smithi107 ceph-mon[32659]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:03.734 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:03.734 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.008 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.008 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.011 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T12:43:04.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T12:43:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:03 smithi132 ceph-mon[37580]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:03 smithi112 ceph-mon[37501]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:04.477 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:05.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:05 smithi107 ceph-mon[32659]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:05 smithi132 ceph-mon[37580]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:05 smithi112 ceph-mon[37501]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:06.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.779+0000 7f61dda00640 1 -- 172.21.15.107:0/2320642585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d81027c0 msgr2=0x7f61d810f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:06.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.779+0000 7f61dda00640 1 --2- 172.21.15.107:0/2320642585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d81027c0 0x7f61d810f4e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f61c4009860 tx=0x7f61c402a3d0 comp rx=0 tx=0).stop 2024-07-26T12:43:06.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.779+0000 7f61dda00640 1 -- 172.21.15.107:0/2320642585 shutdown_connections 2024-07-26T12:43:06.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.779+0000 7f61dda00640 1 --2- 172.21.15.107:0/2320642585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61d810fa20 0x7f61d8111e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:06.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.779+0000 7f61dda00640 1 --2- 172.21.15.107:0/2320642585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d81027c0 0x7f61d810f4e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:06.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.779+0000 7f61dda00640 1 --2- 172.21.15.107:0/2320642585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61d8101e70 0x7f61d8102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:06.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.779+0000 7f61dda00640 1 -- 172.21.15.107:0/2320642585 >> 172.21.15.107:0/2320642585 conn(0x7f61d80fdcf0 msgr2=0x7f61d8100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:06.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.779+0000 7f61dda00640 1 -- 172.21.15.107:0/2320642585 shutdown_connections 2024-07-26T12:43:06.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.779+0000 7f61dda00640 1 -- 172.21.15.107:0/2320642585 wait complete. 2024-07-26T12:43:06.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.781+0000 7f61dda00640 1 Processor -- start 2024-07-26T12:43:06.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.781+0000 7f61dda00640 1 -- start start 2024-07-26T12:43:06.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61dda00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61d8101e70 0x7f61d819bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:06.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61dda00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d81027c0 0x7f61d819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:06.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61dda00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61d810fa20 0x7f61d81a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:06.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61dda00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f61d81154c0 con 0x7f61d81027c0 2024-07-26T12:43:06.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61dda00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f61d8115340 con 0x7f61d8101e70 2024-07-26T12:43:06.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61dda00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f61d8115640 con 0x7f61d810fa20 2024-07-26T12:43:06.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61d7e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61d8101e70 0x7f61d819bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:06.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61d7e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61d8101e70 0x7f61d819bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:33264/0 (socket says 172.21.15.107:33264) 2024-07-26T12:43:06.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61d7e00640 1 -- 172.21.15.107:0/3349999913 learned_addr learned my addr 172.21.15.107:0/3349999913 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:06.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61d7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d81027c0 0x7f61d819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:06.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.782+0000 7f61dd000640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61d810fa20 0x7f61d81a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:06.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61dd000640 1 -- 172.21.15.107:0/3349999913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61d8101e70 msgr2=0x7f61d819bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:06.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61dd000640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61d8101e70 0x7f61d819bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:06.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61dd000640 1 -- 172.21.15.107:0/3349999913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d81027c0 msgr2=0x7f61d819c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:06.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61dd000640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d81027c0 0x7f61d819c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:06.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61dd000640 1 -- 172.21.15.107:0/3349999913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d81a1e40 con 0x7f61d810fa20 2024-07-26T12:43:06.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61d7e00640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61d8101e70 0x7f61d819bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:06.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61dd000640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61d810fa20 0x7f61d81a1710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009fd0 tx=0x7f61d000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:06.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61d4c00640 1 -- 172.21.15.107:0/3349999913 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d000ce20 con 0x7f61d810fa20 2024-07-26T12:43:06.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61d4c00640 1 -- 172.21.15.107:0/3349999913 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61d0005020 con 0x7f61d810fa20 2024-07-26T12:43:06.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61dda00640 1 -- 172.21.15.107:0/3349999913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61d81a2130 con 0x7f61d810fa20 2024-07-26T12:43:06.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61d4c00640 1 -- 172.21.15.107:0/3349999913 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d0010690 con 0x7f61d810fa20 2024-07-26T12:43:06.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.783+0000 7f61dda00640 1 -- 172.21.15.107:0/3349999913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61d81a8890 con 0x7f61d810fa20 2024-07-26T12:43:06.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.785+0000 7f61dda00640 1 -- 172.21.15.107:0/3349999913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61d8102280 con 0x7f61d810fa20 2024-07-26T12:43:06.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.787+0000 7f61d4c00640 1 -- 172.21.15.107:0/3349999913 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f61d00054d0 con 0x7f61d810fa20 2024-07-26T12:43:06.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.788+0000 7f61d4c00640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f61bc0b3ca0 0x7f61bc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:06.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.789+0000 7f61d4c00640 1 -- 172.21.15.107:0/3349999913 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7342+0+0 (secure 0 0 0) 0x7f61d00e3f20 con 0x7f61d810fa20 2024-07-26T12:43:06.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.789+0000 7f61d7e00640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f61bc0b3ca0 0x7f61bc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:06.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.789+0000 7f61d7e00640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f61bc0b3ca0 0x7f61bc0b61b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f61c80058c0 tx=0x7f61c8009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:06.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.790+0000 7f61d4c00640 1 -- 172.21.15.107:0/3349999913 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f61d00ab8a0 con 0x7f61d810fa20 2024-07-26T12:43:06.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:06.988+0000 7f61dda00640 1 -- 172.21.15.107:0/3349999913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f61d819d000 con 0x7f61d810fa20 2024-07-26T12:43:07.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.103+0000 7f61d4c00640 1 -- 172.21.15.107:0/3349999913 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7f61d819d000 con 0x7f61d810fa20 2024-07-26T12:43:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_metadata' created 2024-07-26T12:43:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.111+0000 7f61dda00640 1 -- 172.21.15.107:0/3349999913 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f61bc0b3ca0 msgr2=0x7f61bc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.111+0000 7f61dda00640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f61bc0b3ca0 0x7f61bc0b61b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f61c80058c0 tx=0x7f61c8009260 comp rx=0 tx=0).stop 2024-07-26T12:43:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.111+0000 7f61dda00640 1 -- 172.21.15.107:0/3349999913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61d810fa20 msgr2=0x7f61d81a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:07.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.111+0000 7f61dda00640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61d810fa20 0x7f61d81a1710 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009fd0 tx=0x7f61d000ef10 comp rx=0 tx=0).stop 2024-07-26T12:43:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.111+0000 7f61dda00640 1 -- 172.21.15.107:0/3349999913 shutdown_connections 2024-07-26T12:43:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.111+0000 7f61dda00640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f61bc0b3ca0 0x7f61bc0b61b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.111+0000 7f61dda00640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f61d810fa20 0x7f61d81a1710 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.111+0000 7f61dda00640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61d81027c0 0x7f61d819c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.111+0000 7f61dda00640 1 --2- 172.21.15.107:0/3349999913 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f61d8101e70 0x7f61d819bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.111+0000 7f61dda00640 1 -- 172.21.15.107:0/3349999913 >> 172.21.15.107:0/3349999913 conn(0x7f61d80fdcf0 msgr2=0x7f61d8111b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:07.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.112+0000 7f61dda00640 1 -- 172.21.15.107:0/3349999913 shutdown_connections 2024-07-26T12:43:07.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:07.114+0000 7f61dda00640 1 -- 172.21.15.107:0/3349999913 wait complete. 2024-07-26T12:43:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:07 smithi107 ceph-mon[32659]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:07 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3349999913' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T12:43:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:07 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T12:43:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:07 smithi132 ceph-mon[37580]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:07 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3349999913' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T12:43:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:07 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T12:43:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:43:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:07 smithi112 ceph-mon[37501]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:07 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3349999913' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T12:43:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:07 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T12:43:07.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-07-26T12:43:08.340 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:08 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T12:43:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:08 smithi112 ceph-mon[37501]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T12:43:08.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:08 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T12:43:08.485 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:08 smithi107 ceph-mon[32659]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T12:43:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:08 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T12:43:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:08 smithi132 ceph-mon[37580]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T12:43:09.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:09 smithi112 ceph-mon[37501]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T12:43:09.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:09 smithi112 ceph-mon[37501]: pgmap v289: 33 pgs: 15 creating+peering, 1 active+clean, 17 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:09.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:09 smithi107 ceph-mon[32659]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T12:43:09.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:09 smithi107 ceph-mon[32659]: pgmap v289: 33 pgs: 15 creating+peering, 1 active+clean, 17 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:09.546 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:09 smithi132 ceph-mon[37580]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T12:43:09.546 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:09 smithi132 ceph-mon[37580]: pgmap v289: 33 pgs: 15 creating+peering, 1 active+clean, 17 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:10 smithi107 ceph-mon[32659]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:43:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:10 smithi107 ceph-mon[32659]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T12:43:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:10 smithi112 ceph-mon[37501]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:43:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:10 smithi112 ceph-mon[37501]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T12:43:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:10 smithi132 ceph-mon[37580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:43:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:10 smithi132 ceph-mon[37580]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T12:43:10.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.567+0000 7f84cfc00640 1 -- 172.21.15.107:0/1468443532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84c81057e0 msgr2=0x7f84c8105bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:10.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.567+0000 7f84cfc00640 1 --2- 172.21.15.107:0/1468443532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84c81057e0 0x7f84c8105bf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009900 tx=0x7f84bc02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:43:10.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.568+0000 7f84cfc00640 1 -- 172.21.15.107:0/1468443532 shutdown_connections 2024-07-26T12:43:10.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.568+0000 7f84cfc00640 1 --2- 172.21.15.107:0/1468443532 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84c810ba70 0x7f84c81122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:10.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.568+0000 7f84cfc00640 1 --2- 172.21.15.107:0/1468443532 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c810b120 0x7f84c810b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:10.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.568+0000 7f84cfc00640 1 --2- 172.21.15.107:0/1468443532 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84c81057e0 0x7f84c8105bf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:10.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.568+0000 7f84cfc00640 1 -- 172.21.15.107:0/1468443532 >> 172.21.15.107:0/1468443532 conn(0x7f84c8101210 msgr2=0x7f84c8103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:10.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.568+0000 7f84cfc00640 1 -- 172.21.15.107:0/1468443532 shutdown_connections 2024-07-26T12:43:10.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.568+0000 7f84cfc00640 1 -- 172.21.15.107:0/1468443532 wait complete. 2024-07-26T12:43:10.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.569+0000 7f84cfc00640 1 Processor -- start 2024-07-26T12:43:10.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.569+0000 7f84cfc00640 1 -- start start 2024-07-26T12:43:10.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cfc00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c81057e0 0x7f84c819c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:10.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cfc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84c810b120 0x7f84c819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:10.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cfc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84c810ba70 0x7f84c81a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:10.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cfc00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f84c8115740 con 0x7f84c81057e0 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cfc00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f84c81155c0 con 0x7f84c810b120 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cfc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f84c81158c0 con 0x7f84c810ba70 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cd200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c81057e0 0x7f84c819c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cd200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c81057e0 0x7f84c819c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48334/0 (socket says 172.21.15.107:48334) 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cd200640 1 -- 172.21.15.107:0/3935983902 learned_addr learned my addr 172.21.15.107:0/3935983902 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84c7e00640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84c810b120 0x7f84c819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cdc00640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84c810ba70 0x7f84c81a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cd200640 1 -- 172.21.15.107:0/3935983902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84c810ba70 msgr2=0x7f84c81a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cd200640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84c810ba70 0x7f84c81a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cd200640 1 -- 172.21.15.107:0/3935983902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84c810b120 msgr2=0x7f84c819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cd200640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84c810b120 0x7f84c819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:10.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.570+0000 7f84cd200640 1 -- 172.21.15.107:0/3935983902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c81a20a0 con 0x7f84c81057e0 2024-07-26T12:43:10.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.571+0000 7f84cd200640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c81057e0 0x7f84c819c130 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f84bc005940 tx=0x7f84bc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:10.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.571+0000 7f84c5600640 1 -- 172.21.15.107:0/3935983902 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc03f030 con 0x7f84c81057e0 2024-07-26T12:43:10.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.571+0000 7f84cfc00640 1 -- 172.21.15.107:0/3935983902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c81a2330 con 0x7f84c81057e0 2024-07-26T12:43:10.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.571+0000 7f84c5600640 1 -- 172.21.15.107:0/3935983902 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84bc0044c0 con 0x7f84c81057e0 2024-07-26T12:43:10.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.571+0000 7f84c5600640 1 -- 172.21.15.107:0/3935983902 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc035070 con 0x7f84c81057e0 2024-07-26T12:43:10.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.571+0000 7f84cfc00640 1 -- 172.21.15.107:0/3935983902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c81a8af0 con 0x7f84c81057e0 2024-07-26T12:43:10.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.573+0000 7f84cfc00640 1 -- 172.21.15.107:0/3935983902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c8105bf0 con 0x7f84c81057e0 2024-07-26T12:43:10.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.575+0000 7f84c5600640 1 -- 172.21.15.107:0/3935983902 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f84bc042030 con 0x7f84c81057e0 2024-07-26T12:43:10.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.577+0000 7f84c5600640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84b00b3c20 0x7f84b00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:10.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.577+0000 7f84c5600640 1 -- 172.21.15.107:0/3935983902 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7712+0+0 (secure 0 0 0) 0x7f84bc103f40 con 0x7f84c81057e0 2024-07-26T12:43:10.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.577+0000 7f84c7e00640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84b00b3c20 0x7f84b00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:10.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.578+0000 7f84c7e00640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84b00b3c20 0x7f84b00b6130 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f84b40095f0 tx=0x7f84b4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:10.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.580+0000 7f84c5600640 1 -- 172.21.15.107:0/3935983902 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f84bc0cb1a0 con 0x7f84c81057e0 2024-07-26T12:43:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:10.781+0000 7f84cfc00640 1 -- 172.21.15.107:0/3935983902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f84c819d260 con 0x7f84c81057e0 2024-07-26T12:43:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.141+0000 7f84c5600640 1 -- 172.21.15.107:0/3935983902 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v76) ==== 171+0+0 (secure 0 0 0) 0x7f84bc0cf6f0 con 0x7f84c81057e0 2024-07-26T12:43:11.144 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data' created 2024-07-26T12:43:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.147+0000 7f84cfc00640 1 -- 172.21.15.107:0/3935983902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84b00b3c20 msgr2=0x7f84b00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.147+0000 7f84cfc00640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84b00b3c20 0x7f84b00b6130 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f84b40095f0 tx=0x7f84b4005f90 comp rx=0 tx=0).stop 2024-07-26T12:43:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.147+0000 7f84cfc00640 1 -- 172.21.15.107:0/3935983902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c81057e0 msgr2=0x7f84c819c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:11.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.147+0000 7f84cfc00640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c81057e0 0x7f84c819c130 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f84bc005940 tx=0x7f84bc0040c0 comp rx=0 tx=0).stop 2024-07-26T12:43:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.148+0000 7f84cfc00640 1 -- 172.21.15.107:0/3935983902 shutdown_connections 2024-07-26T12:43:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.148+0000 7f84cfc00640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84b00b3c20 0x7f84b00b6130 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.148+0000 7f84cfc00640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84c810ba70 0x7f84c81a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.148+0000 7f84cfc00640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84c810b120 0x7f84c819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.148+0000 7f84cfc00640 1 --2- 172.21.15.107:0/3935983902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84c81057e0 0x7f84c819c130 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.148+0000 7f84cfc00640 1 -- 172.21.15.107:0/3935983902 >> 172.21.15.107:0/3935983902 conn(0x7f84c8101210 msgr2=0x7f84c8103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.149+0000 7f84cfc00640 1 -- 172.21.15.107:0/3935983902 shutdown_connections 2024-07-26T12:43:11.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:11.149+0000 7f84cfc00640 1 -- 172.21.15.107:0/3935983902 wait complete. 2024-07-26T12:43:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:11 smithi112 ceph-mon[37501]: pgmap v291: 33 pgs: 15 creating+peering, 9 active+clean, 9 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:11 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3935983902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T12:43:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:11 smithi107 ceph-mon[32659]: pgmap v291: 33 pgs: 15 creating+peering, 9 active+clean, 9 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:11 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3935983902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T12:43:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:11 smithi132 ceph-mon[37580]: pgmap v291: 33 pgs: 15 creating+peering, 9 active+clean, 9 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:11 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3935983902' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T12:43:12.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T12:43:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:12 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3935983902' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T12:43:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:12 smithi107 ceph-mon[32659]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T12:43:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:12 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3935983902' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T12:43:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:12 smithi112 ceph-mon[37501]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T12:43:12.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:12 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3935983902' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T12:43:12.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:12 smithi132 ceph-mon[37580]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T12:43:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:13 smithi112 ceph-mon[37501]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T12:43:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:13 smithi112 ceph-mon[37501]: pgmap v294: 97 pgs: 28 creating+peering, 46 active+clean, 23 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:13 smithi107 ceph-mon[32659]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T12:43:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:13 smithi107 ceph-mon[32659]: pgmap v294: 97 pgs: 28 creating+peering, 46 active+clean, 23 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:13 smithi132 ceph-mon[37580]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T12:43:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:13 smithi132 ceph-mon[37580]: pgmap v294: 97 pgs: 28 creating+peering, 46 active+clean, 23 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:14.419 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:14 smithi107 ceph-mon[32659]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T12:43:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:14 smithi112 ceph-mon[37501]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T12:43:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:14 smithi132 ceph-mon[37580]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T12:43:14.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.938+0000 7fad3b200640 1 -- 172.21.15.107:0/3336360657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34109850 msgr2=0x7fad34110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:14.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.938+0000 7fad3b200640 1 --2- 172.21.15.107:0/3336360657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34109850 0x7fad34110080 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009900 tx=0x7fad2c02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:43:14.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.938+0000 7fad3b200640 1 -- 172.21.15.107:0/3336360657 shutdown_connections 2024-07-26T12:43:14.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.938+0000 7fad3b200640 1 --2- 172.21.15.107:0/3336360657 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34109850 0x7fad34110080 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:14.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.938+0000 7fad3b200640 1 --2- 172.21.15.107:0/3336360657 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34108f00 0x7fad34109310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:14.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.938+0000 7fad3b200640 1 --2- 172.21.15.107:0/3336360657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad341035c0 0x7fad341039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:14.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.938+0000 7fad3b200640 1 -- 172.21.15.107:0/3336360657 >> 172.21.15.107:0/3336360657 conn(0x7fad340ff030 msgr2=0x7fad34101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:14.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.938+0000 7fad3b200640 1 -- 172.21.15.107:0/3336360657 shutdown_connections 2024-07-26T12:43:14.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.938+0000 7fad3b200640 1 -- 172.21.15.107:0/3336360657 wait complete. 2024-07-26T12:43:14.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.939+0000 7fad3b200640 1 Processor -- start 2024-07-26T12:43:14.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.939+0000 7fad3b200640 1 -- start start 2024-07-26T12:43:14.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad3b200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad341035c0 0x7fad3419c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:14.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad3b200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34108f00 0x7fad3419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:14.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad3b200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34109850 0x7fad341a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:14.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad3b200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad34113570 con 0x7fad34109850 2024-07-26T12:43:14.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad3b200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fad341133f0 con 0x7fad34108f00 2024-07-26T12:43:14.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad3b200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fad341136f0 con 0x7fad341035c0 2024-07-26T12:43:14.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad33e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad341035c0 0x7fad3419c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad33e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad341035c0 0x7fad3419c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:40096/0 (socket says 172.21.15.107:40096) 2024-07-26T12:43:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad33e00640 1 -- 172.21.15.107:0/4084686747 learned_addr learned my addr 172.21.15.107:0/4084686747 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad33400640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34108f00 0x7fad3419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:14.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.940+0000 7fad39200640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34109850 0x7fad341a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.941+0000 7fad33e00640 1 -- 172.21.15.107:0/4084686747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34108f00 msgr2=0x7fad3419c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.941+0000 7fad33e00640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34108f00 0x7fad3419c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.941+0000 7fad33e00640 1 -- 172.21.15.107:0/4084686747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34109850 msgr2=0x7fad341a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.941+0000 7fad33e00640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34109850 0x7fad341a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.941+0000 7fad33e00640 1 -- 172.21.15.107:0/4084686747 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad341a20d0 con 0x7fad341035c0 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.941+0000 7fad33e00640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad341035c0 0x7fad3419c160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fad2400eb90 tx=0x7fad2400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.941+0000 7fad30c00640 1 -- 172.21.15.107:0/4084686747 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24004d20 con 0x7fad341035c0 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.941+0000 7fad3b200640 1 -- 172.21.15.107:0/4084686747 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad341a23c0 con 0x7fad341035c0 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.942+0000 7fad3b200640 1 -- 172.21.15.107:0/4084686747 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad341a8b20 con 0x7fad341035c0 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.942+0000 7fad30c00640 1 -- 172.21.15.107:0/4084686747 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad2400ee90 con 0x7fad341035c0 2024-07-26T12:43:14.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.942+0000 7fad30c00640 1 -- 172.21.15.107:0/4084686747 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24010690 con 0x7fad341035c0 2024-07-26T12:43:14.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.943+0000 7fad3b200640 1 -- 172.21.15.107:0/4084686747 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad341039d0 con 0x7fad341035c0 2024-07-26T12:43:14.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.945+0000 7fad30c00640 1 -- 172.21.15.107:0/4084686747 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fad24010830 con 0x7fad341035c0 2024-07-26T12:43:14.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.947+0000 7fad30c00640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad180b3c20 0x7fad180b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:14.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.947+0000 7fad30c00640 1 -- 172.21.15.107:0/4084686747 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8094+0+0 (secure 0 0 0) 0x7fad240e43d0 con 0x7fad341035c0 2024-07-26T12:43:14.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.947+0000 7fad33400640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad180b3c20 0x7fad180b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:14.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.948+0000 7fad33400640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad180b3c20 0x7fad180b6130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fad200095f0 tx=0x7fad20005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:14.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:14.948+0000 7fad30c00640 1 -- 172.21.15.107:0/4084686747 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fad24055600 con 0x7fad341035c0 2024-07-26T12:43:15.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:15.195+0000 7fad3b200640 1 -- 172.21.15.107:0/4084686747 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fad341a8e80 con 0x7fad341035c0 2024-07-26T12:43:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:15 smithi112 ceph-mon[37501]: pgmap v296: 97 pgs: 25 creating+peering, 72 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:15 smithi107 ceph-mon[32659]: pgmap v296: 97 pgs: 25 creating+peering, 72 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:15 smithi132 ceph-mon[37580]: pgmap v296: 97 pgs: 25 creating+peering, 72 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:16.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.197+0000 7fad30c00640 1 -- 172.21.15.107:0/4084686747 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fad240aba60 con 0x7fad341035c0 2024-07-26T12:43:16.200 INFO:teuthology.orchestra.run.smithi107.stderr:new fs with metadata pool 2 and data pool 3 2024-07-26T12:43:16.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.202+0000 7fad3b200640 1 -- 172.21.15.107:0/4084686747 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad180b3c20 msgr2=0x7fad180b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:16.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.202+0000 7fad3b200640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad180b3c20 0x7fad180b6130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fad200095f0 tx=0x7fad20005f90 comp rx=0 tx=0).stop 2024-07-26T12:43:16.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.202+0000 7fad3b200640 1 -- 172.21.15.107:0/4084686747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad341035c0 msgr2=0x7fad3419c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:16.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.202+0000 7fad3b200640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad341035c0 0x7fad3419c160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fad2400eb90 tx=0x7fad2400c6a0 comp rx=0 tx=0).stop 2024-07-26T12:43:16.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.203+0000 7fad3b200640 1 -- 172.21.15.107:0/4084686747 shutdown_connections 2024-07-26T12:43:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.203+0000 7fad3b200640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fad180b3c20 0x7fad180b6130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.203+0000 7fad3b200640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad34109850 0x7fad341a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.203+0000 7fad3b200640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fad34108f00 0x7fad3419c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.203+0000 7fad3b200640 1 --2- 172.21.15.107:0/4084686747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad341035c0 0x7fad3419c160 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.203+0000 7fad3b200640 1 -- 172.21.15.107:0/4084686747 >> 172.21.15.107:0/4084686747 conn(0x7fad340ff030 msgr2=0x7fad34106490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.204+0000 7fad3b200640 1 -- 172.21.15.107:0/4084686747 shutdown_connections 2024-07-26T12:43:16.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:16.204+0000 7fad3b200640 1 -- 172.21.15.107:0/4084686747 wait complete. 2024-07-26T12:43:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:16 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/4084686747' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T12:43:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:16 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T12:43:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:16 smithi112 ceph-mon[37501]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:43:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:16 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/4084686747' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T12:43:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:16 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T12:43:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:16 smithi107 ceph-mon[32659]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:43:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:16 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/4084686747' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T12:43:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:16 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T12:43:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:16 smithi132 ceph-mon[37580]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:43:17.139 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-07-26T12:43:17.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-07-26T12:43:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:17 smithi112 ceph-mon[37501]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T12:43:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:17 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T12:43:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:17 smithi112 ceph-mon[37501]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T12:43:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:17 smithi112 ceph-mon[37501]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:43:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:17 smithi112 ceph-mon[37501]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:43:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:17 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] up:active 2024-07-26T12:43:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:17 smithi112 ceph-mon[37501]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:43:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:17 smithi112 ceph-mon[37501]: pgmap v298: 97 pgs: 8 creating+peering, 89 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 186 B/s wr, 0 op/s 2024-07-26T12:43:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:17 smithi107 ceph-mon[32659]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T12:43:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:17 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T12:43:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:17 smithi107 ceph-mon[32659]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T12:43:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:17 smithi107 ceph-mon[32659]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:43:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:17 smithi107 ceph-mon[32659]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:43:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:17 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] up:active 2024-07-26T12:43:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:17 smithi107 ceph-mon[32659]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:43:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:17 smithi107 ceph-mon[32659]: pgmap v298: 97 pgs: 8 creating+peering, 89 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 186 B/s wr, 0 op/s 2024-07-26T12:43:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:17 smithi132 ceph-mon[37580]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T12:43:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:17 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T12:43:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:17 smithi132 ceph-mon[37580]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-07-26T12:43:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:17 smithi132 ceph-mon[37580]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:43:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:17 smithi132 ceph-mon[37580]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:43:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:17 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] up:active 2024-07-26T12:43:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:17 smithi132 ceph-mon[37580]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:43:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:17 smithi132 ceph-mon[37580]: pgmap v298: 97 pgs: 8 creating+peering, 89 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 186 B/s wr, 0 op/s 2024-07-26T12:43:17.596 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:18.185 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:43:17 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2024-07-26T12:43:17.713+0000 7f3578b59640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T12:43:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:18 smithi112 ceph-mon[37501]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T12:43:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:18 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:43:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:18 smithi107 ceph-mon[32659]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T12:43:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:18 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:43:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:18 smithi132 ceph-mon[37580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T12:43:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:18 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:43:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:19 smithi112 ceph-mon[37501]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 788 B/s wr, 5 op/s 2024-07-26T12:43:19.534 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:19 smithi107 ceph-mon[32659]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 788 B/s wr, 5 op/s 2024-07-26T12:43:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:19 smithi132 ceph-mon[37580]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 788 B/s wr, 5 op/s 2024-07-26T12:43:20.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.065+0000 7f5bbaa00640 1 -- 172.21.15.107:0/3443594733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb410af10 msgr2=0x7f5bb410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:20.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.065+0000 7f5bbaa00640 1 --2- 172.21.15.107:0/3443594733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb410af10 0x7f5bb410b320 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0009900 tx=0x7f5ba002a410 comp rx=0 tx=0).stop 2024-07-26T12:43:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.065+0000 7f5bbaa00640 1 -- 172.21.15.107:0/3443594733 shutdown_connections 2024-07-26T12:43:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.065+0000 7f5bbaa00640 1 --2- 172.21.15.107:0/3443594733 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb410b860 0x7f5bb4112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.065+0000 7f5bbaa00640 1 --2- 172.21.15.107:0/3443594733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb410af10 0x7f5bb410b320 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.065+0000 7f5bbaa00640 1 --2- 172.21.15.107:0/3443594733 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb41055d0 0x7f5bb41059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.065+0000 7f5bbaa00640 1 -- 172.21.15.107:0/3443594733 >> 172.21.15.107:0/3443594733 conn(0x7f5bb4101040 msgr2=0x7f5bb4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.065+0000 7f5bbaa00640 1 -- 172.21.15.107:0/3443594733 shutdown_connections 2024-07-26T12:43:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.065+0000 7f5bbaa00640 1 -- 172.21.15.107:0/3443594733 wait complete. 2024-07-26T12:43:20.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.067+0000 7f5bbaa00640 1 Processor -- start 2024-07-26T12:43:20.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.067+0000 7f5bbaa00640 1 -- start start 2024-07-26T12:43:20.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.067+0000 7f5bbaa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb41055d0 0x7f5bb419bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.067+0000 7f5bbaa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb410af10 0x7f5bb419c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.067+0000 7f5bbaa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb410b860 0x7f5bb41a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.067+0000 7f5bbaa00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb4115540 con 0x7f5bb410b860 2024-07-26T12:43:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.067+0000 7f5bbaa00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb41153c0 con 0x7f5bb41055d0 2024-07-26T12:43:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.067+0000 7f5bbaa00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb41156c0 con 0x7f5bb410af10 2024-07-26T12:43:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bb9600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb41055d0 0x7f5bb419bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bba000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb410b860 0x7f5bb41a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bb9600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb41055d0 0x7f5bb419bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:51878/0 (socket says 172.21.15.107:51878) 2024-07-26T12:43:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bb9600640 1 -- 172.21.15.107:0/2736537618 learned_addr learned my addr 172.21.15.107:0/2736537618 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bb8c00640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb410af10 0x7f5bb419c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bb9600640 1 -- 172.21.15.107:0/2736537618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb410af10 msgr2=0x7f5bb419c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:20.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bb9600640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb410af10 0x7f5bb419c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bb9600640 1 -- 172.21.15.107:0/2736537618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb410b860 msgr2=0x7f5bb41a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bb9600640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb410b860 0x7f5bb41a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bb9600640 1 -- 172.21.15.107:0/2736537618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb41a1ea0 con 0x7f5bb41055d0 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bba000640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb410b860 0x7f5bb41a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.068+0000 7f5bb9600640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb41055d0 0x7f5bb419bf30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800e9f0 tx=0x7f5ba800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.069+0000 7f5bb2000640 1 -- 172.21.15.107:0/2736537618 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba800cd80 con 0x7f5bb41055d0 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.069+0000 7f5bbaa00640 1 -- 172.21.15.107:0/2736537618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb41a2190 con 0x7f5bb41055d0 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.069+0000 7f5bb2000640 1 -- 172.21.15.107:0/2736537618 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ba80108e0 con 0x7f5bb41055d0 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.069+0000 7f5bb2000640 1 -- 172.21.15.107:0/2736537618 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8018980 con 0x7f5bb41055d0 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.069+0000 7f5bbaa00640 1 -- 172.21.15.107:0/2736537618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb41a88f0 con 0x7f5bb41055d0 2024-07-26T12:43:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.070+0000 7f5bbaa00640 1 -- 172.21.15.107:0/2736537618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb41059e0 con 0x7f5bb41055d0 2024-07-26T12:43:20.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.073+0000 7f5bb2000640 1 -- 172.21.15.107:0/2736537618 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f5ba8010430 con 0x7f5bb41055d0 2024-07-26T12:43:20.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.074+0000 7f5bb2000640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5b9c0b3c20 0x7f5b9c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:20.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.074+0000 7f5bb2000640 1 -- 172.21.15.107:0/2736537618 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8194+0+0 (secure 0 0 0) 0x7f5ba80e4470 con 0x7f5bb41055d0 2024-07-26T12:43:20.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.074+0000 7f5bb8c00640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5b9c0b3c20 0x7f5b9c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:20.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.075+0000 7f5bb8c00640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5b9c0b3c20 0x7f5b9c0b6130 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ba002aa00 tx=0x7f5ba0005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:20.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.075+0000 7f5bb2000640 1 -- 172.21.15.107:0/2736537618 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5ba80ab430 con 0x7f5bb41055d0 2024-07-26T12:43:20.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.288+0000 7f5bbaa00640 1 -- 172.21.15.107:0/2736537618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f5bb419d060 con 0x7f5bb41055d0 2024-07-26T12:43:20.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.348+0000 7f5bb2000640 1 -- 172.21.15.107:0/2736537618 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v80) ==== 157+0+0 (secure 0 0 0) 0x7f5ba80af980 con 0x7f5bb41055d0 2024-07-26T12:43:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 -- 172.21.15.107:0/2736537618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5b9c0b3c20 msgr2=0x7f5b9c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5b9c0b3c20 0x7f5b9c0b6130 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5ba002aa00 tx=0x7f5ba0005a20 comp rx=0 tx=0).stop 2024-07-26T12:43:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 -- 172.21.15.107:0/2736537618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb41055d0 msgr2=0x7f5bb419bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb41055d0 0x7f5bb419bf30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5ba800e9f0 tx=0x7f5ba800eeb0 comp rx=0 tx=0).stop 2024-07-26T12:43:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 -- 172.21.15.107:0/2736537618 shutdown_connections 2024-07-26T12:43:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5b9c0b3c20 0x7f5b9c0b6130 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5bb410b860 0x7f5bb41a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5bb410af10 0x7f5bb419c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 --2- 172.21.15.107:0/2736537618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5bb41055d0 0x7f5bb419bf30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 -- 172.21.15.107:0/2736537618 >> 172.21.15.107:0/2736537618 conn(0x7f5bb4101040 msgr2=0x7f5bb4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:20.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 -- 172.21.15.107:0/2736537618 shutdown_connections 2024-07-26T12:43:20.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:20.352+0000 7f5bbaa00640 1 -- 172.21.15.107:0/2736537618 wait complete. 2024-07-26T12:43:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:20 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2736537618' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T12:43:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:20 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T12:43:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:20 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2736537618' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T12:43:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:20 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T12:43:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:20 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2736537618' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T12:43:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:20 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T12:43:21.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T12:43:21.604 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:21 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T12:43:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:21 smithi112 ceph-mon[37501]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T12:43:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:21 smithi112 ceph-mon[37501]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-07-26T12:43:21.751 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:21 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T12:43:21.751 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:21 smithi107 ceph-mon[32659]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T12:43:21.751 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:21 smithi107 ceph-mon[32659]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-07-26T12:43:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:21 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T12:43:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:21 smithi132 ceph-mon[37580]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T12:43:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:21 smithi132 ceph-mon[37580]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 5 op/s 2024-07-26T12:43:23.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.982+0000 7ff562e00640 1 -- 172.21.15.107:0/1393353779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55c113c90 msgr2=0x7ff55c116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:23.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.982+0000 7ff562e00640 1 --2- 172.21.15.107:0/1393353779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55c113c90 0x7ff55c116160 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff554009ca0 tx=0x7ff55402a8c0 comp rx=0 tx=0).stop 2024-07-26T12:43:23.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.982+0000 7ff562e00640 1 -- 172.21.15.107:0/1393353779 shutdown_connections 2024-07-26T12:43:23.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.982+0000 7ff562e00640 1 --2- 172.21.15.107:0/1393353779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55c113c90 0x7ff55c116160 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:23.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.982+0000 7ff562e00640 1 --2- 172.21.15.107:0/1393353779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff55c075800 0x7ff55c075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:23.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.982+0000 7ff562e00640 1 --2- 172.21.15.107:0/1393353779 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff55c076db0 0x7ff55c0751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:23.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.982+0000 7ff562e00640 1 -- 172.21.15.107:0/1393353779 >> 172.21.15.107:0/1393353779 conn(0x7ff55c0fee10 msgr2=0x7ff55c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:23.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.982+0000 7ff562e00640 1 -- 172.21.15.107:0/1393353779 shutdown_connections 2024-07-26T12:43:23.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.982+0000 7ff562e00640 1 -- 172.21.15.107:0/1393353779 wait complete. 2024-07-26T12:43:23.985 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:23 smithi107 ceph-mon[32659]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T12:43:23.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.983+0000 7ff562e00640 1 Processor -- start 2024-07-26T12:43:23.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff562e00640 1 -- start start 2024-07-26T12:43:23.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff562e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55c075800 0x7ff55c1a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff562e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff55c076db0 0x7ff55c1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff562e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff55c113c90 0x7ff55c1a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff562e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1198d0 con 0x7ff55c075800 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff562e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c119750 con 0x7ff55c076db0 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff562e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c119a50 con 0x7ff55c113c90 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff561a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55c075800 0x7ff55c1a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff562400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff55c113c90 0x7ff55c1a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff561a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55c075800 0x7ff55c1a02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37622/0 (socket says 172.21.15.107:37622) 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff561a00640 1 -- 172.21.15.107:0/2914901543 learned_addr learned my addr 172.21.15.107:0/2914901543 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.984+0000 7ff561000640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff55c076db0 0x7ff55c1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.985+0000 7ff561a00640 1 -- 172.21.15.107:0/2914901543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff55c113c90 msgr2=0x7ff55c1a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.985+0000 7ff561a00640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff55c113c90 0x7ff55c1a5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.985+0000 7ff561a00640 1 -- 172.21.15.107:0/2914901543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff55c076db0 msgr2=0x7ff55c1a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.985+0000 7ff561a00640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff55c076db0 0x7ff55c1a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.985+0000 7ff561a00640 1 -- 172.21.15.107:0/2914901543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55c1a6210 con 0x7ff55c075800 2024-07-26T12:43:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.985+0000 7ff561a00640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55c075800 0x7ff55c1a02a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff55000e9f0 tx=0x7ff55000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.986+0000 7ff55a000640 1 -- 172.21.15.107:0/2914901543 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55000cd80 con 0x7ff55c075800 2024-07-26T12:43:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.986+0000 7ff562e00640 1 -- 172.21.15.107:0/2914901543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55c1a6500 con 0x7ff55c075800 2024-07-26T12:43:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.986+0000 7ff562e00640 1 -- 172.21.15.107:0/2914901543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff55c1acc60 con 0x7ff55c075800 2024-07-26T12:43:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.986+0000 7ff55a000640 1 -- 172.21.15.107:0/2914901543 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5500108e0 con 0x7ff55c075800 2024-07-26T12:43:23.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.986+0000 7ff55a000640 1 -- 172.21.15.107:0/2914901543 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff550018980 con 0x7ff55c075800 2024-07-26T12:43:23.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.988+0000 7ff562e00640 1 -- 172.21.15.107:0/2914901543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff53c0052c0 con 0x7ff55c075800 2024-07-26T12:43:23.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.990+0000 7ff55a000640 1 -- 172.21.15.107:0/2914901543 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7ff550010430 con 0x7ff55c075800 2024-07-26T12:43:23.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.991+0000 7ff55a000640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff5440b3c10 0x7ff5440b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:23.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.992+0000 7ff55a000640 1 -- 172.21.15.107:0/2914901543 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8492+0+0 (secure 0 0 0) 0x7ff5500e3d60 con 0x7ff55c075800 2024-07-26T12:43:23.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.992+0000 7ff561000640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff5440b3c10 0x7ff5440b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:23.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.992+0000 7ff561000640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff5440b3c10 0x7ff5440b6120 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff55c115b20 tx=0x7ff54800b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:23.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:23.996+0000 7ff55a000640 1 -- 172.21.15.107:0/2914901543 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff5500aac00 con 0x7ff55c075800 2024-07-26T12:43:24.053 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:23 smithi112 ceph-mon[37501]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T12:43:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:23 smithi132 ceph-mon[37580]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T12:43:24.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:24.196+0000 7ff562e00640 1 -- 172.21.15.107:0/2914901543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff53c0055b0 con 0x7ff55c075800 2024-07-26T12:43:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:24 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2914901543' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T12:43:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:24 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2914901543' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T12:43:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:24 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2914901543' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T12:43:25.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.736+0000 7ff55a000640 1 -- 172.21.15.107:0/2914901543 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v82) ==== 228+0+0 (secure 0 0 0) 0x7ff5500af150 con 0x7ff55c075800 2024-07-26T12:43:25.737 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'cephfs_data_ec' created 2024-07-26T12:43:25.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.740+0000 7ff562e00640 1 -- 172.21.15.107:0/2914901543 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff5440b3c10 msgr2=0x7ff5440b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:25.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.740+0000 7ff562e00640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff5440b3c10 0x7ff5440b6120 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff55c115b20 tx=0x7ff54800b410 comp rx=0 tx=0).stop 2024-07-26T12:43:25.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.740+0000 7ff562e00640 1 -- 172.21.15.107:0/2914901543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55c075800 msgr2=0x7ff55c1a02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.740+0000 7ff562e00640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55c075800 0x7ff55c1a02a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff55000e9f0 tx=0x7ff55000eeb0 comp rx=0 tx=0).stop 2024-07-26T12:43:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.741+0000 7ff562e00640 1 -- 172.21.15.107:0/2914901543 shutdown_connections 2024-07-26T12:43:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.741+0000 7ff562e00640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff5440b3c10 0x7ff5440b6120 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:25.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.741+0000 7ff562e00640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff55c113c90 0x7ff55c1a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.741+0000 7ff562e00640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff55c076db0 0x7ff55c1a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.741+0000 7ff562e00640 1 --2- 172.21.15.107:0/2914901543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff55c075800 0x7ff55c1a02a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.741+0000 7ff562e00640 1 -- 172.21.15.107:0/2914901543 >> 172.21.15.107:0/2914901543 conn(0x7ff55c0fee10 msgr2=0x7ff55c10fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.741+0000 7ff562e00640 1 -- 172.21.15.107:0/2914901543 shutdown_connections 2024-07-26T12:43:25.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:25.741+0000 7ff562e00640 1 -- 172.21.15.107:0/2914901543 wait complete. 2024-07-26T12:43:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:25 smithi107 ceph-mon[32659]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T12:43:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:25 smithi107 ceph-mon[32659]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T12:43:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:25 smithi132 ceph-mon[37580]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T12:43:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:25 smithi132 ceph-mon[37580]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T12:43:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:25 smithi112 ceph-mon[37501]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T12:43:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:25 smithi112 ceph-mon[37501]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T12:43:26.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-07-26T12:43:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:26 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2914901543' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T12:43:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:26 smithi107 ceph-mon[32659]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T12:43:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:26 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2914901543' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T12:43:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:26 smithi132 ceph-mon[37580]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T12:43:27.135 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:26 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2914901543' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T12:43:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:26 smithi112 ceph-mon[37501]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T12:43:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:27 smithi107 ceph-mon[32659]: pgmap v306: 129 pgs: 9 creating+peering, 23 unknown, 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-07-26T12:43:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:27 smithi107 ceph-mon[32659]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:43:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:27 smithi107 ceph-mon[32659]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T12:43:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:27 smithi132 ceph-mon[37580]: pgmap v306: 129 pgs: 9 creating+peering, 23 unknown, 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-07-26T12:43:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:27 smithi132 ceph-mon[37580]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:43:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:27 smithi132 ceph-mon[37580]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T12:43:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:27 smithi112 ceph-mon[37501]: pgmap v306: 129 pgs: 9 creating+peering, 23 unknown, 97 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-07-26T12:43:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:27 smithi112 ceph-mon[37501]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T12:43:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:27 smithi112 ceph-mon[37501]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T12:43:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:28 smithi107 ceph-mon[32659]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T12:43:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:28 smithi132 ceph-mon[37580]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T12:43:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:28 smithi112 ceph-mon[37501]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T12:43:29.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.329+0000 7f5fade00640 1 -- 172.21.15.107:0/1810273833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fa810a910 msgr2=0x7f5fa810ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.329+0000 7f5fade00640 1 --2- 172.21.15.107:0/1810273833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fa810a910 0x7f5fa810ad20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009ca0 tx=0x7f5fa002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.329+0000 7f5fade00640 1 -- 172.21.15.107:0/1810273833 shutdown_connections 2024-07-26T12:43:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.329+0000 7f5fade00640 1 --2- 172.21.15.107:0/1810273833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa810e500 0x7f5fa8116f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.329+0000 7f5fade00640 1 --2- 172.21.15.107:0/1810273833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fa810dbb0 0x7f5fa810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.329+0000 7f5fade00640 1 --2- 172.21.15.107:0/1810273833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fa810a910 0x7f5fa810ad20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.329+0000 7f5fade00640 1 -- 172.21.15.107:0/1810273833 >> 172.21.15.107:0/1810273833 conn(0x7f5fa8070d20 msgr2=0x7f5fa8073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.329+0000 7f5fade00640 1 -- 172.21.15.107:0/1810273833 shutdown_connections 2024-07-26T12:43:29.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.329+0000 7f5fade00640 1 -- 172.21.15.107:0/1810273833 wait complete. 2024-07-26T12:43:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.330+0000 7f5fade00640 1 Processor -- start 2024-07-26T12:43:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.330+0000 7f5fade00640 1 -- start start 2024-07-26T12:43:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fade00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fa810a910 0x7f5fa81187f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fade00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa810dbb0 0x7f5fa8118d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fade00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fa810e500 0x7f5fa8119270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fade00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa811bd00 con 0x7f5fa810a910 2024-07-26T12:43:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fade00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa811bb80 con 0x7f5fa810dbb0 2024-07-26T12:43:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fade00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa811be80 con 0x7f5fa810e500 2024-07-26T12:43:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fa6a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fa810a910 0x7f5fa81187f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:29.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fa6a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fa810a910 0x7f5fa81187f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37630/0 (socket says 172.21.15.107:37630) 2024-07-26T12:43:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fa7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fa810e500 0x7f5fa8119270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fa6a00640 1 -- 172.21.15.107:0/2720278237 learned_addr learned my addr 172.21.15.107:0/2720278237 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.331+0000 7f5fa6000640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa810dbb0 0x7f5fa8118d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.332+0000 7f5fa6000640 1 -- 172.21.15.107:0/2720278237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fa810e500 msgr2=0x7f5fa8119270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.332+0000 7f5fa6000640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fa810e500 0x7f5fa8119270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.332+0000 7f5fa6000640 1 -- 172.21.15.107:0/2720278237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fa810a910 msgr2=0x7f5fa81187f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.332+0000 7f5fa6000640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fa810a910 0x7f5fa81187f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.332+0000 7f5fa6000640 1 -- 172.21.15.107:0/2720278237 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa81b0970 con 0x7f5fa810dbb0 2024-07-26T12:43:29.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.332+0000 7f5fa7400640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fa810e500 0x7f5fa8119270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:29.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.332+0000 7f5fa6000640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa810dbb0 0x7f5fa8118d30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800e9c0 tx=0x7f5f9800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:29.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.333+0000 7f5faca00640 1 -- 172.21.15.107:0/2720278237 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9800cd80 con 0x7f5fa810dbb0 2024-07-26T12:43:29.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.333+0000 7f5faca00640 1 -- 172.21.15.107:0/2720278237 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f980035f0 con 0x7f5fa810dbb0 2024-07-26T12:43:29.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.333+0000 7f5fade00640 1 -- 172.21.15.107:0/2720278237 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa81b0c60 con 0x7f5fa810dbb0 2024-07-26T12:43:29.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.333+0000 7f5faca00640 1 -- 172.21.15.107:0/2720278237 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98010690 con 0x7f5fa810dbb0 2024-07-26T12:43:29.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.333+0000 7f5fade00640 1 -- 172.21.15.107:0/2720278237 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa81b11a0 con 0x7f5fa810dbb0 2024-07-26T12:43:29.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.334+0000 7f5fade00640 1 -- 172.21.15.107:0/2720278237 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fa810ad20 con 0x7f5fa810dbb0 2024-07-26T12:43:29.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.336+0000 7f5faca00640 1 -- 172.21.15.107:0/2720278237 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f5f980037a0 con 0x7f5fa810dbb0 2024-07-26T12:43:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.338+0000 7f5faca00640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5f8c0b3c20 0x7f5f8c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.338+0000 7f5faca00640 1 -- 172.21.15.107:0/2720278237 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8985+0+0 (secure 0 0 0) 0x7f5f980e4350 con 0x7f5fa810dbb0 2024-07-26T12:43:29.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.338+0000 7f5fa6a00640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5f8c0b3c20 0x7f5f8c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:29.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.339+0000 7f5fa6a00640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5f8c0b3c20 0x7f5f8c0b6130 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00094e0 tx=0x7f5fa00093f0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:29.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.339+0000 7f5faca00640 1 -- 172.21.15.107:0/2720278237 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5f980ab6e0 con 0x7f5fa810dbb0 2024-07-26T12:43:29.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.537+0000 7f5fade00640 1 -- 172.21.15.107:0/2720278237 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f5fa8119af0 con 0x7f5fa810dbb0 2024-07-26T12:43:29.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.774+0000 7f5faca00640 1 -- 172.21.15.107:0/2720278237 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) ==== 169+0+0 (secure 0 0 0) 0x7f5fa8119af0 con 0x7f5fa810dbb0 2024-07-26T12:43:29.776 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 4 allow_ec_overwrites to true 2024-07-26T12:43:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.778+0000 7f5fade00640 1 -- 172.21.15.107:0/2720278237 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5f8c0b3c20 msgr2=0x7f5f8c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.778+0000 7f5fade00640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5f8c0b3c20 0x7f5f8c0b6130 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00094e0 tx=0x7f5fa00093f0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.778+0000 7f5fade00640 1 -- 172.21.15.107:0/2720278237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa810dbb0 msgr2=0x7f5fa8118d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:29.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.778+0000 7f5fade00640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa810dbb0 0x7f5fa8118d30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800e9c0 tx=0x7f5f9800ee80 comp rx=0 tx=0).stop 2024-07-26T12:43:29.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.778+0000 7f5fade00640 1 -- 172.21.15.107:0/2720278237 shutdown_connections 2024-07-26T12:43:29.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.778+0000 7f5fade00640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5f8c0b3c20 0x7f5f8c0b6130 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.778+0000 7f5fade00640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fa810e500 0x7f5fa8119270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.778+0000 7f5fade00640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fa810dbb0 0x7f5fa8118d30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.778+0000 7f5fade00640 1 --2- 172.21.15.107:0/2720278237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fa810a910 0x7f5fa81187f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:29.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.778+0000 7f5fade00640 1 -- 172.21.15.107:0/2720278237 >> 172.21.15.107:0/2720278237 conn(0x7f5fa8070d20 msgr2=0x7f5fa8074320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:29.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.779+0000 7f5fade00640 1 -- 172.21.15.107:0/2720278237 shutdown_connections 2024-07-26T12:43:29.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:29.779+0000 7f5fade00640 1 -- 172.21.15.107:0/2720278237 wait complete. 2024-07-26T12:43:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:29 smithi107 ceph-mon[32659]: pgmap v309: 129 pgs: 7 creating+activating, 10 creating+peering, 8 unknown, 104 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:29 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2720278237' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T12:43:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:29 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T12:43:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:29 smithi132 ceph-mon[37580]: pgmap v309: 129 pgs: 7 creating+activating, 10 creating+peering, 8 unknown, 104 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:29 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2720278237' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T12:43:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:29 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T12:43:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:29 smithi112 ceph-mon[37501]: pgmap v309: 129 pgs: 7 creating+activating, 10 creating+peering, 8 unknown, 104 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:29 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2720278237' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T12:43:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:29 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T12:43:30.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-07-26T12:43:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:30 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T12:43:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:30 smithi107 ceph-mon[32659]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T12:43:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:30 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T12:43:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:30 smithi132 ceph-mon[37580]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T12:43:31.176 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:30 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T12:43:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:30 smithi112 ceph-mon[37501]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T12:43:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:31 smithi107 ceph-mon[32659]: pgmap v311: 129 pgs: 7 creating+activating, 10 creating+peering, 112 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:31 smithi132 ceph-mon[37580]: pgmap v311: 129 pgs: 7 creating+activating, 10 creating+peering, 112 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:31 smithi112 ceph-mon[37501]: pgmap v311: 129 pgs: 7 creating+activating, 10 creating+peering, 112 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:33.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.373+0000 7feff9c00640 1 -- 172.21.15.107:0/2990859059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feff410af10 msgr2=0x7feff410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:33.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.373+0000 7feff9c00640 1 --2- 172.21.15.107:0/2990859059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feff410af10 0x7feff410b320 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009900 tx=0x7fefdc02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:43:33.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.374+0000 7feff9c00640 1 -- 172.21.15.107:0/2990859059 shutdown_connections 2024-07-26T12:43:33.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.374+0000 7feff9c00640 1 --2- 172.21.15.107:0/2990859059 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feff410b860 0x7feff4112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:33.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.374+0000 7feff9c00640 1 --2- 172.21.15.107:0/2990859059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feff410af10 0x7feff410b320 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:33.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.374+0000 7feff9c00640 1 --2- 172.21.15.107:0/2990859059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff41055d0 0x7feff41059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:33.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.374+0000 7feff9c00640 1 -- 172.21.15.107:0/2990859059 >> 172.21.15.107:0/2990859059 conn(0x7feff4101040 msgr2=0x7feff4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:33.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.374+0000 7feff9c00640 1 -- 172.21.15.107:0/2990859059 shutdown_connections 2024-07-26T12:43:33.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.374+0000 7feff9c00640 1 -- 172.21.15.107:0/2990859059 wait complete. 2024-07-26T12:43:33.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.375+0000 7feff9c00640 1 Processor -- start 2024-07-26T12:43:33.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.375+0000 7feff9c00640 1 -- start start 2024-07-26T12:43:33.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.375+0000 7feff9c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feff41055d0 0x7feff419bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:33.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.375+0000 7feff9c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feff410af10 0x7feff419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:33.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.375+0000 7feff9c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff410b860 0x7feff41a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:33.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.375+0000 7feff9c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feff4115520 con 0x7feff410b860 2024-07-26T12:43:33.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.375+0000 7feff9c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feff41153a0 con 0x7feff410af10 2024-07-26T12:43:33.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.375+0000 7feff9c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7feff41156a0 con 0x7feff41055d0 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7fefeb400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feff410af10 0x7feff419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7fefeb400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feff410af10 0x7feff419c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:55504/0 (socket says 172.21.15.107:55504) 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7fefeb400640 1 -- 172.21.15.107:0/2060259708 learned_addr learned my addr 172.21.15.107:0/2060259708 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7feff9200640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff410b860 0x7feff41a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7feff3e00640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feff41055d0 0x7feff419bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7fefeb400640 1 -- 172.21.15.107:0/2060259708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feff41055d0 msgr2=0x7feff419bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7fefeb400640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feff41055d0 0x7feff419bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7fefeb400640 1 -- 172.21.15.107:0/2060259708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff410b860 msgr2=0x7feff41a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7fefeb400640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff410b860 0x7feff41a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7fefeb400640 1 -- 172.21.15.107:0/2060259708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff41a1ec0 con 0x7feff410af10 2024-07-26T12:43:33.379 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.376+0000 7fefeb400640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feff410af10 0x7feff419c4f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fefdc005ab0 tx=0x7fefdc003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:33.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.377+0000 7feff1600640 1 -- 172.21.15.107:0/2060259708 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc03f070 con 0x7feff410af10 2024-07-26T12:43:33.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.377+0000 7feff1600640 1 -- 172.21.15.107:0/2060259708 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefdc031b50 con 0x7feff410af10 2024-07-26T12:43:33.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.377+0000 7feff9c00640 1 -- 172.21.15.107:0/2060259708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff41a2150 con 0x7feff410af10 2024-07-26T12:43:33.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.377+0000 7feff1600640 1 -- 172.21.15.107:0/2060259708 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc035030 con 0x7feff410af10 2024-07-26T12:43:33.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.377+0000 7feff9c00640 1 -- 172.21.15.107:0/2060259708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff41a8910 con 0x7feff410af10 2024-07-26T12:43:33.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.378+0000 7feff9c00640 1 -- 172.21.15.107:0/2060259708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff41059e0 con 0x7feff410af10 2024-07-26T12:43:33.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.380+0000 7feff1600640 1 -- 172.21.15.107:0/2060259708 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fefdc042070 con 0x7feff410af10 2024-07-26T12:43:33.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.382+0000 7feff1600640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fefd80b3c20 0x7fefd80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:33.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.382+0000 7feff1600640 1 -- 172.21.15.107:0/2060259708 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8985+0+0 (secure 0 0 0) 0x7fefdc1045e0 con 0x7feff410af10 2024-07-26T12:43:33.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.382+0000 7feff3e00640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fefd80b3c20 0x7fefd80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:33.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.383+0000 7feff3e00640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fefd80b3c20 0x7fefd80b6130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009bd0 tx=0x7fefe4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:33.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.384+0000 7feff1600640 1 -- 172.21.15.107:0/2060259708 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fefdc0cb8e0 con 0x7feff410af10 2024-07-26T12:43:33.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.623+0000 7feff9c00640 1 -- 172.21.15.107:0/2060259708 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7feff419d080 con 0x7feff410af10 2024-07-26T12:43:33.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.808+0000 7feff1600640 1 -- 172.21.15.107:0/2060259708 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7feff419d080 con 0x7feff410af10 2024-07-26T12:43:33.809 INFO:teuthology.orchestra.run.smithi107.stderr:added data pool 4 to fsmap 2024-07-26T12:43:33.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.812+0000 7feff9c00640 1 -- 172.21.15.107:0/2060259708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fefd80b3c20 msgr2=0x7fefd80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:33.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.812+0000 7feff9c00640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fefd80b3c20 0x7fefd80b6130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009bd0 tx=0x7fefe4005f90 comp rx=0 tx=0).stop 2024-07-26T12:43:33.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.812+0000 7feff9c00640 1 -- 172.21.15.107:0/2060259708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feff410af10 msgr2=0x7feff419c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:33.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.812+0000 7feff9c00640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feff410af10 0x7feff419c4f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fefdc005ab0 tx=0x7fefdc003b90 comp rx=0 tx=0).stop 2024-07-26T12:43:33.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.812+0000 7feff9c00640 1 -- 172.21.15.107:0/2060259708 shutdown_connections 2024-07-26T12:43:33.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.812+0000 7feff9c00640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fefd80b3c20 0x7fefd80b6130 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:33.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.813+0000 7feff9c00640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff410b860 0x7feff41a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:33.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.813+0000 7feff9c00640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feff410af10 0x7feff419c4f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:33.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.813+0000 7feff9c00640 1 --2- 172.21.15.107:0/2060259708 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feff41055d0 0x7feff419bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:33.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.813+0000 7feff9c00640 1 -- 172.21.15.107:0/2060259708 >> 172.21.15.107:0/2060259708 conn(0x7feff4101040 msgr2=0x7feff4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:33.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.813+0000 7feff9c00640 1 -- 172.21.15.107:0/2060259708 shutdown_connections 2024-07-26T12:43:33.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:33.813+0000 7feff9c00640 1 -- 172.21.15.107:0/2060259708 wait complete. 2024-07-26T12:43:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:33 smithi107 ceph-mon[32659]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:33 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2060259708' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T12:43:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:33 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T12:43:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:33 smithi112 ceph-mon[37501]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:33 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2060259708' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T12:43:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:33 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T12:43:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:33 smithi132 ceph-mon[37580]: pgmap v312: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:33 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2060259708' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T12:43:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:33 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T12:43:34.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:43:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]: dispatch 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:34 smithi107 ceph-mon[32659]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:35.156 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]: dispatch 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-07-26T12:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:34 smithi112 ceph-mon[37501]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T12:43:35.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T12:43:35.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:43:35.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-07-26T12:43:35.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-07-26T12:43:35.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]: dispatch 2024-07-26T12:43:35.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-07-26T12:43:35.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-07-26T12:43:35.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-07-26T12:43:35.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-07-26T12:43:35.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-07-26T12:43:35.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]: dispatch 2024-07-26T12:43:35.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-07-26T12:43:35.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]: dispatch 2024-07-26T12:43:35.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-07-26T12:43:35.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-07-26T12:43:35.314 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-07-26T12:43:35.314 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-07-26T12:43:35.314 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.314 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]: dispatch 2024-07-26T12:43:35.314 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-07-26T12:43:35.315 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:34 smithi132 ceph-mon[37580]: pgmap v314: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T12:43:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:43:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-07-26T12:43:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-07-26T12:43:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]': finished 2024-07-26T12:43:36.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-07-26T12:43:36.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-07-26T12:43:36.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-07-26T12:43:36.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-07-26T12:43:36.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-07-26T12:43:36.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]': finished 2024-07-26T12:43:36.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-07-26T12:43:36.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:35 smithi107 ceph-mon[32659]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T12:43:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]': finished 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]': finished 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-07-26T12:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:35 smithi112 ceph-mon[37501]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T12:43:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T12:43:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:43:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-07-26T12:43:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-07-26T12:43:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 7]}]': finished 2024-07-26T12:43:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-07-26T12:43:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-07-26T12:43:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-07-26T12:43:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-07-26T12:43:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-07-26T12:43:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [5, 10]}]': finished 2024-07-26T12:43:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-07-26T12:43:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:35 smithi132 ceph-mon[37580]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T12:43:37.034 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:36 smithi107 ceph-mon[32659]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T12:43:37.034 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:36 smithi107 ceph-mon[32659]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:36 smithi112 ceph-mon[37501]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T12:43:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:36 smithi112 ceph-mon[37501]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:36 smithi132 ceph-mon[37580]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T12:43:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:36 smithi132 ceph-mon[37580]: pgmap v317: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.394+0000 7fe3ebe00640 1 -- 172.21.15.107:0/238156877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3ec10f330 msgr2=0x7fe3ec10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.394+0000 7fe3ebe00640 1 --2- 172.21.15.107:0/238156877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3ec10f330 0x7fe3ec10f740 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009860 tx=0x7fe3d802a340 comp rx=0 tx=0).stop 2024-07-26T12:43:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.394+0000 7fe3ebe00640 1 -- 172.21.15.107:0/238156877 shutdown_connections 2024-07-26T12:43:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.394+0000 7fe3ebe00640 1 --2- 172.21.15.107:0/238156877 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3ec10fc80 0x7fe3ec1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.394+0000 7fe3ebe00640 1 --2- 172.21.15.107:0/238156877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3ec10f330 0x7fe3ec10f740 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.394+0000 7fe3ebe00640 1 --2- 172.21.15.107:0/238156877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3ec1099f0 0x7fe3ec109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.394+0000 7fe3ebe00640 1 -- 172.21.15.107:0/238156877 >> 172.21.15.107:0/238156877 conn(0x7fe3ec078ff0 msgr2=0x7fe3ec107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:37.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.394+0000 7fe3ebe00640 1 -- 172.21.15.107:0/238156877 shutdown_connections 2024-07-26T12:43:37.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.394+0000 7fe3ebe00640 1 -- 172.21.15.107:0/238156877 wait complete. 2024-07-26T12:43:37.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.395+0000 7fe3ebe00640 1 Processor -- start 2024-07-26T12:43:37.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3ebe00640 1 -- start start 2024-07-26T12:43:37.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3ec1099f0 0x7fe3ec1a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:37.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3ec10f330 0x7fe3ec1a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:37.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3ebe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3ec10fc80 0x7fe3ec1a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:37.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3ebe00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec119940 con 0x7fe3ec1099f0 2024-07-26T12:43:37.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3ebe00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec1197c0 con 0x7fe3ec10f330 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3ebe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ec119ac0 con 0x7fe3ec10fc80 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3eaa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3ec1099f0 0x7fe3ec1a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3ea000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3ec10f330 0x7fe3ec1a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3ea000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3ec10f330 0x7fe3ec1a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:58044/0 (socket says 172.21.15.107:58044) 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3ea000640 1 -- 172.21.15.107:0/219746440 learned_addr learned my addr 172.21.15.107:0/219746440 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.396+0000 7fe3eb400640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3ec10fc80 0x7fe3ec1a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.397+0000 7fe3ea000640 1 -- 172.21.15.107:0/219746440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3ec10fc80 msgr2=0x7fe3ec1a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.397+0000 7fe3ea000640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3ec10fc80 0x7fe3ec1a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.397+0000 7fe3ea000640 1 -- 172.21.15.107:0/219746440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3ec1099f0 msgr2=0x7fe3ec1a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.397+0000 7fe3ea000640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3ec1099f0 0x7fe3ec1a0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.397+0000 7fe3ea000640 1 -- 172.21.15.107:0/219746440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ec1a62c0 con 0x7fe3ec10f330 2024-07-26T12:43:37.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.397+0000 7fe3ea000640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3ec10f330 0x7fe3ec1a08f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8035040 tx=0x7fe3d8032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.397+0000 7fe3e3400640 1 -- 172.21.15.107:0/219746440 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8037040 con 0x7fe3ec10f330 2024-07-26T12:43:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.398+0000 7fe3e3400640 1 -- 172.21.15.107:0/219746440 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3d8004050 con 0x7fe3ec10f330 2024-07-26T12:43:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.398+0000 7fe3ebe00640 1 -- 172.21.15.107:0/219746440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3ec1a6550 con 0x7fe3ec10f330 2024-07-26T12:43:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.398+0000 7fe3e3400640 1 -- 172.21.15.107:0/219746440 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d80035f0 con 0x7fe3ec10f330 2024-07-26T12:43:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.398+0000 7fe3ebe00640 1 -- 172.21.15.107:0/219746440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3ec1a69d0 con 0x7fe3ec10f330 2024-07-26T12:43:37.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.399+0000 7fe3ebe00640 1 -- 172.21.15.107:0/219746440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3ec109e00 con 0x7fe3ec10f330 2024-07-26T12:43:37.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.401+0000 7fe3e3400640 1 -- 172.21.15.107:0/219746440 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fe3d8043020 con 0x7fe3ec10f330 2024-07-26T12:43:37.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.403+0000 7fe3e3400640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3d00b3c20 0x7fe3d00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:37.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.403+0000 7fe3e3400640 1 -- 172.21.15.107:0/219746440 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fe3d8103580 con 0x7fe3ec10f330 2024-07-26T12:43:37.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.403+0000 7fe3eaa00640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3d00b3c20 0x7fe3d00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:37.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.404+0000 7fe3eaa00640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3d00b3c20 0x7fe3d00b6130 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc005fd0 tx=0x7fe3dc009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:37.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.407+0000 7fe3e3400640 1 -- 172.21.15.107:0/219746440 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe3d804a020 con 0x7fe3ec10f330 2024-07-26T12:43:37.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.645+0000 7fe3ebe00640 1 -- 172.21.15.107:0/219746440 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3ec02e0c0 con 0x7fe3ec10f330 2024-07-26T12:43:37.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.646+0000 7fe3e3400640 1 -- 172.21.15.107:0/219746440 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11857 (secure 0 0 0) 0x7fe3d80ca740 con 0x7fe3ec10f330 2024-07-26T12:43:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:43:37.648 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":16,"btime":"2024-07-26T12:43:33:799646+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24632,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24634,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24652,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24662,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24680,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24703,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24715,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:43:33.626554+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24716,"qdb_cluster":[24716]},"id":1}]} 2024-07-26T12:43:37.648 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 16 2024-07-26T12:43:37.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.649+0000 7fe3ebe00640 1 -- 172.21.15.107:0/219746440 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3d00b3c20 msgr2=0x7fe3d00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:37.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.649+0000 7fe3ebe00640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3d00b3c20 0x7fe3d00b6130 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc005fd0 tx=0x7fe3dc009260 comp rx=0 tx=0).stop 2024-07-26T12:43:37.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.649+0000 7fe3ebe00640 1 -- 172.21.15.107:0/219746440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3ec10f330 msgr2=0x7fe3ec1a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:37.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.649+0000 7fe3ebe00640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3ec10f330 0x7fe3ec1a08f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8035040 tx=0x7fe3d8032d50 comp rx=0 tx=0).stop 2024-07-26T12:43:37.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.649+0000 7fe3ebe00640 1 -- 172.21.15.107:0/219746440 shutdown_connections 2024-07-26T12:43:37.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.649+0000 7fe3ebe00640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3d00b3c20 0x7fe3d00b6130 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:37.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.649+0000 7fe3ebe00640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe3ec10fc80 0x7fe3ec1a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:37.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.649+0000 7fe3ebe00640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe3ec10f330 0x7fe3ec1a08f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:37.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.649+0000 7fe3ebe00640 1 --2- 172.21.15.107:0/219746440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3ec1099f0 0x7fe3ec1a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:37.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.649+0000 7fe3ebe00640 1 -- 172.21.15.107:0/219746440 >> 172.21.15.107:0/219746440 conn(0x7fe3ec078ff0 msgr2=0x7fe3ec0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:37.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.650+0000 7fe3ebe00640 1 -- 172.21.15.107:0/219746440 shutdown_connections 2024-07-26T12:43:37.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:37.650+0000 7fe3ebe00640 1 -- 172.21.15.107:0/219746440 wait complete. 2024-07-26T12:43:38.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:43:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:38 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/219746440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:43:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:38 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/219746440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:43:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:38 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/219746440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:43:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:39 smithi112 ceph-mon[37501]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:39 smithi107 ceph-mon[32659]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:39 smithi132 ceph-mon[37580]: pgmap v318: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 5 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:41.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.200+0000 7f21dec00640 1 -- 172.21.15.107:0/1225409116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21d81099f0 msgr2=0x7f21d8109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:41.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.200+0000 7f21dec00640 1 --2- 172.21.15.107:0/1225409116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21d81099f0 0x7f21d8109e00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f21cc009860 tx=0x7f21cc02a340 comp rx=0 tx=0).stop 2024-07-26T12:43:41.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.201+0000 7f21dec00640 1 -- 172.21.15.107:0/1225409116 shutdown_connections 2024-07-26T12:43:41.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.201+0000 7f21dec00640 1 --2- 172.21.15.107:0/1225409116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810fc80 0x7f21d81164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:41.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.201+0000 7f21dec00640 1 --2- 172.21.15.107:0/1225409116 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21d810f330 0x7f21d810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:41.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.201+0000 7f21dec00640 1 --2- 172.21.15.107:0/1225409116 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21d81099f0 0x7f21d8109e00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:41.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.201+0000 7f21dec00640 1 -- 172.21.15.107:0/1225409116 >> 172.21.15.107:0/1225409116 conn(0x7f21d8078ff0 msgr2=0x7f21d8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:41.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.201+0000 7f21dec00640 1 -- 172.21.15.107:0/1225409116 shutdown_connections 2024-07-26T12:43:41.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.201+0000 7f21dec00640 1 -- 172.21.15.107:0/1225409116 wait complete. 2024-07-26T12:43:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.202+0000 7f21dec00640 1 Processor -- start 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.203+0000 7f21dec00640 1 -- start start 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.203+0000 7f21dec00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21d81099f0 0x7f21d81a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.203+0000 7f21dec00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810f330 0x7f21d81a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.203+0000 7f21dec00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21d810fc80 0x7f21d81a5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.203+0000 7f21dec00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f21d8119940 con 0x7f21d810f330 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.203+0000 7f21dec00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f21d81197c0 con 0x7f21d810fc80 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.203+0000 7f21dec00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f21d8119ac0 con 0x7f21d81099f0 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21de200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21d810fc80 0x7f21d81a5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21dd800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21d81099f0 0x7f21d81a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21de200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21d810fc80 0x7f21d81a5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:58076/0 (socket says 172.21.15.107:58076) 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21dce00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810f330 0x7f21d81a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:41.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21de200640 1 -- 172.21.15.107:0/1345771300 learned_addr learned my addr 172.21.15.107:0/1345771300 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21de200640 1 -- 172.21.15.107:0/1345771300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21d81099f0 msgr2=0x7f21d81a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21de200640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21d81099f0 0x7f21d81a0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21de200640 1 -- 172.21.15.107:0/1345771300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810f330 msgr2=0x7f21d81a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21de200640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810f330 0x7f21d81a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21de200640 1 -- 172.21.15.107:0/1345771300 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d81a6380 con 0x7f21d810fc80 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21dd800640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21d81099f0 0x7f21d81a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.204+0000 7f21dce00640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810f330 0x7f21d81a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.205+0000 7f21de200640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21d810fc80 0x7f21d81a5c50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009fd0 tx=0x7f21d000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.205+0000 7f21d6000640 1 -- 172.21.15.107:0/1345771300 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0004b20 con 0x7f21d810fc80 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.205+0000 7f21d6000640 1 -- 172.21.15.107:0/1345771300 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21d0003e20 con 0x7f21d810fc80 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.205+0000 7f21dec00640 1 -- 172.21.15.107:0/1345771300 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d81a6670 con 0x7f21d810fc80 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.205+0000 7f21d6000640 1 -- 172.21.15.107:0/1345771300 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0004820 con 0x7f21d810fc80 2024-07-26T12:43:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.205+0000 7f21dec00640 1 -- 172.21.15.107:0/1345771300 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d81acdd0 con 0x7f21d810fc80 2024-07-26T12:43:41.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.206+0000 7f21dec00640 1 -- 172.21.15.107:0/1345771300 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d8109e00 con 0x7f21d810fc80 2024-07-26T12:43:41.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.208+0000 7f21d6000640 1 -- 172.21.15.107:0/1345771300 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f21d001d020 con 0x7f21d810fc80 2024-07-26T12:43:41.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.210+0000 7f21d6000640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c00b3c20 0x7f21c00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:41.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.210+0000 7f21d6000640 1 -- 172.21.15.107:0/1345771300 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f21d00e4140 con 0x7f21d810fc80 2024-07-26T12:43:41.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.210+0000 7f21dd800640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c00b3c20 0x7f21c00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:41.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.211+0000 7f21dd800640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c00b3c20 0x7f21c00b6130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f21cc02a930 tx=0x7f21cc0058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:41.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.212+0000 7f21d6000640 1 -- 172.21.15.107:0/1345771300 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f21d0004de0 con 0x7f21d810fc80 2024-07-26T12:43:41.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.412+0000 7f21dec00640 1 -- 172.21.15.107:0/1345771300 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f21d81a1540 con 0x7f21d810fc80 2024-07-26T12:43:41.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.414+0000 7f21d6000640 1 -- 172.21.15.107:0/1345771300 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25016 (secure 0 0 0) 0x7f21d81a1540 con 0x7f21d810fc80 2024-07-26T12:43:41.415 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:43:41.416 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:43:35.834161+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 -- 172.21.15.107:0/1345771300 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c00b3c20 msgr2=0x7f21c00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c00b3c20 0x7f21c00b6130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f21cc02a930 tx=0x7f21cc0058d0 comp rx=0 tx=0).stop 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 -- 172.21.15.107:0/1345771300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21d810fc80 msgr2=0x7f21d81a5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21d810fc80 0x7f21d81a5c50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009fd0 tx=0x7f21d000c800 comp rx=0 tx=0).stop 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 -- 172.21.15.107:0/1345771300 shutdown_connections 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c00b3c20 0x7f21c00b6130 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21d810fc80 0x7f21d81a5c50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21d810f330 0x7f21d81a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 --2- 172.21.15.107:0/1345771300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21d81099f0 0x7f21d81a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 -- 172.21.15.107:0/1345771300 >> 172.21.15.107:0/1345771300 conn(0x7f21d8078ff0 msgr2=0x7f21d8107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:41.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 -- 172.21.15.107:0/1345771300 shutdown_connections 2024-07-26T12:43:41.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:41.417+0000 7f21dec00640 1 -- 172.21.15.107:0/1345771300 wait complete. 2024-07-26T12:43:41.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:41 smithi107 ceph-mon[32659]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:41.821 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:41 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1345771300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:43:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:41 smithi132 ceph-mon[37580]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:41 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1345771300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:43:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:41 smithi112 ceph-mon[37501]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:41 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1345771300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:43:42.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:43:42.735 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:44.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:43 smithi112 ceph-mon[37501]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:43 smithi107 ceph-mon[32659]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:43 smithi132 ceph-mon[37580]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:44.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.972+0000 7feb4be00640 1 -- 172.21.15.107:0/701727097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb4c077cc0 msgr2=0x7feb4c113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:44.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.972+0000 7feb4be00640 1 --2- 172.21.15.107:0/701727097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb4c077cc0 0x7feb4c113a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feb34009900 tx=0x7feb3402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:43:44.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.972+0000 7feb4be00640 1 -- 172.21.15.107:0/701727097 shutdown_connections 2024-07-26T12:43:44.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.972+0000 7feb4be00640 1 --2- 172.21.15.107:0/701727097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4c113f80 0x7feb4c1163a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:44.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.972+0000 7feb4be00640 1 --2- 172.21.15.107:0/701727097 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb4c077cc0 0x7feb4c113a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:44.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.972+0000 7feb4be00640 1 --2- 172.21.15.107:0/701727097 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb4c077370 0x7feb4c077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:44.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.972+0000 7feb4be00640 1 -- 172.21.15.107:0/701727097 >> 172.21.15.107:0/701727097 conn(0x7feb4c101010 msgr2=0x7feb4c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:44.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.972+0000 7feb4be00640 1 -- 172.21.15.107:0/701727097 shutdown_connections 2024-07-26T12:43:44.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.972+0000 7feb4be00640 1 -- 172.21.15.107:0/701727097 wait complete. 2024-07-26T12:43:44.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.973+0000 7feb4be00640 1 Processor -- start 2024-07-26T12:43:44.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb4be00640 1 -- start start 2024-07-26T12:43:44.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb4be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb4c077370 0x7feb4c1a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:44.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb4be00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4c077cc0 0x7feb4c1a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:44.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb4be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb4c113f80 0x7feb4c1a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:44.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb4be00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c119930 con 0x7feb4c077cc0 2024-07-26T12:43:44.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb4be00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c1197b0 con 0x7feb4c077370 2024-07-26T12:43:44.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb4be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c119ab0 con 0x7feb4c113f80 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb43e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4c077cc0 0x7feb4c1a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb43e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4c077cc0 0x7feb4c1a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53218/0 (socket says 172.21.15.107:53218) 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb43e00640 1 -- 172.21.15.107:0/856418941 learned_addr learned my addr 172.21.15.107:0/856418941 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.974+0000 7feb4aa00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb4c077370 0x7feb4c1a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.975+0000 7feb4b400640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb4c113f80 0x7feb4c1a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.975+0000 7feb4aa00640 1 -- 172.21.15.107:0/856418941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb4c113f80 msgr2=0x7feb4c1a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.975+0000 7feb4aa00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb4c113f80 0x7feb4c1a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.975+0000 7feb4aa00640 1 -- 172.21.15.107:0/856418941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4c077cc0 msgr2=0x7feb4c1a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.975+0000 7feb4aa00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4c077cc0 0x7feb4c1a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.975+0000 7feb4aa00640 1 -- 172.21.15.107:0/856418941 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c1a62b0 con 0x7feb4c077370 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.975+0000 7feb43e00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4c077cc0 0x7feb4c1a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:44.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.975+0000 7feb4aa00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb4c077370 0x7feb4c1a0340 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00e9f0 tx=0x7feb3c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:44.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.975+0000 7feb43400640 1 -- 172.21.15.107:0/856418941 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c00ccc0 con 0x7feb4c077370 2024-07-26T12:43:44.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.976+0000 7feb43400640 1 -- 172.21.15.107:0/856418941 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb3c005020 con 0x7feb4c077370 2024-07-26T12:43:44.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.976+0000 7feb43400640 1 -- 172.21.15.107:0/856418941 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c004a00 con 0x7feb4c077370 2024-07-26T12:43:44.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.976+0000 7feb4be00640 1 -- 172.21.15.107:0/856418941 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4c1a65a0 con 0x7feb4c077370 2024-07-26T12:43:44.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.976+0000 7feb4be00640 1 -- 172.21.15.107:0/856418941 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4c1acd00 con 0x7feb4c077370 2024-07-26T12:43:44.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.977+0000 7feb4be00640 1 -- 172.21.15.107:0/856418941 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb4c077780 con 0x7feb4c077370 2024-07-26T12:43:44.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.979+0000 7feb43400640 1 -- 172.21.15.107:0/856418941 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7feb3c004ba0 con 0x7feb4c077370 2024-07-26T12:43:44.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.981+0000 7feb43400640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7feb300b3ca0 0x7feb300b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:44.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.981+0000 7feb43400640 1 -- 172.21.15.107:0/856418941 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7feb3c0e40d0 con 0x7feb4c077370 2024-07-26T12:43:44.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.982+0000 7feb43e00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7feb300b3ca0 0x7feb300b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:44.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.982+0000 7feb43400640 1 -- 172.21.15.107:0/856418941 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7feb3c0ab290 con 0x7feb4c077370 2024-07-26T12:43:44.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:44.982+0000 7feb43e00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7feb300b3ca0 0x7feb300b61b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feb34006010 tx=0x7feb34005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:45.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.182+0000 7feb4be00640 1 -- 172.21.15.107:0/856418941 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feb4c1a1470 con 0x7feb4c077370 2024-07-26T12:43:45.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.184+0000 7feb43400640 1 -- 172.21.15.107:0/856418941 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25016 (secure 0 0 0) 0x7feb4c1a1470 con 0x7feb4c077370 2024-07-26T12:43:45.187 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:43:45.187 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:43:35.834161+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:43:45.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 -- 172.21.15.107:0/856418941 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7feb300b3ca0 msgr2=0x7feb300b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:45.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7feb300b3ca0 0x7feb300b61b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feb34006010 tx=0x7feb34005a20 comp rx=0 tx=0).stop 2024-07-26T12:43:45.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 -- 172.21.15.107:0/856418941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb4c077370 msgr2=0x7feb4c1a0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:45.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb4c077370 0x7feb4c1a0340 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00e9f0 tx=0x7feb3c00eeb0 comp rx=0 tx=0).stop 2024-07-26T12:43:45.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 -- 172.21.15.107:0/856418941 shutdown_connections 2024-07-26T12:43:45.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7feb300b3ca0 0x7feb300b61b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:45.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb4c113f80 0x7feb4c1a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:45.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feb4c077cc0 0x7feb4c1a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:45.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 --2- 172.21.15.107:0/856418941 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7feb4c077370 0x7feb4c1a0340 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:45.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 -- 172.21.15.107:0/856418941 >> 172.21.15.107:0/856418941 conn(0x7feb4c101010 msgr2=0x7feb4c116070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:45.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 -- 172.21.15.107:0/856418941 shutdown_connections 2024-07-26T12:43:45.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:45.188+0000 7feb4be00640 1 -- 172.21.15.107:0/856418941 wait complete. 2024-07-26T12:43:45.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:45 smithi107 ceph-mon[32659]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:45.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:45 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/856418941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:43:46.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:43:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:45 smithi132 ceph-mon[37580]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:45 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/856418941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:43:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:45 smithi112 ceph-mon[37501]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3 B/s, 1 keys/s, 0 objects/s recovering 2024-07-26T12:43:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:45 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/856418941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:43:46.500 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:47 smithi107 ceph-mon[32659]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-07-26T12:43:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:47 smithi132 ceph-mon[37580]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-07-26T12:43:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:47 smithi112 ceph-mon[37501]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-07-26T12:43:48.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.746+0000 7f549be00640 1 -- 172.21.15.107:0/3400949679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c10fc80 msgr2=0x7f549c1164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:48.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.746+0000 7f549be00640 1 --2- 172.21.15.107:0/3400949679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c10fc80 0x7f549c1164b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5494009be0 tx=0x7f549402a6c0 comp rx=0 tx=0).stop 2024-07-26T12:43:48.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.746+0000 7f549be00640 1 -- 172.21.15.107:0/3400949679 shutdown_connections 2024-07-26T12:43:48.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.746+0000 7f549be00640 1 --2- 172.21.15.107:0/3400949679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c10fc80 0x7f549c1164b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:48.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.746+0000 7f549be00640 1 --2- 172.21.15.107:0/3400949679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c10f330 0x7f549c10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:48.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.746+0000 7f549be00640 1 --2- 172.21.15.107:0/3400949679 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f549c1099f0 0x7f549c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:48.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.746+0000 7f549be00640 1 -- 172.21.15.107:0/3400949679 >> 172.21.15.107:0/3400949679 conn(0x7f549c078ff0 msgr2=0x7f549c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:48.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.746+0000 7f549be00640 1 -- 172.21.15.107:0/3400949679 shutdown_connections 2024-07-26T12:43:48.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.746+0000 7f549be00640 1 -- 172.21.15.107:0/3400949679 wait complete. 2024-07-26T12:43:48.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549be00640 1 Processor -- start 2024-07-26T12:43:48.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549be00640 1 -- start start 2024-07-26T12:43:48.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549be00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c1099f0 0x7f549c1a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:48.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f549c10f330 0x7f549c1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:48.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c10fc80 0x7f549c1a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:48.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549be00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f549c119940 con 0x7f549c1099f0 2024-07-26T12:43:48.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549be00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1197c0 con 0x7f549c10f330 2024-07-26T12:43:48.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f549c119ac0 con 0x7f549c10fc80 2024-07-26T12:43:48.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549aa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c1099f0 0x7f549c1a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:48.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549aa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c1099f0 0x7f549c1a02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53248/0 (socket says 172.21.15.107:53248) 2024-07-26T12:43:48.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549aa00640 1 -- 172.21.15.107:0/3561024575 learned_addr learned my addr 172.21.15.107:0/3561024575 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:48.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.748+0000 7f549a000640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f549c10f330 0x7f549c1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:48.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.749+0000 7f549b400640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c10fc80 0x7f549c1a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:48.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.749+0000 7f549aa00640 1 -- 172.21.15.107:0/3561024575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c10fc80 msgr2=0x7f549c1a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:48.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.749+0000 7f549aa00640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c10fc80 0x7f549c1a5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:48.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.749+0000 7f549aa00640 1 -- 172.21.15.107:0/3561024575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f549c10f330 msgr2=0x7f549c1a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:48.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.749+0000 7f549aa00640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f549c10f330 0x7f549c1a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:48.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.749+0000 7f549aa00640 1 -- 172.21.15.107:0/3561024575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c1a6220 con 0x7f549c1099f0 2024-07-26T12:43:48.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.749+0000 7f549a000640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f549c10f330 0x7f549c1a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.749+0000 7f549aa00640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c1099f0 0x7f549c1a02b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f548c00d950 tx=0x7f548c00dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.750+0000 7f5493400640 1 -- 172.21.15.107:0/3561024575 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548c017030 con 0x7f549c1099f0 2024-07-26T12:43:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.750+0000 7f549be00640 1 -- 172.21.15.107:0/3561024575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549c1a6510 con 0x7f549c1099f0 2024-07-26T12:43:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.750+0000 7f5493400640 1 -- 172.21.15.107:0/3561024575 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f548c004020 con 0x7f549c1099f0 2024-07-26T12:43:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.750+0000 7f5493400640 1 -- 172.21.15.107:0/3561024575 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f548c005560 con 0x7f549c1099f0 2024-07-26T12:43:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.750+0000 7f549be00640 1 -- 172.21.15.107:0/3561024575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f549c1acc70 con 0x7f549c1099f0 2024-07-26T12:43:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.751+0000 7f549be00640 1 -- 172.21.15.107:0/3561024575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f549c109e00 con 0x7f549c1099f0 2024-07-26T12:43:48.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.756+0000 7f5493400640 1 -- 172.21.15.107:0/3561024575 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f548c009a30 con 0x7f549c1099f0 2024-07-26T12:43:48.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.757+0000 7f5493400640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f54800b3c20 0x7f54800b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:48.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.758+0000 7f5493400640 1 -- 172.21.15.107:0/3561024575 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f548c0e40a0 con 0x7f549c1099f0 2024-07-26T12:43:48.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.758+0000 7f549a000640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f54800b3c20 0x7f54800b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:48.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.758+0000 7f549a000640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f54800b3c20 0x7f54800b6130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f549c10fa50 tx=0x7f5488009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:48.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.760+0000 7f5493400640 1 -- 172.21.15.107:0/3561024575 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f548c0ab260 con 0x7f549c1099f0 2024-07-26T12:43:48.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.996+0000 7f549be00640 1 -- 172.21.15.107:0/3561024575 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f549c1a13e0 con 0x7f549c1099f0 2024-07-26T12:43:48.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:48.998+0000 7f5493400640 1 -- 172.21.15.107:0/3561024575 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11857 (secure 0 0 0) 0x7f548c0aab60 con 0x7f549c1099f0 2024-07-26T12:43:49.000 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:43:49.000 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":16,"btime":"2024-07-26T12:43:33:799646+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24632,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24634,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24652,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24662,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24680,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24694,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24703,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24715,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:43:33.626554+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24716},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24716,"qdb_cluster":[24716]},"id":1}]} 2024-07-26T12:43:49.000 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 16 2024-07-26T12:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 -- 172.21.15.107:0/3561024575 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f54800b3c20 msgr2=0x7f54800b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f54800b3c20 0x7f54800b6130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f549c10fa50 tx=0x7f5488009260 comp rx=0 tx=0).stop 2024-07-26T12:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 -- 172.21.15.107:0/3561024575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c1099f0 msgr2=0x7f549c1a02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c1099f0 0x7f549c1a02b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f548c00d950 tx=0x7f548c00dd50 comp rx=0 tx=0).stop 2024-07-26T12:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 -- 172.21.15.107:0/3561024575 shutdown_connections 2024-07-26T12:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f54800b3c20 0x7f54800b6130 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c10fc80 0x7f549c1a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:49.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f549c10f330 0x7f549c1a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 --2- 172.21.15.107:0/3561024575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f549c1099f0 0x7f549c1a02b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 -- 172.21.15.107:0/3561024575 >> 172.21.15.107:0/3561024575 conn(0x7f549c078ff0 msgr2=0x7f549c0796b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 -- 172.21.15.107:0/3561024575 shutdown_connections 2024-07-26T12:43:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:49.002+0000 7f549be00640 1 -- 172.21.15.107:0/3561024575 wait complete. 2024-07-26T12:43:49.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:43:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:49 smithi107 ceph-mon[32659]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:49 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3561024575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:43:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:49 smithi132 ceph-mon[37580]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:49 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3561024575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:43:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:49 smithi112 ceph-mon[37501]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:49 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3561024575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:43:50.256 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:51.915 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:51 smithi107 ceph-mon[32659]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:51 smithi132 ceph-mon[37580]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:51 smithi112 ceph-mon[37501]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.480+0000 7fc126800640 1 -- 172.21.15.107:0/298190124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc12010fc80 msgr2=0x7fc1201164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.480+0000 7fc126800640 1 --2- 172.21.15.107:0/298190124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc12010fc80 0x7fc1201164b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc118009ca0 tx=0x7fc11802a8c0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.481+0000 7fc126800640 1 -- 172.21.15.107:0/298190124 shutdown_connections 2024-07-26T12:43:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.481+0000 7fc126800640 1 --2- 172.21.15.107:0/298190124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc12010fc80 0x7fc1201164b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.481+0000 7fc126800640 1 --2- 172.21.15.107:0/298190124 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc12010f330 0x7fc12010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.481+0000 7fc126800640 1 --2- 172.21.15.107:0/298190124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1201099f0 0x7fc120109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.481+0000 7fc126800640 1 -- 172.21.15.107:0/298190124 >> 172.21.15.107:0/298190124 conn(0x7fc120078ff0 msgr2=0x7fc120107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.481+0000 7fc126800640 1 -- 172.21.15.107:0/298190124 shutdown_connections 2024-07-26T12:43:52.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.481+0000 7fc126800640 1 -- 172.21.15.107:0/298190124 wait complete. 2024-07-26T12:43:52.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.482+0000 7fc126800640 1 Processor -- start 2024-07-26T12:43:52.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.482+0000 7fc126800640 1 -- start start 2024-07-26T12:43:52.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.482+0000 7fc126800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1201099f0 0x7fc1201a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:52.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.482+0000 7fc126800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc12010f330 0x7fc1201a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:52.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.482+0000 7fc126800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc12010fc80 0x7fc1201a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:52.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc126800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc120119940 con 0x7fc12010fc80 2024-07-26T12:43:52.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc126800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc1201197c0 con 0x7fc12010f330 2024-07-26T12:43:52.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc126800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc120119ac0 con 0x7fc1201099f0 2024-07-26T12:43:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc125e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc12010fc80 0x7fc1201a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc125e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc12010fc80 0x7fc1201a5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53270/0 (socket says 172.21.15.107:53270) 2024-07-26T12:43:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc125e00640 1 -- 172.21.15.107:0/2452259682 learned_addr learned my addr 172.21.15.107:0/2452259682 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc125400640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1201099f0 0x7fc1201a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc124a00640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc12010f330 0x7fc1201a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc125e00640 1 -- 172.21.15.107:0/2452259682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1201099f0 msgr2=0x7fc1201a02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:52.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc125e00640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1201099f0 0x7fc1201a02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc125e00640 1 -- 172.21.15.107:0/2452259682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc12010f330 msgr2=0x7fc1201a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:52.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc125e00640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc12010f330 0x7fc1201a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc125e00640 1 -- 172.21.15.107:0/2452259682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1201a6220 con 0x7fc12010fc80 2024-07-26T12:43:52.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.483+0000 7fc125400640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1201099f0 0x7fc1201a02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:52.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.484+0000 7fc125e00640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc12010fc80 0x7fc1201a5af0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc1180098d0 tx=0x7fc1180317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:52.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.484+0000 7fc11e000640 1 -- 172.21.15.107:0/2452259682 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118003b70 con 0x7fc12010fc80 2024-07-26T12:43:52.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.484+0000 7fc126800640 1 -- 172.21.15.107:0/2452259682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1201a64b0 con 0x7fc12010fc80 2024-07-26T12:43:52.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.484+0000 7fc11e000640 1 -- 172.21.15.107:0/2452259682 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc118003d10 con 0x7fc12010fc80 2024-07-26T12:43:52.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.484+0000 7fc11e000640 1 -- 172.21.15.107:0/2452259682 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11802fa20 con 0x7fc12010fc80 2024-07-26T12:43:52.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.484+0000 7fc126800640 1 -- 172.21.15.107:0/2452259682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1201acc70 con 0x7fc12010fc80 2024-07-26T12:43:52.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.485+0000 7fc126800640 1 -- 172.21.15.107:0/2452259682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc120109e00 con 0x7fc12010fc80 2024-07-26T12:43:52.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.488+0000 7fc11e000640 1 -- 172.21.15.107:0/2452259682 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fc11802fbe0 con 0x7fc12010fc80 2024-07-26T12:43:52.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.490+0000 7fc11e000640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1080b3c20 0x7fc1080b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:52.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.490+0000 7fc11e000640 1 -- 172.21.15.107:0/2452259682 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fc1181056e0 con 0x7fc12010fc80 2024-07-26T12:43:52.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.490+0000 7fc125400640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1080b3c20 0x7fc1080b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:52.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.491+0000 7fc125400640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1080b3c20 0x7fc1080b6130 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc114009fc0 tx=0x7fc114005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:52.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.492+0000 7fc11e000640 1 -- 172.21.15.107:0/2452259682 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc1180cc8a0 con 0x7fc12010fc80 2024-07-26T12:43:52.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.691+0000 7fc126800640 1 -- 172.21.15.107:0/2452259682 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1201a13e0 con 0x7fc12010fc80 2024-07-26T12:43:52.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.695+0000 7fc11e000640 1 -- 172.21.15.107:0/2452259682 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25016 (secure 0 0 0) 0x7fc1201a13e0 con 0x7fc12010fc80 2024-07-26T12:43:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:43:52.697 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:43:35.834161+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.697+0000 7fc126800640 1 -- 172.21.15.107:0/2452259682 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1080b3c20 msgr2=0x7fc1080b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.697+0000 7fc126800640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1080b3c20 0x7fc1080b6130 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc114009fc0 tx=0x7fc114005f90 comp rx=0 tx=0).stop 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.698+0000 7fc126800640 1 -- 172.21.15.107:0/2452259682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc12010fc80 msgr2=0x7fc1201a5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.698+0000 7fc126800640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc12010fc80 0x7fc1201a5af0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc1180098d0 tx=0x7fc1180317b0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.698+0000 7fc126800640 1 -- 172.21.15.107:0/2452259682 shutdown_connections 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.698+0000 7fc126800640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc1080b3c20 0x7fc1080b6130 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.698+0000 7fc126800640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc12010fc80 0x7fc1201a5af0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.698+0000 7fc126800640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc12010f330 0x7fc1201a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.698+0000 7fc126800640 1 --2- 172.21.15.107:0/2452259682 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1201099f0 0x7fc1201a02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.698+0000 7fc126800640 1 -- 172.21.15.107:0/2452259682 >> 172.21.15.107:0/2452259682 conn(0x7fc120078ff0 msgr2=0x7fc120079600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.698+0000 7fc126800640 1 -- 172.21.15.107:0/2452259682 shutdown_connections 2024-07-26T12:43:52.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:52.698+0000 7fc126800640 1 -- 172.21.15.107:0/2452259682 wait complete. 2024-07-26T12:43:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:52 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2452259682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:43:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:52 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2452259682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:43:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:52 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2452259682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:43:53.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T12:43:15.198289+0000', 'modified': '2024-07-26T12:43:33.626554+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24716}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24716': {'gid': 24716, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.112:6839/3779803434', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 3779803434}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 3779803434}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24716, 'qdb_cluster': [24716]} 2024-07-26T12:43:53.553 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T12:43:53.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T12:43:54.018 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:54.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:53 smithi107 ceph-mon[32659]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:53 smithi132 ceph-mon[37580]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:53 smithi112 ceph-mon[37501]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:55.947 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:55 smithi107 ceph-mon[32659]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:55 smithi132 ceph-mon[37580]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:55 smithi112 ceph-mon[37501]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.263+0000 7f4381800640 1 -- 172.21.15.107:0/2858050743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f437c102250 msgr2=0x7f437c102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.263+0000 7f4381800640 1 --2- 172.21.15.107:0/2858050743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f437c102250 0x7f437c102660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f436c009900 tx=0x7f436c02a410 comp rx=0 tx=0).stop 2024-07-26T12:43:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.263+0000 7f4381800640 1 -- 172.21.15.107:0/2858050743 shutdown_connections 2024-07-26T12:43:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.263+0000 7f4381800640 1 --2- 172.21.15.107:0/2858050743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f437c107660 0x7f437c107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.263+0000 7f4381800640 1 --2- 172.21.15.107:0/2858050743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f437c102ba0 0x7f437c1070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.263+0000 7f4381800640 1 --2- 172.21.15.107:0/2858050743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f437c102250 0x7f437c102660 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.263+0000 7f4381800640 1 -- 172.21.15.107:0/2858050743 >> 172.21.15.107:0/2858050743 conn(0x7f437c079190 msgr2=0x7f437c0795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.263+0000 7f4381800640 1 -- 172.21.15.107:0/2858050743 shutdown_connections 2024-07-26T12:43:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.263+0000 7f4381800640 1 -- 172.21.15.107:0/2858050743 wait complete. 2024-07-26T12:43:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.265+0000 7f4381800640 1 Processor -- start 2024-07-26T12:43:56.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.265+0000 7f4381800640 1 -- start start 2024-07-26T12:43:56.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.265+0000 7f4381800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f437c102250 0x7f437c1a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.265+0000 7f4381800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f437c102ba0 0x7f437c1a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.265+0000 7f4381800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f437c107660 0x7f437c1a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.265+0000 7f4381800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f437c119af0 con 0x7f437c102ba0 2024-07-26T12:43:56.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.265+0000 7f4381800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f437c119970 con 0x7f437c107660 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.265+0000 7f4381800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f437c119c70 con 0x7f437c102250 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437a000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f437c102ba0 0x7f437c1a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437a000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f437c102ba0 0x7f437c1a0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48376/0 (socket says 172.21.15.107:48376) 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437a000640 1 -- 172.21.15.107:0/483252228 learned_addr learned my addr 172.21.15.107:0/483252228 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437aa00640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f437c102250 0x7f437c1a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437b400640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f437c107660 0x7f437c1a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437b400640 1 -- 172.21.15.107:0/483252228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f437c102250 msgr2=0x7f437c1a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437b400640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f437c102250 0x7f437c1a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437b400640 1 -- 172.21.15.107:0/483252228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f437c102ba0 msgr2=0x7f437c1a0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437b400640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f437c102ba0 0x7f437c1a0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437b400640 1 -- 172.21.15.107:0/483252228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c1a6490 con 0x7f437c107660 2024-07-26T12:43:56.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437a000640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f437c102ba0 0x7f437c1a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:56.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.266+0000 7f437b400640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f437c107660 0x7f437c1a5d60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f437400ec50 tx=0x7f437400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:56.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.267+0000 7f4373400640 1 -- 172.21.15.107:0/483252228 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374003b70 con 0x7f437c107660 2024-07-26T12:43:56.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.267+0000 7f4381800640 1 -- 172.21.15.107:0/483252228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437c1a6780 con 0x7f437c107660 2024-07-26T12:43:56.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.267+0000 7f4373400640 1 -- 172.21.15.107:0/483252228 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4374003d10 con 0x7f437c107660 2024-07-26T12:43:56.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.267+0000 7f4373400640 1 -- 172.21.15.107:0/483252228 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374010830 con 0x7f437c107660 2024-07-26T12:43:56.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.267+0000 7f4381800640 1 -- 172.21.15.107:0/483252228 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f437c1acee0 con 0x7f437c107660 2024-07-26T12:43:56.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.270+0000 7f4388444740 1 Processor -- start 2024-07-26T12:43:56.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.271+0000 7f4373400640 1 -- 172.21.15.107:0/483252228 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f4374003ec0 con 0x7f437c107660 2024-07-26T12:43:56.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.271+0000 7f4388444740 1 -- start start 2024-07-26T12:43:56.273 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.271+0000 7f4388444740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e55f60 0x55a3121fb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.271+0000 7f4388444740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e2d0a0 0x55a3121fbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.271+0000 7f4388444740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a3121fbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.271+0000 7f4388444740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a312171970 con 0x55a311e2d0a0 2024-07-26T12:43:56.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.271+0000 7f4388444740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a312160e70 con 0x55a311e35280 2024-07-26T12:43:56.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.271+0000 7f4388444740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a312164eb0 con 0x55a311e55f60 2024-07-26T12:43:56.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437aa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e2d0a0 0x55a3121fbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437a000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e55f60 0x55a3121fb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437aa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e2d0a0 0x55a3121fbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48384/0 (socket says 172.21.15.107:48384) 2024-07-26T12:43:56.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437aa00640 1 -- 172.21.15.107:0/2104101886 learned_addr learned my addr 172.21.15.107:0/2104101886 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:56.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437a000640 1 --2- 172.21.15.107:0/2104101886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a3121fbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437aa00640 1 -- 172.21.15.107:0/2104101886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e55f60 msgr2=0x55a3121fb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:43:56.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437aa00640 1 --2- 172.21.15.107:0/2104101886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e55f60 0x55a3121fb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437aa00640 1 -- 172.21.15.107:0/2104101886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 msgr2=0x55a3121fbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:43:56.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437aa00640 1 --2- 172.21.15.107:0/2104101886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a3121fbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437aa00640 1 -- 172.21.15.107:0/2104101886 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a312108110 con 0x55a311e2d0a0 2024-07-26T12:43:56.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437a000640 1 --2- 172.21.15.107:0/2104101886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e55f60 0x55a3121fb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:56.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.272+0000 7f437aa00640 1 --2- 172.21.15.107:0/2104101886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e2d0a0 0x55a3121fbab0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f437c06c2c0 tx=0x7f436c032b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aacb273d53db104 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:56.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.273+0000 7f4373400640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f43600b3c20 0x7f43600b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.273+0000 7f4373400640 1 -- 172.21.15.107:0/483252228 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f43740e45e0 con 0x7f437c107660 2024-07-26T12:43:56.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.273+0000 7f4367e00640 1 -- 172.21.15.107:0/2104101886 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c040070 con 0x55a311e2d0a0 2024-07-26T12:43:56.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.273+0000 7f4367e00640 1 -- 172.21.15.107:0/2104101886 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f436c004030 con 0x55a311e2d0a0 2024-07-26T12:43:56.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.273+0000 7f437a000640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f43600b3c20 0x7f43600b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.273+0000 7f4367e00640 1 -- 172.21.15.107:0/2104101886 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c035040 con 0x55a311e2d0a0 2024-07-26T12:43:56.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.274+0000 7f4388444740 1 -- 172.21.15.107:0/2104101886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e2d0a0 msgr2=0x55a3121fbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:56.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.274+0000 7f437a000640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f43600b3c20 0x7f43600b6130 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f43680095e0 tx=0x7f4368005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:56.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.274+0000 7f4388444740 1 --2- 172.21.15.107:0/2104101886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e2d0a0 0x55a3121fbab0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f437c06c2c0 tx=0x7f436c032b20 comp rx=0 tx=0).stop 2024-07-26T12:43:56.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.274+0000 7f4388444740 1 -- 172.21.15.107:0/2104101886 shutdown_connections 2024-07-26T12:43:56.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.274+0000 7f4388444740 1 --2- 172.21.15.107:0/2104101886 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a3121fbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.274+0000 7f4388444740 1 --2- 172.21.15.107:0/2104101886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e2d0a0 0x55a3121fbab0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.274+0000 7f4388444740 1 --2- 172.21.15.107:0/2104101886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e55f60 0x55a3121fb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.274+0000 7f4388444740 1 -- 172.21.15.107:0/2104101886 >> 172.21.15.107:0/2104101886 conn(0x55a311e4a350 msgr2=0x55a3121f9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.274+0000 7f4388444740 1 -- 172.21.15.107:0/2104101886 shutdown_connections 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.274+0000 7f4388444740 1 -- 172.21.15.107:0/2104101886 wait complete. 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.275+0000 7f4388444740 1 -- start start 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.275+0000 7f4388444740 1 Processor -- start 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f4388444740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e2d0a0 0x55a312206e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f4388444740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e35280 0x55a3122073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f4388444740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a312213010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f4388444740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a311fc45f0 con 0x55a311e55f60 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f4388444740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a3120fd540 con 0x55a311e2d0a0 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f4388444740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a312209cf0 con 0x55a311e35280 2024-07-26T12:43:56.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f4388444740 10 client.0 _ll_register_callbacks cb 0x55a31210cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e2d0a0 0x55a312206e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a312213010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e2d0a0 0x55a312206e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:56172/0 (socket says 172.21.15.107:56172) 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437aa00640 1 -- 172.21.15.107:0/2079999284 learned_addr learned my addr 172.21.15.107:0/2079999284 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437a000640 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e35280 0x55a3122073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437aa00640 1 -- 172.21.15.107:0/2079999284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e35280 msgr2=0x55a3122073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437aa00640 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e35280 0x55a3122073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437aa00640 1 -- 172.21.15.107:0/2079999284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 msgr2=0x55a312213010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437aa00640 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a312213010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437aa00640 1 -- 172.21.15.107:0/2079999284 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a311ec1540 con 0x55a311e2d0a0 2024-07-26T12:43:56.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.276+0000 7f437a000640 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e35280 0x55a3122073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.277+0000 7f437aa00640 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e2d0a0 0x55a312206e20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f436c0095a0 tx=0x7f436c02ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae79f8ce5996575a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.277+0000 7f4364c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.277+0000 7f4364c00640 1 -- 172.21.15.107:0/2079999284 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c035070 con 0x55a311e2d0a0 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.277+0000 7f4388444740 1 -- 172.21.15.107:0/2079999284 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f436c004030 con 0x55a311e2d0a0 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.277+0000 7f4364c00640 1 -- 172.21.15.107:0/2079999284 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f436c0314a0 con 0x55a311e2d0a0 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.277+0000 7f4364c00640 1 -- 172.21.15.107:0/2079999284 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f436c058050 con 0x55a311e2d0a0 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.277+0000 7f4388444740 10 client.24812 fetch_fsmap learned FSMap version 16 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.277+0000 7f4388444740 1 -- 172.21.15.107:0/2079999284 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x55a312215bc0 con 0x55a311e2d0a0 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.278+0000 7f4364c00640 1 -- 172.21.15.107:0/2079999284 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f436c004030 con 0x55a311e2d0a0 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.278+0000 7f4364c00640 1 -- 172.21.15.107:0/2079999284 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 16) ==== 8350+0+0 (secure 0 0 0) 0x7f436c004a50 con 0x55a311e2d0a0 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.278+0000 7f4388444740 10 client.24812 fetch_fsmap finished waiting for FSMap version 16 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.278+0000 7f4388444740 10 client.24812 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.278+0000 7f4388444740 20 client.24812 populate_metadata read hostname 'smithi107' 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.278+0000 7f4388444740 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55a311e148e0 0x55a312219170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.278+0000 7f4388444740 4 client.24812 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:43:56.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.278+0000 7f4388444740 1 -- 172.21.15.107:0/2079999284 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3121738c0 con 0x55a311e148e0 2024-07-26T12:43:56.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.279+0000 7f437a000640 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55a311e148e0 0x55a312219170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.279+0000 7f437a000640 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55a311e148e0 0x55a312219170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f27ffa5d161fc0c3 server_cookie=e0488c0474a23d59 in_seq=0 out_seq=0 2024-07-26T12:43:56.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.279+0000 7f4364c00640 10 client.24812 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:43:56.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4364c00640 1 -- 172.21.15.107:0/2079999284 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a3121738c0 con 0x55a311e148e0 2024-07-26T12:43:56.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4364c00640 10 client.24812 handle_command_reply: tid=0 2024-07-26T12:43:56.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4388444740 1 client.24812 shutdown 2024-07-26T12:43:56.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4388444740 1 -- 172.21.15.107:0/2079999284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e2d0a0 msgr2=0x55a312206e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:56.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4388444740 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e2d0a0 0x55a312206e20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f436c0095a0 tx=0x7f436c02ab10 comp rx=0 tx=0).stop 2024-07-26T12:43:56.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4388444740 1 -- 172.21.15.107:0/2079999284 shutdown_connections 2024-07-26T12:43:56.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4388444740 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55a311e148e0 0x55a312219170 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4388444740 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a312213010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4388444740 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e35280 0x55a3122073e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4388444740 1 --2- 172.21.15.107:0/2079999284 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e2d0a0 0x55a312206e20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4364c00640 0 client.24812 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:43:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.280+0000 7f4388444740 1 -- 172.21.15.107:0/2079999284 >> 172.21.15.107:0/2079999284 conn(0x55a311e4a350 msgr2=0x55a3121fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.281+0000 7f4388444740 1 -- 172.21.15.107:0/2079999284 shutdown_connections 2024-07-26T12:43:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.281+0000 7f4388444740 1 -- 172.21.15.107:0/2079999284 wait complete. 2024-07-26T12:43:56.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.281+0000 7f4388444740 20 client.24812 trim_cache size 0 max 16384 2024-07-26T12:43:56.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.300+0000 7f4388444740 1 Processor -- start 2024-07-26T12:43:56.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.300+0000 7f4388444740 1 -- start start 2024-07-26T12:43:56.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f4388444740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 0x55a3121f9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f4388444740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a312212f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f4388444740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a3122134c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f4388444740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a3122174b0 con 0x55a311e55f60 2024-07-26T12:43:56.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f4388444740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a312217b80 con 0x55a311e35280 2024-07-26T12:43:56.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f4388444740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a312218250 con 0x55a311e2d0a0 2024-07-26T12:43:56.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f437b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a3122134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f437b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a3122134c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48412/0 (socket says 172.21.15.107:48412) 2024-07-26T12:43:56.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f437b400640 1 -- 172.21.15.107:0/60768685 learned_addr learned my addr 172.21.15.107:0/60768685 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:56.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f437aa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 0x55a3121f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.301+0000 7f437a000640 1 --2- 172.21.15.107:0/60768685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a312212f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f437b400640 1 -- 172.21.15.107:0/60768685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 msgr2=0x55a3121f9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:43:56.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f437b400640 1 --2- 172.21.15.107:0/60768685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 0x55a3121f9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f437b400640 1 -- 172.21.15.107:0/60768685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 msgr2=0x55a312212f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:43:56.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f437b400640 1 --2- 172.21.15.107:0/60768685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a312212f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f437b400640 1 -- 172.21.15.107:0/60768685 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3121fde00 con 0x55a311e55f60 2024-07-26T12:43:56.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f437aa00640 1 --2- 172.21.15.107:0/60768685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 0x55a3121f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:56.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f437a000640 1 --2- 172.21.15.107:0/60768685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a312212f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:43:56.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f437b400640 1 --2- 172.21.15.107:0/60768685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a3122134c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f43740a8310 tx=0x7f43740aafa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d445b79f0920db30 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:56.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f4367400640 1 -- 172.21.15.107:0/60768685 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43740ca070 con 0x55a311e55f60 2024-07-26T12:43:56.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f4367400640 1 -- 172.21.15.107:0/60768685 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43740aabb0 con 0x55a311e55f60 2024-07-26T12:43:56.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.302+0000 7f4367400640 1 -- 172.21.15.107:0/60768685 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43740c4030 con 0x55a311e55f60 2024-07-26T12:43:56.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.303+0000 7f4388444740 1 -- 172.21.15.107:0/60768685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 msgr2=0x55a3122134c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:56.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.303+0000 7f4388444740 1 --2- 172.21.15.107:0/60768685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a3122134c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f43740a8310 tx=0x7f43740aafa0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.303+0000 7f4388444740 1 -- 172.21.15.107:0/60768685 shutdown_connections 2024-07-26T12:43:56.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.303+0000 7f4388444740 1 --2- 172.21.15.107:0/60768685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a3122134c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.303+0000 7f4388444740 1 --2- 172.21.15.107:0/60768685 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a312212f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.303+0000 7f4388444740 1 --2- 172.21.15.107:0/60768685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 0x55a3121f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.303+0000 7f4388444740 1 -- 172.21.15.107:0/60768685 >> 172.21.15.107:0/60768685 conn(0x55a311e4a350 msgr2=0x55a3121f8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.303+0000 7f4388444740 1 -- 172.21.15.107:0/60768685 shutdown_connections 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.303+0000 7f4388444740 1 -- 172.21.15.107:0/60768685 wait complete. 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.304+0000 7f4388444740 1 -- start start 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.304+0000 7f4388444740 1 Processor -- start 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f4388444740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 0x55a312221970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f4388444740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a31221ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f4388444740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a31221b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f4388444740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a312218ff0 con 0x55a311e55f60 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f4388444740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55a312218920 con 0x55a311e35280 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f4388444740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55a3122196c0 con 0x55a311e2d0a0 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f4388444740 10 client.0 _ll_register_callbacks cb 0x7f436c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:43:56.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f437b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a31221b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f437b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a31221b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48414/0 (socket says 172.21.15.107:48414) 2024-07-26T12:43:56.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f437b400640 1 -- 172.21.15.107:0/2642089883 learned_addr learned my addr 172.21.15.107:0/2642089883 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:56.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f437aa00640 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 0x55a312221970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.305+0000 7f437a000640 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a31221ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f437aa00640 1 -- 172.21.15.107:0/2642089883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 msgr2=0x55a31221ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:43:56.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f437aa00640 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a31221ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f437aa00640 1 -- 172.21.15.107:0/2642089883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 msgr2=0x55a31221b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f437aa00640 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a31221b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f437aa00640 1 -- 172.21.15.107:0/2642089883 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a31221b970 con 0x55a311e2d0a0 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f437aa00640 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 0x55a312221970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f436c005090 tx=0x7f436c005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebefb3b8dc4562a0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f4364c00640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f4364c00640 1 -- 172.21.15.107:0/2642089883 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c035040 con 0x55a311e2d0a0 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f4388444740 1 -- 172.21.15.107:0/2642089883 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43740aabb0 con 0x55a311e2d0a0 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f4364c00640 1 -- 172.21.15.107:0/2642089883 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f436c031a60 con 0x55a311e2d0a0 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.306+0000 7f4364c00640 1 -- 172.21.15.107:0/2642089883 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f436c06d050 con 0x55a311e2d0a0 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.307+0000 7f4388444740 10 client.24835 fetch_fsmap learned FSMap version 16 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.307+0000 7f4388444740 1 -- 172.21.15.107:0/2642089883 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x55a312221380 con 0x55a311e2d0a0 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.307+0000 7f4364c00640 1 -- 172.21.15.107:0/2642089883 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43740aabb0 con 0x55a311e2d0a0 2024-07-26T12:43:56.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.307+0000 7f4364c00640 1 -- 172.21.15.107:0/2642089883 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 16) ==== 8350+0+0 (secure 0 0 0) 0x7f436c0050c0 con 0x55a311e2d0a0 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.307+0000 7f4388444740 10 client.24835 fetch_fsmap finished waiting for FSMap version 16 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.307+0000 7f4388444740 10 client.24835 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.307+0000 7f4388444740 20 client.24835 populate_metadata read hostname 'smithi107' 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.307+0000 7f4388444740 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55a311de9c90 0x55a312285aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.307+0000 7f4388444740 4 client.24835 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.307+0000 7f4388444740 1 -- 172.21.15.107:0/2642089883 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "status"}) -- 0x55a31216f360 con 0x55a311de9c90 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.308+0000 7f437a000640 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55a311de9c90 0x55a312285aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.308+0000 7f437a000640 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55a311de9c90 0x55a312285aa0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=349df36c09395a7d server_cookie=6fd7d5a846bab72d in_seq=0 out_seq=0 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.308+0000 7f4364c00640 10 client.24835 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.308+0000 7f4364c00640 1 -- 172.21.15.107:0/2642089883 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a31216f360 con 0x55a311de9c90 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.308+0000 7f4364c00640 10 client.24835 handle_command_reply: tid=0 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.308+0000 7f4388444740 1 client.24835 shutdown 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 1 -- 172.21.15.107:0/2642089883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 msgr2=0x55a312221970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:56.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 0x55a312221970 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f436c005090 tx=0x7f436c005e20 comp rx=0 tx=0).stop 2024-07-26T12:43:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 1 -- 172.21.15.107:0/2642089883 shutdown_connections 2024-07-26T12:43:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55a311de9c90 0x55a312285aa0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a311e55f60 0x55a31221b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55a311e35280 0x55a31221ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 1 --2- 172.21.15.107:0/2642089883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55a311e2d0a0 0x55a312221970 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4364c00640 0 client.24835 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:43:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 1 -- 172.21.15.107:0/2642089883 >> 172.21.15.107:0/2642089883 conn(0x55a311e4a350 msgr2=0x55a3121fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 1 -- 172.21.15.107:0/2642089883 shutdown_connections 2024-07-26T12:43:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 1 -- 172.21.15.107:0/2642089883 wait complete. 2024-07-26T12:43:56.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.309+0000 7f4388444740 20 client.24835 trim_cache size 0 max 16384 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24716, 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 40.113460531999998, 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 16, 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 88, 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 85, 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 59.313187372999998 2024-07-26T12:43:56.315 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:43:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 -- 172.21.15.107:0/483252228 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f43600b3c20 msgr2=0x7f43600b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f43600b3c20 0x7f43600b6130 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f43680095e0 tx=0x7f4368005f90 comp rx=0 tx=0).stop 2024-07-26T12:43:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 -- 172.21.15.107:0/483252228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f437c107660 msgr2=0x7f437c1a5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f437c107660 0x7f437c1a5d60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f437400ec50 tx=0x7f437400c6a0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 -- 172.21.15.107:0/483252228 shutdown_connections 2024-07-26T12:43:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f43600b3c20 0x7f43600b6130 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f437c107660 0x7f437c1a5d60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f437c102ba0 0x7f437c1a0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 --2- 172.21.15.107:0/483252228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f437c102250 0x7f437c1a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:56.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 -- 172.21.15.107:0/483252228 >> 172.21.15.107:0/483252228 conn(0x7f437c079190 msgr2=0x7f437c105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:56.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 -- 172.21.15.107:0/483252228 shutdown_connections 2024-07-26T12:43:56.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:56.310+0000 7f4381800640 1 -- 172.21.15.107:0/483252228 wait complete. 2024-07-26T12:43:57.258 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T12:43:57.258 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-07-26T12:43:57.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs ls 2024-07-26T12:43:57.752 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:43:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:57 smithi107 ceph-mon[32659]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:57 smithi132 ceph-mon[37580]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:57 smithi112 ceph-mon[37501]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:43:59.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.931+0000 7f6a53e00640 1 -- 172.21.15.107:0/3095364396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a5410f330 msgr2=0x7f6a5410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:59.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.931+0000 7f6a53e00640 1 --2- 172.21.15.107:0/3095364396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a5410f330 0x7f6a5410f740 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009900 tx=0x7f6a3c02a410 comp rx=0 tx=0).stop 2024-07-26T12:43:59.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.931+0000 7f6a53e00640 1 -- 172.21.15.107:0/3095364396 shutdown_connections 2024-07-26T12:43:59.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.931+0000 7f6a53e00640 1 --2- 172.21.15.107:0/3095364396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a5410fc80 0x7f6a541164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:59.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.931+0000 7f6a53e00640 1 --2- 172.21.15.107:0/3095364396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a5410f330 0x7f6a5410f740 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:59.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.931+0000 7f6a53e00640 1 --2- 172.21.15.107:0/3095364396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a541099f0 0x7f6a54109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:59.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.931+0000 7f6a53e00640 1 -- 172.21.15.107:0/3095364396 >> 172.21.15.107:0/3095364396 conn(0x7f6a54078ff0 msgr2=0x7f6a54107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:43:59.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.931+0000 7f6a53e00640 1 -- 172.21.15.107:0/3095364396 shutdown_connections 2024-07-26T12:43:59.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.931+0000 7f6a53e00640 1 -- 172.21.15.107:0/3095364396 wait complete. 2024-07-26T12:43:59.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.932+0000 7f6a53e00640 1 Processor -- start 2024-07-26T12:43:59.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.932+0000 7f6a53e00640 1 -- start start 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a53e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a541099f0 0x7f6a541a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a53e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a5410f330 0x7f6a541a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a53e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a5410fc80 0x7f6a541a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a53e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6a54119940 con 0x7f6a541099f0 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a53e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f6a541197c0 con 0x7f6a5410fc80 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a53e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f6a54119ac0 con 0x7f6a5410f330 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a52a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a541099f0 0x7f6a541a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a52a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a541099f0 0x7f6a541a0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:48428/0 (socket says 172.21.15.107:48428) 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a52a00640 1 -- 172.21.15.107:0/990799917 learned_addr learned my addr 172.21.15.107:0/990799917 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a4be00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a5410f330 0x7f6a541a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.933+0000 7f6a53400640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a5410fc80 0x7f6a541a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a52a00640 1 -- 172.21.15.107:0/990799917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a5410f330 msgr2=0x7f6a541a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a52a00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a5410f330 0x7f6a541a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a52a00640 1 -- 172.21.15.107:0/990799917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a5410fc80 msgr2=0x7f6a541a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:43:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a52a00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a5410fc80 0x7f6a541a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:43:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a52a00640 1 -- 172.21.15.107:0/990799917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a541a62a0 con 0x7f6a541099f0 2024-07-26T12:43:59.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a52a00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a541099f0 0x7f6a541a0330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6a4400e9f0 tx=0x7f6a4400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a4b400640 1 -- 172.21.15.107:0/990799917 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a4400cd80 con 0x7f6a541099f0 2024-07-26T12:43:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a53e00640 1 -- 172.21.15.107:0/990799917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a541a6590 con 0x7f6a541099f0 2024-07-26T12:43:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a4b400640 1 -- 172.21.15.107:0/990799917 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a440108e0 con 0x7f6a541099f0 2024-07-26T12:43:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a4b400640 1 -- 172.21.15.107:0/990799917 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a44018980 con 0x7f6a541099f0 2024-07-26T12:43:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.934+0000 7f6a53e00640 1 -- 172.21.15.107:0/990799917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a541accf0 con 0x7f6a541099f0 2024-07-26T12:43:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.936+0000 7f6a53e00640 1 -- 172.21.15.107:0/990799917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a54109e00 con 0x7f6a541099f0 2024-07-26T12:43:59.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.939+0000 7f6a4b400640 1 -- 172.21.15.107:0/990799917 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f6a44010430 con 0x7f6a541099f0 2024-07-26T12:43:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.940+0000 7f6a4b400640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6a380b3c20 0x7f6a380b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:43:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.941+0000 7f6a4b400640 1 -- 172.21.15.107:0/990799917 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f6a440e48c0 con 0x7f6a541099f0 2024-07-26T12:43:59.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.941+0000 7f6a4be00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6a380b3c20 0x7f6a380b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:43:59.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.941+0000 7f6a4be00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6a380b3c20 0x7f6a380b6130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009230 tx=0x7f6a3c031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:43:59.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:43:59.941+0000 7f6a4b400640 1 -- 172.21.15.107:0/990799917 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6a440ab430 con 0x7f6a541099f0 2024-07-26T12:44:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:43:59 smithi107 ceph-mon[32659]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:43:59 smithi132 ceph-mon[37580]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:00.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.167+0000 7f6a53e00640 1 -- 172.21.15.107:0/990799917 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6a5402e0c0 con 0x7f6a541099f0 2024-07-26T12:44:00.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.169+0000 7f6a4b400640 1 -- 172.21.15.107:0/990799917 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7f6a440af980 con 0x7f6a541099f0 2024-07-26T12:44:00.171 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-07-26T12:44:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.172+0000 7f6a53e00640 1 -- 172.21.15.107:0/990799917 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6a380b3c20 msgr2=0x7f6a380b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.172+0000 7f6a53e00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6a380b3c20 0x7f6a380b6130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009230 tx=0x7f6a3c031400 comp rx=0 tx=0).stop 2024-07-26T12:44:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.172+0000 7f6a53e00640 1 -- 172.21.15.107:0/990799917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a541099f0 msgr2=0x7f6a541a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.172+0000 7f6a53e00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a541099f0 0x7f6a541a0330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6a4400e9f0 tx=0x7f6a4400eeb0 comp rx=0 tx=0).stop 2024-07-26T12:44:00.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.172+0000 7f6a53e00640 1 -- 172.21.15.107:0/990799917 shutdown_connections 2024-07-26T12:44:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.172+0000 7f6a53e00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f6a380b3c20 0x7f6a380b6130 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.172+0000 7f6a53e00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f6a5410fc80 0x7f6a541a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.172+0000 7f6a53e00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6a5410f330 0x7f6a541a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.172+0000 7f6a53e00640 1 --2- 172.21.15.107:0/990799917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6a541099f0 0x7f6a541a0330 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.172+0000 7f6a53e00640 1 -- 172.21.15.107:0/990799917 >> 172.21.15.107:0/990799917 conn(0x7f6a54078ff0 msgr2=0x7f6a54107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.173+0000 7f6a53e00640 1 -- 172.21.15.107:0/990799917 shutdown_connections 2024-07-26T12:44:00.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:00.173+0000 7f6a53e00640 1 -- 172.21.15.107:0/990799917 wait complete. 2024-07-26T12:44:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:43:59 smithi112 ceph-mon[37501]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:00.946 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:00 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/990799917' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T12:44:01.004 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T12:44:01.004 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T12:44:01.004 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2024-07-26T12:44:01.004 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-07-26T12:44:01.004 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-07-26T12:44:01.004 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T12:44:01.004 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T12:44:01.005 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T12:44:01.005 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-07-26T12:44:01.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:01.005 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2024-07-26T12:44:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:00 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/990799917' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T12:44:01.060 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T12:44:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T12:44:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T12:44:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-07-26T12:44:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2024-07-26T12:44:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-07-26T12:44:01.060 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T12:44:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41971sec preferred_lft 41971sec 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2024-07-26T12:44:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-07-26T12:44:01.063 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T12:44:01.063 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T12:44:01.063 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-07-26T12:44:01.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add name ceph-brx type bridge 2024-07-26T12:44:01.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr flush dev ceph-brx 2024-07-26T12:44:01.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set ceph-brx up 2024-07-26T12:44:01.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T12:44:01.064 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-07-26T12:44:01.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:44:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:00 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/990799917' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T12:44:01.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:44:01.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:01.553 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T12:44:01.623 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-07-26T12:44:01.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:01.628 DEBUG:teuthology.orchestra.run.smithi107:> route 2024-07-26T12:44:01.680 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2024-07-26T12:44:01.680 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T12:44:01.680 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T12:44:01.680 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T12:44:01.681 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T12:44:01.682 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T12:44:01.682 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-07-26T12:44:01.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T12:44:01.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T12:44:01.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T12:44:01.682 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-07-26T12:44:01.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:44:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:01 smithi107 ceph-mon[32659]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:01 smithi132 ceph-mon[37580]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:01 smithi112 ceph-mon[37501]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:02.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:44:02.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:02.445 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-07-26T12:44:02.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:02.493 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list-id 2024-07-26T12:44:02.538 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T12:44:02.538 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-07-26T12:44:02.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T12:44:02.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-07-26T12:44:02.538 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-07-26T12:44:02.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:44:02.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:44:02.769 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-07-26T12:44:02.769 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T12:44:02.769 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-07-26T12:44:02.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-07-26T12:44:02.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T12:44:02.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-07-26T12:44:02.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-07-26T12:44:02.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-07-26T12:44:02.770 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-07-26T12:44:02.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:44:03.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:44:03.064 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T12:44:03.064 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-07-26T12:44:03.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2024-07-26T12:44:03.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T12:44:03.064 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-07-26T12:44:03.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:44:03.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:44:03.260 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-07-26T12:44:03.260 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T12:44:03.260 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-07-26T12:44:03.303 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-07-26T12:44:03.303 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T12:44:03.303 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-07-26T12:44:03.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2024-07-26T12:44:03.463 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T12:44:03.510 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2024-07-26T12:44:03.510 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2024-07-26T12:44:03.510 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2024-07-26T12:44:03.510 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/selinux 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2024-07-26T12:44:03.511 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/59ab343bbb50a4bf5387126900800078d112a4e9df7d3bd33f4356ac164d928b/merged 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/b3695a838ec21094d58f7bc27df98797f0072e5bb84250ebb315c7562a06e803/merged 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/0 2024-07-26T12:44:03.512 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/308b6483c01f4fdf62b720744caee568aa9f82beac51d39d31fe2c3e951c9a2d/merged 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/8a98b2f2cc2a6cc52756c3f2c7441906320c514b80e69a201dc69ba727d151fb/merged 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/a3ccaa206b93c77d87e9dcab20ba7f279cf93dec215b0553e37d3510f9df333e/merged 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/988cb6108666691352afc530941248ca5f91a222a404a058d0f09f7a43710918/merged 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/45028f795b88901767280ddc5493cc74849a77c93a086fba043a660f4ba522a1/merged 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/2e9eaef4784f53415c246c9f41c9d5d8398fbd9075900466045d1b8e3bbdbfb4/merged 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/4d8b28aea637f4c0ab6c1e0d231b9b74fce5732d3bbc9c70a4787c1208c4c8df/merged 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/c7aabdbba75a16a2d2d560fadf206265d702c6d7dee967c50297281b38f5ece7/merged 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T12:44:03.513 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T12:44:03.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:03.514 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2024-07-26T12:44:03.559 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-07-26T12:44:03.560 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-07-26T12:44:03.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2024-07-26T12:44:03.674 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T12:44:03.715 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2024-07-26T12:44:03.715 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2024-07-26T12:44:03.715 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2024-07-26T12:44:03.715 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2024-07-26T12:44:03.715 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2024-07-26T12:44:03.715 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2024-07-26T12:44:03.715 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/selinux 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T12:44:03.716 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/59ab343bbb50a4bf5387126900800078d112a4e9df7d3bd33f4356ac164d928b/merged 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/b3695a838ec21094d58f7bc27df98797f0072e5bb84250ebb315c7562a06e803/merged 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/0 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/308b6483c01f4fdf62b720744caee568aa9f82beac51d39d31fe2c3e951c9a2d/merged 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/8a98b2f2cc2a6cc52756c3f2c7441906320c514b80e69a201dc69ba727d151fb/merged 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/a3ccaa206b93c77d87e9dcab20ba7f279cf93dec215b0553e37d3510f9df333e/merged 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/988cb6108666691352afc530941248ca5f91a222a404a058d0f09f7a43710918/merged 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/45028f795b88901767280ddc5493cc74849a77c93a086fba043a660f4ba522a1/merged 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/2e9eaef4784f53415c246c9f41c9d5d8398fbd9075900466045d1b8e3bbdbfb4/merged 2024-07-26T12:44:03.717 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/4d8b28aea637f4c0ab6c1e0d231b9b74fce5732d3bbc9c70a4787c1208c4c8df/merged 2024-07-26T12:44:03.718 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/c7aabdbba75a16a2d2d560fadf206265d702c6d7dee967c50297281b38f5ece7/merged 2024-07-26T12:44:03.718 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2024-07-26T12:44:03.718 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T12:44:03.718 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T12:44:03.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:03.719 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2024-07-26T12:44:03.882 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:2024-07-26T12:44:03.880+0000 7f6f1afc1480 -1 init, newargv = 0x7f6efc004360 newargc=15 2024-07-26T12:44:03.882 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:ceph-fuse[81296]: starting ceph client 2024-07-26T12:44:03.893 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:ceph-fuse[81296]: starting fuse 2024-07-26T12:44:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:03 smithi107 ceph-mon[32659]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:03 smithi132 ceph-mon[37580]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:03 smithi112 ceph-mon[37501]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:04.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo modprobe fuse 2024-07-26T12:44:04.995 DEBUG:teuthology.orchestra.run.smithi107:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T12:44:05.050 INFO:teuthology.orchestra.run.smithi107.stdout:/proc 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/sys 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/dev 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/security 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/shm 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/pts 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/run 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/cgroup 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/pstore 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/bpf 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/ 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/selinux 2024-07-26T12:44:05.051 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/mqueue 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/debug 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/hugepages 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/tracing 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/fs/fuse/connections 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/sys/kernel/config 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/1000 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/ceph 2024-07-26T12:44:05.052 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/59ab343bbb50a4bf5387126900800078d112a4e9df7d3bd33f4356ac164d928b/merged 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/b3695a838ec21094d58f7bc27df98797f0072e5bb84250ebb315c7562a06e803/merged 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/run/user/0 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/308b6483c01f4fdf62b720744caee568aa9f82beac51d39d31fe2c3e951c9a2d/merged 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/8a98b2f2cc2a6cc52756c3f2c7441906320c514b80e69a201dc69ba727d151fb/merged 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/a3ccaa206b93c77d87e9dcab20ba7f279cf93dec215b0553e37d3510f9df333e/merged 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/988cb6108666691352afc530941248ca5f91a222a404a058d0f09f7a43710918/merged 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/45028f795b88901767280ddc5493cc74849a77c93a086fba043a660f4ba522a1/merged 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/2e9eaef4784f53415c246c9f41c9d5d8398fbd9075900466045d1b8e3bbdbfb4/merged 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/4d8b28aea637f4c0ab6c1e0d231b9b74fce5732d3bbc9c70a4787c1208c4c8df/merged 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/var/lib/containers/storage/overlay/c7aabdbba75a16a2d2d560fadf206265d702c6d7dee967c50297281b38f5ece7/merged 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns 2024-07-26T12:44:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T12:44:05.054 INFO:teuthology.orchestra.run.smithi107.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T12:44:05.054 INFO:teuthology.orchestra.run.smithi107.stdout:/home/ubuntu/cephtest/mnt.admin 2024-07-26T12:44:05.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:05.055 DEBUG:teuthology.orchestra.run.smithi107:> ls /sys/fs/fuse/connections 2024-07-26T12:44:05.106 INFO:teuthology.orchestra.run.smithi107.stdout:119 2024-07-26T12:44:05.107 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-07-26T12:44:05.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo stdin-killer -- python3 -c ' 2024-07-26T12:44:05.107 DEBUG:teuthology.orchestra.run.smithi107:> import glob 2024-07-26T12:44:05.107 DEBUG:teuthology.orchestra.run.smithi107:> import re 2024-07-26T12:44:05.107 DEBUG:teuthology.orchestra.run.smithi107:> import os 2024-07-26T12:44:05.107 DEBUG:teuthology.orchestra.run.smithi107:> import subprocess 2024-07-26T12:44:05.107 DEBUG:teuthology.orchestra.run.smithi107:> 2024-07-26T12:44:05.107 DEBUG:teuthology.orchestra.run.smithi107:> def _find_admin_socket(client_name): 2024-07-26T12:44:05.107 DEBUG:teuthology.orchestra.run.smithi107:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-07-26T12:44:05.107 DEBUG:teuthology.orchestra.run.smithi107:> files = glob.glob(asok_path) 2024-07-26T12:44:05.107 DEBUG:teuthology.orchestra.run.smithi107:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> # Given a non-glob path, it better be there 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> if "*" not in asok_path: 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> assert(len(files) == 1) 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> return files[0] 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> for f in files: 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> if os.path.exists("/proc/{0}".format(pid)): 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> contents = proc_f.read() 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> if mountpoint in contents: 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> return f 2024-07-26T12:44:05.108 DEBUG:teuthology.orchestra.run.smithi107:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-07-26T12:44:05.109 DEBUG:teuthology.orchestra.run.smithi107:> 2024-07-26T12:44:05.109 DEBUG:teuthology.orchestra.run.smithi107:> print(_find_admin_socket("client.admin")) 2024-07-26T12:44:05.109 DEBUG:teuthology.orchestra.run.smithi107:> ' 2024-07-26T12:44:05.271 INFO:teuthology.orchestra.run.smithi107.stdout:/var/run/ceph/ceph-client.admin.81296.asok 2024-07-26T12:44:05.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:44:05.288 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.81296.asok 2024-07-26T12:44:05.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:05.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.81296.asok status 2024-07-26T12:44:05.596 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:44:05.596 INFO:teuthology.orchestra.run.smithi107.stdout: "metadata": { 2024-07-26T12:44:05.596 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_sha1": "e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528", 2024-07-26T12:44:05.596 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T12:44:05.596 INFO:teuthology.orchestra.run.smithi107.stdout: "entity_id": "admin", 2024-07-26T12:44:05.596 INFO:teuthology.orchestra.run.smithi107.stdout: "hostname": "smithi107", 2024-07-26T12:44:05.596 INFO:teuthology.orchestra.run.smithi107.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-07-26T12:44:05.596 INFO:teuthology.orchestra.run.smithi107.stdout: "pid": "81296", 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "root": "/" 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_count": 0, 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "dentry_pinned_count": 0, 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 14691, 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "inst": { 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "name": { 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "client", 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "num": 14691 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-07-26T12:44:05.597 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 567253839 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": { 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: "type": "v1", 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: "addr": "192.168.0.1:0", 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: "nonce": 567253839 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: "inst_str": "client.14691 192.168.0.1:0/567253839", 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: "addr_str": "192.168.0.1:0/567253839", 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: "inode_count": 1, 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_epoch": 16, 2024-07-26T12:44:05.598 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch": 88, 2024-07-26T12:44:05.599 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_epoch_barrier": 0, 2024-07-26T12:44:05.599 INFO:teuthology.orchestra.run.smithi107.stdout: "blocklisted": false, 2024-07-26T12:44:05.599 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs" 2024-07-26T12:44:05.599 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:44:05.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:05.618 DEBUG:teuthology.orchestra.run.smithi107:> stat --file-system '--printf=%T 2024-07-26T12:44:05.618 DEBUG:teuthology.orchestra.run.smithi107:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-07-26T12:44:05.673 INFO:teuthology.orchestra.run.smithi107.stdout:fuseblk 2024-07-26T12:44:05.674 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-07-26T12:44:05.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:05.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-07-26T12:44:05.758 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-07-26T12:44:05.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T12:44:05.900 INFO:teuthology.orchestra.run.smithi107.stdout:# file: . 2024-07-26T12:44:05.900 INFO:teuthology.orchestra.run.smithi107.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-07-26T12:44:05.900 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:44:05.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:44:05.911 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:44:05.911 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T12:44:05.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd blocklist ls 2024-07-26T12:44:06.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:05 smithi107 ceph-mon[32659]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:05 smithi132 ceph-mon[37580]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:05 smithi112 ceph-mon[37501]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:06.454 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:06.893 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:07.071 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:07 smithi132 ceph-mon[37580]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:08.152 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:07 smithi107 ceph-mon[32659]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:08.152 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:08.152 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:07 smithi112 ceph-mon[37501]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:08.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.737+0000 7fd5e7e00640 1 -- 172.21.15.107:0/1021499217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e810af10 msgr2=0x7fd5e810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.737+0000 7fd5e7e00640 1 --2- 172.21.15.107:0/1021499217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e810af10 0x7fd5e810b320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009900 tx=0x7fd5d002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:44:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.737+0000 7fd5e7e00640 1 -- 172.21.15.107:0/1021499217 shutdown_connections 2024-07-26T12:44:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.737+0000 7fd5e7e00640 1 --2- 172.21.15.107:0/1021499217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e810b860 0x7fd5e8112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.737+0000 7fd5e7e00640 1 --2- 172.21.15.107:0/1021499217 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e810af10 0x7fd5e810b320 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.737+0000 7fd5e7e00640 1 --2- 172.21.15.107:0/1021499217 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5e81055d0 0x7fd5e81059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.737+0000 7fd5e7e00640 1 -- 172.21.15.107:0/1021499217 >> 172.21.15.107:0/1021499217 conn(0x7fd5e8101040 msgr2=0x7fd5e8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.737+0000 7fd5e7e00640 1 -- 172.21.15.107:0/1021499217 shutdown_connections 2024-07-26T12:44:08.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.737+0000 7fd5e7e00640 1 -- 172.21.15.107:0/1021499217 wait complete. 2024-07-26T12:44:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.738+0000 7fd5e7e00640 1 Processor -- start 2024-07-26T12:44:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.739+0000 7fd5e7e00640 1 -- start start 2024-07-26T12:44:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.739+0000 7fd5e7e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e81055d0 0x7fd5e819bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.739+0000 7fd5e7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5e810af10 0x7fd5e819c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.739+0000 7fd5e7e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e810b860 0x7fd5e81a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.739+0000 7fd5e7e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e81154d0 con 0x7fd5e81055d0 2024-07-26T12:44:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.739+0000 7fd5e7e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e8115350 con 0x7fd5e810b860 2024-07-26T12:44:08.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.739+0000 7fd5e7e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e8115650 con 0x7fd5e810af10 2024-07-26T12:44:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.739+0000 7fd5e6a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e81055d0 0x7fd5e819bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.739+0000 7fd5e6a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e81055d0 0x7fd5e819bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35704/0 (socket says 172.21.15.107:35704) 2024-07-26T12:44:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.739+0000 7fd5e6a00640 1 -- 172.21.15.107:0/3161200129 learned_addr learned my addr 172.21.15.107:0/3161200129 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.740+0000 7fd5e7400640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e810b860 0x7fd5e81a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:08.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.740+0000 7fd5dfe00640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5e810af10 0x7fd5e819c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:08.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.740+0000 7fd5e7400640 1 -- 172.21.15.107:0/3161200129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5e810af10 msgr2=0x7fd5e819c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:08.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.740+0000 7fd5e7400640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5e810af10 0x7fd5e819c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:08.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.740+0000 7fd5e7400640 1 -- 172.21.15.107:0/3161200129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e81055d0 msgr2=0x7fd5e819bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:08.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.740+0000 7fd5e7400640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e81055d0 0x7fd5e819bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:08.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.740+0000 7fd5e7400640 1 -- 172.21.15.107:0/3161200129 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e81a1e70 con 0x7fd5e810b860 2024-07-26T12:44:08.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.740+0000 7fd5e6a00640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e81055d0 0x7fd5e819bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:44:08.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.740+0000 7fd5e7400640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e810b860 0x7fd5e81a1740 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009fd0 tx=0x7fd5e000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.741+0000 7fd5df400640 1 -- 172.21.15.107:0/3161200129 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e000ce20 con 0x7fd5e810b860 2024-07-26T12:44:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.741+0000 7fd5e7e00640 1 -- 172.21.15.107:0/3161200129 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e81a2160 con 0x7fd5e810b860 2024-07-26T12:44:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.741+0000 7fd5df400640 1 -- 172.21.15.107:0/3161200129 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5e0005020 con 0x7fd5e810b860 2024-07-26T12:44:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.741+0000 7fd5df400640 1 -- 172.21.15.107:0/3161200129 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0018670 con 0x7fd5e810b860 2024-07-26T12:44:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.741+0000 7fd5e7e00640 1 -- 172.21.15.107:0/3161200129 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e81a88c0 con 0x7fd5e810b860 2024-07-26T12:44:08.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.742+0000 7fd5e7e00640 1 -- 172.21.15.107:0/3161200129 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e81059e0 con 0x7fd5e810b860 2024-07-26T12:44:08.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.745+0000 7fd5df400640 1 -- 172.21.15.107:0/3161200129 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fd5e00054d0 con 0x7fd5e810b860 2024-07-26T12:44:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.746+0000 7fd5df400640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5c80b3c20 0x7fd5c80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.747+0000 7fd5df400640 1 -- 172.21.15.107:0/3161200129 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fd5e00e48a0 con 0x7fd5e810b860 2024-07-26T12:44:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.747+0000 7fd5e6a00640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5c80b3c20 0x7fd5c80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.748+0000 7fd5e6a00640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5c80b3c20 0x7fd5c80b6130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8011550 tx=0x7fd5d8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.748+0000 7fd5df400640 1 -- 172.21.15.107:0/3161200129 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd5e0054600 con 0x7fd5e810b860 2024-07-26T12:44:08.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.967+0000 7fd5e7e00640 1 -- 172.21.15.107:0/3161200129 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd5e819d030 con 0x7fd5e810b860 2024-07-26T12:44:08.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.967+0000 7fd5df400640 1 -- 172.21.15.107:0/3161200129 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v88) ==== 81+0+894 (secure 0 0 0) 0x7fd5e00abaf0 con 0x7fd5e810b860 2024-07-26T12:44:08.970 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/3311144690 2024-07-27T12:35:34.485438+0000 2024-07-26T12:44:08.970 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/81377212 2024-07-27T12:35:34.485438+0000 2024-07-26T12:44:08.970 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/634100778 2024-07-27T12:33:46.111763+0000 2024-07-26T12:44:08.970 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4257991258 2024-07-27T12:35:34.485438+0000 2024-07-26T12:44:08.970 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/2773605049 2024-07-27T12:35:34.485438+0000 2024-07-26T12:44:08.970 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/312154060 2024-07-27T12:33:46.111763+0000 2024-07-26T12:44:08.970 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1797109211 2024-07-27T12:33:46.111763+0000 2024-07-26T12:44:08.971 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/670053235 2024-07-27T12:34:14.657397+0000 2024-07-26T12:44:08.971 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/453757265 2024-07-27T12:33:46.111763+0000 2024-07-26T12:44:08.971 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/2773605049 2024-07-27T12:35:34.485438+0000 2024-07-26T12:44:08.971 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/425164471 2024-07-27T12:34:14.657397+0000 2024-07-26T12:44:08.971 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4171514587 2024-07-27T12:34:14.657397+0000 2024-07-26T12:44:08.971 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/634100778 2024-07-27T12:33:46.111763+0000 2024-07-26T12:44:08.971 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/425164471 2024-07-27T12:34:14.657397+0000 2024-07-26T12:44:08.971 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2624080873 2024-07-27T12:34:14.657397+0000 2024-07-26T12:44:08.971 INFO:teuthology.orchestra.run.smithi107.stderr:listed 15 entries 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.970+0000 7fd5e7e00640 1 -- 172.21.15.107:0/3161200129 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5c80b3c20 msgr2=0x7fd5c80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.970+0000 7fd5e7e00640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5c80b3c20 0x7fd5c80b6130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8011550 tx=0x7fd5d8005f90 comp rx=0 tx=0).stop 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.970+0000 7fd5e7e00640 1 -- 172.21.15.107:0/3161200129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e810b860 msgr2=0x7fd5e81a1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.970+0000 7fd5e7e00640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e810b860 0x7fd5e81a1740 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0009fd0 tx=0x7fd5e000ef10 comp rx=0 tx=0).stop 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.970+0000 7fd5e7e00640 1 -- 172.21.15.107:0/3161200129 shutdown_connections 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.970+0000 7fd5e7e00640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5c80b3c20 0x7fd5c80b6130 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.971+0000 7fd5e7e00640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd5e810b860 0x7fd5e81a1740 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.971+0000 7fd5e7e00640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5e810af10 0x7fd5e819c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.971+0000 7fd5e7e00640 1 --2- 172.21.15.107:0/3161200129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e81055d0 0x7fd5e819bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.971+0000 7fd5e7e00640 1 -- 172.21.15.107:0/3161200129 >> 172.21.15.107:0/3161200129 conn(0x7fd5e8101040 msgr2=0x7fd5e8103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.971+0000 7fd5e7e00640 1 -- 172.21.15.107:0/3161200129 shutdown_connections 2024-07-26T12:44:08.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:08.971+0000 7fd5e7e00640 1 -- 172.21.15.107:0/3161200129 wait complete. 2024-07-26T12:44:08.996 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:08.997 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:08.997 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:08.997 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:10.016 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:09 smithi107 ceph-mon[32659]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:10.016 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:09 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3161200129' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T12:44:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:09 smithi132 ceph-mon[37580]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:09 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3161200129' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T12:44:10.078 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:44:10.079 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T12:44:10.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd blocklist ls 2024-07-26T12:44:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:09 smithi112 ceph-mon[37501]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:09 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3161200129' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T12:44:10.657 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:11 smithi107 ceph-mon[32659]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:11 smithi112 ceph-mon[37501]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:11 smithi132 ceph-mon[37580]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.941+0000 7f9f73200640 1 -- 172.21.15.107:0/2071006838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f741055d0 msgr2=0x7f9f741059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.941+0000 7f9f73200640 1 --2- 172.21.15.107:0/2071006838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f741055d0 0x7f9f741059e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9f64009860 tx=0x7f9f6402a340 comp rx=0 tx=0).stop 2024-07-26T12:44:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.941+0000 7f9f73200640 1 -- 172.21.15.107:0/2071006838 shutdown_connections 2024-07-26T12:44:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.941+0000 7f9f73200640 1 --2- 172.21.15.107:0/2071006838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f7410b860 0x7f9f74112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.941+0000 7f9f73200640 1 --2- 172.21.15.107:0/2071006838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f7410af10 0x7f9f7410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.941+0000 7f9f73200640 1 --2- 172.21.15.107:0/2071006838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f741055d0 0x7f9f741059e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.941+0000 7f9f73200640 1 -- 172.21.15.107:0/2071006838 >> 172.21.15.107:0/2071006838 conn(0x7f9f74101040 msgr2=0x7f9f74103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.941+0000 7f9f73200640 1 -- 172.21.15.107:0/2071006838 shutdown_connections 2024-07-26T12:44:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.941+0000 7f9f73200640 1 -- 172.21.15.107:0/2071006838 wait complete. 2024-07-26T12:44:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.942+0000 7f9f73200640 1 Processor -- start 2024-07-26T12:44:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f73200640 1 -- start start 2024-07-26T12:44:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f73200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f741055d0 0x7f9f7419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f73200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f7410af10 0x7f9f7419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:12.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f73200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f7410b860 0x7f9f741a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f73200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9f74115510 con 0x7f9f7410b860 2024-07-26T12:44:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f73200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9f74115390 con 0x7f9f7410af10 2024-07-26T12:44:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f73200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9f74115690 con 0x7f9f741055d0 2024-07-26T12:44:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f71e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f741055d0 0x7f9f7419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f71e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f741055d0 0x7f9f7419bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:41304/0 (socket says 172.21.15.107:41304) 2024-07-26T12:44:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f71e00640 1 -- 172.21.15.107:0/3401391497 learned_addr learned my addr 172.21.15.107:0/3401391497 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.943+0000 7f9f6be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f7410af10 0x7f9f7419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:12.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.944+0000 7f9f72800640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f7410b860 0x7f9f741a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.944+0000 7f9f71e00640 1 -- 172.21.15.107:0/3401391497 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f7410af10 msgr2=0x7f9f7419c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.944+0000 7f9f71e00640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f7410af10 0x7f9f7419c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.944+0000 7f9f71e00640 1 -- 172.21.15.107:0/3401391497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f7410b860 msgr2=0x7f9f741a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.944+0000 7f9f71e00640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f7410b860 0x7f9f741a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:12.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.944+0000 7f9f71e00640 1 -- 172.21.15.107:0/3401391497 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f741a1e90 con 0x7f9f741055d0 2024-07-26T12:44:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.944+0000 7f9f6be00640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f7410af10 0x7f9f7419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:44:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.944+0000 7f9f71e00640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f741055d0 0x7f9f7419bf20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9f64035040 tx=0x7f9f64032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.945+0000 7f9f6aa00640 1 -- 172.21.15.107:0/3401391497 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64037040 con 0x7f9f741055d0 2024-07-26T12:44:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.945+0000 7f9f6aa00640 1 -- 172.21.15.107:0/3401391497 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f64041070 con 0x7f9f741055d0 2024-07-26T12:44:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.945+0000 7f9f6aa00640 1 -- 172.21.15.107:0/3401391497 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f64003850 con 0x7f9f741055d0 2024-07-26T12:44:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.945+0000 7f9f73200640 1 -- 172.21.15.107:0/3401391497 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f741a2120 con 0x7f9f741055d0 2024-07-26T12:44:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.945+0000 7f9f73200640 1 -- 172.21.15.107:0/3401391497 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f741a2550 con 0x7f9f741055d0 2024-07-26T12:44:12.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.946+0000 7f9f73200640 1 -- 172.21.15.107:0/3401391497 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f741059e0 con 0x7f9f741055d0 2024-07-26T12:44:12.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.949+0000 7f9f6aa00640 1 -- 172.21.15.107:0/3401391497 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f9f640039f0 con 0x7f9f741055d0 2024-07-26T12:44:12.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.950+0000 7f9f6aa00640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9f580b3ca0 0x7f9f580b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:12.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.951+0000 7f9f6aa00640 1 -- 172.21.15.107:0/3401391497 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f9f641044d0 con 0x7f9f741055d0 2024-07-26T12:44:12.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.951+0000 7f9f6be00640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9f580b3ca0 0x7f9f580b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:12.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.951+0000 7f9f6be00640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9f580b3ca0 0x7f9f580b61b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9f7410ab30 tx=0x7f9f5c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:12.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:12.952+0000 7f9f6aa00640 1 -- 172.21.15.107:0/3401391497 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9f640cb710 con 0x7f9f741055d0 2024-07-26T12:44:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:13.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.164+0000 7f9f73200640 1 -- 172.21.15.107:0/3401391497 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9f7410d560 con 0x7f9f741055d0 2024-07-26T12:44:13.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.164+0000 7f9f6aa00640 1 -- 172.21.15.107:0/3401391497 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v88) ==== 81+0+894 (secure 0 0 0) 0x7f9f7410d560 con 0x7f9f741055d0 2024-07-26T12:44:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/3311144690 2024-07-27T12:35:34.485438+0000 2024-07-26T12:44:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/81377212 2024-07-27T12:35:34.485438+0000 2024-07-26T12:44:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/634100778 2024-07-27T12:33:46.111763+0000 2024-07-26T12:44:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4257991258 2024-07-27T12:35:34.485438+0000 2024-07-26T12:44:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/2773605049 2024-07-27T12:35:34.485438+0000 2024-07-26T12:44:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/312154060 2024-07-27T12:33:46.111763+0000 2024-07-26T12:44:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/1797109211 2024-07-27T12:33:46.111763+0000 2024-07-26T12:44:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/670053235 2024-07-27T12:34:14.657397+0000 2024-07-26T12:44:13.167 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/453757265 2024-07-27T12:33:46.111763+0000 2024-07-26T12:44:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/2773605049 2024-07-27T12:35:34.485438+0000 2024-07-26T12:44:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/425164471 2024-07-27T12:34:14.657397+0000 2024-07-26T12:44:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/4171514587 2024-07-27T12:34:14.657397+0000 2024-07-26T12:44:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6800/634100778 2024-07-27T12:33:46.111763+0000 2024-07-26T12:44:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:6801/425164471 2024-07-27T12:34:14.657397+0000 2024-07-26T12:44:13.168 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.15.107:0/2624080873 2024-07-27T12:34:14.657397+0000 2024-07-26T12:44:13.168 INFO:teuthology.orchestra.run.smithi107.stderr:listed 15 entries 2024-07-26T12:44:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.167+0000 7f9f73200640 1 -- 172.21.15.107:0/3401391497 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9f580b3ca0 msgr2=0x7f9f580b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.167+0000 7f9f73200640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9f580b3ca0 0x7f9f580b61b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9f7410ab30 tx=0x7f9f5c005f90 comp rx=0 tx=0).stop 2024-07-26T12:44:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.167+0000 7f9f73200640 1 -- 172.21.15.107:0/3401391497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f741055d0 msgr2=0x7f9f7419bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.167+0000 7f9f73200640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f741055d0 0x7f9f7419bf20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9f64035040 tx=0x7f9f64032d50 comp rx=0 tx=0).stop 2024-07-26T12:44:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.167+0000 7f9f73200640 1 -- 172.21.15.107:0/3401391497 shutdown_connections 2024-07-26T12:44:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.167+0000 7f9f73200640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f9f580b3ca0 0x7f9f580b61b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.167+0000 7f9f73200640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f7410b860 0x7f9f741a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.167+0000 7f9f73200640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9f7410af10 0x7f9f7419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.168+0000 7f9f73200640 1 --2- 172.21.15.107:0/3401391497 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9f741055d0 0x7f9f7419bf20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.168+0000 7f9f73200640 1 -- 172.21.15.107:0/3401391497 >> 172.21.15.107:0/3401391497 conn(0x7f9f74101040 msgr2=0x7f9f74103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:13.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.168+0000 7f9f73200640 1 -- 172.21.15.107:0/3401391497 shutdown_connections 2024-07-26T12:44:13.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:13.168+0000 7f9f73200640 1 -- 172.21.15.107:0/3401391497 wait complete. 2024-07-26T12:44:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:44:13.924 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:13 smithi107 ceph-mon[32659]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:13.924 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:13 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3401391497' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T12:44:13.985 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi107.front.sepia.ceph.com... 2024-07-26T12:44:13.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:13.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-07-26T12:44:14.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi107.stderr:ceph-fuse[81296]: fuse finished with error 0 and tester_r 0 2024-07-26T12:44:14.075 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T12:44:14.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:13 smithi112 ceph-mon[37501]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:14.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:13 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3401391497' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T12:44:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:13 smithi132 ceph-mon[37580]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:13 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3401391497' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T12:44:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:14 smithi112 ceph-mon[37501]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:14 smithi107 ceph-mon[32659]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:14 smithi132 ceph-mon[37580]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:17 smithi107 ceph-mon[32659]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:17 smithi132 ceph-mon[37580]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:17 smithi112 ceph-mon[37501]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T12:44:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:19 smithi107 ceph-mon[32659]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T12:44:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:19 smithi132 ceph-mon[37580]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T12:44:20.178 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi107.front.sepia.ceph.com 2024-07-26T12:44:20.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:44:20.178 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-07-26T12:44:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:19 smithi112 ceph-mon[37501]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T12:44:20.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:44:20.702 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:21 smithi107 ceph-mon[32659]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:21 smithi132 ceph-mon[37580]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:21 smithi112 ceph-mon[37501]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:23.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.116+0000 7f02fe200640 1 -- 172.21.15.107:0/1280831767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f810f330 msgr2=0x7f02f810f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:23.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.116+0000 7f02fe200640 1 --2- 172.21.15.107:0/1280831767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f810f330 0x7f02f810f740 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009900 tx=0x7f02e802a410 comp rx=0 tx=0).stop 2024-07-26T12:44:23.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.116+0000 7f02fe200640 1 -- 172.21.15.107:0/1280831767 shutdown_connections 2024-07-26T12:44:23.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.116+0000 7f02fe200640 1 --2- 172.21.15.107:0/1280831767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f810fc80 0x7f02f81164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:23.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.116+0000 7f02fe200640 1 --2- 172.21.15.107:0/1280831767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f810f330 0x7f02f810f740 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:23.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.116+0000 7f02fe200640 1 --2- 172.21.15.107:0/1280831767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02f81099f0 0x7f02f8109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:23.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.116+0000 7f02fe200640 1 -- 172.21.15.107:0/1280831767 >> 172.21.15.107:0/1280831767 conn(0x7f02f8078ff0 msgr2=0x7f02f8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:23.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.116+0000 7f02fe200640 1 -- 172.21.15.107:0/1280831767 shutdown_connections 2024-07-26T12:44:23.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.116+0000 7f02fe200640 1 -- 172.21.15.107:0/1280831767 wait complete. 2024-07-26T12:44:23.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.117+0000 7f02fe200640 1 Processor -- start 2024-07-26T12:44:23.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.117+0000 7f02fe200640 1 -- start start 2024-07-26T12:44:23.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02fe200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f81099f0 0x7f02f81a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:23.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02fe200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f810f330 0x7f02f81a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:23.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02fe200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02f810fc80 0x7f02f81a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:23.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02fe200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8119940 con 0x7f02f810f330 2024-07-26T12:44:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02fe200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f02f81197c0 con 0x7f02f81099f0 2024-07-26T12:44:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02fe200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f02f8119ac0 con 0x7f02f810fc80 2024-07-26T12:44:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02f7e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f810f330 0x7f02f81a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02fce00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f81099f0 0x7f02f81a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02f7e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f810f330 0x7f02f81a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46934/0 (socket says 172.21.15.107:46934) 2024-07-26T12:44:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02f7e00640 1 -- 172.21.15.107:0/3157784694 learned_addr learned my addr 172.21.15.107:0/3157784694 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.118+0000 7f02fd800640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02f810fc80 0x7f02f81a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.119+0000 7f02fce00640 1 -- 172.21.15.107:0/3157784694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02f810fc80 msgr2=0x7f02f81a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.119+0000 7f02fce00640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02f810fc80 0x7f02f81a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.119+0000 7f02fce00640 1 -- 172.21.15.107:0/3157784694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f810f330 msgr2=0x7f02f81a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.119+0000 7f02fce00640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f810f330 0x7f02f81a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.119+0000 7f02fce00640 1 -- 172.21.15.107:0/3157784694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f81a62a0 con 0x7f02f81099f0 2024-07-26T12:44:23.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.119+0000 7f02fce00640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f81099f0 0x7f02f81a0330 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e9f0 tx=0x7f02ec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:23.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.119+0000 7f02f5600640 1 -- 172.21.15.107:0/3157784694 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec00ccc0 con 0x7f02f81099f0 2024-07-26T12:44:23.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.120+0000 7f02fe200640 1 -- 172.21.15.107:0/3157784694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f81a6590 con 0x7f02f81099f0 2024-07-26T12:44:23.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.120+0000 7f02f5600640 1 -- 172.21.15.107:0/3157784694 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02ec0108e0 con 0x7f02f81099f0 2024-07-26T12:44:23.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.120+0000 7f02f5600640 1 -- 172.21.15.107:0/3157784694 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec018880 con 0x7f02f81099f0 2024-07-26T12:44:23.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.120+0000 7f02fe200640 1 -- 172.21.15.107:0/3157784694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f81accf0 con 0x7f02f81099f0 2024-07-26T12:44:23.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.121+0000 7f02fe200640 1 -- 172.21.15.107:0/3157784694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f8109e00 con 0x7f02f81099f0 2024-07-26T12:44:23.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.123+0000 7f02f5600640 1 -- 172.21.15.107:0/3157784694 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f02ec010430 con 0x7f02f81099f0 2024-07-26T12:44:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.125+0000 7f02f5600640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f02e00b3ca0 0x7f02e00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:23.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.125+0000 7f02f5600640 1 -- 172.21.15.107:0/3157784694 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f02ec0e4330 con 0x7f02f81099f0 2024-07-26T12:44:23.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.125+0000 7f02f7e00640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f02e00b3ca0 0x7f02e00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:23.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.126+0000 7f02f7e00640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f02e00b3ca0 0x7f02e00b61b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02f810fae0 tx=0x7f02e8009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:23.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.126+0000 7f02f5600640 1 -- 172.21.15.107:0/3157784694 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f02ec0aaea0 con 0x7f02f81099f0 2024-07-26T12:44:23.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.326+0000 7f02fe200640 1 -- 172.21.15.107:0/3157784694 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02f802e0c0 con 0x7f02f81099f0 2024-07-26T12:44:23.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.328+0000 7f02f5600640 1 -- 172.21.15.107:0/3157784694 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25016 (secure 0 0 0) 0x7f02ec0af3f0 con 0x7f02f81099f0 2024-07-26T12:44:23.330 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:44:23.330 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:43:35.834161+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:44:23.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 -- 172.21.15.107:0/3157784694 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f02e00b3ca0 msgr2=0x7f02e00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:23.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f02e00b3ca0 0x7f02e00b61b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02f810fae0 tx=0x7f02e8009560 comp rx=0 tx=0).stop 2024-07-26T12:44:23.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 -- 172.21.15.107:0/3157784694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f81099f0 msgr2=0x7f02f81a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:23.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f81099f0 0x7f02f81a0330 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e9f0 tx=0x7f02ec00eeb0 comp rx=0 tx=0).stop 2024-07-26T12:44:23.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 -- 172.21.15.107:0/3157784694 shutdown_connections 2024-07-26T12:44:23.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f02e00b3ca0 0x7f02e00b61b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:23.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f02f810fc80 0x7f02f81a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:23.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f02f810f330 0x7f02f81a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:23.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 --2- 172.21.15.107:0/3157784694 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f02f81099f0 0x7f02f81a0330 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:23.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 -- 172.21.15.107:0/3157784694 >> 172.21.15.107:0/3157784694 conn(0x7f02f8078ff0 msgr2=0x7f02f80797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:23.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 -- 172.21.15.107:0/3157784694 shutdown_connections 2024-07-26T12:44:23.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:23.331+0000 7f02fe200640 1 -- 172.21.15.107:0/3157784694 wait complete. 2024-07-26T12:44:23.750 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:23 smithi107 ceph-mon[32659]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:23 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3157784694' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:44:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:23 smithi132 ceph-mon[37580]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:23 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3157784694' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:44:24.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:44:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:23 smithi112 ceph-mon[37501]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:23 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3157784694' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:44:24.632 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:25 smithi107 ceph-mon[32659]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:25 smithi132 ceph-mon[37580]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:25 smithi112 ceph-mon[37501]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:27.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.012+0000 7efcadc00640 1 -- 172.21.15.107:0/1877758348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efca810f330 msgr2=0x7efca810f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:27.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.012+0000 7efcadc00640 1 --2- 172.21.15.107:0/1877758348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efca810f330 0x7efca810f740 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efc94009860 tx=0x7efc9402a340 comp rx=0 tx=0).stop 2024-07-26T12:44:27.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.012+0000 7efcadc00640 1 -- 172.21.15.107:0/1877758348 shutdown_connections 2024-07-26T12:44:27.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.012+0000 7efcadc00640 1 --2- 172.21.15.107:0/1877758348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efca810fc80 0x7efca81164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:27.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.012+0000 7efcadc00640 1 --2- 172.21.15.107:0/1877758348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efca810f330 0x7efca810f740 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:27.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.012+0000 7efcadc00640 1 --2- 172.21.15.107:0/1877758348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efca81099f0 0x7efca8109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:27.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.012+0000 7efcadc00640 1 -- 172.21.15.107:0/1877758348 >> 172.21.15.107:0/1877758348 conn(0x7efca8078ff0 msgr2=0x7efca8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:27.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.012+0000 7efcadc00640 1 -- 172.21.15.107:0/1877758348 shutdown_connections 2024-07-26T12:44:27.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.012+0000 7efcadc00640 1 -- 172.21.15.107:0/1877758348 wait complete. 2024-07-26T12:44:27.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.013+0000 7efcadc00640 1 Processor -- start 2024-07-26T12:44:27.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.014+0000 7efcadc00640 1 -- start start 2024-07-26T12:44:27.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.014+0000 7efcadc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efca81099f0 0x7efca81a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.014+0000 7efcadc00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efca810f330 0x7efca81a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.014+0000 7efcadc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efca810fc80 0x7efca81a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.014+0000 7efcadc00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efca8119940 con 0x7efca810f330 2024-07-26T12:44:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.014+0000 7efcadc00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efca81197c0 con 0x7efca810fc80 2024-07-26T12:44:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.014+0000 7efcadc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efca8119ac0 con 0x7efca81099f0 2024-07-26T12:44:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efca7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efca810f330 0x7efca81a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efcad200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efca810fc80 0x7efca81a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efca7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efca810f330 0x7efca81a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:40156/0 (socket says 172.21.15.107:40156) 2024-07-26T12:44:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efca7400640 1 -- 172.21.15.107:0/3999104203 learned_addr learned my addr 172.21.15.107:0/3999104203 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efca7e00640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efca81099f0 0x7efca81a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:27.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efcad200640 1 -- 172.21.15.107:0/3999104203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efca81099f0 msgr2=0x7efca81a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:27.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efcad200640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efca81099f0 0x7efca81a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:27.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efcad200640 1 -- 172.21.15.107:0/3999104203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efca810f330 msgr2=0x7efca81a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:27.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efcad200640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efca810f330 0x7efca81a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:27.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efcad200640 1 -- 172.21.15.107:0/3999104203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca81a62a0 con 0x7efca810fc80 2024-07-26T12:44:27.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.015+0000 7efca7400640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efca810f330 0x7efca81a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:44:27.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.016+0000 7efcad200640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efca810fc80 0x7efca81a5b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efca000e970 tx=0x7efca000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:27.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.016+0000 7efca4c00640 1 -- 172.21.15.107:0/3999104203 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca000ccc0 con 0x7efca810fc80 2024-07-26T12:44:27.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.016+0000 7efca4c00640 1 -- 172.21.15.107:0/3999104203 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efca0005020 con 0x7efca810fc80 2024-07-26T12:44:27.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.016+0000 7efca4c00640 1 -- 172.21.15.107:0/3999104203 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0004a00 con 0x7efca810fc80 2024-07-26T12:44:27.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.016+0000 7efcadc00640 1 -- 172.21.15.107:0/3999104203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca81a6590 con 0x7efca810fc80 2024-07-26T12:44:27.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.016+0000 7efcadc00640 1 -- 172.21.15.107:0/3999104203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca81accf0 con 0x7efca810fc80 2024-07-26T12:44:27.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.017+0000 7efcadc00640 1 -- 172.21.15.107:0/3999104203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca8109e00 con 0x7efca810fc80 2024-07-26T12:44:27.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.020+0000 7efca4c00640 1 -- 172.21.15.107:0/3999104203 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7efca0004ba0 con 0x7efca810fc80 2024-07-26T12:44:27.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.021+0000 7efca4c00640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc8c0b3ca0 0x7efc8c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:27.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.021+0000 7efca4c00640 1 -- 172.21.15.107:0/3999104203 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7efca00e3ff0 con 0x7efca810fc80 2024-07-26T12:44:27.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.022+0000 7efca7e00640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc8c0b3ca0 0x7efc8c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:27.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.022+0000 7efca7e00640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc8c0b3ca0 0x7efc8c0b61b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efc98005fd0 tx=0x7efc98010040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:27.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.023+0000 7efca4c00640 1 -- 172.21.15.107:0/3999104203 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7efca00ab260 con 0x7efca810fc80 2024-07-26T12:44:27.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.220+0000 7efcadc00640 1 -- 172.21.15.107:0/3999104203 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efca81a1460 con 0x7efca810fc80 2024-07-26T12:44:27.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.222+0000 7efca4c00640 1 -- 172.21.15.107:0/3999104203 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+25016 (secure 0 0 0) 0x7efca81a1460 con 0x7efca810fc80 2024-07-26T12:44:27.224 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:44:27.224 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":88,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:43:35.834161+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:44:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.225+0000 7efcadc00640 1 -- 172.21.15.107:0/3999104203 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc8c0b3ca0 msgr2=0x7efc8c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:27.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.225+0000 7efcadc00640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc8c0b3ca0 0x7efc8c0b61b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efc98005fd0 tx=0x7efc98010040 comp rx=0 tx=0).stop 2024-07-26T12:44:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.225+0000 7efcadc00640 1 -- 172.21.15.107:0/3999104203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efca810fc80 msgr2=0x7efca81a5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.225+0000 7efcadc00640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efca810fc80 0x7efca81a5b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7efca000e970 tx=0x7efca000ee30 comp rx=0 tx=0).stop 2024-07-26T12:44:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.225+0000 7efcadc00640 1 -- 172.21.15.107:0/3999104203 shutdown_connections 2024-07-26T12:44:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.225+0000 7efcadc00640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc8c0b3ca0 0x7efc8c0b61b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.225+0000 7efcadc00640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efca810fc80 0x7efca81a5b70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:27.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.225+0000 7efcadc00640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efca810f330 0x7efca81a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.225+0000 7efcadc00640 1 --2- 172.21.15.107:0/3999104203 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efca81099f0 0x7efca81a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.225+0000 7efcadc00640 1 -- 172.21.15.107:0/3999104203 >> 172.21.15.107:0/3999104203 conn(0x7efca8078ff0 msgr2=0x7efca8107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.226+0000 7efcadc00640 1 -- 172.21.15.107:0/3999104203 shutdown_connections 2024-07-26T12:44:27.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:27.226+0000 7efcadc00640 1 -- 172.21.15.107:0/3999104203 wait complete. 2024-07-26T12:44:27.910 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:27 smithi107 ceph-mon[32659]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:27.910 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:27 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3999104203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:44:28.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-07-26T12:44:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:27 smithi132 ceph-mon[37580]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:27 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3999104203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:44:28.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:27 smithi112 ceph-mon[37501]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:28.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:27 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3999104203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:44:28.545 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:29.894 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:29 smithi107 ceph-mon[32659]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:29 smithi132 ceph-mon[37580]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:29 smithi112 ceph-mon[37501]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T12:44:30.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.798+0000 7fc4d1200640 1 -- 172.21.15.107:0/4012553754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4cc0751e0 msgr2=0x7fc4cc0755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:30.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.798+0000 7fc4d1200640 1 --2- 172.21.15.107:0/4012553754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4cc0751e0 0x7fc4cc0755f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009900 tx=0x7fc4c402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:44:30.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.798+0000 7fc4d1200640 1 -- 172.21.15.107:0/4012553754 shutdown_connections 2024-07-26T12:44:30.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.798+0000 7fc4d1200640 1 --2- 172.21.15.107:0/4012553754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4cc10faf0 0x7fc4cc111f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:30.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.798+0000 7fc4d1200640 1 --2- 172.21.15.107:0/4012553754 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4cc075b30 0x7fc4cc10f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:30.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.798+0000 7fc4d1200640 1 --2- 172.21.15.107:0/4012553754 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4cc0751e0 0x7fc4cc0755f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:30.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.798+0000 7fc4d1200640 1 -- 172.21.15.107:0/4012553754 >> 172.21.15.107:0/4012553754 conn(0x7fc4cc07b280 msgr2=0x7fc4cc07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:30.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.798+0000 7fc4d1200640 1 -- 172.21.15.107:0/4012553754 shutdown_connections 2024-07-26T12:44:30.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.798+0000 7fc4d1200640 1 -- 172.21.15.107:0/4012553754 wait complete. 2024-07-26T12:44:30.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.799+0000 7fc4d1200640 1 Processor -- start 2024-07-26T12:44:30.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.799+0000 7fc4d1200640 1 -- start start 2024-07-26T12:44:30.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.799+0000 7fc4d1200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4cc0751e0 0x7fc4cc19c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:30.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4d1200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4cc075b30 0x7fc4cc19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4d1200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4cc10faf0 0x7fc4cc1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4d1200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1156d0 con 0x7fc4cc075b30 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4d1200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc115550 con 0x7fc4cc0751e0 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4d1200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc115850 con 0x7fc4cc10faf0 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4c9e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4cc0751e0 0x7fc4cc19c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4c9e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4cc0751e0 0x7fc4cc19c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:55022/0 (socket says 172.21.15.107:55022) 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4c9400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4cc075b30 0x7fc4cc19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4c9e00640 1 -- 172.21.15.107:0/1344258719 learned_addr learned my addr 172.21.15.107:0/1344258719 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4ca800640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4cc10faf0 0x7fc4cc1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4c9e00640 1 -- 172.21.15.107:0/1344258719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4cc10faf0 msgr2=0x7fc4cc1a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4c9e00640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4cc10faf0 0x7fc4cc1a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:30.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4c9e00640 1 -- 172.21.15.107:0/1344258719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4cc075b30 msgr2=0x7fc4cc19c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:30.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4c9e00640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4cc075b30 0x7fc4cc19c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:30.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4c9e00640 1 -- 172.21.15.107:0/1344258719 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4cc1a2070 con 0x7fc4cc0751e0 2024-07-26T12:44:30.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.800+0000 7fc4c9400640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4cc075b30 0x7fc4cc19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:44:30.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.801+0000 7fc4c9e00640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4cc0751e0 0x7fc4cc19c100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4005ab0 tx=0x7fc4c4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:30.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.801+0000 7fc4c2a00640 1 -- 172.21.15.107:0/1344258719 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c403f070 con 0x7fc4cc0751e0 2024-07-26T12:44:30.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.801+0000 7fc4c2a00640 1 -- 172.21.15.107:0/1344258719 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4c4031b60 con 0x7fc4cc0751e0 2024-07-26T12:44:30.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.801+0000 7fc4d1200640 1 -- 172.21.15.107:0/1344258719 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4cc1a2300 con 0x7fc4cc0751e0 2024-07-26T12:44:30.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.801+0000 7fc4c2a00640 1 -- 172.21.15.107:0/1344258719 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4035030 con 0x7fc4cc0751e0 2024-07-26T12:44:30.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.801+0000 7fc4d1200640 1 -- 172.21.15.107:0/1344258719 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4cc1a8ac0 con 0x7fc4cc0751e0 2024-07-26T12:44:30.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.803+0000 7fc4d1200640 1 -- 172.21.15.107:0/1344258719 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4cc0755f0 con 0x7fc4cc0751e0 2024-07-26T12:44:30.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.805+0000 7fc4c2a00640 1 -- 172.21.15.107:0/1344258719 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fc4c4042070 con 0x7fc4cc0751e0 2024-07-26T12:44:30.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.806+0000 7fc4c2a00640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc4b00b3c20 0x7fc4b00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:30.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.806+0000 7fc4c2a00640 1 -- 172.21.15.107:0/1344258719 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fc4c41047c0 con 0x7fc4cc0751e0 2024-07-26T12:44:30.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.806+0000 7fc4c9400640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc4b00b3c20 0x7fc4b00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:30.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.807+0000 7fc4c9400640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc4b00b3c20 0x7fc4b00b6130 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc111880 tx=0x7fc4b4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:30.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:30.810+0000 7fc4c2a00640 1 -- 172.21.15.107:0/1344258719 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc4c40cb330 con 0x7fc4cc0751e0 2024-07-26T12:44:31.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.047+0000 7fc4d1200640 1 -- 172.21.15.107:0/1344258719 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc4cc1a8d20 con 0x7fc4cc0751e0 2024-07-26T12:44:31.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.782+0000 7fc4c2a00640 1 -- 172.21.15.107:0/1344258719 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7fc4c40cf880 con 0x7fc4cc0751e0 2024-07-26T12:44:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 -- 172.21.15.107:0/1344258719 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc4b00b3c20 msgr2=0x7fc4b00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:31.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc4b00b3c20 0x7fc4b00b6130 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc111880 tx=0x7fc4b4005f90 comp rx=0 tx=0).stop 2024-07-26T12:44:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 -- 172.21.15.107:0/1344258719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4cc0751e0 msgr2=0x7fc4cc19c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4cc0751e0 0x7fc4cc19c100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4005ab0 tx=0x7fc4c4003c10 comp rx=0 tx=0).stop 2024-07-26T12:44:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 -- 172.21.15.107:0/1344258719 shutdown_connections 2024-07-26T12:44:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc4b00b3c20 0x7fc4b00b6130 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4cc10faf0 0x7fc4cc1a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4cc075b30 0x7fc4cc19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:31.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 --2- 172.21.15.107:0/1344258719 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc4cc0751e0 0x7fc4cc19c100 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:31.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 -- 172.21.15.107:0/1344258719 >> 172.21.15.107:0/1344258719 conn(0x7fc4cc07b280 msgr2=0x7fc4cc100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:31.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 -- 172.21.15.107:0/1344258719 shutdown_connections 2024-07-26T12:44:31.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:31.786+0000 7fc4d1200640 1 -- 172.21.15.107:0/1344258719 wait complete. 2024-07-26T12:44:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:31 smithi107 ceph-mon[32659]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:31 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1344258719' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T12:44:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:31 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T12:44:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:31 smithi132 ceph-mon[37580]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:31 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1344258719' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T12:44:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:31 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T12:44:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:31 smithi112 ceph-mon[37501]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:31 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1344258719' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T12:44:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:31 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T12:44:32.703 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-07-26T12:44:32.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-07-26T12:44:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:32 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T12:44:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:32 smithi107 ceph-mon[32659]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:44:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:32 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T12:44:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:32 smithi132 ceph-mon[37580]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:44:33.176 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:32 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T12:44:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:32 smithi112 ceph-mon[37501]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:44:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:33 smithi107 ceph-mon[32659]: pgmap v345: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:33 smithi132 ceph-mon[37580]: pgmap v345: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:33 smithi112 ceph-mon[37501]: pgmap v345: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:35.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.502+0000 7fda6b200640 1 -- 172.21.15.107:0/2709703718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6c10b860 msgr2=0x7fda6c112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.502+0000 7fda6b200640 1 --2- 172.21.15.107:0/2709703718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6c10b860 0x7fda6c112090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fda64009ca0 tx=0x7fda6402a8b0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.502+0000 7fda6b200640 1 -- 172.21.15.107:0/2709703718 shutdown_connections 2024-07-26T12:44:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.502+0000 7fda6b200640 1 --2- 172.21.15.107:0/2709703718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6c10b860 0x7fda6c112090 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.502+0000 7fda6b200640 1 --2- 172.21.15.107:0/2709703718 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c10af10 0x7fda6c10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.502+0000 7fda6b200640 1 --2- 172.21.15.107:0/2709703718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda6c1055d0 0x7fda6c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.502+0000 7fda6b200640 1 -- 172.21.15.107:0/2709703718 >> 172.21.15.107:0/2709703718 conn(0x7fda6c101040 msgr2=0x7fda6c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.502+0000 7fda6b200640 1 -- 172.21.15.107:0/2709703718 shutdown_connections 2024-07-26T12:44:35.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.502+0000 7fda6b200640 1 -- 172.21.15.107:0/2709703718 wait complete. 2024-07-26T12:44:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.503+0000 7fda6b200640 1 Processor -- start 2024-07-26T12:44:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.503+0000 7fda6b200640 1 -- start start 2024-07-26T12:44:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda6b200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda6c1055d0 0x7fda6c19bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda6b200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c10af10 0x7fda6c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda6b200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6c10b860 0x7fda6c1a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda6b200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c115530 con 0x7fda6c10b860 2024-07-26T12:44:35.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda6b200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c1153b0 con 0x7fda6c10af10 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda6b200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fda6c1156b0 con 0x7fda6c1055d0 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda69e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda6c1055d0 0x7fda6c19bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda6a800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6c10b860 0x7fda6c1a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda69e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda6c1055d0 0x7fda6c19bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:41988/0 (socket says 172.21.15.107:41988) 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda69e00640 1 -- 172.21.15.107:0/3105351394 learned_addr learned my addr 172.21.15.107:0/3105351394 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda6a800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6c10b860 0x7fda6c1a1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41886/0 (socket says 172.21.15.107:41886) 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.504+0000 7fda69400640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c10af10 0x7fda6c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda69e00640 1 -- 172.21.15.107:0/3105351394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c10af10 msgr2=0x7fda6c19c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda69e00640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c10af10 0x7fda6c19c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda69e00640 1 -- 172.21.15.107:0/3105351394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6c10b860 msgr2=0x7fda6c1a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda69e00640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6c10b860 0x7fda6c1a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda69e00640 1 -- 172.21.15.107:0/3105351394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda6c1a1eb0 con 0x7fda6c1055d0 2024-07-26T12:44:35.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda69e00640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda6c1055d0 0x7fda6c19bf40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fda5400e9f0 tx=0x7fda5400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda62a00640 1 -- 172.21.15.107:0/3105351394 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5400cd80 con 0x7fda6c1055d0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda6b200640 1 -- 172.21.15.107:0/3105351394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda6c1a21a0 con 0x7fda6c1055d0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda62a00640 1 -- 172.21.15.107:0/3105351394 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda540054d0 con 0x7fda6c1055d0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda62a00640 1 -- 172.21.15.107:0/3105351394 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54010690 con 0x7fda6c1055d0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.505+0000 7fda6b200640 1 -- 172.21.15.107:0/3105351394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda6c1a8900 con 0x7fda6c1055d0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.507+0000 7fda6b200640 1 -- 172.21.15.107:0/3105351394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda6c1059e0 con 0x7fda6c1055d0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.509+0000 7fda62a00640 1 -- 172.21.15.107:0/3105351394 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fda54004e90 con 0x7fda6c1055d0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.511+0000 7fda62a00640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fda500b3c20 0x7fda500b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.511+0000 7fda62a00640 1 -- 172.21.15.107:0/3105351394 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fda540e45b0 con 0x7fda6c1055d0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.511+0000 7fda69400640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fda500b3c20 0x7fda500b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.512+0000 7fda69400640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fda500b3c20 0x7fda500b6130 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fda6c104740 tx=0x7fda5c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:35.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.512+0000 7fda62a00640 1 -- 172.21.15.107:0/3105351394 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fda540ab770 con 0x7fda6c1055d0 2024-07-26T12:44:35.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.753+0000 7fda6b200640 1 -- 172.21.15.107:0/3105351394 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fda6c02e0c0 con 0x7fda6c1055d0 2024-07-26T12:44:35.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.819+0000 7fda62a00640 1 -- 172.21.15.107:0/3105351394 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7fda540ab070 con 0x7fda6c1055d0 2024-07-26T12:44:35.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.822+0000 7fda6b200640 1 -- 172.21.15.107:0/3105351394 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fda500b3c20 msgr2=0x7fda500b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:35.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.822+0000 7fda6b200640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fda500b3c20 0x7fda500b6130 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fda6c104740 tx=0x7fda5c005f90 comp rx=0 tx=0).stop 2024-07-26T12:44:35.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.822+0000 7fda6b200640 1 -- 172.21.15.107:0/3105351394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda6c1055d0 msgr2=0x7fda6c19bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:35.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.822+0000 7fda6b200640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda6c1055d0 0x7fda6c19bf40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fda5400e9f0 tx=0x7fda5400edf0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.822+0000 7fda6b200640 1 -- 172.21.15.107:0/3105351394 shutdown_connections 2024-07-26T12:44:35.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.822+0000 7fda6b200640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fda500b3c20 0x7fda500b6130 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.822+0000 7fda6b200640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda6c10b860 0x7fda6c1a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.822+0000 7fda6b200640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fda6c10af10 0x7fda6c19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.822+0000 7fda6b200640 1 --2- 172.21.15.107:0/3105351394 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fda6c1055d0 0x7fda6c19bf40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:35.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.822+0000 7fda6b200640 1 -- 172.21.15.107:0/3105351394 >> 172.21.15.107:0/3105351394 conn(0x7fda6c101040 msgr2=0x7fda6c10a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:35.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.823+0000 7fda6b200640 1 -- 172.21.15.107:0/3105351394 shutdown_connections 2024-07-26T12:44:35.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:35.823+0000 7fda6b200640 1 -- 172.21.15.107:0/3105351394 wait complete. 2024-07-26T12:44:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: pgmap v346: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-07-26T12:44:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-07-26T12:44:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-07-26T12:44:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-07-26T12:44:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-07-26T12:44:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-07-26T12:44:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-07-26T12:44:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-07-26T12:44:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3105351394' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T12:44:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:35 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T12:44:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: pgmap v346: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-07-26T12:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-07-26T12:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-07-26T12:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-07-26T12:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-07-26T12:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-07-26T12:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-07-26T12:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-07-26T12:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3105351394' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T12:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:35 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T12:44:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: pgmap v346: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T12:44:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-07-26T12:44:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-07-26T12:44:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-07-26T12:44:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]: dispatch 2024-07-26T12:44:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-07-26T12:44:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-07-26T12:44:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]: dispatch 2024-07-26T12:44:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-07-26T12:44:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3105351394' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T12:44:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:35 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T12:44:36.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-07-26T12:44:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T12:44:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-07-26T12:44:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-07-26T12:44:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-07-26T12:44:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-07-26T12:44:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T12:44:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T12:44:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T12:44:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:44:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-07-26T12:44:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T12:44:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:36 smithi107 ceph-mon[32659]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T12:44:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T12:44:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:36 smithi112 ceph-mon[37501]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T12:44:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T12:44:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.5", "id": [3, 9]}]': finished 2024-07-26T12:44:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-07-26T12:44:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [3, 8]}]': finished 2024-07-26T12:44:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-07-26T12:44:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T12:44:37.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T12:44:37.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T12:44:37.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-07-26T12:44:37.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-07-26T12:44:37.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T12:44:37.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:36 smithi132 ceph-mon[37580]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T12:44:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:37 smithi107 ceph-mon[32659]: pgmap v348: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-07-26T12:44:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:37 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] up:active 2024-07-26T12:44:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:37 smithi107 ceph-mon[32659]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T12:44:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:37 smithi107 ceph-mon[32659]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-07-26T12:44:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:37 smithi107 ceph-mon[32659]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-07-26T12:44:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:37 smithi107 ceph-mon[32659]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-07-26T12:44:38.200 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:44:37 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2024-07-26T12:44:37.720+0000 7f3578b59640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T12:44:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:37 smithi112 ceph-mon[37501]: pgmap v348: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-07-26T12:44:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:37 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] up:active 2024-07-26T12:44:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:37 smithi112 ceph-mon[37501]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T12:44:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:37 smithi112 ceph-mon[37501]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-07-26T12:44:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:37 smithi112 ceph-mon[37501]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-07-26T12:44:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:37 smithi112 ceph-mon[37501]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-07-26T12:44:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:37 smithi132 ceph-mon[37580]: pgmap v348: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-07-26T12:44:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:37 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] up:active 2024-07-26T12:44:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:37 smithi132 ceph-mon[37580]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T12:44:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:37 smithi132 ceph-mon[37580]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-07-26T12:44:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:37 smithi132 ceph-mon[37580]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-07-26T12:44:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:37 smithi132 ceph-mon[37580]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-07-26T12:44:39.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:38 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] up:active 2024-07-26T12:44:39.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:38 smithi107 ceph-mon[32659]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T12:44:39.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:38 smithi107 ceph-mon[32659]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-07-26T12:44:39.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:38 smithi107 ceph-mon[32659]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-07-26T12:44:39.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:38 smithi107 ceph-mon[32659]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-07-26T12:44:39.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:38 smithi107 ceph-mon[32659]: pgmap v350: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-07-26T12:44:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:38 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] up:active 2024-07-26T12:44:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:38 smithi112 ceph-mon[37501]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T12:44:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:38 smithi112 ceph-mon[37501]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-07-26T12:44:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:38 smithi112 ceph-mon[37501]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-07-26T12:44:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:38 smithi112 ceph-mon[37501]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-07-26T12:44:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:38 smithi112 ceph-mon[37501]: pgmap v350: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-07-26T12:44:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:38 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] up:active 2024-07-26T12:44:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:38 smithi132 ceph-mon[37580]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T12:44:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:38 smithi132 ceph-mon[37580]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-07-26T12:44:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:38 smithi132 ceph-mon[37580]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-07-26T12:44:39.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:38 smithi132 ceph-mon[37580]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-07-26T12:44:39.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:38 smithi132 ceph-mon[37580]: pgmap v350: 129 pgs: 1 peering, 128 active+clean; 581 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-07-26T12:44:39.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.477+0000 7f1b8ae00640 1 -- 172.21.15.107:0/3330795156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b841033e0 msgr2=0x7f1b841037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.477+0000 7f1b8ae00640 1 --2- 172.21.15.107:0/3330795156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b841033e0 0x7f1b841037f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1b78009900 tx=0x7f1b7802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.477+0000 7f1b8ae00640 1 -- 172.21.15.107:0/3330795156 shutdown_connections 2024-07-26T12:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.477+0000 7f1b8ae00640 1 --2- 172.21.15.107:0/3330795156 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b84109670 0x7f1b8410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.477+0000 7f1b8ae00640 1 --2- 172.21.15.107:0/3330795156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b84108d20 0x7f1b84109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.477+0000 7f1b8ae00640 1 --2- 172.21.15.107:0/3330795156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b841033e0 0x7f1b841037f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.477+0000 7f1b8ae00640 1 -- 172.21.15.107:0/3330795156 >> 172.21.15.107:0/3330795156 conn(0x7f1b840fee70 msgr2=0x7f1b841012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.478+0000 7f1b8ae00640 1 -- 172.21.15.107:0/3330795156 shutdown_connections 2024-07-26T12:44:39.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.478+0000 7f1b8ae00640 1 -- 172.21.15.107:0/3330795156 wait complete. 2024-07-26T12:44:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.479+0000 7f1b8ae00640 1 Processor -- start 2024-07-26T12:44:39.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.479+0000 7f1b8ae00640 1 -- start start 2024-07-26T12:44:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.479+0000 7f1b8ae00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b841033e0 0x7f1b8419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.479+0000 7f1b8ae00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b84108d20 0x7f1b8419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.479+0000 7f1b8ae00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b84109670 0x7f1b841a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b8ae00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1b84113390 con 0x7f1b84108d20 2024-07-26T12:44:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b8ae00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1b84113210 con 0x7f1b841033e0 2024-07-26T12:44:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b8ae00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1b84113510 con 0x7f1b84109670 2024-07-26T12:44:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b89a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b841033e0 0x7f1b8419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b89a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b841033e0 0x7f1b8419bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:34256/0 (socket says 172.21.15.107:34256) 2024-07-26T12:44:39.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b89a00640 1 -- 172.21.15.107:0/937508584 learned_addr learned my addr 172.21.15.107:0/937508584 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:39.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b89000640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b84108d20 0x7f1b8419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:39.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b8a400640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b84109670 0x7f1b841a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:39.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b89a00640 1 -- 172.21.15.107:0/937508584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b84109670 msgr2=0x7f1b841a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:39.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b89a00640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b84109670 0x7f1b841a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:39.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b89a00640 1 -- 172.21.15.107:0/937508584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b84108d20 msgr2=0x7f1b8419c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:39.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b89a00640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b84108d20 0x7f1b8419c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:39.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.480+0000 7f1b89a00640 1 -- 172.21.15.107:0/937508584 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b841a1e90 con 0x7f1b841033e0 2024-07-26T12:44:39.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.481+0000 7f1b89a00640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b841033e0 0x7f1b8419bf20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1b78005ab0 tx=0x7f1b78003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:39.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.481+0000 7f1b82000640 1 -- 172.21.15.107:0/937508584 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7803f070 con 0x7f1b841033e0 2024-07-26T12:44:39.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.481+0000 7f1b8ae00640 1 -- 172.21.15.107:0/937508584 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b841a2120 con 0x7f1b841033e0 2024-07-26T12:44:39.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.481+0000 7f1b82000640 1 -- 172.21.15.107:0/937508584 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b78031b60 con 0x7f1b841033e0 2024-07-26T12:44:39.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.481+0000 7f1b82000640 1 -- 172.21.15.107:0/937508584 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b78035030 con 0x7f1b841033e0 2024-07-26T12:44:39.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.481+0000 7f1b8ae00640 1 -- 172.21.15.107:0/937508584 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b841a88e0 con 0x7f1b841033e0 2024-07-26T12:44:39.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.483+0000 7f1b8ae00640 1 -- 172.21.15.107:0/937508584 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b841037f0 con 0x7f1b841033e0 2024-07-26T12:44:39.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.485+0000 7f1b82000640 1 -- 172.21.15.107:0/937508584 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f1b78042070 con 0x7f1b841033e0 2024-07-26T12:44:39.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.487+0000 7f1b82000640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1b6c0b3ca0 0x7f1b6c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:39.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.487+0000 7f1b82000640 1 -- 172.21.15.107:0/937508584 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1b78104810 con 0x7f1b841033e0 2024-07-26T12:44:39.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.487+0000 7f1b89000640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1b6c0b3ca0 0x7f1b6c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:39.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.488+0000 7f1b89000640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1b6c0b3ca0 0x7f1b6c0b61b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1b841088d0 tx=0x7f1b70005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:39.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.491+0000 7f1b82000640 1 -- 172.21.15.107:0/937508584 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1b780cb960 con 0x7f1b841033e0 2024-07-26T12:44:39.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.734+0000 7f1b8ae00640 1 -- 172.21.15.107:0/937508584 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f1b8419d050 con 0x7f1b841033e0 2024-07-26T12:44:39.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.868+0000 7f1b82000640 1 -- 172.21.15.107:0/937508584 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v26) ==== 121+0+0 (secure 0 0 0) 0x7f1b780cb260 con 0x7f1b841033e0 2024-07-26T12:44:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.872+0000 7f1b8ae00640 1 -- 172.21.15.107:0/937508584 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1b6c0b3ca0 msgr2=0x7f1b6c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.872+0000 7f1b8ae00640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1b6c0b3ca0 0x7f1b6c0b61b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1b841088d0 tx=0x7f1b70005f90 comp rx=0 tx=0).stop 2024-07-26T12:44:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.872+0000 7f1b8ae00640 1 -- 172.21.15.107:0/937508584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b841033e0 msgr2=0x7f1b8419bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.872+0000 7f1b8ae00640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b841033e0 0x7f1b8419bf20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1b78005ab0 tx=0x7f1b78003c10 comp rx=0 tx=0).stop 2024-07-26T12:44:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.873+0000 7f1b8ae00640 1 -- 172.21.15.107:0/937508584 shutdown_connections 2024-07-26T12:44:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.873+0000 7f1b8ae00640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f1b6c0b3ca0 0x7f1b6c0b61b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.873+0000 7f1b8ae00640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b84109670 0x7f1b841a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.873+0000 7f1b8ae00640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1b84108d20 0x7f1b8419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.873+0000 7f1b8ae00640 1 --2- 172.21.15.107:0/937508584 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1b841033e0 0x7f1b8419bf20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.873+0000 7f1b8ae00640 1 -- 172.21.15.107:0/937508584 >> 172.21.15.107:0/937508584 conn(0x7f1b840fee70 msgr2=0x7f1b8410e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.873+0000 7f1b8ae00640 1 -- 172.21.15.107:0/937508584 shutdown_connections 2024-07-26T12:44:39.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:39.873+0000 7f1b8ae00640 1 -- 172.21.15.107:0/937508584 wait complete. 2024-07-26T12:44:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:39 smithi107 ceph-mon[32659]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-07-26T12:44:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:39 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] up:active 2024-07-26T12:44:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:39 smithi107 ceph-mon[32659]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T12:44:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:39 smithi107 ceph-mon[32659]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T12:44:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:39 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:44:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:39 smithi107 ceph-mon[32659]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-07-26T12:44:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:39 smithi107 ceph-mon[32659]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T12:44:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:39 smithi107 ceph-mon[32659]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-07-26T12:44:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:39 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/937508584' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T12:44:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:39 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T12:44:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:39 smithi112 ceph-mon[37501]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-07-26T12:44:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:39 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] up:active 2024-07-26T12:44:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:39 smithi112 ceph-mon[37501]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T12:44:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:39 smithi112 ceph-mon[37501]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T12:44:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:39 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:44:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:39 smithi112 ceph-mon[37501]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-07-26T12:44:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:39 smithi112 ceph-mon[37501]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T12:44:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:39 smithi112 ceph-mon[37501]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-07-26T12:44:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:39 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/937508584' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T12:44:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:39 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T12:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:39 smithi132 ceph-mon[37580]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-07-26T12:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:39 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] up:active 2024-07-26T12:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:39 smithi132 ceph-mon[37580]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T12:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:39 smithi132 ceph-mon[37580]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T12:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:39 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:39 smithi132 ceph-mon[37580]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-07-26T12:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:39 smithi132 ceph-mon[37580]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T12:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:39 smithi132 ceph-mon[37580]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-07-26T12:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:39 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/937508584' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T12:44:40.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:39 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T12:44:40.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-07-26T12:44:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:40 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T12:44:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:40 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] up:active 2024-07-26T12:44:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:40 smithi107 ceph-mon[32659]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T12:44:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:40 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:44:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:40 smithi107 ceph-mon[32659]: pgmap v351: 129 pgs: 1 peering, 128 active+clean; 584 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s wr, 8 op/s 2024-07-26T12:44:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:40 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T12:44:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:40 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] up:active 2024-07-26T12:44:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:40 smithi112 ceph-mon[37501]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T12:44:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:40 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:44:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:40 smithi112 ceph-mon[37501]: pgmap v351: 129 pgs: 1 peering, 128 active+clean; 584 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s wr, 8 op/s 2024-07-26T12:44:41.274 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:40 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T12:44:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:40 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] up:active 2024-07-26T12:44:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:40 smithi132 ceph-mon[37580]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T12:44:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:40 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:44:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:40 smithi132 ceph-mon[37580]: pgmap v351: 129 pgs: 1 peering, 128 active+clean; 584 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s wr, 8 op/s 2024-07-26T12:44:42.974 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:42 smithi107 ceph-mon[32659]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-07-26T12:44:42.974 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:42 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:44:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:42 smithi132 ceph-mon[37580]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-07-26T12:44:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:42 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:44:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:42 smithi112 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-07-26T12:44:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:42 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:44:43.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.556+0000 7f30a7c00640 1 -- 172.21.15.107:0/249487739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30a0103590 msgr2=0x7f30a01039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:43.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.556+0000 7f30a7c00640 1 --2- 172.21.15.107:0/249487739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30a0103590 0x7f30a01039a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3090009860 tx=0x7f309002a340 comp rx=0 tx=0).stop 2024-07-26T12:44:43.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.556+0000 7f30a7c00640 1 -- 172.21.15.107:0/249487739 shutdown_connections 2024-07-26T12:44:43.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.556+0000 7f30a7c00640 1 --2- 172.21.15.107:0/249487739 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30a0109820 0x7f30a0110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:43.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.556+0000 7f30a7c00640 1 --2- 172.21.15.107:0/249487739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0108ed0 0x7f30a01092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:43.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.556+0000 7f30a7c00640 1 --2- 172.21.15.107:0/249487739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30a0103590 0x7f30a01039a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:43.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.556+0000 7f30a7c00640 1 -- 172.21.15.107:0/249487739 >> 172.21.15.107:0/249487739 conn(0x7f30a007b280 msgr2=0x7f30a007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:43.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.556+0000 7f30a7c00640 1 -- 172.21.15.107:0/249487739 shutdown_connections 2024-07-26T12:44:43.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.556+0000 7f30a7c00640 1 -- 172.21.15.107:0/249487739 wait complete. 2024-07-26T12:44:43.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.557+0000 7f30a7c00640 1 Processor -- start 2024-07-26T12:44:43.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a7c00640 1 -- start start 2024-07-26T12:44:43.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a7c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30a0103590 0x7f30a019c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:43.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a7c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30a0108ed0 0x7f30a019c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:43.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a7c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0109820 0x7f30a01a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:43.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a7c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f30a01134f0 con 0x7f30a0109820 2024-07-26T12:44:43.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a7c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f30a0113370 con 0x7f30a0103590 2024-07-26T12:44:43.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a7c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f30a0113670 con 0x7f30a0108ed0 2024-07-26T12:44:43.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a5c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0109820 0x7f30a01a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:43.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a5200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30a0103590 0x7f30a019c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:43.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a5c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0109820 0x7f30a01a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41932/0 (socket says 172.21.15.107:41932) 2024-07-26T12:44:43.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f30a5c00640 1 -- 172.21.15.107:0/3554515017 learned_addr learned my addr 172.21.15.107:0/3554515017 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:43.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.558+0000 7f3097e00640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30a0108ed0 0x7f30a019c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:43.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.559+0000 7f30a5200640 1 -- 172.21.15.107:0/3554515017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30a0108ed0 msgr2=0x7f30a019c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:43.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.559+0000 7f30a5200640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30a0108ed0 0x7f30a019c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:43.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.559+0000 7f30a5200640 1 -- 172.21.15.107:0/3554515017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0109820 msgr2=0x7f30a01a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:43.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.559+0000 7f30a5200640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0109820 0x7f30a01a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:43.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.559+0000 7f30a5200640 1 -- 172.21.15.107:0/3554515017 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a01a2050 con 0x7f30a0103590 2024-07-26T12:44:43.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.559+0000 7f30a5200640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30a0103590 0x7f30a019c0e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3090035040 tx=0x7f3090032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:43.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.560+0000 7f309e000640 1 -- 172.21.15.107:0/3554515017 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090037040 con 0x7f30a0103590 2024-07-26T12:44:43.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.560+0000 7f309e000640 1 -- 172.21.15.107:0/3554515017 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3090030c10 con 0x7f30a0103590 2024-07-26T12:44:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.560+0000 7f30a7c00640 1 -- 172.21.15.107:0/3554515017 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a01a22e0 con 0x7f30a0103590 2024-07-26T12:44:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.560+0000 7f30a7c00640 1 -- 172.21.15.107:0/3554515017 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a01a8aa0 con 0x7f30a0103590 2024-07-26T12:44:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.560+0000 7f309e000640 1 -- 172.21.15.107:0/3554515017 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30900035f0 con 0x7f30a0103590 2024-07-26T12:44:43.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.561+0000 7f30a7c00640 1 -- 172.21.15.107:0/3554515017 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30a01039a0 con 0x7f30a0103590 2024-07-26T12:44:43.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.563+0000 7f309e000640 1 -- 172.21.15.107:0/3554515017 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f3090043020 con 0x7f30a0103590 2024-07-26T12:44:43.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.565+0000 7f309e000640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f30840b3ca0 0x7f30840b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:43.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.565+0000 7f309e000640 1 -- 172.21.15.107:0/3554515017 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3090103490 con 0x7f30a0103590 2024-07-26T12:44:43.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.565+0000 7f3097e00640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f30840b3ca0 0x7f30840b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:43.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.566+0000 7f3097e00640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f30840b3ca0 0x7f30840b61b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f30a0105fe0 tx=0x7f3088005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:43.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.567+0000 7f309e000640 1 -- 172.21.15.107:0/3554515017 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f30900ca660 con 0x7f30a0103590 2024-07-26T12:44:43.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:43.810+0000 7f30a7c00640 1 -- 172.21.15.107:0/3554515017 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f30a019d210 con 0x7f30a0103590 2024-07-26T12:44:43.812 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:43 smithi107 ceph-mon[32659]: pgmap v352: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5.1 KiB/s wr, 24 op/s 2024-07-26T12:44:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:43 smithi132 ceph-mon[37580]: pgmap v352: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5.1 KiB/s wr, 24 op/s 2024-07-26T12:44:44.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:43 smithi112 ceph-mon[37501]: pgmap v352: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 5.1 KiB/s wr, 24 op/s 2024-07-26T12:44:44.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.813+0000 7f309e000640 1 -- 172.21.15.107:0/3554515017 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v28) ==== 115+0+0 (secure 0 0 0) 0x7f3090030dc0 con 0x7f30a0103590 2024-07-26T12:44:44.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.818+0000 7f30a7c00640 1 -- 172.21.15.107:0/3554515017 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f30840b3ca0 msgr2=0x7f30840b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:44.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.818+0000 7f30a7c00640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f30840b3ca0 0x7f30840b61b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f30a0105fe0 tx=0x7f3088005f90 comp rx=0 tx=0).stop 2024-07-26T12:44:44.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.818+0000 7f30a7c00640 1 -- 172.21.15.107:0/3554515017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30a0103590 msgr2=0x7f30a019c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:44.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.818+0000 7f30a7c00640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30a0103590 0x7f30a019c0e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3090035040 tx=0x7f3090032d50 comp rx=0 tx=0).stop 2024-07-26T12:44:44.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.818+0000 7f30a7c00640 1 -- 172.21.15.107:0/3554515017 shutdown_connections 2024-07-26T12:44:44.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.818+0000 7f30a7c00640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f30840b3ca0 0x7f30840b61b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:44.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.818+0000 7f30a7c00640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30a0109820 0x7f30a01a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:44.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.818+0000 7f30a7c00640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f30a0108ed0 0x7f30a019c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:44.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.818+0000 7f30a7c00640 1 --2- 172.21.15.107:0/3554515017 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f30a0103590 0x7f30a019c0e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:44.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.818+0000 7f30a7c00640 1 -- 172.21.15.107:0/3554515017 >> 172.21.15.107:0/3554515017 conn(0x7f30a007b280 msgr2=0x7f30a0106ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:44.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.819+0000 7f30a7c00640 1 -- 172.21.15.107:0/3554515017 shutdown_connections 2024-07-26T12:44:44.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:44.819+0000 7f30a7c00640 1 -- 172.21.15.107:0/3554515017 wait complete. 2024-07-26T12:44:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:44 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3554515017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T12:44:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:44 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T12:44:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:44 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3554515017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T12:44:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:44 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T12:44:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:44 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3554515017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T12:44:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:44 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T12:44:45.916 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-07-26T12:44:45.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-07-26T12:44:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:45 smithi112 ceph-mon[37501]: pgmap v353: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.4 KiB/s wr, 25 op/s 2024-07-26T12:44:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:45 smithi112 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T12:44:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:45 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:44:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:45 smithi107 ceph-mon[32659]: pgmap v353: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.4 KiB/s wr, 25 op/s 2024-07-26T12:44:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:45 smithi107 ceph-mon[32659]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T12:44:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:45 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:44:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:45 smithi132 ceph-mon[37580]: pgmap v353: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5.4 KiB/s wr, 25 op/s 2024-07-26T12:44:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:45 smithi132 ceph-mon[37580]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T12:44:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:45 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:44:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:47 smithi112 ceph-mon[37501]: pgmap v354: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 4.7 KiB/s wr, 21 op/s 2024-07-26T12:44:48.264 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:47 smithi107 ceph-mon[32659]: pgmap v354: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 4.7 KiB/s wr, 21 op/s 2024-07-26T12:44:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:47 smithi132 ceph-mon[37580]: pgmap v354: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 4.7 KiB/s wr, 21 op/s 2024-07-26T12:44:48.819 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.818+0000 7f85d3000640 1 -- 172.21.15.107:0/627240858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc10af10 msgr2=0x7f85cc10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.818+0000 7f85d3000640 1 --2- 172.21.15.107:0/627240858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc10af10 0x7f85cc10b320 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009900 tx=0x7f85b802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:44:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.818+0000 7f85d3000640 1 -- 172.21.15.107:0/627240858 shutdown_connections 2024-07-26T12:44:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.818+0000 7f85d3000640 1 --2- 172.21.15.107:0/627240858 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85cc10b860 0x7f85cc112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.818+0000 7f85d3000640 1 --2- 172.21.15.107:0/627240858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc10af10 0x7f85cc10b320 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.818+0000 7f85d3000640 1 --2- 172.21.15.107:0/627240858 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85cc1055d0 0x7f85cc1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.818+0000 7f85d3000640 1 -- 172.21.15.107:0/627240858 >> 172.21.15.107:0/627240858 conn(0x7f85cc101040 msgr2=0x7f85cc103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.818+0000 7f85d3000640 1 -- 172.21.15.107:0/627240858 shutdown_connections 2024-07-26T12:44:48.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.818+0000 7f85d3000640 1 -- 172.21.15.107:0/627240858 wait complete. 2024-07-26T12:44:48.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.819+0000 7f85d3000640 1 Processor -- start 2024-07-26T12:44:48.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.819+0000 7f85d3000640 1 -- start start 2024-07-26T12:44:48.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d3000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85cc1055d0 0x7f85cc10f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:48.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d3000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc10af10 0x7f85cc10fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:48.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d3000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85cc10b860 0x7f85cc110010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d3000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc1154f0 con 0x7f85cc10af10 2024-07-26T12:44:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d3000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc115370 con 0x7f85cc10b860 2024-07-26T12:44:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d3000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc115670 con 0x7f85cc1055d0 2024-07-26T12:44:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d1c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85cc1055d0 0x7f85cc10f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d2600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85cc10b860 0x7f85cc110010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d1c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85cc1055d0 0x7f85cc10f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:40546/0 (socket says 172.21.15.107:40546) 2024-07-26T12:44:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d1c00640 1 -- 172.21.15.107:0/3919055597 learned_addr learned my addr 172.21.15.107:0/3919055597 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:48.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.820+0000 7f85d1200640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc10af10 0x7f85cc10fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.821+0000 7f85d1c00640 1 -- 172.21.15.107:0/3919055597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85cc10b860 msgr2=0x7f85cc110010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.821+0000 7f85d1c00640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85cc10b860 0x7f85cc110010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.821+0000 7f85d1c00640 1 -- 172.21.15.107:0/3919055597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc10af10 msgr2=0x7f85cc10fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.821+0000 7f85d1c00640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc10af10 0x7f85cc10fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.821+0000 7f85d1c00640 1 -- 172.21.15.107:0/3919055597 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85cc1ac420 con 0x7f85cc1055d0 2024-07-26T12:44:48.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.821+0000 7f85d1c00640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85cc1055d0 0x7f85cc10f590 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f85c000d770 tx=0x7f85c000dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.824+0000 7f85ca000640 1 -- 172.21.15.107:0/3919055597 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c0017070 con 0x7f85cc1055d0 2024-07-26T12:44:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.824+0000 7f85ca000640 1 -- 172.21.15.107:0/3919055597 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85c00054d0 con 0x7f85cc1055d0 2024-07-26T12:44:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.824+0000 7f85ca000640 1 -- 172.21.15.107:0/3919055597 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c0004c30 con 0x7f85cc1055d0 2024-07-26T12:44:48.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.824+0000 7f85d3000640 1 -- 172.21.15.107:0/3919055597 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85cc1ac710 con 0x7f85cc1055d0 2024-07-26T12:44:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.824+0000 7f85d3000640 1 -- 172.21.15.107:0/3919055597 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85cc1acbd0 con 0x7f85cc1055d0 2024-07-26T12:44:48.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.826+0000 7f85d3000640 1 -- 172.21.15.107:0/3919055597 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85cc1059e0 con 0x7f85cc1055d0 2024-07-26T12:44:48.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.828+0000 7f85ca000640 1 -- 172.21.15.107:0/3919055597 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f85c0004e10 con 0x7f85cc1055d0 2024-07-26T12:44:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.829+0000 7f85ca000640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85b40b3ca0 0x7f85b40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.830+0000 7f85ca000640 1 -- 172.21.15.107:0/3919055597 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f85c00e3d20 con 0x7f85cc1055d0 2024-07-26T12:44:48.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.830+0000 7f85d1200640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85b40b3ca0 0x7f85b40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:48.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.830+0000 7f85d1200640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85b40b3ca0 0x7f85b40b61b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009230 tx=0x7f85b8031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:48.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:48.831+0000 7f85ca000640 1 -- 172.21.15.107:0/3919055597 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f85c00aa8a0 con 0x7f85cc1055d0 2024-07-26T12:44:49.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.055+0000 7f85d3000640 1 -- 172.21.15.107:0/3919055597 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f85cc063230 con 0x7f85b40b3ca0 2024-07-26T12:44:49.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.103+0000 7f85ca000640 1 -- 172.21.15.107:0/3919055597 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f85cc063230 con 0x7f85b40b3ca0 2024-07-26T12:44:49.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.106+0000 7f85d3000640 1 -- 172.21.15.107:0/3919055597 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85b40b3ca0 msgr2=0x7f85b40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:49.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.106+0000 7f85d3000640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85b40b3ca0 0x7f85b40b61b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f85b8009230 tx=0x7f85b8031400 comp rx=0 tx=0).stop 2024-07-26T12:44:49.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.106+0000 7f85d3000640 1 -- 172.21.15.107:0/3919055597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85cc1055d0 msgr2=0x7f85cc10f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:49.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.106+0000 7f85d3000640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85cc1055d0 0x7f85cc10f590 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f85c000d770 tx=0x7f85c000dc30 comp rx=0 tx=0).stop 2024-07-26T12:44:49.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.106+0000 7f85d3000640 1 -- 172.21.15.107:0/3919055597 shutdown_connections 2024-07-26T12:44:49.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.106+0000 7f85d3000640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85b40b3ca0 0x7f85b40b61b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:49.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.106+0000 7f85d3000640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f85cc10b860 0x7f85cc110010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:49.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.106+0000 7f85d3000640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85cc10af10 0x7f85cc10fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:49.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.106+0000 7f85d3000640 1 --2- 172.21.15.107:0/3919055597 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f85cc1055d0 0x7f85cc10f590 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:49.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.106+0000 7f85d3000640 1 -- 172.21.15.107:0/3919055597 >> 172.21.15.107:0/3919055597 conn(0x7f85cc101040 msgr2=0x7f85cc103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:49.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.107+0000 7f85d3000640 1 -- 172.21.15.107:0/3919055597 shutdown_connections 2024-07-26T12:44:49.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:49.107+0000 7f85d3000640 1 -- 172.21.15.107:0/3919055597 wait complete. 2024-07-26T12:44:49.512 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:44:49 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:44:49.069+0000 7f1f78200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T12:44:49.513 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:44:49 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:44:49.069+0000 7f1f78200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T12:44:49.513 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:44:49 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:44:49.069+0000 7f1f78200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T12:44:49.513 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:44:49 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:44:49.069+0000 7f1f78200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T12:44:49.513 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:44:49 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:44:49.069+0000 7f1f78200640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T12:44:50.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-07-26T12:44:50.080 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:49 smithi107 ceph-mon[32659]: pgmap v355: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.4 KiB/s wr, 20 op/s 2024-07-26T12:44:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:49 smithi112 ceph-mon[37501]: pgmap v355: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.4 KiB/s wr, 20 op/s 2024-07-26T12:44:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:49 smithi132 ceph-mon[37580]: pgmap v355: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.4 KiB/s wr, 20 op/s 2024-07-26T12:44:50.489 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:50.955 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:50 smithi107 ceph-mon[32659]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:44:50.955 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:50 smithi107 ceph-mon[32659]: pgmap v356: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.6 KiB/s wr, 18 op/s 2024-07-26T12:44:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:50 smithi112 ceph-mon[37501]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:44:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:50 smithi112 ceph-mon[37501]: pgmap v356: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.6 KiB/s wr, 18 op/s 2024-07-26T12:44:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:50 smithi132 ceph-mon[37580]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:44:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:50 smithi132 ceph-mon[37580]: pgmap v356: 129 pgs: 129 active+clean; 587 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.6 KiB/s wr, 18 op/s 2024-07-26T12:44:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.831+0000 7fc82be00640 1 -- 172.21.15.107:0/292337220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc82c10f940 msgr2=0x7fc82c111d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.831+0000 7fc82be00640 1 --2- 172.21.15.107:0/292337220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc82c10f940 0x7fc82c111d80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc824009900 tx=0x7fc82402a400 comp rx=0 tx=0).stop 2024-07-26T12:44:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.831+0000 7fc82be00640 1 -- 172.21.15.107:0/292337220 shutdown_connections 2024-07-26T12:44:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.831+0000 7fc82be00640 1 --2- 172.21.15.107:0/292337220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc82c10f940 0x7fc82c111d80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.831+0000 7fc82be00640 1 --2- 172.21.15.107:0/292337220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c075b30 0x7fc82c10f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.831+0000 7fc82be00640 1 --2- 172.21.15.107:0/292337220 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc82c0751e0 0x7fc82c0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.831+0000 7fc82be00640 1 -- 172.21.15.107:0/292337220 >> 172.21.15.107:0/292337220 conn(0x7fc82c0fee70 msgr2=0x7fc82c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.831+0000 7fc82be00640 1 -- 172.21.15.107:0/292337220 shutdown_connections 2024-07-26T12:44:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.831+0000 7fc82be00640 1 -- 172.21.15.107:0/292337220 wait complete. 2024-07-26T12:44:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.833+0000 7fc82be00640 1 Processor -- start 2024-07-26T12:44:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.833+0000 7fc82be00640 1 -- start start 2024-07-26T12:44:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.833+0000 7fc82be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c0751e0 0x7fc82c19beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.833+0000 7fc82be00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc82c075b30 0x7fc82c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.833+0000 7fc82be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc82c10f940 0x7fc82c1a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82be00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c115520 con 0x7fc82c075b30 2024-07-26T12:44:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82be00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c1153a0 con 0x7fc82c10f940 2024-07-26T12:44:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c1156a0 con 0x7fc82c0751e0 2024-07-26T12:44:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82a000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc82c075b30 0x7fc82c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82aa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c0751e0 0x7fc82c19beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82a000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc82c075b30 0x7fc82c19c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59914/0 (socket says 172.21.15.107:59914) 2024-07-26T12:44:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82a000640 1 -- 172.21.15.107:0/3333098623 learned_addr learned my addr 172.21.15.107:0/3333098623 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82b400640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc82c10f940 0x7fc82c1a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82aa00640 1 -- 172.21.15.107:0/3333098623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc82c10f940 msgr2=0x7fc82c1a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82aa00640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc82c10f940 0x7fc82c1a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82aa00640 1 -- 172.21.15.107:0/3333098623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc82c075b30 msgr2=0x7fc82c19c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82aa00640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc82c075b30 0x7fc82c19c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.834+0000 7fc82aa00640 1 -- 172.21.15.107:0/3333098623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82c1a1e20 con 0x7fc82c0751e0 2024-07-26T12:44:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.835+0000 7fc82aa00640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c0751e0 0x7fc82c19beb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00e9f0 tx=0x7fc81c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.835+0000 7fc823400640 1 -- 172.21.15.107:0/3333098623 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c00ccc0 con 0x7fc82c0751e0 2024-07-26T12:44:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.835+0000 7fc82be00640 1 -- 172.21.15.107:0/3333098623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc82c1a2110 con 0x7fc82c0751e0 2024-07-26T12:44:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.835+0000 7fc823400640 1 -- 172.21.15.107:0/3333098623 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc81c0108e0 con 0x7fc82c0751e0 2024-07-26T12:44:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.835+0000 7fc823400640 1 -- 172.21.15.107:0/3333098623 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81c018880 con 0x7fc82c0751e0 2024-07-26T12:44:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.835+0000 7fc82be00640 1 -- 172.21.15.107:0/3333098623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc82c1a8870 con 0x7fc82c0751e0 2024-07-26T12:44:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.836+0000 7fc82be00640 1 -- 172.21.15.107:0/3333098623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc82c0755f0 con 0x7fc82c0751e0 2024-07-26T12:44:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.839+0000 7fc823400640 1 -- 172.21.15.107:0/3333098623 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fc81c010430 con 0x7fc82c0751e0 2024-07-26T12:44:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.841+0000 7fc823400640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc80c0b3cb0 0x7fc80c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.841+0000 7fc823400640 1 -- 172.21.15.107:0/3333098623 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc81c0e4ad0 con 0x7fc82c0751e0 2024-07-26T12:44:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.841+0000 7fc82a000640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc80c0b3cb0 0x7fc80c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.842+0000 7fc82a000640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc80c0b3cb0 0x7fc80c0b61c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc8140099a0 tx=0x7fc814009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:52.842+0000 7fc823400640 1 -- 172.21.15.107:0/3333098623 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc81c0ab470 con 0x7fc82c0751e0 2024-07-26T12:44:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:52 smithi107 ceph-mon[32659]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T12:44:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:52 smithi107 ceph-mon[32659]: pgmap v357: 129 pgs: 129 active+clean; 593 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 2.3 KiB/s wr, 20 op/s 2024-07-26T12:44:53.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.077+0000 7fc82be00640 1 -- 172.21.15.107:0/3333098623 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc82c109b60 con 0x7fc80c0b3cb0 2024-07-26T12:44:53.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.117+0000 7fc823400640 1 -- 172.21.15.107:0/3333098623 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc82c109b60 con 0x7fc80c0b3cb0 2024-07-26T12:44:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.119+0000 7fc82be00640 1 -- 172.21.15.107:0/3333098623 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc80c0b3cb0 msgr2=0x7fc80c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.119+0000 7fc82be00640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc80c0b3cb0 0x7fc80c0b61c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc8140099a0 tx=0x7fc814009500 comp rx=0 tx=0).stop 2024-07-26T12:44:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.119+0000 7fc82be00640 1 -- 172.21.15.107:0/3333098623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c0751e0 msgr2=0x7fc82c19beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.119+0000 7fc82be00640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c0751e0 0x7fc82c19beb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00e9f0 tx=0x7fc81c00eeb0 comp rx=0 tx=0).stop 2024-07-26T12:44:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.120+0000 7fc82be00640 1 -- 172.21.15.107:0/3333098623 shutdown_connections 2024-07-26T12:44:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.120+0000 7fc82be00640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc80c0b3cb0 0x7fc80c0b61c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.120+0000 7fc82be00640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc82c10f940 0x7fc82c1a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.120+0000 7fc82be00640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc82c075b30 0x7fc82c19c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:53.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.120+0000 7fc82be00640 1 --2- 172.21.15.107:0/3333098623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc82c0751e0 0x7fc82c19beb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.120+0000 7fc82be00640 1 -- 172.21.15.107:0/3333098623 >> 172.21.15.107:0/3333098623 conn(0x7fc82c0fee70 msgr2=0x7fc82c109450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.120+0000 7fc82be00640 1 -- 172.21.15.107:0/3333098623 shutdown_connections 2024-07-26T12:44:53.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:53.120+0000 7fc82be00640 1 -- 172.21.15.107:0/3333098623 wait complete. 2024-07-26T12:44:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:52 smithi112 ceph-mon[37501]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T12:44:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:52 smithi112 ceph-mon[37501]: pgmap v357: 129 pgs: 129 active+clean; 593 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 2.3 KiB/s wr, 20 op/s 2024-07-26T12:44:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:52 smithi132 ceph-mon[37580]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T12:44:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:52 smithi132 ceph-mon[37580]: pgmap v357: 129 pgs: 129 active+clean; 593 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 2.3 KiB/s wr, 20 op/s 2024-07-26T12:44:54.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-07-26T12:44:54.178 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:53 smithi107 ceph-mon[32659]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:44:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:53 smithi112 ceph-mon[37501]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:44:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:53 smithi132 ceph-mon[37580]: from='client.24907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:44:54.574 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:55.006 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:54 smithi107 ceph-mon[32659]: pgmap v358: 129 pgs: 129 active+clean; 594 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-07-26T12:44:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:54 smithi112 ceph-mon[37501]: pgmap v358: 129 pgs: 129 active+clean; 594 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-07-26T12:44:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:54 smithi132 ceph-mon[37580]: pgmap v358: 129 pgs: 129 active+clean; 594 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-07-26T12:44:57.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.063+0000 7f5aa6e00640 1 -- 172.21.15.107:0/541152906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5aa0102000 msgr2=0x7f5aa0102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:57.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.063+0000 7f5aa6e00640 1 --2- 172.21.15.107:0/541152906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5aa0102000 0x7f5aa0102410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5a90009900 tx=0x7f5a9002a480 comp rx=0 tx=0).stop 2024-07-26T12:44:57.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.063+0000 7f5aa6e00640 1 -- 172.21.15.107:0/541152906 shutdown_connections 2024-07-26T12:44:57.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.063+0000 7f5aa6e00640 1 --2- 172.21.15.107:0/541152906 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5aa010fbb0 0x7f5aa0111fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:57.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.063+0000 7f5aa6e00640 1 --2- 172.21.15.107:0/541152906 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa0102950 0x7f5aa010f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:57.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.063+0000 7f5aa6e00640 1 --2- 172.21.15.107:0/541152906 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5aa0102000 0x7f5aa0102410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:57.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.063+0000 7f5aa6e00640 1 -- 172.21.15.107:0/541152906 >> 172.21.15.107:0/541152906 conn(0x7f5aa0079160 msgr2=0x7f5aa0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:57.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.063+0000 7f5aa6e00640 1 -- 172.21.15.107:0/541152906 shutdown_connections 2024-07-26T12:44:57.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.063+0000 7f5aa6e00640 1 -- 172.21.15.107:0/541152906 wait complete. 2024-07-26T12:44:57.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.065+0000 7f5aa6e00640 1 Processor -- start 2024-07-26T12:44:57.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.065+0000 7f5aa6e00640 1 -- start start 2024-07-26T12:44:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.066+0000 7f5aa6e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5aa0102000 0x7f5aa019c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.066+0000 7f5aa6e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5aa0102950 0x7f5aa019c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.066+0000 7f5aa6e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa010fbb0 0x7f5aa01a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.066+0000 7f5aa6e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa01156b0 con 0x7f5aa010fbb0 2024-07-26T12:44:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.066+0000 7f5aa6e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa0115530 con 0x7f5aa0102950 2024-07-26T12:44:57.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.066+0000 7f5aa6e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa0115830 con 0x7f5aa0102000 2024-07-26T12:44:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.066+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5aa0102000 0x7f5aa019c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.066+0000 7f5a97400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5aa0102950 0x7f5aa019c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.066+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5aa0102000 0x7f5aa019c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:52650/0 (socket says 172.21.15.107:52650) 2024-07-26T12:44:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.066+0000 7f5a9fe00640 1 -- 172.21.15.107:0/775835596 learned_addr learned my addr 172.21.15.107:0/775835596 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:44:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.067+0000 7f5aa4e00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa010fbb0 0x7f5aa01a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:57.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.067+0000 7f5a9fe00640 1 -- 172.21.15.107:0/775835596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5aa0102950 msgr2=0x7f5aa019c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.067+0000 7f5a9fe00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5aa0102950 0x7f5aa019c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.067+0000 7f5a9fe00640 1 -- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa010fbb0 msgr2=0x7f5aa01a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.067+0000 7f5a9fe00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa010fbb0 0x7f5aa01a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.067+0000 7f5a9fe00640 1 -- 172.21.15.107:0/775835596 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa01a1fd0 con 0x7f5aa0102000 2024-07-26T12:44:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.067+0000 7f5a97400640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5aa0102950 0x7f5aa019c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:44:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.067+0000 7f5aa4e00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa010fbb0 0x7f5aa01a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:44:57.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.068+0000 7f5a9fe00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5aa0102000 0x7f5aa019c060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5a90005ab0 tx=0x7f5a90003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.068+0000 7f5a9d600640 1 -- 172.21.15.107:0/775835596 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9003f070 con 0x7f5aa0102000 2024-07-26T12:44:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.068+0000 7f5aa6e00640 1 -- 172.21.15.107:0/775835596 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa01a2260 con 0x7f5aa0102000 2024-07-26T12:44:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.068+0000 7f5aa6e00640 1 -- 172.21.15.107:0/775835596 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa01a8a20 con 0x7f5aa0102000 2024-07-26T12:44:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.068+0000 7f5a9d600640 1 -- 172.21.15.107:0/775835596 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a90031b60 con 0x7f5aa0102000 2024-07-26T12:44:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.068+0000 7f5a9d600640 1 -- 172.21.15.107:0/775835596 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a90049060 con 0x7f5aa0102000 2024-07-26T12:44:57.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.069+0000 7f5aa6e00640 1 -- 172.21.15.107:0/775835596 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aa0102410 con 0x7f5aa0102000 2024-07-26T12:44:57.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.072+0000 7f5a9d600640 1 -- 172.21.15.107:0/775835596 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f5a90042070 con 0x7f5aa0102000 2024-07-26T12:44:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.073+0000 7f5a9d600640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5a840b3d30 0x7f5a840b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:44:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.073+0000 7f5a9d600640 1 -- 172.21.15.107:0/775835596 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5a90104c70 con 0x7f5aa0102000 2024-07-26T12:44:57.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.074+0000 7f5a97400640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5a840b3d30 0x7f5a840b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:44:57.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.074+0000 7f5a97400640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5a840b3d30 0x7f5a840b6240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a880095f0 tx=0x7f5a88005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:44:57.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.075+0000 7f5a9d600640 1 -- 172.21.15.107:0/775835596 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5a900cb610 con 0x7f5aa0102000 2024-07-26T12:44:57.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.301+0000 7f5aa6e00640 1 -- 172.21.15.107:0/775835596 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f5aa01116a0 con 0x7f5a840b3d30 2024-07-26T12:44:57.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.312+0000 7f5a9d600640 1 -- 172.21.15.107:0/775835596 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f5aa01116a0 con 0x7f5a840b3d30 2024-07-26T12:44:57.314 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/qa/sv_0/0422c588-6e0f-4ad5-a831-f7efaf84c035 2024-07-26T12:44:57.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 -- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5a840b3d30 msgr2=0x7f5a840b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:57.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5a840b3d30 0x7f5a840b6240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a880095f0 tx=0x7f5a88005f90 comp rx=0 tx=0).stop 2024-07-26T12:44:57.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 -- 172.21.15.107:0/775835596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5aa0102000 msgr2=0x7f5aa019c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:44:57.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5aa0102000 0x7f5aa019c060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5a90005ab0 tx=0x7f5a90003c10 comp rx=0 tx=0).stop 2024-07-26T12:44:57.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 -- 172.21.15.107:0/775835596 shutdown_connections 2024-07-26T12:44:57.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5a840b3d30 0x7f5a840b6240 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:57.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5aa010fbb0 0x7f5aa01a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:57.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5aa0102950 0x7f5aa019c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:57.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 --2- 172.21.15.107:0/775835596 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5aa0102000 0x7f5aa019c060 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:44:57.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 -- 172.21.15.107:0/775835596 >> 172.21.15.107:0/775835596 conn(0x7f5aa0079160 msgr2=0x7f5aa0110f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:44:57.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 -- 172.21.15.107:0/775835596 shutdown_connections 2024-07-26T12:44:57.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:44:57.315+0000 7f5aa6e00640 1 -- 172.21.15.107:0/775835596 wait complete. 2024-07-26T12:44:58.010 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:57 smithi107 ceph-mon[32659]: pgmap v359: 129 pgs: 129 active+clean; 594 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-07-26T12:44:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:57 smithi132 ceph-mon[37580]: pgmap v359: 129 pgs: 129 active+clean; 594 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-07-26T12:44:58.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-07-26T12:44:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:57 smithi112 ceph-mon[37501]: pgmap v359: 129 pgs: 129 active+clean; 594 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-07-26T12:44:58.578 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:44:59.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:58 smithi107 ceph-mon[32659]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:44:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:58 smithi132 ceph-mon[37580]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:44:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:58 smithi112 ceph-mon[37501]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:44:59.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:44:59 smithi107 ceph-mon[32659]: pgmap v360: 129 pgs: 129 active+clean; 613 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.0 KiB/s wr, 12 op/s 2024-07-26T12:45:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:44:59 smithi132 ceph-mon[37580]: pgmap v360: 129 pgs: 129 active+clean; 613 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.0 KiB/s wr, 12 op/s 2024-07-26T12:45:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:44:59 smithi112 ceph-mon[37501]: pgmap v360: 129 pgs: 129 active+clean; 613 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.0 KiB/s wr, 12 op/s 2024-07-26T12:45:00.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.870+0000 7f4a99600640 1 -- 172.21.15.107:0/1590177183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a9410ba70 msgr2=0x7f4a941122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.870+0000 7f4a99600640 1 --2- 172.21.15.107:0/1590177183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a9410ba70 0x7f4a941122a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c009900 tx=0x7f4a8c02a400 comp rx=0 tx=0).stop 2024-07-26T12:45:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.870+0000 7f4a99600640 1 -- 172.21.15.107:0/1590177183 shutdown_connections 2024-07-26T12:45:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.870+0000 7f4a99600640 1 --2- 172.21.15.107:0/1590177183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a9410ba70 0x7f4a941122a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.870+0000 7f4a99600640 1 --2- 172.21.15.107:0/1590177183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a9410b120 0x7f4a9410b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.870+0000 7f4a99600640 1 --2- 172.21.15.107:0/1590177183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a941057e0 0x7f4a94105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.870+0000 7f4a99600640 1 -- 172.21.15.107:0/1590177183 >> 172.21.15.107:0/1590177183 conn(0x7f4a941011f0 msgr2=0x7f4a94103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.870+0000 7f4a99600640 1 -- 172.21.15.107:0/1590177183 shutdown_connections 2024-07-26T12:45:00.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.870+0000 7f4a99600640 1 -- 172.21.15.107:0/1590177183 wait complete. 2024-07-26T12:45:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.872+0000 7f4a99600640 1 Processor -- start 2024-07-26T12:45:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.872+0000 7f4a99600640 1 -- start start 2024-07-26T12:45:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.872+0000 7f4a99600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a941057e0 0x7f4a9419c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:00.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.872+0000 7f4a99600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a9410b120 0x7f4a9419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.872+0000 7f4a99600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a9410ba70 0x7f4a941a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.872+0000 7f4a99600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4a94115740 con 0x7f4a9410b120 2024-07-26T12:45:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.872+0000 7f4a99600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941155c0 con 0x7f4a9410ba70 2024-07-26T12:45:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.872+0000 7f4a99600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4a941158c0 con 0x7f4a941057e0 2024-07-26T12:45:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a92000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a9410b120 0x7f4a9419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a93400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a9410ba70 0x7f4a941a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a92000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a9410b120 0x7f4a9419c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60070/0 (socket says 172.21.15.107:60070) 2024-07-26T12:45:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a92000640 1 -- 172.21.15.107:0/2562331192 learned_addr learned my addr 172.21.15.107:0/2562331192 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:00.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a92a00640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a941057e0 0x7f4a9419c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a93400640 1 -- 172.21.15.107:0/2562331192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a941057e0 msgr2=0x7f4a9419c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a93400640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a941057e0 0x7f4a9419c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a93400640 1 -- 172.21.15.107:0/2562331192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a9410b120 msgr2=0x7f4a9419c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:00.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a93400640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a9410b120 0x7f4a9419c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a93400640 1 -- 172.21.15.107:0/2562331192 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a941a2020 con 0x7f4a9410ba70 2024-07-26T12:45:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.873+0000 7f4a93400640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a9410ba70 0x7f4a941a18f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c005940 tx=0x7f4a8c004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.874+0000 7f4a8b400640 1 -- 172.21.15.107:0/2562331192 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c03f030 con 0x7f4a9410ba70 2024-07-26T12:45:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.874+0000 7f4a99600640 1 -- 172.21.15.107:0/2562331192 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a941a22b0 con 0x7f4a9410ba70 2024-07-26T12:45:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.874+0000 7f4a8b400640 1 -- 172.21.15.107:0/2562331192 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a8c004440 con 0x7f4a9410ba70 2024-07-26T12:45:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.874+0000 7f4a8b400640 1 -- 172.21.15.107:0/2562331192 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c035070 con 0x7f4a9410ba70 2024-07-26T12:45:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.874+0000 7f4a99600640 1 -- 172.21.15.107:0/2562331192 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a941a8a70 con 0x7f4a9410ba70 2024-07-26T12:45:00.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.875+0000 7f4a99600640 1 -- 172.21.15.107:0/2562331192 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a94105bf0 con 0x7f4a9410ba70 2024-07-26T12:45:00.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.877+0000 7f4a8b400640 1 -- 172.21.15.107:0/2562331192 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f4a8c042030 con 0x7f4a9410ba70 2024-07-26T12:45:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.879+0000 7f4a8b400640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4a780b3d30 0x7f4a780b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.879+0000 7f4a8b400640 1 -- 172.21.15.107:0/2562331192 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4a8c1044f0 con 0x7f4a9410ba70 2024-07-26T12:45:00.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.879+0000 7f4a92a00640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4a780b3d30 0x7f4a780b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:00.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.880+0000 7f4a92a00640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4a780b3d30 0x7f4a780b6240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4a84005fd0 tx=0x7f4a84009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:00.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:00.883+0000 7f4a8b400640 1 -- 172.21.15.107:0/2562331192 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4a8c0cae90 con 0x7f4a9410ba70 2024-07-26T12:45:01.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.110+0000 7f4a99600640 1 -- 172.21.15.107:0/2562331192 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4a9410d8c0 con 0x7f4a780b3d30 2024-07-26T12:45:01.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.144+0000 7f4a8b400640 1 -- 172.21.15.107:0/2562331192 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4a9410d8c0 con 0x7f4a780b3d30 2024-07-26T12:45:01.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.148+0000 7f4a99600640 1 -- 172.21.15.107:0/2562331192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4a780b3d30 msgr2=0x7f4a780b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:01.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.148+0000 7f4a99600640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4a780b3d30 0x7f4a780b6240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4a84005fd0 tx=0x7f4a84009500 comp rx=0 tx=0).stop 2024-07-26T12:45:01.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.148+0000 7f4a99600640 1 -- 172.21.15.107:0/2562331192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a9410ba70 msgr2=0x7f4a941a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:01.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.148+0000 7f4a99600640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a9410ba70 0x7f4a941a18f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c005940 tx=0x7f4a8c004040 comp rx=0 tx=0).stop 2024-07-26T12:45:01.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.148+0000 7f4a99600640 1 -- 172.21.15.107:0/2562331192 shutdown_connections 2024-07-26T12:45:01.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.149+0000 7f4a99600640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4a780b3d30 0x7f4a780b6240 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:01.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.149+0000 7f4a99600640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4a9410ba70 0x7f4a941a18f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:01.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.149+0000 7f4a99600640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a9410b120 0x7f4a9419c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:01.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.149+0000 7f4a99600640 1 --2- 172.21.15.107:0/2562331192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4a941057e0 0x7f4a9419c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:01.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.149+0000 7f4a99600640 1 -- 172.21.15.107:0/2562331192 >> 172.21.15.107:0/2562331192 conn(0x7f4a941011f0 msgr2=0x7f4a9410a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.149+0000 7f4a99600640 1 -- 172.21.15.107:0/2562331192 shutdown_connections 2024-07-26T12:45:01.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:01.149+0000 7f4a99600640 1 -- 172.21.15.107:0/2562331192 wait complete. 2024-07-26T12:45:01.963 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:01 smithi107 ceph-mon[32659]: pgmap v361: 129 pgs: 129 active+clean; 613 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-07-26T12:45:02.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-07-26T12:45:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:01 smithi132 ceph-mon[37580]: pgmap v361: 129 pgs: 129 active+clean; 613 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-07-26T12:45:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:01 smithi112 ceph-mon[37501]: pgmap v361: 129 pgs: 129 active+clean; 613 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-07-26T12:45:02.491 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:02.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:02 smithi107 ceph-mon[32659]: from='client.24905 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:45:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:02 smithi132 ceph-mon[37580]: from='client.24905 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:45:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:02 smithi112 ceph-mon[37501]: from='client.24905 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:45:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:03 smithi132 ceph-mon[37580]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-07-26T12:45:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:03 smithi112 ceph-mon[37501]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-07-26T12:45:04.258 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:03 smithi107 ceph-mon[32659]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-07-26T12:45:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.825+0000 7f804d200640 1 -- 172.21.15.107:0/1756325193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048109870 msgr2=0x7f80481100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.825+0000 7f804d200640 1 --2- 172.21.15.107:0/1756325193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048109870 0x7f80481100a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8040009ca0 tx=0x7f804002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:45:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.826+0000 7f804d200640 1 -- 172.21.15.107:0/1756325193 shutdown_connections 2024-07-26T12:45:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.826+0000 7f804d200640 1 --2- 172.21.15.107:0/1756325193 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048109870 0x7f80481100a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.826+0000 7f804d200640 1 --2- 172.21.15.107:0/1756325193 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8048108f20 0x7f8048109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.826+0000 7f804d200640 1 --2- 172.21.15.107:0/1756325193 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f80481035e0 0x7f80481039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.826+0000 7f804d200640 1 -- 172.21.15.107:0/1756325193 >> 172.21.15.107:0/1756325193 conn(0x7f80480ff030 msgr2=0x7f8048101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:04.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.826+0000 7f804d200640 1 -- 172.21.15.107:0/1756325193 shutdown_connections 2024-07-26T12:45:04.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.826+0000 7f804d200640 1 -- 172.21.15.107:0/1756325193 wait complete. 2024-07-26T12:45:04.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.827+0000 7f804d200640 1 Processor -- start 2024-07-26T12:45:04.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.827+0000 7f804d200640 1 -- start start 2024-07-26T12:45:04.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.827+0000 7f804d200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80481035e0 0x7f8048199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:04.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.827+0000 7f804d200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8048108f20 0x7f804819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:04.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.827+0000 7f804d200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048109870 0x7f804819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.827+0000 7f804d200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8048113570 con 0x7f8048109870 2024-07-26T12:45:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.827+0000 7f804d200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f80481133f0 con 0x7f80481035e0 2024-07-26T12:45:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f804d200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f80481136f0 con 0x7f8048108f20 2024-07-26T12:45:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f8045e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80481035e0 0x7f8048199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f8046800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048109870 0x7f804819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f8045e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80481035e0 0x7f8048199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:55070/0 (socket says 172.21.15.107:55070) 2024-07-26T12:45:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f8045e00640 1 -- 172.21.15.107:0/2296095861 learned_addr learned my addr 172.21.15.107:0/2296095861 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f803d400640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8048108f20 0x7f804819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f803d400640 1 -- 172.21.15.107:0/2296095861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80481035e0 msgr2=0x7f8048199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f803d400640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80481035e0 0x7f8048199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f803d400640 1 -- 172.21.15.107:0/2296095861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048109870 msgr2=0x7f804819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f803d400640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048109870 0x7f804819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.828+0000 7f803d400640 1 -- 172.21.15.107:0/2296095861 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f804819fdf0 con 0x7f8048108f20 2024-07-26T12:45:04.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.829+0000 7f803d400640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8048108f20 0x7f804819a420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f804806c3a0 tx=0x7f803000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:04.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.829+0000 7f803f400640 1 -- 172.21.15.107:0/2296095861 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803000cc40 con 0x7f8048108f20 2024-07-26T12:45:04.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.829+0000 7f804d200640 1 -- 172.21.15.107:0/2296095861 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80481a00e0 con 0x7f8048108f20 2024-07-26T12:45:04.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.829+0000 7f803f400640 1 -- 172.21.15.107:0/2296095861 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f803000cde0 con 0x7f8048108f20 2024-07-26T12:45:04.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.829+0000 7f803f400640 1 -- 172.21.15.107:0/2296095861 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030018760 con 0x7f8048108f20 2024-07-26T12:45:04.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.829+0000 7f804d200640 1 -- 172.21.15.107:0/2296095861 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80481a6840 con 0x7f8048108f20 2024-07-26T12:45:04.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.830+0000 7f804d200640 1 -- 172.21.15.107:0/2296095861 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80481039f0 con 0x7f8048108f20 2024-07-26T12:45:04.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.832+0000 7f803f400640 1 -- 172.21.15.107:0/2296095861 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f8030010a90 con 0x7f8048108f20 2024-07-26T12:45:04.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.834+0000 7f803f400640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f802c0b3cb0 0x7f802c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:04.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.834+0000 7f803f400640 1 -- 172.21.15.107:0/2296095861 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f80300e4530 con 0x7f8048108f20 2024-07-26T12:45:04.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.834+0000 7f8045e00640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f802c0b3cb0 0x7f802c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:04.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.835+0000 7f8045e00640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f802c0b3cb0 0x7f802c0b61c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8038009c90 tx=0x7f8038005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:04.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:04.838+0000 7f803f400640 1 -- 172.21.15.107:0/2296095861 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f80300aaf30 con 0x7f8048108f20 2024-07-26T12:45:05.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.071+0000 7f804d200640 1 -- 172.21.15.107:0/2296095861 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f8048063230 con 0x7f802c0b3cb0 2024-07-26T12:45:05.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.079+0000 7f803f400640 1 -- 172.21.15.107:0/2296095861 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8048063230 con 0x7f802c0b3cb0 2024-07-26T12:45:05.081 INFO:teuthology.orchestra.run.smithi107.stdout:/volumes/qa/sv_1/ff915441-bb90-467d-b025-a122c34b612f 2024-07-26T12:45:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.081+0000 7f804d200640 1 -- 172.21.15.107:0/2296095861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f802c0b3cb0 msgr2=0x7f802c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.081+0000 7f804d200640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f802c0b3cb0 0x7f802c0b61c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8038009c90 tx=0x7f8038005f90 comp rx=0 tx=0).stop 2024-07-26T12:45:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.081+0000 7f804d200640 1 -- 172.21.15.107:0/2296095861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8048108f20 msgr2=0x7f804819a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.081+0000 7f804d200640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8048108f20 0x7f804819a420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f804806c3a0 tx=0x7f803000ee50 comp rx=0 tx=0).stop 2024-07-26T12:45:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.082+0000 7f804d200640 1 -- 172.21.15.107:0/2296095861 shutdown_connections 2024-07-26T12:45:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.082+0000 7f804d200640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f802c0b3cb0 0x7f802c0b61c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.082+0000 7f804d200640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8048109870 0x7f804819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.082+0000 7f804d200640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8048108f20 0x7f804819a420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.082+0000 7f804d200640 1 --2- 172.21.15.107:0/2296095861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f80481035e0 0x7f8048199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.082+0000 7f804d200640 1 -- 172.21.15.107:0/2296095861 >> 172.21.15.107:0/2296095861 conn(0x7f80480ff030 msgr2=0x7f80481074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.082+0000 7f804d200640 1 -- 172.21.15.107:0/2296095861 shutdown_connections 2024-07-26T12:45:05.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:05.082+0000 7f804d200640 1 -- 172.21.15.107:0/2296095861 wait complete. 2024-07-26T12:45:05.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:45:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:05 smithi132 ceph-mon[37580]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T12:45:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:05 smithi112 ceph-mon[37501]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T12:45:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:05 smithi107 ceph-mon[32659]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T12:45:06.346 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:06 smithi112 ceph-mon[37501]: from='client.24925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:45:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:06 smithi107 ceph-mon[32659]: from='client.24925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:45:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:06 smithi132 ceph-mon[37580]: from='client.24925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:45:08.024 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:07 smithi107 ceph-mon[32659]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T12:45:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:07 smithi112 ceph-mon[37501]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T12:45:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:07 smithi132 ceph-mon[37580]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T12:45:08.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.607+0000 7f33dbe00640 1 -- 172.21.15.107:0/3671044720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33dc1099f0 msgr2=0x7f33dc109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.607+0000 7f33dbe00640 1 --2- 172.21.15.107:0/3671044720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33dc1099f0 0x7f33dc109e00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009900 tx=0x7f33cc02a410 comp rx=0 tx=0).stop 2024-07-26T12:45:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.607+0000 7f33dbe00640 1 -- 172.21.15.107:0/3671044720 shutdown_connections 2024-07-26T12:45:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.607+0000 7f33dbe00640 1 --2- 172.21.15.107:0/3671044720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33dc10fc80 0x7f33dc1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.607+0000 7f33dbe00640 1 --2- 172.21.15.107:0/3671044720 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc10f330 0x7f33dc10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.607+0000 7f33dbe00640 1 --2- 172.21.15.107:0/3671044720 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33dc1099f0 0x7f33dc109e00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.607+0000 7f33dbe00640 1 -- 172.21.15.107:0/3671044720 >> 172.21.15.107:0/3671044720 conn(0x7f33dc078ff0 msgr2=0x7f33dc107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.608+0000 7f33dbe00640 1 -- 172.21.15.107:0/3671044720 shutdown_connections 2024-07-26T12:45:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.608+0000 7f33dbe00640 1 -- 172.21.15.107:0/3671044720 wait complete. 2024-07-26T12:45:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.609+0000 7f33dbe00640 1 Processor -- start 2024-07-26T12:45:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.609+0000 7f33dbe00640 1 -- start start 2024-07-26T12:45:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.609+0000 7f33dbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33dc1099f0 0x7f33dc1a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.609+0000 7f33dbe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc10f330 0x7f33dc1a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.609+0000 7f33dbe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33dc10fc80 0x7f33dc1a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.609+0000 7f33dbe00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc119990 con 0x7f33dc10fc80 2024-07-26T12:45:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.609+0000 7f33dbe00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc119810 con 0x7f33dc10f330 2024-07-26T12:45:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.609+0000 7f33dbe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc119b10 con 0x7f33dc1099f0 2024-07-26T12:45:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33daa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33dc1099f0 0x7f33dc1a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33db400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33dc10fc80 0x7f33dc1a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33d3e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc10f330 0x7f33dc1a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33daa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33dc1099f0 0x7f33dc1a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53594/0 (socket says 172.21.15.107:53594) 2024-07-26T12:45:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33daa00640 1 -- 172.21.15.107:0/244589176 learned_addr learned my addr 172.21.15.107:0/244589176 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33daa00640 1 -- 172.21.15.107:0/244589176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc10f330 msgr2=0x7f33dc1a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33daa00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc10f330 0x7f33dc1a0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33daa00640 1 -- 172.21.15.107:0/244589176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33dc10fc80 msgr2=0x7f33dc1a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33daa00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33dc10fc80 0x7f33dc1a5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33daa00640 1 -- 172.21.15.107:0/244589176 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc1a62d0 con 0x7f33dc1099f0 2024-07-26T12:45:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.610+0000 7f33db400640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33dc10fc80 0x7f33dc1a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:45:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.611+0000 7f33daa00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33dc1099f0 0x7f33dc1a0360 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f33cc005ab0 tx=0x7f33cc003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.611+0000 7f33d3400640 1 -- 172.21.15.107:0/244589176 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc03f070 con 0x7f33dc1099f0 2024-07-26T12:45:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.611+0000 7f33d3400640 1 -- 172.21.15.107:0/244589176 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33cc031b60 con 0x7f33dc1099f0 2024-07-26T12:45:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.611+0000 7f33dbe00640 1 -- 172.21.15.107:0/244589176 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33dc1a6560 con 0x7f33dc1099f0 2024-07-26T12:45:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.611+0000 7f33d3400640 1 -- 172.21.15.107:0/244589176 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc035030 con 0x7f33dc1099f0 2024-07-26T12:45:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.611+0000 7f33dbe00640 1 -- 172.21.15.107:0/244589176 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33dc1acd20 con 0x7f33dc1099f0 2024-07-26T12:45:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.612+0000 7f33dbe00640 1 -- 172.21.15.107:0/244589176 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33dc109e00 con 0x7f33dc1099f0 2024-07-26T12:45:08.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.615+0000 7f33d3400640 1 -- 172.21.15.107:0/244589176 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f33cc042070 con 0x7f33dc1099f0 2024-07-26T12:45:08.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.616+0000 7f33d3400640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f33c00b3cb0 0x7f33c00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:08.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.616+0000 7f33d3400640 1 -- 172.21.15.107:0/244589176 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f33cc105070 con 0x7f33dc1099f0 2024-07-26T12:45:08.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.617+0000 7f33d3e00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f33c00b3cb0 0x7f33c00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:08.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.617+0000 7f33d3e00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f33c00b3cb0 0x7f33c00b61c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f33dc10fa50 tx=0x7f33c4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:08.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.618+0000 7f33d3400640 1 -- 172.21.15.107:0/244589176 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f33cc0cba10 con 0x7f33dc1099f0 2024-07-26T12:45:08.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.858+0000 7f33dbe00640 1 -- 172.21.15.107:0/244589176 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33dc111a80 con 0x7f33dc1099f0 2024-07-26T12:45:08.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.859+0000 7f33d3400640 1 -- 172.21.15.107:0/244589176 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11993 (secure 0 0 0) 0x7f33cc0cff60 con 0x7f33dc1099f0 2024-07-26T12:45:08.861 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:45:08.861 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"btime":"2024-07-26T12:44:44:796279+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:44:43.812461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24716,"mds_1":24715,"mds_2":24703,"mds_3":24694,"mds_4":24682},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24632":{"gid":24632,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24634":{"gid":24634,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24652":{"gid":24652,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24662":{"gid":24662,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":32,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24694":{"gid":24694,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":31,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":29,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":28,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24716,"qdb_cluster":[24682,24694,24703,24716,24715]},"id":1}]} 2024-07-26T12:45:08.861 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-07-26T12:45:08.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 -- 172.21.15.107:0/244589176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f33c00b3cb0 msgr2=0x7f33c00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:08.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f33c00b3cb0 0x7f33c00b61c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f33dc10fa50 tx=0x7f33c4005f90 comp rx=0 tx=0).stop 2024-07-26T12:45:08.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 -- 172.21.15.107:0/244589176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33dc1099f0 msgr2=0x7f33dc1a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:08.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33dc1099f0 0x7f33dc1a0360 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f33cc005ab0 tx=0x7f33cc003c10 comp rx=0 tx=0).stop 2024-07-26T12:45:08.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 -- 172.21.15.107:0/244589176 shutdown_connections 2024-07-26T12:45:08.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f33c00b3cb0 0x7f33c00b61c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:08.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33dc10fc80 0x7f33dc1a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f33dc10f330 0x7f33dc1a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 --2- 172.21.15.107:0/244589176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f33dc1099f0 0x7f33dc1a0360 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 -- 172.21.15.107:0/244589176 >> 172.21.15.107:0/244589176 conn(0x7f33dc078ff0 msgr2=0x7f33dc0796f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 -- 172.21.15.107:0/244589176 shutdown_connections 2024-07-26T12:45:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:08.862+0000 7f33dbe00640 1 -- 172.21.15.107:0/244589176 wait complete. 2024-07-26T12:45:09.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:45:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:09 smithi107 ceph-mon[32659]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 4.9 KiB/s wr, 13 op/s 2024-07-26T12:45:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:09 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/244589176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:45:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:09 smithi112 ceph-mon[37501]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 4.9 KiB/s wr, 13 op/s 2024-07-26T12:45:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:09 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/244589176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:45:10.209 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:09 smithi132 ceph-mon[37580]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 4.9 KiB/s wr, 13 op/s 2024-07-26T12:45:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:09 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/244589176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:45:10.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:10 smithi107 ceph-mon[32659]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-07-26T12:45:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:10 smithi112 ceph-mon[37501]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-07-26T12:45:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:10 smithi132 ceph-mon[37580]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.0 KiB/s wr, 11 op/s 2024-07-26T12:45:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.461+0000 7f3607e00640 1 -- 172.21.15.107:0/858334536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3608109670 msgr2=0x7f360810fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.461+0000 7f3607e00640 1 --2- 172.21.15.107:0/858334536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3608109670 0x7f360810fea0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3600009ca0 tx=0x7f360002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:45:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.461+0000 7f3607e00640 1 -- 172.21.15.107:0/858334536 shutdown_connections 2024-07-26T12:45:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.461+0000 7f3607e00640 1 --2- 172.21.15.107:0/858334536 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3608109670 0x7f360810fea0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.461+0000 7f3607e00640 1 --2- 172.21.15.107:0/858334536 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3608108d20 0x7f3608109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.461+0000 7f3607e00640 1 --2- 172.21.15.107:0/858334536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36081033e0 0x7f36081037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:12.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.461+0000 7f3607e00640 1 -- 172.21.15.107:0/858334536 >> 172.21.15.107:0/858334536 conn(0x7f36080fee30 msgr2=0x7f3608101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:12.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.461+0000 7f3607e00640 1 -- 172.21.15.107:0/858334536 shutdown_connections 2024-07-26T12:45:12.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.461+0000 7f3607e00640 1 -- 172.21.15.107:0/858334536 wait complete. 2024-07-26T12:45:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.462+0000 7f3607e00640 1 Processor -- start 2024-07-26T12:45:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f3607e00640 1 -- start start 2024-07-26T12:45:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f3607e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36081033e0 0x7f3608199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f3607e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3608108d20 0x7f360819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f3607e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3608109670 0x7f360819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f3607e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3608113390 con 0x7f36081033e0 2024-07-26T12:45:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f3607e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3608113210 con 0x7f3608109670 2024-07-26T12:45:12.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f3607e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3608113510 con 0x7f3608108d20 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f35fe000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3608108d20 0x7f360819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f3606a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36081033e0 0x7f3608199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f35fe000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3608108d20 0x7f360819a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53610/0 (socket says 172.21.15.107:53610) 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f35fe000640 1 -- 172.21.15.107:0/2917546336 learned_addr learned my addr 172.21.15.107:0/2917546336 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.463+0000 7f3607400640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3608109670 0x7f360819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.464+0000 7f35fe000640 1 -- 172.21.15.107:0/2917546336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3608109670 msgr2=0x7f360819f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.464+0000 7f35fe000640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3608109670 0x7f360819f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.464+0000 7f35fe000640 1 -- 172.21.15.107:0/2917546336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36081033e0 msgr2=0x7f3608199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.464+0000 7f35fe000640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36081033e0 0x7f3608199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.464+0000 7f35fe000640 1 -- 172.21.15.107:0/2917546336 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360819fcb0 con 0x7f3608108d20 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.464+0000 7f3607400640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3608109670 0x7f360819f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:45:12.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.464+0000 7f35fe000640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3608108d20 0x7f360819a2e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f35f000e9c0 tx=0x7f35f000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:12.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.465+0000 7f35ffe00640 1 -- 172.21.15.107:0/2917546336 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f000ccc0 con 0x7f3608108d20 2024-07-26T12:45:12.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.465+0000 7f35ffe00640 1 -- 172.21.15.107:0/2917546336 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35f00054d0 con 0x7f3608108d20 2024-07-26T12:45:12.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.465+0000 7f3607e00640 1 -- 172.21.15.107:0/2917546336 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f360819ffa0 con 0x7f3608108d20 2024-07-26T12:45:12.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.465+0000 7f35ffe00640 1 -- 172.21.15.107:0/2917546336 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f0010430 con 0x7f3608108d20 2024-07-26T12:45:12.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.465+0000 7f3607e00640 1 -- 172.21.15.107:0/2917546336 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36081a6700 con 0x7f3608108d20 2024-07-26T12:45:12.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.466+0000 7f3607e00640 1 -- 172.21.15.107:0/2917546336 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36081037f0 con 0x7f3608108d20 2024-07-26T12:45:12.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.468+0000 7f35ffe00640 1 -- 172.21.15.107:0/2917546336 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f35f00105d0 con 0x7f3608108d20 2024-07-26T12:45:12.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.470+0000 7f35ffe00640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35ec0b3d30 0x7f35ec0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:12.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.470+0000 7f35ffe00640 1 -- 172.21.15.107:0/2917546336 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f35f00e4530 con 0x7f3608108d20 2024-07-26T12:45:12.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.470+0000 7f3606a00640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35ec0b3d30 0x7f35ec0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:12.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.471+0000 7f3606a00640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35ec0b3d30 0x7f35ec0b6240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009bd0 tx=0x7f35f8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:12.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.473+0000 7f35ffe00640 1 -- 172.21.15.107:0/2917546336 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f35f00ab580 con 0x7f3608108d20 2024-07-26T12:45:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:12.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.679+0000 7f3607e00640 1 -- 172.21.15.107:0/2917546336 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f360810aed0 con 0x7f3608108d20 2024-07-26T12:45:12.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.680+0000 7f35ffe00640 1 -- 172.21.15.107:0/2917546336 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25203 (secure 0 0 0) 0x7f35f00aae80 con 0x7f3608108d20 2024-07-26T12:45:12.683 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:45:12.683 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:44:36.797948+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:45:12.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 -- 172.21.15.107:0/2917546336 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35ec0b3d30 msgr2=0x7f35ec0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:12.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35ec0b3d30 0x7f35ec0b6240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f35f8009bd0 tx=0x7f35f8005f90 comp rx=0 tx=0).stop 2024-07-26T12:45:12.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 -- 172.21.15.107:0/2917546336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3608108d20 msgr2=0x7f360819a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3608108d20 0x7f360819a2e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f35f000e9c0 tx=0x7f35f000ee80 comp rx=0 tx=0).stop 2024-07-26T12:45:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 -- 172.21.15.107:0/2917546336 shutdown_connections 2024-07-26T12:45:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35ec0b3d30 0x7f35ec0b6240 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3608109670 0x7f360819f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3608108d20 0x7f360819a2e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 --2- 172.21.15.107:0/2917546336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36081033e0 0x7f3608199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 -- 172.21.15.107:0/2917546336 >> 172.21.15.107:0/2917546336 conn(0x7f36080fee30 msgr2=0x7f3608108330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.683+0000 7f3607e00640 1 -- 172.21.15.107:0/2917546336 shutdown_connections 2024-07-26T12:45:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:12.684+0000 7f3607e00640 1 -- 172.21.15.107:0/2917546336 wait complete. 2024-07-26T12:45:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:13.506 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T12:45:13.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T12:45:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:13 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2917546336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:45:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:13 smithi107 ceph-mon[32659]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-07-26T12:45:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:13 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2917546336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:45:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:13 smithi112 ceph-mon[37501]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-07-26T12:45:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:13 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2917546336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:45:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:13 smithi132 ceph-mon[37580]: pgmap v367: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 3.6 KiB/s wr, 13 op/s 2024-07-26T12:45:14.114 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:14 smithi112 ceph-mon[37501]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-07-26T12:45:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:14 smithi107 ceph-mon[32659]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-07-26T12:45:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:14 smithi132 ceph-mon[37580]: pgmap v368: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-07-26T12:45:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.779+0000 7f5c65200640 1 -- 172.21.15.107:0/2807051238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c60108ef0 msgr2=0x7f5c60109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:16.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.779+0000 7f5c65200640 1 --2- 172.21.15.107:0/2807051238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c60108ef0 0x7f5c60109300 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5c48009900 tx=0x7f5c4802a410 comp rx=0 tx=0).stop 2024-07-26T12:45:16.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.779+0000 7f5c65200640 1 -- 172.21.15.107:0/2807051238 shutdown_connections 2024-07-26T12:45:16.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.779+0000 7f5c65200640 1 --2- 172.21.15.107:0/2807051238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60109840 0x7f5c60110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:16.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.779+0000 7f5c65200640 1 --2- 172.21.15.107:0/2807051238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c60108ef0 0x7f5c60109300 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:16.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.779+0000 7f5c65200640 1 --2- 172.21.15.107:0/2807051238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c601035b0 0x7f5c601039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:16.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.779+0000 7f5c65200640 1 -- 172.21.15.107:0/2807051238 >> 172.21.15.107:0/2807051238 conn(0x7f5c600fefc0 msgr2=0x7f5c60101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:16.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.779+0000 7f5c65200640 1 -- 172.21.15.107:0/2807051238 shutdown_connections 2024-07-26T12:45:16.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.779+0000 7f5c65200640 1 -- 172.21.15.107:0/2807051238 wait complete. 2024-07-26T12:45:16.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.780+0000 7f5c65200640 1 Processor -- start 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.780+0000 7f5c65200640 1 -- start start 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c65200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c601035b0 0x7f5c6019c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c65200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60108ef0 0x7f5c6019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c65200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c60109840 0x7f5c601a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c65200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5c60113500 con 0x7f5c60108ef0 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c65200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5c60113380 con 0x7f5c60109840 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c65200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5c60113680 con 0x7f5c601035b0 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c55400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60108ef0 0x7f5c6019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c5de00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c601035b0 0x7f5c6019c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c5de00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c601035b0 0x7f5c6019c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33272/0 (socket says 172.21.15.107:33272) 2024-07-26T12:45:16.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c5de00640 1 -- 172.21.15.107:0/1279355348 learned_addr learned my addr 172.21.15.107:0/1279355348 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.781+0000 7f5c5e800640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c60109840 0x7f5c601a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.782+0000 7f5c5de00640 1 -- 172.21.15.107:0/1279355348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c60109840 msgr2=0x7f5c601a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.782+0000 7f5c5de00640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c60109840 0x7f5c601a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.782+0000 7f5c5de00640 1 -- 172.21.15.107:0/1279355348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60108ef0 msgr2=0x7f5c6019c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.782+0000 7f5c5de00640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60108ef0 0x7f5c6019c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:16.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.782+0000 7f5c5de00640 1 -- 172.21.15.107:0/1279355348 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c601a20a0 con 0x7f5c601035b0 2024-07-26T12:45:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.782+0000 7f5c5de00640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c601035b0 0x7f5c6019c130 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000d950 tx=0x7f5c5000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.782+0000 7f5c57400640 1 -- 172.21.15.107:0/1279355348 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c50017070 con 0x7f5c601035b0 2024-07-26T12:45:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.782+0000 7f5c57400640 1 -- 172.21.15.107:0/1279355348 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c50009c30 con 0x7f5c601035b0 2024-07-26T12:45:16.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.783+0000 7f5c57400640 1 -- 172.21.15.107:0/1279355348 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5000f620 con 0x7f5c601035b0 2024-07-26T12:45:16.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.783+0000 7f5c65200640 1 -- 172.21.15.107:0/1279355348 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c601a2390 con 0x7f5c601035b0 2024-07-26T12:45:16.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.783+0000 7f5c65200640 1 -- 172.21.15.107:0/1279355348 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c601a8af0 con 0x7f5c601035b0 2024-07-26T12:45:16.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.784+0000 7f5c65200640 1 -- 172.21.15.107:0/1279355348 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c60104830 con 0x7f5c601035b0 2024-07-26T12:45:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.786+0000 7f5c57400640 1 -- 172.21.15.107:0/1279355348 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f5c50005030 con 0x7f5c601035b0 2024-07-26T12:45:16.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.788+0000 7f5c57400640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5c440b3d30 0x7f5c440b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.788+0000 7f5c57400640 1 -- 172.21.15.107:0/1279355348 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5c500e3da0 con 0x7f5c601035b0 2024-07-26T12:45:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.788+0000 7f5c55400640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5c440b3d30 0x7f5c440b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:16.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.789+0000 7f5c55400640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5c440b3d30 0x7f5c440b6240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5c4802aa00 tx=0x7f5c48005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:16.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:16.789+0000 7f5c57400640 1 -- 172.21.15.107:0/1279355348 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5c50009de0 con 0x7f5c601035b0 2024-07-26T12:45:17.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.019+0000 7f5c65200640 1 -- 172.21.15.107:0/1279355348 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5c60063230 con 0x7f5c440b3d30 2024-07-26T12:45:17.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.028+0000 7f5c57400640 1 -- 172.21.15.107:0/1279355348 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340392 (secure 0 0 0) 0x7f5c60063230 con 0x7f5c440b3d30 2024-07-26T12:45:17.034 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:45:17.036 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-07-26T12:45:17.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.033+0000 7f5c65200640 1 -- 172.21.15.107:0/1279355348 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5c440b3d30 msgr2=0x7f5c440b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.033+0000 7f5c65200640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5c440b3d30 0x7f5c440b6240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5c4802aa00 tx=0x7f5c48005a20 comp rx=0 tx=0).stop 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.033+0000 7f5c65200640 1 -- 172.21.15.107:0/1279355348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c601035b0 msgr2=0x7f5c6019c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.033+0000 7f5c65200640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c601035b0 0x7f5c6019c130 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000d950 tx=0x7f5c5000de10 comp rx=0 tx=0).stop 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.033+0000 7f5c65200640 1 -- 172.21.15.107:0/1279355348 shutdown_connections 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.033+0000 7f5c65200640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5c440b3d30 0x7f5c440b6240 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.033+0000 7f5c65200640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5c60109840 0x7f5c601a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.033+0000 7f5c65200640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c60108ef0 0x7f5c6019c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.033+0000 7f5c65200640 1 --2- 172.21.15.107:0/1279355348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c601035b0 0x7f5c6019c130 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.034+0000 7f5c65200640 1 -- 172.21.15.107:0/1279355348 >> 172.21.15.107:0/1279355348 conn(0x7f5c600fefc0 msgr2=0x7f5c601074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.034+0000 7f5c65200640 1 -- 172.21.15.107:0/1279355348 shutdown_connections 2024-07-26T12:45:17.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:17.034+0000 7f5c65200640 1 -- 172.21.15.107:0/1279355348 wait complete. 2024-07-26T12:45:17.865 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-07-26T12:45:16.705935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661942,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":459,"num_read_kb":331,"num_write":376,"num_write_kb":2296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2951874,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":310,"ondisk_log_size":310,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376060,"kb_used_data":14424,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565764,"statfs":{"total":1151940427776,"available":1151555342336,"internally_reserved":0,"allocated":14770176,"data_stored":8378575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11656,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":69,"num_write":13,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006182"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":97,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:15.953478+0000","last_change":"2024-07-26T12:43:08.129417+0000","last_active":"2024-07-26T12:45:15.953478+0000","last_peered":"2024-07-26T12:45:15.953478+0000","last_clean":"2024-07-26T12:45:15.953478+0000","last_became_active":"2024-07-26T12:43:08.128919+0000","last_became_peered":"2024-07-26T12:43:08.128919+0000","last_unstale":"2024-07-26T12:45:15.953478+0000","last_undegraded":"2024-07-26T12:45:15.953478+0000","last_fullsized":"2024-07-26T12:45:15.953478+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:35:55.414185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805971+0000","last_change":"2024-07-26T12:43:12.172058+0000","last_active":"2024-07-26T12:44:36.805971+0000","last_peered":"2024-07-26T12:44:36.805971+0000","last_clean":"2024-07-26T12:44:36.805971+0000","last_became_active":"2024-07-26T12:43:12.171631+0000","last_became_peered":"2024-07-26T12:43:12.171631+0000","last_unstale":"2024-07-26T12:44:36.805971+0000","last_undegraded":"2024-07-26T12:44:36.805971+0000","last_fullsized":"2024-07-26T12:44:36.805971+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:51.972962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.802628+0000","last_change":"2024-07-26T12:43:26.773739+0000","last_active":"2024-07-26T12:44:36.802628+0000","last_peered":"2024-07-26T12:44:36.802628+0000","last_clean":"2024-07-26T12:44:36.802628+0000","last_became_active":"2024-07-26T12:43:26.773254+0000","last_became_peered":"2024-07-26T12:43:26.773254+0000","last_unstale":"2024-07-26T12:44:36.802628+0000","last_undegraded":"2024-07-26T12:44:36.802628+0000","last_fullsized":"2024-07-26T12:44:36.802628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:03.125497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.852995+0000","last_change":"2024-07-26T12:43:08.227329+0000","last_active":"2024-07-26T12:44:36.852995+0000","last_peered":"2024-07-26T12:44:36.852995+0000","last_clean":"2024-07-26T12:44:36.852995+0000","last_became_active":"2024-07-26T12:43:08.226674+0000","last_became_peered":"2024-07-26T12:43:08.226674+0000","last_unstale":"2024-07-26T12:44:36.852995+0000","last_undegraded":"2024-07-26T12:44:36.852995+0000","last_fullsized":"2024-07-26T12:44:36.852995+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.222684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813652+0000","last_change":"2024-07-26T12:43:12.190616+0000","last_active":"2024-07-26T12:44:36.813652+0000","last_peered":"2024-07-26T12:44:36.813652+0000","last_clean":"2024-07-26T12:44:36.813652+0000","last_became_active":"2024-07-26T12:43:12.189717+0000","last_became_peered":"2024-07-26T12:43:12.189717+0000","last_unstale":"2024-07-26T12:44:36.813652+0000","last_undegraded":"2024-07-26T12:44:36.813652+0000","last_fullsized":"2024-07-26T12:44:36.813652+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:03.509592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850258+0000","last_change":"2024-07-26T12:43:26.774486+0000","last_active":"2024-07-26T12:44:36.850258+0000","last_peered":"2024-07-26T12:44:36.850258+0000","last_clean":"2024-07-26T12:44:36.850258+0000","last_became_active":"2024-07-26T12:43:26.774046+0000","last_became_peered":"2024-07-26T12:43:26.774046+0000","last_unstale":"2024-07-26T12:44:36.850258+0000","last_undegraded":"2024-07-26T12:44:36.850258+0000","last_fullsized":"2024-07-26T12:44:36.850258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.847072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.868396+0000","last_change":"2024-07-26T12:43:08.123084+0000","last_active":"2024-07-26T12:44:39.868396+0000","last_peered":"2024-07-26T12:44:39.868396+0000","last_clean":"2024-07-26T12:44:39.868396+0000","last_became_active":"2024-07-26T12:43:08.122371+0000","last_became_peered":"2024-07-26T12:43:08.122371+0000","last_unstale":"2024-07-26T12:44:39.868396+0000","last_undegraded":"2024-07-26T12:44:39.868396+0000","last_fullsized":"2024-07-26T12:44:39.868396+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:27:09.665660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807575+0000","last_change":"2024-07-26T12:43:12.170290+0000","last_active":"2024-07-26T12:44:36.807575+0000","last_peered":"2024-07-26T12:44:36.807575+0000","last_clean":"2024-07-26T12:44:36.807575+0000","last_became_active":"2024-07-26T12:43:12.169849+0000","last_became_peered":"2024-07-26T12:43:12.169849+0000","last_unstale":"2024-07-26T12:44:36.807575+0000","last_undegraded":"2024-07-26T12:44:36.807575+0000","last_fullsized":"2024-07-26T12:44:36.807575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:46.044921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850501+0000","last_change":"2024-07-26T12:43:26.764804+0000","last_active":"2024-07-26T12:44:36.850501+0000","last_peered":"2024-07-26T12:44:36.850501+0000","last_clean":"2024-07-26T12:44:36.850501+0000","last_became_active":"2024-07-26T12:43:26.764065+0000","last_became_peered":"2024-07-26T12:43:26.764065+0000","last_unstale":"2024-07-26T12:44:36.850501+0000","last_undegraded":"2024-07-26T12:44:36.850501+0000","last_fullsized":"2024-07-26T12:44:36.850501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:16:59.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.874869+0000","last_change":"2024-07-26T12:43:08.121496+0000","last_active":"2024-07-26T12:44:38.874869+0000","last_peered":"2024-07-26T12:44:38.874869+0000","last_clean":"2024-07-26T12:44:38.874869+0000","last_became_active":"2024-07-26T12:43:08.120962+0000","last_became_peered":"2024-07-26T12:43:08.120962+0000","last_unstale":"2024-07-26T12:44:38.874869+0000","last_undegraded":"2024-07-26T12:44:38.874869+0000","last_fullsized":"2024-07-26T12:44:38.874869+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:42:02.998023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850048+0000","last_change":"2024-07-26T12:43:12.168707+0000","last_active":"2024-07-26T12:44:36.850048+0000","last_peered":"2024-07-26T12:44:36.850048+0000","last_clean":"2024-07-26T12:44:36.850048+0000","last_became_active":"2024-07-26T12:43:12.168338+0000","last_became_peered":"2024-07-26T12:43:12.168338+0000","last_unstale":"2024-07-26T12:44:36.850048+0000","last_undegraded":"2024-07-26T12:44:36.850048+0000","last_fullsized":"2024-07-26T12:44:36.850048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:39:44.724864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850427+0000","last_change":"2024-07-26T12:43:26.770863+0000","last_active":"2024-07-26T12:44:36.850427+0000","last_peered":"2024-07-26T12:44:36.850427+0000","last_clean":"2024-07-26T12:44:36.850427+0000","last_became_active":"2024-07-26T12:43:26.770480+0000","last_became_peered":"2024-07-26T12:43:26.770480+0000","last_unstale":"2024-07-26T12:44:36.850427+0000","last_undegraded":"2024-07-26T12:44:36.850427+0000","last_fullsized":"2024-07-26T12:44:36.850427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:09.128364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805647+0000","last_change":"2024-07-26T12:43:12.181659+0000","last_active":"2024-07-26T12:44:36.805647+0000","last_peered":"2024-07-26T12:44:36.805647+0000","last_clean":"2024-07-26T12:44:36.805647+0000","last_became_active":"2024-07-26T12:43:12.181268+0000","last_became_peered":"2024-07-26T12:43:12.181268+0000","last_unstale":"2024-07-26T12:44:36.805647+0000","last_undegraded":"2024-07-26T12:44:36.805647+0000","last_fullsized":"2024-07-26T12:44:36.805647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:38:28.445981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850732+0000","last_change":"2024-07-26T12:43:35.856273+0000","last_active":"2024-07-26T12:44:36.850732+0000","last_peered":"2024-07-26T12:44:36.850732+0000","last_clean":"2024-07-26T12:44:36.850732+0000","last_became_active":"2024-07-26T12:43:35.849625+0000","last_became_peered":"2024-07-26T12:43:35.849625+0000","last_unstale":"2024-07-26T12:44:36.850732+0000","last_undegraded":"2024-07-26T12:44:36.850732+0000","last_fullsized":"2024-07-26T12:44:36.850732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:34:45.261474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850624+0000","last_change":"2024-07-26T12:43:26.793431+0000","last_active":"2024-07-26T12:44:36.850624+0000","last_peered":"2024-07-26T12:44:36.850624+0000","last_clean":"2024-07-26T12:44:36.850624+0000","last_became_active":"2024-07-26T12:43:26.792931+0000","last_became_peered":"2024-07-26T12:43:26.792931+0000","last_unstale":"2024-07-26T12:44:36.850624+0000","last_undegraded":"2024-07-26T12:44:36.850624+0000","last_fullsized":"2024-07-26T12:44:36.850624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:31.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805205+0000","last_change":"2024-07-26T12:43:12.183784+0000","last_active":"2024-07-26T12:44:36.805205+0000","last_peered":"2024-07-26T12:44:36.805205+0000","last_clean":"2024-07-26T12:44:36.805205+0000","last_became_active":"2024-07-26T12:43:12.182696+0000","last_became_peered":"2024-07-26T12:43:12.182696+0000","last_unstale":"2024-07-26T12:44:36.805205+0000","last_undegraded":"2024-07-26T12:44:36.805205+0000","last_fullsized":"2024-07-26T12:44:36.805205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:33:19.918180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.873813+0000","last_change":"2024-07-26T12:43:08.131069+0000","last_active":"2024-07-26T12:44:37.873813+0000","last_peered":"2024-07-26T12:44:37.873813+0000","last_clean":"2024-07-26T12:44:37.873813+0000","last_became_active":"2024-07-26T12:43:08.130637+0000","last_became_peered":"2024-07-26T12:43:08.130637+0000","last_unstale":"2024-07-26T12:44:37.873813+0000","last_undegraded":"2024-07-26T12:44:37.873813+0000","last_fullsized":"2024-07-26T12:44:37.873813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:04:16.461900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850617+0000","last_change":"2024-07-26T12:43:26.763388+0000","last_active":"2024-07-26T12:44:36.850617+0000","last_peered":"2024-07-26T12:44:36.850617+0000","last_clean":"2024-07-26T12:44:36.850617+0000","last_became_active":"2024-07-26T12:43:26.762464+0000","last_became_peered":"2024-07-26T12:43:26.762464+0000","last_unstale":"2024-07-26T12:44:36.850617+0000","last_undegraded":"2024-07-26T12:44:36.850617+0000","last_fullsized":"2024-07-26T12:44:36.850617+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:10:17.598266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850536+0000","last_change":"2024-07-26T12:43:12.191640+0000","last_active":"2024-07-26T12:44:36.850536+0000","last_peered":"2024-07-26T12:44:36.850536+0000","last_clean":"2024-07-26T12:44:36.850536+0000","last_became_active":"2024-07-26T12:43:12.191260+0000","last_became_peered":"2024-07-26T12:43:12.191260+0000","last_unstale":"2024-07-26T12:44:36.850536+0000","last_undegraded":"2024-07-26T12:44:36.850536+0000","last_fullsized":"2024-07-26T12:44:36.850536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:47:18.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.876184+0000","last_change":"2024-07-26T12:43:08.135035+0000","last_active":"2024-07-26T12:44:38.876184+0000","last_peered":"2024-07-26T12:44:38.876184+0000","last_clean":"2024-07-26T12:44:38.876184+0000","last_became_active":"2024-07-26T12:43:08.134204+0000","last_became_peered":"2024-07-26T12:43:08.134204+0000","last_unstale":"2024-07-26T12:44:38.876184+0000","last_undegraded":"2024-07-26T12:44:38.876184+0000","last_fullsized":"2024-07-26T12:44:38.876184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:38.954460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840930+0000","last_change":"2024-07-26T12:43:35.853244+0000","last_active":"2024-07-26T12:44:36.840930+0000","last_peered":"2024-07-26T12:44:36.840930+0000","last_clean":"2024-07-26T12:44:36.840930+0000","last_became_active":"2024-07-26T12:43:35.852728+0000","last_became_peered":"2024-07-26T12:43:35.852728+0000","last_unstale":"2024-07-26T12:44:36.840930+0000","last_undegraded":"2024-07-26T12:44:36.840930+0000","last_fullsized":"2024-07-26T12:44:36.840930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:20:39.080000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.839777+0000","last_change":"2024-07-26T12:43:12.172457+0000","last_active":"2024-07-26T12:44:36.839777+0000","last_peered":"2024-07-26T12:44:36.839777+0000","last_clean":"2024-07-26T12:44:36.839777+0000","last_became_active":"2024-07-26T12:43:12.172034+0000","last_became_peered":"2024-07-26T12:43:12.172034+0000","last_unstale":"2024-07-26T12:44:36.839777+0000","last_undegraded":"2024-07-26T12:44:36.839777+0000","last_fullsized":"2024-07-26T12:44:36.839777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:37:02.237852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807446+0000","last_change":"2024-07-26T12:43:08.227152+0000","last_active":"2024-07-26T12:44:36.807446+0000","last_peered":"2024-07-26T12:44:36.807446+0000","last_clean":"2024-07-26T12:44:36.807446+0000","last_became_active":"2024-07-26T12:43:08.226722+0000","last_became_peered":"2024-07-26T12:43:08.226722+0000","last_unstale":"2024-07-26T12:44:36.807446+0000","last_undegraded":"2024-07-26T12:44:36.807446+0000","last_fullsized":"2024-07-26T12:44:36.807446+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:16.293211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850777+0000","last_change":"2024-07-26T12:43:26.792999+0000","last_active":"2024-07-26T12:44:36.850777+0000","last_peered":"2024-07-26T12:44:36.850777+0000","last_clean":"2024-07-26T12:44:36.850777+0000","last_became_active":"2024-07-26T12:43:26.792515+0000","last_became_peered":"2024-07-26T12:43:26.792515+0000","last_unstale":"2024-07-26T12:44:36.850777+0000","last_undegraded":"2024-07-26T12:44:36.850777+0000","last_fullsized":"2024-07-26T12:44:36.850777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:49:35.104600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813498+0000","last_change":"2024-07-26T12:43:12.185381+0000","last_active":"2024-07-26T12:44:36.813498+0000","last_peered":"2024-07-26T12:44:36.813498+0000","last_clean":"2024-07-26T12:44:36.813498+0000","last_became_active":"2024-07-26T12:43:12.184892+0000","last_became_peered":"2024-07-26T12:43:12.184892+0000","last_unstale":"2024-07-26T12:44:36.813498+0000","last_undegraded":"2024-07-26T12:44:36.813498+0000","last_fullsized":"2024-07-26T12:44:36.813498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:38:53.992586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.877700+0000","last_change":"2024-07-26T12:43:08.895079+0000","last_active":"2024-07-26T12:44:38.877700+0000","last_peered":"2024-07-26T12:44:38.877700+0000","last_clean":"2024-07-26T12:44:38.877700+0000","last_became_active":"2024-07-26T12:43:08.894653+0000","last_became_peered":"2024-07-26T12:43:08.894653+0000","last_unstale":"2024-07-26T12:44:38.877700+0000","last_undegraded":"2024-07-26T12:44:38.877700+0000","last_fullsized":"2024-07-26T12:44:38.877700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:02:53.137905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803224+0000","last_change":"2024-07-26T12:43:26.762802+0000","last_active":"2024-07-26T12:44:36.803224+0000","last_peered":"2024-07-26T12:44:36.803224+0000","last_clean":"2024-07-26T12:44:36.803224+0000","last_became_active":"2024-07-26T12:43:26.762211+0000","last_became_peered":"2024-07-26T12:43:26.762211+0000","last_unstale":"2024-07-26T12:44:36.803224+0000","last_undegraded":"2024-07-26T12:44:36.803224+0000","last_fullsized":"2024-07-26T12:44:36.803224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:35:19.623168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.806698+0000","last_change":"2024-07-26T12:43:12.176062+0000","last_active":"2024-07-26T12:44:36.806698+0000","last_peered":"2024-07-26T12:44:36.806698+0000","last_clean":"2024-07-26T12:44:36.806698+0000","last_became_active":"2024-07-26T12:43:12.175671+0000","last_became_peered":"2024-07-26T12:43:12.175671+0000","last_unstale":"2024-07-26T12:44:36.806698+0000","last_undegraded":"2024-07-26T12:44:36.806698+0000","last_fullsized":"2024-07-26T12:44:36.806698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:48.374964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'16","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:07.219585+0000","last_change":"2024-07-26T12:43:08.131453+0000","last_active":"2024-07-26T12:45:07.219585+0000","last_peered":"2024-07-26T12:45:07.219585+0000","last_clean":"2024-07-26T12:45:07.219585+0000","last_became_active":"2024-07-26T12:43:08.130928+0000","last_became_peered":"2024-07-26T12:43:08.130928+0000","last_unstale":"2024-07-26T12:45:07.219585+0000","last_undegraded":"2024-07-26T12:45:07.219585+0000","last_fullsized":"2024-07-26T12:45:07.219585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:43.190860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804235+0000","last_change":"2024-07-26T12:43:26.792942+0000","last_active":"2024-07-26T12:44:36.804235+0000","last_peered":"2024-07-26T12:44:36.804235+0000","last_clean":"2024-07-26T12:44:36.804235+0000","last_became_active":"2024-07-26T12:43:26.792045+0000","last_became_peered":"2024-07-26T12:43:26.792045+0000","last_unstale":"2024-07-26T12:44:36.804235+0000","last_undegraded":"2024-07-26T12:44:36.804235+0000","last_fullsized":"2024-07-26T12:44:36.804235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:56.005295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810937+0000","last_change":"2024-07-26T12:43:12.187601+0000","last_active":"2024-07-26T12:44:36.810937+0000","last_peered":"2024-07-26T12:44:36.810937+0000","last_clean":"2024-07-26T12:44:36.810937+0000","last_became_active":"2024-07-26T12:43:12.187175+0000","last_became_peered":"2024-07-26T12:43:12.187175+0000","last_unstale":"2024-07-26T12:44:36.810937+0000","last_undegraded":"2024-07-26T12:44:36.810937+0000","last_fullsized":"2024-07-26T12:44:36.810937+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:06.992854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881924+0000","last_change":"2024-07-26T12:43:08.227279+0000","last_active":"2024-07-26T12:44:39.881924+0000","last_peered":"2024-07-26T12:44:39.881924+0000","last_clean":"2024-07-26T12:44:39.881924+0000","last_became_active":"2024-07-26T12:43:08.226634+0000","last_became_peered":"2024-07-26T12:43:08.226634+0000","last_unstale":"2024-07-26T12:44:39.881924+0000","last_undegraded":"2024-07-26T12:44:39.881924+0000","last_fullsized":"2024-07-26T12:44:39.881924+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:19.845664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850980+0000","last_change":"2024-07-26T12:43:26.774537+0000","last_active":"2024-07-26T12:44:36.850980+0000","last_peered":"2024-07-26T12:44:36.850980+0000","last_clean":"2024-07-26T12:44:36.850980+0000","last_became_active":"2024-07-26T12:43:26.774036+0000","last_became_peered":"2024-07-26T12:43:26.774036+0000","last_unstale":"2024-07-26T12:44:36.850980+0000","last_undegraded":"2024-07-26T12:44:36.850980+0000","last_fullsized":"2024-07-26T12:44:36.850980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:21.774498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.857503+0000","last_change":"2024-07-26T12:44:36.857443+0000","last_active":"2024-07-26T12:44:36.857503+0000","last_peered":"2024-07-26T12:44:36.857503+0000","last_clean":"2024-07-26T12:44:36.857503+0000","last_became_active":"2024-07-26T12:44:36.857001+0000","last_became_peered":"2024-07-26T12:44:36.857001+0000","last_unstale":"2024-07-26T12:44:36.857503+0000","last_undegraded":"2024-07-26T12:44:36.857503+0000","last_fullsized":"2024-07-26T12:44:36.857503+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:08.236404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.875394+0000","last_change":"2024-07-26T12:43:08.227432+0000","last_active":"2024-07-26T12:44:37.875394+0000","last_peered":"2024-07-26T12:44:37.875394+0000","last_clean":"2024-07-26T12:44:37.875394+0000","last_became_active":"2024-07-26T12:43:08.226645+0000","last_became_peered":"2024-07-26T12:43:08.226645+0000","last_unstale":"2024-07-26T12:44:37.875394+0000","last_undegraded":"2024-07-26T12:44:37.875394+0000","last_fullsized":"2024-07-26T12:44:37.875394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:23.498855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840264+0000","last_change":"2024-07-26T12:43:26.792874+0000","last_active":"2024-07-26T12:44:36.840264+0000","last_peered":"2024-07-26T12:44:36.840264+0000","last_clean":"2024-07-26T12:44:36.840264+0000","last_became_active":"2024-07-26T12:43:26.792371+0000","last_became_peered":"2024-07-26T12:43:26.792371+0000","last_unstale":"2024-07-26T12:44:36.840264+0000","last_undegraded":"2024-07-26T12:44:36.840264+0000","last_fullsized":"2024-07-26T12:44:36.840264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:25.753111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809487+0000","last_change":"2024-07-26T12:43:12.185733+0000","last_active":"2024-07-26T12:44:36.809487+0000","last_peered":"2024-07-26T12:44:36.809487+0000","last_clean":"2024-07-26T12:44:36.809487+0000","last_became_active":"2024-07-26T12:43:12.185305+0000","last_became_peered":"2024-07-26T12:43:12.185305+0000","last_unstale":"2024-07-26T12:44:36.809487+0000","last_undegraded":"2024-07-26T12:44:36.809487+0000","last_fullsized":"2024-07-26T12:44:36.809487+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:44.634607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:11.947551+0000","last_change":"2024-07-26T12:43:08.118508+0000","last_active":"2024-07-26T12:45:11.947551+0000","last_peered":"2024-07-26T12:45:11.947551+0000","last_clean":"2024-07-26T12:45:11.947551+0000","last_became_active":"2024-07-26T12:43:08.118088+0000","last_became_peered":"2024-07-26T12:43:08.118088+0000","last_unstale":"2024-07-26T12:45:11.947551+0000","last_undegraded":"2024-07-26T12:45:11.947551+0000","last_fullsized":"2024-07-26T12:45:11.947551+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:23:46.474298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810206+0000","last_change":"2024-07-26T12:43:26.779328+0000","last_active":"2024-07-26T12:44:36.810206+0000","last_peered":"2024-07-26T12:44:36.810206+0000","last_clean":"2024-07-26T12:44:36.810206+0000","last_became_active":"2024-07-26T12:43:26.778912+0000","last_became_peered":"2024-07-26T12:43:26.778912+0000","last_unstale":"2024-07-26T12:44:36.810206+0000","last_undegraded":"2024-07-26T12:44:36.810206+0000","last_fullsized":"2024-07-26T12:44:36.810206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:58.364775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850823+0000","last_change":"2024-07-26T12:43:35.854997+0000","last_active":"2024-07-26T12:44:36.850823+0000","last_peered":"2024-07-26T12:44:36.850823+0000","last_clean":"2024-07-26T12:44:36.850823+0000","last_became_active":"2024-07-26T12:43:35.854387+0000","last_became_peered":"2024-07-26T12:43:35.854387+0000","last_unstale":"2024-07-26T12:44:36.850823+0000","last_undegraded":"2024-07-26T12:44:36.850823+0000","last_fullsized":"2024-07-26T12:44:36.850823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:04:20.489388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:12.968092+0000","last_change":"2024-07-26T12:43:08.227628+0000","last_active":"2024-07-26T12:45:12.968092+0000","last_peered":"2024-07-26T12:45:12.968092+0000","last_clean":"2024-07-26T12:45:12.968092+0000","last_became_active":"2024-07-26T12:43:08.227133+0000","last_became_peered":"2024-07-26T12:43:08.227133+0000","last_unstale":"2024-07-26T12:45:12.968092+0000","last_undegraded":"2024-07-26T12:45:12.968092+0000","last_fullsized":"2024-07-26T12:45:12.968092+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:28:28.484315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810768+0000","last_change":"2024-07-26T12:43:26.794016+0000","last_active":"2024-07-26T12:44:36.810768+0000","last_peered":"2024-07-26T12:44:36.810768+0000","last_clean":"2024-07-26T12:44:36.810768+0000","last_became_active":"2024-07-26T12:43:26.792562+0000","last_became_peered":"2024-07-26T12:43:26.792562+0000","last_unstale":"2024-07-26T12:44:36.810768+0000","last_undegraded":"2024-07-26T12:44:36.810768+0000","last_fullsized":"2024-07-26T12:44:36.810768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:26.162840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809072+0000","last_change":"2024-07-26T12:43:12.173014+0000","last_active":"2024-07-26T12:44:36.809072+0000","last_peered":"2024-07-26T12:44:36.809072+0000","last_clean":"2024-07-26T12:44:36.809072+0000","last_became_active":"2024-07-26T12:43:12.172600+0000","last_became_peered":"2024-07-26T12:43:12.172600+0000","last_unstale":"2024-07-26T12:44:36.809072+0000","last_undegraded":"2024-07-26T12:44:36.809072+0000","last_fullsized":"2024-07-26T12:44:36.809072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:59.842744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.872830+0000","last_change":"2024-07-26T12:43:08.119000+0000","last_active":"2024-07-26T12:44:38.872830+0000","last_peered":"2024-07-26T12:44:38.872830+0000","last_clean":"2024-07-26T12:44:38.872830+0000","last_became_active":"2024-07-26T12:43:08.118608+0000","last_became_peered":"2024-07-26T12:43:08.118608+0000","last_unstale":"2024-07-26T12:44:38.872830+0000","last_undegraded":"2024-07-26T12:44:38.872830+0000","last_fullsized":"2024-07-26T12:44:38.872830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:29:47.663449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851268+0000","last_change":"2024-07-26T12:43:35.914716+0000","last_active":"2024-07-26T12:44:36.851268+0000","last_peered":"2024-07-26T12:44:36.851268+0000","last_clean":"2024-07-26T12:44:36.851268+0000","last_became_active":"2024-07-26T12:43:35.914354+0000","last_became_peered":"2024-07-26T12:43:35.914354+0000","last_unstale":"2024-07-26T12:44:36.851268+0000","last_undegraded":"2024-07-26T12:44:36.851268+0000","last_fullsized":"2024-07-26T12:44:36.851268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:09:10.994141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805160+0000","last_change":"2024-07-26T12:43:12.169997+0000","last_active":"2024-07-26T12:44:36.805160+0000","last_peered":"2024-07-26T12:44:36.805160+0000","last_clean":"2024-07-26T12:44:36.805160+0000","last_became_active":"2024-07-26T12:43:12.169044+0000","last_became_peered":"2024-07-26T12:43:12.169044+0000","last_unstale":"2024-07-26T12:44:36.805160+0000","last_undegraded":"2024-07-26T12:44:36.805160+0000","last_fullsized":"2024-07-26T12:44:36.805160+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:41.791434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881551+0000","last_change":"2024-07-26T12:43:08.129025+0000","last_active":"2024-07-26T12:44:39.881551+0000","last_peered":"2024-07-26T12:44:39.881551+0000","last_clean":"2024-07-26T12:44:39.881551+0000","last_became_active":"2024-07-26T12:43:08.128608+0000","last_became_peered":"2024-07-26T12:43:08.128608+0000","last_unstale":"2024-07-26T12:44:39.881551+0000","last_undegraded":"2024-07-26T12:44:39.881551+0000","last_fullsized":"2024-07-26T12:44:39.881551+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:43.749473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":121,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805787+0000","last_change":"2024-07-26T12:41:42.411595+0000","last_active":"2024-07-26T12:44:36.805787+0000","last_peered":"2024-07-26T12:44:36.805787+0000","last_clean":"2024-07-26T12:44:36.805787+0000","last_became_active":"2024-07-26T12:41:42.407119+0000","last_became_peered":"2024-07-26T12:41:42.407119+0000","last_unstale":"2024-07-26T12:44:36.805787+0000","last_undegraded":"2024-07-26T12:44:36.805787+0000","last_fullsized":"2024-07-26T12:44:36.805787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_clean_scrub_stamp":"2024-07-26T12:37:59.605754+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:21.376418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805538+0000","last_change":"2024-07-26T12:43:26.763955+0000","last_active":"2024-07-26T12:44:36.805538+0000","last_peered":"2024-07-26T12:44:36.805538+0000","last_clean":"2024-07-26T12:44:36.805538+0000","last_became_active":"2024-07-26T12:43:26.763368+0000","last_became_peered":"2024-07-26T12:43:26.763368+0000","last_unstale":"2024-07-26T12:44:36.805538+0000","last_undegraded":"2024-07-26T12:44:36.805538+0000","last_fullsized":"2024-07-26T12:44:36.805538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:48:14.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805985+0000","last_change":"2024-07-26T12:43:12.174780+0000","last_active":"2024-07-26T12:44:36.805985+0000","last_peered":"2024-07-26T12:44:36.805985+0000","last_clean":"2024-07-26T12:44:36.805985+0000","last_became_active":"2024-07-26T12:43:12.174379+0000","last_became_peered":"2024-07-26T12:43:12.174379+0000","last_unstale":"2024-07-26T12:44:36.805985+0000","last_undegraded":"2024-07-26T12:44:36.805985+0000","last_fullsized":"2024-07-26T12:44:36.805985+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:42:22.783259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'6","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:12.942224+0000","last_change":"2024-07-26T12:43:08.894655+0000","last_active":"2024-07-26T12:45:12.942224+0000","last_peered":"2024-07-26T12:45:12.942224+0000","last_clean":"2024-07-26T12:45:12.942224+0000","last_became_active":"2024-07-26T12:43:08.894270+0000","last_became_peered":"2024-07-26T12:43:08.894270+0000","last_unstale":"2024-07-26T12:45:12.942224+0000","last_undegraded":"2024-07-26T12:45:12.942224+0000","last_fullsized":"2024-07-26T12:45:12.942224+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:34.431204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807841+0000","last_change":"2024-07-26T12:43:26.769716+0000","last_active":"2024-07-26T12:44:36.807841+0000","last_peered":"2024-07-26T12:44:36.807841+0000","last_clean":"2024-07-26T12:44:36.807841+0000","last_became_active":"2024-07-26T12:43:26.769154+0000","last_became_peered":"2024-07-26T12:43:26.769154+0000","last_unstale":"2024-07-26T12:44:36.807841+0000","last_undegraded":"2024-07-26T12:44:36.807841+0000","last_fullsized":"2024-07-26T12:44:36.807841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:43.102498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805540+0000","last_change":"2024-07-26T12:43:12.179488+0000","last_active":"2024-07-26T12:44:36.805540+0000","last_peered":"2024-07-26T12:44:36.805540+0000","last_clean":"2024-07-26T12:44:36.805540+0000","last_became_active":"2024-07-26T12:43:12.179171+0000","last_became_peered":"2024-07-26T12:43:12.179171+0000","last_unstale":"2024-07-26T12:44:36.805540+0000","last_undegraded":"2024-07-26T12:44:36.805540+0000","last_fullsized":"2024-07-26T12:44:36.805540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:43.695582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.877080+0000","last_change":"2024-07-26T12:43:08.130838+0000","last_active":"2024-07-26T12:44:38.877080+0000","last_peered":"2024-07-26T12:44:38.877080+0000","last_clean":"2024-07-26T12:44:38.877080+0000","last_became_active":"2024-07-26T12:43:08.130446+0000","last_became_peered":"2024-07-26T12:43:08.130446+0000","last_unstale":"2024-07-26T12:44:38.877080+0000","last_undegraded":"2024-07-26T12:44:38.877080+0000","last_fullsized":"2024-07-26T12:44:38.877080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:33:55.271038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.812600+0000","last_change":"2024-07-26T12:43:26.781982+0000","last_active":"2024-07-26T12:44:36.812600+0000","last_peered":"2024-07-26T12:44:36.812600+0000","last_clean":"2024-07-26T12:44:36.812600+0000","last_became_active":"2024-07-26T12:43:26.777102+0000","last_became_peered":"2024-07-26T12:43:26.777102+0000","last_unstale":"2024-07-26T12:44:36.812600+0000","last_undegraded":"2024-07-26T12:44:36.812600+0000","last_fullsized":"2024-07-26T12:44:36.812600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:49.553124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.806033+0000","last_change":"2024-07-26T12:43:12.177043+0000","last_active":"2024-07-26T12:44:36.806033+0000","last_peered":"2024-07-26T12:44:36.806033+0000","last_clean":"2024-07-26T12:44:36.806033+0000","last_became_active":"2024-07-26T12:43:12.176634+0000","last_became_peered":"2024-07-26T12:43:12.176634+0000","last_unstale":"2024-07-26T12:44:36.806033+0000","last_undegraded":"2024-07-26T12:44:36.806033+0000","last_fullsized":"2024-07-26T12:44:36.806033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:20.210176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.877336+0000","last_change":"2024-07-26T12:43:08.131279+0000","last_active":"2024-07-26T12:44:38.877336+0000","last_peered":"2024-07-26T12:44:38.877336+0000","last_clean":"2024-07-26T12:44:38.877336+0000","last_became_active":"2024-07-26T12:43:08.130755+0000","last_became_peered":"2024-07-26T12:43:08.130755+0000","last_unstale":"2024-07-26T12:44:38.877336+0000","last_undegraded":"2024-07-26T12:44:38.877336+0000","last_fullsized":"2024-07-26T12:44:38.877336+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:03.234905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.849647+0000","last_change":"2024-07-26T12:43:26.764407+0000","last_active":"2024-07-26T12:44:36.849647+0000","last_peered":"2024-07-26T12:44:36.849647+0000","last_clean":"2024-07-26T12:44:36.849647+0000","last_became_active":"2024-07-26T12:43:26.762408+0000","last_became_peered":"2024-07-26T12:43:26.762408+0000","last_unstale":"2024-07-26T12:44:36.849647+0000","last_undegraded":"2024-07-26T12:44:36.849647+0000","last_fullsized":"2024-07-26T12:44:36.849647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:35:29.139476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851249+0000","last_change":"2024-07-26T12:43:12.174587+0000","last_active":"2024-07-26T12:44:36.851249+0000","last_peered":"2024-07-26T12:44:36.851249+0000","last_clean":"2024-07-26T12:44:36.851249+0000","last_became_active":"2024-07-26T12:43:12.173929+0000","last_became_peered":"2024-07-26T12:43:12.173929+0000","last_unstale":"2024-07-26T12:44:36.851249+0000","last_undegraded":"2024-07-26T12:44:36.851249+0000","last_fullsized":"2024-07-26T12:44:36.851249+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:16.775198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.845786+0000","last_change":"2024-07-26T12:43:08.119825+0000","last_active":"2024-07-26T12:44:37.845786+0000","last_peered":"2024-07-26T12:44:37.845786+0000","last_clean":"2024-07-26T12:44:37.845786+0000","last_became_active":"2024-07-26T12:43:08.119273+0000","last_became_peered":"2024-07-26T12:43:08.119273+0000","last_unstale":"2024-07-26T12:44:37.845786+0000","last_undegraded":"2024-07-26T12:44:37.845786+0000","last_fullsized":"2024-07-26T12:44:37.845786+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:39.973020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.849943+0000","last_change":"2024-07-26T12:43:26.773226+0000","last_active":"2024-07-26T12:44:36.849943+0000","last_peered":"2024-07-26T12:44:36.849943+0000","last_clean":"2024-07-26T12:44:36.849943+0000","last_became_active":"2024-07-26T12:43:26.772745+0000","last_became_peered":"2024-07-26T12:43:26.772745+0000","last_unstale":"2024-07-26T12:44:36.849943+0000","last_undegraded":"2024-07-26T12:44:36.849943+0000","last_fullsized":"2024-07-26T12:44:36.849943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:35.407138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850158+0000","last_change":"2024-07-26T12:43:12.176178+0000","last_active":"2024-07-26T12:44:36.850158+0000","last_peered":"2024-07-26T12:44:36.850158+0000","last_clean":"2024-07-26T12:44:36.850158+0000","last_became_active":"2024-07-26T12:43:12.175791+0000","last_became_peered":"2024-07-26T12:43:12.175791+0000","last_unstale":"2024-07-26T12:44:36.850158+0000","last_undegraded":"2024-07-26T12:44:36.850158+0000","last_fullsized":"2024-07-26T12:44:36.850158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:48.885831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.874508+0000","last_change":"2024-07-26T12:43:08.226345+0000","last_active":"2024-07-26T12:44:38.874508+0000","last_peered":"2024-07-26T12:44:38.874508+0000","last_clean":"2024-07-26T12:44:38.874508+0000","last_became_active":"2024-07-26T12:43:08.225897+0000","last_became_peered":"2024-07-26T12:43:08.225897+0000","last_unstale":"2024-07-26T12:44:38.874508+0000","last_undegraded":"2024-07-26T12:44:38.874508+0000","last_fullsized":"2024-07-26T12:44:38.874508+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:59.351076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810637+0000","last_change":"2024-07-26T12:43:26.774800+0000","last_active":"2024-07-26T12:44:36.810637+0000","last_peered":"2024-07-26T12:44:36.810637+0000","last_clean":"2024-07-26T12:44:36.810637+0000","last_became_active":"2024-07-26T12:43:26.774192+0000","last_became_peered":"2024-07-26T12:43:26.774192+0000","last_unstale":"2024-07-26T12:44:36.810637+0000","last_undegraded":"2024-07-26T12:44:36.810637+0000","last_fullsized":"2024-07-26T12:44:36.810637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:18.379619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805223+0000","last_change":"2024-07-26T12:43:12.171720+0000","last_active":"2024-07-26T12:44:36.805223+0000","last_peered":"2024-07-26T12:44:36.805223+0000","last_clean":"2024-07-26T12:44:36.805223+0000","last_became_active":"2024-07-26T12:43:12.171334+0000","last_became_peered":"2024-07-26T12:43:12.171334+0000","last_unstale":"2024-07-26T12:44:36.805223+0000","last_undegraded":"2024-07-26T12:44:36.805223+0000","last_fullsized":"2024-07-26T12:44:36.805223+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:35.156430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:09.962169+0000","last_change":"2024-07-26T12:43:08.120325+0000","last_active":"2024-07-26T12:45:09.962169+0000","last_peered":"2024-07-26T12:45:09.962169+0000","last_clean":"2024-07-26T12:45:09.962169+0000","last_became_active":"2024-07-26T12:43:08.119896+0000","last_became_peered":"2024-07-26T12:43:08.119896+0000","last_unstale":"2024-07-26T12:45:09.962169+0000","last_undegraded":"2024-07-26T12:45:09.962169+0000","last_fullsized":"2024-07-26T12:45:09.962169+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:23:19.470491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.856160+0000","last_change":"2024-07-26T12:43:26.778688+0000","last_active":"2024-07-26T12:44:36.856160+0000","last_peered":"2024-07-26T12:44:36.856160+0000","last_clean":"2024-07-26T12:44:36.856160+0000","last_became_active":"2024-07-26T12:43:26.778239+0000","last_became_peered":"2024-07-26T12:43:26.778239+0000","last_unstale":"2024-07-26T12:44:36.856160+0000","last_undegraded":"2024-07-26T12:44:36.856160+0000","last_fullsized":"2024-07-26T12:44:36.856160+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:50:48.409636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805469+0000","last_change":"2024-07-26T12:43:12.182323+0000","last_active":"2024-07-26T12:44:36.805469+0000","last_peered":"2024-07-26T12:44:36.805469+0000","last_clean":"2024-07-26T12:44:36.805469+0000","last_became_active":"2024-07-26T12:43:12.181860+0000","last_became_peered":"2024-07-26T12:43:12.181860+0000","last_unstale":"2024-07-26T12:44:36.805469+0000","last_undegraded":"2024-07-26T12:44:36.805469+0000","last_fullsized":"2024-07-26T12:44:36.805469+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:08.764979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.877164+0000","last_change":"2024-07-26T12:43:08.130823+0000","last_active":"2024-07-26T12:44:38.877164+0000","last_peered":"2024-07-26T12:44:38.877164+0000","last_clean":"2024-07-26T12:44:38.877164+0000","last_became_active":"2024-07-26T12:43:08.130424+0000","last_became_peered":"2024-07-26T12:43:08.130424+0000","last_unstale":"2024-07-26T12:44:38.877164+0000","last_undegraded":"2024-07-26T12:44:38.877164+0000","last_fullsized":"2024-07-26T12:44:38.877164+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:49:35.829624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850635+0000","last_change":"2024-07-26T12:43:26.770070+0000","last_active":"2024-07-26T12:44:36.850635+0000","last_peered":"2024-07-26T12:44:36.850635+0000","last_clean":"2024-07-26T12:44:36.850635+0000","last_became_active":"2024-07-26T12:43:26.769594+0000","last_became_peered":"2024-07-26T12:43:26.769594+0000","last_unstale":"2024-07-26T12:44:36.850635+0000","last_undegraded":"2024-07-26T12:44:36.850635+0000","last_fullsized":"2024-07-26T12:44:36.850635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:18.342739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809942+0000","last_change":"2024-07-26T12:43:12.183134+0000","last_active":"2024-07-26T12:44:36.809942+0000","last_peered":"2024-07-26T12:44:36.809942+0000","last_clean":"2024-07-26T12:44:36.809942+0000","last_became_active":"2024-07-26T12:43:12.182406+0000","last_became_peered":"2024-07-26T12:43:12.182406+0000","last_unstale":"2024-07-26T12:44:36.809942+0000","last_undegraded":"2024-07-26T12:44:36.809942+0000","last_fullsized":"2024-07-26T12:44:36.809942+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:54:24.038971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881522+0000","last_change":"2024-07-26T12:43:35.862005+0000","last_active":"2024-07-26T12:44:39.881522+0000","last_peered":"2024-07-26T12:44:39.881522+0000","last_clean":"2024-07-26T12:44:39.881522+0000","last_became_active":"2024-07-26T12:43:35.850911+0000","last_became_peered":"2024-07-26T12:43:35.850911+0000","last_unstale":"2024-07-26T12:44:39.881522+0000","last_undegraded":"2024-07-26T12:44:39.881522+0000","last_fullsized":"2024-07-26T12:44:39.881522+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:18:16.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850128+0000","last_change":"2024-07-26T12:43:26.774599+0000","last_active":"2024-07-26T12:44:36.850128+0000","last_peered":"2024-07-26T12:44:36.850128+0000","last_clean":"2024-07-26T12:44:36.850128+0000","last_became_active":"2024-07-26T12:43:26.774186+0000","last_became_peered":"2024-07-26T12:43:26.774186+0000","last_unstale":"2024-07-26T12:44:36.850128+0000","last_undegraded":"2024-07-26T12:44:36.850128+0000","last_fullsized":"2024-07-26T12:44:36.850128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:02:42.252093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840792+0000","last_change":"2024-07-26T12:43:12.174676+0000","last_active":"2024-07-26T12:44:36.840792+0000","last_peered":"2024-07-26T12:44:36.840792+0000","last_clean":"2024-07-26T12:44:36.840792+0000","last_became_active":"2024-07-26T12:43:12.174310+0000","last_became_peered":"2024-07-26T12:43:12.174310+0000","last_unstale":"2024-07-26T12:44:36.840792+0000","last_undegraded":"2024-07-26T12:44:36.840792+0000","last_fullsized":"2024-07-26T12:44:36.840792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:44.381289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":155,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:12.003755+0000","last_change":"2024-07-26T12:43:08.122675+0000","last_active":"2024-07-26T12:45:12.003755+0000","last_peered":"2024-07-26T12:45:12.003755+0000","last_clean":"2024-07-26T12:45:12.003755+0000","last_became_active":"2024-07-26T12:43:08.122174+0000","last_became_peered":"2024-07-26T12:43:08.122174+0000","last_unstale":"2024-07-26T12:45:12.003755+0000","last_undegraded":"2024-07-26T12:45:12.003755+0000","last_fullsized":"2024-07-26T12:45:12.003755+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:54.513306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64136,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":74,"num_read_kb":93,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.854038+0000","last_change":"2024-07-26T12:43:26.764283+0000","last_active":"2024-07-26T12:44:36.854038+0000","last_peered":"2024-07-26T12:44:36.854038+0000","last_clean":"2024-07-26T12:44:36.854038+0000","last_became_active":"2024-07-26T12:43:26.763821+0000","last_became_peered":"2024-07-26T12:43:26.763821+0000","last_unstale":"2024-07-26T12:44:36.854038+0000","last_undegraded":"2024-07-26T12:44:36.854038+0000","last_fullsized":"2024-07-26T12:44:36.854038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:56.753016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807726+0000","last_change":"2024-07-26T12:43:35.915277+0000","last_active":"2024-07-26T12:44:36.807726+0000","last_peered":"2024-07-26T12:44:36.807726+0000","last_clean":"2024-07-26T12:44:36.807726+0000","last_became_active":"2024-07-26T12:43:35.914848+0000","last_became_peered":"2024-07-26T12:43:35.914848+0000","last_unstale":"2024-07-26T12:44:36.807726+0000","last_undegraded":"2024-07-26T12:44:36.807726+0000","last_fullsized":"2024-07-26T12:44:36.807726+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:56:35.639076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.874973+0000","last_change":"2024-07-26T12:43:08.227352+0000","last_active":"2024-07-26T12:44:37.874973+0000","last_peered":"2024-07-26T12:44:37.874973+0000","last_clean":"2024-07-26T12:44:37.874973+0000","last_became_active":"2024-07-26T12:43:08.226857+0000","last_became_peered":"2024-07-26T12:43:08.226857+0000","last_unstale":"2024-07-26T12:44:37.874973+0000","last_undegraded":"2024-07-26T12:44:37.874973+0000","last_fullsized":"2024-07-26T12:44:37.874973+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:20.156503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850687+0000","last_change":"2024-07-26T12:43:26.762943+0000","last_active":"2024-07-26T12:44:36.850687+0000","last_peered":"2024-07-26T12:44:36.850687+0000","last_clean":"2024-07-26T12:44:36.850687+0000","last_became_active":"2024-07-26T12:43:26.762436+0000","last_became_peered":"2024-07-26T12:43:26.762436+0000","last_unstale":"2024-07-26T12:44:36.850687+0000","last_undegraded":"2024-07-26T12:44:36.850687+0000","last_fullsized":"2024-07-26T12:44:36.850687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:05:23.924397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.854788+0000","last_change":"2024-07-26T12:44:36.854705+0000","last_active":"2024-07-26T12:44:36.854788+0000","last_peered":"2024-07-26T12:44:36.854788+0000","last_clean":"2024-07-26T12:44:36.854788+0000","last_became_active":"2024-07-26T12:44:36.854268+0000","last_became_peered":"2024-07-26T12:44:36.854268+0000","last_unstale":"2024-07-26T12:44:36.854788+0000","last_undegraded":"2024-07-26T12:44:36.854788+0000","last_fullsized":"2024-07-26T12:44:36.854788+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:24:30.768905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.874500+0000","last_change":"2024-07-26T12:43:08.118597+0000","last_active":"2024-07-26T12:44:38.874500+0000","last_peered":"2024-07-26T12:44:38.874500+0000","last_clean":"2024-07-26T12:44:38.874500+0000","last_became_active":"2024-07-26T12:43:08.118145+0000","last_became_peered":"2024-07-26T12:43:08.118145+0000","last_unstale":"2024-07-26T12:44:38.874500+0000","last_undegraded":"2024-07-26T12:44:38.874500+0000","last_fullsized":"2024-07-26T12:44:38.874500+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:52:45.511693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.853564+0000","last_change":"2024-07-26T12:43:26.771861+0000","last_active":"2024-07-26T12:44:36.853564+0000","last_peered":"2024-07-26T12:44:36.853564+0000","last_clean":"2024-07-26T12:44:36.853564+0000","last_became_active":"2024-07-26T12:43:26.771323+0000","last_became_peered":"2024-07-26T12:43:26.771323+0000","last_unstale":"2024-07-26T12:44:36.853564+0000","last_undegraded":"2024-07-26T12:44:36.853564+0000","last_fullsized":"2024-07-26T12:44:36.853564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:34.207514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.808442+0000","last_change":"2024-07-26T12:43:12.169437+0000","last_active":"2024-07-26T12:44:36.808442+0000","last_peered":"2024-07-26T12:44:36.808442+0000","last_clean":"2024-07-26T12:44:36.808442+0000","last_became_active":"2024-07-26T12:43:12.168387+0000","last_became_peered":"2024-07-26T12:43:12.168387+0000","last_unstale":"2024-07-26T12:44:36.808442+0000","last_undegraded":"2024-07-26T12:44:36.808442+0000","last_fullsized":"2024-07-26T12:44:36.808442+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:57:06.148965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":115,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:11.939808+0000","last_change":"2024-07-26T12:43:08.129618+0000","last_active":"2024-07-26T12:45:11.939808+0000","last_peered":"2024-07-26T12:45:11.939808+0000","last_clean":"2024-07-26T12:45:11.939808+0000","last_became_active":"2024-07-26T12:43:08.129064+0000","last_became_peered":"2024-07-26T12:43:08.129064+0000","last_unstale":"2024-07-26T12:45:11.939808+0000","last_undegraded":"2024-07-26T12:45:11.939808+0000","last_fullsized":"2024-07-26T12:45:11.939808+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:26:22.406075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":66,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804352+0000","last_change":"2024-07-26T12:43:35.854710+0000","last_active":"2024-07-26T12:44:36.804352+0000","last_peered":"2024-07-26T12:44:36.804352+0000","last_clean":"2024-07-26T12:44:36.804352+0000","last_became_active":"2024-07-26T12:43:35.852772+0000","last_became_peered":"2024-07-26T12:43:35.852772+0000","last_unstale":"2024-07-26T12:44:36.804352+0000","last_undegraded":"2024-07-26T12:44:36.804352+0000","last_fullsized":"2024-07-26T12:44:36.804352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:24:54.015879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813169+0000","last_change":"2024-07-26T12:43:26.783182+0000","last_active":"2024-07-26T12:44:36.813169+0000","last_peered":"2024-07-26T12:44:36.813169+0000","last_clean":"2024-07-26T12:44:36.813169+0000","last_became_active":"2024-07-26T12:43:26.782697+0000","last_became_peered":"2024-07-26T12:43:26.782697+0000","last_unstale":"2024-07-26T12:44:36.813169+0000","last_undegraded":"2024-07-26T12:44:36.813169+0000","last_fullsized":"2024-07-26T12:44:36.813169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:57.169222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.853634+0000","last_change":"2024-07-26T12:43:12.166163+0000","last_active":"2024-07-26T12:44:36.853634+0000","last_peered":"2024-07-26T12:44:36.853634+0000","last_clean":"2024-07-26T12:44:36.853634+0000","last_became_active":"2024-07-26T12:43:12.164427+0000","last_became_peered":"2024-07-26T12:43:12.164427+0000","last_unstale":"2024-07-26T12:44:36.853634+0000","last_undegraded":"2024-07-26T12:44:36.853634+0000","last_fullsized":"2024-07-26T12:44:36.853634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:25.335508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.874150+0000","last_change":"2024-07-26T12:43:08.118224+0000","last_active":"2024-07-26T12:44:38.874150+0000","last_peered":"2024-07-26T12:44:38.874150+0000","last_clean":"2024-07-26T12:44:38.874150+0000","last_became_active":"2024-07-26T12:43:08.117821+0000","last_became_peered":"2024-07-26T12:43:08.117821+0000","last_unstale":"2024-07-26T12:44:38.874150+0000","last_undegraded":"2024-07-26T12:44:38.874150+0000","last_fullsized":"2024-07-26T12:44:38.874150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:56:14.380931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851063+0000","last_change":"2024-07-26T12:43:35.907740+0000","last_active":"2024-07-26T12:44:36.851063+0000","last_peered":"2024-07-26T12:44:36.851063+0000","last_clean":"2024-07-26T12:44:36.851063+0000","last_became_active":"2024-07-26T12:43:35.907115+0000","last_became_peered":"2024-07-26T12:43:35.907115+0000","last_unstale":"2024-07-26T12:44:36.851063+0000","last_undegraded":"2024-07-26T12:44:36.851063+0000","last_fullsized":"2024-07-26T12:44:36.851063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:53:24.783872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804682+0000","last_change":"2024-07-26T12:43:12.177579+0000","last_active":"2024-07-26T12:44:36.804682+0000","last_peered":"2024-07-26T12:44:36.804682+0000","last_clean":"2024-07-26T12:44:36.804682+0000","last_became_active":"2024-07-26T12:43:12.171433+0000","last_became_peered":"2024-07-26T12:43:12.171433+0000","last_unstale":"2024-07-26T12:44:36.804682+0000","last_undegraded":"2024-07-26T12:44:36.804682+0000","last_fullsized":"2024-07-26T12:44:36.804682+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:50.972151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.853291+0000","last_change":"2024-07-26T12:43:08.133446+0000","last_active":"2024-07-26T12:44:36.853291+0000","last_peered":"2024-07-26T12:44:36.853291+0000","last_clean":"2024-07-26T12:44:36.853291+0000","last_became_active":"2024-07-26T12:43:08.133008+0000","last_became_peered":"2024-07-26T12:43:08.133008+0000","last_unstale":"2024-07-26T12:44:36.853291+0000","last_undegraded":"2024-07-26T12:44:36.853291+0000","last_fullsized":"2024-07-26T12:44:36.853291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:05:21.283738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.873296+0000","last_change":"2024-07-26T12:43:08.124976+0000","last_active":"2024-07-26T12:44:37.873296+0000","last_peered":"2024-07-26T12:44:37.873296+0000","last_clean":"2024-07-26T12:44:37.873296+0000","last_became_active":"2024-07-26T12:43:08.124664+0000","last_became_peered":"2024-07-26T12:43:08.124664+0000","last_unstale":"2024-07-26T12:44:37.873296+0000","last_undegraded":"2024-07-26T12:44:37.873296+0000","last_fullsized":"2024-07-26T12:44:37.873296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:51.798657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850488+0000","last_change":"2024-07-26T12:43:35.856266+0000","last_active":"2024-07-26T12:44:36.850488+0000","last_peered":"2024-07-26T12:44:36.850488+0000","last_clean":"2024-07-26T12:44:36.850488+0000","last_became_active":"2024-07-26T12:43:35.855802+0000","last_became_peered":"2024-07-26T12:43:35.855802+0000","last_unstale":"2024-07-26T12:44:36.850488+0000","last_undegraded":"2024-07-26T12:44:36.850488+0000","last_fullsized":"2024-07-26T12:44:36.850488+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:16.432983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.802797+0000","last_change":"2024-07-26T12:43:26.778626+0000","last_active":"2024-07-26T12:44:36.802797+0000","last_peered":"2024-07-26T12:44:36.802797+0000","last_clean":"2024-07-26T12:44:36.802797+0000","last_became_active":"2024-07-26T12:43:26.778165+0000","last_became_peered":"2024-07-26T12:43:26.778165+0000","last_unstale":"2024-07-26T12:44:36.802797+0000","last_undegraded":"2024-07-26T12:44:36.802797+0000","last_fullsized":"2024-07-26T12:44:36.802797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:06:08.766495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:12.942144+0000","last_change":"2024-07-26T12:43:08.119702+0000","last_active":"2024-07-26T12:45:12.942144+0000","last_peered":"2024-07-26T12:45:12.942144+0000","last_clean":"2024-07-26T12:45:12.942144+0000","last_became_active":"2024-07-26T12:43:08.119225+0000","last_became_peered":"2024-07-26T12:43:08.119225+0000","last_unstale":"2024-07-26T12:45:12.942144+0000","last_undegraded":"2024-07-26T12:45:12.942144+0000","last_fullsized":"2024-07-26T12:45:12.942144+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:27.616873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840532+0000","last_change":"2024-07-26T12:43:12.166239+0000","last_active":"2024-07-26T12:44:36.840532+0000","last_peered":"2024-07-26T12:44:36.840532+0000","last_clean":"2024-07-26T12:44:36.840532+0000","last_became_active":"2024-07-26T12:43:12.165770+0000","last_became_peered":"2024-07-26T12:43:12.165770+0000","last_unstale":"2024-07-26T12:44:36.840532+0000","last_undegraded":"2024-07-26T12:44:36.840532+0000","last_fullsized":"2024-07-26T12:44:36.840532+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:11:15.290898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.802985+0000","last_change":"2024-07-26T12:43:26.765905+0000","last_active":"2024-07-26T12:44:36.802985+0000","last_peered":"2024-07-26T12:44:36.802985+0000","last_clean":"2024-07-26T12:44:36.802985+0000","last_became_active":"2024-07-26T12:43:26.765425+0000","last_became_peered":"2024-07-26T12:43:26.765425+0000","last_unstale":"2024-07-26T12:44:36.802985+0000","last_undegraded":"2024-07-26T12:44:36.802985+0000","last_fullsized":"2024-07-26T12:44:36.802985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:16.786604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850553+0000","last_change":"2024-07-26T12:43:12.171400+0000","last_active":"2024-07-26T12:44:36.850553+0000","last_peered":"2024-07-26T12:44:36.850553+0000","last_clean":"2024-07-26T12:44:36.850553+0000","last_became_active":"2024-07-26T12:43:12.170998+0000","last_became_peered":"2024-07-26T12:43:12.170998+0000","last_unstale":"2024-07-26T12:44:36.850553+0000","last_undegraded":"2024-07-26T12:44:36.850553+0000","last_fullsized":"2024-07-26T12:44:36.850553+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:43.251975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803384+0000","last_change":"2024-07-26T12:43:35.912729+0000","last_active":"2024-07-26T12:44:36.803384+0000","last_peered":"2024-07-26T12:44:36.803384+0000","last_clean":"2024-07-26T12:44:36.803384+0000","last_became_active":"2024-07-26T12:43:35.912318+0000","last_became_peered":"2024-07-26T12:43:35.912318+0000","last_unstale":"2024-07-26T12:44:36.803384+0000","last_undegraded":"2024-07-26T12:44:36.803384+0000","last_fullsized":"2024-07-26T12:44:36.803384+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:14.473536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803150+0000","last_change":"2024-07-26T12:43:12.182933+0000","last_active":"2024-07-26T12:44:36.803150+0000","last_peered":"2024-07-26T12:44:36.803150+0000","last_clean":"2024-07-26T12:44:36.803150+0000","last_became_active":"2024-07-26T12:43:12.182656+0000","last_became_peered":"2024-07-26T12:43:12.182656+0000","last_unstale":"2024-07-26T12:44:36.803150+0000","last_undegraded":"2024-07-26T12:44:36.803150+0000","last_fullsized":"2024-07-26T12:44:36.803150+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:05:59.717662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.812721+0000","last_change":"2024-07-26T12:43:12.181196+0000","last_active":"2024-07-26T12:44:36.812721+0000","last_peered":"2024-07-26T12:44:36.812721+0000","last_clean":"2024-07-26T12:44:36.812721+0000","last_became_active":"2024-07-26T12:43:12.180765+0000","last_became_peered":"2024-07-26T12:43:12.180765+0000","last_unstale":"2024-07-26T12:44:36.812721+0000","last_undegraded":"2024-07-26T12:44:36.812721+0000","last_fullsized":"2024-07-26T12:44:36.812721+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:20.590023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810869+0000","last_change":"2024-07-26T12:43:12.185408+0000","last_active":"2024-07-26T12:44:36.810869+0000","last_peered":"2024-07-26T12:44:36.810869+0000","last_clean":"2024-07-26T12:44:36.810869+0000","last_became_active":"2024-07-26T12:43:12.185038+0000","last_became_peered":"2024-07-26T12:43:12.185038+0000","last_unstale":"2024-07-26T12:44:36.810869+0000","last_undegraded":"2024-07-26T12:44:36.810869+0000","last_fullsized":"2024-07-26T12:44:36.810869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:55:30.604995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851270+0000","last_change":"2024-07-26T12:43:12.169301+0000","last_active":"2024-07-26T12:44:36.851270+0000","last_peered":"2024-07-26T12:44:36.851270+0000","last_clean":"2024-07-26T12:44:36.851270+0000","last_became_active":"2024-07-26T12:43:12.168923+0000","last_became_peered":"2024-07-26T12:43:12.168923+0000","last_unstale":"2024-07-26T12:44:36.851270+0000","last_undegraded":"2024-07-26T12:44:36.851270+0000","last_fullsized":"2024-07-26T12:44:36.851270+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:04.506485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803145+0000","last_change":"2024-07-26T12:43:12.166663+0000","last_active":"2024-07-26T12:44:36.803145+0000","last_peered":"2024-07-26T12:44:36.803145+0000","last_clean":"2024-07-26T12:44:36.803145+0000","last_became_active":"2024-07-26T12:43:12.166200+0000","last_became_peered":"2024-07-26T12:43:12.166200+0000","last_unstale":"2024-07-26T12:44:36.803145+0000","last_undegraded":"2024-07-26T12:44:36.803145+0000","last_fullsized":"2024-07-26T12:44:36.803145+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:14.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.806579+0000","last_change":"2024-07-26T12:43:12.171273+0000","last_active":"2024-07-26T12:44:36.806579+0000","last_peered":"2024-07-26T12:44:36.806579+0000","last_clean":"2024-07-26T12:44:36.806579+0000","last_became_active":"2024-07-26T12:43:12.167680+0000","last_became_peered":"2024-07-26T12:43:12.167680+0000","last_unstale":"2024-07-26T12:44:36.806579+0000","last_undegraded":"2024-07-26T12:44:36.806579+0000","last_fullsized":"2024-07-26T12:44:36.806579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:44.797644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850681+0000","last_change":"2024-07-26T12:43:12.179891+0000","last_active":"2024-07-26T12:44:36.850681+0000","last_peered":"2024-07-26T12:44:36.850681+0000","last_clean":"2024-07-26T12:44:36.850681+0000","last_became_active":"2024-07-26T12:43:12.179518+0000","last_became_peered":"2024-07-26T12:43:12.179518+0000","last_unstale":"2024-07-26T12:44:36.850681+0000","last_undegraded":"2024-07-26T12:44:36.850681+0000","last_fullsized":"2024-07-26T12:44:36.850681+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:24.246023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804512+0000","last_change":"2024-07-26T12:43:12.168521+0000","last_active":"2024-07-26T12:44:36.804512+0000","last_peered":"2024-07-26T12:44:36.804512+0000","last_clean":"2024-07-26T12:44:36.804512+0000","last_became_active":"2024-07-26T12:43:12.168126+0000","last_became_peered":"2024-07-26T12:43:12.168126+0000","last_unstale":"2024-07-26T12:44:36.804512+0000","last_undegraded":"2024-07-26T12:44:36.804512+0000","last_fullsized":"2024-07-26T12:44:36.804512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:00.135090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.843425+0000","last_change":"2024-07-26T12:44:36.843322+0000","last_active":"2024-07-26T12:44:36.843425+0000","last_peered":"2024-07-26T12:44:36.843425+0000","last_clean":"2024-07-26T12:44:36.843425+0000","last_became_active":"2024-07-26T12:44:36.842821+0000","last_became_peered":"2024-07-26T12:44:36.842821+0000","last_unstale":"2024-07-26T12:44:36.843425+0000","last_undegraded":"2024-07-26T12:44:36.843425+0000","last_fullsized":"2024-07-26T12:44:36.843425+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:05.609298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809985+0000","last_change":"2024-07-26T12:43:12.185565+0000","last_active":"2024-07-26T12:44:36.809985+0000","last_peered":"2024-07-26T12:44:36.809985+0000","last_clean":"2024-07-26T12:44:36.809985+0000","last_became_active":"2024-07-26T12:43:12.185057+0000","last_became_peered":"2024-07-26T12:43:12.185057+0000","last_unstale":"2024-07-26T12:44:36.809985+0000","last_undegraded":"2024-07-26T12:44:36.809985+0000","last_fullsized":"2024-07-26T12:44:36.809985+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:58.108311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851009+0000","last_change":"2024-07-26T12:43:12.173301+0000","last_active":"2024-07-26T12:44:36.851009+0000","last_peered":"2024-07-26T12:44:36.851009+0000","last_clean":"2024-07-26T12:44:36.851009+0000","last_became_active":"2024-07-26T12:43:12.172955+0000","last_became_peered":"2024-07-26T12:43:12.172955+0000","last_unstale":"2024-07-26T12:44:36.851009+0000","last_undegraded":"2024-07-26T12:44:36.851009+0000","last_fullsized":"2024-07-26T12:44:36.851009+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:41.731141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803067+0000","last_change":"2024-07-26T12:43:12.176169+0000","last_active":"2024-07-26T12:44:36.803067+0000","last_peered":"2024-07-26T12:44:36.803067+0000","last_clean":"2024-07-26T12:44:36.803067+0000","last_became_active":"2024-07-26T12:43:12.175745+0000","last_became_peered":"2024-07-26T12:43:12.175745+0000","last_unstale":"2024-07-26T12:44:36.803067+0000","last_undegraded":"2024-07-26T12:44:36.803067+0000","last_fullsized":"2024-07-26T12:44:36.803067+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:20.393207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.856834+0000","last_change":"2024-07-26T12:43:12.168182+0000","last_active":"2024-07-26T12:44:36.856834+0000","last_peered":"2024-07-26T12:44:36.856834+0000","last_clean":"2024-07-26T12:44:36.856834+0000","last_became_active":"2024-07-26T12:43:12.163771+0000","last_became_peered":"2024-07-26T12:43:12.163771+0000","last_unstale":"2024-07-26T12:44:36.856834+0000","last_undegraded":"2024-07-26T12:44:36.856834+0000","last_fullsized":"2024-07-26T12:44:36.856834+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:40:30.960153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851443+0000","last_change":"2024-07-26T12:43:12.189524+0000","last_active":"2024-07-26T12:44:36.851443+0000","last_peered":"2024-07-26T12:44:36.851443+0000","last_clean":"2024-07-26T12:44:36.851443+0000","last_became_active":"2024-07-26T12:43:12.189165+0000","last_became_peered":"2024-07-26T12:43:12.189165+0000","last_unstale":"2024-07-26T12:44:36.851443+0000","last_undegraded":"2024-07-26T12:44:36.851443+0000","last_fullsized":"2024-07-26T12:44:36.851443+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:11:08.479058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807287+0000","last_change":"2024-07-26T12:43:12.180603+0000","last_active":"2024-07-26T12:44:36.807287+0000","last_peered":"2024-07-26T12:44:36.807287+0000","last_clean":"2024-07-26T12:44:36.807287+0000","last_became_active":"2024-07-26T12:43:12.180204+0000","last_became_peered":"2024-07-26T12:43:12.180204+0000","last_unstale":"2024-07-26T12:44:36.807287+0000","last_undegraded":"2024-07-26T12:44:36.807287+0000","last_fullsized":"2024-07-26T12:44:36.807287+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:46.768213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.812128+0000","last_change":"2024-07-26T12:43:12.185347+0000","last_active":"2024-07-26T12:44:36.812128+0000","last_peered":"2024-07-26T12:44:36.812128+0000","last_clean":"2024-07-26T12:44:36.812128+0000","last_became_active":"2024-07-26T12:43:12.180190+0000","last_became_peered":"2024-07-26T12:43:12.180190+0000","last_unstale":"2024-07-26T12:44:36.812128+0000","last_undegraded":"2024-07-26T12:44:36.812128+0000","last_fullsized":"2024-07-26T12:44:36.812128+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:07.654967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.812348+0000","last_change":"2024-07-26T12:43:12.177258+0000","last_active":"2024-07-26T12:44:36.812348+0000","last_peered":"2024-07-26T12:44:36.812348+0000","last_clean":"2024-07-26T12:44:36.812348+0000","last_became_active":"2024-07-26T12:43:12.176879+0000","last_became_peered":"2024-07-26T12:43:12.176879+0000","last_unstale":"2024-07-26T12:44:36.812348+0000","last_undegraded":"2024-07-26T12:44:36.812348+0000","last_fullsized":"2024-07-26T12:44:36.812348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:24.244759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.857476+0000","last_change":"2024-07-26T12:44:36.857403+0000","last_active":"2024-07-26T12:44:36.857476+0000","last_peered":"2024-07-26T12:44:36.857476+0000","last_clean":"2024-07-26T12:44:36.857476+0000","last_became_active":"2024-07-26T12:44:36.856980+0000","last_became_peered":"2024-07-26T12:44:36.856980+0000","last_unstale":"2024-07-26T12:44:36.857476+0000","last_undegraded":"2024-07-26T12:44:36.857476+0000","last_fullsized":"2024-07-26T12:44:36.857476+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.166383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850844+0000","last_change":"2024-07-26T12:43:12.166827+0000","last_active":"2024-07-26T12:44:36.850844+0000","last_peered":"2024-07-26T12:44:36.850844+0000","last_clean":"2024-07-26T12:44:36.850844+0000","last_became_active":"2024-07-26T12:43:12.165222+0000","last_became_peered":"2024-07-26T12:43:12.165222+0000","last_unstale":"2024-07-26T12:44:36.850844+0000","last_undegraded":"2024-07-26T12:44:36.850844+0000","last_fullsized":"2024-07-26T12:44:36.850844+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:37:42.219844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805845+0000","last_change":"2024-07-26T12:43:12.176961+0000","last_active":"2024-07-26T12:44:36.805845+0000","last_peered":"2024-07-26T12:44:36.805845+0000","last_clean":"2024-07-26T12:44:36.805845+0000","last_became_active":"2024-07-26T12:43:12.176550+0000","last_became_peered":"2024-07-26T12:43:12.176550+0000","last_unstale":"2024-07-26T12:44:36.805845+0000","last_undegraded":"2024-07-26T12:44:36.805845+0000","last_fullsized":"2024-07-26T12:44:36.805845+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:36:19.720801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809896+0000","last_change":"2024-07-26T12:43:12.173560+0000","last_active":"2024-07-26T12:44:36.809896+0000","last_peered":"2024-07-26T12:44:36.809896+0000","last_clean":"2024-07-26T12:44:36.809896+0000","last_became_active":"2024-07-26T12:43:12.173187+0000","last_became_peered":"2024-07-26T12:43:12.173187+0000","last_unstale":"2024-07-26T12:44:36.809896+0000","last_undegraded":"2024-07-26T12:44:36.809896+0000","last_fullsized":"2024-07-26T12:44:36.809896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:53.929210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.808767+0000","last_change":"2024-07-26T12:43:12.169829+0000","last_active":"2024-07-26T12:44:36.808767+0000","last_peered":"2024-07-26T12:44:36.808767+0000","last_clean":"2024-07-26T12:44:36.808767+0000","last_became_active":"2024-07-26T12:43:12.169444+0000","last_became_peered":"2024-07-26T12:43:12.169444+0000","last_unstale":"2024-07-26T12:44:36.808767+0000","last_undegraded":"2024-07-26T12:44:36.808767+0000","last_fullsized":"2024-07-26T12:44:36.808767+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:29.111483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804682+0000","last_change":"2024-07-26T12:43:12.171857+0000","last_active":"2024-07-26T12:44:36.804682+0000","last_peered":"2024-07-26T12:44:36.804682+0000","last_clean":"2024-07-26T12:44:36.804682+0000","last_became_active":"2024-07-26T12:43:12.170752+0000","last_became_peered":"2024-07-26T12:43:12.170752+0000","last_unstale":"2024-07-26T12:44:36.804682+0000","last_undegraded":"2024-07-26T12:44:36.804682+0000","last_fullsized":"2024-07-26T12:44:36.804682+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:22:09.442916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805276+0000","last_change":"2024-07-26T12:43:12.178646+0000","last_active":"2024-07-26T12:44:36.805276+0000","last_peered":"2024-07-26T12:44:36.805276+0000","last_clean":"2024-07-26T12:44:36.805276+0000","last_became_active":"2024-07-26T12:43:12.178259+0000","last_became_peered":"2024-07-26T12:43:12.178259+0000","last_unstale":"2024-07-26T12:44:36.805276+0000","last_undegraded":"2024-07-26T12:44:36.805276+0000","last_fullsized":"2024-07-26T12:44:36.805276+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:30:59.417843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809164+0000","last_change":"2024-07-26T12:43:12.164579+0000","last_active":"2024-07-26T12:44:36.809164+0000","last_peered":"2024-07-26T12:44:36.809164+0000","last_clean":"2024-07-26T12:44:36.809164+0000","last_became_active":"2024-07-26T12:43:12.164099+0000","last_became_peered":"2024-07-26T12:43:12.164099+0000","last_unstale":"2024-07-26T12:44:36.809164+0000","last_undegraded":"2024-07-26T12:44:36.809164+0000","last_fullsized":"2024-07-26T12:44:36.809164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:51:45.293190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840446+0000","last_change":"2024-07-26T12:43:12.180838+0000","last_active":"2024-07-26T12:44:36.840446+0000","last_peered":"2024-07-26T12:44:36.840446+0000","last_clean":"2024-07-26T12:44:36.840446+0000","last_became_active":"2024-07-26T12:43:12.180543+0000","last_became_peered":"2024-07-26T12:43:12.180543+0000","last_unstale":"2024-07-26T12:44:36.840446+0000","last_undegraded":"2024-07-26T12:44:36.840446+0000","last_fullsized":"2024-07-26T12:44:36.840446+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:27.185082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850779+0000","last_change":"2024-07-26T12:43:12.167235+0000","last_active":"2024-07-26T12:44:36.850779+0000","last_peered":"2024-07-26T12:44:36.850779+0000","last_clean":"2024-07-26T12:44:36.850779+0000","last_became_active":"2024-07-26T12:43:12.166489+0000","last_became_peered":"2024-07-26T12:43:12.166489+0000","last_unstale":"2024-07-26T12:44:36.850779+0000","last_undegraded":"2024-07-26T12:44:36.850779+0000","last_fullsized":"2024-07-26T12:44:36.850779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:13.868870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813042+0000","last_change":"2024-07-26T12:43:12.176717+0000","last_active":"2024-07-26T12:44:36.813042+0000","last_peered":"2024-07-26T12:44:36.813042+0000","last_clean":"2024-07-26T12:44:36.813042+0000","last_became_active":"2024-07-26T12:43:12.172699+0000","last_became_peered":"2024-07-26T12:43:12.172699+0000","last_unstale":"2024-07-26T12:44:36.813042+0000","last_undegraded":"2024-07-26T12:44:36.813042+0000","last_fullsized":"2024-07-26T12:44:36.813042+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:23:36.853999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813407+0000","last_change":"2024-07-26T12:43:12.177093+0000","last_active":"2024-07-26T12:44:36.813407+0000","last_peered":"2024-07-26T12:44:36.813407+0000","last_clean":"2024-07-26T12:44:36.813407+0000","last_became_active":"2024-07-26T12:43:12.176693+0000","last_became_peered":"2024-07-26T12:43:12.176693+0000","last_unstale":"2024-07-26T12:44:36.813407+0000","last_undegraded":"2024-07-26T12:44:36.813407+0000","last_fullsized":"2024-07-26T12:44:36.813407+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:56.875143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851429+0000","last_change":"2024-07-26T12:43:12.189531+0000","last_active":"2024-07-26T12:44:36.851429+0000","last_peered":"2024-07-26T12:44:36.851429+0000","last_clean":"2024-07-26T12:44:36.851429+0000","last_became_active":"2024-07-26T12:43:12.189162+0000","last_became_peered":"2024-07-26T12:43:12.189162+0000","last_unstale":"2024-07-26T12:44:36.851429+0000","last_undegraded":"2024-07-26T12:44:36.851429+0000","last_fullsized":"2024-07-26T12:44:36.851429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:11:00.160614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71340,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":378,"num_read_kb":262,"num_write":251,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678056,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":536325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874285,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37032,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708120,"statfs":{"total":95995035648,"available":95957114880,"internally_reserved":0,"allocated":1736704,"data_stored":1180565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037809,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":539892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234039,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":536174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430268,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36964,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708188,"statfs":{"total":95995035648,"available":95957184512,"internally_reserved":0,"allocated":1667072,"data_stored":1132671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593794,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":531300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1128253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953549,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":535374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":531008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378712,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":597400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542239,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1118208,"data_stored":596466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705762,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":533147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":66822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T12:45:17.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T12:45:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:17 smithi107 ceph-mon[32659]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-07-26T12:45:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:17 smithi132 ceph-mon[37580]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-07-26T12:45:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:17 smithi112 ceph-mon[37501]: pgmap v369: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-07-26T12:45:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:18.315 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:18 smithi107 ceph-mon[32659]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:45:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:18 smithi132 ceph-mon[37580]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:45:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:18 smithi112 ceph-mon[37501]: from='client.24943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:45:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:45:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:45:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:45:20.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:19 smithi107 ceph-mon[32659]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T12:45:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:19 smithi132 ceph-mon[37580]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T12:45:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:19 smithi112 ceph-mon[37501]: pgmap v370: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T12:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.581+0000 7f861ac00640 1 -- 172.21.15.107:0/62159551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8614109df0 msgr2=0x7f861410a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.581+0000 7f861ac00640 1 --2- 172.21.15.107:0/62159551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8614109df0 0x7f861410a200 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8600009900 tx=0x7f860002a410 comp rx=0 tx=0).stop 2024-07-26T12:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.582+0000 7f861ac00640 1 -- 172.21.15.107:0/62159551 shutdown_connections 2024-07-26T12:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.582+0000 7f861ac00640 1 --2- 172.21.15.107:0/62159551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f861410a740 0x7f86141169f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:20.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.582+0000 7f861ac00640 1 --2- 172.21.15.107:0/62159551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8614109df0 0x7f861410a200 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.582+0000 7f861ac00640 1 --2- 172.21.15.107:0/62159551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f861410eb80 0x7f861410ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.582+0000 7f861ac00640 1 -- 172.21.15.107:0/62159551 >> 172.21.15.107:0/62159551 conn(0x7f8614070d20 msgr2=0x7f8614073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.582+0000 7f861ac00640 1 -- 172.21.15.107:0/62159551 shutdown_connections 2024-07-26T12:45:20.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.582+0000 7f861ac00640 1 -- 172.21.15.107:0/62159551 wait complete. 2024-07-26T12:45:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.583+0000 7f861ac00640 1 Processor -- start 2024-07-26T12:45:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.583+0000 7f861ac00640 1 -- start start 2024-07-26T12:45:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f861ac00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8614109df0 0x7f86141a2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f861ac00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861410a740 0x7f86141a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f861ac00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f861410eb80 0x7f86141a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f861ac00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f861411ac10 con 0x7f861410a740 2024-07-26T12:45:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f861ac00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f861411aa90 con 0x7f8614109df0 2024-07-26T12:45:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f861ac00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f861411ad90 con 0x7f861410eb80 2024-07-26T12:45:20.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f8618c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f861410eb80 0x7f86141a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:20.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f8613e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8614109df0 0x7f86141a2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:20.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f8613400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861410a740 0x7f86141a2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:20.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f8613e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8614109df0 0x7f86141a2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44998/0 (socket says 172.21.15.107:44998) 2024-07-26T12:45:20.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f8613e00640 1 -- 172.21.15.107:0/58458090 learned_addr learned my addr 172.21.15.107:0/58458090 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:20.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.584+0000 7f8618c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f861410eb80 0x7f86141a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33288/0 (socket says 172.21.15.107:33288) 2024-07-26T12:45:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f8613e00640 1 -- 172.21.15.107:0/58458090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f861410eb80 msgr2=0x7f86141a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f8613e00640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f861410eb80 0x7f86141a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f8613e00640 1 -- 172.21.15.107:0/58458090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861410a740 msgr2=0x7f86141a2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f8613e00640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861410a740 0x7f86141a2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f8613e00640 1 -- 172.21.15.107:0/58458090 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86141a8680 con 0x7f8614109df0 2024-07-26T12:45:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f8613e00640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8614109df0 0x7f86141a2710 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f860400eb90 tx=0x7f860400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f8610c00640 1 -- 172.21.15.107:0/58458090 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604004d20 con 0x7f8614109df0 2024-07-26T12:45:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f8610c00640 1 -- 172.21.15.107:0/58458090 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f860400ee90 con 0x7f8614109df0 2024-07-26T12:45:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f861ac00640 1 -- 172.21.15.107:0/58458090 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86141a8970 con 0x7f8614109df0 2024-07-26T12:45:20.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f8610c00640 1 -- 172.21.15.107:0/58458090 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604010670 con 0x7f8614109df0 2024-07-26T12:45:20.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.585+0000 7f861ac00640 1 -- 172.21.15.107:0/58458090 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86141af0d0 con 0x7f8614109df0 2024-07-26T12:45:20.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.587+0000 7f861ac00640 1 -- 172.21.15.107:0/58458090 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f861410a200 con 0x7f8614109df0 2024-07-26T12:45:20.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.589+0000 7f8610c00640 1 -- 172.21.15.107:0/58458090 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f8604010930 con 0x7f8614109df0 2024-07-26T12:45:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.590+0000 7f8610c00640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85f80b3cb0 0x7f85f80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:20.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.591+0000 7f8610c00640 1 -- 172.21.15.107:0/58458090 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f86040e4b40 con 0x7f8614109df0 2024-07-26T12:45:20.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.591+0000 7f8613400640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85f80b3cb0 0x7f85f80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:20.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.591+0000 7f8613400640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85f80b3cb0 0x7f85f80b61c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8600009230 tx=0x7f8600031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:20.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.592+0000 7f8610c00640 1 -- 172.21.15.107:0/58458090 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8604055600 con 0x7f8614109df0 2024-07-26T12:45:20.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.787+0000 7f861ac00640 1 -- 172.21.15.107:0/58458090 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8614117af0 con 0x7f85f80b3cb0 2024-07-26T12:45:20.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.795+0000 7f8610c00640 1 -- 172.21.15.107:0/58458090 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340390 (secure 0 0 0) 0x7f8614117af0 con 0x7f85f80b3cb0 2024-07-26T12:45:20.801 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:45:20.803 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-07-26T12:45:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 -- 172.21.15.107:0/58458090 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85f80b3cb0 msgr2=0x7f85f80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85f80b3cb0 0x7f85f80b61c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8600009230 tx=0x7f8600031400 comp rx=0 tx=0).stop 2024-07-26T12:45:20.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 -- 172.21.15.107:0/58458090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8614109df0 msgr2=0x7f86141a2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8614109df0 0x7f86141a2710 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f860400eb90 tx=0x7f860400c6a0 comp rx=0 tx=0).stop 2024-07-26T12:45:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 -- 172.21.15.107:0/58458090 shutdown_connections 2024-07-26T12:45:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f85f80b3cb0 0x7f85f80b61c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f861410eb80 0x7f86141a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f861410a740 0x7f86141a2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 --2- 172.21.15.107:0/58458090 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8614109df0 0x7f86141a2710 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 -- 172.21.15.107:0/58458090 >> 172.21.15.107:0/58458090 conn(0x7f8614070d20 msgr2=0x7f861410be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 -- 172.21.15.107:0/58458090 shutdown_connections 2024-07-26T12:45:20.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:20.801+0000 7f861ac00640 1 -- 172.21.15.107:0/58458090 wait complete. 2024-07-26T12:45:21.661 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-07-26T12:45:20.708426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661942,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":505,"num_read_kb":358,"num_write":376,"num_write_kb":2296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2951874,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":310,"ondisk_log_size":310,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376068,"kb_used_data":14432,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565756,"statfs":{"total":1151940427776,"available":1151555334144,"internally_reserved":0,"allocated":14778368,"data_stored":8384828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6253,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":63,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006465"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":97,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:15.953478+0000","last_change":"2024-07-26T12:43:08.129417+0000","last_active":"2024-07-26T12:45:15.953478+0000","last_peered":"2024-07-26T12:45:15.953478+0000","last_clean":"2024-07-26T12:45:15.953478+0000","last_became_active":"2024-07-26T12:43:08.128919+0000","last_became_peered":"2024-07-26T12:43:08.128919+0000","last_unstale":"2024-07-26T12:45:15.953478+0000","last_undegraded":"2024-07-26T12:45:15.953478+0000","last_fullsized":"2024-07-26T12:45:15.953478+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:35:55.414185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805971+0000","last_change":"2024-07-26T12:43:12.172058+0000","last_active":"2024-07-26T12:44:36.805971+0000","last_peered":"2024-07-26T12:44:36.805971+0000","last_clean":"2024-07-26T12:44:36.805971+0000","last_became_active":"2024-07-26T12:43:12.171631+0000","last_became_peered":"2024-07-26T12:43:12.171631+0000","last_unstale":"2024-07-26T12:44:36.805971+0000","last_undegraded":"2024-07-26T12:44:36.805971+0000","last_fullsized":"2024-07-26T12:44:36.805971+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:51.972962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.802628+0000","last_change":"2024-07-26T12:43:26.773739+0000","last_active":"2024-07-26T12:44:36.802628+0000","last_peered":"2024-07-26T12:44:36.802628+0000","last_clean":"2024-07-26T12:44:36.802628+0000","last_became_active":"2024-07-26T12:43:26.773254+0000","last_became_peered":"2024-07-26T12:43:26.773254+0000","last_unstale":"2024-07-26T12:44:36.802628+0000","last_undegraded":"2024-07-26T12:44:36.802628+0000","last_fullsized":"2024-07-26T12:44:36.802628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:03.125497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.852995+0000","last_change":"2024-07-26T12:43:08.227329+0000","last_active":"2024-07-26T12:44:36.852995+0000","last_peered":"2024-07-26T12:44:36.852995+0000","last_clean":"2024-07-26T12:44:36.852995+0000","last_became_active":"2024-07-26T12:43:08.226674+0000","last_became_peered":"2024-07-26T12:43:08.226674+0000","last_unstale":"2024-07-26T12:44:36.852995+0000","last_undegraded":"2024-07-26T12:44:36.852995+0000","last_fullsized":"2024-07-26T12:44:36.852995+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.222684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813756+0000","last_change":"2024-07-26T12:43:12.190616+0000","last_active":"2024-07-26T12:44:36.813756+0000","last_peered":"2024-07-26T12:44:36.813756+0000","last_clean":"2024-07-26T12:44:36.813756+0000","last_became_active":"2024-07-26T12:43:12.189717+0000","last_became_peered":"2024-07-26T12:43:12.189717+0000","last_unstale":"2024-07-26T12:44:36.813756+0000","last_undegraded":"2024-07-26T12:44:36.813756+0000","last_fullsized":"2024-07-26T12:44:36.813756+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:03.509592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850258+0000","last_change":"2024-07-26T12:43:26.774486+0000","last_active":"2024-07-26T12:44:36.850258+0000","last_peered":"2024-07-26T12:44:36.850258+0000","last_clean":"2024-07-26T12:44:36.850258+0000","last_became_active":"2024-07-26T12:43:26.774046+0000","last_became_peered":"2024-07-26T12:43:26.774046+0000","last_unstale":"2024-07-26T12:44:36.850258+0000","last_undegraded":"2024-07-26T12:44:36.850258+0000","last_fullsized":"2024-07-26T12:44:36.850258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.847072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.868396+0000","last_change":"2024-07-26T12:43:08.123084+0000","last_active":"2024-07-26T12:44:39.868396+0000","last_peered":"2024-07-26T12:44:39.868396+0000","last_clean":"2024-07-26T12:44:39.868396+0000","last_became_active":"2024-07-26T12:43:08.122371+0000","last_became_peered":"2024-07-26T12:43:08.122371+0000","last_unstale":"2024-07-26T12:44:39.868396+0000","last_undegraded":"2024-07-26T12:44:39.868396+0000","last_fullsized":"2024-07-26T12:44:39.868396+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:27:09.665660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807575+0000","last_change":"2024-07-26T12:43:12.170290+0000","last_active":"2024-07-26T12:44:36.807575+0000","last_peered":"2024-07-26T12:44:36.807575+0000","last_clean":"2024-07-26T12:44:36.807575+0000","last_became_active":"2024-07-26T12:43:12.169849+0000","last_became_peered":"2024-07-26T12:43:12.169849+0000","last_unstale":"2024-07-26T12:44:36.807575+0000","last_undegraded":"2024-07-26T12:44:36.807575+0000","last_fullsized":"2024-07-26T12:44:36.807575+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:46.044921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850501+0000","last_change":"2024-07-26T12:43:26.764804+0000","last_active":"2024-07-26T12:44:36.850501+0000","last_peered":"2024-07-26T12:44:36.850501+0000","last_clean":"2024-07-26T12:44:36.850501+0000","last_became_active":"2024-07-26T12:43:26.764065+0000","last_became_peered":"2024-07-26T12:43:26.764065+0000","last_unstale":"2024-07-26T12:44:36.850501+0000","last_undegraded":"2024-07-26T12:44:36.850501+0000","last_fullsized":"2024-07-26T12:44:36.850501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:16:59.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.874869+0000","last_change":"2024-07-26T12:43:08.121496+0000","last_active":"2024-07-26T12:44:38.874869+0000","last_peered":"2024-07-26T12:44:38.874869+0000","last_clean":"2024-07-26T12:44:38.874869+0000","last_became_active":"2024-07-26T12:43:08.120962+0000","last_became_peered":"2024-07-26T12:43:08.120962+0000","last_unstale":"2024-07-26T12:44:38.874869+0000","last_undegraded":"2024-07-26T12:44:38.874869+0000","last_fullsized":"2024-07-26T12:44:38.874869+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:42:02.998023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850048+0000","last_change":"2024-07-26T12:43:12.168707+0000","last_active":"2024-07-26T12:44:36.850048+0000","last_peered":"2024-07-26T12:44:36.850048+0000","last_clean":"2024-07-26T12:44:36.850048+0000","last_became_active":"2024-07-26T12:43:12.168338+0000","last_became_peered":"2024-07-26T12:43:12.168338+0000","last_unstale":"2024-07-26T12:44:36.850048+0000","last_undegraded":"2024-07-26T12:44:36.850048+0000","last_fullsized":"2024-07-26T12:44:36.850048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:39:44.724864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850427+0000","last_change":"2024-07-26T12:43:26.770863+0000","last_active":"2024-07-26T12:44:36.850427+0000","last_peered":"2024-07-26T12:44:36.850427+0000","last_clean":"2024-07-26T12:44:36.850427+0000","last_became_active":"2024-07-26T12:43:26.770480+0000","last_became_peered":"2024-07-26T12:43:26.770480+0000","last_unstale":"2024-07-26T12:44:36.850427+0000","last_undegraded":"2024-07-26T12:44:36.850427+0000","last_fullsized":"2024-07-26T12:44:36.850427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:09.128364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805647+0000","last_change":"2024-07-26T12:43:12.181659+0000","last_active":"2024-07-26T12:44:36.805647+0000","last_peered":"2024-07-26T12:44:36.805647+0000","last_clean":"2024-07-26T12:44:36.805647+0000","last_became_active":"2024-07-26T12:43:12.181268+0000","last_became_peered":"2024-07-26T12:43:12.181268+0000","last_unstale":"2024-07-26T12:44:36.805647+0000","last_undegraded":"2024-07-26T12:44:36.805647+0000","last_fullsized":"2024-07-26T12:44:36.805647+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:38:28.445981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850732+0000","last_change":"2024-07-26T12:43:35.856273+0000","last_active":"2024-07-26T12:44:36.850732+0000","last_peered":"2024-07-26T12:44:36.850732+0000","last_clean":"2024-07-26T12:44:36.850732+0000","last_became_active":"2024-07-26T12:43:35.849625+0000","last_became_peered":"2024-07-26T12:43:35.849625+0000","last_unstale":"2024-07-26T12:44:36.850732+0000","last_undegraded":"2024-07-26T12:44:36.850732+0000","last_fullsized":"2024-07-26T12:44:36.850732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:34:45.261474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850624+0000","last_change":"2024-07-26T12:43:26.793431+0000","last_active":"2024-07-26T12:44:36.850624+0000","last_peered":"2024-07-26T12:44:36.850624+0000","last_clean":"2024-07-26T12:44:36.850624+0000","last_became_active":"2024-07-26T12:43:26.792931+0000","last_became_peered":"2024-07-26T12:43:26.792931+0000","last_unstale":"2024-07-26T12:44:36.850624+0000","last_undegraded":"2024-07-26T12:44:36.850624+0000","last_fullsized":"2024-07-26T12:44:36.850624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:31.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805205+0000","last_change":"2024-07-26T12:43:12.183784+0000","last_active":"2024-07-26T12:44:36.805205+0000","last_peered":"2024-07-26T12:44:36.805205+0000","last_clean":"2024-07-26T12:44:36.805205+0000","last_became_active":"2024-07-26T12:43:12.182696+0000","last_became_peered":"2024-07-26T12:43:12.182696+0000","last_unstale":"2024-07-26T12:44:36.805205+0000","last_undegraded":"2024-07-26T12:44:36.805205+0000","last_fullsized":"2024-07-26T12:44:36.805205+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:33:19.918180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.860356+0000","last_change":"2024-07-26T12:43:08.131069+0000","last_active":"2024-07-26T12:44:38.860356+0000","last_peered":"2024-07-26T12:44:38.860356+0000","last_clean":"2024-07-26T12:44:38.860356+0000","last_became_active":"2024-07-26T12:43:08.130637+0000","last_became_peered":"2024-07-26T12:43:08.130637+0000","last_unstale":"2024-07-26T12:44:38.860356+0000","last_undegraded":"2024-07-26T12:44:38.860356+0000","last_fullsized":"2024-07-26T12:44:38.860356+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:04:16.461900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850617+0000","last_change":"2024-07-26T12:43:26.763388+0000","last_active":"2024-07-26T12:44:36.850617+0000","last_peered":"2024-07-26T12:44:36.850617+0000","last_clean":"2024-07-26T12:44:36.850617+0000","last_became_active":"2024-07-26T12:43:26.762464+0000","last_became_peered":"2024-07-26T12:43:26.762464+0000","last_unstale":"2024-07-26T12:44:36.850617+0000","last_undegraded":"2024-07-26T12:44:36.850617+0000","last_fullsized":"2024-07-26T12:44:36.850617+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:10:17.598266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850536+0000","last_change":"2024-07-26T12:43:12.191640+0000","last_active":"2024-07-26T12:44:36.850536+0000","last_peered":"2024-07-26T12:44:36.850536+0000","last_clean":"2024-07-26T12:44:36.850536+0000","last_became_active":"2024-07-26T12:43:12.191260+0000","last_became_peered":"2024-07-26T12:43:12.191260+0000","last_unstale":"2024-07-26T12:44:36.850536+0000","last_undegraded":"2024-07-26T12:44:36.850536+0000","last_fullsized":"2024-07-26T12:44:36.850536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:47:18.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.876184+0000","last_change":"2024-07-26T12:43:08.135035+0000","last_active":"2024-07-26T12:44:38.876184+0000","last_peered":"2024-07-26T12:44:38.876184+0000","last_clean":"2024-07-26T12:44:38.876184+0000","last_became_active":"2024-07-26T12:43:08.134204+0000","last_became_peered":"2024-07-26T12:43:08.134204+0000","last_unstale":"2024-07-26T12:44:38.876184+0000","last_undegraded":"2024-07-26T12:44:38.876184+0000","last_fullsized":"2024-07-26T12:44:38.876184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:38.954460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840930+0000","last_change":"2024-07-26T12:43:35.853244+0000","last_active":"2024-07-26T12:44:36.840930+0000","last_peered":"2024-07-26T12:44:36.840930+0000","last_clean":"2024-07-26T12:44:36.840930+0000","last_became_active":"2024-07-26T12:43:35.852728+0000","last_became_peered":"2024-07-26T12:43:35.852728+0000","last_unstale":"2024-07-26T12:44:36.840930+0000","last_undegraded":"2024-07-26T12:44:36.840930+0000","last_fullsized":"2024-07-26T12:44:36.840930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:20:39.080000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.839777+0000","last_change":"2024-07-26T12:43:12.172457+0000","last_active":"2024-07-26T12:44:36.839777+0000","last_peered":"2024-07-26T12:44:36.839777+0000","last_clean":"2024-07-26T12:44:36.839777+0000","last_became_active":"2024-07-26T12:43:12.172034+0000","last_became_peered":"2024-07-26T12:43:12.172034+0000","last_unstale":"2024-07-26T12:44:36.839777+0000","last_undegraded":"2024-07-26T12:44:36.839777+0000","last_fullsized":"2024-07-26T12:44:36.839777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:37:02.237852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807446+0000","last_change":"2024-07-26T12:43:08.227152+0000","last_active":"2024-07-26T12:44:36.807446+0000","last_peered":"2024-07-26T12:44:36.807446+0000","last_clean":"2024-07-26T12:44:36.807446+0000","last_became_active":"2024-07-26T12:43:08.226722+0000","last_became_peered":"2024-07-26T12:43:08.226722+0000","last_unstale":"2024-07-26T12:44:36.807446+0000","last_undegraded":"2024-07-26T12:44:36.807446+0000","last_fullsized":"2024-07-26T12:44:36.807446+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:16.293211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850777+0000","last_change":"2024-07-26T12:43:26.792999+0000","last_active":"2024-07-26T12:44:36.850777+0000","last_peered":"2024-07-26T12:44:36.850777+0000","last_clean":"2024-07-26T12:44:36.850777+0000","last_became_active":"2024-07-26T12:43:26.792515+0000","last_became_peered":"2024-07-26T12:43:26.792515+0000","last_unstale":"2024-07-26T12:44:36.850777+0000","last_undegraded":"2024-07-26T12:44:36.850777+0000","last_fullsized":"2024-07-26T12:44:36.850777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:49:35.104600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813498+0000","last_change":"2024-07-26T12:43:12.185381+0000","last_active":"2024-07-26T12:44:36.813498+0000","last_peered":"2024-07-26T12:44:36.813498+0000","last_clean":"2024-07-26T12:44:36.813498+0000","last_became_active":"2024-07-26T12:43:12.184892+0000","last_became_peered":"2024-07-26T12:43:12.184892+0000","last_unstale":"2024-07-26T12:44:36.813498+0000","last_undegraded":"2024-07-26T12:44:36.813498+0000","last_fullsized":"2024-07-26T12:44:36.813498+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:38:53.992586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.877700+0000","last_change":"2024-07-26T12:43:08.895079+0000","last_active":"2024-07-26T12:44:38.877700+0000","last_peered":"2024-07-26T12:44:38.877700+0000","last_clean":"2024-07-26T12:44:38.877700+0000","last_became_active":"2024-07-26T12:43:08.894653+0000","last_became_peered":"2024-07-26T12:43:08.894653+0000","last_unstale":"2024-07-26T12:44:38.877700+0000","last_undegraded":"2024-07-26T12:44:38.877700+0000","last_fullsized":"2024-07-26T12:44:38.877700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:02:53.137905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803224+0000","last_change":"2024-07-26T12:43:26.762802+0000","last_active":"2024-07-26T12:44:36.803224+0000","last_peered":"2024-07-26T12:44:36.803224+0000","last_clean":"2024-07-26T12:44:36.803224+0000","last_became_active":"2024-07-26T12:43:26.762211+0000","last_became_peered":"2024-07-26T12:43:26.762211+0000","last_unstale":"2024-07-26T12:44:36.803224+0000","last_undegraded":"2024-07-26T12:44:36.803224+0000","last_fullsized":"2024-07-26T12:44:36.803224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:35:19.623168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.806698+0000","last_change":"2024-07-26T12:43:12.176062+0000","last_active":"2024-07-26T12:44:36.806698+0000","last_peered":"2024-07-26T12:44:36.806698+0000","last_clean":"2024-07-26T12:44:36.806698+0000","last_became_active":"2024-07-26T12:43:12.175671+0000","last_became_peered":"2024-07-26T12:43:12.175671+0000","last_unstale":"2024-07-26T12:44:36.806698+0000","last_undegraded":"2024-07-26T12:44:36.806698+0000","last_fullsized":"2024-07-26T12:44:36.806698+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:48.374964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'16","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:07.219585+0000","last_change":"2024-07-26T12:43:08.131453+0000","last_active":"2024-07-26T12:45:07.219585+0000","last_peered":"2024-07-26T12:45:07.219585+0000","last_clean":"2024-07-26T12:45:07.219585+0000","last_became_active":"2024-07-26T12:43:08.130928+0000","last_became_peered":"2024-07-26T12:43:08.130928+0000","last_unstale":"2024-07-26T12:45:07.219585+0000","last_undegraded":"2024-07-26T12:45:07.219585+0000","last_fullsized":"2024-07-26T12:45:07.219585+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:43.190860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804235+0000","last_change":"2024-07-26T12:43:26.792942+0000","last_active":"2024-07-26T12:44:36.804235+0000","last_peered":"2024-07-26T12:44:36.804235+0000","last_clean":"2024-07-26T12:44:36.804235+0000","last_became_active":"2024-07-26T12:43:26.792045+0000","last_became_peered":"2024-07-26T12:43:26.792045+0000","last_unstale":"2024-07-26T12:44:36.804235+0000","last_undegraded":"2024-07-26T12:44:36.804235+0000","last_fullsized":"2024-07-26T12:44:36.804235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:56.005295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.811044+0000","last_change":"2024-07-26T12:43:12.187601+0000","last_active":"2024-07-26T12:44:36.811044+0000","last_peered":"2024-07-26T12:44:36.811044+0000","last_clean":"2024-07-26T12:44:36.811044+0000","last_became_active":"2024-07-26T12:43:12.187175+0000","last_became_peered":"2024-07-26T12:43:12.187175+0000","last_unstale":"2024-07-26T12:44:36.811044+0000","last_undegraded":"2024-07-26T12:44:36.811044+0000","last_fullsized":"2024-07-26T12:44:36.811044+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:06.992854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881924+0000","last_change":"2024-07-26T12:43:08.227279+0000","last_active":"2024-07-26T12:44:39.881924+0000","last_peered":"2024-07-26T12:44:39.881924+0000","last_clean":"2024-07-26T12:44:39.881924+0000","last_became_active":"2024-07-26T12:43:08.226634+0000","last_became_peered":"2024-07-26T12:43:08.226634+0000","last_unstale":"2024-07-26T12:44:39.881924+0000","last_undegraded":"2024-07-26T12:44:39.881924+0000","last_fullsized":"2024-07-26T12:44:39.881924+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:19.845664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850980+0000","last_change":"2024-07-26T12:43:26.774537+0000","last_active":"2024-07-26T12:44:36.850980+0000","last_peered":"2024-07-26T12:44:36.850980+0000","last_clean":"2024-07-26T12:44:36.850980+0000","last_became_active":"2024-07-26T12:43:26.774036+0000","last_became_peered":"2024-07-26T12:43:26.774036+0000","last_unstale":"2024-07-26T12:44:36.850980+0000","last_undegraded":"2024-07-26T12:44:36.850980+0000","last_fullsized":"2024-07-26T12:44:36.850980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:21.774498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.857982+0000","last_change":"2024-07-26T12:44:36.857443+0000","last_active":"2024-07-26T12:44:36.857982+0000","last_peered":"2024-07-26T12:44:36.857982+0000","last_clean":"2024-07-26T12:44:36.857982+0000","last_became_active":"2024-07-26T12:44:36.857001+0000","last_became_peered":"2024-07-26T12:44:36.857001+0000","last_unstale":"2024-07-26T12:44:36.857982+0000","last_undegraded":"2024-07-26T12:44:36.857982+0000","last_fullsized":"2024-07-26T12:44:36.857982+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:08.236404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.875394+0000","last_change":"2024-07-26T12:43:08.227432+0000","last_active":"2024-07-26T12:44:37.875394+0000","last_peered":"2024-07-26T12:44:37.875394+0000","last_clean":"2024-07-26T12:44:37.875394+0000","last_became_active":"2024-07-26T12:43:08.226645+0000","last_became_peered":"2024-07-26T12:43:08.226645+0000","last_unstale":"2024-07-26T12:44:37.875394+0000","last_undegraded":"2024-07-26T12:44:37.875394+0000","last_fullsized":"2024-07-26T12:44:37.875394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:23.498855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840264+0000","last_change":"2024-07-26T12:43:26.792874+0000","last_active":"2024-07-26T12:44:36.840264+0000","last_peered":"2024-07-26T12:44:36.840264+0000","last_clean":"2024-07-26T12:44:36.840264+0000","last_became_active":"2024-07-26T12:43:26.792371+0000","last_became_peered":"2024-07-26T12:43:26.792371+0000","last_unstale":"2024-07-26T12:44:36.840264+0000","last_undegraded":"2024-07-26T12:44:36.840264+0000","last_fullsized":"2024-07-26T12:44:36.840264+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:25.753111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809546+0000","last_change":"2024-07-26T12:43:12.185733+0000","last_active":"2024-07-26T12:44:36.809546+0000","last_peered":"2024-07-26T12:44:36.809546+0000","last_clean":"2024-07-26T12:44:36.809546+0000","last_became_active":"2024-07-26T12:43:12.185305+0000","last_became_peered":"2024-07-26T12:43:12.185305+0000","last_unstale":"2024-07-26T12:44:36.809546+0000","last_undegraded":"2024-07-26T12:44:36.809546+0000","last_fullsized":"2024-07-26T12:44:36.809546+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:44.634607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:15.954127+0000","last_change":"2024-07-26T12:43:08.118508+0000","last_active":"2024-07-26T12:45:15.954127+0000","last_peered":"2024-07-26T12:45:15.954127+0000","last_clean":"2024-07-26T12:45:15.954127+0000","last_became_active":"2024-07-26T12:43:08.118088+0000","last_became_peered":"2024-07-26T12:43:08.118088+0000","last_unstale":"2024-07-26T12:45:15.954127+0000","last_undegraded":"2024-07-26T12:45:15.954127+0000","last_fullsized":"2024-07-26T12:45:15.954127+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:23:46.474298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810206+0000","last_change":"2024-07-26T12:43:26.779328+0000","last_active":"2024-07-26T12:44:36.810206+0000","last_peered":"2024-07-26T12:44:36.810206+0000","last_clean":"2024-07-26T12:44:36.810206+0000","last_became_active":"2024-07-26T12:43:26.778912+0000","last_became_peered":"2024-07-26T12:43:26.778912+0000","last_unstale":"2024-07-26T12:44:36.810206+0000","last_undegraded":"2024-07-26T12:44:36.810206+0000","last_fullsized":"2024-07-26T12:44:36.810206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:58.364775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850823+0000","last_change":"2024-07-26T12:43:35.854997+0000","last_active":"2024-07-26T12:44:36.850823+0000","last_peered":"2024-07-26T12:44:36.850823+0000","last_clean":"2024-07-26T12:44:36.850823+0000","last_became_active":"2024-07-26T12:43:35.854387+0000","last_became_peered":"2024-07-26T12:43:35.854387+0000","last_unstale":"2024-07-26T12:44:36.850823+0000","last_undegraded":"2024-07-26T12:44:36.850823+0000","last_fullsized":"2024-07-26T12:44:36.850823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:04:20.489388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:16.974383+0000","last_change":"2024-07-26T12:43:08.227628+0000","last_active":"2024-07-26T12:45:16.974383+0000","last_peered":"2024-07-26T12:45:16.974383+0000","last_clean":"2024-07-26T12:45:16.974383+0000","last_became_active":"2024-07-26T12:43:08.227133+0000","last_became_peered":"2024-07-26T12:43:08.227133+0000","last_unstale":"2024-07-26T12:45:16.974383+0000","last_undegraded":"2024-07-26T12:45:16.974383+0000","last_fullsized":"2024-07-26T12:45:16.974383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:28:28.484315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810768+0000","last_change":"2024-07-26T12:43:26.794016+0000","last_active":"2024-07-26T12:44:36.810768+0000","last_peered":"2024-07-26T12:44:36.810768+0000","last_clean":"2024-07-26T12:44:36.810768+0000","last_became_active":"2024-07-26T12:43:26.792562+0000","last_became_peered":"2024-07-26T12:43:26.792562+0000","last_unstale":"2024-07-26T12:44:36.810768+0000","last_undegraded":"2024-07-26T12:44:36.810768+0000","last_fullsized":"2024-07-26T12:44:36.810768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:26.162840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809241+0000","last_change":"2024-07-26T12:43:12.173014+0000","last_active":"2024-07-26T12:44:36.809241+0000","last_peered":"2024-07-26T12:44:36.809241+0000","last_clean":"2024-07-26T12:44:36.809241+0000","last_became_active":"2024-07-26T12:43:12.172600+0000","last_became_peered":"2024-07-26T12:43:12.172600+0000","last_unstale":"2024-07-26T12:44:36.809241+0000","last_undegraded":"2024-07-26T12:44:36.809241+0000","last_fullsized":"2024-07-26T12:44:36.809241+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:59.842744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.872830+0000","last_change":"2024-07-26T12:43:08.119000+0000","last_active":"2024-07-26T12:44:38.872830+0000","last_peered":"2024-07-26T12:44:38.872830+0000","last_clean":"2024-07-26T12:44:38.872830+0000","last_became_active":"2024-07-26T12:43:08.118608+0000","last_became_peered":"2024-07-26T12:43:08.118608+0000","last_unstale":"2024-07-26T12:44:38.872830+0000","last_undegraded":"2024-07-26T12:44:38.872830+0000","last_fullsized":"2024-07-26T12:44:38.872830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:29:47.663449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851268+0000","last_change":"2024-07-26T12:43:35.914716+0000","last_active":"2024-07-26T12:44:36.851268+0000","last_peered":"2024-07-26T12:44:36.851268+0000","last_clean":"2024-07-26T12:44:36.851268+0000","last_became_active":"2024-07-26T12:43:35.914354+0000","last_became_peered":"2024-07-26T12:43:35.914354+0000","last_unstale":"2024-07-26T12:44:36.851268+0000","last_undegraded":"2024-07-26T12:44:36.851268+0000","last_fullsized":"2024-07-26T12:44:36.851268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:09:10.994141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805160+0000","last_change":"2024-07-26T12:43:12.169997+0000","last_active":"2024-07-26T12:44:36.805160+0000","last_peered":"2024-07-26T12:44:36.805160+0000","last_clean":"2024-07-26T12:44:36.805160+0000","last_became_active":"2024-07-26T12:43:12.169044+0000","last_became_peered":"2024-07-26T12:43:12.169044+0000","last_unstale":"2024-07-26T12:44:36.805160+0000","last_undegraded":"2024-07-26T12:44:36.805160+0000","last_fullsized":"2024-07-26T12:44:36.805160+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:41.791434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.882037+0000","last_change":"2024-07-26T12:43:08.129025+0000","last_active":"2024-07-26T12:44:39.882037+0000","last_peered":"2024-07-26T12:44:39.882037+0000","last_clean":"2024-07-26T12:44:39.882037+0000","last_became_active":"2024-07-26T12:43:08.128608+0000","last_became_peered":"2024-07-26T12:43:08.128608+0000","last_unstale":"2024-07-26T12:44:39.882037+0000","last_undegraded":"2024-07-26T12:44:39.882037+0000","last_fullsized":"2024-07-26T12:44:39.882037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:43.749473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":121,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805787+0000","last_change":"2024-07-26T12:41:42.411595+0000","last_active":"2024-07-26T12:44:36.805787+0000","last_peered":"2024-07-26T12:44:36.805787+0000","last_clean":"2024-07-26T12:44:36.805787+0000","last_became_active":"2024-07-26T12:41:42.407119+0000","last_became_peered":"2024-07-26T12:41:42.407119+0000","last_unstale":"2024-07-26T12:44:36.805787+0000","last_undegraded":"2024-07-26T12:44:36.805787+0000","last_fullsized":"2024-07-26T12:44:36.805787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_clean_scrub_stamp":"2024-07-26T12:37:59.605754+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:21.376418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805538+0000","last_change":"2024-07-26T12:43:26.763955+0000","last_active":"2024-07-26T12:44:36.805538+0000","last_peered":"2024-07-26T12:44:36.805538+0000","last_clean":"2024-07-26T12:44:36.805538+0000","last_became_active":"2024-07-26T12:43:26.763368+0000","last_became_peered":"2024-07-26T12:43:26.763368+0000","last_unstale":"2024-07-26T12:44:36.805538+0000","last_undegraded":"2024-07-26T12:44:36.805538+0000","last_fullsized":"2024-07-26T12:44:36.805538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:48:14.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805985+0000","last_change":"2024-07-26T12:43:12.174780+0000","last_active":"2024-07-26T12:44:36.805985+0000","last_peered":"2024-07-26T12:44:36.805985+0000","last_clean":"2024-07-26T12:44:36.805985+0000","last_became_active":"2024-07-26T12:43:12.174379+0000","last_became_peered":"2024-07-26T12:43:12.174379+0000","last_unstale":"2024-07-26T12:44:36.805985+0000","last_undegraded":"2024-07-26T12:44:36.805985+0000","last_fullsized":"2024-07-26T12:44:36.805985+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:42:22.783259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'6","reported_seq":94,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:17.949277+0000","last_change":"2024-07-26T12:43:08.894655+0000","last_active":"2024-07-26T12:45:17.949277+0000","last_peered":"2024-07-26T12:45:17.949277+0000","last_clean":"2024-07-26T12:45:17.949277+0000","last_became_active":"2024-07-26T12:43:08.894270+0000","last_became_peered":"2024-07-26T12:43:08.894270+0000","last_unstale":"2024-07-26T12:45:17.949277+0000","last_undegraded":"2024-07-26T12:45:17.949277+0000","last_fullsized":"2024-07-26T12:45:17.949277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:34.431204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807841+0000","last_change":"2024-07-26T12:43:26.769716+0000","last_active":"2024-07-26T12:44:36.807841+0000","last_peered":"2024-07-26T12:44:36.807841+0000","last_clean":"2024-07-26T12:44:36.807841+0000","last_became_active":"2024-07-26T12:43:26.769154+0000","last_became_peered":"2024-07-26T12:43:26.769154+0000","last_unstale":"2024-07-26T12:44:36.807841+0000","last_undegraded":"2024-07-26T12:44:36.807841+0000","last_fullsized":"2024-07-26T12:44:36.807841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:43.102498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805540+0000","last_change":"2024-07-26T12:43:12.179488+0000","last_active":"2024-07-26T12:44:36.805540+0000","last_peered":"2024-07-26T12:44:36.805540+0000","last_clean":"2024-07-26T12:44:36.805540+0000","last_became_active":"2024-07-26T12:43:12.179171+0000","last_became_peered":"2024-07-26T12:43:12.179171+0000","last_unstale":"2024-07-26T12:44:36.805540+0000","last_undegraded":"2024-07-26T12:44:36.805540+0000","last_fullsized":"2024-07-26T12:44:36.805540+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:43.695582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.877080+0000","last_change":"2024-07-26T12:43:08.130838+0000","last_active":"2024-07-26T12:44:38.877080+0000","last_peered":"2024-07-26T12:44:38.877080+0000","last_clean":"2024-07-26T12:44:38.877080+0000","last_became_active":"2024-07-26T12:43:08.130446+0000","last_became_peered":"2024-07-26T12:43:08.130446+0000","last_unstale":"2024-07-26T12:44:38.877080+0000","last_undegraded":"2024-07-26T12:44:38.877080+0000","last_fullsized":"2024-07-26T12:44:38.877080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:33:55.271038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.812600+0000","last_change":"2024-07-26T12:43:26.781982+0000","last_active":"2024-07-26T12:44:36.812600+0000","last_peered":"2024-07-26T12:44:36.812600+0000","last_clean":"2024-07-26T12:44:36.812600+0000","last_became_active":"2024-07-26T12:43:26.777102+0000","last_became_peered":"2024-07-26T12:43:26.777102+0000","last_unstale":"2024-07-26T12:44:36.812600+0000","last_undegraded":"2024-07-26T12:44:36.812600+0000","last_fullsized":"2024-07-26T12:44:36.812600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:49.553124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.806033+0000","last_change":"2024-07-26T12:43:12.177043+0000","last_active":"2024-07-26T12:44:36.806033+0000","last_peered":"2024-07-26T12:44:36.806033+0000","last_clean":"2024-07-26T12:44:36.806033+0000","last_became_active":"2024-07-26T12:43:12.176634+0000","last_became_peered":"2024-07-26T12:43:12.176634+0000","last_unstale":"2024-07-26T12:44:36.806033+0000","last_undegraded":"2024-07-26T12:44:36.806033+0000","last_fullsized":"2024-07-26T12:44:36.806033+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:20.210176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.877336+0000","last_change":"2024-07-26T12:43:08.131279+0000","last_active":"2024-07-26T12:44:38.877336+0000","last_peered":"2024-07-26T12:44:38.877336+0000","last_clean":"2024-07-26T12:44:38.877336+0000","last_became_active":"2024-07-26T12:43:08.130755+0000","last_became_peered":"2024-07-26T12:43:08.130755+0000","last_unstale":"2024-07-26T12:44:38.877336+0000","last_undegraded":"2024-07-26T12:44:38.877336+0000","last_fullsized":"2024-07-26T12:44:38.877336+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:03.234905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.849647+0000","last_change":"2024-07-26T12:43:26.764407+0000","last_active":"2024-07-26T12:44:36.849647+0000","last_peered":"2024-07-26T12:44:36.849647+0000","last_clean":"2024-07-26T12:44:36.849647+0000","last_became_active":"2024-07-26T12:43:26.762408+0000","last_became_peered":"2024-07-26T12:43:26.762408+0000","last_unstale":"2024-07-26T12:44:36.849647+0000","last_undegraded":"2024-07-26T12:44:36.849647+0000","last_fullsized":"2024-07-26T12:44:36.849647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:35:29.139476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851249+0000","last_change":"2024-07-26T12:43:12.174587+0000","last_active":"2024-07-26T12:44:36.851249+0000","last_peered":"2024-07-26T12:44:36.851249+0000","last_clean":"2024-07-26T12:44:36.851249+0000","last_became_active":"2024-07-26T12:43:12.173929+0000","last_became_peered":"2024-07-26T12:43:12.173929+0000","last_unstale":"2024-07-26T12:44:36.851249+0000","last_undegraded":"2024-07-26T12:44:36.851249+0000","last_fullsized":"2024-07-26T12:44:36.851249+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:16.775198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.845786+0000","last_change":"2024-07-26T12:43:08.119825+0000","last_active":"2024-07-26T12:44:37.845786+0000","last_peered":"2024-07-26T12:44:37.845786+0000","last_clean":"2024-07-26T12:44:37.845786+0000","last_became_active":"2024-07-26T12:43:08.119273+0000","last_became_peered":"2024-07-26T12:43:08.119273+0000","last_unstale":"2024-07-26T12:44:37.845786+0000","last_undegraded":"2024-07-26T12:44:37.845786+0000","last_fullsized":"2024-07-26T12:44:37.845786+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:39.973020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.849943+0000","last_change":"2024-07-26T12:43:26.773226+0000","last_active":"2024-07-26T12:44:36.849943+0000","last_peered":"2024-07-26T12:44:36.849943+0000","last_clean":"2024-07-26T12:44:36.849943+0000","last_became_active":"2024-07-26T12:43:26.772745+0000","last_became_peered":"2024-07-26T12:43:26.772745+0000","last_unstale":"2024-07-26T12:44:36.849943+0000","last_undegraded":"2024-07-26T12:44:36.849943+0000","last_fullsized":"2024-07-26T12:44:36.849943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:35.407138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850158+0000","last_change":"2024-07-26T12:43:12.176178+0000","last_active":"2024-07-26T12:44:36.850158+0000","last_peered":"2024-07-26T12:44:36.850158+0000","last_clean":"2024-07-26T12:44:36.850158+0000","last_became_active":"2024-07-26T12:43:12.175791+0000","last_became_peered":"2024-07-26T12:43:12.175791+0000","last_unstale":"2024-07-26T12:44:36.850158+0000","last_undegraded":"2024-07-26T12:44:36.850158+0000","last_fullsized":"2024-07-26T12:44:36.850158+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:48.885831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.874508+0000","last_change":"2024-07-26T12:43:08.226345+0000","last_active":"2024-07-26T12:44:38.874508+0000","last_peered":"2024-07-26T12:44:38.874508+0000","last_clean":"2024-07-26T12:44:38.874508+0000","last_became_active":"2024-07-26T12:43:08.225897+0000","last_became_peered":"2024-07-26T12:43:08.225897+0000","last_unstale":"2024-07-26T12:44:38.874508+0000","last_undegraded":"2024-07-26T12:44:38.874508+0000","last_fullsized":"2024-07-26T12:44:38.874508+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:59.351076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810637+0000","last_change":"2024-07-26T12:43:26.774800+0000","last_active":"2024-07-26T12:44:36.810637+0000","last_peered":"2024-07-26T12:44:36.810637+0000","last_clean":"2024-07-26T12:44:36.810637+0000","last_became_active":"2024-07-26T12:43:26.774192+0000","last_became_peered":"2024-07-26T12:43:26.774192+0000","last_unstale":"2024-07-26T12:44:36.810637+0000","last_undegraded":"2024-07-26T12:44:36.810637+0000","last_fullsized":"2024-07-26T12:44:36.810637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:18.379619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805223+0000","last_change":"2024-07-26T12:43:12.171720+0000","last_active":"2024-07-26T12:44:36.805223+0000","last_peered":"2024-07-26T12:44:36.805223+0000","last_clean":"2024-07-26T12:44:36.805223+0000","last_became_active":"2024-07-26T12:43:12.171334+0000","last_became_peered":"2024-07-26T12:43:12.171334+0000","last_unstale":"2024-07-26T12:44:36.805223+0000","last_undegraded":"2024-07-26T12:44:36.805223+0000","last_fullsized":"2024-07-26T12:44:36.805223+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:35.156430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:14.971833+0000","last_change":"2024-07-26T12:43:08.120325+0000","last_active":"2024-07-26T12:45:14.971833+0000","last_peered":"2024-07-26T12:45:14.971833+0000","last_clean":"2024-07-26T12:45:14.971833+0000","last_became_active":"2024-07-26T12:43:08.119896+0000","last_became_peered":"2024-07-26T12:43:08.119896+0000","last_unstale":"2024-07-26T12:45:14.971833+0000","last_undegraded":"2024-07-26T12:45:14.971833+0000","last_fullsized":"2024-07-26T12:45:14.971833+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:23:19.470491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.856160+0000","last_change":"2024-07-26T12:43:26.778688+0000","last_active":"2024-07-26T12:44:36.856160+0000","last_peered":"2024-07-26T12:44:36.856160+0000","last_clean":"2024-07-26T12:44:36.856160+0000","last_became_active":"2024-07-26T12:43:26.778239+0000","last_became_peered":"2024-07-26T12:43:26.778239+0000","last_unstale":"2024-07-26T12:44:36.856160+0000","last_undegraded":"2024-07-26T12:44:36.856160+0000","last_fullsized":"2024-07-26T12:44:36.856160+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:50:48.409636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805469+0000","last_change":"2024-07-26T12:43:12.182323+0000","last_active":"2024-07-26T12:44:36.805469+0000","last_peered":"2024-07-26T12:44:36.805469+0000","last_clean":"2024-07-26T12:44:36.805469+0000","last_became_active":"2024-07-26T12:43:12.181860+0000","last_became_peered":"2024-07-26T12:43:12.181860+0000","last_unstale":"2024-07-26T12:44:36.805469+0000","last_undegraded":"2024-07-26T12:44:36.805469+0000","last_fullsized":"2024-07-26T12:44:36.805469+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:08.764979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.877164+0000","last_change":"2024-07-26T12:43:08.130823+0000","last_active":"2024-07-26T12:44:38.877164+0000","last_peered":"2024-07-26T12:44:38.877164+0000","last_clean":"2024-07-26T12:44:38.877164+0000","last_became_active":"2024-07-26T12:43:08.130424+0000","last_became_peered":"2024-07-26T12:43:08.130424+0000","last_unstale":"2024-07-26T12:44:38.877164+0000","last_undegraded":"2024-07-26T12:44:38.877164+0000","last_fullsized":"2024-07-26T12:44:38.877164+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:49:35.829624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850635+0000","last_change":"2024-07-26T12:43:26.770070+0000","last_active":"2024-07-26T12:44:36.850635+0000","last_peered":"2024-07-26T12:44:36.850635+0000","last_clean":"2024-07-26T12:44:36.850635+0000","last_became_active":"2024-07-26T12:43:26.769594+0000","last_became_peered":"2024-07-26T12:43:26.769594+0000","last_unstale":"2024-07-26T12:44:36.850635+0000","last_undegraded":"2024-07-26T12:44:36.850635+0000","last_fullsized":"2024-07-26T12:44:36.850635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:18.342739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809942+0000","last_change":"2024-07-26T12:43:12.183134+0000","last_active":"2024-07-26T12:44:36.809942+0000","last_peered":"2024-07-26T12:44:36.809942+0000","last_clean":"2024-07-26T12:44:36.809942+0000","last_became_active":"2024-07-26T12:43:12.182406+0000","last_became_peered":"2024-07-26T12:43:12.182406+0000","last_unstale":"2024-07-26T12:44:36.809942+0000","last_undegraded":"2024-07-26T12:44:36.809942+0000","last_fullsized":"2024-07-26T12:44:36.809942+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:54:24.038971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881522+0000","last_change":"2024-07-26T12:43:35.862005+0000","last_active":"2024-07-26T12:44:39.881522+0000","last_peered":"2024-07-26T12:44:39.881522+0000","last_clean":"2024-07-26T12:44:39.881522+0000","last_became_active":"2024-07-26T12:43:35.850911+0000","last_became_peered":"2024-07-26T12:43:35.850911+0000","last_unstale":"2024-07-26T12:44:39.881522+0000","last_undegraded":"2024-07-26T12:44:39.881522+0000","last_fullsized":"2024-07-26T12:44:39.881522+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:18:16.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850128+0000","last_change":"2024-07-26T12:43:26.774599+0000","last_active":"2024-07-26T12:44:36.850128+0000","last_peered":"2024-07-26T12:44:36.850128+0000","last_clean":"2024-07-26T12:44:36.850128+0000","last_became_active":"2024-07-26T12:43:26.774186+0000","last_became_peered":"2024-07-26T12:43:26.774186+0000","last_unstale":"2024-07-26T12:44:36.850128+0000","last_undegraded":"2024-07-26T12:44:36.850128+0000","last_fullsized":"2024-07-26T12:44:36.850128+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:02:42.252093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840792+0000","last_change":"2024-07-26T12:43:12.174676+0000","last_active":"2024-07-26T12:44:36.840792+0000","last_peered":"2024-07-26T12:44:36.840792+0000","last_clean":"2024-07-26T12:44:36.840792+0000","last_became_active":"2024-07-26T12:43:12.174310+0000","last_became_peered":"2024-07-26T12:43:12.174310+0000","last_unstale":"2024-07-26T12:44:36.840792+0000","last_undegraded":"2024-07-26T12:44:36.840792+0000","last_fullsized":"2024-07-26T12:44:36.840792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:44.381289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":166,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:16.946988+0000","last_change":"2024-07-26T12:43:08.122675+0000","last_active":"2024-07-26T12:45:16.946988+0000","last_peered":"2024-07-26T12:45:16.946988+0000","last_clean":"2024-07-26T12:45:16.946988+0000","last_became_active":"2024-07-26T12:43:08.122174+0000","last_became_peered":"2024-07-26T12:43:08.122174+0000","last_unstale":"2024-07-26T12:45:16.946988+0000","last_undegraded":"2024-07-26T12:45:16.946988+0000","last_fullsized":"2024-07-26T12:45:16.946988+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:54.513306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64136,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":85,"num_read_kb":105,"num_write":38,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.854038+0000","last_change":"2024-07-26T12:43:26.764283+0000","last_active":"2024-07-26T12:44:36.854038+0000","last_peered":"2024-07-26T12:44:36.854038+0000","last_clean":"2024-07-26T12:44:36.854038+0000","last_became_active":"2024-07-26T12:43:26.763821+0000","last_became_peered":"2024-07-26T12:43:26.763821+0000","last_unstale":"2024-07-26T12:44:36.854038+0000","last_undegraded":"2024-07-26T12:44:36.854038+0000","last_fullsized":"2024-07-26T12:44:36.854038+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:56.753016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807726+0000","last_change":"2024-07-26T12:43:35.915277+0000","last_active":"2024-07-26T12:44:36.807726+0000","last_peered":"2024-07-26T12:44:36.807726+0000","last_clean":"2024-07-26T12:44:36.807726+0000","last_became_active":"2024-07-26T12:43:35.914848+0000","last_became_peered":"2024-07-26T12:43:35.914848+0000","last_unstale":"2024-07-26T12:44:36.807726+0000","last_undegraded":"2024-07-26T12:44:36.807726+0000","last_fullsized":"2024-07-26T12:44:36.807726+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:56:35.639076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.874973+0000","last_change":"2024-07-26T12:43:08.227352+0000","last_active":"2024-07-26T12:44:37.874973+0000","last_peered":"2024-07-26T12:44:37.874973+0000","last_clean":"2024-07-26T12:44:37.874973+0000","last_became_active":"2024-07-26T12:43:08.226857+0000","last_became_peered":"2024-07-26T12:43:08.226857+0000","last_unstale":"2024-07-26T12:44:37.874973+0000","last_undegraded":"2024-07-26T12:44:37.874973+0000","last_fullsized":"2024-07-26T12:44:37.874973+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:20.156503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850687+0000","last_change":"2024-07-26T12:43:26.762943+0000","last_active":"2024-07-26T12:44:36.850687+0000","last_peered":"2024-07-26T12:44:36.850687+0000","last_clean":"2024-07-26T12:44:36.850687+0000","last_became_active":"2024-07-26T12:43:26.762436+0000","last_became_peered":"2024-07-26T12:43:26.762436+0000","last_unstale":"2024-07-26T12:44:36.850687+0000","last_undegraded":"2024-07-26T12:44:36.850687+0000","last_fullsized":"2024-07-26T12:44:36.850687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:05:23.924397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.854788+0000","last_change":"2024-07-26T12:44:36.854705+0000","last_active":"2024-07-26T12:44:36.854788+0000","last_peered":"2024-07-26T12:44:36.854788+0000","last_clean":"2024-07-26T12:44:36.854788+0000","last_became_active":"2024-07-26T12:44:36.854268+0000","last_became_peered":"2024-07-26T12:44:36.854268+0000","last_unstale":"2024-07-26T12:44:36.854788+0000","last_undegraded":"2024-07-26T12:44:36.854788+0000","last_fullsized":"2024-07-26T12:44:36.854788+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:24:30.768905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.874500+0000","last_change":"2024-07-26T12:43:08.118597+0000","last_active":"2024-07-26T12:44:38.874500+0000","last_peered":"2024-07-26T12:44:38.874500+0000","last_clean":"2024-07-26T12:44:38.874500+0000","last_became_active":"2024-07-26T12:43:08.118145+0000","last_became_peered":"2024-07-26T12:43:08.118145+0000","last_unstale":"2024-07-26T12:44:38.874500+0000","last_undegraded":"2024-07-26T12:44:38.874500+0000","last_fullsized":"2024-07-26T12:44:38.874500+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:52:45.511693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.853564+0000","last_change":"2024-07-26T12:43:26.771861+0000","last_active":"2024-07-26T12:44:36.853564+0000","last_peered":"2024-07-26T12:44:36.853564+0000","last_clean":"2024-07-26T12:44:36.853564+0000","last_became_active":"2024-07-26T12:43:26.771323+0000","last_became_peered":"2024-07-26T12:43:26.771323+0000","last_unstale":"2024-07-26T12:44:36.853564+0000","last_undegraded":"2024-07-26T12:44:36.853564+0000","last_fullsized":"2024-07-26T12:44:36.853564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:34.207514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.808442+0000","last_change":"2024-07-26T12:43:12.169437+0000","last_active":"2024-07-26T12:44:36.808442+0000","last_peered":"2024-07-26T12:44:36.808442+0000","last_clean":"2024-07-26T12:44:36.808442+0000","last_became_active":"2024-07-26T12:43:12.168387+0000","last_became_peered":"2024-07-26T12:43:12.168387+0000","last_unstale":"2024-07-26T12:44:36.808442+0000","last_undegraded":"2024-07-26T12:44:36.808442+0000","last_fullsized":"2024-07-26T12:44:36.808442+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:57:06.148965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":125,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:16.947135+0000","last_change":"2024-07-26T12:43:08.129618+0000","last_active":"2024-07-26T12:45:16.947135+0000","last_peered":"2024-07-26T12:45:16.947135+0000","last_clean":"2024-07-26T12:45:16.947135+0000","last_became_active":"2024-07-26T12:43:08.129064+0000","last_became_peered":"2024-07-26T12:43:08.129064+0000","last_unstale":"2024-07-26T12:45:16.947135+0000","last_undegraded":"2024-07-26T12:45:16.947135+0000","last_fullsized":"2024-07-26T12:45:16.947135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:26:22.406075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804352+0000","last_change":"2024-07-26T12:43:35.854710+0000","last_active":"2024-07-26T12:44:36.804352+0000","last_peered":"2024-07-26T12:44:36.804352+0000","last_clean":"2024-07-26T12:44:36.804352+0000","last_became_active":"2024-07-26T12:43:35.852772+0000","last_became_peered":"2024-07-26T12:43:35.852772+0000","last_unstale":"2024-07-26T12:44:36.804352+0000","last_undegraded":"2024-07-26T12:44:36.804352+0000","last_fullsized":"2024-07-26T12:44:36.804352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:24:54.015879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813169+0000","last_change":"2024-07-26T12:43:26.783182+0000","last_active":"2024-07-26T12:44:36.813169+0000","last_peered":"2024-07-26T12:44:36.813169+0000","last_clean":"2024-07-26T12:44:36.813169+0000","last_became_active":"2024-07-26T12:43:26.782697+0000","last_became_peered":"2024-07-26T12:43:26.782697+0000","last_unstale":"2024-07-26T12:44:36.813169+0000","last_undegraded":"2024-07-26T12:44:36.813169+0000","last_fullsized":"2024-07-26T12:44:36.813169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:57.169222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.853634+0000","last_change":"2024-07-26T12:43:12.166163+0000","last_active":"2024-07-26T12:44:36.853634+0000","last_peered":"2024-07-26T12:44:36.853634+0000","last_clean":"2024-07-26T12:44:36.853634+0000","last_became_active":"2024-07-26T12:43:12.164427+0000","last_became_peered":"2024-07-26T12:43:12.164427+0000","last_unstale":"2024-07-26T12:44:36.853634+0000","last_undegraded":"2024-07-26T12:44:36.853634+0000","last_fullsized":"2024-07-26T12:44:36.853634+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:25.335508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.874150+0000","last_change":"2024-07-26T12:43:08.118224+0000","last_active":"2024-07-26T12:44:38.874150+0000","last_peered":"2024-07-26T12:44:38.874150+0000","last_clean":"2024-07-26T12:44:38.874150+0000","last_became_active":"2024-07-26T12:43:08.117821+0000","last_became_peered":"2024-07-26T12:43:08.117821+0000","last_unstale":"2024-07-26T12:44:38.874150+0000","last_undegraded":"2024-07-26T12:44:38.874150+0000","last_fullsized":"2024-07-26T12:44:38.874150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:56:14.380931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851063+0000","last_change":"2024-07-26T12:43:35.907740+0000","last_active":"2024-07-26T12:44:36.851063+0000","last_peered":"2024-07-26T12:44:36.851063+0000","last_clean":"2024-07-26T12:44:36.851063+0000","last_became_active":"2024-07-26T12:43:35.907115+0000","last_became_peered":"2024-07-26T12:43:35.907115+0000","last_unstale":"2024-07-26T12:44:36.851063+0000","last_undegraded":"2024-07-26T12:44:36.851063+0000","last_fullsized":"2024-07-26T12:44:36.851063+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:53:24.783872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804682+0000","last_change":"2024-07-26T12:43:12.177579+0000","last_active":"2024-07-26T12:44:36.804682+0000","last_peered":"2024-07-26T12:44:36.804682+0000","last_clean":"2024-07-26T12:44:36.804682+0000","last_became_active":"2024-07-26T12:43:12.171433+0000","last_became_peered":"2024-07-26T12:43:12.171433+0000","last_unstale":"2024-07-26T12:44:36.804682+0000","last_undegraded":"2024-07-26T12:44:36.804682+0000","last_fullsized":"2024-07-26T12:44:36.804682+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:50.972151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.853291+0000","last_change":"2024-07-26T12:43:08.133446+0000","last_active":"2024-07-26T12:44:36.853291+0000","last_peered":"2024-07-26T12:44:36.853291+0000","last_clean":"2024-07-26T12:44:36.853291+0000","last_became_active":"2024-07-26T12:43:08.133008+0000","last_became_peered":"2024-07-26T12:43:08.133008+0000","last_unstale":"2024-07-26T12:44:36.853291+0000","last_undegraded":"2024-07-26T12:44:36.853291+0000","last_fullsized":"2024-07-26T12:44:36.853291+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:05:21.283738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.873296+0000","last_change":"2024-07-26T12:43:08.124976+0000","last_active":"2024-07-26T12:44:37.873296+0000","last_peered":"2024-07-26T12:44:37.873296+0000","last_clean":"2024-07-26T12:44:37.873296+0000","last_became_active":"2024-07-26T12:43:08.124664+0000","last_became_peered":"2024-07-26T12:43:08.124664+0000","last_unstale":"2024-07-26T12:44:37.873296+0000","last_undegraded":"2024-07-26T12:44:37.873296+0000","last_fullsized":"2024-07-26T12:44:37.873296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:51.798657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850488+0000","last_change":"2024-07-26T12:43:35.856266+0000","last_active":"2024-07-26T12:44:36.850488+0000","last_peered":"2024-07-26T12:44:36.850488+0000","last_clean":"2024-07-26T12:44:36.850488+0000","last_became_active":"2024-07-26T12:43:35.855802+0000","last_became_peered":"2024-07-26T12:43:35.855802+0000","last_unstale":"2024-07-26T12:44:36.850488+0000","last_undegraded":"2024-07-26T12:44:36.850488+0000","last_fullsized":"2024-07-26T12:44:36.850488+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:16.432983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.802797+0000","last_change":"2024-07-26T12:43:26.778626+0000","last_active":"2024-07-26T12:44:36.802797+0000","last_peered":"2024-07-26T12:44:36.802797+0000","last_clean":"2024-07-26T12:44:36.802797+0000","last_became_active":"2024-07-26T12:43:26.778165+0000","last_became_peered":"2024-07-26T12:43:26.778165+0000","last_unstale":"2024-07-26T12:44:36.802797+0000","last_undegraded":"2024-07-26T12:44:36.802797+0000","last_fullsized":"2024-07-26T12:44:36.802797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:06:08.766495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":94,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:17.949026+0000","last_change":"2024-07-26T12:43:08.119702+0000","last_active":"2024-07-26T12:45:17.949026+0000","last_peered":"2024-07-26T12:45:17.949026+0000","last_clean":"2024-07-26T12:45:17.949026+0000","last_became_active":"2024-07-26T12:43:08.119225+0000","last_became_peered":"2024-07-26T12:43:08.119225+0000","last_unstale":"2024-07-26T12:45:17.949026+0000","last_undegraded":"2024-07-26T12:45:17.949026+0000","last_fullsized":"2024-07-26T12:45:17.949026+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:27.616873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840532+0000","last_change":"2024-07-26T12:43:12.166239+0000","last_active":"2024-07-26T12:44:36.840532+0000","last_peered":"2024-07-26T12:44:36.840532+0000","last_clean":"2024-07-26T12:44:36.840532+0000","last_became_active":"2024-07-26T12:43:12.165770+0000","last_became_peered":"2024-07-26T12:43:12.165770+0000","last_unstale":"2024-07-26T12:44:36.840532+0000","last_undegraded":"2024-07-26T12:44:36.840532+0000","last_fullsized":"2024-07-26T12:44:36.840532+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:11:15.290898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.802985+0000","last_change":"2024-07-26T12:43:26.765905+0000","last_active":"2024-07-26T12:44:36.802985+0000","last_peered":"2024-07-26T12:44:36.802985+0000","last_clean":"2024-07-26T12:44:36.802985+0000","last_became_active":"2024-07-26T12:43:26.765425+0000","last_became_peered":"2024-07-26T12:43:26.765425+0000","last_unstale":"2024-07-26T12:44:36.802985+0000","last_undegraded":"2024-07-26T12:44:36.802985+0000","last_fullsized":"2024-07-26T12:44:36.802985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:16.786604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850553+0000","last_change":"2024-07-26T12:43:12.171400+0000","last_active":"2024-07-26T12:44:36.850553+0000","last_peered":"2024-07-26T12:44:36.850553+0000","last_clean":"2024-07-26T12:44:36.850553+0000","last_became_active":"2024-07-26T12:43:12.170998+0000","last_became_peered":"2024-07-26T12:43:12.170998+0000","last_unstale":"2024-07-26T12:44:36.850553+0000","last_undegraded":"2024-07-26T12:44:36.850553+0000","last_fullsized":"2024-07-26T12:44:36.850553+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:43.251975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803434+0000","last_change":"2024-07-26T12:43:35.912729+0000","last_active":"2024-07-26T12:44:36.803434+0000","last_peered":"2024-07-26T12:44:36.803434+0000","last_clean":"2024-07-26T12:44:36.803434+0000","last_became_active":"2024-07-26T12:43:35.912318+0000","last_became_peered":"2024-07-26T12:43:35.912318+0000","last_unstale":"2024-07-26T12:44:36.803434+0000","last_undegraded":"2024-07-26T12:44:36.803434+0000","last_fullsized":"2024-07-26T12:44:36.803434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:14.473536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803226+0000","last_change":"2024-07-26T12:43:12.182933+0000","last_active":"2024-07-26T12:44:36.803226+0000","last_peered":"2024-07-26T12:44:36.803226+0000","last_clean":"2024-07-26T12:44:36.803226+0000","last_became_active":"2024-07-26T12:43:12.182656+0000","last_became_peered":"2024-07-26T12:43:12.182656+0000","last_unstale":"2024-07-26T12:44:36.803226+0000","last_undegraded":"2024-07-26T12:44:36.803226+0000","last_fullsized":"2024-07-26T12:44:36.803226+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:05:59.717662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.812876+0000","last_change":"2024-07-26T12:43:12.181196+0000","last_active":"2024-07-26T12:44:36.812876+0000","last_peered":"2024-07-26T12:44:36.812876+0000","last_clean":"2024-07-26T12:44:36.812876+0000","last_became_active":"2024-07-26T12:43:12.180765+0000","last_became_peered":"2024-07-26T12:43:12.180765+0000","last_unstale":"2024-07-26T12:44:36.812876+0000","last_undegraded":"2024-07-26T12:44:36.812876+0000","last_fullsized":"2024-07-26T12:44:36.812876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:20.590023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810869+0000","last_change":"2024-07-26T12:43:12.185408+0000","last_active":"2024-07-26T12:44:36.810869+0000","last_peered":"2024-07-26T12:44:36.810869+0000","last_clean":"2024-07-26T12:44:36.810869+0000","last_became_active":"2024-07-26T12:43:12.185038+0000","last_became_peered":"2024-07-26T12:43:12.185038+0000","last_unstale":"2024-07-26T12:44:36.810869+0000","last_undegraded":"2024-07-26T12:44:36.810869+0000","last_fullsized":"2024-07-26T12:44:36.810869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:55:30.604995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851270+0000","last_change":"2024-07-26T12:43:12.169301+0000","last_active":"2024-07-26T12:44:36.851270+0000","last_peered":"2024-07-26T12:44:36.851270+0000","last_clean":"2024-07-26T12:44:36.851270+0000","last_became_active":"2024-07-26T12:43:12.168923+0000","last_became_peered":"2024-07-26T12:43:12.168923+0000","last_unstale":"2024-07-26T12:44:36.851270+0000","last_undegraded":"2024-07-26T12:44:36.851270+0000","last_fullsized":"2024-07-26T12:44:36.851270+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:04.506485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803145+0000","last_change":"2024-07-26T12:43:12.166663+0000","last_active":"2024-07-26T12:44:36.803145+0000","last_peered":"2024-07-26T12:44:36.803145+0000","last_clean":"2024-07-26T12:44:36.803145+0000","last_became_active":"2024-07-26T12:43:12.166200+0000","last_became_peered":"2024-07-26T12:43:12.166200+0000","last_unstale":"2024-07-26T12:44:36.803145+0000","last_undegraded":"2024-07-26T12:44:36.803145+0000","last_fullsized":"2024-07-26T12:44:36.803145+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:14.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.806579+0000","last_change":"2024-07-26T12:43:12.171273+0000","last_active":"2024-07-26T12:44:36.806579+0000","last_peered":"2024-07-26T12:44:36.806579+0000","last_clean":"2024-07-26T12:44:36.806579+0000","last_became_active":"2024-07-26T12:43:12.167680+0000","last_became_peered":"2024-07-26T12:43:12.167680+0000","last_unstale":"2024-07-26T12:44:36.806579+0000","last_undegraded":"2024-07-26T12:44:36.806579+0000","last_fullsized":"2024-07-26T12:44:36.806579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:44.797644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850681+0000","last_change":"2024-07-26T12:43:12.179891+0000","last_active":"2024-07-26T12:44:36.850681+0000","last_peered":"2024-07-26T12:44:36.850681+0000","last_clean":"2024-07-26T12:44:36.850681+0000","last_became_active":"2024-07-26T12:43:12.179518+0000","last_became_peered":"2024-07-26T12:43:12.179518+0000","last_unstale":"2024-07-26T12:44:36.850681+0000","last_undegraded":"2024-07-26T12:44:36.850681+0000","last_fullsized":"2024-07-26T12:44:36.850681+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:24.246023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804512+0000","last_change":"2024-07-26T12:43:12.168521+0000","last_active":"2024-07-26T12:44:36.804512+0000","last_peered":"2024-07-26T12:44:36.804512+0000","last_clean":"2024-07-26T12:44:36.804512+0000","last_became_active":"2024-07-26T12:43:12.168126+0000","last_became_peered":"2024-07-26T12:43:12.168126+0000","last_unstale":"2024-07-26T12:44:36.804512+0000","last_undegraded":"2024-07-26T12:44:36.804512+0000","last_fullsized":"2024-07-26T12:44:36.804512+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:00.135090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.843425+0000","last_change":"2024-07-26T12:44:36.843322+0000","last_active":"2024-07-26T12:44:36.843425+0000","last_peered":"2024-07-26T12:44:36.843425+0000","last_clean":"2024-07-26T12:44:36.843425+0000","last_became_active":"2024-07-26T12:44:36.842821+0000","last_became_peered":"2024-07-26T12:44:36.842821+0000","last_unstale":"2024-07-26T12:44:36.843425+0000","last_undegraded":"2024-07-26T12:44:36.843425+0000","last_fullsized":"2024-07-26T12:44:36.843425+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:05.609298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809985+0000","last_change":"2024-07-26T12:43:12.185565+0000","last_active":"2024-07-26T12:44:36.809985+0000","last_peered":"2024-07-26T12:44:36.809985+0000","last_clean":"2024-07-26T12:44:36.809985+0000","last_became_active":"2024-07-26T12:43:12.185057+0000","last_became_peered":"2024-07-26T12:43:12.185057+0000","last_unstale":"2024-07-26T12:44:36.809985+0000","last_undegraded":"2024-07-26T12:44:36.809985+0000","last_fullsized":"2024-07-26T12:44:36.809985+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:58.108311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851009+0000","last_change":"2024-07-26T12:43:12.173301+0000","last_active":"2024-07-26T12:44:36.851009+0000","last_peered":"2024-07-26T12:44:36.851009+0000","last_clean":"2024-07-26T12:44:36.851009+0000","last_became_active":"2024-07-26T12:43:12.172955+0000","last_became_peered":"2024-07-26T12:43:12.172955+0000","last_unstale":"2024-07-26T12:44:36.851009+0000","last_undegraded":"2024-07-26T12:44:36.851009+0000","last_fullsized":"2024-07-26T12:44:36.851009+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:41.731141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803067+0000","last_change":"2024-07-26T12:43:12.176169+0000","last_active":"2024-07-26T12:44:36.803067+0000","last_peered":"2024-07-26T12:44:36.803067+0000","last_clean":"2024-07-26T12:44:36.803067+0000","last_became_active":"2024-07-26T12:43:12.175745+0000","last_became_peered":"2024-07-26T12:43:12.175745+0000","last_unstale":"2024-07-26T12:44:36.803067+0000","last_undegraded":"2024-07-26T12:44:36.803067+0000","last_fullsized":"2024-07-26T12:44:36.803067+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:20.393207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.856834+0000","last_change":"2024-07-26T12:43:12.168182+0000","last_active":"2024-07-26T12:44:36.856834+0000","last_peered":"2024-07-26T12:44:36.856834+0000","last_clean":"2024-07-26T12:44:36.856834+0000","last_became_active":"2024-07-26T12:43:12.163771+0000","last_became_peered":"2024-07-26T12:43:12.163771+0000","last_unstale":"2024-07-26T12:44:36.856834+0000","last_undegraded":"2024-07-26T12:44:36.856834+0000","last_fullsized":"2024-07-26T12:44:36.856834+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:40:30.960153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851443+0000","last_change":"2024-07-26T12:43:12.189524+0000","last_active":"2024-07-26T12:44:36.851443+0000","last_peered":"2024-07-26T12:44:36.851443+0000","last_clean":"2024-07-26T12:44:36.851443+0000","last_became_active":"2024-07-26T12:43:12.189165+0000","last_became_peered":"2024-07-26T12:43:12.189165+0000","last_unstale":"2024-07-26T12:44:36.851443+0000","last_undegraded":"2024-07-26T12:44:36.851443+0000","last_fullsized":"2024-07-26T12:44:36.851443+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:11:08.479058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807287+0000","last_change":"2024-07-26T12:43:12.180603+0000","last_active":"2024-07-26T12:44:36.807287+0000","last_peered":"2024-07-26T12:44:36.807287+0000","last_clean":"2024-07-26T12:44:36.807287+0000","last_became_active":"2024-07-26T12:43:12.180204+0000","last_became_peered":"2024-07-26T12:43:12.180204+0000","last_unstale":"2024-07-26T12:44:36.807287+0000","last_undegraded":"2024-07-26T12:44:36.807287+0000","last_fullsized":"2024-07-26T12:44:36.807287+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:46.768213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.812128+0000","last_change":"2024-07-26T12:43:12.185347+0000","last_active":"2024-07-26T12:44:36.812128+0000","last_peered":"2024-07-26T12:44:36.812128+0000","last_clean":"2024-07-26T12:44:36.812128+0000","last_became_active":"2024-07-26T12:43:12.180190+0000","last_became_peered":"2024-07-26T12:43:12.180190+0000","last_unstale":"2024-07-26T12:44:36.812128+0000","last_undegraded":"2024-07-26T12:44:36.812128+0000","last_fullsized":"2024-07-26T12:44:36.812128+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:07.654967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.812348+0000","last_change":"2024-07-26T12:43:12.177258+0000","last_active":"2024-07-26T12:44:36.812348+0000","last_peered":"2024-07-26T12:44:36.812348+0000","last_clean":"2024-07-26T12:44:36.812348+0000","last_became_active":"2024-07-26T12:43:12.176879+0000","last_became_peered":"2024-07-26T12:43:12.176879+0000","last_unstale":"2024-07-26T12:44:36.812348+0000","last_undegraded":"2024-07-26T12:44:36.812348+0000","last_fullsized":"2024-07-26T12:44:36.812348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:24.244759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.857476+0000","last_change":"2024-07-26T12:44:36.857403+0000","last_active":"2024-07-26T12:44:36.857476+0000","last_peered":"2024-07-26T12:44:36.857476+0000","last_clean":"2024-07-26T12:44:36.857476+0000","last_became_active":"2024-07-26T12:44:36.856980+0000","last_became_peered":"2024-07-26T12:44:36.856980+0000","last_unstale":"2024-07-26T12:44:36.857476+0000","last_undegraded":"2024-07-26T12:44:36.857476+0000","last_fullsized":"2024-07-26T12:44:36.857476+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.166383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850844+0000","last_change":"2024-07-26T12:43:12.166827+0000","last_active":"2024-07-26T12:44:36.850844+0000","last_peered":"2024-07-26T12:44:36.850844+0000","last_clean":"2024-07-26T12:44:36.850844+0000","last_became_active":"2024-07-26T12:43:12.165222+0000","last_became_peered":"2024-07-26T12:43:12.165222+0000","last_unstale":"2024-07-26T12:44:36.850844+0000","last_undegraded":"2024-07-26T12:44:36.850844+0000","last_fullsized":"2024-07-26T12:44:36.850844+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:37:42.219844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805845+0000","last_change":"2024-07-26T12:43:12.176961+0000","last_active":"2024-07-26T12:44:36.805845+0000","last_peered":"2024-07-26T12:44:36.805845+0000","last_clean":"2024-07-26T12:44:36.805845+0000","last_became_active":"2024-07-26T12:43:12.176550+0000","last_became_peered":"2024-07-26T12:43:12.176550+0000","last_unstale":"2024-07-26T12:44:36.805845+0000","last_undegraded":"2024-07-26T12:44:36.805845+0000","last_fullsized":"2024-07-26T12:44:36.805845+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:36:19.720801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809896+0000","last_change":"2024-07-26T12:43:12.173560+0000","last_active":"2024-07-26T12:44:36.809896+0000","last_peered":"2024-07-26T12:44:36.809896+0000","last_clean":"2024-07-26T12:44:36.809896+0000","last_became_active":"2024-07-26T12:43:12.173187+0000","last_became_peered":"2024-07-26T12:43:12.173187+0000","last_unstale":"2024-07-26T12:44:36.809896+0000","last_undegraded":"2024-07-26T12:44:36.809896+0000","last_fullsized":"2024-07-26T12:44:36.809896+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:53.929210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.808767+0000","last_change":"2024-07-26T12:43:12.169829+0000","last_active":"2024-07-26T12:44:36.808767+0000","last_peered":"2024-07-26T12:44:36.808767+0000","last_clean":"2024-07-26T12:44:36.808767+0000","last_became_active":"2024-07-26T12:43:12.169444+0000","last_became_peered":"2024-07-26T12:43:12.169444+0000","last_unstale":"2024-07-26T12:44:36.808767+0000","last_undegraded":"2024-07-26T12:44:36.808767+0000","last_fullsized":"2024-07-26T12:44:36.808767+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:29.111483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804682+0000","last_change":"2024-07-26T12:43:12.171857+0000","last_active":"2024-07-26T12:44:36.804682+0000","last_peered":"2024-07-26T12:44:36.804682+0000","last_clean":"2024-07-26T12:44:36.804682+0000","last_became_active":"2024-07-26T12:43:12.170752+0000","last_became_peered":"2024-07-26T12:43:12.170752+0000","last_unstale":"2024-07-26T12:44:36.804682+0000","last_undegraded":"2024-07-26T12:44:36.804682+0000","last_fullsized":"2024-07-26T12:44:36.804682+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:22:09.442916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805276+0000","last_change":"2024-07-26T12:43:12.178646+0000","last_active":"2024-07-26T12:44:36.805276+0000","last_peered":"2024-07-26T12:44:36.805276+0000","last_clean":"2024-07-26T12:44:36.805276+0000","last_became_active":"2024-07-26T12:43:12.178259+0000","last_became_peered":"2024-07-26T12:43:12.178259+0000","last_unstale":"2024-07-26T12:44:36.805276+0000","last_undegraded":"2024-07-26T12:44:36.805276+0000","last_fullsized":"2024-07-26T12:44:36.805276+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:30:59.417843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.809164+0000","last_change":"2024-07-26T12:43:12.164579+0000","last_active":"2024-07-26T12:44:36.809164+0000","last_peered":"2024-07-26T12:44:36.809164+0000","last_clean":"2024-07-26T12:44:36.809164+0000","last_became_active":"2024-07-26T12:43:12.164099+0000","last_became_peered":"2024-07-26T12:43:12.164099+0000","last_unstale":"2024-07-26T12:44:36.809164+0000","last_undegraded":"2024-07-26T12:44:36.809164+0000","last_fullsized":"2024-07-26T12:44:36.809164+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:51:45.293190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840446+0000","last_change":"2024-07-26T12:43:12.180838+0000","last_active":"2024-07-26T12:44:36.840446+0000","last_peered":"2024-07-26T12:44:36.840446+0000","last_clean":"2024-07-26T12:44:36.840446+0000","last_became_active":"2024-07-26T12:43:12.180543+0000","last_became_peered":"2024-07-26T12:43:12.180543+0000","last_unstale":"2024-07-26T12:44:36.840446+0000","last_undegraded":"2024-07-26T12:44:36.840446+0000","last_fullsized":"2024-07-26T12:44:36.840446+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:27.185082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850779+0000","last_change":"2024-07-26T12:43:12.167235+0000","last_active":"2024-07-26T12:44:36.850779+0000","last_peered":"2024-07-26T12:44:36.850779+0000","last_clean":"2024-07-26T12:44:36.850779+0000","last_became_active":"2024-07-26T12:43:12.166489+0000","last_became_peered":"2024-07-26T12:43:12.166489+0000","last_unstale":"2024-07-26T12:44:36.850779+0000","last_undegraded":"2024-07-26T12:44:36.850779+0000","last_fullsized":"2024-07-26T12:44:36.850779+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:13.868870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813194+0000","last_change":"2024-07-26T12:43:12.176717+0000","last_active":"2024-07-26T12:44:36.813194+0000","last_peered":"2024-07-26T12:44:36.813194+0000","last_clean":"2024-07-26T12:44:36.813194+0000","last_became_active":"2024-07-26T12:43:12.172699+0000","last_became_peered":"2024-07-26T12:43:12.172699+0000","last_unstale":"2024-07-26T12:44:36.813194+0000","last_undegraded":"2024-07-26T12:44:36.813194+0000","last_fullsized":"2024-07-26T12:44:36.813194+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:23:36.853999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813580+0000","last_change":"2024-07-26T12:43:12.177093+0000","last_active":"2024-07-26T12:44:36.813580+0000","last_peered":"2024-07-26T12:44:36.813580+0000","last_clean":"2024-07-26T12:44:36.813580+0000","last_became_active":"2024-07-26T12:43:12.176693+0000","last_became_peered":"2024-07-26T12:43:12.176693+0000","last_unstale":"2024-07-26T12:44:36.813580+0000","last_undegraded":"2024-07-26T12:44:36.813580+0000","last_fullsized":"2024-07-26T12:44:36.813580+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:56.875143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851429+0000","last_change":"2024-07-26T12:43:12.189531+0000","last_active":"2024-07-26T12:44:36.851429+0000","last_peered":"2024-07-26T12:44:36.851429+0000","last_clean":"2024-07-26T12:44:36.851429+0000","last_became_active":"2024-07-26T12:43:12.189162+0000","last_became_peered":"2024-07-26T12:43:12.189162+0000","last_unstale":"2024-07-26T12:44:36.851429+0000","last_undegraded":"2024-07-26T12:44:36.851429+0000","last_fullsized":"2024-07-26T12:44:36.851429+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:11:00.160614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71340,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":424,"num_read_kb":289,"num_write":251,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":214020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678056,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":536325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874286,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1744896,"data_stored":1186818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037809,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":539892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234040,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":536174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430269,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36964,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708188,"statfs":{"total":95995035648,"available":95957184512,"internally_reserved":0,"allocated":1667072,"data_stored":1132671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593795,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":531300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790026,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1128253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":535374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":531008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378713,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":597400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542239,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1118208,"data_stored":596466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":533147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":66822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T12:45:21.666 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T12:45:21.667 INFO:tasks.cephadm:Setting up client nodes... 2024-07-26T12:45:21.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-07-26T12:45:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:21 smithi107 ceph-mon[32659]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T12:45:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:21 smithi107 ceph-mon[32659]: from='client.24935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:45:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:21 smithi132 ceph-mon[37580]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T12:45:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:21 smithi132 ceph-mon[37580]: from='client.24935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:45:22.166 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:21 smithi112 ceph-mon[37501]: pgmap v371: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T12:45:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:21 smithi112 ceph-mon[37501]: from='client.24935 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:45:23.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:23 smithi107 ceph-mon[32659]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T12:45:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:23 smithi132 ceph-mon[37580]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T12:45:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:23 smithi112 ceph-mon[37501]: pgmap v372: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-07-26T12:45:24.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.499+0000 7fd325000640 1 -- 172.21.15.107:0/262839189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd32010af40 msgr2=0x7fd32010b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:24.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.499+0000 7fd325000640 1 --2- 172.21.15.107:0/262839189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd32010af40 0x7fd32010b350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009860 tx=0x7fd30c02a340 comp rx=0 tx=0).stop 2024-07-26T12:45:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.499+0000 7fd325000640 1 -- 172.21.15.107:0/262839189 shutdown_connections 2024-07-26T12:45:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.499+0000 7fd325000640 1 --2- 172.21.15.107:0/262839189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd32010b890 0x7fd3201120c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.499+0000 7fd325000640 1 --2- 172.21.15.107:0/262839189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd32010af40 0x7fd32010b350 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.499+0000 7fd325000640 1 --2- 172.21.15.107:0/262839189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd320105600 0x7fd320105a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.499+0000 7fd325000640 1 -- 172.21.15.107:0/262839189 >> 172.21.15.107:0/262839189 conn(0x7fd320101070 msgr2=0x7fd3201034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.500+0000 7fd325000640 1 -- 172.21.15.107:0/262839189 shutdown_connections 2024-07-26T12:45:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.500+0000 7fd325000640 1 -- 172.21.15.107:0/262839189 wait complete. 2024-07-26T12:45:24.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.501+0000 7fd325000640 1 Processor -- start 2024-07-26T12:45:24.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.501+0000 7fd325000640 1 -- start start 2024-07-26T12:45:24.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.501+0000 7fd325000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd320105600 0x7fd32019bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:24.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.501+0000 7fd325000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd32010af40 0x7fd32019c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:24.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.501+0000 7fd325000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd32010b890 0x7fd3201a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:24.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.501+0000 7fd325000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd320115520 con 0x7fd320105600 2024-07-26T12:45:24.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.501+0000 7fd325000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd3201153a0 con 0x7fd32010b890 2024-07-26T12:45:24.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.501+0000 7fd325000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd3201156a0 con 0x7fd32010af40 2024-07-26T12:45:24.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd320105600 0x7fd32019bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:24.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd320105600 0x7fd32019bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41014/0 (socket says 172.21.15.107:41014) 2024-07-26T12:45:24.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31f400640 1 -- 172.21.15.107:0/2720769537 learned_addr learned my addr 172.21.15.107:0/2720769537 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:24.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31ea00640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd32010af40 0x7fd32019c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:24.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31fe00640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd32010b890 0x7fd3201a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:24.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31f400640 1 -- 172.21.15.107:0/2720769537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd32010af40 msgr2=0x7fd32019c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:24.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31f400640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd32010af40 0x7fd32019c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:24.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31f400640 1 -- 172.21.15.107:0/2720769537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd32010b890 msgr2=0x7fd3201a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31f400640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd32010b890 0x7fd3201a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31f400640 1 -- 172.21.15.107:0/2720769537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3201a1e80 con 0x7fd320105600 2024-07-26T12:45:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.502+0000 7fd31ea00640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd32010af40 0x7fd32019c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:45:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.503+0000 7fd31f400640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd320105600 0x7fd32019bf10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd31000e9c0 tx=0x7fd31000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.503+0000 7fd317e00640 1 -- 172.21.15.107:0/2720769537 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31000ccc0 con 0x7fd320105600 2024-07-26T12:45:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.503+0000 7fd325000640 1 -- 172.21.15.107:0/2720769537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3201a2170 con 0x7fd320105600 2024-07-26T12:45:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.503+0000 7fd317e00640 1 -- 172.21.15.107:0/2720769537 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3100054d0 con 0x7fd320105600 2024-07-26T12:45:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.503+0000 7fd317e00640 1 -- 172.21.15.107:0/2720769537 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310010430 con 0x7fd320105600 2024-07-26T12:45:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.503+0000 7fd325000640 1 -- 172.21.15.107:0/2720769537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3201a88d0 con 0x7fd320105600 2024-07-26T12:45:24.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.505+0000 7fd325000640 1 -- 172.21.15.107:0/2720769537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd320105a10 con 0x7fd320105600 2024-07-26T12:45:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.507+0000 7fd317e00640 1 -- 172.21.15.107:0/2720769537 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fd310004f80 con 0x7fd320105600 2024-07-26T12:45:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.508+0000 7fd317e00640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3040b3ca0 0x7fd3040b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:24.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.509+0000 7fd317e00640 1 -- 172.21.15.107:0/2720769537 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd3100e46f0 con 0x7fd320105600 2024-07-26T12:45:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.509+0000 7fd31ea00640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3040b3ca0 0x7fd3040b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:24.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.509+0000 7fd31ea00640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3040b3ca0 0x7fd3040b61b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd30c005940 tx=0x7fd30c0058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:24.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.513+0000 7fd317e00640 1 -- 172.21.15.107:0/2720769537 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd3100ab6e0 con 0x7fd320105600 2024-07-26T12:45:24.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.779+0000 7fd325000640 1 -- 172.21.15.107:0/2720769537 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fd32002e0c0 con 0x7fd320105600 2024-07-26T12:45:24.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.789+0000 7fd317e00640 1 -- 172.21.15.107:0/2720769537 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v33) ==== 170+0+59 (secure 0 0 0) 0x7fd3100aafe0 con 0x7fd320105600 2024-07-26T12:45:24.790 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-07-26T12:45:24.791 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBkmqNmg2aILhAAeASNnaaNnuYPPtBJLcXTfw== 2024-07-26T12:45:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 -- 172.21.15.107:0/2720769537 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3040b3ca0 msgr2=0x7fd3040b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3040b3ca0 0x7fd3040b61b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd30c005940 tx=0x7fd30c0058d0 comp rx=0 tx=0).stop 2024-07-26T12:45:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 -- 172.21.15.107:0/2720769537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd320105600 msgr2=0x7fd32019bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:24.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd320105600 0x7fd32019bf10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd31000e9c0 tx=0x7fd31000ee80 comp rx=0 tx=0).stop 2024-07-26T12:45:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 -- 172.21.15.107:0/2720769537 shutdown_connections 2024-07-26T12:45:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3040b3ca0 0x7fd3040b61b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd32010b890 0x7fd3201a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd32010af40 0x7fd32019c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 --2- 172.21.15.107:0/2720769537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd320105600 0x7fd32019bf10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 -- 172.21.15.107:0/2720769537 >> 172.21.15.107:0/2720769537 conn(0x7fd320101070 msgr2=0x7fd320103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.792+0000 7fd325000640 1 -- 172.21.15.107:0/2720769537 shutdown_connections 2024-07-26T12:45:24.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:24.793+0000 7fd325000640 1 -- 172.21.15.107:0/2720769537 wait complete. 2024-07-26T12:45:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:24 smithi107 ceph-mon[32659]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T12:45:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:24 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2720769537' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T12:45:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:24 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2720769537' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T12:45:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:24 smithi112 ceph-mon[37501]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T12:45:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:24 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2720769537' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T12:45:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:24 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2720769537' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T12:45:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:24 smithi132 ceph-mon[37580]: pgmap v373: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T12:45:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:24 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2720769537' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T12:45:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:24 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2720769537' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T12:45:25.553 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:45:25.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-07-26T12:45:25.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-07-26T12:45:25.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph config log 1 --format=json 2024-07-26T12:45:26.199 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:27 smithi107 ceph-mon[32659]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8 op/s 2024-07-26T12:45:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:27 smithi132 ceph-mon[37580]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8 op/s 2024-07-26T12:45:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:27 smithi112 ceph-mon[37501]: pgmap v374: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 8 op/s 2024-07-26T12:45:28.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.489+0000 7fb23b400640 1 -- 172.21.15.107:0/1245185523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb234109820 msgr2=0x7fb234110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:28.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.489+0000 7fb23b400640 1 --2- 172.21.15.107:0/1245185523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb234109820 0x7fb234110050 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009ca0 tx=0x7fb22c02a8c0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.489+0000 7fb23b400640 1 -- 172.21.15.107:0/1245185523 shutdown_connections 2024-07-26T12:45:28.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.489+0000 7fb23b400640 1 --2- 172.21.15.107:0/1245185523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb234109820 0x7fb234110050 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.489+0000 7fb23b400640 1 --2- 172.21.15.107:0/1245185523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234108ed0 0x7fb2341092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.489+0000 7fb23b400640 1 --2- 172.21.15.107:0/1245185523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb234103590 0x7fb2341039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.489+0000 7fb23b400640 1 -- 172.21.15.107:0/1245185523 >> 172.21.15.107:0/1245185523 conn(0x7fb23407b280 msgr2=0x7fb23407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:28.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.489+0000 7fb23b400640 1 -- 172.21.15.107:0/1245185523 shutdown_connections 2024-07-26T12:45:28.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.489+0000 7fb23b400640 1 -- 172.21.15.107:0/1245185523 wait complete. 2024-07-26T12:45:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.490+0000 7fb23b400640 1 Processor -- start 2024-07-26T12:45:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.490+0000 7fb23b400640 1 -- start start 2024-07-26T12:45:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.491+0000 7fb23b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234103590 0x7fb23419c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.491+0000 7fb23b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb234108ed0 0x7fb23419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.491+0000 7fb23b400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb234109820 0x7fb2341a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.491+0000 7fb23b400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb2341134f0 con 0x7fb234108ed0 2024-07-26T12:45:28.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.491+0000 7fb23b400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb234113370 con 0x7fb234109820 2024-07-26T12:45:28.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.491+0000 7fb23b400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb234113670 con 0x7fb234103590 2024-07-26T12:45:28.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.491+0000 7fb233e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb234108ed0 0x7fb23419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:28.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.491+0000 7fb238a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234103590 0x7fb23419c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:28.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.492+0000 7fb233e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb234108ed0 0x7fb23419c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41030/0 (socket says 172.21.15.107:41030) 2024-07-26T12:45:28.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.492+0000 7fb233e00640 1 -- 172.21.15.107:0/3641264198 learned_addr learned my addr 172.21.15.107:0/3641264198 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:28.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.492+0000 7fb239400640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb234109820 0x7fb2341a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:28.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.492+0000 7fb238a00640 1 -- 172.21.15.107:0/3641264198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb234109820 msgr2=0x7fb2341a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:28.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.492+0000 7fb238a00640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb234109820 0x7fb2341a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.492+0000 7fb238a00640 1 -- 172.21.15.107:0/3641264198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb234108ed0 msgr2=0x7fb23419c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:28.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.492+0000 7fb238a00640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb234108ed0 0x7fb23419c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.492+0000 7fb238a00640 1 -- 172.21.15.107:0/3641264198 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2341a1ff0 con 0x7fb234103590 2024-07-26T12:45:28.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.492+0000 7fb238a00640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234103590 0x7fb23419c080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb22800e9f0 tx=0x7fb22800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:28.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.493+0000 7fb231600640 1 -- 172.21.15.107:0/3641264198 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22800ccc0 con 0x7fb234103590 2024-07-26T12:45:28.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.493+0000 7fb23b400640 1 -- 172.21.15.107:0/3641264198 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2341a22e0 con 0x7fb234103590 2024-07-26T12:45:28.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.493+0000 7fb231600640 1 -- 172.21.15.107:0/3641264198 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2280108e0 con 0x7fb234103590 2024-07-26T12:45:28.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.493+0000 7fb231600640 1 -- 172.21.15.107:0/3641264198 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb228018880 con 0x7fb234103590 2024-07-26T12:45:28.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.493+0000 7fb23b400640 1 -- 172.21.15.107:0/3641264198 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2341a8a40 con 0x7fb234103590 2024-07-26T12:45:28.495 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.494+0000 7fb23b400640 1 -- 172.21.15.107:0/3641264198 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2341039a0 con 0x7fb234103590 2024-07-26T12:45:28.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.497+0000 7fb231600640 1 -- 172.21.15.107:0/3641264198 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fb228010430 con 0x7fb234103590 2024-07-26T12:45:28.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.498+0000 7fb231600640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb21c0b3cb0 0x7fb21c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:28.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.499+0000 7fb231600640 1 -- 172.21.15.107:0/3641264198 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb2280e4a90 con 0x7fb234103590 2024-07-26T12:45:28.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.499+0000 7fb233e00640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb21c0b3cb0 0x7fb21c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:28.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.499+0000 7fb233e00640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb21c0b3cb0 0x7fb21c0b61c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb234108b60 tx=0x7fb220005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:28.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.500+0000 7fb231600640 1 -- 172.21.15.107:0/3641264198 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb2280ab430 con 0x7fb234103590 2024-07-26T12:45:28.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.699+0000 7fb23b400640 1 -- 172.21.15.107:0/3641264198 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fb23419d1b0 con 0x7fb234103590 2024-07-26T12:45:28.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.700+0000 7fb231600640 1 -- 172.21.15.107:0/3641264198 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v53) ==== 86+0+133 (secure 0 0 0) 0x7fb23419d1b0 con 0x7fb234103590 2024-07-26T12:45:28.702 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:45:28.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.704+0000 7fb23b400640 1 -- 172.21.15.107:0/3641264198 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb21c0b3cb0 msgr2=0x7fb21c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:28.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.704+0000 7fb23b400640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb21c0b3cb0 0x7fb21c0b61c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb234108b60 tx=0x7fb220005f90 comp rx=0 tx=0).stop 2024-07-26T12:45:28.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.704+0000 7fb23b400640 1 -- 172.21.15.107:0/3641264198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234103590 msgr2=0x7fb23419c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:28.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.704+0000 7fb23b400640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234103590 0x7fb23419c080 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb22800e9f0 tx=0x7fb22800eeb0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.704+0000 7fb23b400640 1 -- 172.21.15.107:0/3641264198 shutdown_connections 2024-07-26T12:45:28.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.704+0000 7fb23b400640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb21c0b3cb0 0x7fb21c0b61c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.704+0000 7fb23b400640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb234109820 0x7fb2341a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.704+0000 7fb23b400640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb234108ed0 0x7fb23419c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.704+0000 7fb23b400640 1 --2- 172.21.15.107:0/3641264198 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb234103590 0x7fb23419c080 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:28.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.704+0000 7fb23b400640 1 -- 172.21.15.107:0/3641264198 >> 172.21.15.107:0/3641264198 conn(0x7fb23407b280 msgr2=0x7fb234106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:28.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.705+0000 7fb23b400640 1 -- 172.21.15.107:0/3641264198 shutdown_connections 2024-07-26T12:45:28.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:28.705+0000 7fb23b400640 1 -- 172.21.15.107:0/3641264198 wait complete. 2024-07-26T12:45:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:28 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3641264198' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T12:45:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:28 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3641264198' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T12:45:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:28 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3641264198' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T12:45:29.546 INFO:teuthology.orchestra.run.smithi107.stdout:[{"version":53,"timestamp":"2024-07-26T12:42:57.149466+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-07-26T12:45:29.546 INFO:tasks.ceph_manager:config epoch is 53 2024-07-26T12:45:29.546 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-07-26T12:45:29.547 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-07-26T12:45:29.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph mgr dump --format=json 2024-07-26T12:45:29.982 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:29 smithi107 ceph-mon[32659]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s 2024-07-26T12:45:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:29 smithi132 ceph-mon[37580]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s 2024-07-26T12:45:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:29 smithi112 ceph-mon[37501]: pgmap v375: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s 2024-07-26T12:45:31.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:31 smithi107 ceph-mon[32659]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:45:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:31 smithi132 ceph-mon[37580]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:45:32.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.189+0000 7f1476c00640 1 -- 172.21.15.107:0/475366150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14701022a0 msgr2=0x7f14701026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.189+0000 7f1476c00640 1 --2- 172.21.15.107:0/475366150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14701022a0 0x7f14701026b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1460009900 tx=0x7f146002a410 comp rx=0 tx=0).stop 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.189+0000 7f1476c00640 1 -- 172.21.15.107:0/475366150 shutdown_connections 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.189+0000 7f1476c00640 1 --2- 172.21.15.107:0/475366150 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1470108530 0x7f147010ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.189+0000 7f1476c00640 1 --2- 172.21.15.107:0/475366150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1470107be0 0x7f1470107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.189+0000 7f1476c00640 1 --2- 172.21.15.107:0/475366150 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f14701022a0 0x7f14701026b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.189+0000 7f1476c00640 1 -- 172.21.15.107:0/475366150 >> 172.21.15.107:0/475366150 conn(0x7f14700fdcf0 msgr2=0x7f1470100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.190+0000 7f1476c00640 1 -- 172.21.15.107:0/475366150 shutdown_connections 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.190+0000 7f1476c00640 1 -- 172.21.15.107:0/475366150 wait complete. 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.191+0000 7f1476c00640 1 Processor -- start 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.191+0000 7f1476c00640 1 -- start start 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.191+0000 7f1476c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14701022a0 0x7f147019bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.191+0000 7f1476c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1470107be0 0x7f147019c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.191+0000 7f1476c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1470108530 0x7f14701a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.191+0000 7f1476c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1470112200 con 0x7f14701022a0 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.191+0000 7f1476c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1470112080 con 0x7f1470107be0 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.191+0000 7f1476c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1470112380 con 0x7f1470108530 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.192+0000 7f1475800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14701022a0 0x7f147019bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.192+0000 7f1475800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14701022a0 0x7f147019bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41042/0 (socket says 172.21.15.107:41042) 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.192+0000 7f1475800640 1 -- 172.21.15.107:0/1282654762 learned_addr learned my addr 172.21.15.107:0/1282654762 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.192+0000 7f1474e00640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1470107be0 0x7f147019c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.192+0000 7f1476200640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1470108530 0x7f14701a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.192+0000 7f1475800640 1 -- 172.21.15.107:0/1282654762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1470108530 msgr2=0x7f14701a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:32.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.192+0000 7f1475800640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1470108530 0x7f14701a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.192+0000 7f1475800640 1 -- 172.21.15.107:0/1282654762 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1470107be0 msgr2=0x7f147019c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.192+0000 7f1475800640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1470107be0 0x7f147019c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.192+0000 7f1475800640 1 -- 172.21.15.107:0/1282654762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14701a1f30 con 0x7f14701022a0 2024-07-26T12:45:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.193+0000 7f1475800640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14701022a0 0x7f147019bfc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1460005940 tx=0x7f14600040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.193+0000 7f146e000640 1 -- 172.21.15.107:0/1282654762 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146003f030 con 0x7f14701022a0 2024-07-26T12:45:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.193+0000 7f146e000640 1 -- 172.21.15.107:0/1282654762 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14600044c0 con 0x7f14701022a0 2024-07-26T12:45:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.193+0000 7f146e000640 1 -- 172.21.15.107:0/1282654762 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460035070 con 0x7f14701022a0 2024-07-26T12:45:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.193+0000 7f1476c00640 1 -- 172.21.15.107:0/1282654762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14701a21c0 con 0x7f14701022a0 2024-07-26T12:45:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.193+0000 7f1476c00640 1 -- 172.21.15.107:0/1282654762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14701a8980 con 0x7f14701022a0 2024-07-26T12:45:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.194+0000 7f1476c00640 1 -- 172.21.15.107:0/1282654762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14701026b0 con 0x7f14701022a0 2024-07-26T12:45:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.197+0000 7f146e000640 1 -- 172.21.15.107:0/1282654762 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f1460042030 con 0x7f14701022a0 2024-07-26T12:45:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.198+0000 7f146e000640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14580b3d30 0x7f14580b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.198+0000 7f146e000640 1 -- 172.21.15.107:0/1282654762 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1460104820 con 0x7f14701022a0 2024-07-26T12:45:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.199+0000 7f1474e00640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14580b3d30 0x7f14580b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:32.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.200+0000 7f1474e00640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14580b3d30 0x7f14580b6240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f14701043f0 tx=0x7f1464005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:31 smithi112 ceph-mon[37501]: pgmap v376: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:45:32.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.201+0000 7f146e000640 1 -- 172.21.15.107:0/1282654762 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f14600cb1c0 con 0x7f14701022a0 2024-07-26T12:45:32.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.451+0000 7f1476c00640 1 -- 172.21.15.107:0/1282654762 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f147002e0c0 con 0x7f14701022a0 2024-07-26T12:45:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.458+0000 7f146e000640 1 -- 172.21.15.107:0/1282654762 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v21) ==== 74+0+291043 (secure 0 0 0) 0x7f14600cf710 con 0x7f14701022a0 2024-07-26T12:45:32.465 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:45:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 -- 172.21.15.107:0/1282654762 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14580b3d30 msgr2=0x7f14580b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:32.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14580b3d30 0x7f14580b6240 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f14701043f0 tx=0x7f1464005f90 comp rx=0 tx=0).stop 2024-07-26T12:45:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 -- 172.21.15.107:0/1282654762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14701022a0 msgr2=0x7f147019bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14701022a0 0x7f147019bfc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1460005940 tx=0x7f14600040c0 comp rx=0 tx=0).stop 2024-07-26T12:45:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 -- 172.21.15.107:0/1282654762 shutdown_connections 2024-07-26T12:45:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f14580b3d30 0x7f14580b6240 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1470108530 0x7f14701a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1470107be0 0x7f147019c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 --2- 172.21.15.107:0/1282654762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14701022a0 0x7f147019bfc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 -- 172.21.15.107:0/1282654762 >> 172.21.15.107:0/1282654762 conn(0x7f14700fdcf0 msgr2=0x7f14701072f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 -- 172.21.15.107:0/1282654762 shutdown_connections 2024-07-26T12:45:32.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:32.466+0000 7f1476c00640 1 -- 172.21.15.107:0/1282654762 wait complete. 2024-07-26T12:45:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:32 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1282654762' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T12:45:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:32 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1282654762' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T12:45:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:32 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1282654762' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T12:45:33.315 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"flags":0,"active_gid":14211,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":3577384363},{"type":"v1","addr":"172.21.15.107:6801","nonce":3577384363}]},"active_addr":"172.21.15.107:6801/3577384363","active_change":"2024-07-26T12:35:34.485808+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24145,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24152,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.107:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":4176847987}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":536086364}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":465048417}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":1184483456}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3868452635}]}]} 2024-07-26T12:45:33.322 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-07-26T12:45:33.322 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-07-26T12:45:33.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:45:33.830 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:33 smithi132 ceph-mon[37580]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:34.155 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:33 smithi107 ceph-mon[32659]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:33 smithi112 ceph-mon[37501]: pgmap v377: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:35.961 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:35 smithi107 ceph-mon[32659]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:45:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:35 smithi132 ceph-mon[37580]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:45:36.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.165+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1925098509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ed4109670 msgr2=0x7f3ed410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:36.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.165+0000 7f3ed8a00640 1 --2- 172.21.15.107:0/1925098509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ed4109670 0x7f3ed410fea0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009ca0 tx=0x7f3ecc02a8c0 comp rx=0 tx=0).stop 2024-07-26T12:45:36.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.165+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1925098509 shutdown_connections 2024-07-26T12:45:36.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.165+0000 7f3ed8a00640 1 --2- 172.21.15.107:0/1925098509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ed4109670 0x7f3ed410fea0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:36.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.165+0000 7f3ed8a00640 1 --2- 172.21.15.107:0/1925098509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed4108d20 0x7f3ed4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:36.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.165+0000 7f3ed8a00640 1 --2- 172.21.15.107:0/1925098509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ed41033e0 0x7f3ed41037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:36.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.165+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1925098509 >> 172.21.15.107:0/1925098509 conn(0x7f3ed40fee70 msgr2=0x7f3ed41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:36.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.165+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1925098509 shutdown_connections 2024-07-26T12:45:36.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.165+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1925098509 wait complete. 2024-07-26T12:45:36.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.166+0000 7f3ed8a00640 1 Processor -- start 2024-07-26T12:45:36.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.167+0000 7f3ed8a00640 1 -- start start 2024-07-26T12:45:36.168 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.167+0000 7f3ed8a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ed41033e0 0x7f3ed4075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.167+0000 7f3ed8a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ed4108d20 0x7f3ed4075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.167+0000 7f3ed8a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed4109670 0x7f3ed4075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.167+0000 7f3ed8a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed4113320 con 0x7f3ed4108d20 2024-07-26T12:45:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.167+0000 7f3ed8a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed41131a0 con 0x7f3ed41033e0 2024-07-26T12:45:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.167+0000 7f3ed8a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed41134a0 con 0x7f3ed4109670 2024-07-26T12:45:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.167+0000 7f3ecbe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ed4108d20 0x7f3ed4075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.168+0000 7f3ed3e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed4109670 0x7f3ed4075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.168+0000 7f3ecbe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ed4108d20 0x7f3ed4075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47112/0 (socket says 172.21.15.107:47112) 2024-07-26T12:45:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.168+0000 7f3ed3400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ed41033e0 0x7f3ed4075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.168+0000 7f3ecbe00640 1 -- 172.21.15.107:0/1848738226 learned_addr learned my addr 172.21.15.107:0/1848738226 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.168+0000 7f3ed3400640 1 -- 172.21.15.107:0/1848738226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed4109670 msgr2=0x7f3ed4075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.168+0000 7f3ed3400640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed4109670 0x7f3ed4075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.168+0000 7f3ed3400640 1 -- 172.21.15.107:0/1848738226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ed4108d20 msgr2=0x7f3ed4075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.168+0000 7f3ed3400640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ed4108d20 0x7f3ed4075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.168+0000 7f3ed3400640 1 -- 172.21.15.107:0/1848738226 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed41a8010 con 0x7f3ed41033e0 2024-07-26T12:45:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.168+0000 7f3ecbe00640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ed4108d20 0x7f3ed4075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:45:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.169+0000 7f3ed3400640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ed41033e0 0x7f3ed4075140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400d6d0 tx=0x7f3ec400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.169+0000 7f3ed0c00640 1 -- 172.21.15.107:0/1848738226 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4017070 con 0x7f3ed41033e0 2024-07-26T12:45:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.169+0000 7f3ed0c00640 1 -- 172.21.15.107:0/1848738226 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ec4009820 con 0x7f3ed41033e0 2024-07-26T12:45:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.169+0000 7f3ed0c00640 1 -- 172.21.15.107:0/1848738226 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4004c40 con 0x7f3ed41033e0 2024-07-26T12:45:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.169+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1848738226 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed41a8300 con 0x7f3ed41033e0 2024-07-26T12:45:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.169+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1848738226 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed41a8840 con 0x7f3ed41033e0 2024-07-26T12:45:36.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.170+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1848738226 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ed41037f0 con 0x7f3ed41033e0 2024-07-26T12:45:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.173+0000 7f3ed0c00640 1 -- 172.21.15.107:0/1848738226 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f3ec4005220 con 0x7f3ed41033e0 2024-07-26T12:45:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.175+0000 7f3ed0c00640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3eb80b3cb0 0x7f3eb80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.175+0000 7f3ed0c00640 1 -- 172.21.15.107:0/1848738226 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3ec40e4180 con 0x7f3ed41033e0 2024-07-26T12:45:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.175+0000 7f3ecbe00640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3eb80b3cb0 0x7f3eb80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.176+0000 7f3ecbe00640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3eb80b3cb0 0x7f3eb80b61c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc0095f0 tx=0x7f3ebc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:36.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.177+0000 7f3ed0c00640 1 -- 172.21.15.107:0/1848738226 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3ec40aab20 con 0x7f3ed41033e0 2024-07-26T12:45:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:35 smithi112 ceph-mon[37501]: pgmap v378: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:45:36.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.381+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1848738226 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ed402e0c0 con 0x7f3ed41033e0 2024-07-26T12:45:36.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.383+0000 7f3ed0c00640 1 -- 172.21.15.107:0/1848738226 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25203 (secure 0 0 0) 0x7f3ec40af070 con 0x7f3ed41033e0 2024-07-26T12:45:36.384 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:45:36.385 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:44:36.797948+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:45:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1848738226 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3eb80b3cb0 msgr2=0x7f3eb80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3eb80b3cb0 0x7f3eb80b61c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc0095f0 tx=0x7f3ebc005f90 comp rx=0 tx=0).stop 2024-07-26T12:45:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1848738226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ed41033e0 msgr2=0x7f3ed4075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ed41033e0 0x7f3ed4075140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400d6d0 tx=0x7f3ec400db90 comp rx=0 tx=0).stop 2024-07-26T12:45:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1848738226 shutdown_connections 2024-07-26T12:45:36.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3eb80b3cb0 0x7f3eb80b61c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed4109670 0x7f3ed4075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ed4108d20 0x7f3ed4075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 --2- 172.21.15.107:0/1848738226 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3ed41033e0 0x7f3ed4075140 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1848738226 >> 172.21.15.107:0/1848738226 conn(0x7f3ed40fee70 msgr2=0x7f3ed4107340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1848738226 shutdown_connections 2024-07-26T12:45:36.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:36.386+0000 7f3ed8a00640 1 -- 172.21.15.107:0/1848738226 wait complete. 2024-07-26T12:45:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:36 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1848738226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:45:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:36 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1848738226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:45:37.229 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-07-26T12:45:37.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:45:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:36 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1848738226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:45:37.692 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:38.158 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:37 smithi107 ceph-mon[32659]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:37 smithi112 ceph-mon[37501]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:37 smithi132 ceph-mon[37580]: pgmap v379: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:39.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.871+0000 7fb35ec00640 1 -- 172.21.15.107:0/2160703236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3581116e0 msgr2=0x7fb358111af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:39.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.871+0000 7fb35ec00640 1 --2- 172.21.15.107:0/2160703236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3581116e0 0x7fb358111af0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb340009900 tx=0x7fb34002a410 comp rx=0 tx=0).stop 2024-07-26T12:45:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.871+0000 7fb35ec00640 1 -- 172.21.15.107:0/2160703236 shutdown_connections 2024-07-26T12:45:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.871+0000 7fb35ec00640 1 --2- 172.21.15.107:0/2160703236 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb358112030 0x7fb358118860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.871+0000 7fb35ec00640 1 --2- 172.21.15.107:0/2160703236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3581116e0 0x7fb358111af0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.871+0000 7fb35ec00640 1 --2- 172.21.15.107:0/2160703236 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb35810bda0 0x7fb35810c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.871+0000 7fb35ec00640 1 -- 172.21.15.107:0/2160703236 >> 172.21.15.107:0/2160703236 conn(0x7fb3580714b0 msgr2=0x7fb358109df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.871+0000 7fb35ec00640 1 -- 172.21.15.107:0/2160703236 shutdown_connections 2024-07-26T12:45:39.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.871+0000 7fb35ec00640 1 -- 172.21.15.107:0/2160703236 wait complete. 2024-07-26T12:45:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.872+0000 7fb35ec00640 1 Processor -- start 2024-07-26T12:45:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb35ec00640 1 -- start start 2024-07-26T12:45:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb35ec00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35810bda0 0x7fb35806c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb35ec00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3581116e0 0x7fb35806ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb35ec00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb358112030 0x7fb35806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb35ec00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb35811bd40 con 0x7fb35810bda0 2024-07-26T12:45:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb35ec00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb35811bbc0 con 0x7fb3581116e0 2024-07-26T12:45:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb35ec00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb35811bec0 con 0x7fb358112030 2024-07-26T12:45:39.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb357e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35810bda0 0x7fb35806c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb34f400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3581116e0 0x7fb35806ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb34f400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3581116e0 0x7fb35806ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:49220/0 (socket says 172.21.15.107:49220) 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb357e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35810bda0 0x7fb35806c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47128/0 (socket says 172.21.15.107:47128) 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb34f400640 1 -- 172.21.15.107:0/1157096309 learned_addr learned my addr 172.21.15.107:0/1157096309 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.873+0000 7fb35cc00640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb358112030 0x7fb35806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.874+0000 7fb34f400640 1 -- 172.21.15.107:0/1157096309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb358112030 msgr2=0x7fb35806d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.874+0000 7fb34f400640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb358112030 0x7fb35806d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.874+0000 7fb34f400640 1 -- 172.21.15.107:0/1157096309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35810bda0 msgr2=0x7fb35806c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.874+0000 7fb34f400640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35810bda0 0x7fb35806c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.874+0000 7fb34f400640 1 -- 172.21.15.107:0/1157096309 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3581b0a10 con 0x7fb3581116e0 2024-07-26T12:45:39.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.874+0000 7fb34f400640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3581116e0 0x7fb35806ce10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb340005ab0 tx=0x7fb340003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.875+0000 7fb355600640 1 -- 172.21.15.107:0/1157096309 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34003f070 con 0x7fb3581116e0 2024-07-26T12:45:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.875+0000 7fb35ec00640 1 -- 172.21.15.107:0/1157096309 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3581b0ca0 con 0x7fb3581116e0 2024-07-26T12:45:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.875+0000 7fb355600640 1 -- 172.21.15.107:0/1157096309 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb340031b50 con 0x7fb3581116e0 2024-07-26T12:45:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.875+0000 7fb355600640 1 -- 172.21.15.107:0/1157096309 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb340035030 con 0x7fb3581116e0 2024-07-26T12:45:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.875+0000 7fb35ec00640 1 -- 172.21.15.107:0/1157096309 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3581b1180 con 0x7fb3581116e0 2024-07-26T12:45:39.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.876+0000 7fb35ec00640 1 -- 172.21.15.107:0/1157096309 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb35810c1b0 con 0x7fb3581116e0 2024-07-26T12:45:39.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.878+0000 7fb355600640 1 -- 172.21.15.107:0/1157096309 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fb340042070 con 0x7fb3581116e0 2024-07-26T12:45:39.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.880+0000 7fb355600640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb33c0b3cb0 0x7fb33c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:39.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.880+0000 7fb355600640 1 -- 172.21.15.107:0/1157096309 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb340104aa0 con 0x7fb3581116e0 2024-07-26T12:45:39.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.880+0000 7fb357e00640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb33c0b3cb0 0x7fb33c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:39.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.881+0000 7fb357e00640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb33c0b3cb0 0x7fb33c0b61c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb35806e320 tx=0x7fb348005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:39.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:39.881+0000 7fb355600640 1 -- 172.21.15.107:0/1157096309 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb3400cba90 con 0x7fb3581116e0 2024-07-26T12:45:40.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.081+0000 7fb35ec00640 1 -- 172.21.15.107:0/1157096309 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb35802e0c0 con 0x7fb3581116e0 2024-07-26T12:45:40.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:39 smithi107 ceph-mon[32659]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:40.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.083+0000 7fb355600640 1 -- 172.21.15.107:0/1157096309 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25203 (secure 0 0 0) 0x7fb3400cb390 con 0x7fb3581116e0 2024-07-26T12:45:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:45:40.085 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:44:36.797948+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:45:40.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 -- 172.21.15.107:0/1157096309 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb33c0b3cb0 msgr2=0x7fb33c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb33c0b3cb0 0x7fb33c0b61c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb35806e320 tx=0x7fb348005f90 comp rx=0 tx=0).stop 2024-07-26T12:45:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 -- 172.21.15.107:0/1157096309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3581116e0 msgr2=0x7fb35806ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3581116e0 0x7fb35806ce10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb340005ab0 tx=0x7fb340003b90 comp rx=0 tx=0).stop 2024-07-26T12:45:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 -- 172.21.15.107:0/1157096309 shutdown_connections 2024-07-26T12:45:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb33c0b3cb0 0x7fb33c0b61c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb358112030 0x7fb35806d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb3581116e0 0x7fb35806ce10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 --2- 172.21.15.107:0/1157096309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb35810bda0 0x7fb35806c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:40.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 -- 172.21.15.107:0/1157096309 >> 172.21.15.107:0/1157096309 conn(0x7fb3580714b0 msgr2=0x7fb358071d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:40.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 -- 172.21.15.107:0/1157096309 shutdown_connections 2024-07-26T12:45:40.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:40.086+0000 7fb35ec00640 1 -- 172.21.15.107:0/1157096309 wait complete. 2024-07-26T12:45:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:39 smithi112 ceph-mon[37501]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:39 smithi132 ceph-mon[37580]: pgmap v380: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:41.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-07-26T12:45:41.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-07-26T12:45:41.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-07-26T12:45:41.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-07-26T12:45:41.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-07-26T12:45:41.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-07-26T12:45:41.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-07-26T12:45:41.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-07-26T12:45:41.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-07-26T12:45:41.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-07-26T12:45:41.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-07-26T12:45:41.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-07-26T12:45:41.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:40 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1157096309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:45:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:40 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1157096309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:45:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:40 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1157096309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:45:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:41 smithi132 ceph-mon[37580]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:45:42.125 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:41 smithi107 ceph-mon[32659]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:45:42.131 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:42.158 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:41 smithi112 ceph-mon[37501]: pgmap v381: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:45:42.340 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:42.989 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:42.989 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:42.989 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:42.996 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:42.996 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:43.004 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:43.106 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:42 smithi107 ceph-mon[32659]: pgmap v382: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:42 smithi112 ceph-mon[37501]: pgmap v382: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:42 smithi132 ceph-mon[37580]: pgmap v382: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:45 smithi107 ceph-mon[32659]: pgmap v383: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:45:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:45 smithi132 ceph-mon[37580]: pgmap v383: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:45:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:45 smithi112 ceph-mon[37501]: pgmap v383: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:45:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:47 smithi107 ceph-mon[32659]: pgmap v384: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:47 smithi132 ceph-mon[37580]: pgmap v384: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:47 smithi112 ceph-mon[37501]: pgmap v384: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:49 smithi107 ceph-mon[32659]: pgmap v385: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:49 smithi132 ceph-mon[37580]: pgmap v385: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:49 smithi112 ceph-mon[37501]: pgmap v385: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:51.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.498+0000 7fdfcbe00640 1 -- 172.21.15.107:0/1500653713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfcc075180 msgr2=0x7fdfcc075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:51.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.498+0000 7fdfcbe00640 1 --2- 172.21.15.107:0/1500653713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfcc075180 0x7fdfcc075590 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009900 tx=0x7fdfb402a410 comp rx=0 tx=0).stop 2024-07-26T12:45:51.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.498+0000 7fdfcbe00640 1 -- 172.21.15.107:0/1500653713 shutdown_connections 2024-07-26T12:45:51.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.498+0000 7fdfcbe00640 1 --2- 172.21.15.107:0/1500653713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfcc10f980 0x7fdfcc111da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:51.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.498+0000 7fdfcbe00640 1 --2- 172.21.15.107:0/1500653713 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfcc075ad0 0x7fdfcc10f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:51.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.498+0000 7fdfcbe00640 1 --2- 172.21.15.107:0/1500653713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfcc075180 0x7fdfcc075590 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:51.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.498+0000 7fdfcbe00640 1 -- 172.21.15.107:0/1500653713 >> 172.21.15.107:0/1500653713 conn(0x7fdfcc0fee10 msgr2=0x7fdfcc101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.498+0000 7fdfcbe00640 1 -- 172.21.15.107:0/1500653713 shutdown_connections 2024-07-26T12:45:51.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.498+0000 7fdfcbe00640 1 -- 172.21.15.107:0/1500653713 wait complete. 2024-07-26T12:45:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.500+0000 7fdfcbe00640 1 Processor -- start 2024-07-26T12:45:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.500+0000 7fdfcbe00640 1 -- start start 2024-07-26T12:45:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfcbe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfcc075180 0x7fdfcc19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfcbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfcc075ad0 0x7fdfcc19c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:51.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfcbe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfcc10f980 0x7fdfcc1a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfcbe00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc115500 con 0x7fdfcc075180 2024-07-26T12:45:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfcbe00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc115380 con 0x7fdfcc10f980 2024-07-26T12:45:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfcbe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdfcc115680 con 0x7fdfcc075ad0 2024-07-26T12:45:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfcaa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfcc075180 0x7fdfcc19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfc3e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfcc075ad0 0x7fdfcc19c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfcaa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfcc075180 0x7fdfcc19bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46902/0 (socket says 172.21.15.107:46902) 2024-07-26T12:45:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfc3e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfcc075ad0 0x7fdfcc19c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:39622/0 (socket says 172.21.15.107:39622) 2024-07-26T12:45:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfcaa00640 1 -- 172.21.15.107:0/3872615847 learned_addr learned my addr 172.21.15.107:0/3872615847 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:51.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.501+0000 7fdfcb400640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfcc10f980 0x7fdfcc1a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:51.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.502+0000 7fdfc3e00640 1 -- 172.21.15.107:0/3872615847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfcc10f980 msgr2=0x7fdfcc1a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.502+0000 7fdfc3e00640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfcc10f980 0x7fdfcc1a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.502+0000 7fdfc3e00640 1 -- 172.21.15.107:0/3872615847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfcc075180 msgr2=0x7fdfcc19bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.502+0000 7fdfc3e00640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfcc075180 0x7fdfcc19bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.502+0000 7fdfc3e00640 1 -- 172.21.15.107:0/3872615847 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfcc1a1e60 con 0x7fdfcc075ad0 2024-07-26T12:45:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.502+0000 7fdfcaa00640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdfcc075180 0x7fdfcc19bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:45:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.502+0000 7fdfcb400640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdfcc10f980 0x7fdfcc1a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:45:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.502+0000 7fdfc3e00640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfcc075ad0 0x7fdfcc19c490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc077200 tx=0x7fdfbc00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.503+0000 7fdfc3400640 1 -- 172.21.15.107:0/3872615847 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc00cd00 con 0x7fdfcc075ad0 2024-07-26T12:45:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.503+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfcc1a2150 con 0x7fdfcc075ad0 2024-07-26T12:45:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.503+0000 7fdfc3400640 1 -- 172.21.15.107:0/3872615847 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfbc0035f0 con 0x7fdfcc075ad0 2024-07-26T12:45:51.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.503+0000 7fdfc3400640 1 -- 172.21.15.107:0/3872615847 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc010690 con 0x7fdfcc075ad0 2024-07-26T12:45:51.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.503+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfcc1a88b0 con 0x7fdfcc075ad0 2024-07-26T12:45:51.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.504+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdfcc1023a0 con 0x7fdfcc075ad0 2024-07-26T12:45:51.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.507+0000 7fdfc3400640 1 -- 172.21.15.107:0/3872615847 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fdfbc0037a0 con 0x7fdfcc075ad0 2024-07-26T12:45:51.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.508+0000 7fdfc3400640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdfb00b3cb0 0x7fdfb00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:51.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.508+0000 7fdfc3400640 1 -- 172.21.15.107:0/3872615847 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fdfbc0e4930 con 0x7fdfcc075ad0 2024-07-26T12:45:51.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.508+0000 7fdfcaa00640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdfb00b3cb0 0x7fdfb00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:51.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.509+0000 7fdfc3400640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] conn(0x7fdfb00bf350 0x7fdfb00c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:51.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.509+0000 7fdfc3400640 1 -- 172.21.15.107:0/3872615847 --> [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdfbc022e50 con 0x7fdfb00bf350 2024-07-26T12:45:51.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.509+0000 7fdfc3400640 1 -- 172.21.15.107:0/3872615847 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdfbc0e4ce0 con 0x7fdfcc075ad0 2024-07-26T12:45:51.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.509+0000 7fdfcb400640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] conn(0x7fdfb00bf350 0x7fdfb00c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:51.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.509+0000 7fdfcaa00640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdfb00b3cb0 0x7fdfb00b61c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4031440 tx=0x7fdfb40058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:51.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.510+0000 7fdfcb400640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] conn(0x7fdfb00bf350 0x7fdfb00c1840 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:51.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.510+0000 7fdfc3400640 1 -- 172.21.15.107:0/3872615847 <== osd.1 v2:172.21.15.112:6800/3125120524 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfbc022e50 con 0x7fdfb00bf350 2024-07-26T12:45:51.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.527+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 --> [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdfcc076690 con 0x7fdfb00bf350 2024-07-26T12:45:51.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.528+0000 7fdfc3400640 1 -- 172.21.15.107:0/3872615847 <== osd.1 v2:172.21.15.112:6800/3125120524 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdfcc076690 con 0x7fdfb00bf350 2024-07-26T12:45:51.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.528+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 >> [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] conn(0x7fdfb00bf350 msgr2=0x7fdfb00c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:51.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.528+0000 7fdfcbe00640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] conn(0x7fdfb00bf350 0x7fdfb00c1840 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:51.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.529+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdfb00b3cb0 msgr2=0x7fdfb00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:51.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.529+0000 7fdfcbe00640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdfb00b3cb0 0x7fdfb00b61c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4031440 tx=0x7fdfb40058d0 comp rx=0 tx=0).stop 2024-07-26T12:45:51.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.529+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfcc075ad0 msgr2=0x7fdfcc19c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:51.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.529+0000 7fdfcbe00640 1 --2- 172.21.15.107:0/3872615847 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfcc075ad0 0x7fdfcc19c490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc077200 tx=0x7fdfbc00ee50 comp rx=0 tx=0).stop 2024-07-26T12:45:51.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.529+0000 7fdfcb400640 1 -- 172.21.15.107:0/3872615847 reap_dead start 2024-07-26T12:45:51.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.529+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 shutdown_connections 2024-07-26T12:45:51.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.529+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 >> 172.21.15.107:0/3872615847 conn(0x7fdfcc0fee10 msgr2=0x7fdfcc101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:51.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.529+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 shutdown_connections 2024-07-26T12:45:51.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:51.529+0000 7fdfcbe00640 1 -- 172.21.15.107:0/3872615847 wait complete. 2024-07-26T12:45:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:51 smithi132 ceph-mon[37580]: pgmap v386: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:45:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:51 smithi112 ceph-mon[37501]: pgmap v386: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:45:52.246 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:51 smithi107 ceph-mon[32659]: pgmap v386: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:45:53.803 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:53 smithi107 ceph-mon[32659]: pgmap v387: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:53 smithi132 ceph-mon[37580]: pgmap v387: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:53 smithi112 ceph-mon[37501]: pgmap v387: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:45:54.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.641+0000 7f5fe6a00640 1 -- 172.21.15.107:0/726626598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe01022a0 msgr2=0x7f5fe01026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.641+0000 7f5fe6a00640 1 --2- 172.21.15.107:0/726626598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe01022a0 0x7f5fe01026b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009900 tx=0x7f5fd402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.641+0000 7f5fe6a00640 1 -- 172.21.15.107:0/726626598 shutdown_connections 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.641+0000 7f5fe6a00640 1 --2- 172.21.15.107:0/726626598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fe0108530 0x7f5fe010ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.641+0000 7f5fe6a00640 1 --2- 172.21.15.107:0/726626598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fe0107be0 0x7f5fe0107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.641+0000 7f5fe6a00640 1 --2- 172.21.15.107:0/726626598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe01022a0 0x7f5fe01026b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.641+0000 7f5fe6a00640 1 -- 172.21.15.107:0/726626598 >> 172.21.15.107:0/726626598 conn(0x7f5fe00fdcf0 msgr2=0x7f5fe0100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.641+0000 7f5fe6a00640 1 -- 172.21.15.107:0/726626598 shutdown_connections 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.641+0000 7f5fe6a00640 1 -- 172.21.15.107:0/726626598 wait complete. 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.642+0000 7f5fe6a00640 1 Processor -- start 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.642+0000 7f5fe6a00640 1 -- start start 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe6a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe01022a0 0x7f5fe00fe220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe6a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fe0107be0 0x7f5fe00fe760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:54.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe6a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fe0108530 0x7f5fe00feca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe6a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe0112250 con 0x7f5fe01022a0 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe6a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe01120d0 con 0x7f5fe0108530 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe6a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe01123d0 con 0x7f5fe0107be0 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe5600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe01022a0 0x7f5fe00fe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe4c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fe0107be0 0x7f5fe00fe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe4c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fe0107be0 0x7f5fe00fe760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:34056/0 (socket says 172.21.15.107:34056) 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe5600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe01022a0 0x7f5fe00fe220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45342/0 (socket says 172.21.15.107:45342) 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.643+0000 7f5fe4c00640 1 -- 172.21.15.107:0/828096200 learned_addr learned my addr 172.21.15.107:0/828096200 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.644+0000 7f5fe6000640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fe0108530 0x7f5fe00feca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.644+0000 7f5fe5600640 1 -- 172.21.15.107:0/828096200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fe0107be0 msgr2=0x7f5fe00fe760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.644+0000 7f5fe5600640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fe0107be0 0x7f5fe00fe760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.644+0000 7f5fe5600640 1 -- 172.21.15.107:0/828096200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fe0108530 msgr2=0x7f5fe00feca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.644+0000 7f5fe5600640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5fe0108530 0x7f5fe00feca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:54.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.644+0000 7f5fe5600640 1 -- 172.21.15.107:0/828096200 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe01a8050 con 0x7f5fe01022a0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.644+0000 7f5fe4c00640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5fe0107be0 0x7f5fe00fe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.644+0000 7f5fe5600640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe01022a0 0x7f5fe00fe220 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5fd402a9e0 tx=0x7f5fd4003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.645+0000 7f5fde000640 1 -- 172.21.15.107:0/828096200 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd403f070 con 0x7f5fe01022a0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.645+0000 7f5fde000640 1 -- 172.21.15.107:0/828096200 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fd4031b50 con 0x7f5fe01022a0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.645+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe01a82e0 con 0x7f5fe01022a0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.645+0000 7f5fde000640 1 -- 172.21.15.107:0/828096200 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd4035030 con 0x7f5fe01022a0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.645+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe01a8790 con 0x7f5fe01022a0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.646+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5fe0106df0 con 0x7f5fe01022a0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.649+0000 7f5fde000640 1 -- 172.21.15.107:0/828096200 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f5fd4042070 con 0x7f5fe01022a0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.650+0000 7f5fde000640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5fc80b3cb0 0x7f5fc80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.651+0000 7f5fde000640 1 -- 172.21.15.107:0/828096200 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5fd4104c80 con 0x7f5fe01022a0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.651+0000 7f5fe4c00640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5fc80b3cb0 0x7f5fc80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.651+0000 7f5fde000640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] conn(0x7f5fc80bf350 0x7f5fc80c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.651+0000 7f5fde000640 1 -- 172.21.15.107:0/828096200 --> [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5fc80c1f10 con 0x7f5fc80bf350 2024-07-26T12:45:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.651+0000 7f5fde000640 1 -- 172.21.15.107:0/828096200 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f5fd4105030 con 0x7f5fe01022a0 2024-07-26T12:45:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.651+0000 7f5fe6000640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] conn(0x7f5fc80bf350 0x7f5fc80c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.652+0000 7f5fe4c00640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5fc80b3cb0 0x7f5fc80b61c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5fe01017b0 tx=0x7f5fcc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.652+0000 7f5fe6000640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] conn(0x7f5fc80bf350 0x7f5fc80c1840 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.653+0000 7f5fde000640 1 -- 172.21.15.107:0/828096200 <== osd.3 v2:172.21.15.107:6810/1620179989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5fc80c1f10 con 0x7f5fc80bf350 2024-07-26T12:45:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.668+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 --> [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5fe0103520 con 0x7f5fc80bf350 2024-07-26T12:45:54.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.668+0000 7f5fde000640 1 -- 172.21.15.107:0/828096200 <== osd.3 v2:172.21.15.107:6810/1620179989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5fe0103520 con 0x7f5fc80bf350 2024-07-26T12:45:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.669+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] conn(0x7f5fc80bf350 msgr2=0x7f5fc80c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.669+0000 7f5fe6a00640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] conn(0x7f5fc80bf350 0x7f5fc80c1840 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.669+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5fc80b3cb0 msgr2=0x7f5fc80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.669+0000 7f5fe6a00640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f5fc80b3cb0 0x7f5fc80b61c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5fe01017b0 tx=0x7f5fcc005f90 comp rx=0 tx=0).stop 2024-07-26T12:45:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.669+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe01022a0 msgr2=0x7f5fe00fe220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.669+0000 7f5fe6a00640 1 --2- 172.21.15.107:0/828096200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5fe01022a0 0x7f5fe00fe220 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5fd402a9e0 tx=0x7f5fd4003b90 comp rx=0 tx=0).stop 2024-07-26T12:45:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.669+0000 7f5fe6000640 1 -- 172.21.15.107:0/828096200 reap_dead start 2024-07-26T12:45:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.669+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 shutdown_connections 2024-07-26T12:45:54.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.670+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 >> 172.21.15.107:0/828096200 conn(0x7f5fe00fdcf0 msgr2=0x7f5fe006ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:54.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.670+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 shutdown_connections 2024-07-26T12:45:54.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:54.670+0000 7f5fe6a00640 1 -- 172.21.15.107:0/828096200 wait complete. 2024-07-26T12:45:54.941 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:45:55.977 INFO:teuthology.orchestra.run.smithi107.stdout:60129542246 2024-07-26T12:45:55.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-07-26T12:45:56.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:55 smithi107 ceph-mon[32659]: pgmap v388: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:45:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:55 smithi132 ceph-mon[37580]: pgmap v388: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:45:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:55 smithi112 ceph-mon[37501]: pgmap v388: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:45:57.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:57 smithi107 ceph-mon[32659]: pgmap v389: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:57 smithi112 ceph-mon[37501]: pgmap v389: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:58.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.231+0000 7fb61d200640 1 -- 172.21.15.107:0/3621185412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6181024a0 msgr2=0x7fb6181028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.231+0000 7fb61d200640 1 --2- 172.21.15.107:0/3621185412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6181024a0 0x7fb6181028b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb608009900 tx=0x7fb60802a410 comp rx=0 tx=0).stop 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.231+0000 7fb61d200640 1 -- 172.21.15.107:0/3621185412 shutdown_connections 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.231+0000 7fb61d200640 1 --2- 172.21.15.107:0/3621185412 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb618108730 0x7fb61810ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.231+0000 7fb61d200640 1 --2- 172.21.15.107:0/3621185412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb618107de0 0x7fb6181081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.231+0000 7fb61d200640 1 --2- 172.21.15.107:0/3621185412 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6181024a0 0x7fb6181028b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.231+0000 7fb61d200640 1 -- 172.21.15.107:0/3621185412 >> 172.21.15.107:0/3621185412 conn(0x7fb618079160 msgr2=0x7fb618079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.232+0000 7fb61d200640 1 -- 172.21.15.107:0/3621185412 shutdown_connections 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.232+0000 7fb61d200640 1 -- 172.21.15.107:0/3621185412 wait complete. 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.233+0000 7fb61d200640 1 Processor -- start 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.233+0000 7fb61d200640 1 -- start start 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb61d200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6181024a0 0x7fb61819c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:58.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb61d200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb618107de0 0x7fb61819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb61d200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb618108730 0x7fb6181a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb61d200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb6181123b0 con 0x7fb6181024a0 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb61d200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb618112230 con 0x7fb618107de0 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb61d200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb618112530 con 0x7fb618108730 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb615400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb618107de0 0x7fb61819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb615e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6181024a0 0x7fb61819c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb615400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb618107de0 0x7fb61819c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:47724/0 (socket says 172.21.15.107:47724) 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb615400640 1 -- 172.21.15.107:0/3116006288 learned_addr learned my addr 172.21.15.107:0/3116006288 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.234+0000 7fb616800640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb618108730 0x7fb6181a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb615400640 1 -- 172.21.15.107:0/3116006288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb618108730 msgr2=0x7fb6181a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb615400640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb618108730 0x7fb6181a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb615400640 1 -- 172.21.15.107:0/3116006288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6181024a0 msgr2=0x7fb61819c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb615400640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6181024a0 0x7fb61819c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb615400640 1 -- 172.21.15.107:0/3116006288 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6181a2110 con 0x7fb618107de0 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb616800640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb618108730 0x7fb6181a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb615400640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb618107de0 0x7fb61819c740 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb60400e9c0 tx=0x7fb60400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb60ea00640 1 -- 172.21.15.107:0/3116006288 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60400cd80 con 0x7fb618107de0 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb60ea00640 1 -- 172.21.15.107:0/3116006288 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6040054d0 con 0x7fb618107de0 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb60ea00640 1 -- 172.21.15.107:0/3116006288 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb604010430 con 0x7fb618107de0 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.235+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6181a2400 con 0x7fb618107de0 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.236+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6181a8b60 con 0x7fb618107de0 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.237+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb61806bf40 con 0x7fb618107de0 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.239+0000 7fb60ea00640 1 -- 172.21.15.107:0/3116006288 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fb604004fb0 con 0x7fb618107de0 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.241+0000 7fb60ea00640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb5fc0b3d30 0x7fb5fc0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:58.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.241+0000 7fb60ea00640 1 -- 172.21.15.107:0/3116006288 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb6040e4ab0 con 0x7fb618107de0 2024-07-26T12:45:58.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.241+0000 7fb615e00640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb5fc0b3d30 0x7fb5fc0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:58.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.241+0000 7fb60ea00640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] conn(0x7fb5fc0bf3d0 0x7fb5fc0c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:58.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.241+0000 7fb60ea00640 1 -- 172.21.15.107:0/3116006288 --> [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5fc0c1f90 con 0x7fb5fc0bf3d0 2024-07-26T12:45:58.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.241+0000 7fb60ea00640 1 -- 172.21.15.107:0/3116006288 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb6040e4e60 con 0x7fb618107de0 2024-07-26T12:45:58.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.242+0000 7fb616800640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] conn(0x7fb5fc0bf3d0 0x7fb5fc0c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:58.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.242+0000 7fb615e00640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb5fc0b3d30 0x7fb5fc0b6240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb608031440 tx=0x7fb608009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:58.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.242+0000 7fb616800640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] conn(0x7fb5fc0bf3d0 0x7fb5fc0c18c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.243+0000 7fb60ea00640 1 -- 172.21.15.107:0/3116006288 <== osd.7 v2:172.21.15.112:6816/3770813146 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5fc0c1f90 con 0x7fb5fc0bf3d0 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.258+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 --> [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb61806adc0 con 0x7fb5fc0bf3d0 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.259+0000 7fb60ea00640 1 -- 172.21.15.107:0/3116006288 <== osd.7 v2:172.21.15.112:6816/3770813146 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb61806adc0 con 0x7fb5fc0bf3d0 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.259+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 >> [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] conn(0x7fb5fc0bf3d0 msgr2=0x7fb5fc0c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.259+0000 7fb61d200640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] conn(0x7fb5fc0bf3d0 0x7fb5fc0c18c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.260+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb5fc0b3d30 msgr2=0x7fb5fc0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.260+0000 7fb61d200640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb5fc0b3d30 0x7fb5fc0b6240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb608031440 tx=0x7fb608009560 comp rx=0 tx=0).stop 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.260+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb618107de0 msgr2=0x7fb61819c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.260+0000 7fb61d200640 1 --2- 172.21.15.107:0/3116006288 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb618107de0 0x7fb61819c740 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb60400e9c0 tx=0x7fb60400ee80 comp rx=0 tx=0).stop 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.260+0000 7fb616800640 1 -- 172.21.15.107:0/3116006288 reap_dead start 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.260+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 shutdown_connections 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.260+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 >> 172.21.15.107:0/3116006288 conn(0x7fb618079160 msgr2=0x7fb61806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.260+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 shutdown_connections 2024-07-26T12:45:58.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.260+0000 7fb61d200640 1 -- 172.21.15.107:0/3116006288 wait complete. 2024-07-26T12:45:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:57 smithi132 ceph-mon[37580]: pgmap v389: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:58.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.382+0000 7f240ac00640 1 -- 172.21.15.107:0/2743959646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2404077cc0 msgr2=0x7f2404113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.382+0000 7f240ac00640 1 --2- 172.21.15.107:0/2743959646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2404077cc0 0x7f2404113a50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f23f0009900 tx=0x7f23f002a410 comp rx=0 tx=0).stop 2024-07-26T12:45:58.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.382+0000 7f240ac00640 1 -- 172.21.15.107:0/2743959646 shutdown_connections 2024-07-26T12:45:58.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.382+0000 7f240ac00640 1 --2- 172.21.15.107:0/2743959646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2404113f90 0x7f24041163b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.382+0000 7f240ac00640 1 --2- 172.21.15.107:0/2743959646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2404077cc0 0x7f2404113a50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.382+0000 7f240ac00640 1 --2- 172.21.15.107:0/2743959646 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2404077370 0x7f2404077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.382+0000 7f240ac00640 1 -- 172.21.15.107:0/2743959646 >> 172.21.15.107:0/2743959646 conn(0x7f2404101040 msgr2=0x7f2404103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:58.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.382+0000 7f240ac00640 1 -- 172.21.15.107:0/2743959646 shutdown_connections 2024-07-26T12:45:58.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.382+0000 7f240ac00640 1 -- 172.21.15.107:0/2743959646 wait complete. 2024-07-26T12:45:58.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.383+0000 7f240ac00640 1 Processor -- start 2024-07-26T12:45:58.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f240ac00640 1 -- start start 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f240ac00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2404077370 0x7f24041a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f240ac00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2404077cc0 0x7f24041a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f240ac00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2404113f90 0x7f24041a5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f240ac00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f24041198f0 con 0x7f2404077cc0 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f240ac00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2404119770 con 0x7f2404113f90 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f240ac00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2404119a70 con 0x7f2404077370 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f2408e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2404077cc0 0x7f24041a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f2408e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2404077cc0 0x7f24041a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45386/0 (socket says 172.21.15.107:45386) 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f2408e00640 1 -- 172.21.15.107:0/202595276 learned_addr learned my addr 172.21.15.107:0/202595276 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f2409800640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2404077370 0x7f24041a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.384+0000 7f240a200640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2404113f90 0x7f24041a5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:58.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.385+0000 7f2409800640 1 -- 172.21.15.107:0/202595276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2404113f90 msgr2=0x7f24041a5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.385+0000 7f2409800640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2404113f90 0x7f24041a5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.385+0000 7f2409800640 1 -- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2404077cc0 msgr2=0x7f24041a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.385+0000 7f2409800640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2404077cc0 0x7f24041a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.385+0000 7f2409800640 1 -- 172.21.15.107:0/202595276 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24041a62f0 con 0x7f2404077370 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.385+0000 7f2409800640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2404077370 0x7f24041a0380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009fd0 tx=0x7f23f800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.385+0000 7f2402000640 1 -- 172.21.15.107:0/202595276 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f8004b20 con 0x7f2404077370 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.386+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24041a65e0 con 0x7f2404077370 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.386+0000 7f2402000640 1 -- 172.21.15.107:0/202595276 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23f8004cc0 con 0x7f2404077370 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.386+0000 7f2402000640 1 -- 172.21.15.107:0/202595276 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f8010720 con 0x7f2404077370 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.386+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24041acd40 con 0x7f2404077370 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.386+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2404063540 con 0x7f2404077370 2024-07-26T12:45:58.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.389+0000 7f2402000640 1 -- 172.21.15.107:0/202595276 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f23f8005020 con 0x7f2404077370 2024-07-26T12:45:58.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.391+0000 7f2402000640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f23ec0b3cb0 0x7f23ec0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:58.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.391+0000 7f2402000640 1 -- 172.21.15.107:0/202595276 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f23f80e4c90 con 0x7f2404077370 2024-07-26T12:45:58.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.391+0000 7f2408e00640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f23ec0b3cb0 0x7f23ec0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:58.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.391+0000 7f2402000640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] conn(0x7f23ec0bf350 0x7f23ec0c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:45:58.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.391+0000 7f2402000640 1 -- 172.21.15.107:0/202595276 --> [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23ec0c1f10 con 0x7f23ec0bf350 2024-07-26T12:45:58.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.391+0000 7f2402000640 1 -- 172.21.15.107:0/202595276 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f23f80e5040 con 0x7f2404077370 2024-07-26T12:45:58.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.391+0000 7f240a200640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] conn(0x7f23ec0bf350 0x7f23ec0c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:45:58.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.392+0000 7f2408e00640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f23ec0b3cb0 0x7f23ec0b61c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2404114530 tx=0x7f23f0009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:58.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.392+0000 7f240a200640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] conn(0x7f23ec0bf350 0x7f23ec0c1840 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:45:58.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.393+0000 7f2402000640 1 -- 172.21.15.107:0/202595276 <== osd.0 v2:172.21.15.107:6802/2354859122 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23ec0c1f10 con 0x7f23ec0bf350 2024-07-26T12:45:58.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.410+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 --> [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2404063940 con 0x7f23ec0bf350 2024-07-26T12:45:58.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.410+0000 7f2402000640 1 -- 172.21.15.107:0/202595276 <== osd.0 v2:172.21.15.107:6802/2354859122 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2404063940 con 0x7f23ec0bf350 2024-07-26T12:45:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.411+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] conn(0x7f23ec0bf350 msgr2=0x7f23ec0c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.411+0000 7f240ac00640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] conn(0x7f23ec0bf350 0x7f23ec0c1840 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:45:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.411+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f23ec0b3cb0 msgr2=0x7f23ec0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.412+0000 7f240ac00640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f23ec0b3cb0 0x7f23ec0b61c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2404114530 tx=0x7f23f0009560 comp rx=0 tx=0).stop 2024-07-26T12:45:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.412+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2404077370 msgr2=0x7f24041a0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:45:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.412+0000 7f240ac00640 1 --2- 172.21.15.107:0/202595276 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2404077370 0x7f24041a0380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009fd0 tx=0x7f23f800ef90 comp rx=0 tx=0).stop 2024-07-26T12:45:58.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.412+0000 7f240a200640 1 -- 172.21.15.107:0/202595276 reap_dead start 2024-07-26T12:45:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.412+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 shutdown_connections 2024-07-26T12:45:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.412+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 >> 172.21.15.107:0/202595276 conn(0x7f2404101040 msgr2=0x7f2404116090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:45:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.413+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 shutdown_connections 2024-07-26T12:45:58.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:58.413+0000 7f240ac00640 1 -- 172.21.15.107:0/202595276 wait complete. 2024-07-26T12:45:59.908 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:45:59 smithi107 ceph-mon[32659]: pgmap v390: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:00.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:45:59 smithi132 ceph-mon[37580]: pgmap v390: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:45:59 smithi112 ceph-mon[37501]: pgmap v390: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:01.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.065+0000 7fec8e600640 1 -- 172.21.15.107:0/1005251955 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec88113f80 msgr2=0x7fec881163a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.065+0000 7fec8e600640 1 --2- 172.21.15.107:0/1005251955 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec88113f80 0x7fec881163a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fec80009ca0 tx=0x7fec8002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.065+0000 7fec8e600640 1 -- 172.21.15.107:0/1005251955 shutdown_connections 2024-07-26T12:46:01.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.065+0000 7fec8e600640 1 --2- 172.21.15.107:0/1005251955 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec88113f80 0x7fec881163a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.065+0000 7fec8e600640 1 --2- 172.21.15.107:0/1005251955 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec88077cc0 0x7fec88113a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.065+0000 7fec8e600640 1 --2- 172.21.15.107:0/1005251955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec88077370 0x7fec88077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.065+0000 7fec8e600640 1 -- 172.21.15.107:0/1005251955 >> 172.21.15.107:0/1005251955 conn(0x7fec88101010 msgr2=0x7fec88103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:01.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.066+0000 7fec8e600640 1 -- 172.21.15.107:0/1005251955 shutdown_connections 2024-07-26T12:46:01.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.066+0000 7fec8e600640 1 -- 172.21.15.107:0/1005251955 wait complete. 2024-07-26T12:46:01.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.067+0000 7fec8e600640 1 Processor -- start 2024-07-26T12:46:01.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.067+0000 7fec8e600640 1 -- start start 2024-07-26T12:46:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.068+0000 7fec8e600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec88077370 0x7fec881a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.068+0000 7fec8e600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec88077cc0 0x7fec881a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.068+0000 7fec8e600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec88113f80 0x7fec881a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.068+0000 7fec8e600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fec88119900 con 0x7fec88113f80 2024-07-26T12:46:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.068+0000 7fec8e600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fec88119780 con 0x7fec88077370 2024-07-26T12:46:01.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.068+0000 7fec8e600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fec88119a80 con 0x7fec88077cc0 2024-07-26T12:46:01.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.068+0000 7fec8d200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec88077370 0x7fec881a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:01.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.068+0000 7fec8d200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec88077370 0x7fec881a02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:47768/0 (socket says 172.21.15.107:47768) 2024-07-26T12:46:01.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.069+0000 7fec8d200640 1 -- 172.21.15.107:0/1094767684 learned_addr learned my addr 172.21.15.107:0/1094767684 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:01.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.068+0000 7fec87e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec88077cc0 0x7fec881a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:01.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.069+0000 7fec8dc00640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec88113f80 0x7fec881a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:01.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.069+0000 7fec87e00640 1 -- 172.21.15.107:0/1094767684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec88077370 msgr2=0x7fec881a02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.069+0000 7fec87e00640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fec88077370 0x7fec881a02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.069+0000 7fec87e00640 1 -- 172.21.15.107:0/1094767684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec88113f80 msgr2=0x7fec881a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.069+0000 7fec87e00640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec88113f80 0x7fec881a5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.069+0000 7fec87e00640 1 -- 172.21.15.107:0/1094767684 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec881a6220 con 0x7fec88077cc0 2024-07-26T12:46:01.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.070+0000 7fec87e00640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec88077cc0 0x7fec881a0850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fec8806c310 tx=0x7fec7800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:01.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.070+0000 7fec85600640 1 -- 172.21.15.107:0/1094767684 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7800cc40 con 0x7fec88077cc0 2024-07-26T12:46:01.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.070+0000 7fec85600640 1 -- 172.21.15.107:0/1094767684 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec7800cde0 con 0x7fec88077cc0 2024-07-26T12:46:01.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.070+0000 7fec85600640 1 -- 172.21.15.107:0/1094767684 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78004a00 con 0x7fec88077cc0 2024-07-26T12:46:01.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.070+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec881a6510 con 0x7fec88077cc0 2024-07-26T12:46:01.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.070+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec881acc70 con 0x7fec88077cc0 2024-07-26T12:46:01.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.071+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec88063540 con 0x7fec88077cc0 2024-07-26T12:46:01.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.074+0000 7fec85600640 1 -- 172.21.15.107:0/1094767684 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fec78004ba0 con 0x7fec88077cc0 2024-07-26T12:46:01.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.075+0000 7fec85600640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fec700b3d30 0x7fec700b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:01.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.076+0000 7fec85600640 1 -- 172.21.15.107:0/1094767684 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fec780e46f0 con 0x7fec88077cc0 2024-07-26T12:46:01.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.076+0000 7fec8d200640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fec700b3d30 0x7fec700b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:01.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.076+0000 7fec85600640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] conn(0x7fec700bf3d0 0x7fec700c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:01.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.076+0000 7fec85600640 1 -- 172.21.15.107:0/1094767684 --> [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec7801de50 con 0x7fec700bf3d0 2024-07-26T12:46:01.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.076+0000 7fec85600640 1 -- 172.21.15.107:0/1094767684 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fec780e4aa0 con 0x7fec88077cc0 2024-07-26T12:46:01.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.077+0000 7fec8dc00640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] conn(0x7fec700bf3d0 0x7fec700c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:01.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.077+0000 7fec8d200640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fec700b3d30 0x7fec700b6240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009c90 tx=0x7fec7c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:01.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.077+0000 7fec8dc00640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] conn(0x7fec700bf3d0 0x7fec700c18c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:01.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.078+0000 7fec85600640 1 -- 172.21.15.107:0/1094767684 <== osd.10 v2:172.21.15.112:6824/3415013149 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fec7801de50 con 0x7fec700bf3d0 2024-07-26T12:46:01.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.094+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 --> [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fec880639c0 con 0x7fec700bf3d0 2024-07-26T12:46:01.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.095+0000 7fec85600640 1 -- 172.21.15.107:0/1094767684 <== osd.10 v2:172.21.15.112:6824/3415013149 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fec880639c0 con 0x7fec700bf3d0 2024-07-26T12:46:01.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.095+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 >> [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] conn(0x7fec700bf3d0 msgr2=0x7fec700c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.095+0000 7fec8e600640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] conn(0x7fec700bf3d0 0x7fec700c18c0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.096+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fec700b3d30 msgr2=0x7fec700b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.096+0000 7fec8e600640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fec700b3d30 0x7fec700b6240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009c90 tx=0x7fec7c005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:01.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.096+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec88077cc0 msgr2=0x7fec881a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.096+0000 7fec8e600640 1 --2- 172.21.15.107:0/1094767684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec88077cc0 0x7fec881a0850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fec8806c310 tx=0x7fec7800ee50 comp rx=0 tx=0).stop 2024-07-26T12:46:01.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.096+0000 7fec8dc00640 1 -- 172.21.15.107:0/1094767684 reap_dead start 2024-07-26T12:46:01.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.096+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 shutdown_connections 2024-07-26T12:46:01.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.096+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 >> 172.21.15.107:0/1094767684 conn(0x7fec88101010 msgr2=0x7fec88115ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:01.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.096+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 shutdown_connections 2024-07-26T12:46:01.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:01.096+0000 7fec8e600640 1 -- 172.21.15.107:0/1094767684 wait complete. 2024-07-26T12:46:01.123 INFO:teuthology.orchestra.run.smithi107.stdout:111669149787 2024-07-26T12:46:01.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-07-26T12:46:01.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.316+0000 7f78ab400640 1 -- 172.21.15.107:0/3123543570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a41035f0 msgr2=0x7f78a4103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.316+0000 7f78ab400640 1 --2- 172.21.15.107:0/3123543570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a41035f0 0x7f78a4103a00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7898009900 tx=0x7f789802a410 comp rx=0 tx=0).stop 2024-07-26T12:46:01.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.316+0000 7f78ab400640 1 -- 172.21.15.107:0/3123543570 shutdown_connections 2024-07-26T12:46:01.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.316+0000 7f78ab400640 1 --2- 172.21.15.107:0/3123543570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a4109830 0x7f78a4110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.316+0000 7f78ab400640 1 --2- 172.21.15.107:0/3123543570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4108ee0 0x7f78a41092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.316+0000 7f78ab400640 1 --2- 172.21.15.107:0/3123543570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a41035f0 0x7f78a4103a00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.316+0000 7f78ab400640 1 -- 172.21.15.107:0/3123543570 >> 172.21.15.107:0/3123543570 conn(0x7f78a407b340 msgr2=0x7f78a407b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.316+0000 7f78ab400640 1 -- 172.21.15.107:0/3123543570 shutdown_connections 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.317+0000 7f78ab400640 1 -- 172.21.15.107:0/3123543570 wait complete. 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.318+0000 7f78ab400640 1 Processor -- start 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.318+0000 7f78ab400640 1 -- start start 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.318+0000 7f78ab400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a41035f0 0x7f78a4075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.318+0000 7f78ab400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4108ee0 0x7f78a4075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.318+0000 7f78ab400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a4109830 0x7f78a4075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.318+0000 7f78ab400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41134e0 con 0x7f78a41035f0 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.318+0000 7f78ab400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4113360 con 0x7f78a4109830 2024-07-26T12:46:01.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.318+0000 7f78ab400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4113660 con 0x7f78a4108ee0 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.319+0000 7f78a8a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a41035f0 0x7f78a4075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.319+0000 7f78a8a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a41035f0 0x7f78a4075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45362/0 (socket says 172.21.15.107:45362) 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.319+0000 7f78a8a00640 1 -- 172.21.15.107:0/2290167598 learned_addr learned my addr 172.21.15.107:0/2290167598 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.319+0000 7f78a9400640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a4109830 0x7f78a4075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.319+0000 7f78a3e00640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4108ee0 0x7f78a4075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.319+0000 7f78a8a00640 1 -- 172.21.15.107:0/2290167598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4108ee0 msgr2=0x7f78a4075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.319+0000 7f78a8a00640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f78a4108ee0 0x7f78a4075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.319+0000 7f78a8a00640 1 -- 172.21.15.107:0/2290167598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a4109830 msgr2=0x7f78a4075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.319+0000 7f78a8a00640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f78a4109830 0x7f78a4075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.319+0000 7f78a8a00640 1 -- 172.21.15.107:0/2290167598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a41a8230 con 0x7f78a41035f0 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.320+0000 7f78a8a00640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a41035f0 0x7f78a4075140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7898005940 tx=0x7f78980040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:01.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.320+0000 7f78a1600640 1 -- 172.21.15.107:0/2290167598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789803f030 con 0x7f78a41035f0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.320+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a41a84c0 con 0x7f78a41035f0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.320+0000 7f78a1600640 1 -- 172.21.15.107:0/2290167598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f78980044c0 con 0x7f78a41035f0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.320+0000 7f78a1600640 1 -- 172.21.15.107:0/2290167598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7898035070 con 0x7f78a41035f0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.320+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a41a89a0 con 0x7f78a41035f0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.321+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f78a4063540 con 0x7f78a41035f0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.324+0000 7f78a1600640 1 -- 172.21.15.107:0/2290167598 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f7898042030 con 0x7f78a41035f0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.326+0000 7f78a1600640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f788c0b3cb0 0x7f788c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.326+0000 7f78a1600640 1 -- 172.21.15.107:0/2290167598 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7898104d30 con 0x7f78a41035f0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.326+0000 7f78a3e00640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f788c0b3cb0 0x7f788c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.326+0000 7f78a1600640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] conn(0x7f788c0bf350 0x7f788c0c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.326+0000 7f78a1600640 1 -- 172.21.15.107:0/2290167598 --> [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7898046e50 con 0x7f788c0bf350 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.326+0000 7f78a1600640 1 -- 172.21.15.107:0/2290167598 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f7898105180 con 0x7f78a41035f0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.327+0000 7f78a9400640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] conn(0x7f788c0bf350 0x7f788c0c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:01.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.327+0000 7f78a3e00640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f788c0b3cb0 0x7f788c0b61c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f78a41055b0 tx=0x7f7890005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:01.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.327+0000 7f78a9400640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] conn(0x7f788c0bf350 0x7f788c0c1840 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:01.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.328+0000 7f78a1600640 1 -- 172.21.15.107:0/2290167598 <== osd.8 v2:172.21.15.132:6816/2677294944 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7898046e50 con 0x7f788c0bf350 2024-07-26T12:46:01.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.343+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 --> [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f78a40639c0 con 0x7f788c0bf350 2024-07-26T12:46:01.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.344+0000 7f78a1600640 1 -- 172.21.15.107:0/2290167598 <== osd.8 v2:172.21.15.132:6816/2677294944 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f78a40639c0 con 0x7f788c0bf350 2024-07-26T12:46:01.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.344+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 >> [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] conn(0x7f788c0bf350 msgr2=0x7f788c0c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.344+0000 7f78ab400640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] conn(0x7f788c0bf350 0x7f788c0c1840 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.344+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f788c0b3cb0 msgr2=0x7f788c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.344+0000 7f78ab400640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f788c0b3cb0 0x7f788c0b61c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f78a41055b0 tx=0x7f7890005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:01.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.344+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a41035f0 msgr2=0x7f78a4075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:01.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.344+0000 7f78ab400640 1 --2- 172.21.15.107:0/2290167598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a41035f0 0x7f78a4075140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7898005940 tx=0x7f78980040c0 comp rx=0 tx=0).stop 2024-07-26T12:46:01.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.344+0000 7f78a9400640 1 -- 172.21.15.107:0/2290167598 reap_dead start 2024-07-26T12:46:01.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.344+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 shutdown_connections 2024-07-26T12:46:01.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.345+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 >> 172.21.15.107:0/2290167598 conn(0x7f78a407b340 msgr2=0x7f78a4107030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:01.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.345+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 shutdown_connections 2024-07-26T12:46:01.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:45:57.345+0000 7f78ab400640 1 -- 172.21.15.107:0/2290167598 wait complete. 2024-07-26T12:46:02.148 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:01 smithi107 ceph-mon[32659]: pgmap v391: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:46:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:01 smithi112 ceph-mon[37501]: pgmap v391: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:46:02.211 INFO:teuthology.orchestra.run.smithi107.stdout:38654705772 2024-07-26T12:46:02.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-07-26T12:46:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:01 smithi132 ceph-mon[37580]: pgmap v391: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:46:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.492+0000 7fe361200640 1 -- 172.21.15.107:0/3817540298 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe35c10ba40 msgr2=0x7fe35c112270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.492+0000 7fe361200640 1 --2- 172.21.15.107:0/3817540298 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe35c10ba40 0x7fe35c112270 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe354009be0 tx=0x7fe35402a6c0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.492+0000 7fe361200640 1 -- 172.21.15.107:0/3817540298 shutdown_connections 2024-07-26T12:46:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.492+0000 7fe361200640 1 --2- 172.21.15.107:0/3817540298 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe35c10ba40 0x7fe35c112270 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.492+0000 7fe361200640 1 --2- 172.21.15.107:0/3817540298 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe35c10b0f0 0x7fe35c10b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.492+0000 7fe361200640 1 --2- 172.21.15.107:0/3817540298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c1057b0 0x7fe35c105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.492+0000 7fe361200640 1 -- 172.21.15.107:0/3817540298 >> 172.21.15.107:0/3817540298 conn(0x7fe35c101240 msgr2=0x7fe35c103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.493+0000 7fe361200640 1 -- 172.21.15.107:0/3817540298 shutdown_connections 2024-07-26T12:46:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.493+0000 7fe361200640 1 -- 172.21.15.107:0/3817540298 wait complete. 2024-07-26T12:46:03.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.494+0000 7fe361200640 1 Processor -- start 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.494+0000 7fe361200640 1 -- start start 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.494+0000 7fe361200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe35c1057b0 0x7fe35c19c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.494+0000 7fe361200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c10b0f0 0x7fe35c19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.494+0000 7fe361200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe35c10ba40 0x7fe35c1a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.494+0000 7fe361200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c115710 con 0x7fe35c10b0f0 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.494+0000 7fe361200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c115590 con 0x7fe35c10ba40 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe361200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe35c115890 con 0x7fe35c1057b0 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe359400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c10b0f0 0x7fe35c19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe359e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe35c1057b0 0x7fe35c19c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe359e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe35c1057b0 0x7fe35c19c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:34144/0 (socket says 172.21.15.107:34144) 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe359400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c10b0f0 0x7fe35c19c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45430/0 (socket says 172.21.15.107:45430) 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe359e00640 1 -- 172.21.15.107:0/286599173 learned_addr learned my addr 172.21.15.107:0/286599173 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe35a800640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe35c10ba40 0x7fe35c1a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:03.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe359400640 1 -- 172.21.15.107:0/286599173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe35c1057b0 msgr2=0x7fe35c19c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe359400640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe35c1057b0 0x7fe35c19c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe359400640 1 -- 172.21.15.107:0/286599173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe35c10ba40 msgr2=0x7fe35c1a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe359400640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe35c10ba40 0x7fe35c1a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.495+0000 7fe359400640 1 -- 172.21.15.107:0/286599173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35c1a2090 con 0x7fe35c10b0f0 2024-07-26T12:46:03.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.496+0000 7fe359400640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c10b0f0 0x7fe35c19c6c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe35c06c4d0 tx=0x7fe34800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:03.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.496+0000 7fe352a00640 1 -- 172.21.15.107:0/286599173 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34800cc90 con 0x7fe35c10b0f0 2024-07-26T12:46:03.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.496+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe35c1a2380 con 0x7fe35c10b0f0 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.496+0000 7fe352a00640 1 -- 172.21.15.107:0/286599173 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe34800ee70 con 0x7fe35c10b0f0 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.496+0000 7fe352a00640 1 -- 172.21.15.107:0/286599173 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe348004a00 con 0x7fe35c10b0f0 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.496+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe35c1a8ae0 con 0x7fe35c10b0f0 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.497+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe35c063580 con 0x7fe35c10b0f0 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.501+0000 7fe352a00640 1 -- 172.21.15.107:0/286599173 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fe348004ba0 con 0x7fe35c10b0f0 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.502+0000 7fe352a00640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3400b3cb0 0x7fe3400b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.502+0000 7fe352a00640 1 -- 172.21.15.107:0/286599173 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe3480e4330 con 0x7fe35c10b0f0 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.503+0000 7fe359e00640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3400b3cb0 0x7fe3400b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.503+0000 7fe352a00640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] conn(0x7fe3400bf350 0x7fe3400c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.503+0000 7fe352a00640 1 -- 172.21.15.107:0/286599173 --> [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe348022e50 con 0x7fe3400bf350 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.503+0000 7fe352a00640 1 -- 172.21.15.107:0/286599173 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe3480e46e0 con 0x7fe35c10b0f0 2024-07-26T12:46:03.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.503+0000 7fe35a800640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] conn(0x7fe3400bf350 0x7fe3400c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.503+0000 7fe359e00640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3400b3cb0 0x7fe3400b61c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0095f0 tx=0x7fe34c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.504+0000 7fe35a800640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] conn(0x7fe3400bf350 0x7fe3400c1840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.505+0000 7fe352a00640 1 -- 172.21.15.107:0/286599173 <== osd.11 v2:172.21.15.132:6824/3905824310 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe348022e50 con 0x7fe3400bf350 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.527+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 --> [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe35c063a00 con 0x7fe3400bf350 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.528+0000 7fe352a00640 1 -- 172.21.15.107:0/286599173 <== osd.11 v2:172.21.15.132:6824/3905824310 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe35c063a00 con 0x7fe3400bf350 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.528+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 >> [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] conn(0x7fe3400bf350 msgr2=0x7fe3400c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.528+0000 7fe361200640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] conn(0x7fe3400bf350 0x7fe3400c1840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.529+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3400b3cb0 msgr2=0x7fe3400b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.529+0000 7fe361200640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe3400b3cb0 0x7fe3400b61c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0095f0 tx=0x7fe34c005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.529+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c10b0f0 msgr2=0x7fe35c19c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.529+0000 7fe361200640 1 --2- 172.21.15.107:0/286599173 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe35c10b0f0 0x7fe35c19c6c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe35c06c4d0 tx=0x7fe34800ec20 comp rx=0 tx=0).stop 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.529+0000 7fe35a800640 1 -- 172.21.15.107:0/286599173 reap_dead start 2024-07-26T12:46:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.529+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 shutdown_connections 2024-07-26T12:46:03.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.529+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 >> 172.21.15.107:0/286599173 conn(0x7fe35c101240 msgr2=0x7fe35c10a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:03.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.529+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 shutdown_connections 2024-07-26T12:46:03.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.530+0000 7fe361200640 1 -- 172.21.15.107:0/286599173 wait complete. 2024-07-26T12:46:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.926+0000 7fdad3000640 1 -- 172.21.15.107:0/1883407538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdacc10b120 msgr2=0x7fdacc10b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.926+0000 7fdad3000640 1 --2- 172.21.15.107:0/1883407538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdacc10b120 0x7fdacc10b530 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009860 tx=0x7fdab802a340 comp rx=0 tx=0).stop 2024-07-26T12:46:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.926+0000 7fdad3000640 1 -- 172.21.15.107:0/1883407538 shutdown_connections 2024-07-26T12:46:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.926+0000 7fdad3000640 1 --2- 172.21.15.107:0/1883407538 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdacc10ba70 0x7fdacc1122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.926+0000 7fdad3000640 1 --2- 172.21.15.107:0/1883407538 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdacc10b120 0x7fdacc10b530 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.926+0000 7fdad3000640 1 --2- 172.21.15.107:0/1883407538 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc1057e0 0x7fdacc105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.926+0000 7fdad3000640 1 -- 172.21.15.107:0/1883407538 >> 172.21.15.107:0/1883407538 conn(0x7fdacc1011f0 msgr2=0x7fdacc103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.926+0000 7fdad3000640 1 -- 172.21.15.107:0/1883407538 shutdown_connections 2024-07-26T12:46:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.926+0000 7fdad3000640 1 -- 172.21.15.107:0/1883407538 wait complete. 2024-07-26T12:46:03.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.928+0000 7fdad3000640 1 Processor -- start 2024-07-26T12:46:03.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.928+0000 7fdad3000640 1 -- start start 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.928+0000 7fdad3000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdacc1057e0 0x7fdacc19c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.928+0000 7fdad3000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdacc10b120 0x7fdacc19c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.928+0000 7fdad3000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc10ba70 0x7fdacc1a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.928+0000 7fdad3000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc115700 con 0x7fdacc10ba70 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.928+0000 7fdad3000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc115580 con 0x7fdacc10b120 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.928+0000 7fdad3000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc115880 con 0x7fdacc1057e0 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdacbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdacc1057e0 0x7fdacc19c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdad1000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc10ba70 0x7fdacc1a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdacbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdacc1057e0 0x7fdacc19c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53652/0 (socket says 172.21.15.107:53652) 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdacbe00640 1 -- 172.21.15.107:0/40783002 learned_addr learned my addr 172.21.15.107:0/40783002 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdacb400640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdacc10b120 0x7fdacc19c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdacbe00640 1 -- 172.21.15.107:0/40783002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdacc10b120 msgr2=0x7fdacc19c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdacbe00640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fdacc10b120 0x7fdacc19c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdacbe00640 1 -- 172.21.15.107:0/40783002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc10ba70 msgr2=0x7fdacc1a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdacbe00640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdacc10ba70 0x7fdacc1a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdacbe00640 1 -- 172.21.15.107:0/40783002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdacc1a20e0 con 0x7fdacc1057e0 2024-07-26T12:46:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.929+0000 7fdacbe00640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdacc1057e0 0x7fdacc19c170 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00e9c0 tx=0x7fdabc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.930+0000 7fdac8c00640 1 -- 172.21.15.107:0/40783002 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc00ccc0 con 0x7fdacc1057e0 2024-07-26T12:46:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.930+0000 7fdac8c00640 1 -- 172.21.15.107:0/40783002 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdabc0054d0 con 0x7fdacc1057e0 2024-07-26T12:46:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.930+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdacc1a23d0 con 0x7fdacc1057e0 2024-07-26T12:46:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.930+0000 7fdac8c00640 1 -- 172.21.15.107:0/40783002 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc010430 con 0x7fdacc1057e0 2024-07-26T12:46:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.930+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdacc1a8b30 con 0x7fdacc1057e0 2024-07-26T12:46:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.931+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdacc10a310 con 0x7fdacc1057e0 2024-07-26T12:46:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.933+0000 7fdac8c00640 1 -- 172.21.15.107:0/40783002 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fdabc004f80 con 0x7fdacc1057e0 2024-07-26T12:46:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.935+0000 7fdac8c00640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdab00b3cb0 0x7fdab00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:03.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.935+0000 7fdac8c00640 1 -- 172.21.15.107:0/40783002 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fdabc0e4d10 con 0x7fdacc1057e0 2024-07-26T12:46:03.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.935+0000 7fdacb400640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdab00b3cb0 0x7fdab00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:03.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.936+0000 7fdac8c00640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] conn(0x7fdab00bf350 0x7fdab00c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:03.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.936+0000 7fdac8c00640 1 -- 172.21.15.107:0/40783002 --> [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdabc015e80 con 0x7fdab00bf350 2024-07-26T12:46:03.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.936+0000 7fdac8c00640 1 -- 172.21.15.107:0/40783002 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fdabc0e50c0 con 0x7fdacc1057e0 2024-07-26T12:46:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.936+0000 7fdad1000640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] conn(0x7fdab00bf350 0x7fdab00c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.936+0000 7fdacb400640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdab00b3cb0 0x7fdab00b61c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdab8035040 tx=0x7fdab8030340 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.937+0000 7fdad1000640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] conn(0x7fdab00bf350 0x7fdab00c1840 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.938+0000 7fdac8c00640 1 -- 172.21.15.107:0/40783002 <== osd.2 v2:172.21.15.132:6800/28260456 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdabc015e80 con 0x7fdab00bf350 2024-07-26T12:46:03.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.952+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 --> [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdacc106a60 con 0x7fdab00bf350 2024-07-26T12:46:03.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.954+0000 7fdac8c00640 1 -- 172.21.15.107:0/40783002 <== osd.2 v2:172.21.15.132:6800/28260456 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdacc106a60 con 0x7fdab00bf350 2024-07-26T12:46:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.954+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 >> [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] conn(0x7fdab00bf350 msgr2=0x7fdab00c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.954+0000 7fdad3000640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] conn(0x7fdab00bf350 0x7fdab00c1840 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.954+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdab00b3cb0 msgr2=0x7fdab00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.954+0000 7fdad3000640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fdab00b3cb0 0x7fdab00b61c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdab8035040 tx=0x7fdab8030340 comp rx=0 tx=0).stop 2024-07-26T12:46:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.954+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdacc1057e0 msgr2=0x7fdacc19c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.954+0000 7fdad3000640 1 --2- 172.21.15.107:0/40783002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdacc1057e0 0x7fdacc19c170 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00e9c0 tx=0x7fdabc00ee80 comp rx=0 tx=0).stop 2024-07-26T12:46:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.954+0000 7fdad1000640 1 -- 172.21.15.107:0/40783002 reap_dead start 2024-07-26T12:46:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.954+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 shutdown_connections 2024-07-26T12:46:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.954+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 >> 172.21.15.107:0/40783002 conn(0x7fdacc1011f0 msgr2=0x7fdacc1098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.955+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 shutdown_connections 2024-07-26T12:46:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:03.955+0000 7fdad3000640 1 -- 172.21.15.107:0/40783002 wait complete. 2024-07-26T12:46:03.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:03 smithi107 ceph-mon[32659]: pgmap v392: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:03 smithi112 ceph-mon[37501]: pgmap v392: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.209+0000 7fd2c6c00640 1 -- 172.21.15.107:0/633052002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2c010ba70 msgr2=0x7fd2c01122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.209+0000 7fd2c6c00640 1 --2- 172.21.15.107:0/633052002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2c010ba70 0x7fd2c01122a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009be0 tx=0x7fd2b802a6c0 comp rx=0 tx=0).stop 2024-07-26T12:46:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.209+0000 7fd2c6c00640 1 -- 172.21.15.107:0/633052002 shutdown_connections 2024-07-26T12:46:04.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.209+0000 7fd2c6c00640 1 --2- 172.21.15.107:0/633052002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2c010ba70 0x7fd2c01122a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.209+0000 7fd2c6c00640 1 --2- 172.21.15.107:0/633052002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c010b120 0x7fd2c010b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.209+0000 7fd2c6c00640 1 --2- 172.21.15.107:0/633052002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2c01057e0 0x7fd2c0105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.209+0000 7fd2c6c00640 1 -- 172.21.15.107:0/633052002 >> 172.21.15.107:0/633052002 conn(0x7fd2c01011f0 msgr2=0x7fd2c0103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.209+0000 7fd2c6c00640 1 -- 172.21.15.107:0/633052002 shutdown_connections 2024-07-26T12:46:04.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.209+0000 7fd2c6c00640 1 -- 172.21.15.107:0/633052002 wait complete. 2024-07-26T12:46:04.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.210+0000 7fd2c6c00640 1 Processor -- start 2024-07-26T12:46:04.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2c6c00640 1 -- start start 2024-07-26T12:46:04.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2c6c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c01057e0 0x7fd2c019c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:04.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2c6c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2c010b120 0x7fd2c019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:04.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2c6c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2c010ba70 0x7fd2c01a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:04.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2c6c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c0115740 con 0x7fd2c01057e0 2024-07-26T12:46:04.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2c6c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c01155c0 con 0x7fd2c010b120 2024-07-26T12:46:04.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2c6c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c01158c0 con 0x7fd2c010ba70 2024-07-26T12:46:04.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2bfe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c01057e0 0x7fd2c019c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2bfe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c01057e0 0x7fd2c019c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46454/0 (socket says 172.21.15.107:46454) 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2bfe00640 1 -- 172.21.15.107:0/2918314989 learned_addr learned my addr 172.21.15.107:0/2918314989 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.211+0000 7fd2bf400640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2c010b120 0x7fd2c019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.212+0000 7fd2c4c00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2c010ba70 0x7fd2c01a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.212+0000 7fd2bfe00640 1 -- 172.21.15.107:0/2918314989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2c010ba70 msgr2=0x7fd2c01a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.212+0000 7fd2bfe00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd2c010ba70 0x7fd2c01a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.212+0000 7fd2bfe00640 1 -- 172.21.15.107:0/2918314989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2c010b120 msgr2=0x7fd2c019c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.212+0000 7fd2bfe00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2c010b120 0x7fd2c019c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.212+0000 7fd2bfe00640 1 -- 172.21.15.107:0/2918314989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c01a2020 con 0x7fd2c01057e0 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.212+0000 7fd2bf400640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd2c010b120 0x7fd2c019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.212+0000 7fd2bfe00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c01057e0 0x7fd2c019c0b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd2b000e9f0 tx=0x7fd2b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.213+0000 7fd2bcc00640 1 -- 172.21.15.107:0/2918314989 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b000cd80 con 0x7fd2c01057e0 2024-07-26T12:46:04.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.213+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c01a2310 con 0x7fd2c01057e0 2024-07-26T12:46:04.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.213+0000 7fd2bcc00640 1 -- 172.21.15.107:0/2918314989 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2b00054d0 con 0x7fd2c01057e0 2024-07-26T12:46:04.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.213+0000 7fd2bcc00640 1 -- 172.21.15.107:0/2918314989 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0010690 con 0x7fd2c01057e0 2024-07-26T12:46:04.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.213+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c01a8a70 con 0x7fd2c01057e0 2024-07-26T12:46:04.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.214+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd2c0063580 con 0x7fd2c01057e0 2024-07-26T12:46:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.216+0000 7fd2bcc00640 1 -- 172.21.15.107:0/2918314989 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fd2b0004e10 con 0x7fd2c01057e0 2024-07-26T12:46:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.218+0000 7fd2bcc00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd2a40b3cb0 0x7fd2a40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.218+0000 7fd2bcc00640 1 -- 172.21.15.107:0/2918314989 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd2b00e4b10 con 0x7fd2c01057e0 2024-07-26T12:46:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.218+0000 7fd2bf400640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd2a40b3cb0 0x7fd2a40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.218+0000 7fd2bcc00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] conn(0x7fd2a40bf350 0x7fd2a40c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.218+0000 7fd2bcc00640 1 -- 172.21.15.107:0/2918314989 --> [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2b0015e80 con 0x7fd2a40bf350 2024-07-26T12:46:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.218+0000 7fd2bcc00640 1 -- 172.21.15.107:0/2918314989 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd2b00e4ec0 con 0x7fd2c01057e0 2024-07-26T12:46:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.219+0000 7fd2c4c00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] conn(0x7fd2a40bf350 0x7fd2a40c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.219+0000 7fd2bf400640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd2a40b3cb0 0x7fd2a40b61c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0096c0 tx=0x7fd2ac009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:04.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.220+0000 7fd2c4c00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] conn(0x7fd2a40bf350 0x7fd2a40c1840 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:04.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.221+0000 7fd2bcc00640 1 -- 172.21.15.107:0/2918314989 <== osd.6 v2:172.21.15.107:6818/2132739845 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2b0015e80 con 0x7fd2a40bf350 2024-07-26T12:46:04.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.236+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 --> [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd2c0063a00 con 0x7fd2a40bf350 2024-07-26T12:46:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.237+0000 7fd2bcc00640 1 -- 172.21.15.107:0/2918314989 <== osd.6 v2:172.21.15.107:6818/2132739845 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd2c0063a00 con 0x7fd2a40bf350 2024-07-26T12:46:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.238+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] conn(0x7fd2a40bf350 msgr2=0x7fd2a40c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:04.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.238+0000 7fd2c6c00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] conn(0x7fd2a40bf350 0x7fd2a40c1840 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.238+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd2a40b3cb0 msgr2=0x7fd2a40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.238+0000 7fd2c6c00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd2a40b3cb0 0x7fd2a40b61c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac0096c0 tx=0x7fd2ac009260 comp rx=0 tx=0).stop 2024-07-26T12:46:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.238+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c01057e0 msgr2=0x7fd2c019c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.238+0000 7fd2c6c00640 1 --2- 172.21.15.107:0/2918314989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2c01057e0 0x7fd2c019c0b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd2b000e9f0 tx=0x7fd2b000edf0 comp rx=0 tx=0).stop 2024-07-26T12:46:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.238+0000 7fd2c4c00640 1 -- 172.21.15.107:0/2918314989 reap_dead start 2024-07-26T12:46:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.238+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 shutdown_connections 2024-07-26T12:46:04.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.238+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 >> 172.21.15.107:0/2918314989 conn(0x7fd2c01011f0 msgr2=0x7fd2c0108620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.239+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 shutdown_connections 2024-07-26T12:46:04.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.239+0000 7fd2c6c00640 1 -- 172.21.15.107:0/2918314989 wait complete. 2024-07-26T12:46:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:03 smithi132 ceph-mon[37580]: pgmap v392: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:05.416 INFO:teuthology.orchestra.run.smithi107.stdout:206158430278 2024-07-26T12:46:05.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-07-26T12:46:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.682+0000 7fb619c00640 1 -- 172.21.15.107:0/1531372180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb61410b860 msgr2=0x7fb614112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.682+0000 7fb619c00640 1 --2- 172.21.15.107:0/1531372180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb61410b860 0x7fb614112090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009ca0 tx=0x7fb60c02a8b0 comp rx=0 tx=0).stop 2024-07-26T12:46:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.683+0000 7fb619c00640 1 -- 172.21.15.107:0/1531372180 shutdown_connections 2024-07-26T12:46:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.683+0000 7fb619c00640 1 --2- 172.21.15.107:0/1531372180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb61410b860 0x7fb614112090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.683+0000 7fb619c00640 1 --2- 172.21.15.107:0/1531372180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb61410af10 0x7fb61410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.683+0000 7fb619c00640 1 --2- 172.21.15.107:0/1531372180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6141055d0 0x7fb6141059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.683+0000 7fb619c00640 1 -- 172.21.15.107:0/1531372180 >> 172.21.15.107:0/1531372180 conn(0x7fb614101040 msgr2=0x7fb614103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.683+0000 7fb619c00640 1 -- 172.21.15.107:0/1531372180 shutdown_connections 2024-07-26T12:46:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.683+0000 7fb619c00640 1 -- 172.21.15.107:0/1531372180 wait complete. 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.684+0000 7fb619c00640 1 Processor -- start 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb619c00640 1 -- start start 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb619c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6141055d0 0x7fb61419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb619c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb61410af10 0x7fb61419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb619c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb61410b860 0x7fb6141a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb619c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb614115510 con 0x7fb61410af10 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb619c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb614115390 con 0x7fb6141055d0 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb619c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb614115690 con 0x7fb61410b860 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb619200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb61410b860 0x7fb6141a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb60b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb61410af10 0x7fb61419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb619200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb61410b860 0x7fb6141a16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53684/0 (socket says 172.21.15.107:53684) 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb60b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb61410af10 0x7fb61419c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46478/0 (socket says 172.21.15.107:46478) 2024-07-26T12:46:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb619200640 1 -- 172.21.15.107:0/4157052645 learned_addr learned my addr 172.21.15.107:0/4157052645 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.685+0000 7fb613e00640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6141055d0 0x7fb61419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.686+0000 7fb60b400640 1 -- 172.21.15.107:0/4157052645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb61410b860 msgr2=0x7fb6141a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.686+0000 7fb60b400640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb61410b860 0x7fb6141a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.686+0000 7fb60b400640 1 -- 172.21.15.107:0/4157052645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6141055d0 msgr2=0x7fb61419bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.686+0000 7fb60b400640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb6141055d0 0x7fb61419bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.686+0000 7fb60b400640 1 -- 172.21.15.107:0/4157052645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6141a1e10 con 0x7fb61410af10 2024-07-26T12:46:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.686+0000 7fb60b400640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb61410af10 0x7fb61419c440 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb61406c270 tx=0x7fb5fc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.686+0000 7fb611600640 1 -- 172.21.15.107:0/4157052645 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc00cd00 con 0x7fb61410af10 2024-07-26T12:46:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.687+0000 7fb611600640 1 -- 172.21.15.107:0/4157052645 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5fc0035f0 con 0x7fb61410af10 2024-07-26T12:46:05.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.687+0000 7fb611600640 1 -- 172.21.15.107:0/4157052645 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc010690 con 0x7fb61410af10 2024-07-26T12:46:05.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:05 smithi107 ceph-mon[32659]: pgmap v393: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T12:46:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.687+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6141a2100 con 0x7fb61410af10 2024-07-26T12:46:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.687+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6141a8860 con 0x7fb61410af10 2024-07-26T12:46:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.688+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb614063540 con 0x7fb61410af10 2024-07-26T12:46:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.690+0000 7fb611600640 1 -- 172.21.15.107:0/4157052645 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fb5fc0037a0 con 0x7fb61410af10 2024-07-26T12:46:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.692+0000 7fb611600640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb5f80b3cb0 0x7fb5f80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.692+0000 7fb611600640 1 -- 172.21.15.107:0/4157052645 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb5fc0e48e0 con 0x7fb61410af10 2024-07-26T12:46:05.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.692+0000 7fb613e00640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb5f80b3cb0 0x7fb5f80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.692+0000 7fb611600640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] conn(0x7fb5f80bf350 0x7fb5f80c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.693+0000 7fb611600640 1 -- 172.21.15.107:0/4157052645 --> [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5fc022e50 con 0x7fb5f80bf350 2024-07-26T12:46:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.693+0000 7fb611600640 1 -- 172.21.15.107:0/4157052645 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fb5fc0e4d30 con 0x7fb61410af10 2024-07-26T12:46:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.693+0000 7fb619200640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] conn(0x7fb5f80bf350 0x7fb5f80c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.693+0000 7fb613e00640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb5f80b3cb0 0x7fb5f80b61c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb6040095f0 tx=0x7fb604005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.694+0000 7fb619200640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] conn(0x7fb5f80bf350 0x7fb5f80c1840 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.695+0000 7fb611600640 1 -- 172.21.15.107:0/4157052645 <== osd.5 v2:172.21.15.132:6808/1054197844 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5fc022e50 con 0x7fb5f80bf350 2024-07-26T12:46:05.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.710+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 --> [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb6140639c0 con 0x7fb5f80bf350 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.711+0000 7fb611600640 1 -- 172.21.15.107:0/4157052645 <== osd.5 v2:172.21.15.132:6808/1054197844 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb6140639c0 con 0x7fb5f80bf350 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 >> [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] conn(0x7fb5f80bf350 msgr2=0x7fb5f80c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619c00640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] conn(0x7fb5f80bf350 0x7fb5f80c1840 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb5f80b3cb0 msgr2=0x7fb5f80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619c00640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb5f80b3cb0 0x7fb5f80b61c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb6040095f0 tx=0x7fb604005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb61410af10 msgr2=0x7fb61419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619c00640 1 --2- 172.21.15.107:0/4157052645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb61410af10 0x7fb61419c440 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb61406c270 tx=0x7fb5fc00ee50 comp rx=0 tx=0).stop 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619200640 1 -- 172.21.15.107:0/4157052645 reap_dead start 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 shutdown_connections 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 >> 172.21.15.107:0/4157052645 conn(0x7fb614101040 msgr2=0x7fb614109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 shutdown_connections 2024-07-26T12:46:05.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:04.712+0000 7fb619c00640 1 -- 172.21.15.107:0/4157052645 wait complete. 2024-07-26T12:46:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:05 smithi112 ceph-mon[37501]: pgmap v393: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T12:46:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:05 smithi132 ceph-mon[37580]: pgmap v393: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T12:46:06.738 INFO:teuthology.orchestra.run.smithi107.stdout:231928234049 2024-07-26T12:46:06.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-07-26T12:46:07.421 INFO:teuthology.orchestra.run.smithi107.stdout:184683593805 2024-07-26T12:46:07.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-07-26T12:46:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:07 smithi107 ceph-mon[32659]: pgmap v394: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:46:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:07 smithi112 ceph-mon[37501]: pgmap v394: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:46:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:07 smithi132 ceph-mon[37580]: pgmap v394: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:46:08.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.633+0000 7f166e000640 1 -- 172.21.15.107:0/2077124019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16681141a0 msgr2=0x7f16681165c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:08.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.633+0000 7f166e000640 1 --2- 172.21.15.107:0/2077124019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16681141a0 0x7f16681165c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1660009ca0 tx=0x7f166002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:46:08.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.633+0000 7f166e000640 1 -- 172.21.15.107:0/2077124019 shutdown_connections 2024-07-26T12:46:08.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.633+0000 7f166e000640 1 --2- 172.21.15.107:0/2077124019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16681141a0 0x7f16681165c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:08.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.633+0000 7f166e000640 1 --2- 172.21.15.107:0/2077124019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1668077cc0 0x7f1668113c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:08.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.633+0000 7f166e000640 1 --2- 172.21.15.107:0/2077124019 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1668077370 0x7f1668077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:08.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.633+0000 7f166e000640 1 -- 172.21.15.107:0/2077124019 >> 172.21.15.107:0/2077124019 conn(0x7f16681011f0 msgr2=0x7f1668103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.633+0000 7f166e000640 1 -- 172.21.15.107:0/2077124019 shutdown_connections 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.633+0000 7f166e000640 1 -- 172.21.15.107:0/2077124019 wait complete. 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.634+0000 7f166e000640 1 Processor -- start 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.634+0000 7f166e000640 1 -- start start 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f166e000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1668077370 0x7f16681a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f166e000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1668077cc0 0x7f16681a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f166e000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16681141a0 0x7f16681a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f166e000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1668119b50 con 0x7f16681141a0 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f166e000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f16681199d0 con 0x7f1668077370 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f166e000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1668119cd0 con 0x7f1668077cc0 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1667400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1668077370 0x7f16681a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:08.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1667e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16681141a0 0x7f16681a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1667400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1668077370 0x7f16681a04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:53632/0 (socket says 172.21.15.107:53632) 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1667400640 1 -- 172.21.15.107:0/2275224786 learned_addr learned my addr 172.21.15.107:0/2275224786 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1666a00640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1668077cc0 0x7f16681a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1667400640 1 -- 172.21.15.107:0/2275224786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1668077cc0 msgr2=0x7f16681a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1667400640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1668077cc0 0x7f16681a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1667400640 1 -- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16681141a0 msgr2=0x7f16681a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1667400640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16681141a0 0x7f16681a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1667400640 1 -- 172.21.15.107:0/2275224786 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16681a6450 con 0x7f1668077370 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.635+0000 7f1666a00640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1668077cc0 0x7f16681a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.636+0000 7f1667400640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1668077370 0x7f16681a04e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f165800e9f0 tx=0x7f165800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.636+0000 7f165fe00640 1 -- 172.21.15.107:0/2275224786 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165800cd80 con 0x7f1668077370 2024-07-26T12:46:08.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.636+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16681a6740 con 0x7f1668077370 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.636+0000 7f165fe00640 1 -- 172.21.15.107:0/2275224786 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16580108e0 con 0x7f1668077370 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.636+0000 7f165fe00640 1 -- 172.21.15.107:0/2275224786 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658018980 con 0x7f1668077370 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.636+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16681acea0 con 0x7f1668077370 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.637+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f16681165d0 con 0x7f1668077370 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.640+0000 7f165fe00640 1 -- 172.21.15.107:0/2275224786 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f1658010430 con 0x7f1668077370 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.641+0000 7f165fe00640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f164c0b3cb0 0x7f164c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.642+0000 7f165fe00640 1 -- 172.21.15.107:0/2275224786 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f16580e4d30 con 0x7f1668077370 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.642+0000 7f1666a00640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f164c0b3cb0 0x7f164c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.642+0000 7f165fe00640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] conn(0x7f164c0bf350 0x7f164c0c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.642+0000 7f165fe00640 1 -- 172.21.15.107:0/2275224786 --> [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f164c0c1f10 con 0x7f164c0bf350 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.642+0000 7f165fe00640 1 -- 172.21.15.107:0/2275224786 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f16580e50e0 con 0x7f1668077370 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.642+0000 7f1667e00640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] conn(0x7f164c0bf350 0x7f164c0c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:08.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.643+0000 7f1666a00640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f164c0b3cb0 0x7f164c0b61c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1668079c10 tx=0x7f1654005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:08.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.643+0000 7f1667e00640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] conn(0x7f164c0bf350 0x7f164c0c1840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:08.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.644+0000 7f165fe00640 1 -- 172.21.15.107:0/2275224786 <== osd.9 v2:172.21.15.107:6826/3840896542 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f164c0c1f10 con 0x7f164c0bf350 2024-07-26T12:46:08.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.659+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 --> [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1668078880 con 0x7f164c0bf350 2024-07-26T12:46:08.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.660+0000 7f165fe00640 1 -- 172.21.15.107:0/2275224786 <== osd.9 v2:172.21.15.107:6826/3840896542 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1668078880 con 0x7f164c0bf350 2024-07-26T12:46:08.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.660+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] conn(0x7f164c0bf350 msgr2=0x7f164c0c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:08.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.660+0000 7f166e000640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] conn(0x7f164c0bf350 0x7f164c0c1840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:08.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.661+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f164c0b3cb0 msgr2=0x7f164c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:08.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.661+0000 7f166e000640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f164c0b3cb0 0x7f164c0b61c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1668079c10 tx=0x7f1654005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:08.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.661+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1668077370 msgr2=0x7f16681a04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:08.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.661+0000 7f166e000640 1 --2- 172.21.15.107:0/2275224786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f1668077370 0x7f16681a04e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f165800e9f0 tx=0x7f165800eeb0 comp rx=0 tx=0).stop 2024-07-26T12:46:08.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.661+0000 7f1667e00640 1 -- 172.21.15.107:0/2275224786 reap_dead start 2024-07-26T12:46:08.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.661+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 shutdown_connections 2024-07-26T12:46:08.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.661+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 >> 172.21.15.107:0/2275224786 conn(0x7f16681011f0 msgr2=0x7f1668103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:08.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.661+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 shutdown_connections 2024-07-26T12:46:08.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:08.661+0000 7f166e000640 1 -- 172.21.15.107:0/2275224786 wait complete. 2024-07-26T12:46:09.153 INFO:teuthology.orchestra.run.smithi107.stdout:279172874295 2024-07-26T12:46:09.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-07-26T12:46:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.773+0000 7f31ffe00640 1 -- 172.21.15.107:0/3045837253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f320010af10 msgr2=0x7f320010b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.773+0000 7f31ffe00640 1 --2- 172.21.15.107:0/3045837253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f320010af10 0x7f320010b320 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f31ec009900 tx=0x7f31ec02a410 comp rx=0 tx=0).stop 2024-07-26T12:46:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.774+0000 7f31ffe00640 1 -- 172.21.15.107:0/3045837253 shutdown_connections 2024-07-26T12:46:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.774+0000 7f31ffe00640 1 --2- 172.21.15.107:0/3045837253 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f320010b860 0x7f3200112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.774+0000 7f31ffe00640 1 --2- 172.21.15.107:0/3045837253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f320010af10 0x7f320010b320 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.774+0000 7f31ffe00640 1 --2- 172.21.15.107:0/3045837253 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32001055d0 0x7f32001059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.774+0000 7f31ffe00640 1 -- 172.21.15.107:0/3045837253 >> 172.21.15.107:0/3045837253 conn(0x7f3200101040 msgr2=0x7f3200103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:09.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.774+0000 7f31ffe00640 1 -- 172.21.15.107:0/3045837253 shutdown_connections 2024-07-26T12:46:09.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.774+0000 7f31ffe00640 1 -- 172.21.15.107:0/3045837253 wait complete. 2024-07-26T12:46:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.775+0000 7f31ffe00640 1 Processor -- start 2024-07-26T12:46:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.775+0000 7f31ffe00640 1 -- start start 2024-07-26T12:46:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31ffe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32001055d0 0x7f320019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31ffe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f320010af10 0x7f320019c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31ffe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f320010b860 0x7f32001a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31ffe00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f32001154f0 con 0x7f32001055d0 2024-07-26T12:46:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31ffe00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3200115370 con 0x7f320010af10 2024-07-26T12:46:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31ffe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3200115670 con 0x7f320010b860 2024-07-26T12:46:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31fea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32001055d0 0x7f320019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31fe000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f320010af10 0x7f320019c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31fea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32001055d0 0x7f320019bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46494/0 (socket says 172.21.15.107:46494) 2024-07-26T12:46:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31fea00640 1 -- 172.21.15.107:0/3521624019 learned_addr learned my addr 172.21.15.107:0/3521624019 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:09.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.776+0000 7f31ff400640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f320010b860 0x7f32001a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31fe000640 1 -- 172.21.15.107:0/3521624019 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f320010b860 msgr2=0x7f32001a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31fe000640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f320010b860 0x7f32001a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31fe000640 1 -- 172.21.15.107:0/3521624019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32001055d0 msgr2=0x7f320019bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31fe000640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32001055d0 0x7f320019bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31fe000640 1 -- 172.21.15.107:0/3521624019 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32001a1eb0 con 0x7f320010af10 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31fea00640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32001055d0 0x7f320019bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31fe000640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f320010af10 0x7f320019c4e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f31ec02a930 tx=0x7f31ec004060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31f7400640 1 -- 172.21.15.107:0/3521624019 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec03f070 con 0x7f320010af10 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32001a2140 con 0x7f320010af10 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31f7400640 1 -- 172.21.15.107:0/3521624019 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31ec003ca0 con 0x7f320010af10 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31f7400640 1 -- 172.21.15.107:0/3521624019 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ec035070 con 0x7f320010af10 2024-07-26T12:46:09.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.777+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32001a8900 con 0x7f320010af10 2024-07-26T12:46:09.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.778+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3200063540 con 0x7f320010af10 2024-07-26T12:46:09.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.781+0000 7f31f7400640 1 -- 172.21.15.107:0/3521624019 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f31ec03b460 con 0x7f320010af10 2024-07-26T12:46:09.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.783+0000 7f31f7400640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31e40b3cb0 0x7f31e40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:09.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.783+0000 7f31f7400640 1 -- 172.21.15.107:0/3521624019 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f31ec104d80 con 0x7f320010af10 2024-07-26T12:46:09.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.783+0000 7f31fea00640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31e40b3cb0 0x7f31e40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:09.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.783+0000 7f31f7400640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] conn(0x7f31e40bf350 0x7f31e40c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:09.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.783+0000 7f31f7400640 1 -- 172.21.15.107:0/3521624019 --> [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31ec032970 con 0x7f31e40bf350 2024-07-26T12:46:09.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.783+0000 7f31f7400640 1 -- 172.21.15.107:0/3521624019 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f31ec105130 con 0x7f320010af10 2024-07-26T12:46:09.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.784+0000 7f31ff400640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] conn(0x7f31e40bf350 0x7f31e40c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:09.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.784+0000 7f31fea00640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31e40b3cb0 0x7f31e40b61c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f31f00104e0 tx=0x7f31f000b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:09.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.784+0000 7f31ff400640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] conn(0x7f31e40bf350 0x7f31e40c1840 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:09.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.785+0000 7f31f7400640 1 -- 172.21.15.107:0/3521624019 <== osd.4 v2:172.21.15.112:6808/95536584 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31ec032970 con 0x7f31e40bf350 2024-07-26T12:46:09.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.800+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 --> [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f32000639c0 con 0x7f31e40bf350 2024-07-26T12:46:09.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.801+0000 7f31f7400640 1 -- 172.21.15.107:0/3521624019 <== osd.4 v2:172.21.15.112:6808/95536584 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f32000639c0 con 0x7f31e40bf350 2024-07-26T12:46:09.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 >> [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] conn(0x7f31e40bf350 msgr2=0x7f31e40c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:09.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ffe00640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] conn(0x7f31e40bf350 0x7f31e40c1840 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31e40b3cb0 msgr2=0x7f31e40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ffe00640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31e40b3cb0 0x7f31e40b61c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f31f00104e0 tx=0x7f31f000b410 comp rx=0 tx=0).stop 2024-07-26T12:46:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f320010af10 msgr2=0x7f320019c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ffe00640 1 --2- 172.21.15.107:0/3521624019 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f320010af10 0x7f320019c4e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f31ec02a930 tx=0x7f31ec004060 comp rx=0 tx=0).stop 2024-07-26T12:46:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ff400640 1 -- 172.21.15.107:0/3521624019 reap_dead start 2024-07-26T12:46:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 shutdown_connections 2024-07-26T12:46:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 >> 172.21.15.107:0/3521624019 conn(0x7f3200101040 msgr2=0x7f3200103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 shutdown_connections 2024-07-26T12:46:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:09.802+0000 7f31ffe00640 1 -- 172.21.15.107:0/3521624019 wait complete. 2024-07-26T12:46:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:09 smithi112 ceph-mon[37501]: pgmap v395: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-07-26T12:46:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:09 smithi107 ceph-mon[32659]: pgmap v395: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-07-26T12:46:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:09 smithi132 ceph-mon[37580]: pgmap v395: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-07-26T12:46:10.441 INFO:teuthology.orchestra.run.smithi107.stdout:304942678066 2024-07-26T12:46:10.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-07-26T12:46:11.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:10 smithi107 ceph-mon[32659]: pgmap v396: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:46:11.066 INFO:teuthology.orchestra.run.smithi107.stdout:137438953561 2024-07-26T12:46:11.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-07-26T12:46:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:10 smithi112 ceph-mon[37501]: pgmap v396: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:46:11.233 INFO:teuthology.orchestra.run.smithi107.stdout:158913790036 2024-07-26T12:46:11.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-07-26T12:46:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:10 smithi132 ceph-mon[37580]: pgmap v396: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:46:11.582 INFO:teuthology.orchestra.run.smithi107.stdout:81604378722 2024-07-26T12:46:11.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-07-26T12:46:12.096 INFO:teuthology.orchestra.run.smithi107.stdout:257698037820 2024-07-26T12:46:12.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-07-26T12:46:12.123 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:12.192 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:12.209 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:12.239 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:12.264 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:12.308 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:12.324 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:12.333 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:12.335 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:12.338 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:13.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:13 smithi107 ceph-mon[32659]: pgmap v397: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:46:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:13 smithi132 ceph-mon[37580]: pgmap v397: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:46:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:13 smithi112 ceph-mon[37501]: pgmap v397: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:46:15.245 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:15 smithi107 ceph-mon[32659]: pgmap v398: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:46:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:15 smithi132 ceph-mon[37580]: pgmap v398: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:46:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:15 smithi112 ceph-mon[37501]: pgmap v398: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:46:18.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:17 smithi107 ceph-mon[32659]: pgmap v399: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:17 smithi132 ceph-mon[37580]: pgmap v399: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:17 smithi112 ceph-mon[37501]: pgmap v399: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:46:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:19 smithi107 ceph-mon[32659]: pgmap v400: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:46:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:46:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:19 smithi112 ceph-mon[37501]: pgmap v400: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:46:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:46:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:19 smithi132 ceph-mon[37580]: pgmap v400: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:46:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:46:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:21 smithi107 ceph-mon[32659]: pgmap v401: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:46:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:21 smithi112 ceph-mon[37501]: pgmap v401: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:46:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:21 smithi132 ceph-mon[37580]: pgmap v401: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:46:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:23 smithi112 ceph-mon[37501]: pgmap v402: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:46:24.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:23 smithi107 ceph-mon[32659]: pgmap v402: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:46:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:23 smithi132 ceph-mon[37580]: pgmap v402: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:46:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.787+0000 7f16eea00640 1 -- 172.21.15.107:0/572215977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16e8113cd0 msgr2=0x7f16e81161a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.787+0000 7f16eea00640 1 --2- 172.21.15.107:0/572215977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16e8113cd0 0x7f16e81161a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009be0 tx=0x7f16e002a6d0 comp rx=0 tx=0).stop 2024-07-26T12:46:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.787+0000 7f16eea00640 1 -- 172.21.15.107:0/572215977 shutdown_connections 2024-07-26T12:46:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.787+0000 7f16eea00640 1 --2- 172.21.15.107:0/572215977 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16e8113cd0 0x7f16e81161a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.787+0000 7f16eea00640 1 --2- 172.21.15.107:0/572215977 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16e8075800 0x7f16e8075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.787+0000 7f16eea00640 1 --2- 172.21.15.107:0/572215977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16e8076db0 0x7f16e80751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.787+0000 7f16eea00640 1 -- 172.21.15.107:0/572215977 >> 172.21.15.107:0/572215977 conn(0x7f16e80fee90 msgr2=0x7f16e81012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.787+0000 7f16eea00640 1 -- 172.21.15.107:0/572215977 shutdown_connections 2024-07-26T12:46:25.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.788+0000 7f16eea00640 1 -- 172.21.15.107:0/572215977 wait complete. 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.789+0000 7f16eea00640 1 Processor -- start 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.789+0000 7f16eea00640 1 -- start start 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.789+0000 7f16eea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16e8075800 0x7f16e81a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16e8076db0 0x7f16e81a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16e8113cd0 0x7f16e81a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eea00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f16e8119910 con 0x7f16e8076db0 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eea00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f16e8119790 con 0x7f16e8113cd0 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eea00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f16e8119a90 con 0x7f16e8075800 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16e7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16e8076db0 0x7f16e81a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eca00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16e8113cd0 0x7f16e81a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16e7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16e8076db0 0x7f16e81a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47162/0 (socket says 172.21.15.107:47162) 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16e7400640 1 -- 172.21.15.107:0/188455031 learned_addr learned my addr 172.21.15.107:0/188455031 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:25.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16e7e00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16e8075800 0x7f16e81a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:25.362 INFO:teuthology.orchestra.run.smithi107.stdout:304942678070 2024-07-26T12:46:25.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eca00640 1 -- 172.21.15.107:0/188455031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16e8075800 msgr2=0x7f16e81a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:25.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eca00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16e8075800 0x7f16e81a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:25.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eca00640 1 -- 172.21.15.107:0/188455031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16e8076db0 msgr2=0x7f16e81a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:25.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eca00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16e8076db0 0x7f16e81a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:25.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.791+0000 7f16eca00640 1 -- 172.21.15.107:0/188455031 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e81a6270 con 0x7f16e8113cd0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.792+0000 7f16eca00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16e8113cd0 0x7f16e81a5b40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009b30 tx=0x7f16e0003b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.792+0000 7f16e4c00640 1 -- 172.21.15.107:0/188455031 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e0041020 con 0x7f16e8113cd0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.792+0000 7f16e4c00640 1 -- 172.21.15.107:0/188455031 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16e00318f0 con 0x7f16e8113cd0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.792+0000 7f16eea00640 1 -- 172.21.15.107:0/188455031 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16e81a6500 con 0x7f16e8113cd0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.792+0000 7f16e4c00640 1 -- 172.21.15.107:0/188455031 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e0036040 con 0x7f16e8113cd0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.792+0000 7f16eea00640 1 -- 172.21.15.107:0/188455031 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16e81accc0 con 0x7f16e8113cd0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.794+0000 7f16eea00640 1 -- 172.21.15.107:0/188455031 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16e80766f0 con 0x7f16e8113cd0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.797+0000 7f16e4c00640 1 -- 172.21.15.107:0/188455031 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f16e0044020 con 0x7f16e8113cd0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.798+0000 7f16e4c00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f16cc0b3cb0 0x7f16cc0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.799+0000 7f16e4c00640 1 -- 172.21.15.107:0/188455031 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f16e0105250 con 0x7f16e8113cd0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.799+0000 7f16e7e00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f16cc0b3cb0 0x7f16cc0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.800+0000 7f16e7e00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f16cc0b3cb0 0x7f16cc0b61c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f16d80095f0 tx=0x7f16d8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:25.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:24.800+0000 7f16e4c00640 1 -- 172.21.15.107:0/188455031 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f16e004b020 con 0x7f16e8113cd0 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.004+0000 7f16eea00640 1 -- 172.21.15.107:0/188455031 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f16e8112a00 con 0x7f16e8113cd0 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.005+0000 7f16e4c00640 1 -- 172.21.15.107:0/188455031 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f16e00cc350 con 0x7f16e8113cd0 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.008+0000 7f16eea00640 1 -- 172.21.15.107:0/188455031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f16cc0b3cb0 msgr2=0x7f16cc0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.008+0000 7f16eea00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f16cc0b3cb0 0x7f16cc0b61c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f16d80095f0 tx=0x7f16d8005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.008+0000 7f16eea00640 1 -- 172.21.15.107:0/188455031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16e8113cd0 msgr2=0x7f16e81a5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.008+0000 7f16eea00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16e8113cd0 0x7f16e81a5b40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009b30 tx=0x7f16e0003b80 comp rx=0 tx=0).stop 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.008+0000 7f16eea00640 1 -- 172.21.15.107:0/188455031 shutdown_connections 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.008+0000 7f16eea00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f16cc0b3cb0 0x7f16cc0b61c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.008+0000 7f16eea00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f16e8113cd0 0x7f16e81a5b40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.008+0000 7f16eea00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16e8076db0 0x7f16e81a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.008+0000 7f16eea00640 1 --2- 172.21.15.107:0/188455031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f16e8075800 0x7f16e81a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.008+0000 7f16eea00640 1 -- 172.21.15.107:0/188455031 >> 172.21.15.107:0/188455031 conn(0x7f16e80fee90 msgr2=0x7f16e810fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.009+0000 7f16eea00640 1 -- 172.21.15.107:0/188455031 shutdown_connections 2024-07-26T12:46:25.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:25.009+0000 7f16eea00640 1 -- 172.21.15.107:0/188455031 wait complete. 2024-07-26T12:46:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:25 smithi107 ceph-mon[32659]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:46:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:25 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/188455031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T12:46:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:25 smithi112 ceph-mon[37501]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:46:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:25 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/188455031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T12:46:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:25 smithi132 ceph-mon[37580]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:46:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:25 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/188455031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T12:46:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.787+0000 7f0074a00640 1 -- 172.21.15.107:0/45326592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0070102490 msgr2=0x7f00701028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.787+0000 7f0074a00640 1 --2- 172.21.15.107:0/45326592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0070102490 0x7f00701028a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0060009900 tx=0x7f006002a400 comp rx=0 tx=0).stop 2024-07-26T12:46:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.787+0000 7f0074a00640 1 -- 172.21.15.107:0/45326592 shutdown_connections 2024-07-26T12:46:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.787+0000 7f0074a00640 1 --2- 172.21.15.107:0/45326592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0070108720 0x7f007010ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.787+0000 7f0074a00640 1 --2- 172.21.15.107:0/45326592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0070107dd0 0x7f00701081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.787+0000 7f0074a00640 1 --2- 172.21.15.107:0/45326592 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0070102490 0x7f00701028a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.787+0000 7f0074a00640 1 -- 172.21.15.107:0/45326592 >> 172.21.15.107:0/45326592 conn(0x7f0070079160 msgr2=0x7f0070079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:26.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.787+0000 7f0074a00640 1 -- 172.21.15.107:0/45326592 shutdown_connections 2024-07-26T12:46:26.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.787+0000 7f0074a00640 1 -- 172.21.15.107:0/45326592 wait complete. 2024-07-26T12:46:26.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.788+0000 7f0074a00640 1 Processor -- start 2024-07-26T12:46:26.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.789+0000 7f0074a00640 1 -- start start 2024-07-26T12:46:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.789+0000 7f0074a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0070102490 0x7f0070199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.789+0000 7f0074a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0070107dd0 0x7f007019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.789+0000 7f0074a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0070108720 0x7f007019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.789+0000 7f0074a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f00701123f0 con 0x7f0070107dd0 2024-07-26T12:46:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.789+0000 7f0074a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0070112270 con 0x7f0070102490 2024-07-26T12:46:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.789+0000 7f0074a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0070112570 con 0x7f0070108720 2024-07-26T12:46:26.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.789+0000 7f006d400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0070107dd0 0x7f007019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:26.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.789+0000 7f006de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0070102490 0x7f0070199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:26.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.790+0000 7f006d400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0070107dd0 0x7f007019a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47174/0 (socket says 172.21.15.107:47174) 2024-07-26T12:46:26.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.790+0000 7f006d400640 1 -- 172.21.15.107:0/3427619262 learned_addr learned my addr 172.21.15.107:0/3427619262 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:26.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.790+0000 7f006e800640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0070108720 0x7f007019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.790+0000 7f006de00640 1 -- 172.21.15.107:0/3427619262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0070108720 msgr2=0x7f007019f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.790+0000 7f006de00640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0070108720 0x7f007019f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.790+0000 7f006de00640 1 -- 172.21.15.107:0/3427619262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0070107dd0 msgr2=0x7f007019a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.790+0000 7f006de00640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0070107dd0 0x7f007019a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.790+0000 7f006de00640 1 -- 172.21.15.107:0/3427619262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007019fe20 con 0x7f0070102490 2024-07-26T12:46:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.791+0000 7f006d400640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0070107dd0 0x7f007019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:46:26.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.791+0000 7f006de00640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0070102490 0x7f0070199eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f006002a9f0 tx=0x7f006003ac10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:26.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.791+0000 7f0066a00640 1 -- 172.21.15.107:0/3427619262 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060035070 con 0x7f0070102490 2024-07-26T12:46:26.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.791+0000 7f0066a00640 1 -- 172.21.15.107:0/3427619262 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0060003cb0 con 0x7f0070102490 2024-07-26T12:46:26.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.792+0000 7f0074a00640 1 -- 172.21.15.107:0/3427619262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00701a00b0 con 0x7f0070102490 2024-07-26T12:46:26.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.792+0000 7f0066a00640 1 -- 172.21.15.107:0/3427619262 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060031cc0 con 0x7f0070102490 2024-07-26T12:46:26.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.792+0000 7f0074a00640 1 -- 172.21.15.107:0/3427619262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0070103b60 con 0x7f0070102490 2024-07-26T12:46:26.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.793+0000 7f0074a00640 1 -- 172.21.15.107:0/3427619262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0070103710 con 0x7f0070102490 2024-07-26T12:46:26.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.797+0000 7f0066a00640 1 -- 172.21.15.107:0/3427619262 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f00600314a0 con 0x7f0070102490 2024-07-26T12:46:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.799+0000 7f0066a00640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f00540b3d30 0x7f00540b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:26.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.799+0000 7f0066a00640 1 -- 172.21.15.107:0/3427619262 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0060108730 con 0x7f0070102490 2024-07-26T12:46:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.799+0000 7f006d400640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f00540b3d30 0x7f00540b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.800+0000 7f0066a00640 1 -- 172.21.15.107:0/3427619262 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f00600cf720 con 0x7f0070102490 2024-07-26T12:46:26.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.800+0000 7f006d400640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f00540b3d30 0x7f00540b6240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f00701005a0 tx=0x7f005c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:26.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.883+0000 7f2f0b200640 1 -- 172.21.15.107:0/2658380828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f0c10d630 msgr2=0x7f2f0c10fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.883+0000 7f2f0b200640 1 --2- 172.21.15.107:0/2658380828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f0c10d630 0x7f2f0c10fb40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009be0 tx=0x7f2f0402a6d0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.883+0000 7f2f0b200640 1 -- 172.21.15.107:0/2658380828 shutdown_connections 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.883+0000 7f2f0b200640 1 --2- 172.21.15.107:0/2658380828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f0c10d630 0x7f2f0c10fb40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.883+0000 7f2f0b200640 1 --2- 172.21.15.107:0/2658380828 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f0c104ab0 0x7f2f0c10d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.883+0000 7f2f0b200640 1 --2- 172.21.15.107:0/2658380828 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f0c104080 0x7f2f0c104490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.883+0000 7f2f0b200640 1 -- 172.21.15.107:0/2658380828 >> 172.21.15.107:0/2658380828 conn(0x7f2f0c0fdcf0 msgr2=0x7f2f0c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.884+0000 7f2f0b200640 1 -- 172.21.15.107:0/2658380828 shutdown_connections 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.884+0000 7f2f0b200640 1 -- 172.21.15.107:0/2658380828 wait complete. 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.885+0000 7f2f0b200640 1 Processor -- start 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.885+0000 7f2f0b200640 1 -- start start 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.885+0000 7f2f0b200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f0c104080 0x7f2f0c19bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:26.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.885+0000 7f2f0b200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f0c104ab0 0x7f2f0c19c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.885+0000 7f2f0b200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f0c10d630 0x7f2f0c1a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.885+0000 7f2f0b200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2f0c1132c0 con 0x7f2f0c104080 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.885+0000 7f2f0b200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2f0c113140 con 0x7f2f0c104ab0 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.885+0000 7f2f0b200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2f0c113440 con 0x7f2f0c10d630 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f09e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f0c104080 0x7f2f0c19bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f0a800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f0c10d630 0x7f2f0c1a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f0a800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f0c10d630 0x7f2f0c1a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:39064/0 (socket says 172.21.15.107:39064) 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f09e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f0c104080 0x7f2f0c19bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47192/0 (socket says 172.21.15.107:47192) 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f0a800640 1 -- 172.21.15.107:0/3999893123 learned_addr learned my addr 172.21.15.107:0/3999893123 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f09400640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f0c104ab0 0x7f2f0c19c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f0a800640 1 -- 172.21.15.107:0/3999893123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f0c104ab0 msgr2=0x7f2f0c19c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:26.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f0a800640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f0c104ab0 0x7f2f0c19c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f0a800640 1 -- 172.21.15.107:0/3999893123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f0c104080 msgr2=0x7f2f0c19bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f0a800640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f0c104080 0x7f2f0c19bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f0a800640 1 -- 172.21.15.107:0/3999893123 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f0c1a1e80 con 0x7f2f0c10d630 2024-07-26T12:46:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.886+0000 7f2f09e00640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f0c104080 0x7f2f0c19bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.887+0000 7f2f0a800640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f0c10d630 0x7f2f0c1a1750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2f04004af0 tx=0x7f2f04003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.887+0000 7f2f02a00640 1 -- 172.21.15.107:0/3999893123 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04041020 con 0x7f2f0c10d630 2024-07-26T12:46:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.887+0000 7f2f0b200640 1 -- 172.21.15.107:0/3999893123 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f0c1a2110 con 0x7f2f0c10d630 2024-07-26T12:46:26.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.887+0000 7f2f02a00640 1 -- 172.21.15.107:0/3999893123 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f04003e10 con 0x7f2f0c10d630 2024-07-26T12:46:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.887+0000 7f2f02a00640 1 -- 172.21.15.107:0/3999893123 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f04036040 con 0x7f2f0c10d630 2024-07-26T12:46:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.887+0000 7f2f0b200640 1 -- 172.21.15.107:0/3999893123 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f0c1a88d0 con 0x7f2f0c10d630 2024-07-26T12:46:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.889+0000 7f2f0b200640 1 -- 172.21.15.107:0/3999893123 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f0c104490 con 0x7f2f0c10d630 2024-07-26T12:46:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.891+0000 7f2f02a00640 1 -- 172.21.15.107:0/3999893123 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f2f04044020 con 0x7f2f0c10d630 2024-07-26T12:46:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.892+0000 7f2f02a00640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2ef00b3d30 0x7f2ef00b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.893+0000 7f2f02a00640 1 -- 172.21.15.107:0/3999893123 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2f04105270 con 0x7f2f0c10d630 2024-07-26T12:46:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.893+0000 7f2f09e00640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2ef00b3d30 0x7f2ef00b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.894+0000 7f2f09e00640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2ef00b3d30 0x7f2ef00b6240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0096c0 tx=0x7f2efc009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:26.894+0000 7f2f02a00640 1 -- 172.21.15.107:0/3999893123 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2f0404b020 con 0x7f2f0c10d630 2024-07-26T12:46:27.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.026+0000 7f0074a00640 1 -- 172.21.15.107:0/3427619262 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0070103e20 con 0x7f0070102490 2024-07-26T12:46:27.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.026+0000 7f0066a00640 1 -- 172.21.15.107:0/3427619262 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f00600cf020 con 0x7f0070102490 2024-07-26T12:46:27.027 INFO:teuthology.orchestra.run.smithi107.stdout:60129542254 2024-07-26T12:46:27.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.030+0000 7f0074a00640 1 -- 172.21.15.107:0/3427619262 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f00540b3d30 msgr2=0x7f00540b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:27.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.030+0000 7f0074a00640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f00540b3d30 0x7f00540b6240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f00701005a0 tx=0x7f005c005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:27.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.030+0000 7f0074a00640 1 -- 172.21.15.107:0/3427619262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0070102490 msgr2=0x7f0070199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:27.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.030+0000 7f0074a00640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0070102490 0x7f0070199eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f006002a9f0 tx=0x7f006003ac10 comp rx=0 tx=0).stop 2024-07-26T12:46:27.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.030+0000 7f0074a00640 1 -- 172.21.15.107:0/3427619262 shutdown_connections 2024-07-26T12:46:27.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.030+0000 7f0074a00640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f00540b3d30 0x7f00540b6240 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:27.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.030+0000 7f0074a00640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0070108720 0x7f007019f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:27.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.031+0000 7f0074a00640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0070107dd0 0x7f007019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:27.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.031+0000 7f0074a00640 1 --2- 172.21.15.107:0/3427619262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0070102490 0x7f0070199eb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:27.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.031+0000 7f0074a00640 1 -- 172.21.15.107:0/3427619262 >> 172.21.15.107:0/3427619262 conn(0x7f0070079160 msgr2=0x7f0070106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:27.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.031+0000 7f0074a00640 1 -- 172.21.15.107:0/3427619262 shutdown_connections 2024-07-26T12:46:27.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.031+0000 7f0074a00640 1 -- 172.21.15.107:0/3427619262 wait complete. 2024-07-26T12:46:27.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.132+0000 7f2f0b200640 1 -- 172.21.15.107:0/3999893123 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f2f0c1a8aa0 con 0x7f2f0c10d630 2024-07-26T12:46:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.133+0000 7f2f02a00640 1 -- 172.21.15.107:0/3999893123 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2f04030970 con 0x7f2f0c10d630 2024-07-26T12:46:27.135 INFO:teuthology.orchestra.run.smithi107.stdout:231928234054 2024-07-26T12:46:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 -- 172.21.15.107:0/3999893123 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2ef00b3d30 msgr2=0x7f2ef00b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2ef00b3d30 0x7f2ef00b6240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0096c0 tx=0x7f2efc009260 comp rx=0 tx=0).stop 2024-07-26T12:46:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 -- 172.21.15.107:0/3999893123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f0c10d630 msgr2=0x7f2f0c1a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f0c10d630 0x7f2f0c1a1750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2f04004af0 tx=0x7f2f04003c20 comp rx=0 tx=0).stop 2024-07-26T12:46:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 -- 172.21.15.107:0/3999893123 shutdown_connections 2024-07-26T12:46:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2ef00b3d30 0x7f2ef00b6240 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2f0c10d630 0x7f2f0c1a1750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2f0c104ab0 0x7f2f0c19c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 --2- 172.21.15.107:0/3999893123 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2f0c104080 0x7f2f0c19bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:27.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 -- 172.21.15.107:0/3999893123 >> 172.21.15.107:0/3999893123 conn(0x7f2f0c0fdcf0 msgr2=0x7f2f0c10c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 -- 172.21.15.107:0/3999893123 shutdown_connections 2024-07-26T12:46:27.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:27.137+0000 7f2f0b200640 1 -- 172.21.15.107:0/3999893123 wait complete. 2024-07-26T12:46:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:27 smithi112 ceph-mon[37501]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:27 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3427619262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T12:46:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:27 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3999893123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T12:46:28.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:27 smithi107 ceph-mon[32659]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:28.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:27 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3427619262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T12:46:28.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:27 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3999893123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T12:46:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:27 smithi132 ceph-mon[37580]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:27 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3427619262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T12:46:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:27 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3999893123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T12:46:28.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.048+0000 7f8d30a00640 1 -- 172.21.15.107:0/2120270952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2c10b860 msgr2=0x7f8d2c112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:28.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.048+0000 7f8d30a00640 1 --2- 172.21.15.107:0/2120270952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2c10b860 0x7f8d2c112090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009ca0 tx=0x7f8d2402a8c0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.049+0000 7f8d30a00640 1 -- 172.21.15.107:0/2120270952 shutdown_connections 2024-07-26T12:46:28.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.049+0000 7f8d30a00640 1 --2- 172.21.15.107:0/2120270952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2c10b860 0x7f8d2c112090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.049+0000 7f8d30a00640 1 --2- 172.21.15.107:0/2120270952 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d2c10af10 0x7f8d2c10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.049+0000 7f8d30a00640 1 --2- 172.21.15.107:0/2120270952 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d2c1055d0 0x7f8d2c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.049+0000 7f8d30a00640 1 -- 172.21.15.107:0/2120270952 >> 172.21.15.107:0/2120270952 conn(0x7f8d2c101040 msgr2=0x7f8d2c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.049+0000 7f8d30a00640 1 -- 172.21.15.107:0/2120270952 shutdown_connections 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.049+0000 7f8d30a00640 1 -- 172.21.15.107:0/2120270952 wait complete. 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.050+0000 7f8d30a00640 1 Processor -- start 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.050+0000 7f8d30a00640 1 -- start start 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.050+0000 7f8d30a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d2c1055d0 0x7f8d2c19beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.050+0000 7f8d30a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d2c10af10 0x7f8d2c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.050+0000 7f8d30a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2c10b860 0x7f8d2c1a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.050+0000 7f8d30a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c115560 con 0x7f8d2c10b860 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.050+0000 7f8d30a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c1153e0 con 0x7f8d2c10af10 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.050+0000 7f8d30a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c1156e0 con 0x7f8d2c1055d0 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d2c10af10 0x7f8d2c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2be00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2c10b860 0x7f8d2c1a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:28.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d2c10af10 0x7f8d2c19c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:45708/0 (socket says 172.21.15.107:45708) 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2be00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2c10b860 0x7f8d2c1a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47220/0 (socket says 172.21.15.107:47220) 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2aa00640 1 -- 172.21.15.107:0/1154256525 learned_addr learned my addr 172.21.15.107:0/1154256525 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2b400640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d2c1055d0 0x7f8d2c19beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2aa00640 1 -- 172.21.15.107:0/1154256525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d2c1055d0 msgr2=0x7f8d2c19beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2aa00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d2c1055d0 0x7f8d2c19beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2aa00640 1 -- 172.21.15.107:0/1154256525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2c10b860 msgr2=0x7f8d2c1a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2aa00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2c10b860 0x7f8d2c1a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2aa00640 1 -- 172.21.15.107:0/1154256525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2c1a1e20 con 0x7f8d2c10af10 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.051+0000 7f8d2be00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2c10b860 0x7f8d2c1a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.052+0000 7f8d2aa00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d2c10af10 0x7f8d2c19c450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800e9f0 tx=0x7f8d1800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.052+0000 7f8d23e00640 1 -- 172.21.15.107:0/1154256525 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1800ccc0 con 0x7f8d2c10af10 2024-07-26T12:46:28.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.052+0000 7f8d30a00640 1 -- 172.21.15.107:0/1154256525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2c1a2110 con 0x7f8d2c10af10 2024-07-26T12:46:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.052+0000 7f8d23e00640 1 -- 172.21.15.107:0/1154256525 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d180108e0 con 0x7f8d2c10af10 2024-07-26T12:46:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.052+0000 7f8d23e00640 1 -- 172.21.15.107:0/1154256525 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d18018860 con 0x7f8d2c10af10 2024-07-26T12:46:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.052+0000 7f8d30a00640 1 -- 172.21.15.107:0/1154256525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d2c1a8870 con 0x7f8d2c10af10 2024-07-26T12:46:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.054+0000 7f8d30a00640 1 -- 172.21.15.107:0/1154256525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d2c1059e0 con 0x7f8d2c10af10 2024-07-26T12:46:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.056+0000 7f8d23e00640 1 -- 172.21.15.107:0/1154256525 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f8d18010430 con 0x7f8d2c10af10 2024-07-26T12:46:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.058+0000 7f8d23e00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8d100b3d30 0x7f8d100b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.058+0000 7f8d23e00640 1 -- 172.21.15.107:0/1154256525 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8d180e4530 con 0x7f8d2c10af10 2024-07-26T12:46:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.059+0000 7f8d2b400640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8d100b3d30 0x7f8d100b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.060+0000 7f8d2b400640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8d100b3d30 0x7f8d100b6240 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009bd0 tx=0x7f8d1c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:28.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.062+0000 7f8d23e00640 1 -- 172.21.15.107:0/1154256525 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8d180aaed0 con 0x7f8d2c10af10 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stdout:279172874300 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.273+0000 7f8d30a00640 1 -- 172.21.15.107:0/1154256525 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f8d2c10d0c0 con 0x7f8d2c10af10 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.273+0000 7f8d23e00640 1 -- 172.21.15.107:0/1154256525 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f8d180af420 con 0x7f8d2c10af10 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.277+0000 7f8d30a00640 1 -- 172.21.15.107:0/1154256525 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8d100b3d30 msgr2=0x7f8d100b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.277+0000 7f8d30a00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8d100b3d30 0x7f8d100b6240 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009bd0 tx=0x7f8d1c005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.277+0000 7f8d30a00640 1 -- 172.21.15.107:0/1154256525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d2c10af10 msgr2=0x7f8d2c19c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.277+0000 7f8d30a00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d2c10af10 0x7f8d2c19c450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800e9f0 tx=0x7f8d1800eeb0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.277+0000 7f8d30a00640 1 -- 172.21.15.107:0/1154256525 shutdown_connections 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.277+0000 7f8d30a00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8d100b3d30 0x7f8d100b6240 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.277+0000 7f8d30a00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d2c10b860 0x7f8d2c1a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.277+0000 7f8d30a00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8d2c10af10 0x7f8d2c19c450 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.277+0000 7f8d30a00640 1 --2- 172.21.15.107:0/1154256525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8d2c1055d0 0x7f8d2c19beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:28.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.277+0000 7f8d30a00640 1 -- 172.21.15.107:0/1154256525 >> 172.21.15.107:0/1154256525 conn(0x7f8d2c101040 msgr2=0x7f8d2c10a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:28.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.278+0000 7f8d30a00640 1 -- 172.21.15.107:0/1154256525 shutdown_connections 2024-07-26T12:46:28.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.278+0000 7f8d30a00640 1 -- 172.21.15.107:0/1154256525 wait complete. 2024-07-26T12:46:29.104 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:28 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1154256525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T12:46:29.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:28 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1772949149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T12:46:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:28 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1154256525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T12:46:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:28 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1772949149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T12:46:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:28 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1154256525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T12:46:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:28 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1772949149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T12:46:29.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.432+0000 7f4c16800640 1 -- 172.21.15.107:0/416944974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c10104ce0 msgr2=0x7f4c1010d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.432+0000 7f4c16800640 1 --2- 172.21.15.107:0/416944974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c10104ce0 0x7f4c1010d320 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009900 tx=0x7f4bf802a410 comp rx=0 tx=0).stop 2024-07-26T12:46:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.433+0000 7f4c16800640 1 -- 172.21.15.107:0/416944974 shutdown_connections 2024-07-26T12:46:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.433+0000 7f4c16800640 1 --2- 172.21.15.107:0/416944974 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c1010d860 0x7f4c1010fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.433+0000 7f4c16800640 1 --2- 172.21.15.107:0/416944974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c10104ce0 0x7f4c1010d320 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.433+0000 7f4c16800640 1 --2- 172.21.15.107:0/416944974 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c101042b0 0x7f4c101046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.433+0000 7f4c16800640 1 -- 172.21.15.107:0/416944974 >> 172.21.15.107:0/416944974 conn(0x7f4c10079160 msgr2=0x7f4c10079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.433+0000 7f4c16800640 1 -- 172.21.15.107:0/416944974 shutdown_connections 2024-07-26T12:46:29.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.433+0000 7f4c16800640 1 -- 172.21.15.107:0/416944974 wait complete. 2024-07-26T12:46:29.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.434+0000 7f4c16800640 1 Processor -- start 2024-07-26T12:46:29.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.434+0000 7f4c16800640 1 -- start start 2024-07-26T12:46:29.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c16800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c101042b0 0x7f4c1019c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c16800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c10104ce0 0x7f4c1019c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c16800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c1010d860 0x7f4c101a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c16800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4c10113460 con 0x7f4c1010d860 2024-07-26T12:46:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c16800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4c101132e0 con 0x7f4c101042b0 2024-07-26T12:46:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c16800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4c101135e0 con 0x7f4c10104ce0 2024-07-26T12:46:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c07e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c10104ce0 0x7f4c1019c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c0fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c1010d860 0x7f4c101a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c07e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c10104ce0 0x7f4c1019c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:39132/0 (socket says 172.21.15.107:39132) 2024-07-26T12:46:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c0fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c1010d860 0x7f4c101a1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47258/0 (socket says 172.21.15.107:47258) 2024-07-26T12:46:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c07e00640 1 -- 172.21.15.107:0/4092371786 learned_addr learned my addr 172.21.15.107:0/4092371786 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.435+0000 7f4c0f400640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c101042b0 0x7f4c1019c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.436+0000 7f4c0fe00640 1 -- 172.21.15.107:0/4092371786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c10104ce0 msgr2=0x7f4c1019c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.436+0000 7f4c0fe00640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c10104ce0 0x7f4c1019c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.436+0000 7f4c0fe00640 1 -- 172.21.15.107:0/4092371786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c101042b0 msgr2=0x7f4c1019c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.436+0000 7f4c0fe00640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c101042b0 0x7f4c1019c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.436+0000 7f4c0fe00640 1 -- 172.21.15.107:0/4092371786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c101a2080 con 0x7f4c1010d860 2024-07-26T12:46:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.436+0000 7f4c07e00640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c10104ce0 0x7f4c1019c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:29.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.436+0000 7f4c0fe00640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c1010d860 0x7f4c101a1950 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4c08009fd0 tx=0x7f4c0800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.437+0000 7f4c0cc00640 1 -- 172.21.15.107:0/4092371786 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0800ce20 con 0x7f4c1010d860 2024-07-26T12:46:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.437+0000 7f4c0cc00640 1 -- 172.21.15.107:0/4092371786 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c08005020 con 0x7f4c1010d860 2024-07-26T12:46:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.437+0000 7f4c16800640 1 -- 172.21.15.107:0/4092371786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c101a2370 con 0x7f4c1010d860 2024-07-26T12:46:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.437+0000 7f4c0cc00640 1 -- 172.21.15.107:0/4092371786 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08010690 con 0x7f4c1010d860 2024-07-26T12:46:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.437+0000 7f4c16800640 1 -- 172.21.15.107:0/4092371786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c101a8ad0 con 0x7f4c1010d860 2024-07-26T12:46:29.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.438+0000 7f4c16800640 1 -- 172.21.15.107:0/4092371786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c101046c0 con 0x7f4c1010d860 2024-07-26T12:46:29.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.441+0000 7f4c0cc00640 1 -- 172.21.15.107:0/4092371786 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f4c080054d0 con 0x7f4c1010d860 2024-07-26T12:46:29.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.443+0000 7f4c0cc00640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4bf40b3cb0 0x7f4bf40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:29.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.443+0000 7f4c0cc00640 1 -- 172.21.15.107:0/4092371786 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4c080e48b0 con 0x7f4c1010d860 2024-07-26T12:46:29.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.443+0000 7f4c0f400640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4bf40b3cb0 0x7f4bf40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:29.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.444+0000 7f4c0f400640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4bf40b3cb0 0x7f4bf40b61c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4c1006b5f0 tx=0x7f4c00005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:29.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.445+0000 7f4c0cc00640 1 -- 172.21.15.107:0/4092371786 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4c080ab900 con 0x7f4c1010d860 2024-07-26T12:46:29.541 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678066 got 304942678070 for osd.11 2024-07-26T12:46:29.541 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:29.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.663+0000 7f4c16800640 1 -- 172.21.15.107:0/4092371786 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f4c1019d240 con 0x7f4c1010d860 2024-07-26T12:46:29.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.664+0000 7f4c0cc00640 1 -- 172.21.15.107:0/4092371786 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4c1019d240 con 0x7f4c1010d860 2024-07-26T12:46:29.665 INFO:teuthology.orchestra.run.smithi107.stdout:81604378727 2024-07-26T12:46:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.667+0000 7f4c16800640 1 -- 172.21.15.107:0/4092371786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4bf40b3cb0 msgr2=0x7f4bf40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.667+0000 7f4c16800640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4bf40b3cb0 0x7f4bf40b61c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4c1006b5f0 tx=0x7f4c00005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.667+0000 7f4c16800640 1 -- 172.21.15.107:0/4092371786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c1010d860 msgr2=0x7f4c101a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:29.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.667+0000 7f4c16800640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c1010d860 0x7f4c101a1950 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4c08009fd0 tx=0x7f4c0800ef10 comp rx=0 tx=0).stop 2024-07-26T12:46:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.667+0000 7f4c16800640 1 -- 172.21.15.107:0/4092371786 shutdown_connections 2024-07-26T12:46:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.667+0000 7f4c16800640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4bf40b3cb0 0x7f4bf40b61c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.667+0000 7f4c16800640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c1010d860 0x7f4c101a1950 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.667+0000 7f4c16800640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4c10104ce0 0x7f4c1019c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.667+0000 7f4c16800640 1 --2- 172.21.15.107:0/4092371786 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4c101042b0 0x7f4c1019c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.667+0000 7f4c16800640 1 -- 172.21.15.107:0/4092371786 >> 172.21.15.107:0/4092371786 conn(0x7f4c10079160 msgr2=0x7f4c100fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.668+0000 7f4c16800640 1 -- 172.21.15.107:0/4092371786 shutdown_connections 2024-07-26T12:46:29.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:29.668+0000 7f4c16800640 1 -- 172.21.15.107:0/4092371786 wait complete. 2024-07-26T12:46:29.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.252+0000 7f3e0f200640 1 -- 172.21.15.107:0/3396666816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e10108530 msgr2=0x7f3e1010ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:29.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.252+0000 7f3e0f200640 1 --2- 172.21.15.107:0/3396666816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e10108530 0x7f3e1010ed60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009ca0 tx=0x7f3e0802a8c0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.252+0000 7f3e0f200640 1 -- 172.21.15.107:0/3396666816 shutdown_connections 2024-07-26T12:46:29.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.252+0000 7f3e0f200640 1 --2- 172.21.15.107:0/3396666816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e10108530 0x7f3e1010ed60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.252+0000 7f3e0f200640 1 --2- 172.21.15.107:0/3396666816 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e10107be0 0x7f3e10107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.252+0000 7f3e0f200640 1 --2- 172.21.15.107:0/3396666816 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e101022a0 0x7f3e101026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.252+0000 7f3e0f200640 1 -- 172.21.15.107:0/3396666816 >> 172.21.15.107:0/3396666816 conn(0x7f3e100fdcf0 msgr2=0x7f3e10100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.252+0000 7f3e0f200640 1 -- 172.21.15.107:0/3396666816 shutdown_connections 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.253+0000 7f3e0f200640 1 -- 172.21.15.107:0/3396666816 wait complete. 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.254+0000 7f3e0f200640 1 Processor -- start 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.254+0000 7f3e0f200640 1 -- start start 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0f200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e101022a0 0x7f3e1019bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0f200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e10107be0 0x7f3e1019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0f200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e10108530 0x7f3e101a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0f200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3e10113310 con 0x7f3e10108530 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0f200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3e10113190 con 0x7f3e101022a0 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0f200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3e10113490 con 0x7f3e10107be0 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0e800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e10108530 0x7f3e101a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0e800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e10108530 0x7f3e101a16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47236/0 (socket says 172.21.15.107:47236) 2024-07-26T12:46:29.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0e800640 1 -- 172.21.15.107:0/1772949149 learned_addr learned my addr 172.21.15.107:0/1772949149 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0d400640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e10107be0 0x7f3e1019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.255+0000 7f3e0de00640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e101022a0 0x7f3e1019bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.256+0000 7f3e0d400640 1 -- 172.21.15.107:0/1772949149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e101022a0 msgr2=0x7f3e1019bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.256+0000 7f3e0d400640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e101022a0 0x7f3e1019bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.256+0000 7f3e0d400640 1 -- 172.21.15.107:0/1772949149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e10108530 msgr2=0x7f3e101a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.256+0000 7f3e0d400640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e10108530 0x7f3e101a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.256+0000 7f3e0d400640 1 -- 172.21.15.107:0/1772949149 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e101a1e10 con 0x7f3e10107be0 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.256+0000 7f3e0e800640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e10108530 0x7f3e101a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.256+0000 7f3e0d400640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e10107be0 0x7f3e1019c440 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3e1006a4c0 tx=0x7f3dfc00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.256+0000 7f3e06a00640 1 -- 172.21.15.107:0/1772949149 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc00cc40 con 0x7f3e10107be0 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.257+0000 7f3e0f200640 1 -- 172.21.15.107:0/1772949149 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e101a2100 con 0x7f3e10107be0 2024-07-26T12:46:29.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.257+0000 7f3e06a00640 1 -- 172.21.15.107:0/1772949149 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dfc00cde0 con 0x7f3e10107be0 2024-07-26T12:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.257+0000 7f3e06a00640 1 -- 172.21.15.107:0/1772949149 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc018760 con 0x7f3e10107be0 2024-07-26T12:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.257+0000 7f3e0f200640 1 -- 172.21.15.107:0/1772949149 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e101a8860 con 0x7f3e10107be0 2024-07-26T12:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.258+0000 7f3e0f200640 1 -- 172.21.15.107:0/1772949149 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e101026b0 con 0x7f3e10107be0 2024-07-26T12:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.260+0000 7f3e06a00640 1 -- 172.21.15.107:0/1772949149 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f3dfc010a90 con 0x7f3e10107be0 2024-07-26T12:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.262+0000 7f3e06a00640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3df40b3cb0 0x7f3df40b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.262+0000 7f3e06a00640 1 -- 172.21.15.107:0/1772949149 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3dfc0e4a70 con 0x7f3e10107be0 2024-07-26T12:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.262+0000 7f3e0de00640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3df40b3cb0 0x7f3df40b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.263+0000 7f3e0de00640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3df40b3cb0 0x7f3df40b61c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3e000095f0 tx=0x7f3e00005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:29.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.263+0000 7f3e06a00640 1 -- 172.21.15.107:0/1772949149 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3dfc0ab410 con 0x7f3e10107be0 2024-07-26T12:46:29.705 INFO:teuthology.orchestra.run.smithi107.stdout:111669149794 2024-07-26T12:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.482+0000 7f3e0f200640 1 -- 172.21.15.107:0/1772949149 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3e101a8b90 con 0x7f3e10107be0 2024-07-26T12:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.483+0000 7f3e06a00640 1 -- 172.21.15.107:0/1772949149 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3dfc0af960 con 0x7f3e10107be0 2024-07-26T12:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 -- 172.21.15.107:0/1772949149 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3df40b3cb0 msgr2=0x7f3df40b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3df40b3cb0 0x7f3df40b61c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3e000095f0 tx=0x7f3e00005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 -- 172.21.15.107:0/1772949149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e10107be0 msgr2=0x7f3e1019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e10107be0 0x7f3e1019c440 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3e1006a4c0 tx=0x7f3dfc00ee50 comp rx=0 tx=0).stop 2024-07-26T12:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 -- 172.21.15.107:0/1772949149 shutdown_connections 2024-07-26T12:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3df40b3cb0 0x7f3df40b61c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3e10108530 0x7f3e101a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e10107be0 0x7f3e1019c440 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 --2- 172.21.15.107:0/1772949149 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3e101022a0 0x7f3e1019bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:29.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 -- 172.21.15.107:0/1772949149 >> 172.21.15.107:0/1772949149 conn(0x7f3e100fdcf0 msgr2=0x7f3e1010d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:29.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 -- 172.21.15.107:0/1772949149 shutdown_connections 2024-07-26T12:46:29.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:28.486+0000 7f3e0f200640 1 -- 172.21.15.107:0/1772949149 wait complete. 2024-07-26T12:46:29.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:29 smithi107 ceph-mon[32659]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:29.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:29 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/4092371786' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T12:46:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:29 smithi112 ceph-mon[37501]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:29 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/4092371786' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T12:46:30.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:29 smithi132 ceph-mon[37580]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:30.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:29 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/4092371786' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T12:46:31.152 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:30 smithi107 ceph-mon[32659]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:46:31.198 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149787 got 111669149794 for osd.3 2024-07-26T12:46:31.198 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:30 smithi112 ceph-mon[37501]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:46:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:30 smithi132 ceph-mon[37580]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:46:31.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.870+0000 7feaa1200640 1 -- 172.21.15.107:0/1224139662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea9c10f9a0 msgr2=0x7fea9c111dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:31.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.870+0000 7feaa1200640 1 --2- 172.21.15.107:0/1224139662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea9c10f9a0 0x7fea9c111dc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fea94009ca0 tx=0x7fea9402a8b0 comp rx=0 tx=0).stop 2024-07-26T12:46:31.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.870+0000 7feaa1200640 1 -- 172.21.15.107:0/1224139662 shutdown_connections 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.870+0000 7feaa1200640 1 --2- 172.21.15.107:0/1224139662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea9c10f9a0 0x7fea9c111dc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.870+0000 7feaa1200640 1 --2- 172.21.15.107:0/1224139662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea9c075ad0 0x7fea9c10f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.870+0000 7feaa1200640 1 --2- 172.21.15.107:0/1224139662 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea9c075180 0x7fea9c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.870+0000 7feaa1200640 1 -- 172.21.15.107:0/1224139662 >> 172.21.15.107:0/1224139662 conn(0x7fea9c0fee30 msgr2=0x7fea9c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.870+0000 7feaa1200640 1 -- 172.21.15.107:0/1224139662 shutdown_connections 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.870+0000 7feaa1200640 1 -- 172.21.15.107:0/1224139662 wait complete. 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.871+0000 7feaa1200640 1 Processor -- start 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.871+0000 7feaa1200640 1 -- start start 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7feaa1200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea9c075180 0x7fea9c19beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7feaa1200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea9c075ad0 0x7fea9c19c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7feaa1200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea9c10f9a0 0x7fea9c1a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:31.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7feaa1200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c115500 con 0x7fea9c075ad0 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7feaa1200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c115380 con 0x7fea9c10f9a0 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7feaa1200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fea9c115680 con 0x7fea9c075180 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7fea9b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea9c075180 0x7fea9c19beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7fea9aa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea9c075ad0 0x7fea9c19c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7fea9b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea9c075180 0x7fea9c19beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:39178/0 (socket says 172.21.15.107:39178) 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7fea9aa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea9c075ad0 0x7fea9c19c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47288/0 (socket says 172.21.15.107:47288) 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7fea9b400640 1 -- 172.21.15.107:0/664545405 learned_addr learned my addr 172.21.15.107:0/664545405 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.872+0000 7fea9be00640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea9c10f9a0 0x7fea9c1a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7fea9aa00640 1 -- 172.21.15.107:0/664545405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea9c075180 msgr2=0x7fea9c19beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7fea9aa00640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea9c075180 0x7fea9c19beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7fea9aa00640 1 -- 172.21.15.107:0/664545405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea9c10f9a0 msgr2=0x7fea9c1a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7fea9aa00640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea9c10f9a0 0x7fea9c1a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:31.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7fea9aa00640 1 -- 172.21.15.107:0/664545405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9c1a1e20 con 0x7fea9c075ad0 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7fea9b400640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea9c075180 0x7fea9c19beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7fea9aa00640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea9c075ad0 0x7fea9c19c450 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fea9c077200 tx=0x7fea8800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7fea93e00640 1 -- 172.21.15.107:0/664545405 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8800cd00 con 0x7fea9c075ad0 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7fea93e00640 1 -- 172.21.15.107:0/664545405 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea880035f0 con 0x7fea9c075ad0 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7fea93e00640 1 -- 172.21.15.107:0/664545405 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88010690 con 0x7fea9c075ad0 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7feaa1200640 1 -- 172.21.15.107:0/664545405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea9c1a2110 con 0x7fea9c075ad0 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.873+0000 7feaa1200640 1 -- 172.21.15.107:0/664545405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea9c1a8870 con 0x7fea9c075ad0 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.875+0000 7feaa1200640 1 -- 172.21.15.107:0/664545405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea9c075590 con 0x7fea9c075ad0 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.878+0000 7fea93e00640 1 -- 172.21.15.107:0/664545405 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fea880037a0 con 0x7fea9c075ad0 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.879+0000 7fea93e00640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fea800b3cb0 0x7fea800b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:31.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.880+0000 7fea93e00640 1 -- 172.21.15.107:0/664545405 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fea880e46f0 con 0x7fea9c075ad0 2024-07-26T12:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.880+0000 7fea9b400640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fea800b3cb0 0x7fea800b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.881+0000 7fea9b400640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fea800b3cb0 0x7fea800b61c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea8c0095f0 tx=0x7fea8c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.881+0000 7fea93e00640 1 -- 172.21.15.107:0/664545405 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fea88055600 con 0x7fea9c075ad0 2024-07-26T12:46:32.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:31 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3868838140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T12:46:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.086+0000 7feaa1200640 1 -- 172.21.15.107:0/664545405 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fea9c19cfe0 con 0x7fea9c075ad0 2024-07-26T12:46:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.087+0000 7fea93e00640 1 -- 172.21.15.107:0/664545405 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fea880ab6e0 con 0x7fea9c075ad0 2024-07-26T12:46:32.088 INFO:teuthology.orchestra.run.smithi107.stdout:137438953565 2024-07-26T12:46:32.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.090+0000 7feaa1200640 1 -- 172.21.15.107:0/664545405 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fea800b3cb0 msgr2=0x7fea800b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:32.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.090+0000 7feaa1200640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fea800b3cb0 0x7fea800b61c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea8c0095f0 tx=0x7fea8c005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:32.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.090+0000 7feaa1200640 1 -- 172.21.15.107:0/664545405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea9c075ad0 msgr2=0x7fea9c19c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:32.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.090+0000 7feaa1200640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea9c075ad0 0x7fea9c19c450 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fea9c077200 tx=0x7fea8800ee50 comp rx=0 tx=0).stop 2024-07-26T12:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.090+0000 7feaa1200640 1 -- 172.21.15.107:0/664545405 shutdown_connections 2024-07-26T12:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.090+0000 7feaa1200640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fea800b3cb0 0x7fea800b61c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.090+0000 7feaa1200640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea9c10f9a0 0x7fea9c1a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.090+0000 7feaa1200640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea9c075ad0 0x7fea9c19c450 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.090+0000 7feaa1200640 1 --2- 172.21.15.107:0/664545405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea9c075180 0x7fea9c19beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.090+0000 7feaa1200640 1 -- 172.21.15.107:0/664545405 >> 172.21.15.107:0/664545405 conn(0x7fea9c0fee30 msgr2=0x7fea9c1094b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.091+0000 7feaa1200640 1 -- 172.21.15.107:0/664545405 shutdown_connections 2024-07-26T12:46:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.091+0000 7feaa1200640 1 -- 172.21.15.107:0/664545405 wait complete. 2024-07-26T12:46:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:31 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3868838140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T12:46:32.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.231+0000 7f3d45800640 1 -- 172.21.15.107:0/218728186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d4010d650 msgr2=0x7f3d4010fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:32.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.231+0000 7f3d45800640 1 --2- 172.21.15.107:0/218728186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d4010d650 0x7f3d4010fb40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009900 tx=0x7f3d3802a420 comp rx=0 tx=0).stop 2024-07-26T12:46:32.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.232+0000 7f3d45800640 1 -- 172.21.15.107:0/218728186 shutdown_connections 2024-07-26T12:46:32.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.232+0000 7f3d45800640 1 --2- 172.21.15.107:0/218728186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d4010d650 0x7f3d4010fb40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.232+0000 7f3d45800640 1 --2- 172.21.15.107:0/218728186 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d40104b40 0x7f3d4010d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.232+0000 7f3d45800640 1 --2- 172.21.15.107:0/218728186 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d401041f0 0x7f3d40104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.232+0000 7f3d45800640 1 -- 172.21.15.107:0/218728186 >> 172.21.15.107:0/218728186 conn(0x7f3d400fdcf0 msgr2=0x7f3d40100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:32.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.232+0000 7f3d45800640 1 -- 172.21.15.107:0/218728186 shutdown_connections 2024-07-26T12:46:32.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.232+0000 7f3d45800640 1 -- 172.21.15.107:0/218728186 wait complete. 2024-07-26T12:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.233+0000 7f3d45800640 1 Processor -- start 2024-07-26T12:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.233+0000 7f3d45800640 1 -- start start 2024-07-26T12:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.233+0000 7f3d45800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d401041f0 0x7f3d40199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:32.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.233+0000 7f3d45800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d40104b40 0x7f3d4019a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:32.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.233+0000 7f3d45800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d4010d650 0x7f3d4019f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:32.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.233+0000 7f3d45800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3d40113350 con 0x7f3d4010d650 2024-07-26T12:46:32.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.233+0000 7f3d45800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3d401131d0 con 0x7f3d401041f0 2024-07-26T12:46:32.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.233+0000 7f3d45800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3d401134d0 con 0x7f3d40104b40 2024-07-26T12:46:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.234+0000 7f3d3fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d401041f0 0x7f3d40199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.234+0000 7f3d44e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d4010d650 0x7f3d4019f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.234+0000 7f3d3fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d401041f0 0x7f3d40199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:45812/0 (socket says 172.21.15.107:45812) 2024-07-26T12:46:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.234+0000 7f3d3fe00640 1 -- 172.21.15.107:0/3650850699 learned_addr learned my addr 172.21.15.107:0/3650850699 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.234+0000 7f3d37400640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d40104b40 0x7f3d4019a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.234+0000 7f3d44e00640 1 -- 172.21.15.107:0/3650850699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d40104b40 msgr2=0x7f3d4019a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.234+0000 7f3d44e00640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d40104b40 0x7f3d4019a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.235+0000 7f3d44e00640 1 -- 172.21.15.107:0/3650850699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d401041f0 msgr2=0x7f3d40199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.235+0000 7f3d44e00640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d401041f0 0x7f3d40199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.235+0000 7f3d44e00640 1 -- 172.21.15.107:0/3650850699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4019fd10 con 0x7f3d4010d650 2024-07-26T12:46:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.235+0000 7f3d37400640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d40104b40 0x7f3d4019a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.235+0000 7f3d44e00640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d4010d650 0x7f3d4019f5e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3d38005940 tx=0x7f3d38031a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.235+0000 7f3d3d600640 1 -- 172.21.15.107:0/3650850699 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38048070 con 0x7f3d4010d650 2024-07-26T12:46:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.235+0000 7f3d3d600640 1 -- 172.21.15.107:0/3650850699 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d380043d0 con 0x7f3d4010d650 2024-07-26T12:46:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.235+0000 7f3d45800640 1 -- 172.21.15.107:0/3650850699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d4019ffa0 con 0x7f3d4010d650 2024-07-26T12:46:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.235+0000 7f3d3d600640 1 -- 172.21.15.107:0/3650850699 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38035050 con 0x7f3d4010d650 2024-07-26T12:46:32.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.235+0000 7f3d45800640 1 -- 172.21.15.107:0/3650850699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d40101e10 con 0x7f3d4010d650 2024-07-26T12:46:32.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.237+0000 7f3d45800640 1 -- 172.21.15.107:0/3650850699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d4006af90 con 0x7f3d4010d650 2024-07-26T12:46:32.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.239+0000 7f3d3d600640 1 -- 172.21.15.107:0/3650850699 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f3d3804b070 con 0x7f3d4010d650 2024-07-26T12:46:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.240+0000 7f3d3d600640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3d240b3cb0 0x7f3d240b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.241+0000 7f3d3d600640 1 -- 172.21.15.107:0/3650850699 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3d38107e70 con 0x7f3d4010d650 2024-07-26T12:46:32.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.241+0000 7f3d3fe00640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3d240b3cb0 0x7f3d240b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:32.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.242+0000 7f3d3fe00640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3d240b3cb0 0x7f3d240b61c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009a30 tx=0x7f3d30009190 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:32.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.245+0000 7f3d3d600640 1 -- 172.21.15.107:0/3650850699 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3d380ce810 con 0x7f3d4010d650 2024-07-26T12:46:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:31 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3868838140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T12:46:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.453+0000 7f3d45800640 1 -- 172.21.15.107:0/3650850699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3d40102070 con 0x7f3d4010d650 2024-07-26T12:46:32.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.454+0000 7f3d3d600640 1 -- 172.21.15.107:0/3650850699 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3d380d2d60 con 0x7f3d4010d650 2024-07-26T12:46:32.455 INFO:teuthology.orchestra.run.smithi107.stdout:158913790041 2024-07-26T12:46:32.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 -- 172.21.15.107:0/3650850699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3d240b3cb0 msgr2=0x7f3d240b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3d240b3cb0 0x7f3d240b61c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009a30 tx=0x7f3d30009190 comp rx=0 tx=0).stop 2024-07-26T12:46:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 -- 172.21.15.107:0/3650850699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d4010d650 msgr2=0x7f3d4019f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d4010d650 0x7f3d4019f5e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3d38005940 tx=0x7f3d38031a40 comp rx=0 tx=0).stop 2024-07-26T12:46:32.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 -- 172.21.15.107:0/3650850699 shutdown_connections 2024-07-26T12:46:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3d240b3cb0 0x7f3d240b61c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d4010d650 0x7f3d4019f5e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3d40104b40 0x7f3d4019a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 --2- 172.21.15.107:0/3650850699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3d401041f0 0x7f3d40199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 -- 172.21.15.107:0/3650850699 >> 172.21.15.107:0/3650850699 conn(0x7f3d400fdcf0 msgr2=0x7f3d40107100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 -- 172.21.15.107:0/3650850699 shutdown_connections 2024-07-26T12:46:32.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:32.457+0000 7f3d45800640 1 -- 172.21.15.107:0/3650850699 wait complete. 2024-07-26T12:46:33.152 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:32 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/664545405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T12:46:33.152 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:32 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3650850699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T12:46:33.152 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:32 smithi107 ceph-mon[32659]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:32 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/664545405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T12:46:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:32 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3650850699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T12:46:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:32 smithi112 ceph-mon[37501]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:32 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/664545405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T12:46:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:32 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3650850699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T12:46:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:32 smithi132 ceph-mon[37580]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:33.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542246 got 60129542254 for osd.1 2024-07-26T12:46:33.524 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:34.529 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378722 got 81604378727 for osd.2 2024-07-26T12:46:34.529 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:34.656 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234049 got 231928234054 for osd.8 2024-07-26T12:46:34.656 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:35.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.257+0000 7f0ea6800640 1 -- 172.21.15.107:0/4170169077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea01042b0 msgr2=0x7f0ea01046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:35.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.257+0000 7f0ea6800640 1 --2- 172.21.15.107:0/4170169077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea01042b0 0x7f0ea01046c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009900 tx=0x7f0e9002a410 comp rx=0 tx=0).stop 2024-07-26T12:46:35.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.257+0000 7f0ea6800640 1 -- 172.21.15.107:0/4170169077 shutdown_connections 2024-07-26T12:46:35.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.257+0000 7f0ea6800640 1 --2- 172.21.15.107:0/4170169077 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea010d860 0x7f0ea010fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:35.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.257+0000 7f0ea6800640 1 --2- 172.21.15.107:0/4170169077 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ea0104ce0 0x7f0ea010d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:35.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.257+0000 7f0ea6800640 1 --2- 172.21.15.107:0/4170169077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea01042b0 0x7f0ea01046c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:35.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.257+0000 7f0ea6800640 1 -- 172.21.15.107:0/4170169077 >> 172.21.15.107:0/4170169077 conn(0x7f0ea0079170 msgr2=0x7f0ea0079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:35.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.257+0000 7f0ea6800640 1 -- 172.21.15.107:0/4170169077 shutdown_connections 2024-07-26T12:46:35.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.257+0000 7f0ea6800640 1 -- 172.21.15.107:0/4170169077 wait complete. 2024-07-26T12:46:35.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.259+0000 7f0ea6800640 1 Processor -- start 2024-07-26T12:46:35.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.259+0000 7f0ea6800640 1 -- start start 2024-07-26T12:46:35.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.259+0000 7f0ea6800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ea01042b0 0x7f0ea0199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:35.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.259+0000 7f0ea6800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea0104ce0 0x7f0ea019a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:35.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0ea6800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea010d860 0x7f0ea019f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:35.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0ea6800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea01134d0 con 0x7f0ea0104ce0 2024-07-26T12:46:35.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0ea6800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea0113350 con 0x7f0ea010d860 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0ea6800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea0113650 con 0x7f0ea01042b0 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9ea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea0104ce0 0x7f0ea019a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ea01042b0 0x7f0ea0199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9ea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea0104ce0 0x7f0ea019a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44358/0 (socket says 172.21.15.107:44358) 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ea01042b0 0x7f0ea0199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:35224/0 (socket says 172.21.15.107:35224) 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9ea00640 1 -- 172.21.15.107:0/3076931085 learned_addr learned my addr 172.21.15.107:0/3076931085 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9fe00640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea010d860 0x7f0ea019f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9f400640 1 -- 172.21.15.107:0/3076931085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea010d860 msgr2=0x7f0ea019f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9f400640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea010d860 0x7f0ea019f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9f400640 1 -- 172.21.15.107:0/3076931085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea0104ce0 msgr2=0x7f0ea019a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9f400640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea0104ce0 0x7f0ea019a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:35.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9f400640 1 -- 172.21.15.107:0/3076931085 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea019fe50 con 0x7f0ea01042b0 2024-07-26T12:46:35.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.260+0000 7f0e9fe00640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea010d860 0x7f0ea019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:35.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.261+0000 7f0e9f400640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ea01042b0 0x7f0ea0199ee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0e9002a930 tx=0x7f0e90031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:35.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.261+0000 7f0e97e00640 1 -- 172.21.15.107:0/3076931085 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9003f030 con 0x7f0ea01042b0 2024-07-26T12:46:35.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.261+0000 7f0e97e00640 1 -- 172.21.15.107:0/3076931085 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e900043d0 con 0x7f0ea01042b0 2024-07-26T12:46:35.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.261+0000 7f0ea6800640 1 -- 172.21.15.107:0/3076931085 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea01a00e0 con 0x7f0ea01042b0 2024-07-26T12:46:35.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.261+0000 7f0e97e00640 1 -- 172.21.15.107:0/3076931085 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90035070 con 0x7f0ea01042b0 2024-07-26T12:46:35.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.261+0000 7f0ea6800640 1 -- 172.21.15.107:0/3076931085 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea01a68a0 con 0x7f0ea01042b0 2024-07-26T12:46:35.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.262+0000 7f0ea6800640 1 -- 172.21.15.107:0/3076931085 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ea006bbf0 con 0x7f0ea01042b0 2024-07-26T12:46:35.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.265+0000 7f0e97e00640 1 -- 172.21.15.107:0/3076931085 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f0e90042030 con 0x7f0ea01042b0 2024-07-26T12:46:35.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.267+0000 7f0e97e00640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0e840b3d30 0x7f0e840b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:35.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.267+0000 7f0e97e00640 1 -- 172.21.15.107:0/3076931085 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0e901046d0 con 0x7f0ea01042b0 2024-07-26T12:46:35.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.267+0000 7f0e9ea00640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0e840b3d30 0x7f0e840b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:35.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.268+0000 7f0e9ea00640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0e840b3d30 0x7f0e840b6240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0101e60 tx=0x7f0e88005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.268+0000 7f0e97e00640 1 -- 172.21.15.107:0/3076931085 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0e900cb6c0 con 0x7f0ea01042b0 2024-07-26T12:46:35.317 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874295 got 279172874300 for osd.10 2024-07-26T12:46:35.317 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:35.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.472+0000 7f0ea6800640 1 -- 172.21.15.107:0/3076931085 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f0ea0108a10 con 0x7f0ea01042b0 2024-07-26T12:46:35.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.473+0000 7f0e97e00640 1 -- 172.21.15.107:0/3076931085 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0e900cafc0 con 0x7f0ea01042b0 2024-07-26T12:46:35.474 INFO:teuthology.orchestra.run.smithi107.stdout:184683593811 2024-07-26T12:46:35.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 -- 172.21.15.107:0/3076931085 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0e840b3d30 msgr2=0x7f0e840b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:35.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0e840b3d30 0x7f0e840b6240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0101e60 tx=0x7f0e88005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 -- 172.21.15.107:0/3076931085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ea01042b0 msgr2=0x7f0ea0199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ea01042b0 0x7f0ea0199ee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0e9002a930 tx=0x7f0e90031a20 comp rx=0 tx=0).stop 2024-07-26T12:46:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 -- 172.21.15.107:0/3076931085 shutdown_connections 2024-07-26T12:46:35.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0e840b3d30 0x7f0e840b6240 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:35.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0ea010d860 0x7f0ea019f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:35.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ea0104ce0 0x7f0ea019a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:35.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 --2- 172.21.15.107:0/3076931085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ea01042b0 0x7f0ea0199ee0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:35.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 -- 172.21.15.107:0/3076931085 >> 172.21.15.107:0/3076931085 conn(0x7f0ea0079170 msgr2=0x7f0ea01002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:35.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 -- 172.21.15.107:0/3076931085 shutdown_connections 2024-07-26T12:46:35.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:35.477+0000 7f0ea6800640 1 -- 172.21.15.107:0/3076931085 wait complete. 2024-07-26T12:46:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:35 smithi132 ceph-mon[37580]: pgmap v408: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:46:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:35 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3076931085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T12:46:36.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:35 smithi107 ceph-mon[32659]: pgmap v408: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:46:36.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:35 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3076931085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T12:46:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:35 smithi112 ceph-mon[37501]: pgmap v408: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:46:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:35 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3076931085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T12:46:36.681 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593805 got 184683593811 for osd.6 2024-07-26T12:46:36.682 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:36.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.882+0000 7f29cf600640 1 -- 172.21.15.107:0/3774161223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29c810ba70 msgr2=0x7f29c81122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.882+0000 7f29cf600640 1 --2- 172.21.15.107:0/3774161223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29c810ba70 0x7f29c81122a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009be0 tx=0x7f29c002a6c0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.882+0000 7f29cf600640 1 -- 172.21.15.107:0/3774161223 shutdown_connections 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.882+0000 7f29cf600640 1 --2- 172.21.15.107:0/3774161223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29c810ba70 0x7f29c81122a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.882+0000 7f29cf600640 1 --2- 172.21.15.107:0/3774161223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29c810b120 0x7f29c810b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.882+0000 7f29cf600640 1 --2- 172.21.15.107:0/3774161223 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29c81057e0 0x7f29c8105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.882+0000 7f29cf600640 1 -- 172.21.15.107:0/3774161223 >> 172.21.15.107:0/3774161223 conn(0x7f29c81011f0 msgr2=0x7f29c8103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.882+0000 7f29cf600640 1 -- 172.21.15.107:0/3774161223 shutdown_connections 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.882+0000 7f29cf600640 1 -- 172.21.15.107:0/3774161223 wait complete. 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.884+0000 7f29cf600640 1 Processor -- start 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.884+0000 7f29cf600640 1 -- start start 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29cf600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29c81057e0 0x7f29c819c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:36.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29cf600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29c810b120 0x7f29c819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29cf600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29c810ba70 0x7f29c81a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29cf600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f29c8115740 con 0x7f29c81057e0 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29cf600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f29c81155c0 con 0x7f29c810b120 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29cf600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f29c81158c0 con 0x7f29c810ba70 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29ccc00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29c81057e0 0x7f29c819c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29ccc00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29c81057e0 0x7f29c819c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47280/0 (socket says 172.21.15.107:47280) 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29ccc00640 1 -- 172.21.15.107:0/3868838140 learned_addr learned my addr 172.21.15.107:0/3868838140 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29cd600640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29c810ba70 0x7f29c81a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29c7e00640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29c810b120 0x7f29c819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29ccc00640 1 -- 172.21.15.107:0/3868838140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29c810ba70 msgr2=0x7f29c81a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.885+0000 7f29ccc00640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29c810ba70 0x7f29c81a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.886+0000 7f29ccc00640 1 -- 172.21.15.107:0/3868838140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29c810b120 msgr2=0x7f29c819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:36.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.886+0000 7f29ccc00640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29c810b120 0x7f29c819c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.886+0000 7f29ccc00640 1 -- 172.21.15.107:0/3868838140 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c81a20c0 con 0x7f29c81057e0 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.886+0000 7f29c7e00640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29c810b120 0x7f29c819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.886+0000 7f29ccc00640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29c81057e0 0x7f29c819c150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00e9f0 tx=0x7f29bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.886+0000 7f29c5600640 1 -- 172.21.15.107:0/3868838140 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc00ccc0 con 0x7f29c81057e0 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.886+0000 7f29c5600640 1 -- 172.21.15.107:0/3868838140 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29bc0054d0 con 0x7f29c81057e0 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.886+0000 7f29c5600640 1 -- 172.21.15.107:0/3868838140 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc010430 con 0x7f29c81057e0 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.886+0000 7f29cf600640 1 -- 172.21.15.107:0/3868838140 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c81a23b0 con 0x7f29c81057e0 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.886+0000 7f29cf600640 1 -- 172.21.15.107:0/3868838140 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c81a8b10 con 0x7f29c81057e0 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.888+0000 7f29cf600640 1 -- 172.21.15.107:0/3868838140 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c8105bf0 con 0x7f29c81057e0 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.890+0000 7f29c5600640 1 -- 172.21.15.107:0/3868838140 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f29bc0046c0 con 0x7f29c81057e0 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.891+0000 7f29c5600640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f29b00b3d30 0x7f29b00b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:36.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.892+0000 7f29c5600640 1 -- 172.21.15.107:0/3868838140 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f29bc0e45c0 con 0x7f29c81057e0 2024-07-26T12:46:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.892+0000 7f29c7e00640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f29b00b3d30 0x7f29b00b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.893+0000 7f29c7e00640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f29b00b3d30 0x7f29b00b6240 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f29c8103a90 tx=0x7f29b8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:30.896+0000 7f29c5600640 1 -- 172.21.15.107:0/3868838140 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f29bc0ab5b0 con 0x7f29c81057e0 2024-07-26T12:46:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.101+0000 7f29cf600640 1 -- 172.21.15.107:0/3868838140 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f29c819d280 con 0x7f29c81057e0 2024-07-26T12:46:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.102+0000 7f29c5600640 1 -- 172.21.15.107:0/3868838140 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f29bc0aaeb0 con 0x7f29c81057e0 2024-07-26T12:46:36.847 INFO:teuthology.orchestra.run.smithi107.stdout:206158430284 2024-07-26T12:46:36.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 -- 172.21.15.107:0/3868838140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f29b00b3d30 msgr2=0x7f29b00b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f29b00b3d30 0x7f29b00b6240 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f29c8103a90 tx=0x7f29b8005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 -- 172.21.15.107:0/3868838140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29c81057e0 msgr2=0x7f29c819c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29c81057e0 0x7f29c819c150 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00e9f0 tx=0x7f29bc00eeb0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 -- 172.21.15.107:0/3868838140 shutdown_connections 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f29b00b3d30 0x7f29b00b6240 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29c810ba70 0x7f29c81a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f29c810b120 0x7f29c819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 --2- 172.21.15.107:0/3868838140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29c81057e0 0x7f29c819c150 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 -- 172.21.15.107:0/3868838140 >> 172.21.15.107:0/3868838140 conn(0x7f29c81011f0 msgr2=0x7f29c8104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 -- 172.21.15.107:0/3868838140 shutdown_connections 2024-07-26T12:46:36.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:31.106+0000 7f29cf600640 1 -- 172.21.15.107:0/3868838140 wait complete. 2024-07-26T12:46:37.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:36 smithi107 ceph-mon[32659]: mgrmap e22: x(active, since 11m), standbys: z, y 2024-07-26T12:46:37.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:36 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1747187321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T12:46:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:36 smithi112 ceph-mon[37501]: mgrmap e22: x(active, since 11m), standbys: z, y 2024-07-26T12:46:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:36 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1747187321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T12:46:37.212 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790036 got 158913790041 for osd.5 2024-07-26T12:46:37.212 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:36 smithi132 ceph-mon[37580]: mgrmap e22: x(active, since 11m), standbys: z, y 2024-07-26T12:46:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:36 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1747187321' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T12:46:37.706 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430278 got 206158430284 for osd.7 2024-07-26T12:46:37.707 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:37 smithi107 ceph-mon[32659]: pgmap v409: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-07-26T12:46:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.493+0000 7fce13e00640 1 -- 172.21.15.107:0/1505460357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c10b0f0 msgr2=0x7fce0c10b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.493+0000 7fce13e00640 1 --2- 172.21.15.107:0/1505460357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c10b0f0 0x7fce0c10b500 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009900 tx=0x7fcdf802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:46:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.493+0000 7fce13e00640 1 -- 172.21.15.107:0/1505460357 shutdown_connections 2024-07-26T12:46:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.493+0000 7fce13e00640 1 --2- 172.21.15.107:0/1505460357 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce0c10ba40 0x7fce0c112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.493+0000 7fce13e00640 1 --2- 172.21.15.107:0/1505460357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c10b0f0 0x7fce0c10b500 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.493+0000 7fce13e00640 1 --2- 172.21.15.107:0/1505460357 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce0c1057b0 0x7fce0c105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:38.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.493+0000 7fce13e00640 1 -- 172.21.15.107:0/1505460357 >> 172.21.15.107:0/1505460357 conn(0x7fce0c101240 msgr2=0x7fce0c103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.493+0000 7fce13e00640 1 -- 172.21.15.107:0/1505460357 shutdown_connections 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.493+0000 7fce13e00640 1 -- 172.21.15.107:0/1505460357 wait complete. 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.494+0000 7fce13e00640 1 Processor -- start 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.495+0000 7fce13e00640 1 -- start start 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.495+0000 7fce13e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce0c1057b0 0x7fce0c19c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.495+0000 7fce13e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c10b0f0 0x7fce0c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.495+0000 7fce13e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce0c10ba40 0x7fce0c1a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.495+0000 7fce13e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c115700 con 0x7fce0c10b0f0 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.495+0000 7fce13e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c115580 con 0x7fce0c1057b0 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.495+0000 7fce13e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c115880 con 0x7fce0c10ba40 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.495+0000 7fce10a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c10b0f0 0x7fce0c19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:38.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.496+0000 7fce11400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce0c1057b0 0x7fce0c19c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.496+0000 7fce10a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c10b0f0 0x7fce0c19c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44372/0 (socket says 172.21.15.107:44372) 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.496+0000 7fce10a00640 1 -- 172.21.15.107:0/1747187321 learned_addr learned my addr 172.21.15.107:0/1747187321 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.496+0000 7fce11e00640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce0c10ba40 0x7fce0c1a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.496+0000 7fce10a00640 1 -- 172.21.15.107:0/1747187321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce0c10ba40 msgr2=0x7fce0c1a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.496+0000 7fce10a00640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce0c10ba40 0x7fce0c1a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.496+0000 7fce10a00640 1 -- 172.21.15.107:0/1747187321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce0c1057b0 msgr2=0x7fce0c19c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.496+0000 7fce10a00640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce0c1057b0 0x7fce0c19c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.496+0000 7fce10a00640 1 -- 172.21.15.107:0/1747187321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce0c1a2060 con 0x7fce0c10b0f0 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.497+0000 7fce10a00640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c10b0f0 0x7fce0c19c690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8005ab0 tx=0x7fcdf8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.497+0000 7fce0a000640 1 -- 172.21.15.107:0/1747187321 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf803f070 con 0x7fce0c10b0f0 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.497+0000 7fce0a000640 1 -- 172.21.15.107:0/1747187321 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdf8031b50 con 0x7fce0c10b0f0 2024-07-26T12:46:38.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.497+0000 7fce13e00640 1 -- 172.21.15.107:0/1747187321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce0c1a22f0 con 0x7fce0c10b0f0 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.497+0000 7fce0a000640 1 -- 172.21.15.107:0/1747187321 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8035030 con 0x7fce0c10b0f0 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.497+0000 7fce13e00640 1 -- 172.21.15.107:0/1747187321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce0c1a8ab0 con 0x7fce0c10b0f0 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.499+0000 7fce13e00640 1 -- 172.21.15.107:0/1747187321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce0c105bc0 con 0x7fce0c10b0f0 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.501+0000 7fce0a000640 1 -- 172.21.15.107:0/1747187321 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fcdf8042070 con 0x7fce0c10b0f0 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.503+0000 7fce0a000640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcdf00b3ca0 0x7fcdf00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.503+0000 7fce0a000640 1 -- 172.21.15.107:0/1747187321 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcdf81049c0 con 0x7fce0c10b0f0 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.503+0000 7fce11400640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcdf00b3ca0 0x7fcdf00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.504+0000 7fce11400640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcdf00b3ca0 0x7fcdf00b61b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc009bd0 tx=0x7fcdfc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.505+0000 7fce0a000640 1 -- 172.21.15.107:0/1747187321 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcdf80cb4c0 con 0x7fce0c10b0f0 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.711+0000 7fce13e00640 1 -- 172.21.15.107:0/1747187321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fce0c02e0c0 con 0x7fce0c10b0f0 2024-07-26T12:46:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.712+0000 7fce0a000640 1 -- 172.21.15.107:0/1747187321 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcdf80cfa10 con 0x7fce0c10b0f0 2024-07-26T12:46:38.069 INFO:teuthology.orchestra.run.smithi107.stdout:38654705779 2024-07-26T12:46:38.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 -- 172.21.15.107:0/1747187321 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcdf00b3ca0 msgr2=0x7fcdf00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:38.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcdf00b3ca0 0x7fcdf00b61b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc009bd0 tx=0x7fcdfc005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:38.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 -- 172.21.15.107:0/1747187321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c10b0f0 msgr2=0x7fce0c19c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:38.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c10b0f0 0x7fce0c19c690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8005ab0 tx=0x7fcdf8003b90 comp rx=0 tx=0).stop 2024-07-26T12:46:38.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 -- 172.21.15.107:0/1747187321 shutdown_connections 2024-07-26T12:46:38.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcdf00b3ca0 0x7fcdf00b61b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:38.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fce0c10ba40 0x7fce0c1a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:38.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce0c10b0f0 0x7fce0c19c690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:38.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 --2- 172.21.15.107:0/1747187321 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fce0c1057b0 0x7fce0c19c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:38.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 -- 172.21.15.107:0/1747187321 >> 172.21.15.107:0/1747187321 conn(0x7fce0c101240 msgr2=0x7fce0c103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:38.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 -- 172.21.15.107:0/1747187321 shutdown_connections 2024-07-26T12:46:38.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:36.715+0000 7fce13e00640 1 -- 172.21.15.107:0/1747187321 wait complete. 2024-07-26T12:46:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:37 smithi112 ceph-mon[37501]: pgmap v409: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-07-26T12:46:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:37 smithi132 ceph-mon[37580]: pgmap v409: 129 pgs: 129 active+clean; 646 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-07-26T12:46:38.805 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705772 got 38654705779 for osd.0 2024-07-26T12:46:38.805 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:38.895 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:38.936 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953561 got 137438953565 for osd.4 2024-07-26T12:46:38.936 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:40.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:39 smithi107 ceph-mon[32659]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T12:46:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:39 smithi112 ceph-mon[37501]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T12:46:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:39 smithi132 ceph-mon[37580]: pgmap v410: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T12:46:41.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.290+0000 7f7724c00640 1 -- 172.21.15.107:0/375737182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f772010f330 msgr2=0x7f772010f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:41.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.290+0000 7f7724c00640 1 --2- 172.21.15.107:0/375737182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f772010f330 0x7f772010f740 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7708009900 tx=0x7f770802a410 comp rx=0 tx=0).stop 2024-07-26T12:46:41.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.290+0000 7f7724c00640 1 -- 172.21.15.107:0/375737182 shutdown_connections 2024-07-26T12:46:41.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.290+0000 7f7724c00640 1 --2- 172.21.15.107:0/375737182 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010fc80 0x7f77201164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:41.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.290+0000 7f7724c00640 1 --2- 172.21.15.107:0/375737182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f772010f330 0x7f772010f740 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:41.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.290+0000 7f7724c00640 1 --2- 172.21.15.107:0/375737182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77201099f0 0x7f7720109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:41.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.290+0000 7f7724c00640 1 -- 172.21.15.107:0/375737182 >> 172.21.15.107:0/375737182 conn(0x7f7720078ff0 msgr2=0x7f7720107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:41.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.290+0000 7f7724c00640 1 -- 172.21.15.107:0/375737182 shutdown_connections 2024-07-26T12:46:41.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.290+0000 7f7724c00640 1 -- 172.21.15.107:0/375737182 wait complete. 2024-07-26T12:46:41.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.291+0000 7f7724c00640 1 Processor -- start 2024-07-26T12:46:41.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.291+0000 7f7724c00640 1 -- start start 2024-07-26T12:46:41.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f7724c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77201099f0 0x7f77201a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:41.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f7724c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010f330 0x7f77201a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:41.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f7724c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f772010fc80 0x7f77201a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:41.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f7724c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7720119940 con 0x7f77201099f0 2024-07-26T12:46:41.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f7724c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f77201197c0 con 0x7f772010f330 2024-07-26T12:46:41.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f7724c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7720119ac0 con 0x7f772010fc80 2024-07-26T12:46:41.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f771f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77201099f0 0x7f77201a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:41.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f771f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77201099f0 0x7f77201a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:44394/0 (socket says 172.21.15.107:44394) 2024-07-26T12:46:41.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f771f400640 1 -- 172.21.15.107:0/1428755106 learned_addr learned my addr 172.21.15.107:0/1428755106 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:41.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f7716a00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010f330 0x7f77201a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:41.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.292+0000 7f771fe00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f772010fc80 0x7f77201a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.293+0000 7f7716a00640 1 -- 172.21.15.107:0/1428755106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f772010fc80 msgr2=0x7f77201a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.293+0000 7f7716a00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f772010fc80 0x7f77201a5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.293+0000 7f7716a00640 1 -- 172.21.15.107:0/1428755106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77201099f0 msgr2=0x7f77201a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.293+0000 7f7716a00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77201099f0 0x7f77201a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.293+0000 7f7716a00640 1 -- 172.21.15.107:0/1428755106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77201a6340 con 0x7f772010f330 2024-07-26T12:46:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.293+0000 7f771f400640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77201099f0 0x7f77201a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.293+0000 7f7716a00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010f330 0x7f77201a0970 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f770802a930 tx=0x7f7708031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:41.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.294+0000 7f771cc00640 1 -- 172.21.15.107:0/1428755106 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770803f070 con 0x7f772010f330 2024-07-26T12:46:41.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.294+0000 7f7724c00640 1 -- 172.21.15.107:0/1428755106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77201a65d0 con 0x7f772010f330 2024-07-26T12:46:41.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.294+0000 7f771cc00640 1 -- 172.21.15.107:0/1428755106 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7708004490 con 0x7f772010f330 2024-07-26T12:46:41.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.294+0000 7f771cc00640 1 -- 172.21.15.107:0/1428755106 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7708035070 con 0x7f772010f330 2024-07-26T12:46:41.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.294+0000 7f7724c00640 1 -- 172.21.15.107:0/1428755106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77201acd90 con 0x7f772010f330 2024-07-26T12:46:41.296 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.295+0000 7f7724c00640 1 -- 172.21.15.107:0/1428755106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7720109e00 con 0x7f772010f330 2024-07-26T12:46:41.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.298+0000 7f771cc00640 1 -- 172.21.15.107:0/1428755106 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f770803b460 con 0x7f772010f330 2024-07-26T12:46:41.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.300+0000 7f771cc00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f77040b3c20 0x7f77040b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:41.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.300+0000 7f771cc00640 1 -- 172.21.15.107:0/1428755106 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7708104610 con 0x7f772010f330 2024-07-26T12:46:41.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.300+0000 7f771f400640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f77040b3c20 0x7f77040b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:41.301 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.301+0000 7f771f400640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f77040b3c20 0x7f77040b6130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7710011500 tx=0x7f7710005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:41.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.301+0000 7f771cc00640 1 -- 172.21.15.107:0/1428755106 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f77080cb110 con 0x7f772010f330 2024-07-26T12:46:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.504+0000 7f7724c00640 1 -- 172.21.15.107:0/1428755106 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f7720111a50 con 0x7f772010f330 2024-07-26T12:46:41.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.505+0000 7f771cc00640 1 -- 172.21.15.107:0/1428755106 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f77080cf660 con 0x7f772010f330 2024-07-26T12:46:41.506 INFO:teuthology.orchestra.run.smithi107.stdout:257698037826 2024-07-26T12:46:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 -- 172.21.15.107:0/1428755106 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f77040b3c20 msgr2=0x7f77040b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f77040b3c20 0x7f77040b6130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7710011500 tx=0x7f7710005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 -- 172.21.15.107:0/1428755106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010f330 msgr2=0x7f77201a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010f330 0x7f77201a0970 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f770802a930 tx=0x7f7708031a20 comp rx=0 tx=0).stop 2024-07-26T12:46:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 -- 172.21.15.107:0/1428755106 shutdown_connections 2024-07-26T12:46:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f77040b3c20 0x7f77040b6130 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f772010fc80 0x7f77201a5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f772010f330 0x7f77201a0970 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 --2- 172.21.15.107:0/1428755106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77201099f0 0x7f77201a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 -- 172.21.15.107:0/1428755106 >> 172.21.15.107:0/1428755106 conn(0x7f7720078ff0 msgr2=0x7f77200797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.508+0000 7f7724c00640 1 -- 172.21.15.107:0/1428755106 shutdown_connections 2024-07-26T12:46:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:41.509+0000 7f7724c00640 1 -- 172.21.15.107:0/1428755106 wait complete. 2024-07-26T12:46:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:41 smithi112 ceph-mon[37501]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T12:46:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:41 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1428755106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T12:46:42.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:41 smithi107 ceph-mon[32659]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T12:46:42.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:41 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1428755106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T12:46:42.308 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037820 got 257698037826 for osd.9 2024-07-26T12:46:42.308 DEBUG:teuthology.parallel:result is None 2024-07-26T12:46:42.309 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T12:46:42.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T12:46:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:41 smithi132 ceph-mon[37580]: pgmap v411: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T12:46:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:41 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1428755106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T12:46:42.729 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:43.141 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:42 smithi107 ceph-mon[32659]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 596 B/s wr, 12 op/s 2024-07-26T12:46:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:42 smithi112 ceph-mon[37501]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 596 B/s wr, 12 op/s 2024-07-26T12:46:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:42 smithi132 ceph-mon[37580]: pgmap v412: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 596 B/s wr, 12 op/s 2024-07-26T12:46:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.056+0000 7fe490c00640 1 -- 172.21.15.107:0/1809877440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe48c069b10 msgr2=0x7fe48c106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.056+0000 7fe490c00640 1 --2- 172.21.15.107:0/1809877440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe48c069b10 0x7fe48c106f30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009900 tx=0x7fe47c02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:46:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.056+0000 7fe490c00640 1 -- 172.21.15.107:0/1809877440 shutdown_connections 2024-07-26T12:46:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.056+0000 7fe490c00640 1 --2- 172.21.15.107:0/1809877440 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe48c10c0a0 0x7fe48c10e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.056+0000 7fe490c00640 1 --2- 172.21.15.107:0/1809877440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48c101f60 0x7fe48c107470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.056+0000 7fe490c00640 1 --2- 172.21.15.107:0/1809877440 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe48c069b10 0x7fe48c106f30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.056+0000 7fe490c00640 1 -- 172.21.15.107:0/1809877440 >> 172.21.15.107:0/1809877440 conn(0x7fe48c0780b0 msgr2=0x7fe48c0784f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.056+0000 7fe490c00640 1 -- 172.21.15.107:0/1809877440 shutdown_connections 2024-07-26T12:46:45.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.056+0000 7fe490c00640 1 -- 172.21.15.107:0/1809877440 wait complete. 2024-07-26T12:46:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.058+0000 7fe490c00640 1 Processor -- start 2024-07-26T12:46:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.058+0000 7fe490c00640 1 -- start start 2024-07-26T12:46:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.058+0000 7fe490c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe48c069b10 0x7fe48c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.058+0000 7fe490c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe48c101f60 0x7fe48c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.058+0000 7fe490c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48c10c0a0 0x7fe48c19f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.058+0000 7fe490c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c113340 con 0x7fe48c10c0a0 2024-07-26T12:46:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.058+0000 7fe490c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c1131c0 con 0x7fe48c069b10 2024-07-26T12:46:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.058+0000 7fe490c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe48c1134c0 con 0x7fe48c101f60 2024-07-26T12:46:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe483e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe48c101f60 0x7fe48c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe48b400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe48c069b10 0x7fe48c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe483e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe48c101f60 0x7fe48c19a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:57660/0 (socket says 172.21.15.107:57660) 2024-07-26T12:46:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe483e00640 1 -- 172.21.15.107:0/3974223618 learned_addr learned my addr 172.21.15.107:0/3974223618 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe48be00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48c10c0a0 0x7fe48c19f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe48b400640 1 -- 172.21.15.107:0/3974223618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe48c101f60 msgr2=0x7fe48c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe48b400640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe48c101f60 0x7fe48c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:45.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe48b400640 1 -- 172.21.15.107:0/3974223618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48c10c0a0 msgr2=0x7fe48c19f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe48b400640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48c10c0a0 0x7fe48c19f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe48b400640 1 -- 172.21.15.107:0/3974223618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe48c19fc60 con 0x7fe48c069b10 2024-07-26T12:46:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.059+0000 7fe483e00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe48c101f60 0x7fe48c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.060+0000 7fe48b400640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe48c069b10 0x7fe48c199cf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe47c02a9e0 tx=0x7fe47c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.060+0000 7fe488c00640 1 -- 172.21.15.107:0/3974223618 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c03f070 con 0x7fe48c069b10 2024-07-26T12:46:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.060+0000 7fe490c00640 1 -- 172.21.15.107:0/3974223618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe48c19fef0 con 0x7fe48c069b10 2024-07-26T12:46:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.060+0000 7fe488c00640 1 -- 172.21.15.107:0/3974223618 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe47c031b50 con 0x7fe48c069b10 2024-07-26T12:46:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.060+0000 7fe488c00640 1 -- 172.21.15.107:0/3974223618 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c035030 con 0x7fe48c069b10 2024-07-26T12:46:45.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.060+0000 7fe490c00640 1 -- 172.21.15.107:0/3974223618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe48c1a66b0 con 0x7fe48c069b10 2024-07-26T12:46:45.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.061+0000 7fe490c00640 1 -- 172.21.15.107:0/3974223618 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe48c106f30 con 0x7fe48c069b10 2024-07-26T12:46:45.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.064+0000 7fe488c00640 1 -- 172.21.15.107:0/3974223618 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fe47c042070 con 0x7fe48c069b10 2024-07-26T12:46:45.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.065+0000 7fe488c00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4700b3c20 0x7fe4700b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:45.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.065+0000 7fe488c00640 1 -- 172.21.15.107:0/3974223618 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe47c104900 con 0x7fe48c069b10 2024-07-26T12:46:45.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.066+0000 7fe483e00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4700b3c20 0x7fe4700b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:45.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.066+0000 7fe483e00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4700b3c20 0x7fe4700b6130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe48c10e7c0 tx=0x7fe474005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:45.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.067+0000 7fe488c00640 1 -- 172.21.15.107:0/3974223618 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe47c031d00 con 0x7fe48c069b10 2024-07-26T12:46:45.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.264+0000 7fe490c00640 1 -- 172.21.15.107:0/3974223618 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe48c105450 con 0x7fe4700b3c20 2024-07-26T12:46:45.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.275+0000 7fe488c00640 1 -- 172.21.15.107:0/3974223618 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340515 (secure 0 0 0) 0x7fe48c105450 con 0x7fe4700b3c20 2024-07-26T12:46:45.280 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:46:45.284 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 -- 172.21.15.107:0/3974223618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4700b3c20 msgr2=0x7fe4700b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4700b3c20 0x7fe4700b6130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe48c10e7c0 tx=0x7fe474005f90 comp rx=0 tx=0).stop 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 -- 172.21.15.107:0/3974223618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe48c069b10 msgr2=0x7fe48c199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe48c069b10 0x7fe48c199cf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe47c02a9e0 tx=0x7fe47c003b90 comp rx=0 tx=0).stop 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 -- 172.21.15.107:0/3974223618 shutdown_connections 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe4700b3c20 0x7fe4700b6130 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe48c10c0a0 0x7fe48c19f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe48c101f60 0x7fe48c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 --2- 172.21.15.107:0/3974223618 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe48c069b10 0x7fe48c199cf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 -- 172.21.15.107:0/3974223618 >> 172.21.15.107:0/3974223618 conn(0x7fe48c0780b0 msgr2=0x7fe48c104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 -- 172.21.15.107:0/3974223618 shutdown_connections 2024-07-26T12:46:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:45.284+0000 7fe490c00640 1 -- 172.21.15.107:0/3974223618 wait complete. 2024-07-26T12:46:46.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:45 smithi107 ceph-mon[32659]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T12:46:46.105 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2024-07-26T12:46:44.752342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662473,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1356,"num_read_kb":785,"num_write":385,"num_write_kb":2303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2951874,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":818,"ondisk_log_size":818,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376064,"kb_used_data":14432,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565760,"statfs":{"total":1151940427776,"available":1151555338240,"internally_reserved":0,"allocated":14778368,"data_stored":8386421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":118,"num_read_kb":58,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006347"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":182,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:40.125743+0000","last_change":"2024-07-26T12:43:08.129417+0000","last_active":"2024-07-26T12:46:40.125743+0000","last_peered":"2024-07-26T12:46:40.125743+0000","last_clean":"2024-07-26T12:46:40.125743+0000","last_became_active":"2024-07-26T12:43:08.128919+0000","last_became_peered":"2024-07-26T12:43:08.128919+0000","last_unstale":"2024-07-26T12:46:40.125743+0000","last_undegraded":"2024-07-26T12:46:40.125743+0000","last_fullsized":"2024-07-26T12:46:40.125743+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:35:55.414185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":123,"num_read_kb":123,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.657938+0000","last_change":"2024-07-26T12:43:12.172058+0000","last_active":"2024-07-26T12:46:34.657938+0000","last_peered":"2024-07-26T12:46:34.657938+0000","last_clean":"2024-07-26T12:46:34.657938+0000","last_became_active":"2024-07-26T12:43:12.171631+0000","last_became_peered":"2024-07-26T12:43:12.171631+0000","last_unstale":"2024-07-26T12:46:34.657938+0000","last_undegraded":"2024-07-26T12:46:34.657938+0000","last_fullsized":"2024-07-26T12:46:34.657938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:51.972962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.802710+0000","last_change":"2024-07-26T12:43:26.773739+0000","last_active":"2024-07-26T12:44:36.802710+0000","last_peered":"2024-07-26T12:44:36.802710+0000","last_clean":"2024-07-26T12:44:36.802710+0000","last_became_active":"2024-07-26T12:43:26.773254+0000","last_became_peered":"2024-07-26T12:43:26.773254+0000","last_unstale":"2024-07-26T12:44:36.802710+0000","last_undegraded":"2024-07-26T12:44:36.802710+0000","last_fullsized":"2024-07-26T12:44:36.802710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:03.125497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.852995+0000","last_change":"2024-07-26T12:43:08.227329+0000","last_active":"2024-07-26T12:44:36.852995+0000","last_peered":"2024-07-26T12:44:36.852995+0000","last_clean":"2024-07-26T12:44:36.852995+0000","last_became_active":"2024-07-26T12:43:08.226674+0000","last_became_peered":"2024-07-26T12:43:08.226674+0000","last_unstale":"2024-07-26T12:44:36.852995+0000","last_undegraded":"2024-07-26T12:44:36.852995+0000","last_fullsized":"2024-07-26T12:44:36.852995+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.222684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674400+0000","last_change":"2024-07-26T12:43:12.190616+0000","last_active":"2024-07-26T12:46:34.674400+0000","last_peered":"2024-07-26T12:46:34.674400+0000","last_clean":"2024-07-26T12:46:34.674400+0000","last_became_active":"2024-07-26T12:43:12.189717+0000","last_became_peered":"2024-07-26T12:43:12.189717+0000","last_unstale":"2024-07-26T12:46:34.674400+0000","last_undegraded":"2024-07-26T12:46:34.674400+0000","last_fullsized":"2024-07-26T12:46:34.674400+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:03.509592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850323+0000","last_change":"2024-07-26T12:43:26.774486+0000","last_active":"2024-07-26T12:44:36.850323+0000","last_peered":"2024-07-26T12:44:36.850323+0000","last_clean":"2024-07-26T12:44:36.850323+0000","last_became_active":"2024-07-26T12:43:26.774046+0000","last_became_peered":"2024-07-26T12:43:26.774046+0000","last_unstale":"2024-07-26T12:44:36.850323+0000","last_undegraded":"2024-07-26T12:44:36.850323+0000","last_fullsized":"2024-07-26T12:44:36.850323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.847072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881403+0000","last_change":"2024-07-26T12:43:08.123084+0000","last_active":"2024-07-26T12:44:39.881403+0000","last_peered":"2024-07-26T12:44:39.881403+0000","last_clean":"2024-07-26T12:44:39.881403+0000","last_became_active":"2024-07-26T12:43:08.122371+0000","last_became_peered":"2024-07-26T12:43:08.122371+0000","last_unstale":"2024-07-26T12:44:39.881403+0000","last_undegraded":"2024-07-26T12:44:39.881403+0000","last_fullsized":"2024-07-26T12:44:39.881403+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:27:09.665660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.670134+0000","last_change":"2024-07-26T12:43:12.170290+0000","last_active":"2024-07-26T12:46:34.670134+0000","last_peered":"2024-07-26T12:46:34.670134+0000","last_clean":"2024-07-26T12:46:34.670134+0000","last_became_active":"2024-07-26T12:43:12.169849+0000","last_became_peered":"2024-07-26T12:43:12.169849+0000","last_unstale":"2024-07-26T12:46:34.670134+0000","last_undegraded":"2024-07-26T12:46:34.670134+0000","last_fullsized":"2024-07-26T12:46:34.670134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:46.044921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850576+0000","last_change":"2024-07-26T12:43:26.764804+0000","last_active":"2024-07-26T12:44:36.850576+0000","last_peered":"2024-07-26T12:44:36.850576+0000","last_clean":"2024-07-26T12:44:36.850576+0000","last_became_active":"2024-07-26T12:43:26.764065+0000","last_became_peered":"2024-07-26T12:43:26.764065+0000","last_unstale":"2024-07-26T12:44:36.850576+0000","last_undegraded":"2024-07-26T12:44:36.850576+0000","last_fullsized":"2024-07-26T12:44:36.850576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:16:59.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.886417+0000","last_change":"2024-07-26T12:43:08.121496+0000","last_active":"2024-07-26T12:44:39.886417+0000","last_peered":"2024-07-26T12:44:39.886417+0000","last_clean":"2024-07-26T12:44:39.886417+0000","last_became_active":"2024-07-26T12:43:08.120962+0000","last_became_peered":"2024-07-26T12:43:08.120962+0000","last_unstale":"2024-07-26T12:44:39.886417+0000","last_undegraded":"2024-07-26T12:44:39.886417+0000","last_fullsized":"2024-07-26T12:44:39.886417+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:42:02.998023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655057+0000","last_change":"2024-07-26T12:43:12.168707+0000","last_active":"2024-07-26T12:46:34.655057+0000","last_peered":"2024-07-26T12:46:34.655057+0000","last_clean":"2024-07-26T12:46:34.655057+0000","last_became_active":"2024-07-26T12:43:12.168338+0000","last_became_peered":"2024-07-26T12:43:12.168338+0000","last_unstale":"2024-07-26T12:46:34.655057+0000","last_undegraded":"2024-07-26T12:46:34.655057+0000","last_fullsized":"2024-07-26T12:46:34.655057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:39:44.724864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850496+0000","last_change":"2024-07-26T12:43:26.770863+0000","last_active":"2024-07-26T12:44:36.850496+0000","last_peered":"2024-07-26T12:44:36.850496+0000","last_clean":"2024-07-26T12:44:36.850496+0000","last_became_active":"2024-07-26T12:43:26.770480+0000","last_became_peered":"2024-07-26T12:43:26.770480+0000","last_unstale":"2024-07-26T12:44:36.850496+0000","last_undegraded":"2024-07-26T12:44:36.850496+0000","last_fullsized":"2024-07-26T12:44:36.850496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:09.128364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672434+0000","last_change":"2024-07-26T12:43:12.181659+0000","last_active":"2024-07-26T12:46:34.672434+0000","last_peered":"2024-07-26T12:46:34.672434+0000","last_clean":"2024-07-26T12:46:34.672434+0000","last_became_active":"2024-07-26T12:43:12.181268+0000","last_became_peered":"2024-07-26T12:43:12.181268+0000","last_unstale":"2024-07-26T12:46:34.672434+0000","last_undegraded":"2024-07-26T12:46:34.672434+0000","last_fullsized":"2024-07-26T12:46:34.672434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:38:28.445981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881363+0000","last_change":"2024-07-26T12:43:35.856273+0000","last_active":"2024-07-26T12:44:39.881363+0000","last_peered":"2024-07-26T12:44:39.881363+0000","last_clean":"2024-07-26T12:44:39.881363+0000","last_became_active":"2024-07-26T12:43:35.849625+0000","last_became_peered":"2024-07-26T12:43:35.849625+0000","last_unstale":"2024-07-26T12:44:39.881363+0000","last_undegraded":"2024-07-26T12:44:39.881363+0000","last_fullsized":"2024-07-26T12:44:39.881363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:34:45.261474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850698+0000","last_change":"2024-07-26T12:43:26.793431+0000","last_active":"2024-07-26T12:44:36.850698+0000","last_peered":"2024-07-26T12:44:36.850698+0000","last_clean":"2024-07-26T12:44:36.850698+0000","last_became_active":"2024-07-26T12:43:26.792931+0000","last_became_peered":"2024-07-26T12:43:26.792931+0000","last_unstale":"2024-07-26T12:44:36.850698+0000","last_undegraded":"2024-07-26T12:44:36.850698+0000","last_fullsized":"2024-07-26T12:44:36.850698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:31.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669836+0000","last_change":"2024-07-26T12:43:12.183784+0000","last_active":"2024-07-26T12:46:34.669836+0000","last_peered":"2024-07-26T12:46:34.669836+0000","last_clean":"2024-07-26T12:46:34.669836+0000","last_became_active":"2024-07-26T12:43:12.182696+0000","last_became_peered":"2024-07-26T12:43:12.182696+0000","last_unstale":"2024-07-26T12:46:34.669836+0000","last_undegraded":"2024-07-26T12:46:34.669836+0000","last_fullsized":"2024-07-26T12:46:34.669836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:33:19.918180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.860356+0000","last_change":"2024-07-26T12:43:08.131069+0000","last_active":"2024-07-26T12:44:38.860356+0000","last_peered":"2024-07-26T12:44:38.860356+0000","last_clean":"2024-07-26T12:44:38.860356+0000","last_became_active":"2024-07-26T12:43:08.130637+0000","last_became_peered":"2024-07-26T12:43:08.130637+0000","last_unstale":"2024-07-26T12:44:38.860356+0000","last_undegraded":"2024-07-26T12:44:38.860356+0000","last_fullsized":"2024-07-26T12:44:38.860356+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:04:16.461900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:01.141272+0000","last_change":"2024-07-26T12:43:26.763388+0000","last_active":"2024-07-26T12:45:01.141272+0000","last_peered":"2024-07-26T12:45:01.141272+0000","last_clean":"2024-07-26T12:45:01.141272+0000","last_became_active":"2024-07-26T12:43:26.762464+0000","last_became_peered":"2024-07-26T12:43:26.762464+0000","last_unstale":"2024-07-26T12:45:01.141272+0000","last_undegraded":"2024-07-26T12:45:01.141272+0000","last_fullsized":"2024-07-26T12:45:01.141272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:10:17.598266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.658539+0000","last_change":"2024-07-26T12:43:12.191640+0000","last_active":"2024-07-26T12:46:34.658539+0000","last_peered":"2024-07-26T12:46:34.658539+0000","last_clean":"2024-07-26T12:46:34.658539+0000","last_became_active":"2024-07-26T12:43:12.191260+0000","last_became_peered":"2024-07-26T12:43:12.191260+0000","last_unstale":"2024-07-26T12:46:34.658539+0000","last_undegraded":"2024-07-26T12:46:34.658539+0000","last_fullsized":"2024-07-26T12:46:34.658539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:47:18.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.876436+0000","last_change":"2024-07-26T12:43:08.135035+0000","last_active":"2024-07-26T12:44:38.876436+0000","last_peered":"2024-07-26T12:44:38.876436+0000","last_clean":"2024-07-26T12:44:38.876436+0000","last_became_active":"2024-07-26T12:43:08.134204+0000","last_became_peered":"2024-07-26T12:43:08.134204+0000","last_unstale":"2024-07-26T12:44:38.876436+0000","last_undegraded":"2024-07-26T12:44:38.876436+0000","last_fullsized":"2024-07-26T12:44:38.876436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:38.954460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.841001+0000","last_change":"2024-07-26T12:43:35.853244+0000","last_active":"2024-07-26T12:44:36.841001+0000","last_peered":"2024-07-26T12:44:36.841001+0000","last_clean":"2024-07-26T12:44:36.841001+0000","last_became_active":"2024-07-26T12:43:35.852728+0000","last_became_peered":"2024-07-26T12:43:35.852728+0000","last_unstale":"2024-07-26T12:44:36.841001+0000","last_undegraded":"2024-07-26T12:44:36.841001+0000","last_fullsized":"2024-07-26T12:44:36.841001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:20:39.080000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.634371+0000","last_change":"2024-07-26T12:43:12.172457+0000","last_active":"2024-07-26T12:46:34.634371+0000","last_peered":"2024-07-26T12:46:34.634371+0000","last_clean":"2024-07-26T12:46:34.634371+0000","last_became_active":"2024-07-26T12:43:12.172034+0000","last_became_peered":"2024-07-26T12:43:12.172034+0000","last_unstale":"2024-07-26T12:46:34.634371+0000","last_undegraded":"2024-07-26T12:46:34.634371+0000","last_fullsized":"2024-07-26T12:46:34.634371+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:37:02.237852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807509+0000","last_change":"2024-07-26T12:43:08.227152+0000","last_active":"2024-07-26T12:44:36.807509+0000","last_peered":"2024-07-26T12:44:36.807509+0000","last_clean":"2024-07-26T12:44:36.807509+0000","last_became_active":"2024-07-26T12:43:08.226722+0000","last_became_peered":"2024-07-26T12:43:08.226722+0000","last_unstale":"2024-07-26T12:44:36.807509+0000","last_undegraded":"2024-07-26T12:44:36.807509+0000","last_fullsized":"2024-07-26T12:44:36.807509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:16.293211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850844+0000","last_change":"2024-07-26T12:43:26.792999+0000","last_active":"2024-07-26T12:44:36.850844+0000","last_peered":"2024-07-26T12:44:36.850844+0000","last_clean":"2024-07-26T12:44:36.850844+0000","last_became_active":"2024-07-26T12:43:26.792515+0000","last_became_peered":"2024-07-26T12:43:26.792515+0000","last_unstale":"2024-07-26T12:44:36.850844+0000","last_undegraded":"2024-07-26T12:44:36.850844+0000","last_fullsized":"2024-07-26T12:44:36.850844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:49:35.104600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.673077+0000","last_change":"2024-07-26T12:43:12.185381+0000","last_active":"2024-07-26T12:46:34.673077+0000","last_peered":"2024-07-26T12:46:34.673077+0000","last_clean":"2024-07-26T12:46:34.673077+0000","last_became_active":"2024-07-26T12:43:12.184892+0000","last_became_peered":"2024-07-26T12:43:12.184892+0000","last_unstale":"2024-07-26T12:46:34.673077+0000","last_undegraded":"2024-07-26T12:46:34.673077+0000","last_fullsized":"2024-07-26T12:46:34.673077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:38:53.992586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881366+0000","last_change":"2024-07-26T12:43:08.895079+0000","last_active":"2024-07-26T12:44:39.881366+0000","last_peered":"2024-07-26T12:44:39.881366+0000","last_clean":"2024-07-26T12:44:39.881366+0000","last_became_active":"2024-07-26T12:43:08.894653+0000","last_became_peered":"2024-07-26T12:43:08.894653+0000","last_unstale":"2024-07-26T12:44:39.881366+0000","last_undegraded":"2024-07-26T12:44:39.881366+0000","last_fullsized":"2024-07-26T12:44:39.881366+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:02:53.137905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803288+0000","last_change":"2024-07-26T12:43:26.762802+0000","last_active":"2024-07-26T12:44:36.803288+0000","last_peered":"2024-07-26T12:44:36.803288+0000","last_clean":"2024-07-26T12:44:36.803288+0000","last_became_active":"2024-07-26T12:43:26.762211+0000","last_became_peered":"2024-07-26T12:43:26.762211+0000","last_unstale":"2024-07-26T12:44:36.803288+0000","last_undegraded":"2024-07-26T12:44:36.803288+0000","last_fullsized":"2024-07-26T12:44:36.803288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:35:19.623168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662724+0000","last_change":"2024-07-26T12:43:12.176062+0000","last_active":"2024-07-26T12:46:34.662724+0000","last_peered":"2024-07-26T12:46:34.662724+0000","last_clean":"2024-07-26T12:46:34.662724+0000","last_became_active":"2024-07-26T12:43:12.175671+0000","last_became_peered":"2024-07-26T12:43:12.175671+0000","last_unstale":"2024-07-26T12:46:34.662724+0000","last_undegraded":"2024-07-26T12:46:34.662724+0000","last_fullsized":"2024-07-26T12:46:34.662724+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:48.374964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'17","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.262028+0000","last_change":"2024-07-26T12:43:08.131453+0000","last_active":"2024-07-26T12:45:21.262028+0000","last_peered":"2024-07-26T12:45:21.262028+0000","last_clean":"2024-07-26T12:45:21.262028+0000","last_became_active":"2024-07-26T12:43:08.130928+0000","last_became_peered":"2024-07-26T12:43:08.130928+0000","last_unstale":"2024-07-26T12:45:21.262028+0000","last_undegraded":"2024-07-26T12:45:21.262028+0000","last_fullsized":"2024-07-26T12:45:21.262028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:43.190860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804408+0000","last_change":"2024-07-26T12:43:26.792942+0000","last_active":"2024-07-26T12:44:36.804408+0000","last_peered":"2024-07-26T12:44:36.804408+0000","last_clean":"2024-07-26T12:44:36.804408+0000","last_became_active":"2024-07-26T12:43:26.792045+0000","last_became_peered":"2024-07-26T12:43:26.792045+0000","last_unstale":"2024-07-26T12:44:36.804408+0000","last_undegraded":"2024-07-26T12:44:36.804408+0000","last_fullsized":"2024-07-26T12:44:36.804408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:56.005295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662815+0000","last_change":"2024-07-26T12:43:12.187601+0000","last_active":"2024-07-26T12:46:34.662815+0000","last_peered":"2024-07-26T12:46:34.662815+0000","last_clean":"2024-07-26T12:46:34.662815+0000","last_became_active":"2024-07-26T12:43:12.187175+0000","last_became_peered":"2024-07-26T12:43:12.187175+0000","last_unstale":"2024-07-26T12:46:34.662815+0000","last_undegraded":"2024-07-26T12:46:34.662815+0000","last_fullsized":"2024-07-26T12:46:34.662815+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:06.992854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881924+0000","last_change":"2024-07-26T12:43:08.227279+0000","last_active":"2024-07-26T12:44:39.881924+0000","last_peered":"2024-07-26T12:44:39.881924+0000","last_clean":"2024-07-26T12:44:39.881924+0000","last_became_active":"2024-07-26T12:43:08.226634+0000","last_became_peered":"2024-07-26T12:43:08.226634+0000","last_unstale":"2024-07-26T12:44:39.881924+0000","last_undegraded":"2024-07-26T12:44:39.881924+0000","last_fullsized":"2024-07-26T12:44:39.881924+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:19.845664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851516+0000","last_change":"2024-07-26T12:43:26.774537+0000","last_active":"2024-07-26T12:44:36.851516+0000","last_peered":"2024-07-26T12:44:36.851516+0000","last_clean":"2024-07-26T12:44:36.851516+0000","last_became_active":"2024-07-26T12:43:26.774036+0000","last_became_peered":"2024-07-26T12:43:26.774036+0000","last_unstale":"2024-07-26T12:44:36.851516+0000","last_undegraded":"2024-07-26T12:44:36.851516+0000","last_fullsized":"2024-07-26T12:44:36.851516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:21.774498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662351+0000","last_change":"2024-07-26T12:44:36.857443+0000","last_active":"2024-07-26T12:46:34.662351+0000","last_peered":"2024-07-26T12:46:34.662351+0000","last_clean":"2024-07-26T12:46:34.662351+0000","last_became_active":"2024-07-26T12:44:36.857001+0000","last_became_peered":"2024-07-26T12:44:36.857001+0000","last_unstale":"2024-07-26T12:46:34.662351+0000","last_undegraded":"2024-07-26T12:46:34.662351+0000","last_fullsized":"2024-07-26T12:46:34.662351+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:08.236404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.875394+0000","last_change":"2024-07-26T12:43:08.227432+0000","last_active":"2024-07-26T12:44:37.875394+0000","last_peered":"2024-07-26T12:44:37.875394+0000","last_clean":"2024-07-26T12:44:37.875394+0000","last_became_active":"2024-07-26T12:43:08.226645+0000","last_became_peered":"2024-07-26T12:43:08.226645+0000","last_unstale":"2024-07-26T12:44:37.875394+0000","last_undegraded":"2024-07-26T12:44:37.875394+0000","last_fullsized":"2024-07-26T12:44:37.875394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:23.498855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840331+0000","last_change":"2024-07-26T12:43:26.792874+0000","last_active":"2024-07-26T12:44:36.840331+0000","last_peered":"2024-07-26T12:44:36.840331+0000","last_clean":"2024-07-26T12:44:36.840331+0000","last_became_active":"2024-07-26T12:43:26.792371+0000","last_became_peered":"2024-07-26T12:43:26.792371+0000","last_unstale":"2024-07-26T12:44:36.840331+0000","last_undegraded":"2024-07-26T12:44:36.840331+0000","last_fullsized":"2024-07-26T12:44:36.840331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:25.753111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.653961+0000","last_change":"2024-07-26T12:43:12.185733+0000","last_active":"2024-07-26T12:46:34.653961+0000","last_peered":"2024-07-26T12:46:34.653961+0000","last_clean":"2024-07-26T12:46:34.653961+0000","last_became_active":"2024-07-26T12:43:12.185305+0000","last_became_peered":"2024-07-26T12:43:12.185305+0000","last_unstale":"2024-07-26T12:46:34.653961+0000","last_undegraded":"2024-07-26T12:46:34.653961+0000","last_fullsized":"2024-07-26T12:46:34.653961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:44.634607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:41.128840+0000","last_change":"2024-07-26T12:43:08.118508+0000","last_active":"2024-07-26T12:46:41.128840+0000","last_peered":"2024-07-26T12:46:41.128840+0000","last_clean":"2024-07-26T12:46:41.128840+0000","last_became_active":"2024-07-26T12:43:08.118088+0000","last_became_peered":"2024-07-26T12:43:08.118088+0000","last_unstale":"2024-07-26T12:46:41.128840+0000","last_undegraded":"2024-07-26T12:46:41.128840+0000","last_fullsized":"2024-07-26T12:46:41.128840+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:23:46.474298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810356+0000","last_change":"2024-07-26T12:43:26.779328+0000","last_active":"2024-07-26T12:44:36.810356+0000","last_peered":"2024-07-26T12:44:36.810356+0000","last_clean":"2024-07-26T12:44:36.810356+0000","last_became_active":"2024-07-26T12:43:26.778912+0000","last_became_peered":"2024-07-26T12:43:26.778912+0000","last_unstale":"2024-07-26T12:44:36.810356+0000","last_undegraded":"2024-07-26T12:44:36.810356+0000","last_fullsized":"2024-07-26T12:44:36.810356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:58.364775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650368+0000","last_change":"2024-07-26T12:43:35.854997+0000","last_active":"2024-07-26T12:46:34.650368+0000","last_peered":"2024-07-26T12:46:34.650368+0000","last_clean":"2024-07-26T12:46:34.650368+0000","last_became_active":"2024-07-26T12:43:35.854387+0000","last_became_peered":"2024-07-26T12:43:35.854387+0000","last_unstale":"2024-07-26T12:46:34.650368+0000","last_undegraded":"2024-07-26T12:46:34.650368+0000","last_fullsized":"2024-07-26T12:46:34.650368+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:04:20.489388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:41.196387+0000","last_change":"2024-07-26T12:43:08.227628+0000","last_active":"2024-07-26T12:46:41.196387+0000","last_peered":"2024-07-26T12:46:41.196387+0000","last_clean":"2024-07-26T12:46:41.196387+0000","last_became_active":"2024-07-26T12:43:08.227133+0000","last_became_peered":"2024-07-26T12:43:08.227133+0000","last_unstale":"2024-07-26T12:46:41.196387+0000","last_undegraded":"2024-07-26T12:46:41.196387+0000","last_fullsized":"2024-07-26T12:46:41.196387+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:28:28.484315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810904+0000","last_change":"2024-07-26T12:43:26.794016+0000","last_active":"2024-07-26T12:44:36.810904+0000","last_peered":"2024-07-26T12:44:36.810904+0000","last_clean":"2024-07-26T12:44:36.810904+0000","last_became_active":"2024-07-26T12:43:26.792562+0000","last_became_peered":"2024-07-26T12:43:26.792562+0000","last_unstale":"2024-07-26T12:44:36.810904+0000","last_undegraded":"2024-07-26T12:44:36.810904+0000","last_fullsized":"2024-07-26T12:44:36.810904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:26.162840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.652950+0000","last_change":"2024-07-26T12:43:12.173014+0000","last_active":"2024-07-26T12:46:34.652950+0000","last_peered":"2024-07-26T12:46:34.652950+0000","last_clean":"2024-07-26T12:46:34.652950+0000","last_became_active":"2024-07-26T12:43:12.172600+0000","last_became_peered":"2024-07-26T12:43:12.172600+0000","last_unstale":"2024-07-26T12:46:34.652950+0000","last_undegraded":"2024-07-26T12:46:34.652950+0000","last_fullsized":"2024-07-26T12:46:34.652950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:59.842744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.872830+0000","last_change":"2024-07-26T12:43:08.119000+0000","last_active":"2024-07-26T12:44:38.872830+0000","last_peered":"2024-07-26T12:44:38.872830+0000","last_clean":"2024-07-26T12:44:38.872830+0000","last_became_active":"2024-07-26T12:43:08.118608+0000","last_became_peered":"2024-07-26T12:43:08.118608+0000","last_unstale":"2024-07-26T12:44:38.872830+0000","last_undegraded":"2024-07-26T12:44:38.872830+0000","last_fullsized":"2024-07-26T12:44:38.872830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:29:47.663449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851317+0000","last_change":"2024-07-26T12:43:35.914716+0000","last_active":"2024-07-26T12:44:36.851317+0000","last_peered":"2024-07-26T12:44:36.851317+0000","last_clean":"2024-07-26T12:44:36.851317+0000","last_became_active":"2024-07-26T12:43:35.914354+0000","last_became_peered":"2024-07-26T12:43:35.914354+0000","last_unstale":"2024-07-26T12:44:36.851317+0000","last_undegraded":"2024-07-26T12:44:36.851317+0000","last_fullsized":"2024-07-26T12:44:36.851317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:09:10.994141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.660577+0000","last_change":"2024-07-26T12:43:12.169997+0000","last_active":"2024-07-26T12:46:34.660577+0000","last_peered":"2024-07-26T12:46:34.660577+0000","last_clean":"2024-07-26T12:46:34.660577+0000","last_became_active":"2024-07-26T12:43:12.169044+0000","last_became_peered":"2024-07-26T12:43:12.169044+0000","last_unstale":"2024-07-26T12:46:34.660577+0000","last_undegraded":"2024-07-26T12:46:34.660577+0000","last_fullsized":"2024-07-26T12:46:34.660577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:41.791434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.882037+0000","last_change":"2024-07-26T12:43:08.129025+0000","last_active":"2024-07-26T12:44:39.882037+0000","last_peered":"2024-07-26T12:44:39.882037+0000","last_clean":"2024-07-26T12:44:39.882037+0000","last_became_active":"2024-07-26T12:43:08.128608+0000","last_became_peered":"2024-07-26T12:43:08.128608+0000","last_unstale":"2024-07-26T12:44:39.882037+0000","last_undegraded":"2024-07-26T12:44:39.882037+0000","last_fullsized":"2024-07-26T12:44:39.882037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:43.749473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805850+0000","last_change":"2024-07-26T12:41:42.411595+0000","last_active":"2024-07-26T12:44:36.805850+0000","last_peered":"2024-07-26T12:44:36.805850+0000","last_clean":"2024-07-26T12:44:36.805850+0000","last_became_active":"2024-07-26T12:41:42.407119+0000","last_became_peered":"2024-07-26T12:41:42.407119+0000","last_unstale":"2024-07-26T12:44:36.805850+0000","last_undegraded":"2024-07-26T12:44:36.805850+0000","last_fullsized":"2024-07-26T12:44:36.805850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_clean_scrub_stamp":"2024-07-26T12:37:59.605754+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:21.376418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:53.111058+0000","last_change":"2024-07-26T12:43:26.763955+0000","last_active":"2024-07-26T12:44:53.111058+0000","last_peered":"2024-07-26T12:44:53.111058+0000","last_clean":"2024-07-26T12:44:53.111058+0000","last_became_active":"2024-07-26T12:43:26.763368+0000","last_became_peered":"2024-07-26T12:43:26.763368+0000","last_unstale":"2024-07-26T12:44:53.111058+0000","last_undegraded":"2024-07-26T12:44:53.111058+0000","last_fullsized":"2024-07-26T12:44:53.111058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:48:14.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662998+0000","last_change":"2024-07-26T12:43:12.174780+0000","last_active":"2024-07-26T12:46:34.662998+0000","last_peered":"2024-07-26T12:46:34.662998+0000","last_clean":"2024-07-26T12:46:34.662998+0000","last_became_active":"2024-07-26T12:43:12.174379+0000","last_became_peered":"2024-07-26T12:43:12.174379+0000","last_unstale":"2024-07-26T12:46:34.662998+0000","last_undegraded":"2024-07-26T12:46:34.662998+0000","last_fullsized":"2024-07-26T12:46:34.662998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:42:22.783259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'7","reported_seq":182,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:43.087466+0000","last_change":"2024-07-26T12:43:08.894655+0000","last_active":"2024-07-26T12:46:43.087466+0000","last_peered":"2024-07-26T12:46:43.087466+0000","last_clean":"2024-07-26T12:46:43.087466+0000","last_became_active":"2024-07-26T12:43:08.894270+0000","last_became_peered":"2024-07-26T12:43:08.894270+0000","last_unstale":"2024-07-26T12:46:43.087466+0000","last_undegraded":"2024-07-26T12:46:43.087466+0000","last_fullsized":"2024-07-26T12:46:43.087466+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:34.431204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":129,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807895+0000","last_change":"2024-07-26T12:43:26.769716+0000","last_active":"2024-07-26T12:44:36.807895+0000","last_peered":"2024-07-26T12:44:36.807895+0000","last_clean":"2024-07-26T12:44:36.807895+0000","last_became_active":"2024-07-26T12:43:26.769154+0000","last_became_peered":"2024-07-26T12:43:26.769154+0000","last_unstale":"2024-07-26T12:44:36.807895+0000","last_undegraded":"2024-07-26T12:44:36.807895+0000","last_fullsized":"2024-07-26T12:44:36.807895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:43.102498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.663935+0000","last_change":"2024-07-26T12:43:12.179488+0000","last_active":"2024-07-26T12:46:34.663935+0000","last_peered":"2024-07-26T12:46:34.663935+0000","last_clean":"2024-07-26T12:46:34.663935+0000","last_became_active":"2024-07-26T12:43:12.179171+0000","last_became_peered":"2024-07-26T12:43:12.179171+0000","last_unstale":"2024-07-26T12:46:34.663935+0000","last_undegraded":"2024-07-26T12:46:34.663935+0000","last_fullsized":"2024-07-26T12:46:34.663935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:43.695582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881415+0000","last_change":"2024-07-26T12:43:08.130838+0000","last_active":"2024-07-26T12:44:39.881415+0000","last_peered":"2024-07-26T12:44:39.881415+0000","last_clean":"2024-07-26T12:44:39.881415+0000","last_became_active":"2024-07-26T12:43:08.130446+0000","last_became_peered":"2024-07-26T12:43:08.130446+0000","last_unstale":"2024-07-26T12:44:39.881415+0000","last_undegraded":"2024-07-26T12:44:39.881415+0000","last_fullsized":"2024-07-26T12:44:39.881415+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:33:55.271038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.812663+0000","last_change":"2024-07-26T12:43:26.781982+0000","last_active":"2024-07-26T12:44:36.812663+0000","last_peered":"2024-07-26T12:44:36.812663+0000","last_clean":"2024-07-26T12:44:36.812663+0000","last_became_active":"2024-07-26T12:43:26.777102+0000","last_became_peered":"2024-07-26T12:43:26.777102+0000","last_unstale":"2024-07-26T12:44:36.812663+0000","last_undegraded":"2024-07-26T12:44:36.812663+0000","last_fullsized":"2024-07-26T12:44:36.812663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:49.553124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659275+0000","last_change":"2024-07-26T12:43:12.177043+0000","last_active":"2024-07-26T12:46:34.659275+0000","last_peered":"2024-07-26T12:46:34.659275+0000","last_clean":"2024-07-26T12:46:34.659275+0000","last_became_active":"2024-07-26T12:43:12.176634+0000","last_became_peered":"2024-07-26T12:43:12.176634+0000","last_unstale":"2024-07-26T12:46:34.659275+0000","last_undegraded":"2024-07-26T12:46:34.659275+0000","last_fullsized":"2024-07-26T12:46:34.659275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:20.210176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.878038+0000","last_change":"2024-07-26T12:43:08.131279+0000","last_active":"2024-07-26T12:44:38.878038+0000","last_peered":"2024-07-26T12:44:38.878038+0000","last_clean":"2024-07-26T12:44:38.878038+0000","last_became_active":"2024-07-26T12:43:08.130755+0000","last_became_peered":"2024-07-26T12:43:08.130755+0000","last_unstale":"2024-07-26T12:44:38.878038+0000","last_undegraded":"2024-07-26T12:44:38.878038+0000","last_fullsized":"2024-07-26T12:44:38.878038+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:03.234905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.849753+0000","last_change":"2024-07-26T12:43:26.764407+0000","last_active":"2024-07-26T12:44:36.849753+0000","last_peered":"2024-07-26T12:44:36.849753+0000","last_clean":"2024-07-26T12:44:36.849753+0000","last_became_active":"2024-07-26T12:43:26.762408+0000","last_became_peered":"2024-07-26T12:43:26.762408+0000","last_unstale":"2024-07-26T12:44:36.849753+0000","last_undegraded":"2024-07-26T12:44:36.849753+0000","last_fullsized":"2024-07-26T12:44:36.849753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:35:29.139476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.647937+0000","last_change":"2024-07-26T12:43:12.174587+0000","last_active":"2024-07-26T12:46:34.647937+0000","last_peered":"2024-07-26T12:46:34.647937+0000","last_clean":"2024-07-26T12:46:34.647937+0000","last_became_active":"2024-07-26T12:43:12.173929+0000","last_became_peered":"2024-07-26T12:43:12.173929+0000","last_unstale":"2024-07-26T12:46:34.647937+0000","last_undegraded":"2024-07-26T12:46:34.647937+0000","last_fullsized":"2024-07-26T12:46:34.647937+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:16.775198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881722+0000","last_change":"2024-07-26T12:43:08.119825+0000","last_active":"2024-07-26T12:44:39.881722+0000","last_peered":"2024-07-26T12:44:39.881722+0000","last_clean":"2024-07-26T12:44:39.881722+0000","last_became_active":"2024-07-26T12:43:08.119273+0000","last_became_peered":"2024-07-26T12:43:08.119273+0000","last_unstale":"2024-07-26T12:44:39.881722+0000","last_undegraded":"2024-07-26T12:44:39.881722+0000","last_fullsized":"2024-07-26T12:44:39.881722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:39.973020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850064+0000","last_change":"2024-07-26T12:43:26.773226+0000","last_active":"2024-07-26T12:44:36.850064+0000","last_peered":"2024-07-26T12:44:36.850064+0000","last_clean":"2024-07-26T12:44:36.850064+0000","last_became_active":"2024-07-26T12:43:26.772745+0000","last_became_peered":"2024-07-26T12:43:26.772745+0000","last_unstale":"2024-07-26T12:44:36.850064+0000","last_undegraded":"2024-07-26T12:44:36.850064+0000","last_fullsized":"2024-07-26T12:44:36.850064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:35.407138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.648152+0000","last_change":"2024-07-26T12:43:12.176178+0000","last_active":"2024-07-26T12:46:34.648152+0000","last_peered":"2024-07-26T12:46:34.648152+0000","last_clean":"2024-07-26T12:46:34.648152+0000","last_became_active":"2024-07-26T12:43:12.175791+0000","last_became_peered":"2024-07-26T12:43:12.175791+0000","last_unstale":"2024-07-26T12:46:34.648152+0000","last_undegraded":"2024-07-26T12:46:34.648152+0000","last_fullsized":"2024-07-26T12:46:34.648152+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:48.885831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881993+0000","last_change":"2024-07-26T12:43:08.226345+0000","last_active":"2024-07-26T12:44:39.881993+0000","last_peered":"2024-07-26T12:44:39.881993+0000","last_clean":"2024-07-26T12:44:39.881993+0000","last_became_active":"2024-07-26T12:43:08.225897+0000","last_became_peered":"2024-07-26T12:43:08.225897+0000","last_unstale":"2024-07-26T12:44:39.881993+0000","last_undegraded":"2024-07-26T12:44:39.881993+0000","last_fullsized":"2024-07-26T12:44:39.881993+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:59.351076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810778+0000","last_change":"2024-07-26T12:43:26.774800+0000","last_active":"2024-07-26T12:44:36.810778+0000","last_peered":"2024-07-26T12:44:36.810778+0000","last_clean":"2024-07-26T12:44:36.810778+0000","last_became_active":"2024-07-26T12:43:26.774192+0000","last_became_peered":"2024-07-26T12:43:26.774192+0000","last_unstale":"2024-07-26T12:44:36.810778+0000","last_undegraded":"2024-07-26T12:44:36.810778+0000","last_fullsized":"2024-07-26T12:44:36.810778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:18.379619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659090+0000","last_change":"2024-07-26T12:43:12.171720+0000","last_active":"2024-07-26T12:46:34.659090+0000","last_peered":"2024-07-26T12:46:34.659090+0000","last_clean":"2024-07-26T12:46:34.659090+0000","last_became_active":"2024-07-26T12:43:12.171334+0000","last_became_peered":"2024-07-26T12:43:12.171334+0000","last_unstale":"2024-07-26T12:46:34.659090+0000","last_undegraded":"2024-07-26T12:46:34.659090+0000","last_fullsized":"2024-07-26T12:46:34.659090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:35.156430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:40.194468+0000","last_change":"2024-07-26T12:43:08.120325+0000","last_active":"2024-07-26T12:46:40.194468+0000","last_peered":"2024-07-26T12:46:40.194468+0000","last_clean":"2024-07-26T12:46:40.194468+0000","last_became_active":"2024-07-26T12:43:08.119896+0000","last_became_peered":"2024-07-26T12:43:08.119896+0000","last_unstale":"2024-07-26T12:46:40.194468+0000","last_undegraded":"2024-07-26T12:46:40.194468+0000","last_fullsized":"2024-07-26T12:46:40.194468+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:23:19.470491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":125,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.856219+0000","last_change":"2024-07-26T12:43:26.778688+0000","last_active":"2024-07-26T12:44:36.856219+0000","last_peered":"2024-07-26T12:44:36.856219+0000","last_clean":"2024-07-26T12:44:36.856219+0000","last_became_active":"2024-07-26T12:43:26.778239+0000","last_became_peered":"2024-07-26T12:43:26.778239+0000","last_unstale":"2024-07-26T12:44:36.856219+0000","last_undegraded":"2024-07-26T12:44:36.856219+0000","last_fullsized":"2024-07-26T12:44:36.856219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:50:48.409636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.676452+0000","last_change":"2024-07-26T12:43:12.182323+0000","last_active":"2024-07-26T12:46:34.676452+0000","last_peered":"2024-07-26T12:46:34.676452+0000","last_clean":"2024-07-26T12:46:34.676452+0000","last_became_active":"2024-07-26T12:43:12.181860+0000","last_became_peered":"2024-07-26T12:43:12.181860+0000","last_unstale":"2024-07-26T12:46:34.676452+0000","last_undegraded":"2024-07-26T12:46:34.676452+0000","last_fullsized":"2024-07-26T12:46:34.676452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:08.764979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881366+0000","last_change":"2024-07-26T12:43:08.130823+0000","last_active":"2024-07-26T12:44:39.881366+0000","last_peered":"2024-07-26T12:44:39.881366+0000","last_clean":"2024-07-26T12:44:39.881366+0000","last_became_active":"2024-07-26T12:43:08.130424+0000","last_became_peered":"2024-07-26T12:43:08.130424+0000","last_unstale":"2024-07-26T12:44:39.881366+0000","last_undegraded":"2024-07-26T12:44:39.881366+0000","last_fullsized":"2024-07-26T12:44:39.881366+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:49:35.829624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850734+0000","last_change":"2024-07-26T12:43:26.770070+0000","last_active":"2024-07-26T12:44:36.850734+0000","last_peered":"2024-07-26T12:44:36.850734+0000","last_clean":"2024-07-26T12:44:36.850734+0000","last_became_active":"2024-07-26T12:43:26.769594+0000","last_became_peered":"2024-07-26T12:43:26.769594+0000","last_unstale":"2024-07-26T12:44:36.850734+0000","last_undegraded":"2024-07-26T12:44:36.850734+0000","last_fullsized":"2024-07-26T12:44:36.850734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:18.342739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669154+0000","last_change":"2024-07-26T12:43:12.183134+0000","last_active":"2024-07-26T12:46:34.669154+0000","last_peered":"2024-07-26T12:46:34.669154+0000","last_clean":"2024-07-26T12:46:34.669154+0000","last_became_active":"2024-07-26T12:43:12.182406+0000","last_became_peered":"2024-07-26T12:43:12.182406+0000","last_unstale":"2024-07-26T12:46:34.669154+0000","last_undegraded":"2024-07-26T12:46:34.669154+0000","last_fullsized":"2024-07-26T12:46:34.669154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:54:24.038971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881861+0000","last_change":"2024-07-26T12:43:35.862005+0000","last_active":"2024-07-26T12:44:39.881861+0000","last_peered":"2024-07-26T12:44:39.881861+0000","last_clean":"2024-07-26T12:44:39.881861+0000","last_became_active":"2024-07-26T12:43:35.850911+0000","last_became_peered":"2024-07-26T12:43:35.850911+0000","last_unstale":"2024-07-26T12:44:39.881861+0000","last_undegraded":"2024-07-26T12:44:39.881861+0000","last_fullsized":"2024-07-26T12:44:39.881861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:18:16.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850203+0000","last_change":"2024-07-26T12:43:26.774599+0000","last_active":"2024-07-26T12:44:36.850203+0000","last_peered":"2024-07-26T12:44:36.850203+0000","last_clean":"2024-07-26T12:44:36.850203+0000","last_became_active":"2024-07-26T12:43:26.774186+0000","last_became_peered":"2024-07-26T12:43:26.774186+0000","last_unstale":"2024-07-26T12:44:36.850203+0000","last_undegraded":"2024-07-26T12:44:36.850203+0000","last_fullsized":"2024-07-26T12:44:36.850203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:02:42.252093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.633333+0000","last_change":"2024-07-26T12:43:12.174676+0000","last_active":"2024-07-26T12:46:34.633333+0000","last_peered":"2024-07-26T12:46:34.633333+0000","last_clean":"2024-07-26T12:46:34.633333+0000","last_became_active":"2024-07-26T12:43:12.174310+0000","last_became_peered":"2024-07-26T12:43:12.174310+0000","last_unstale":"2024-07-26T12:46:34.633333+0000","last_undegraded":"2024-07-26T12:46:34.633333+0000","last_fullsized":"2024-07-26T12:46:34.633333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:44.381289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":341,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:42.084521+0000","last_change":"2024-07-26T12:43:08.122675+0000","last_active":"2024-07-26T12:46:42.084521+0000","last_peered":"2024-07-26T12:46:42.084521+0000","last_clean":"2024-07-26T12:46:42.084521+0000","last_became_active":"2024-07-26T12:43:08.122174+0000","last_became_peered":"2024-07-26T12:43:08.122174+0000","last_unstale":"2024-07-26T12:46:42.084521+0000","last_undegraded":"2024-07-26T12:46:42.084521+0000","last_fullsized":"2024-07-26T12:46:42.084521+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:54.513306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64385,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":256,"num_read_kb":191,"num_write":41,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.854144+0000","last_change":"2024-07-26T12:43:26.764283+0000","last_active":"2024-07-26T12:44:36.854144+0000","last_peered":"2024-07-26T12:44:36.854144+0000","last_clean":"2024-07-26T12:44:36.854144+0000","last_became_active":"2024-07-26T12:43:26.763821+0000","last_became_peered":"2024-07-26T12:43:26.763821+0000","last_unstale":"2024-07-26T12:44:36.854144+0000","last_undegraded":"2024-07-26T12:44:36.854144+0000","last_fullsized":"2024-07-26T12:44:36.854144+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:56.753016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672115+0000","last_change":"2024-07-26T12:43:35.915277+0000","last_active":"2024-07-26T12:46:34.672115+0000","last_peered":"2024-07-26T12:46:34.672115+0000","last_clean":"2024-07-26T12:46:34.672115+0000","last_became_active":"2024-07-26T12:43:35.914848+0000","last_became_peered":"2024-07-26T12:43:35.914848+0000","last_unstale":"2024-07-26T12:46:34.672115+0000","last_undegraded":"2024-07-26T12:46:34.672115+0000","last_fullsized":"2024-07-26T12:46:34.672115+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:56:35.639076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.885252+0000","last_change":"2024-07-26T12:43:08.227352+0000","last_active":"2024-07-26T12:44:39.885252+0000","last_peered":"2024-07-26T12:44:39.885252+0000","last_clean":"2024-07-26T12:44:39.885252+0000","last_became_active":"2024-07-26T12:43:08.226857+0000","last_became_peered":"2024-07-26T12:43:08.226857+0000","last_unstale":"2024-07-26T12:44:39.885252+0000","last_undegraded":"2024-07-26T12:44:39.885252+0000","last_fullsized":"2024-07-26T12:44:39.885252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:20.156503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850749+0000","last_change":"2024-07-26T12:43:26.762943+0000","last_active":"2024-07-26T12:44:36.850749+0000","last_peered":"2024-07-26T12:44:36.850749+0000","last_clean":"2024-07-26T12:44:36.850749+0000","last_became_active":"2024-07-26T12:43:26.762436+0000","last_became_peered":"2024-07-26T12:43:26.762436+0000","last_unstale":"2024-07-26T12:44:36.850749+0000","last_undegraded":"2024-07-26T12:44:36.850749+0000","last_fullsized":"2024-07-26T12:44:36.850749+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:05:23.924397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.673105+0000","last_change":"2024-07-26T12:44:36.854705+0000","last_active":"2024-07-26T12:46:34.673105+0000","last_peered":"2024-07-26T12:46:34.673105+0000","last_clean":"2024-07-26T12:46:34.673105+0000","last_became_active":"2024-07-26T12:44:36.854268+0000","last_became_peered":"2024-07-26T12:44:36.854268+0000","last_unstale":"2024-07-26T12:46:34.673105+0000","last_undegraded":"2024-07-26T12:46:34.673105+0000","last_fullsized":"2024-07-26T12:46:34.673105+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:24:30.768905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.874653+0000","last_change":"2024-07-26T12:43:08.118597+0000","last_active":"2024-07-26T12:44:38.874653+0000","last_peered":"2024-07-26T12:44:38.874653+0000","last_clean":"2024-07-26T12:44:38.874653+0000","last_became_active":"2024-07-26T12:43:08.118145+0000","last_became_peered":"2024-07-26T12:43:08.118145+0000","last_unstale":"2024-07-26T12:44:38.874653+0000","last_undegraded":"2024-07-26T12:44:38.874653+0000","last_fullsized":"2024-07-26T12:44:38.874653+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:52:45.511693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.853684+0000","last_change":"2024-07-26T12:43:26.771861+0000","last_active":"2024-07-26T12:44:36.853684+0000","last_peered":"2024-07-26T12:44:36.853684+0000","last_clean":"2024-07-26T12:44:36.853684+0000","last_became_active":"2024-07-26T12:43:26.771323+0000","last_became_peered":"2024-07-26T12:43:26.771323+0000","last_unstale":"2024-07-26T12:44:36.853684+0000","last_undegraded":"2024-07-26T12:44:36.853684+0000","last_fullsized":"2024-07-26T12:44:36.853684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:34.207514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656373+0000","last_change":"2024-07-26T12:43:12.169437+0000","last_active":"2024-07-26T12:46:34.656373+0000","last_peered":"2024-07-26T12:46:34.656373+0000","last_clean":"2024-07-26T12:46:34.656373+0000","last_became_active":"2024-07-26T12:43:12.168387+0000","last_became_peered":"2024-07-26T12:43:12.168387+0000","last_unstale":"2024-07-26T12:46:34.656373+0000","last_undegraded":"2024-07-26T12:46:34.656373+0000","last_fullsized":"2024-07-26T12:46:34.656373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:57:06.148965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'6","reported_seq":298,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:42.084648+0000","last_change":"2024-07-26T12:43:08.129618+0000","last_active":"2024-07-26T12:46:42.084648+0000","last_peered":"2024-07-26T12:46:42.084648+0000","last_clean":"2024-07-26T12:46:42.084648+0000","last_became_active":"2024-07-26T12:43:08.129064+0000","last_became_peered":"2024-07-26T12:43:08.129064+0000","last_unstale":"2024-07-26T12:46:42.084648+0000","last_undegraded":"2024-07-26T12:46:42.084648+0000","last_fullsized":"2024-07-26T12:46:42.084648+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:26:22.406075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":246,"num_read_kb":246,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804420+0000","last_change":"2024-07-26T12:43:35.854710+0000","last_active":"2024-07-26T12:44:36.804420+0000","last_peered":"2024-07-26T12:44:36.804420+0000","last_clean":"2024-07-26T12:44:36.804420+0000","last_became_active":"2024-07-26T12:43:35.852772+0000","last_became_peered":"2024-07-26T12:43:35.852772+0000","last_unstale":"2024-07-26T12:44:36.804420+0000","last_undegraded":"2024-07-26T12:44:36.804420+0000","last_fullsized":"2024-07-26T12:44:36.804420+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:24:54.015879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813340+0000","last_change":"2024-07-26T12:43:26.783182+0000","last_active":"2024-07-26T12:44:36.813340+0000","last_peered":"2024-07-26T12:44:36.813340+0000","last_clean":"2024-07-26T12:44:36.813340+0000","last_became_active":"2024-07-26T12:43:26.782697+0000","last_became_peered":"2024-07-26T12:43:26.782697+0000","last_unstale":"2024-07-26T12:44:36.813340+0000","last_undegraded":"2024-07-26T12:44:36.813340+0000","last_fullsized":"2024-07-26T12:44:36.813340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:57.169222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.666849+0000","last_change":"2024-07-26T12:43:12.166163+0000","last_active":"2024-07-26T12:46:34.666849+0000","last_peered":"2024-07-26T12:46:34.666849+0000","last_clean":"2024-07-26T12:46:34.666849+0000","last_became_active":"2024-07-26T12:43:12.164427+0000","last_became_peered":"2024-07-26T12:43:12.164427+0000","last_unstale":"2024-07-26T12:46:34.666849+0000","last_undegraded":"2024-07-26T12:46:34.666849+0000","last_fullsized":"2024-07-26T12:46:34.666849+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:25.335508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.885502+0000","last_change":"2024-07-26T12:43:08.118224+0000","last_active":"2024-07-26T12:44:39.885502+0000","last_peered":"2024-07-26T12:44:39.885502+0000","last_clean":"2024-07-26T12:44:39.885502+0000","last_became_active":"2024-07-26T12:43:08.117821+0000","last_became_peered":"2024-07-26T12:43:08.117821+0000","last_unstale":"2024-07-26T12:44:39.885502+0000","last_undegraded":"2024-07-26T12:44:39.885502+0000","last_fullsized":"2024-07-26T12:44:39.885502+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:56:14.380931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851235+0000","last_change":"2024-07-26T12:43:35.907740+0000","last_active":"2024-07-26T12:44:36.851235+0000","last_peered":"2024-07-26T12:44:36.851235+0000","last_clean":"2024-07-26T12:44:36.851235+0000","last_became_active":"2024-07-26T12:43:35.907115+0000","last_became_peered":"2024-07-26T12:43:35.907115+0000","last_unstale":"2024-07-26T12:44:36.851235+0000","last_undegraded":"2024-07-26T12:44:36.851235+0000","last_fullsized":"2024-07-26T12:44:36.851235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:53:24.783872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.670083+0000","last_change":"2024-07-26T12:43:12.177579+0000","last_active":"2024-07-26T12:46:34.670083+0000","last_peered":"2024-07-26T12:46:34.670083+0000","last_clean":"2024-07-26T12:46:34.670083+0000","last_became_active":"2024-07-26T12:43:12.171433+0000","last_became_peered":"2024-07-26T12:43:12.171433+0000","last_unstale":"2024-07-26T12:46:34.670083+0000","last_undegraded":"2024-07-26T12:46:34.670083+0000","last_fullsized":"2024-07-26T12:46:34.670083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:50.972151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.877099+0000","last_change":"2024-07-26T12:43:08.133446+0000","last_active":"2024-07-26T12:44:38.877099+0000","last_peered":"2024-07-26T12:44:38.877099+0000","last_clean":"2024-07-26T12:44:38.877099+0000","last_became_active":"2024-07-26T12:43:08.133008+0000","last_became_peered":"2024-07-26T12:43:08.133008+0000","last_unstale":"2024-07-26T12:44:38.877099+0000","last_undegraded":"2024-07-26T12:44:38.877099+0000","last_fullsized":"2024-07-26T12:44:38.877099+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:05:21.283738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.874509+0000","last_change":"2024-07-26T12:43:08.124976+0000","last_active":"2024-07-26T12:44:37.874509+0000","last_peered":"2024-07-26T12:44:37.874509+0000","last_clean":"2024-07-26T12:44:37.874509+0000","last_became_active":"2024-07-26T12:43:08.124664+0000","last_became_peered":"2024-07-26T12:43:08.124664+0000","last_unstale":"2024-07-26T12:44:37.874509+0000","last_undegraded":"2024-07-26T12:44:37.874509+0000","last_fullsized":"2024-07-26T12:44:37.874509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:51.798657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664094+0000","last_change":"2024-07-26T12:43:35.856266+0000","last_active":"2024-07-26T12:46:34.664094+0000","last_peered":"2024-07-26T12:46:34.664094+0000","last_clean":"2024-07-26T12:46:34.664094+0000","last_became_active":"2024-07-26T12:43:35.855802+0000","last_became_peered":"2024-07-26T12:43:35.855802+0000","last_unstale":"2024-07-26T12:46:34.664094+0000","last_undegraded":"2024-07-26T12:46:34.664094+0000","last_fullsized":"2024-07-26T12:46:34.664094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:16.432983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.802866+0000","last_change":"2024-07-26T12:43:26.778626+0000","last_active":"2024-07-26T12:44:36.802866+0000","last_peered":"2024-07-26T12:44:36.802866+0000","last_clean":"2024-07-26T12:44:36.802866+0000","last_became_active":"2024-07-26T12:43:26.778165+0000","last_became_peered":"2024-07-26T12:43:26.778165+0000","last_unstale":"2024-07-26T12:44:36.802866+0000","last_undegraded":"2024-07-26T12:44:36.802866+0000","last_fullsized":"2024-07-26T12:44:36.802866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:06:08.766495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":181,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:42.085334+0000","last_change":"2024-07-26T12:43:08.119702+0000","last_active":"2024-07-26T12:46:42.085334+0000","last_peered":"2024-07-26T12:46:42.085334+0000","last_clean":"2024-07-26T12:46:42.085334+0000","last_became_active":"2024-07-26T12:43:08.119225+0000","last_became_peered":"2024-07-26T12:43:08.119225+0000","last_unstale":"2024-07-26T12:46:42.085334+0000","last_undegraded":"2024-07-26T12:46:42.085334+0000","last_fullsized":"2024-07-26T12:46:42.085334+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:27.616873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.640444+0000","last_change":"2024-07-26T12:43:12.166239+0000","last_active":"2024-07-26T12:46:34.640444+0000","last_peered":"2024-07-26T12:46:34.640444+0000","last_clean":"2024-07-26T12:46:34.640444+0000","last_became_active":"2024-07-26T12:43:12.165770+0000","last_became_peered":"2024-07-26T12:43:12.165770+0000","last_unstale":"2024-07-26T12:46:34.640444+0000","last_undegraded":"2024-07-26T12:46:34.640444+0000","last_fullsized":"2024-07-26T12:46:34.640444+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:11:15.290898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803067+0000","last_change":"2024-07-26T12:43:26.765905+0000","last_active":"2024-07-26T12:44:36.803067+0000","last_peered":"2024-07-26T12:44:36.803067+0000","last_clean":"2024-07-26T12:44:36.803067+0000","last_became_active":"2024-07-26T12:43:26.765425+0000","last_became_peered":"2024-07-26T12:43:26.765425+0000","last_unstale":"2024-07-26T12:44:36.803067+0000","last_undegraded":"2024-07-26T12:44:36.803067+0000","last_fullsized":"2024-07-26T12:44:36.803067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:16.786604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644926+0000","last_change":"2024-07-26T12:43:12.171400+0000","last_active":"2024-07-26T12:46:34.644926+0000","last_peered":"2024-07-26T12:46:34.644926+0000","last_clean":"2024-07-26T12:46:34.644926+0000","last_became_active":"2024-07-26T12:43:12.170998+0000","last_became_peered":"2024-07-26T12:43:12.170998+0000","last_unstale":"2024-07-26T12:46:34.644926+0000","last_undegraded":"2024-07-26T12:46:34.644926+0000","last_fullsized":"2024-07-26T12:46:34.644926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:43.251975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656641+0000","last_change":"2024-07-26T12:43:35.912729+0000","last_active":"2024-07-26T12:46:34.656641+0000","last_peered":"2024-07-26T12:46:34.656641+0000","last_clean":"2024-07-26T12:46:34.656641+0000","last_became_active":"2024-07-26T12:43:35.912318+0000","last_became_peered":"2024-07-26T12:43:35.912318+0000","last_unstale":"2024-07-26T12:46:34.656641+0000","last_undegraded":"2024-07-26T12:46:34.656641+0000","last_fullsized":"2024-07-26T12:46:34.656641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:14.473536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661247+0000","last_change":"2024-07-26T12:43:12.182933+0000","last_active":"2024-07-26T12:46:34.661247+0000","last_peered":"2024-07-26T12:46:34.661247+0000","last_clean":"2024-07-26T12:46:34.661247+0000","last_became_active":"2024-07-26T12:43:12.182656+0000","last_became_peered":"2024-07-26T12:43:12.182656+0000","last_unstale":"2024-07-26T12:46:34.661247+0000","last_undegraded":"2024-07-26T12:46:34.661247+0000","last_fullsized":"2024-07-26T12:46:34.661247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:05:59.717662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674579+0000","last_change":"2024-07-26T12:43:12.181196+0000","last_active":"2024-07-26T12:46:34.674579+0000","last_peered":"2024-07-26T12:46:34.674579+0000","last_clean":"2024-07-26T12:46:34.674579+0000","last_became_active":"2024-07-26T12:43:12.180765+0000","last_became_peered":"2024-07-26T12:43:12.180765+0000","last_unstale":"2024-07-26T12:46:34.674579+0000","last_undegraded":"2024-07-26T12:46:34.674579+0000","last_fullsized":"2024-07-26T12:46:34.674579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:20.590023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.653653+0000","last_change":"2024-07-26T12:43:12.185408+0000","last_active":"2024-07-26T12:46:34.653653+0000","last_peered":"2024-07-26T12:46:34.653653+0000","last_clean":"2024-07-26T12:46:34.653653+0000","last_became_active":"2024-07-26T12:43:12.185038+0000","last_became_peered":"2024-07-26T12:43:12.185038+0000","last_unstale":"2024-07-26T12:46:34.653653+0000","last_undegraded":"2024-07-26T12:46:34.653653+0000","last_fullsized":"2024-07-26T12:46:34.653653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:55:30.604995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644286+0000","last_change":"2024-07-26T12:43:12.169301+0000","last_active":"2024-07-26T12:46:34.644286+0000","last_peered":"2024-07-26T12:46:34.644286+0000","last_clean":"2024-07-26T12:46:34.644286+0000","last_became_active":"2024-07-26T12:43:12.168923+0000","last_became_peered":"2024-07-26T12:43:12.168923+0000","last_unstale":"2024-07-26T12:46:34.644286+0000","last_undegraded":"2024-07-26T12:46:34.644286+0000","last_fullsized":"2024-07-26T12:46:34.644286+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:04.506485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.647408+0000","last_change":"2024-07-26T12:43:12.166663+0000","last_active":"2024-07-26T12:46:34.647408+0000","last_peered":"2024-07-26T12:46:34.647408+0000","last_clean":"2024-07-26T12:46:34.647408+0000","last_became_active":"2024-07-26T12:43:12.166200+0000","last_became_peered":"2024-07-26T12:43:12.166200+0000","last_unstale":"2024-07-26T12:46:34.647408+0000","last_undegraded":"2024-07-26T12:46:34.647408+0000","last_fullsized":"2024-07-26T12:46:34.647408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:14.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.660632+0000","last_change":"2024-07-26T12:43:12.171273+0000","last_active":"2024-07-26T12:46:34.660632+0000","last_peered":"2024-07-26T12:46:34.660632+0000","last_clean":"2024-07-26T12:46:34.660632+0000","last_became_active":"2024-07-26T12:43:12.167680+0000","last_became_peered":"2024-07-26T12:43:12.167680+0000","last_unstale":"2024-07-26T12:46:34.660632+0000","last_undegraded":"2024-07-26T12:46:34.660632+0000","last_fullsized":"2024-07-26T12:46:34.660632+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:44.797644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.667348+0000","last_change":"2024-07-26T12:43:12.179891+0000","last_active":"2024-07-26T12:46:34.667348+0000","last_peered":"2024-07-26T12:46:34.667348+0000","last_clean":"2024-07-26T12:46:34.667348+0000","last_became_active":"2024-07-26T12:43:12.179518+0000","last_became_peered":"2024-07-26T12:43:12.179518+0000","last_unstale":"2024-07-26T12:46:34.667348+0000","last_undegraded":"2024-07-26T12:46:34.667348+0000","last_fullsized":"2024-07-26T12:46:34.667348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:24.246023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668993+0000","last_change":"2024-07-26T12:43:12.168521+0000","last_active":"2024-07-26T12:46:34.668993+0000","last_peered":"2024-07-26T12:46:34.668993+0000","last_clean":"2024-07-26T12:46:34.668993+0000","last_became_active":"2024-07-26T12:43:12.168126+0000","last_became_peered":"2024-07-26T12:43:12.168126+0000","last_unstale":"2024-07-26T12:46:34.668993+0000","last_undegraded":"2024-07-26T12:46:34.668993+0000","last_fullsized":"2024-07-26T12:46:34.668993+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:00.135090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661477+0000","last_change":"2024-07-26T12:44:36.843322+0000","last_active":"2024-07-26T12:46:34.661477+0000","last_peered":"2024-07-26T12:46:34.661477+0000","last_clean":"2024-07-26T12:46:34.661477+0000","last_became_active":"2024-07-26T12:44:36.842821+0000","last_became_peered":"2024-07-26T12:44:36.842821+0000","last_unstale":"2024-07-26T12:46:34.661477+0000","last_undegraded":"2024-07-26T12:46:34.661477+0000","last_fullsized":"2024-07-26T12:46:34.661477+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:05.609298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674453+0000","last_change":"2024-07-26T12:43:12.185565+0000","last_active":"2024-07-26T12:46:34.674453+0000","last_peered":"2024-07-26T12:46:34.674453+0000","last_clean":"2024-07-26T12:46:34.674453+0000","last_became_active":"2024-07-26T12:43:12.185057+0000","last_became_peered":"2024-07-26T12:43:12.185057+0000","last_unstale":"2024-07-26T12:46:34.674453+0000","last_undegraded":"2024-07-26T12:46:34.674453+0000","last_fullsized":"2024-07-26T12:46:34.674453+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:58.108311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644559+0000","last_change":"2024-07-26T12:43:12.173301+0000","last_active":"2024-07-26T12:46:34.644559+0000","last_peered":"2024-07-26T12:46:34.644559+0000","last_clean":"2024-07-26T12:46:34.644559+0000","last_became_active":"2024-07-26T12:43:12.172955+0000","last_became_peered":"2024-07-26T12:43:12.172955+0000","last_unstale":"2024-07-26T12:46:34.644559+0000","last_undegraded":"2024-07-26T12:46:34.644559+0000","last_fullsized":"2024-07-26T12:46:34.644559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:41.731141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661375+0000","last_change":"2024-07-26T12:43:12.176169+0000","last_active":"2024-07-26T12:46:34.661375+0000","last_peered":"2024-07-26T12:46:34.661375+0000","last_clean":"2024-07-26T12:46:34.661375+0000","last_became_active":"2024-07-26T12:43:12.175745+0000","last_became_peered":"2024-07-26T12:43:12.175745+0000","last_unstale":"2024-07-26T12:46:34.661375+0000","last_undegraded":"2024-07-26T12:46:34.661375+0000","last_fullsized":"2024-07-26T12:46:34.661375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:20.393207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.654280+0000","last_change":"2024-07-26T12:43:12.168182+0000","last_active":"2024-07-26T12:46:34.654280+0000","last_peered":"2024-07-26T12:46:34.654280+0000","last_clean":"2024-07-26T12:46:34.654280+0000","last_became_active":"2024-07-26T12:43:12.163771+0000","last_became_peered":"2024-07-26T12:43:12.163771+0000","last_unstale":"2024-07-26T12:46:34.654280+0000","last_undegraded":"2024-07-26T12:46:34.654280+0000","last_fullsized":"2024-07-26T12:46:34.654280+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:40:30.960153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659998+0000","last_change":"2024-07-26T12:43:12.189524+0000","last_active":"2024-07-26T12:46:34.659998+0000","last_peered":"2024-07-26T12:46:34.659998+0000","last_clean":"2024-07-26T12:46:34.659998+0000","last_became_active":"2024-07-26T12:43:12.189165+0000","last_became_peered":"2024-07-26T12:43:12.189165+0000","last_unstale":"2024-07-26T12:46:34.659998+0000","last_undegraded":"2024-07-26T12:46:34.659998+0000","last_fullsized":"2024-07-26T12:46:34.659998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:11:08.479058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.665264+0000","last_change":"2024-07-26T12:43:12.180603+0000","last_active":"2024-07-26T12:46:34.665264+0000","last_peered":"2024-07-26T12:46:34.665264+0000","last_clean":"2024-07-26T12:46:34.665264+0000","last_became_active":"2024-07-26T12:43:12.180204+0000","last_became_peered":"2024-07-26T12:43:12.180204+0000","last_unstale":"2024-07-26T12:46:34.665264+0000","last_undegraded":"2024-07-26T12:46:34.665264+0000","last_fullsized":"2024-07-26T12:46:34.665264+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:46.768213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.667944+0000","last_change":"2024-07-26T12:43:12.185347+0000","last_active":"2024-07-26T12:46:34.667944+0000","last_peered":"2024-07-26T12:46:34.667944+0000","last_clean":"2024-07-26T12:46:34.667944+0000","last_became_active":"2024-07-26T12:43:12.180190+0000","last_became_peered":"2024-07-26T12:43:12.180190+0000","last_unstale":"2024-07-26T12:46:34.667944+0000","last_undegraded":"2024-07-26T12:46:34.667944+0000","last_fullsized":"2024-07-26T12:46:34.667944+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:07.654967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.665258+0000","last_change":"2024-07-26T12:43:12.177258+0000","last_active":"2024-07-26T12:46:34.665258+0000","last_peered":"2024-07-26T12:46:34.665258+0000","last_clean":"2024-07-26T12:46:34.665258+0000","last_became_active":"2024-07-26T12:43:12.176879+0000","last_became_peered":"2024-07-26T12:43:12.176879+0000","last_unstale":"2024-07-26T12:46:34.665258+0000","last_undegraded":"2024-07-26T12:46:34.665258+0000","last_fullsized":"2024-07-26T12:46:34.665258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:24.244759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674886+0000","last_change":"2024-07-26T12:44:36.857403+0000","last_active":"2024-07-26T12:46:34.674886+0000","last_peered":"2024-07-26T12:46:34.674886+0000","last_clean":"2024-07-26T12:46:34.674886+0000","last_became_active":"2024-07-26T12:44:36.856980+0000","last_became_peered":"2024-07-26T12:44:36.856980+0000","last_unstale":"2024-07-26T12:46:34.674886+0000","last_undegraded":"2024-07-26T12:46:34.674886+0000","last_fullsized":"2024-07-26T12:46:34.674886+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.166383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.652345+0000","last_change":"2024-07-26T12:43:12.166827+0000","last_active":"2024-07-26T12:46:34.652345+0000","last_peered":"2024-07-26T12:46:34.652345+0000","last_clean":"2024-07-26T12:46:34.652345+0000","last_became_active":"2024-07-26T12:43:12.165222+0000","last_became_peered":"2024-07-26T12:43:12.165222+0000","last_unstale":"2024-07-26T12:46:34.652345+0000","last_undegraded":"2024-07-26T12:46:34.652345+0000","last_fullsized":"2024-07-26T12:46:34.652345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:37:42.219844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662861+0000","last_change":"2024-07-26T12:43:12.176961+0000","last_active":"2024-07-26T12:46:34.662861+0000","last_peered":"2024-07-26T12:46:34.662861+0000","last_clean":"2024-07-26T12:46:34.662861+0000","last_became_active":"2024-07-26T12:43:12.176550+0000","last_became_peered":"2024-07-26T12:43:12.176550+0000","last_unstale":"2024-07-26T12:46:34.662861+0000","last_undegraded":"2024-07-26T12:46:34.662861+0000","last_fullsized":"2024-07-26T12:46:34.662861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:36:19.720801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650354+0000","last_change":"2024-07-26T12:43:12.173560+0000","last_active":"2024-07-26T12:46:34.650354+0000","last_peered":"2024-07-26T12:46:34.650354+0000","last_clean":"2024-07-26T12:46:34.650354+0000","last_became_active":"2024-07-26T12:43:12.173187+0000","last_became_peered":"2024-07-26T12:43:12.173187+0000","last_unstale":"2024-07-26T12:46:34.650354+0000","last_undegraded":"2024-07-26T12:46:34.650354+0000","last_fullsized":"2024-07-26T12:46:34.650354+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:53.929210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655391+0000","last_change":"2024-07-26T12:43:12.169829+0000","last_active":"2024-07-26T12:46:34.655391+0000","last_peered":"2024-07-26T12:46:34.655391+0000","last_clean":"2024-07-26T12:46:34.655391+0000","last_became_active":"2024-07-26T12:43:12.169444+0000","last_became_peered":"2024-07-26T12:43:12.169444+0000","last_unstale":"2024-07-26T12:46:34.655391+0000","last_undegraded":"2024-07-26T12:46:34.655391+0000","last_fullsized":"2024-07-26T12:46:34.655391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:29.111483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668876+0000","last_change":"2024-07-26T12:43:12.171857+0000","last_active":"2024-07-26T12:46:34.668876+0000","last_peered":"2024-07-26T12:46:34.668876+0000","last_clean":"2024-07-26T12:46:34.668876+0000","last_became_active":"2024-07-26T12:43:12.170752+0000","last_became_peered":"2024-07-26T12:43:12.170752+0000","last_unstale":"2024-07-26T12:46:34.668876+0000","last_undegraded":"2024-07-26T12:46:34.668876+0000","last_fullsized":"2024-07-26T12:46:34.668876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:22:09.442916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656394+0000","last_change":"2024-07-26T12:43:12.178646+0000","last_active":"2024-07-26T12:46:34.656394+0000","last_peered":"2024-07-26T12:46:34.656394+0000","last_clean":"2024-07-26T12:46:34.656394+0000","last_became_active":"2024-07-26T12:43:12.178259+0000","last_became_peered":"2024-07-26T12:43:12.178259+0000","last_unstale":"2024-07-26T12:46:34.656394+0000","last_undegraded":"2024-07-26T12:46:34.656394+0000","last_fullsized":"2024-07-26T12:46:34.656394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:30:59.417843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.652610+0000","last_change":"2024-07-26T12:43:12.164579+0000","last_active":"2024-07-26T12:46:34.652610+0000","last_peered":"2024-07-26T12:46:34.652610+0000","last_clean":"2024-07-26T12:46:34.652610+0000","last_became_active":"2024-07-26T12:43:12.164099+0000","last_became_peered":"2024-07-26T12:43:12.164099+0000","last_unstale":"2024-07-26T12:46:34.652610+0000","last_undegraded":"2024-07-26T12:46:34.652610+0000","last_fullsized":"2024-07-26T12:46:34.652610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:51:45.293190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.652843+0000","last_change":"2024-07-26T12:43:12.180838+0000","last_active":"2024-07-26T12:46:34.652843+0000","last_peered":"2024-07-26T12:46:34.652843+0000","last_clean":"2024-07-26T12:46:34.652843+0000","last_became_active":"2024-07-26T12:43:12.180543+0000","last_became_peered":"2024-07-26T12:43:12.180543+0000","last_unstale":"2024-07-26T12:46:34.652843+0000","last_undegraded":"2024-07-26T12:46:34.652843+0000","last_fullsized":"2024-07-26T12:46:34.652843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:27.185082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.648877+0000","last_change":"2024-07-26T12:43:12.167235+0000","last_active":"2024-07-26T12:46:34.648877+0000","last_peered":"2024-07-26T12:46:34.648877+0000","last_clean":"2024-07-26T12:46:34.648877+0000","last_became_active":"2024-07-26T12:43:12.166489+0000","last_became_peered":"2024-07-26T12:43:12.166489+0000","last_unstale":"2024-07-26T12:46:34.648877+0000","last_undegraded":"2024-07-26T12:46:34.648877+0000","last_fullsized":"2024-07-26T12:46:34.648877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:13.868870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668677+0000","last_change":"2024-07-26T12:43:12.176717+0000","last_active":"2024-07-26T12:46:34.668677+0000","last_peered":"2024-07-26T12:46:34.668677+0000","last_clean":"2024-07-26T12:46:34.668677+0000","last_became_active":"2024-07-26T12:43:12.172699+0000","last_became_peered":"2024-07-26T12:43:12.172699+0000","last_unstale":"2024-07-26T12:46:34.668677+0000","last_undegraded":"2024-07-26T12:46:34.668677+0000","last_fullsized":"2024-07-26T12:46:34.668677+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:23:36.853999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.667623+0000","last_change":"2024-07-26T12:43:12.177093+0000","last_active":"2024-07-26T12:46:34.667623+0000","last_peered":"2024-07-26T12:46:34.667623+0000","last_clean":"2024-07-26T12:46:34.667623+0000","last_became_active":"2024-07-26T12:43:12.176693+0000","last_became_peered":"2024-07-26T12:43:12.176693+0000","last_unstale":"2024-07-26T12:46:34.667623+0000","last_undegraded":"2024-07-26T12:46:34.667623+0000","last_fullsized":"2024-07-26T12:46:34.667623+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:56.875143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659723+0000","last_change":"2024-07-26T12:43:12.189531+0000","last_active":"2024-07-26T12:46:34.659723+0000","last_peered":"2024-07-26T12:46:34.659723+0000","last_clean":"2024-07-26T12:46:34.659723+0000","last_became_active":"2024-07-26T12:43:12.189162+0000","last_became_peered":"2024-07-26T12:43:12.189162+0000","last_unstale":"2024-07-26T12:46:34.659723+0000","last_undegraded":"2024-07-26T12:46:34.659723+0000","last_fullsized":"2024-07-26T12:46:34.659723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:11:00.160614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71871,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1275,"num_read_kb":716,"num_write":260,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":215613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678074,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":536466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874304,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1744896,"data_stored":1187067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037827,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":539892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234058,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":536174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430287,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36964,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708188,"statfs":{"total":95995035648,"available":95957184512,"internally_reserved":0,"allocated":1667072,"data_stored":1132812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593813,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":531300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790044,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1128394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953568,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":535374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":531008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378731,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":597931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542257,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1118208,"data_stored":596715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705781,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":533288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T12:46:46.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T12:46:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:45 smithi112 ceph-mon[37501]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T12:46:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:45 smithi132 ceph-mon[37580]: pgmap v413: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-07-26T12:46:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:46 smithi107 ceph-mon[32659]: from='client.25115 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:46:47.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:46 smithi112 ceph-mon[37501]: from='client.25115 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:46:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:46 smithi132 ceph-mon[37580]: from='client.25115 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:46:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:47 smithi107 ceph-mon[32659]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T12:46:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:47 smithi112 ceph-mon[37501]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T12:46:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:47 smithi132 ceph-mon[37580]: pgmap v414: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-07-26T12:46:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.033+0000 7f607f200640 1 -- 172.21.15.107:0/3982323501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608010af10 msgr2=0x7f608010b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.033+0000 7f607f200640 1 --2- 172.21.15.107:0/3982323501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608010af10 0x7f608010b320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6068009900 tx=0x7f606802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:46:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.034+0000 7f607f200640 1 -- 172.21.15.107:0/3982323501 shutdown_connections 2024-07-26T12:46:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.034+0000 7f607f200640 1 --2- 172.21.15.107:0/3982323501 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f608010b860 0x7f6080112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.034+0000 7f607f200640 1 --2- 172.21.15.107:0/3982323501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608010af10 0x7f608010b320 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.034+0000 7f607f200640 1 --2- 172.21.15.107:0/3982323501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f60801055d0 0x7f60801059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.034+0000 7f607f200640 1 -- 172.21.15.107:0/3982323501 >> 172.21.15.107:0/3982323501 conn(0x7f6080101040 msgr2=0x7f6080103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.034+0000 7f607f200640 1 -- 172.21.15.107:0/3982323501 shutdown_connections 2024-07-26T12:46:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.034+0000 7f607f200640 1 -- 172.21.15.107:0/3982323501 wait complete. 2024-07-26T12:46:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.035+0000 7f607f200640 1 Processor -- start 2024-07-26T12:46:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.035+0000 7f607f200640 1 -- start start 2024-07-26T12:46:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607f200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60801055d0 0x7f608019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607f200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f608010af10 0x7f608019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607f200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608010b860 0x7f60801a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607f200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6080115520 con 0x7f608010b860 2024-07-26T12:46:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607f200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f60801153a0 con 0x7f60801055d0 2024-07-26T12:46:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607f200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f60801156a0 con 0x7f608010af10 2024-07-26T12:46:49.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60801055d0 0x7f608019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:49.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60801055d0 0x7f608019bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:50344/0 (socket says 172.21.15.107:50344) 2024-07-26T12:46:49.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607de00640 1 -- 172.21.15.107:0/3225315366 learned_addr learned my addr 172.21.15.107:0/3225315366 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:49.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607e800640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608010b860 0x7f60801a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:49.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607d400640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f608010af10 0x7f608019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:49.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607de00640 1 -- 172.21.15.107:0/3225315366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f608010af10 msgr2=0x7f608019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607de00640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f608010af10 0x7f608019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607de00640 1 -- 172.21.15.107:0/3225315366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608010b860 msgr2=0x7f60801a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607de00640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608010b860 0x7f60801a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.036+0000 7f607de00640 1 -- 172.21.15.107:0/3225315366 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60801a1ec0 con 0x7f60801055d0 2024-07-26T12:46:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.037+0000 7f607de00640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60801055d0 0x7f608019bf50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f607000eab0 tx=0x7f607000ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.037+0000 7f6076a00640 1 -- 172.21.15.107:0/3225315366 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607000ce40 con 0x7f60801055d0 2024-07-26T12:46:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.037+0000 7f6076a00640 1 -- 172.21.15.107:0/3225315366 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6070011920 con 0x7f60801055d0 2024-07-26T12:46:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.037+0000 7f607f200640 1 -- 172.21.15.107:0/3225315366 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60801a21b0 con 0x7f60801055d0 2024-07-26T12:46:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.037+0000 7f6076a00640 1 -- 172.21.15.107:0/3225315366 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070010a30 con 0x7f60801055d0 2024-07-26T12:46:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.037+0000 7f607f200640 1 -- 172.21.15.107:0/3225315366 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60801a8910 con 0x7f60801055d0 2024-07-26T12:46:49.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.038+0000 7f607f200640 1 -- 172.21.15.107:0/3225315366 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60801059e0 con 0x7f60801055d0 2024-07-26T12:46:49.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.041+0000 7f6076a00640 1 -- 172.21.15.107:0/3225315366 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f6070010bd0 con 0x7f60801055d0 2024-07-26T12:46:49.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.042+0000 7f6076a00640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f60640b3c20 0x7f60640b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:49.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.042+0000 7f6076a00640 1 -- 172.21.15.107:0/3225315366 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f60700e4920 con 0x7f60801055d0 2024-07-26T12:46:49.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.042+0000 7f607d400640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f60640b3c20 0x7f60640b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:49.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.043+0000 7f607d400640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f60640b3c20 0x7f60640b6130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f606802a9e0 tx=0x7f6068005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:49.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.044+0000 7f6076a00640 1 -- 172.21.15.107:0/3225315366 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6070016030 con 0x7f60801055d0 2024-07-26T12:46:49.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.236+0000 7f607f200640 1 -- 172.21.15.107:0/3225315366 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6080063230 con 0x7f60640b3c20 2024-07-26T12:46:49.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.255+0000 7f6076a00640 1 -- 172.21.15.107:0/3225315366 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340515 (secure 0 0 0) 0x7f6080063230 con 0x7f60640b3c20 2024-07-26T12:46:49.261 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:46:49.265 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-07-26T12:46:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 -- 172.21.15.107:0/3225315366 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f60640b3c20 msgr2=0x7f60640b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f60640b3c20 0x7f60640b6130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f606802a9e0 tx=0x7f6068005a20 comp rx=0 tx=0).stop 2024-07-26T12:46:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 -- 172.21.15.107:0/3225315366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60801055d0 msgr2=0x7f608019bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60801055d0 0x7f608019bf50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f607000eab0 tx=0x7f607000ef70 comp rx=0 tx=0).stop 2024-07-26T12:46:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 -- 172.21.15.107:0/3225315366 shutdown_connections 2024-07-26T12:46:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f60640b3c20 0x7f60640b6130 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f608010b860 0x7f60801a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f608010af10 0x7f608019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 --2- 172.21.15.107:0/3225315366 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f60801055d0 0x7f608019bf50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 -- 172.21.15.107:0/3225315366 >> 172.21.15.107:0/3225315366 conn(0x7f6080101040 msgr2=0x7f6080103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 -- 172.21.15.107:0/3225315366 shutdown_connections 2024-07-26T12:46:49.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:49.261+0000 7f607f200640 1 -- 172.21.15.107:0/3225315366 wait complete. 2024-07-26T12:46:50.131 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:49 smithi107 ceph-mon[32659]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-07-26T12:46:50.135 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-07-26T12:46:48.754654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662473,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1396,"num_read_kb":810,"num_write":385,"num_write_kb":2303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2951874,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":818,"ondisk_log_size":818,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376064,"kb_used_data":14432,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565760,"statfs":{"total":1151940427776,"available":1151555338240,"internally_reserved":0,"allocated":14778368,"data_stored":8386421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":138,"num_read_kb":73,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006640"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":187,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:45.137164+0000","last_change":"2024-07-26T12:43:08.129417+0000","last_active":"2024-07-26T12:46:45.137164+0000","last_peered":"2024-07-26T12:46:45.137164+0000","last_clean":"2024-07-26T12:46:45.137164+0000","last_became_active":"2024-07-26T12:43:08.128919+0000","last_became_peered":"2024-07-26T12:43:08.128919+0000","last_unstale":"2024-07-26T12:46:45.137164+0000","last_undegraded":"2024-07-26T12:46:45.137164+0000","last_fullsized":"2024-07-26T12:46:45.137164+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:35:55.414185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":128,"num_read_kb":128,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.657938+0000","last_change":"2024-07-26T12:43:12.172058+0000","last_active":"2024-07-26T12:46:34.657938+0000","last_peered":"2024-07-26T12:46:34.657938+0000","last_clean":"2024-07-26T12:46:34.657938+0000","last_became_active":"2024-07-26T12:43:12.171631+0000","last_became_peered":"2024-07-26T12:43:12.171631+0000","last_unstale":"2024-07-26T12:46:34.657938+0000","last_undegraded":"2024-07-26T12:46:34.657938+0000","last_fullsized":"2024-07-26T12:46:34.657938+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:51.972962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.802710+0000","last_change":"2024-07-26T12:43:26.773739+0000","last_active":"2024-07-26T12:44:36.802710+0000","last_peered":"2024-07-26T12:44:36.802710+0000","last_clean":"2024-07-26T12:44:36.802710+0000","last_became_active":"2024-07-26T12:43:26.773254+0000","last_became_peered":"2024-07-26T12:43:26.773254+0000","last_unstale":"2024-07-26T12:44:36.802710+0000","last_undegraded":"2024-07-26T12:44:36.802710+0000","last_fullsized":"2024-07-26T12:44:36.802710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:03.125497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.852995+0000","last_change":"2024-07-26T12:43:08.227329+0000","last_active":"2024-07-26T12:44:36.852995+0000","last_peered":"2024-07-26T12:44:36.852995+0000","last_clean":"2024-07-26T12:44:36.852995+0000","last_became_active":"2024-07-26T12:43:08.226674+0000","last_became_peered":"2024-07-26T12:43:08.226674+0000","last_unstale":"2024-07-26T12:44:36.852995+0000","last_undegraded":"2024-07-26T12:44:36.852995+0000","last_fullsized":"2024-07-26T12:44:36.852995+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.222684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674400+0000","last_change":"2024-07-26T12:43:12.190616+0000","last_active":"2024-07-26T12:46:34.674400+0000","last_peered":"2024-07-26T12:46:34.674400+0000","last_clean":"2024-07-26T12:46:34.674400+0000","last_became_active":"2024-07-26T12:43:12.189717+0000","last_became_peered":"2024-07-26T12:43:12.189717+0000","last_unstale":"2024-07-26T12:46:34.674400+0000","last_undegraded":"2024-07-26T12:46:34.674400+0000","last_fullsized":"2024-07-26T12:46:34.674400+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:03.509592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850323+0000","last_change":"2024-07-26T12:43:26.774486+0000","last_active":"2024-07-26T12:44:36.850323+0000","last_peered":"2024-07-26T12:44:36.850323+0000","last_clean":"2024-07-26T12:44:36.850323+0000","last_became_active":"2024-07-26T12:43:26.774046+0000","last_became_peered":"2024-07-26T12:43:26.774046+0000","last_unstale":"2024-07-26T12:44:36.850323+0000","last_undegraded":"2024-07-26T12:44:36.850323+0000","last_fullsized":"2024-07-26T12:44:36.850323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.847072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881403+0000","last_change":"2024-07-26T12:43:08.123084+0000","last_active":"2024-07-26T12:44:39.881403+0000","last_peered":"2024-07-26T12:44:39.881403+0000","last_clean":"2024-07-26T12:44:39.881403+0000","last_became_active":"2024-07-26T12:43:08.122371+0000","last_became_peered":"2024-07-26T12:43:08.122371+0000","last_unstale":"2024-07-26T12:44:39.881403+0000","last_undegraded":"2024-07-26T12:44:39.881403+0000","last_fullsized":"2024-07-26T12:44:39.881403+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:27:09.665660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.670134+0000","last_change":"2024-07-26T12:43:12.170290+0000","last_active":"2024-07-26T12:46:34.670134+0000","last_peered":"2024-07-26T12:46:34.670134+0000","last_clean":"2024-07-26T12:46:34.670134+0000","last_became_active":"2024-07-26T12:43:12.169849+0000","last_became_peered":"2024-07-26T12:43:12.169849+0000","last_unstale":"2024-07-26T12:46:34.670134+0000","last_undegraded":"2024-07-26T12:46:34.670134+0000","last_fullsized":"2024-07-26T12:46:34.670134+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:46.044921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850576+0000","last_change":"2024-07-26T12:43:26.764804+0000","last_active":"2024-07-26T12:44:36.850576+0000","last_peered":"2024-07-26T12:44:36.850576+0000","last_clean":"2024-07-26T12:44:36.850576+0000","last_became_active":"2024-07-26T12:43:26.764065+0000","last_became_peered":"2024-07-26T12:43:26.764065+0000","last_unstale":"2024-07-26T12:44:36.850576+0000","last_undegraded":"2024-07-26T12:44:36.850576+0000","last_fullsized":"2024-07-26T12:44:36.850576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:16:59.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.886417+0000","last_change":"2024-07-26T12:43:08.121496+0000","last_active":"2024-07-26T12:44:39.886417+0000","last_peered":"2024-07-26T12:44:39.886417+0000","last_clean":"2024-07-26T12:44:39.886417+0000","last_became_active":"2024-07-26T12:43:08.120962+0000","last_became_peered":"2024-07-26T12:43:08.120962+0000","last_unstale":"2024-07-26T12:44:39.886417+0000","last_undegraded":"2024-07-26T12:44:39.886417+0000","last_fullsized":"2024-07-26T12:44:39.886417+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:42:02.998023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655057+0000","last_change":"2024-07-26T12:43:12.168707+0000","last_active":"2024-07-26T12:46:34.655057+0000","last_peered":"2024-07-26T12:46:34.655057+0000","last_clean":"2024-07-26T12:46:34.655057+0000","last_became_active":"2024-07-26T12:43:12.168338+0000","last_became_peered":"2024-07-26T12:43:12.168338+0000","last_unstale":"2024-07-26T12:46:34.655057+0000","last_undegraded":"2024-07-26T12:46:34.655057+0000","last_fullsized":"2024-07-26T12:46:34.655057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:39:44.724864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850496+0000","last_change":"2024-07-26T12:43:26.770863+0000","last_active":"2024-07-26T12:44:36.850496+0000","last_peered":"2024-07-26T12:44:36.850496+0000","last_clean":"2024-07-26T12:44:36.850496+0000","last_became_active":"2024-07-26T12:43:26.770480+0000","last_became_peered":"2024-07-26T12:43:26.770480+0000","last_unstale":"2024-07-26T12:44:36.850496+0000","last_undegraded":"2024-07-26T12:44:36.850496+0000","last_fullsized":"2024-07-26T12:44:36.850496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:09.128364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672434+0000","last_change":"2024-07-26T12:43:12.181659+0000","last_active":"2024-07-26T12:46:34.672434+0000","last_peered":"2024-07-26T12:46:34.672434+0000","last_clean":"2024-07-26T12:46:34.672434+0000","last_became_active":"2024-07-26T12:43:12.181268+0000","last_became_peered":"2024-07-26T12:43:12.181268+0000","last_unstale":"2024-07-26T12:46:34.672434+0000","last_undegraded":"2024-07-26T12:46:34.672434+0000","last_fullsized":"2024-07-26T12:46:34.672434+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:38:28.445981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881363+0000","last_change":"2024-07-26T12:43:35.856273+0000","last_active":"2024-07-26T12:44:39.881363+0000","last_peered":"2024-07-26T12:44:39.881363+0000","last_clean":"2024-07-26T12:44:39.881363+0000","last_became_active":"2024-07-26T12:43:35.849625+0000","last_became_peered":"2024-07-26T12:43:35.849625+0000","last_unstale":"2024-07-26T12:44:39.881363+0000","last_undegraded":"2024-07-26T12:44:39.881363+0000","last_fullsized":"2024-07-26T12:44:39.881363+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:34:45.261474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850698+0000","last_change":"2024-07-26T12:43:26.793431+0000","last_active":"2024-07-26T12:44:36.850698+0000","last_peered":"2024-07-26T12:44:36.850698+0000","last_clean":"2024-07-26T12:44:36.850698+0000","last_became_active":"2024-07-26T12:43:26.792931+0000","last_became_peered":"2024-07-26T12:43:26.792931+0000","last_unstale":"2024-07-26T12:44:36.850698+0000","last_undegraded":"2024-07-26T12:44:36.850698+0000","last_fullsized":"2024-07-26T12:44:36.850698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:31.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669836+0000","last_change":"2024-07-26T12:43:12.183784+0000","last_active":"2024-07-26T12:46:34.669836+0000","last_peered":"2024-07-26T12:46:34.669836+0000","last_clean":"2024-07-26T12:46:34.669836+0000","last_became_active":"2024-07-26T12:43:12.182696+0000","last_became_peered":"2024-07-26T12:43:12.182696+0000","last_unstale":"2024-07-26T12:46:34.669836+0000","last_undegraded":"2024-07-26T12:46:34.669836+0000","last_fullsized":"2024-07-26T12:46:34.669836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:33:19.918180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.860356+0000","last_change":"2024-07-26T12:43:08.131069+0000","last_active":"2024-07-26T12:44:38.860356+0000","last_peered":"2024-07-26T12:44:38.860356+0000","last_clean":"2024-07-26T12:44:38.860356+0000","last_became_active":"2024-07-26T12:43:08.130637+0000","last_became_peered":"2024-07-26T12:43:08.130637+0000","last_unstale":"2024-07-26T12:44:38.860356+0000","last_undegraded":"2024-07-26T12:44:38.860356+0000","last_fullsized":"2024-07-26T12:44:38.860356+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:04:16.461900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:01.141272+0000","last_change":"2024-07-26T12:43:26.763388+0000","last_active":"2024-07-26T12:45:01.141272+0000","last_peered":"2024-07-26T12:45:01.141272+0000","last_clean":"2024-07-26T12:45:01.141272+0000","last_became_active":"2024-07-26T12:43:26.762464+0000","last_became_peered":"2024-07-26T12:43:26.762464+0000","last_unstale":"2024-07-26T12:45:01.141272+0000","last_undegraded":"2024-07-26T12:45:01.141272+0000","last_fullsized":"2024-07-26T12:45:01.141272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:10:17.598266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.658539+0000","last_change":"2024-07-26T12:43:12.191640+0000","last_active":"2024-07-26T12:46:34.658539+0000","last_peered":"2024-07-26T12:46:34.658539+0000","last_clean":"2024-07-26T12:46:34.658539+0000","last_became_active":"2024-07-26T12:43:12.191260+0000","last_became_peered":"2024-07-26T12:43:12.191260+0000","last_unstale":"2024-07-26T12:46:34.658539+0000","last_undegraded":"2024-07-26T12:46:34.658539+0000","last_fullsized":"2024-07-26T12:46:34.658539+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:47:18.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.876436+0000","last_change":"2024-07-26T12:43:08.135035+0000","last_active":"2024-07-26T12:44:38.876436+0000","last_peered":"2024-07-26T12:44:38.876436+0000","last_clean":"2024-07-26T12:44:38.876436+0000","last_became_active":"2024-07-26T12:43:08.134204+0000","last_became_peered":"2024-07-26T12:43:08.134204+0000","last_unstale":"2024-07-26T12:44:38.876436+0000","last_undegraded":"2024-07-26T12:44:38.876436+0000","last_fullsized":"2024-07-26T12:44:38.876436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:38.954460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.841001+0000","last_change":"2024-07-26T12:43:35.853244+0000","last_active":"2024-07-26T12:44:36.841001+0000","last_peered":"2024-07-26T12:44:36.841001+0000","last_clean":"2024-07-26T12:44:36.841001+0000","last_became_active":"2024-07-26T12:43:35.852728+0000","last_became_peered":"2024-07-26T12:43:35.852728+0000","last_unstale":"2024-07-26T12:44:36.841001+0000","last_undegraded":"2024-07-26T12:44:36.841001+0000","last_fullsized":"2024-07-26T12:44:36.841001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:20:39.080000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.634371+0000","last_change":"2024-07-26T12:43:12.172457+0000","last_active":"2024-07-26T12:46:34.634371+0000","last_peered":"2024-07-26T12:46:34.634371+0000","last_clean":"2024-07-26T12:46:34.634371+0000","last_became_active":"2024-07-26T12:43:12.172034+0000","last_became_peered":"2024-07-26T12:43:12.172034+0000","last_unstale":"2024-07-26T12:46:34.634371+0000","last_undegraded":"2024-07-26T12:46:34.634371+0000","last_fullsized":"2024-07-26T12:46:34.634371+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:37:02.237852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807509+0000","last_change":"2024-07-26T12:43:08.227152+0000","last_active":"2024-07-26T12:44:36.807509+0000","last_peered":"2024-07-26T12:44:36.807509+0000","last_clean":"2024-07-26T12:44:36.807509+0000","last_became_active":"2024-07-26T12:43:08.226722+0000","last_became_peered":"2024-07-26T12:43:08.226722+0000","last_unstale":"2024-07-26T12:44:36.807509+0000","last_undegraded":"2024-07-26T12:44:36.807509+0000","last_fullsized":"2024-07-26T12:44:36.807509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:16.293211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850844+0000","last_change":"2024-07-26T12:43:26.792999+0000","last_active":"2024-07-26T12:44:36.850844+0000","last_peered":"2024-07-26T12:44:36.850844+0000","last_clean":"2024-07-26T12:44:36.850844+0000","last_became_active":"2024-07-26T12:43:26.792515+0000","last_became_peered":"2024-07-26T12:43:26.792515+0000","last_unstale":"2024-07-26T12:44:36.850844+0000","last_undegraded":"2024-07-26T12:44:36.850844+0000","last_fullsized":"2024-07-26T12:44:36.850844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:49:35.104600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.673077+0000","last_change":"2024-07-26T12:43:12.185381+0000","last_active":"2024-07-26T12:46:34.673077+0000","last_peered":"2024-07-26T12:46:34.673077+0000","last_clean":"2024-07-26T12:46:34.673077+0000","last_became_active":"2024-07-26T12:43:12.184892+0000","last_became_peered":"2024-07-26T12:43:12.184892+0000","last_unstale":"2024-07-26T12:46:34.673077+0000","last_undegraded":"2024-07-26T12:46:34.673077+0000","last_fullsized":"2024-07-26T12:46:34.673077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:38:53.992586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881366+0000","last_change":"2024-07-26T12:43:08.895079+0000","last_active":"2024-07-26T12:44:39.881366+0000","last_peered":"2024-07-26T12:44:39.881366+0000","last_clean":"2024-07-26T12:44:39.881366+0000","last_became_active":"2024-07-26T12:43:08.894653+0000","last_became_peered":"2024-07-26T12:43:08.894653+0000","last_unstale":"2024-07-26T12:44:39.881366+0000","last_undegraded":"2024-07-26T12:44:39.881366+0000","last_fullsized":"2024-07-26T12:44:39.881366+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:02:53.137905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803288+0000","last_change":"2024-07-26T12:43:26.762802+0000","last_active":"2024-07-26T12:44:36.803288+0000","last_peered":"2024-07-26T12:44:36.803288+0000","last_clean":"2024-07-26T12:44:36.803288+0000","last_became_active":"2024-07-26T12:43:26.762211+0000","last_became_peered":"2024-07-26T12:43:26.762211+0000","last_unstale":"2024-07-26T12:44:36.803288+0000","last_undegraded":"2024-07-26T12:44:36.803288+0000","last_fullsized":"2024-07-26T12:44:36.803288+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:35:19.623168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662724+0000","last_change":"2024-07-26T12:43:12.176062+0000","last_active":"2024-07-26T12:46:34.662724+0000","last_peered":"2024-07-26T12:46:34.662724+0000","last_clean":"2024-07-26T12:46:34.662724+0000","last_became_active":"2024-07-26T12:43:12.175671+0000","last_became_peered":"2024-07-26T12:43:12.175671+0000","last_unstale":"2024-07-26T12:46:34.662724+0000","last_undegraded":"2024-07-26T12:46:34.662724+0000","last_fullsized":"2024-07-26T12:46:34.662724+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:48.374964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'17","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.262028+0000","last_change":"2024-07-26T12:43:08.131453+0000","last_active":"2024-07-26T12:45:21.262028+0000","last_peered":"2024-07-26T12:45:21.262028+0000","last_clean":"2024-07-26T12:45:21.262028+0000","last_became_active":"2024-07-26T12:43:08.130928+0000","last_became_peered":"2024-07-26T12:43:08.130928+0000","last_unstale":"2024-07-26T12:45:21.262028+0000","last_undegraded":"2024-07-26T12:45:21.262028+0000","last_fullsized":"2024-07-26T12:45:21.262028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:43.190860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804408+0000","last_change":"2024-07-26T12:43:26.792942+0000","last_active":"2024-07-26T12:44:36.804408+0000","last_peered":"2024-07-26T12:44:36.804408+0000","last_clean":"2024-07-26T12:44:36.804408+0000","last_became_active":"2024-07-26T12:43:26.792045+0000","last_became_peered":"2024-07-26T12:43:26.792045+0000","last_unstale":"2024-07-26T12:44:36.804408+0000","last_undegraded":"2024-07-26T12:44:36.804408+0000","last_fullsized":"2024-07-26T12:44:36.804408+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:56.005295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662815+0000","last_change":"2024-07-26T12:43:12.187601+0000","last_active":"2024-07-26T12:46:34.662815+0000","last_peered":"2024-07-26T12:46:34.662815+0000","last_clean":"2024-07-26T12:46:34.662815+0000","last_became_active":"2024-07-26T12:43:12.187175+0000","last_became_peered":"2024-07-26T12:43:12.187175+0000","last_unstale":"2024-07-26T12:46:34.662815+0000","last_undegraded":"2024-07-26T12:46:34.662815+0000","last_fullsized":"2024-07-26T12:46:34.662815+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:06.992854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881924+0000","last_change":"2024-07-26T12:43:08.227279+0000","last_active":"2024-07-26T12:44:39.881924+0000","last_peered":"2024-07-26T12:44:39.881924+0000","last_clean":"2024-07-26T12:44:39.881924+0000","last_became_active":"2024-07-26T12:43:08.226634+0000","last_became_peered":"2024-07-26T12:43:08.226634+0000","last_unstale":"2024-07-26T12:44:39.881924+0000","last_undegraded":"2024-07-26T12:44:39.881924+0000","last_fullsized":"2024-07-26T12:44:39.881924+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:19.845664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851516+0000","last_change":"2024-07-26T12:43:26.774537+0000","last_active":"2024-07-26T12:44:36.851516+0000","last_peered":"2024-07-26T12:44:36.851516+0000","last_clean":"2024-07-26T12:44:36.851516+0000","last_became_active":"2024-07-26T12:43:26.774036+0000","last_became_peered":"2024-07-26T12:43:26.774036+0000","last_unstale":"2024-07-26T12:44:36.851516+0000","last_undegraded":"2024-07-26T12:44:36.851516+0000","last_fullsized":"2024-07-26T12:44:36.851516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:21.774498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662351+0000","last_change":"2024-07-26T12:44:36.857443+0000","last_active":"2024-07-26T12:46:34.662351+0000","last_peered":"2024-07-26T12:46:34.662351+0000","last_clean":"2024-07-26T12:46:34.662351+0000","last_became_active":"2024-07-26T12:44:36.857001+0000","last_became_peered":"2024-07-26T12:44:36.857001+0000","last_unstale":"2024-07-26T12:46:34.662351+0000","last_undegraded":"2024-07-26T12:46:34.662351+0000","last_fullsized":"2024-07-26T12:46:34.662351+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:08.236404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.875394+0000","last_change":"2024-07-26T12:43:08.227432+0000","last_active":"2024-07-26T12:44:37.875394+0000","last_peered":"2024-07-26T12:44:37.875394+0000","last_clean":"2024-07-26T12:44:37.875394+0000","last_became_active":"2024-07-26T12:43:08.226645+0000","last_became_peered":"2024-07-26T12:43:08.226645+0000","last_unstale":"2024-07-26T12:44:37.875394+0000","last_undegraded":"2024-07-26T12:44:37.875394+0000","last_fullsized":"2024-07-26T12:44:37.875394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:23.498855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.840331+0000","last_change":"2024-07-26T12:43:26.792874+0000","last_active":"2024-07-26T12:44:36.840331+0000","last_peered":"2024-07-26T12:44:36.840331+0000","last_clean":"2024-07-26T12:44:36.840331+0000","last_became_active":"2024-07-26T12:43:26.792371+0000","last_became_peered":"2024-07-26T12:43:26.792371+0000","last_unstale":"2024-07-26T12:44:36.840331+0000","last_undegraded":"2024-07-26T12:44:36.840331+0000","last_fullsized":"2024-07-26T12:44:36.840331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:25.753111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.653961+0000","last_change":"2024-07-26T12:43:12.185733+0000","last_active":"2024-07-26T12:46:34.653961+0000","last_peered":"2024-07-26T12:46:34.653961+0000","last_clean":"2024-07-26T12:46:34.653961+0000","last_became_active":"2024-07-26T12:43:12.185305+0000","last_became_peered":"2024-07-26T12:43:12.185305+0000","last_unstale":"2024-07-26T12:46:34.653961+0000","last_undegraded":"2024-07-26T12:46:34.653961+0000","last_fullsized":"2024-07-26T12:46:34.653961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:44.634607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":180,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:46.140249+0000","last_change":"2024-07-26T12:43:08.118508+0000","last_active":"2024-07-26T12:46:46.140249+0000","last_peered":"2024-07-26T12:46:46.140249+0000","last_clean":"2024-07-26T12:46:46.140249+0000","last_became_active":"2024-07-26T12:43:08.118088+0000","last_became_peered":"2024-07-26T12:43:08.118088+0000","last_unstale":"2024-07-26T12:46:46.140249+0000","last_undegraded":"2024-07-26T12:46:46.140249+0000","last_fullsized":"2024-07-26T12:46:46.140249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:23:46.474298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":129,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810356+0000","last_change":"2024-07-26T12:43:26.779328+0000","last_active":"2024-07-26T12:44:36.810356+0000","last_peered":"2024-07-26T12:44:36.810356+0000","last_clean":"2024-07-26T12:44:36.810356+0000","last_became_active":"2024-07-26T12:43:26.778912+0000","last_became_peered":"2024-07-26T12:43:26.778912+0000","last_unstale":"2024-07-26T12:44:36.810356+0000","last_undegraded":"2024-07-26T12:44:36.810356+0000","last_fullsized":"2024-07-26T12:44:36.810356+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:58.364775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650368+0000","last_change":"2024-07-26T12:43:35.854997+0000","last_active":"2024-07-26T12:46:34.650368+0000","last_peered":"2024-07-26T12:46:34.650368+0000","last_clean":"2024-07-26T12:46:34.650368+0000","last_became_active":"2024-07-26T12:43:35.854387+0000","last_became_peered":"2024-07-26T12:43:35.854387+0000","last_unstale":"2024-07-26T12:46:34.650368+0000","last_undegraded":"2024-07-26T12:46:34.650368+0000","last_fullsized":"2024-07-26T12:46:34.650368+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:04:20.489388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":180,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:46.208623+0000","last_change":"2024-07-26T12:43:08.227628+0000","last_active":"2024-07-26T12:46:46.208623+0000","last_peered":"2024-07-26T12:46:46.208623+0000","last_clean":"2024-07-26T12:46:46.208623+0000","last_became_active":"2024-07-26T12:43:08.227133+0000","last_became_peered":"2024-07-26T12:43:08.227133+0000","last_unstale":"2024-07-26T12:46:46.208623+0000","last_undegraded":"2024-07-26T12:46:46.208623+0000","last_fullsized":"2024-07-26T12:46:46.208623+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:28:28.484315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":129,"num_read_kb":129,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810904+0000","last_change":"2024-07-26T12:43:26.794016+0000","last_active":"2024-07-26T12:44:36.810904+0000","last_peered":"2024-07-26T12:44:36.810904+0000","last_clean":"2024-07-26T12:44:36.810904+0000","last_became_active":"2024-07-26T12:43:26.792562+0000","last_became_peered":"2024-07-26T12:43:26.792562+0000","last_unstale":"2024-07-26T12:44:36.810904+0000","last_undegraded":"2024-07-26T12:44:36.810904+0000","last_fullsized":"2024-07-26T12:44:36.810904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:26.162840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.652950+0000","last_change":"2024-07-26T12:43:12.173014+0000","last_active":"2024-07-26T12:46:34.652950+0000","last_peered":"2024-07-26T12:46:34.652950+0000","last_clean":"2024-07-26T12:46:34.652950+0000","last_became_active":"2024-07-26T12:43:12.172600+0000","last_became_peered":"2024-07-26T12:43:12.172600+0000","last_unstale":"2024-07-26T12:46:34.652950+0000","last_undegraded":"2024-07-26T12:46:34.652950+0000","last_fullsized":"2024-07-26T12:46:34.652950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:59.842744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.872830+0000","last_change":"2024-07-26T12:43:08.119000+0000","last_active":"2024-07-26T12:44:38.872830+0000","last_peered":"2024-07-26T12:44:38.872830+0000","last_clean":"2024-07-26T12:44:38.872830+0000","last_became_active":"2024-07-26T12:43:08.118608+0000","last_became_peered":"2024-07-26T12:43:08.118608+0000","last_unstale":"2024-07-26T12:44:38.872830+0000","last_undegraded":"2024-07-26T12:44:38.872830+0000","last_fullsized":"2024-07-26T12:44:38.872830+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:29:47.663449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851317+0000","last_change":"2024-07-26T12:43:35.914716+0000","last_active":"2024-07-26T12:44:36.851317+0000","last_peered":"2024-07-26T12:44:36.851317+0000","last_clean":"2024-07-26T12:44:36.851317+0000","last_became_active":"2024-07-26T12:43:35.914354+0000","last_became_peered":"2024-07-26T12:43:35.914354+0000","last_unstale":"2024-07-26T12:44:36.851317+0000","last_undegraded":"2024-07-26T12:44:36.851317+0000","last_fullsized":"2024-07-26T12:44:36.851317+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:09:10.994141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.660577+0000","last_change":"2024-07-26T12:43:12.169997+0000","last_active":"2024-07-26T12:46:34.660577+0000","last_peered":"2024-07-26T12:46:34.660577+0000","last_clean":"2024-07-26T12:46:34.660577+0000","last_became_active":"2024-07-26T12:43:12.169044+0000","last_became_peered":"2024-07-26T12:43:12.169044+0000","last_unstale":"2024-07-26T12:46:34.660577+0000","last_undegraded":"2024-07-26T12:46:34.660577+0000","last_fullsized":"2024-07-26T12:46:34.660577+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:41.791434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.882037+0000","last_change":"2024-07-26T12:43:08.129025+0000","last_active":"2024-07-26T12:44:39.882037+0000","last_peered":"2024-07-26T12:44:39.882037+0000","last_clean":"2024-07-26T12:44:39.882037+0000","last_became_active":"2024-07-26T12:43:08.128608+0000","last_became_peered":"2024-07-26T12:43:08.128608+0000","last_unstale":"2024-07-26T12:44:39.882037+0000","last_undegraded":"2024-07-26T12:44:39.882037+0000","last_fullsized":"2024-07-26T12:44:39.882037+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:43.749473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":122,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.805850+0000","last_change":"2024-07-26T12:41:42.411595+0000","last_active":"2024-07-26T12:44:36.805850+0000","last_peered":"2024-07-26T12:44:36.805850+0000","last_clean":"2024-07-26T12:44:36.805850+0000","last_became_active":"2024-07-26T12:41:42.407119+0000","last_became_peered":"2024-07-26T12:41:42.407119+0000","last_unstale":"2024-07-26T12:44:36.805850+0000","last_undegraded":"2024-07-26T12:44:36.805850+0000","last_fullsized":"2024-07-26T12:44:36.805850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_clean_scrub_stamp":"2024-07-26T12:37:59.605754+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:21.376418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:53.111058+0000","last_change":"2024-07-26T12:43:26.763955+0000","last_active":"2024-07-26T12:44:53.111058+0000","last_peered":"2024-07-26T12:44:53.111058+0000","last_clean":"2024-07-26T12:44:53.111058+0000","last_became_active":"2024-07-26T12:43:26.763368+0000","last_became_peered":"2024-07-26T12:43:26.763368+0000","last_unstale":"2024-07-26T12:44:53.111058+0000","last_undegraded":"2024-07-26T12:44:53.111058+0000","last_fullsized":"2024-07-26T12:44:53.111058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:48:14.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662998+0000","last_change":"2024-07-26T12:43:12.174780+0000","last_active":"2024-07-26T12:46:34.662998+0000","last_peered":"2024-07-26T12:46:34.662998+0000","last_clean":"2024-07-26T12:46:34.662998+0000","last_became_active":"2024-07-26T12:43:12.174379+0000","last_became_peered":"2024-07-26T12:43:12.174379+0000","last_unstale":"2024-07-26T12:46:34.662998+0000","last_undegraded":"2024-07-26T12:46:34.662998+0000","last_fullsized":"2024-07-26T12:46:34.662998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:42:22.783259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'7","reported_seq":182,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:43.087466+0000","last_change":"2024-07-26T12:43:08.894655+0000","last_active":"2024-07-26T12:46:43.087466+0000","last_peered":"2024-07-26T12:46:43.087466+0000","last_clean":"2024-07-26T12:46:43.087466+0000","last_became_active":"2024-07-26T12:43:08.894270+0000","last_became_peered":"2024-07-26T12:43:08.894270+0000","last_unstale":"2024-07-26T12:46:43.087466+0000","last_undegraded":"2024-07-26T12:46:43.087466+0000","last_fullsized":"2024-07-26T12:46:43.087466+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:34.431204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":129,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.807895+0000","last_change":"2024-07-26T12:43:26.769716+0000","last_active":"2024-07-26T12:44:36.807895+0000","last_peered":"2024-07-26T12:44:36.807895+0000","last_clean":"2024-07-26T12:44:36.807895+0000","last_became_active":"2024-07-26T12:43:26.769154+0000","last_became_peered":"2024-07-26T12:43:26.769154+0000","last_unstale":"2024-07-26T12:44:36.807895+0000","last_undegraded":"2024-07-26T12:44:36.807895+0000","last_fullsized":"2024-07-26T12:44:36.807895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:43.102498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.663935+0000","last_change":"2024-07-26T12:43:12.179488+0000","last_active":"2024-07-26T12:46:34.663935+0000","last_peered":"2024-07-26T12:46:34.663935+0000","last_clean":"2024-07-26T12:46:34.663935+0000","last_became_active":"2024-07-26T12:43:12.179171+0000","last_became_peered":"2024-07-26T12:43:12.179171+0000","last_unstale":"2024-07-26T12:46:34.663935+0000","last_undegraded":"2024-07-26T12:46:34.663935+0000","last_fullsized":"2024-07-26T12:46:34.663935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:43.695582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881415+0000","last_change":"2024-07-26T12:43:08.130838+0000","last_active":"2024-07-26T12:44:39.881415+0000","last_peered":"2024-07-26T12:44:39.881415+0000","last_clean":"2024-07-26T12:44:39.881415+0000","last_became_active":"2024-07-26T12:43:08.130446+0000","last_became_peered":"2024-07-26T12:43:08.130446+0000","last_unstale":"2024-07-26T12:44:39.881415+0000","last_undegraded":"2024-07-26T12:44:39.881415+0000","last_fullsized":"2024-07-26T12:44:39.881415+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:33:55.271038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.812663+0000","last_change":"2024-07-26T12:43:26.781982+0000","last_active":"2024-07-26T12:44:36.812663+0000","last_peered":"2024-07-26T12:44:36.812663+0000","last_clean":"2024-07-26T12:44:36.812663+0000","last_became_active":"2024-07-26T12:43:26.777102+0000","last_became_peered":"2024-07-26T12:43:26.777102+0000","last_unstale":"2024-07-26T12:44:36.812663+0000","last_undegraded":"2024-07-26T12:44:36.812663+0000","last_fullsized":"2024-07-26T12:44:36.812663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:49.553124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659275+0000","last_change":"2024-07-26T12:43:12.177043+0000","last_active":"2024-07-26T12:46:34.659275+0000","last_peered":"2024-07-26T12:46:34.659275+0000","last_clean":"2024-07-26T12:46:34.659275+0000","last_became_active":"2024-07-26T12:43:12.176634+0000","last_became_peered":"2024-07-26T12:43:12.176634+0000","last_unstale":"2024-07-26T12:46:34.659275+0000","last_undegraded":"2024-07-26T12:46:34.659275+0000","last_fullsized":"2024-07-26T12:46:34.659275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:20.210176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.878038+0000","last_change":"2024-07-26T12:43:08.131279+0000","last_active":"2024-07-26T12:44:38.878038+0000","last_peered":"2024-07-26T12:44:38.878038+0000","last_clean":"2024-07-26T12:44:38.878038+0000","last_became_active":"2024-07-26T12:43:08.130755+0000","last_became_peered":"2024-07-26T12:43:08.130755+0000","last_unstale":"2024-07-26T12:44:38.878038+0000","last_undegraded":"2024-07-26T12:44:38.878038+0000","last_fullsized":"2024-07-26T12:44:38.878038+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:03.234905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.849753+0000","last_change":"2024-07-26T12:43:26.764407+0000","last_active":"2024-07-26T12:44:36.849753+0000","last_peered":"2024-07-26T12:44:36.849753+0000","last_clean":"2024-07-26T12:44:36.849753+0000","last_became_active":"2024-07-26T12:43:26.762408+0000","last_became_peered":"2024-07-26T12:43:26.762408+0000","last_unstale":"2024-07-26T12:44:36.849753+0000","last_undegraded":"2024-07-26T12:44:36.849753+0000","last_fullsized":"2024-07-26T12:44:36.849753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:35:29.139476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.647937+0000","last_change":"2024-07-26T12:43:12.174587+0000","last_active":"2024-07-26T12:46:34.647937+0000","last_peered":"2024-07-26T12:46:34.647937+0000","last_clean":"2024-07-26T12:46:34.647937+0000","last_became_active":"2024-07-26T12:43:12.173929+0000","last_became_peered":"2024-07-26T12:43:12.173929+0000","last_unstale":"2024-07-26T12:46:34.647937+0000","last_undegraded":"2024-07-26T12:46:34.647937+0000","last_fullsized":"2024-07-26T12:46:34.647937+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:16.775198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881722+0000","last_change":"2024-07-26T12:43:08.119825+0000","last_active":"2024-07-26T12:44:39.881722+0000","last_peered":"2024-07-26T12:44:39.881722+0000","last_clean":"2024-07-26T12:44:39.881722+0000","last_became_active":"2024-07-26T12:43:08.119273+0000","last_became_peered":"2024-07-26T12:43:08.119273+0000","last_unstale":"2024-07-26T12:44:39.881722+0000","last_undegraded":"2024-07-26T12:44:39.881722+0000","last_fullsized":"2024-07-26T12:44:39.881722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:39.973020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850064+0000","last_change":"2024-07-26T12:43:26.773226+0000","last_active":"2024-07-26T12:44:36.850064+0000","last_peered":"2024-07-26T12:44:36.850064+0000","last_clean":"2024-07-26T12:44:36.850064+0000","last_became_active":"2024-07-26T12:43:26.772745+0000","last_became_peered":"2024-07-26T12:43:26.772745+0000","last_unstale":"2024-07-26T12:44:36.850064+0000","last_undegraded":"2024-07-26T12:44:36.850064+0000","last_fullsized":"2024-07-26T12:44:36.850064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:35.407138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.648152+0000","last_change":"2024-07-26T12:43:12.176178+0000","last_active":"2024-07-26T12:46:34.648152+0000","last_peered":"2024-07-26T12:46:34.648152+0000","last_clean":"2024-07-26T12:46:34.648152+0000","last_became_active":"2024-07-26T12:43:12.175791+0000","last_became_peered":"2024-07-26T12:43:12.175791+0000","last_unstale":"2024-07-26T12:46:34.648152+0000","last_undegraded":"2024-07-26T12:46:34.648152+0000","last_fullsized":"2024-07-26T12:46:34.648152+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:48.885831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881993+0000","last_change":"2024-07-26T12:43:08.226345+0000","last_active":"2024-07-26T12:44:39.881993+0000","last_peered":"2024-07-26T12:44:39.881993+0000","last_clean":"2024-07-26T12:44:39.881993+0000","last_became_active":"2024-07-26T12:43:08.225897+0000","last_became_peered":"2024-07-26T12:43:08.225897+0000","last_unstale":"2024-07-26T12:44:39.881993+0000","last_undegraded":"2024-07-26T12:44:39.881993+0000","last_fullsized":"2024-07-26T12:44:39.881993+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:59.351076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.810778+0000","last_change":"2024-07-26T12:43:26.774800+0000","last_active":"2024-07-26T12:44:36.810778+0000","last_peered":"2024-07-26T12:44:36.810778+0000","last_clean":"2024-07-26T12:44:36.810778+0000","last_became_active":"2024-07-26T12:43:26.774192+0000","last_became_peered":"2024-07-26T12:43:26.774192+0000","last_unstale":"2024-07-26T12:44:36.810778+0000","last_undegraded":"2024-07-26T12:44:36.810778+0000","last_fullsized":"2024-07-26T12:44:36.810778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:18.379619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659090+0000","last_change":"2024-07-26T12:43:12.171720+0000","last_active":"2024-07-26T12:46:34.659090+0000","last_peered":"2024-07-26T12:46:34.659090+0000","last_clean":"2024-07-26T12:46:34.659090+0000","last_became_active":"2024-07-26T12:43:12.171334+0000","last_became_peered":"2024-07-26T12:43:12.171334+0000","last_unstale":"2024-07-26T12:46:34.659090+0000","last_undegraded":"2024-07-26T12:46:34.659090+0000","last_fullsized":"2024-07-26T12:46:34.659090+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:35.156430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":180,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:45.206928+0000","last_change":"2024-07-26T12:43:08.120325+0000","last_active":"2024-07-26T12:46:45.206928+0000","last_peered":"2024-07-26T12:46:45.206928+0000","last_clean":"2024-07-26T12:46:45.206928+0000","last_became_active":"2024-07-26T12:43:08.119896+0000","last_became_peered":"2024-07-26T12:43:08.119896+0000","last_unstale":"2024-07-26T12:46:45.206928+0000","last_undegraded":"2024-07-26T12:46:45.206928+0000","last_fullsized":"2024-07-26T12:46:45.206928+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:23:19.470491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":130,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.856219+0000","last_change":"2024-07-26T12:43:26.778688+0000","last_active":"2024-07-26T12:44:36.856219+0000","last_peered":"2024-07-26T12:44:36.856219+0000","last_clean":"2024-07-26T12:44:36.856219+0000","last_became_active":"2024-07-26T12:43:26.778239+0000","last_became_peered":"2024-07-26T12:43:26.778239+0000","last_unstale":"2024-07-26T12:44:36.856219+0000","last_undegraded":"2024-07-26T12:44:36.856219+0000","last_fullsized":"2024-07-26T12:44:36.856219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:50:48.409636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.676452+0000","last_change":"2024-07-26T12:43:12.182323+0000","last_active":"2024-07-26T12:46:34.676452+0000","last_peered":"2024-07-26T12:46:34.676452+0000","last_clean":"2024-07-26T12:46:34.676452+0000","last_became_active":"2024-07-26T12:43:12.181860+0000","last_became_peered":"2024-07-26T12:43:12.181860+0000","last_unstale":"2024-07-26T12:46:34.676452+0000","last_undegraded":"2024-07-26T12:46:34.676452+0000","last_fullsized":"2024-07-26T12:46:34.676452+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:08.764979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881366+0000","last_change":"2024-07-26T12:43:08.130823+0000","last_active":"2024-07-26T12:44:39.881366+0000","last_peered":"2024-07-26T12:44:39.881366+0000","last_clean":"2024-07-26T12:44:39.881366+0000","last_became_active":"2024-07-26T12:43:08.130424+0000","last_became_peered":"2024-07-26T12:43:08.130424+0000","last_unstale":"2024-07-26T12:44:39.881366+0000","last_undegraded":"2024-07-26T12:44:39.881366+0000","last_fullsized":"2024-07-26T12:44:39.881366+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:49:35.829624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850734+0000","last_change":"2024-07-26T12:43:26.770070+0000","last_active":"2024-07-26T12:44:36.850734+0000","last_peered":"2024-07-26T12:44:36.850734+0000","last_clean":"2024-07-26T12:44:36.850734+0000","last_became_active":"2024-07-26T12:43:26.769594+0000","last_became_peered":"2024-07-26T12:43:26.769594+0000","last_unstale":"2024-07-26T12:44:36.850734+0000","last_undegraded":"2024-07-26T12:44:36.850734+0000","last_fullsized":"2024-07-26T12:44:36.850734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:18.342739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669154+0000","last_change":"2024-07-26T12:43:12.183134+0000","last_active":"2024-07-26T12:46:34.669154+0000","last_peered":"2024-07-26T12:46:34.669154+0000","last_clean":"2024-07-26T12:46:34.669154+0000","last_became_active":"2024-07-26T12:43:12.182406+0000","last_became_peered":"2024-07-26T12:43:12.182406+0000","last_unstale":"2024-07-26T12:46:34.669154+0000","last_undegraded":"2024-07-26T12:46:34.669154+0000","last_fullsized":"2024-07-26T12:46:34.669154+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:54:24.038971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.881861+0000","last_change":"2024-07-26T12:43:35.862005+0000","last_active":"2024-07-26T12:44:39.881861+0000","last_peered":"2024-07-26T12:44:39.881861+0000","last_clean":"2024-07-26T12:44:39.881861+0000","last_became_active":"2024-07-26T12:43:35.850911+0000","last_became_peered":"2024-07-26T12:43:35.850911+0000","last_unstale":"2024-07-26T12:44:39.881861+0000","last_undegraded":"2024-07-26T12:44:39.881861+0000","last_fullsized":"2024-07-26T12:44:39.881861+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:18:16.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850203+0000","last_change":"2024-07-26T12:43:26.774599+0000","last_active":"2024-07-26T12:44:36.850203+0000","last_peered":"2024-07-26T12:44:36.850203+0000","last_clean":"2024-07-26T12:44:36.850203+0000","last_became_active":"2024-07-26T12:43:26.774186+0000","last_became_peered":"2024-07-26T12:43:26.774186+0000","last_unstale":"2024-07-26T12:44:36.850203+0000","last_undegraded":"2024-07-26T12:44:36.850203+0000","last_fullsized":"2024-07-26T12:44:36.850203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:02:42.252093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.633333+0000","last_change":"2024-07-26T12:43:12.174676+0000","last_active":"2024-07-26T12:46:34.633333+0000","last_peered":"2024-07-26T12:46:34.633333+0000","last_clean":"2024-07-26T12:46:34.633333+0000","last_became_active":"2024-07-26T12:43:12.174310+0000","last_became_peered":"2024-07-26T12:43:12.174310+0000","last_unstale":"2024-07-26T12:46:34.633333+0000","last_undegraded":"2024-07-26T12:46:34.633333+0000","last_fullsized":"2024-07-26T12:46:34.633333+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:44.381289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'39","reported_seq":351,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:47.092709+0000","last_change":"2024-07-26T12:43:08.122675+0000","last_active":"2024-07-26T12:46:47.092709+0000","last_peered":"2024-07-26T12:46:47.092709+0000","last_clean":"2024-07-26T12:46:47.092709+0000","last_became_active":"2024-07-26T12:43:08.122174+0000","last_became_peered":"2024-07-26T12:43:08.122174+0000","last_unstale":"2024-07-26T12:46:47.092709+0000","last_undegraded":"2024-07-26T12:46:47.092709+0000","last_fullsized":"2024-07-26T12:46:47.092709+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:54.513306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64385,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":196,"num_write":41,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.854144+0000","last_change":"2024-07-26T12:43:26.764283+0000","last_active":"2024-07-26T12:44:36.854144+0000","last_peered":"2024-07-26T12:44:36.854144+0000","last_clean":"2024-07-26T12:44:36.854144+0000","last_became_active":"2024-07-26T12:43:26.763821+0000","last_became_peered":"2024-07-26T12:43:26.763821+0000","last_unstale":"2024-07-26T12:44:36.854144+0000","last_undegraded":"2024-07-26T12:44:36.854144+0000","last_fullsized":"2024-07-26T12:44:36.854144+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:56.753016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672115+0000","last_change":"2024-07-26T12:43:35.915277+0000","last_active":"2024-07-26T12:46:34.672115+0000","last_peered":"2024-07-26T12:46:34.672115+0000","last_clean":"2024-07-26T12:46:34.672115+0000","last_became_active":"2024-07-26T12:43:35.914848+0000","last_became_peered":"2024-07-26T12:43:35.914848+0000","last_unstale":"2024-07-26T12:46:34.672115+0000","last_undegraded":"2024-07-26T12:46:34.672115+0000","last_fullsized":"2024-07-26T12:46:34.672115+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:56:35.639076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.885252+0000","last_change":"2024-07-26T12:43:08.227352+0000","last_active":"2024-07-26T12:44:39.885252+0000","last_peered":"2024-07-26T12:44:39.885252+0000","last_clean":"2024-07-26T12:44:39.885252+0000","last_became_active":"2024-07-26T12:43:08.226857+0000","last_became_peered":"2024-07-26T12:43:08.226857+0000","last_unstale":"2024-07-26T12:44:39.885252+0000","last_undegraded":"2024-07-26T12:44:39.885252+0000","last_fullsized":"2024-07-26T12:44:39.885252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:20.156503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.850749+0000","last_change":"2024-07-26T12:43:26.762943+0000","last_active":"2024-07-26T12:44:36.850749+0000","last_peered":"2024-07-26T12:44:36.850749+0000","last_clean":"2024-07-26T12:44:36.850749+0000","last_became_active":"2024-07-26T12:43:26.762436+0000","last_became_peered":"2024-07-26T12:43:26.762436+0000","last_unstale":"2024-07-26T12:44:36.850749+0000","last_undegraded":"2024-07-26T12:44:36.850749+0000","last_fullsized":"2024-07-26T12:44:36.850749+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:05:23.924397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.673105+0000","last_change":"2024-07-26T12:44:36.854705+0000","last_active":"2024-07-26T12:46:34.673105+0000","last_peered":"2024-07-26T12:46:34.673105+0000","last_clean":"2024-07-26T12:46:34.673105+0000","last_became_active":"2024-07-26T12:44:36.854268+0000","last_became_peered":"2024-07-26T12:44:36.854268+0000","last_unstale":"2024-07-26T12:46:34.673105+0000","last_undegraded":"2024-07-26T12:46:34.673105+0000","last_fullsized":"2024-07-26T12:46:34.673105+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:24:30.768905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.874653+0000","last_change":"2024-07-26T12:43:08.118597+0000","last_active":"2024-07-26T12:44:38.874653+0000","last_peered":"2024-07-26T12:44:38.874653+0000","last_clean":"2024-07-26T12:44:38.874653+0000","last_became_active":"2024-07-26T12:43:08.118145+0000","last_became_peered":"2024-07-26T12:43:08.118145+0000","last_unstale":"2024-07-26T12:44:38.874653+0000","last_undegraded":"2024-07-26T12:44:38.874653+0000","last_fullsized":"2024-07-26T12:44:38.874653+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:52:45.511693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.853684+0000","last_change":"2024-07-26T12:43:26.771861+0000","last_active":"2024-07-26T12:44:36.853684+0000","last_peered":"2024-07-26T12:44:36.853684+0000","last_clean":"2024-07-26T12:44:36.853684+0000","last_became_active":"2024-07-26T12:43:26.771323+0000","last_became_peered":"2024-07-26T12:43:26.771323+0000","last_unstale":"2024-07-26T12:44:36.853684+0000","last_undegraded":"2024-07-26T12:44:36.853684+0000","last_fullsized":"2024-07-26T12:44:36.853684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:34.207514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656373+0000","last_change":"2024-07-26T12:43:12.169437+0000","last_active":"2024-07-26T12:46:34.656373+0000","last_peered":"2024-07-26T12:46:34.656373+0000","last_clean":"2024-07-26T12:46:34.656373+0000","last_became_active":"2024-07-26T12:43:12.168387+0000","last_became_peered":"2024-07-26T12:43:12.168387+0000","last_unstale":"2024-07-26T12:46:34.656373+0000","last_undegraded":"2024-07-26T12:46:34.656373+0000","last_fullsized":"2024-07-26T12:46:34.656373+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:57:06.148965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'6","reported_seq":308,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:47.092705+0000","last_change":"2024-07-26T12:43:08.129618+0000","last_active":"2024-07-26T12:46:47.092705+0000","last_peered":"2024-07-26T12:46:47.092705+0000","last_clean":"2024-07-26T12:46:47.092705+0000","last_became_active":"2024-07-26T12:43:08.129064+0000","last_became_peered":"2024-07-26T12:43:08.129064+0000","last_unstale":"2024-07-26T12:46:47.092705+0000","last_undegraded":"2024-07-26T12:46:47.092705+0000","last_fullsized":"2024-07-26T12:46:47.092705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:26:22.406075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":256,"num_read_kb":256,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.804420+0000","last_change":"2024-07-26T12:43:35.854710+0000","last_active":"2024-07-26T12:44:36.804420+0000","last_peered":"2024-07-26T12:44:36.804420+0000","last_clean":"2024-07-26T12:44:36.804420+0000","last_became_active":"2024-07-26T12:43:35.852772+0000","last_became_peered":"2024-07-26T12:43:35.852772+0000","last_unstale":"2024-07-26T12:44:36.804420+0000","last_undegraded":"2024-07-26T12:44:36.804420+0000","last_fullsized":"2024-07-26T12:44:36.804420+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:24:54.015879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.813340+0000","last_change":"2024-07-26T12:43:26.783182+0000","last_active":"2024-07-26T12:44:36.813340+0000","last_peered":"2024-07-26T12:44:36.813340+0000","last_clean":"2024-07-26T12:44:36.813340+0000","last_became_active":"2024-07-26T12:43:26.782697+0000","last_became_peered":"2024-07-26T12:43:26.782697+0000","last_unstale":"2024-07-26T12:44:36.813340+0000","last_undegraded":"2024-07-26T12:44:36.813340+0000","last_fullsized":"2024-07-26T12:44:36.813340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:57.169222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.666849+0000","last_change":"2024-07-26T12:43:12.166163+0000","last_active":"2024-07-26T12:46:34.666849+0000","last_peered":"2024-07-26T12:46:34.666849+0000","last_clean":"2024-07-26T12:46:34.666849+0000","last_became_active":"2024-07-26T12:43:12.164427+0000","last_became_peered":"2024-07-26T12:43:12.164427+0000","last_unstale":"2024-07-26T12:46:34.666849+0000","last_undegraded":"2024-07-26T12:46:34.666849+0000","last_fullsized":"2024-07-26T12:46:34.666849+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:25.335508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:39.885502+0000","last_change":"2024-07-26T12:43:08.118224+0000","last_active":"2024-07-26T12:44:39.885502+0000","last_peered":"2024-07-26T12:44:39.885502+0000","last_clean":"2024-07-26T12:44:39.885502+0000","last_became_active":"2024-07-26T12:43:08.117821+0000","last_became_peered":"2024-07-26T12:43:08.117821+0000","last_unstale":"2024-07-26T12:44:39.885502+0000","last_undegraded":"2024-07-26T12:44:39.885502+0000","last_fullsized":"2024-07-26T12:44:39.885502+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:56:14.380931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.851235+0000","last_change":"2024-07-26T12:43:35.907740+0000","last_active":"2024-07-26T12:44:36.851235+0000","last_peered":"2024-07-26T12:44:36.851235+0000","last_clean":"2024-07-26T12:44:36.851235+0000","last_became_active":"2024-07-26T12:43:35.907115+0000","last_became_peered":"2024-07-26T12:43:35.907115+0000","last_unstale":"2024-07-26T12:44:36.851235+0000","last_undegraded":"2024-07-26T12:44:36.851235+0000","last_fullsized":"2024-07-26T12:44:36.851235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:53:24.783872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.670083+0000","last_change":"2024-07-26T12:43:12.177579+0000","last_active":"2024-07-26T12:46:34.670083+0000","last_peered":"2024-07-26T12:46:34.670083+0000","last_clean":"2024-07-26T12:46:34.670083+0000","last_became_active":"2024-07-26T12:43:12.171433+0000","last_became_peered":"2024-07-26T12:43:12.171433+0000","last_unstale":"2024-07-26T12:46:34.670083+0000","last_undegraded":"2024-07-26T12:46:34.670083+0000","last_fullsized":"2024-07-26T12:46:34.670083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:50.972151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:38.877099+0000","last_change":"2024-07-26T12:43:08.133446+0000","last_active":"2024-07-26T12:44:38.877099+0000","last_peered":"2024-07-26T12:44:38.877099+0000","last_clean":"2024-07-26T12:44:38.877099+0000","last_became_active":"2024-07-26T12:43:08.133008+0000","last_became_peered":"2024-07-26T12:43:08.133008+0000","last_unstale":"2024-07-26T12:44:38.877099+0000","last_undegraded":"2024-07-26T12:44:38.877099+0000","last_fullsized":"2024-07-26T12:44:38.877099+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:05:21.283738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:37.874509+0000","last_change":"2024-07-26T12:43:08.124976+0000","last_active":"2024-07-26T12:44:37.874509+0000","last_peered":"2024-07-26T12:44:37.874509+0000","last_clean":"2024-07-26T12:44:37.874509+0000","last_became_active":"2024-07-26T12:43:08.124664+0000","last_became_peered":"2024-07-26T12:43:08.124664+0000","last_unstale":"2024-07-26T12:44:37.874509+0000","last_undegraded":"2024-07-26T12:44:37.874509+0000","last_fullsized":"2024-07-26T12:44:37.874509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:51.798657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664094+0000","last_change":"2024-07-26T12:43:35.856266+0000","last_active":"2024-07-26T12:46:34.664094+0000","last_peered":"2024-07-26T12:46:34.664094+0000","last_clean":"2024-07-26T12:46:34.664094+0000","last_became_active":"2024-07-26T12:43:35.855802+0000","last_became_peered":"2024-07-26T12:43:35.855802+0000","last_unstale":"2024-07-26T12:46:34.664094+0000","last_undegraded":"2024-07-26T12:46:34.664094+0000","last_fullsized":"2024-07-26T12:46:34.664094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:16.432983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.802866+0000","last_change":"2024-07-26T12:43:26.778626+0000","last_active":"2024-07-26T12:44:36.802866+0000","last_peered":"2024-07-26T12:44:36.802866+0000","last_clean":"2024-07-26T12:44:36.802866+0000","last_became_active":"2024-07-26T12:43:26.778165+0000","last_became_peered":"2024-07-26T12:43:26.778165+0000","last_unstale":"2024-07-26T12:44:36.802866+0000","last_undegraded":"2024-07-26T12:44:36.802866+0000","last_fullsized":"2024-07-26T12:44:36.802866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:06:08.766495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":181,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:42.085334+0000","last_change":"2024-07-26T12:43:08.119702+0000","last_active":"2024-07-26T12:46:42.085334+0000","last_peered":"2024-07-26T12:46:42.085334+0000","last_clean":"2024-07-26T12:46:42.085334+0000","last_became_active":"2024-07-26T12:43:08.119225+0000","last_became_peered":"2024-07-26T12:43:08.119225+0000","last_unstale":"2024-07-26T12:46:42.085334+0000","last_undegraded":"2024-07-26T12:46:42.085334+0000","last_fullsized":"2024-07-26T12:46:42.085334+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:27.616873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.640444+0000","last_change":"2024-07-26T12:43:12.166239+0000","last_active":"2024-07-26T12:46:34.640444+0000","last_peered":"2024-07-26T12:46:34.640444+0000","last_clean":"2024-07-26T12:46:34.640444+0000","last_became_active":"2024-07-26T12:43:12.165770+0000","last_became_peered":"2024-07-26T12:43:12.165770+0000","last_unstale":"2024-07-26T12:46:34.640444+0000","last_undegraded":"2024-07-26T12:46:34.640444+0000","last_fullsized":"2024-07-26T12:46:34.640444+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:11:15.290898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:44:36.803067+0000","last_change":"2024-07-26T12:43:26.765905+0000","last_active":"2024-07-26T12:44:36.803067+0000","last_peered":"2024-07-26T12:44:36.803067+0000","last_clean":"2024-07-26T12:44:36.803067+0000","last_became_active":"2024-07-26T12:43:26.765425+0000","last_became_peered":"2024-07-26T12:43:26.765425+0000","last_unstale":"2024-07-26T12:44:36.803067+0000","last_undegraded":"2024-07-26T12:44:36.803067+0000","last_fullsized":"2024-07-26T12:44:36.803067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:16.786604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644926+0000","last_change":"2024-07-26T12:43:12.171400+0000","last_active":"2024-07-26T12:46:34.644926+0000","last_peered":"2024-07-26T12:46:34.644926+0000","last_clean":"2024-07-26T12:46:34.644926+0000","last_became_active":"2024-07-26T12:43:12.170998+0000","last_became_peered":"2024-07-26T12:43:12.170998+0000","last_unstale":"2024-07-26T12:46:34.644926+0000","last_undegraded":"2024-07-26T12:46:34.644926+0000","last_fullsized":"2024-07-26T12:46:34.644926+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:43.251975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656641+0000","last_change":"2024-07-26T12:43:35.912729+0000","last_active":"2024-07-26T12:46:34.656641+0000","last_peered":"2024-07-26T12:46:34.656641+0000","last_clean":"2024-07-26T12:46:34.656641+0000","last_became_active":"2024-07-26T12:43:35.912318+0000","last_became_peered":"2024-07-26T12:43:35.912318+0000","last_unstale":"2024-07-26T12:46:34.656641+0000","last_undegraded":"2024-07-26T12:46:34.656641+0000","last_fullsized":"2024-07-26T12:46:34.656641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:14.473536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661247+0000","last_change":"2024-07-26T12:43:12.182933+0000","last_active":"2024-07-26T12:46:34.661247+0000","last_peered":"2024-07-26T12:46:34.661247+0000","last_clean":"2024-07-26T12:46:34.661247+0000","last_became_active":"2024-07-26T12:43:12.182656+0000","last_became_peered":"2024-07-26T12:43:12.182656+0000","last_unstale":"2024-07-26T12:46:34.661247+0000","last_undegraded":"2024-07-26T12:46:34.661247+0000","last_fullsized":"2024-07-26T12:46:34.661247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:05:59.717662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674579+0000","last_change":"2024-07-26T12:43:12.181196+0000","last_active":"2024-07-26T12:46:34.674579+0000","last_peered":"2024-07-26T12:46:34.674579+0000","last_clean":"2024-07-26T12:46:34.674579+0000","last_became_active":"2024-07-26T12:43:12.180765+0000","last_became_peered":"2024-07-26T12:43:12.180765+0000","last_unstale":"2024-07-26T12:46:34.674579+0000","last_undegraded":"2024-07-26T12:46:34.674579+0000","last_fullsized":"2024-07-26T12:46:34.674579+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:20.590023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.653653+0000","last_change":"2024-07-26T12:43:12.185408+0000","last_active":"2024-07-26T12:46:34.653653+0000","last_peered":"2024-07-26T12:46:34.653653+0000","last_clean":"2024-07-26T12:46:34.653653+0000","last_became_active":"2024-07-26T12:43:12.185038+0000","last_became_peered":"2024-07-26T12:43:12.185038+0000","last_unstale":"2024-07-26T12:46:34.653653+0000","last_undegraded":"2024-07-26T12:46:34.653653+0000","last_fullsized":"2024-07-26T12:46:34.653653+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:55:30.604995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644286+0000","last_change":"2024-07-26T12:43:12.169301+0000","last_active":"2024-07-26T12:46:34.644286+0000","last_peered":"2024-07-26T12:46:34.644286+0000","last_clean":"2024-07-26T12:46:34.644286+0000","last_became_active":"2024-07-26T12:43:12.168923+0000","last_became_peered":"2024-07-26T12:43:12.168923+0000","last_unstale":"2024-07-26T12:46:34.644286+0000","last_undegraded":"2024-07-26T12:46:34.644286+0000","last_fullsized":"2024-07-26T12:46:34.644286+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:04.506485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.647408+0000","last_change":"2024-07-26T12:43:12.166663+0000","last_active":"2024-07-26T12:46:34.647408+0000","last_peered":"2024-07-26T12:46:34.647408+0000","last_clean":"2024-07-26T12:46:34.647408+0000","last_became_active":"2024-07-26T12:43:12.166200+0000","last_became_peered":"2024-07-26T12:43:12.166200+0000","last_unstale":"2024-07-26T12:46:34.647408+0000","last_undegraded":"2024-07-26T12:46:34.647408+0000","last_fullsized":"2024-07-26T12:46:34.647408+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:14.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.660632+0000","last_change":"2024-07-26T12:43:12.171273+0000","last_active":"2024-07-26T12:46:34.660632+0000","last_peered":"2024-07-26T12:46:34.660632+0000","last_clean":"2024-07-26T12:46:34.660632+0000","last_became_active":"2024-07-26T12:43:12.167680+0000","last_became_peered":"2024-07-26T12:43:12.167680+0000","last_unstale":"2024-07-26T12:46:34.660632+0000","last_undegraded":"2024-07-26T12:46:34.660632+0000","last_fullsized":"2024-07-26T12:46:34.660632+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:44.797644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.667348+0000","last_change":"2024-07-26T12:43:12.179891+0000","last_active":"2024-07-26T12:46:34.667348+0000","last_peered":"2024-07-26T12:46:34.667348+0000","last_clean":"2024-07-26T12:46:34.667348+0000","last_became_active":"2024-07-26T12:43:12.179518+0000","last_became_peered":"2024-07-26T12:43:12.179518+0000","last_unstale":"2024-07-26T12:46:34.667348+0000","last_undegraded":"2024-07-26T12:46:34.667348+0000","last_fullsized":"2024-07-26T12:46:34.667348+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:24.246023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668993+0000","last_change":"2024-07-26T12:43:12.168521+0000","last_active":"2024-07-26T12:46:34.668993+0000","last_peered":"2024-07-26T12:46:34.668993+0000","last_clean":"2024-07-26T12:46:34.668993+0000","last_became_active":"2024-07-26T12:43:12.168126+0000","last_became_peered":"2024-07-26T12:43:12.168126+0000","last_unstale":"2024-07-26T12:46:34.668993+0000","last_undegraded":"2024-07-26T12:46:34.668993+0000","last_fullsized":"2024-07-26T12:46:34.668993+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:00.135090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661477+0000","last_change":"2024-07-26T12:44:36.843322+0000","last_active":"2024-07-26T12:46:34.661477+0000","last_peered":"2024-07-26T12:46:34.661477+0000","last_clean":"2024-07-26T12:46:34.661477+0000","last_became_active":"2024-07-26T12:44:36.842821+0000","last_became_peered":"2024-07-26T12:44:36.842821+0000","last_unstale":"2024-07-26T12:46:34.661477+0000","last_undegraded":"2024-07-26T12:46:34.661477+0000","last_fullsized":"2024-07-26T12:46:34.661477+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:05.609298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674453+0000","last_change":"2024-07-26T12:43:12.185565+0000","last_active":"2024-07-26T12:46:34.674453+0000","last_peered":"2024-07-26T12:46:34.674453+0000","last_clean":"2024-07-26T12:46:34.674453+0000","last_became_active":"2024-07-26T12:43:12.185057+0000","last_became_peered":"2024-07-26T12:43:12.185057+0000","last_unstale":"2024-07-26T12:46:34.674453+0000","last_undegraded":"2024-07-26T12:46:34.674453+0000","last_fullsized":"2024-07-26T12:46:34.674453+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:58.108311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644559+0000","last_change":"2024-07-26T12:43:12.173301+0000","last_active":"2024-07-26T12:46:34.644559+0000","last_peered":"2024-07-26T12:46:34.644559+0000","last_clean":"2024-07-26T12:46:34.644559+0000","last_became_active":"2024-07-26T12:43:12.172955+0000","last_became_peered":"2024-07-26T12:43:12.172955+0000","last_unstale":"2024-07-26T12:46:34.644559+0000","last_undegraded":"2024-07-26T12:46:34.644559+0000","last_fullsized":"2024-07-26T12:46:34.644559+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:41.731141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661375+0000","last_change":"2024-07-26T12:43:12.176169+0000","last_active":"2024-07-26T12:46:34.661375+0000","last_peered":"2024-07-26T12:46:34.661375+0000","last_clean":"2024-07-26T12:46:34.661375+0000","last_became_active":"2024-07-26T12:43:12.175745+0000","last_became_peered":"2024-07-26T12:43:12.175745+0000","last_unstale":"2024-07-26T12:46:34.661375+0000","last_undegraded":"2024-07-26T12:46:34.661375+0000","last_fullsized":"2024-07-26T12:46:34.661375+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:20.393207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.654280+0000","last_change":"2024-07-26T12:43:12.168182+0000","last_active":"2024-07-26T12:46:34.654280+0000","last_peered":"2024-07-26T12:46:34.654280+0000","last_clean":"2024-07-26T12:46:34.654280+0000","last_became_active":"2024-07-26T12:43:12.163771+0000","last_became_peered":"2024-07-26T12:43:12.163771+0000","last_unstale":"2024-07-26T12:46:34.654280+0000","last_undegraded":"2024-07-26T12:46:34.654280+0000","last_fullsized":"2024-07-26T12:46:34.654280+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:40:30.960153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659998+0000","last_change":"2024-07-26T12:43:12.189524+0000","last_active":"2024-07-26T12:46:34.659998+0000","last_peered":"2024-07-26T12:46:34.659998+0000","last_clean":"2024-07-26T12:46:34.659998+0000","last_became_active":"2024-07-26T12:43:12.189165+0000","last_became_peered":"2024-07-26T12:43:12.189165+0000","last_unstale":"2024-07-26T12:46:34.659998+0000","last_undegraded":"2024-07-26T12:46:34.659998+0000","last_fullsized":"2024-07-26T12:46:34.659998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:11:08.479058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.665264+0000","last_change":"2024-07-26T12:43:12.180603+0000","last_active":"2024-07-26T12:46:34.665264+0000","last_peered":"2024-07-26T12:46:34.665264+0000","last_clean":"2024-07-26T12:46:34.665264+0000","last_became_active":"2024-07-26T12:43:12.180204+0000","last_became_peered":"2024-07-26T12:43:12.180204+0000","last_unstale":"2024-07-26T12:46:34.665264+0000","last_undegraded":"2024-07-26T12:46:34.665264+0000","last_fullsized":"2024-07-26T12:46:34.665264+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:46.768213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.667944+0000","last_change":"2024-07-26T12:43:12.185347+0000","last_active":"2024-07-26T12:46:34.667944+0000","last_peered":"2024-07-26T12:46:34.667944+0000","last_clean":"2024-07-26T12:46:34.667944+0000","last_became_active":"2024-07-26T12:43:12.180190+0000","last_became_peered":"2024-07-26T12:43:12.180190+0000","last_unstale":"2024-07-26T12:46:34.667944+0000","last_undegraded":"2024-07-26T12:46:34.667944+0000","last_fullsized":"2024-07-26T12:46:34.667944+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:07.654967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.665258+0000","last_change":"2024-07-26T12:43:12.177258+0000","last_active":"2024-07-26T12:46:34.665258+0000","last_peered":"2024-07-26T12:46:34.665258+0000","last_clean":"2024-07-26T12:46:34.665258+0000","last_became_active":"2024-07-26T12:43:12.176879+0000","last_became_peered":"2024-07-26T12:43:12.176879+0000","last_unstale":"2024-07-26T12:46:34.665258+0000","last_undegraded":"2024-07-26T12:46:34.665258+0000","last_fullsized":"2024-07-26T12:46:34.665258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:24.244759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674886+0000","last_change":"2024-07-26T12:44:36.857403+0000","last_active":"2024-07-26T12:46:34.674886+0000","last_peered":"2024-07-26T12:46:34.674886+0000","last_clean":"2024-07-26T12:46:34.674886+0000","last_became_active":"2024-07-26T12:44:36.856980+0000","last_became_peered":"2024-07-26T12:44:36.856980+0000","last_unstale":"2024-07-26T12:46:34.674886+0000","last_undegraded":"2024-07-26T12:46:34.674886+0000","last_fullsized":"2024-07-26T12:46:34.674886+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.166383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.652345+0000","last_change":"2024-07-26T12:43:12.166827+0000","last_active":"2024-07-26T12:46:34.652345+0000","last_peered":"2024-07-26T12:46:34.652345+0000","last_clean":"2024-07-26T12:46:34.652345+0000","last_became_active":"2024-07-26T12:43:12.165222+0000","last_became_peered":"2024-07-26T12:43:12.165222+0000","last_unstale":"2024-07-26T12:46:34.652345+0000","last_undegraded":"2024-07-26T12:46:34.652345+0000","last_fullsized":"2024-07-26T12:46:34.652345+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:37:42.219844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662861+0000","last_change":"2024-07-26T12:43:12.176961+0000","last_active":"2024-07-26T12:46:34.662861+0000","last_peered":"2024-07-26T12:46:34.662861+0000","last_clean":"2024-07-26T12:46:34.662861+0000","last_became_active":"2024-07-26T12:43:12.176550+0000","last_became_peered":"2024-07-26T12:43:12.176550+0000","last_unstale":"2024-07-26T12:46:34.662861+0000","last_undegraded":"2024-07-26T12:46:34.662861+0000","last_fullsized":"2024-07-26T12:46:34.662861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:36:19.720801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650354+0000","last_change":"2024-07-26T12:43:12.173560+0000","last_active":"2024-07-26T12:46:34.650354+0000","last_peered":"2024-07-26T12:46:34.650354+0000","last_clean":"2024-07-26T12:46:34.650354+0000","last_became_active":"2024-07-26T12:43:12.173187+0000","last_became_peered":"2024-07-26T12:43:12.173187+0000","last_unstale":"2024-07-26T12:46:34.650354+0000","last_undegraded":"2024-07-26T12:46:34.650354+0000","last_fullsized":"2024-07-26T12:46:34.650354+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:53.929210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655391+0000","last_change":"2024-07-26T12:43:12.169829+0000","last_active":"2024-07-26T12:46:34.655391+0000","last_peered":"2024-07-26T12:46:34.655391+0000","last_clean":"2024-07-26T12:46:34.655391+0000","last_became_active":"2024-07-26T12:43:12.169444+0000","last_became_peered":"2024-07-26T12:43:12.169444+0000","last_unstale":"2024-07-26T12:46:34.655391+0000","last_undegraded":"2024-07-26T12:46:34.655391+0000","last_fullsized":"2024-07-26T12:46:34.655391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:29.111483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668876+0000","last_change":"2024-07-26T12:43:12.171857+0000","last_active":"2024-07-26T12:46:34.668876+0000","last_peered":"2024-07-26T12:46:34.668876+0000","last_clean":"2024-07-26T12:46:34.668876+0000","last_became_active":"2024-07-26T12:43:12.170752+0000","last_became_peered":"2024-07-26T12:43:12.170752+0000","last_unstale":"2024-07-26T12:46:34.668876+0000","last_undegraded":"2024-07-26T12:46:34.668876+0000","last_fullsized":"2024-07-26T12:46:34.668876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:22:09.442916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656394+0000","last_change":"2024-07-26T12:43:12.178646+0000","last_active":"2024-07-26T12:46:34.656394+0000","last_peered":"2024-07-26T12:46:34.656394+0000","last_clean":"2024-07-26T12:46:34.656394+0000","last_became_active":"2024-07-26T12:43:12.178259+0000","last_became_peered":"2024-07-26T12:43:12.178259+0000","last_unstale":"2024-07-26T12:46:34.656394+0000","last_undegraded":"2024-07-26T12:46:34.656394+0000","last_fullsized":"2024-07-26T12:46:34.656394+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:30:59.417843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.652610+0000","last_change":"2024-07-26T12:43:12.164579+0000","last_active":"2024-07-26T12:46:34.652610+0000","last_peered":"2024-07-26T12:46:34.652610+0000","last_clean":"2024-07-26T12:46:34.652610+0000","last_became_active":"2024-07-26T12:43:12.164099+0000","last_became_peered":"2024-07-26T12:43:12.164099+0000","last_unstale":"2024-07-26T12:46:34.652610+0000","last_undegraded":"2024-07-26T12:46:34.652610+0000","last_fullsized":"2024-07-26T12:46:34.652610+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:51:45.293190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.652843+0000","last_change":"2024-07-26T12:43:12.180838+0000","last_active":"2024-07-26T12:46:34.652843+0000","last_peered":"2024-07-26T12:46:34.652843+0000","last_clean":"2024-07-26T12:46:34.652843+0000","last_became_active":"2024-07-26T12:43:12.180543+0000","last_became_peered":"2024-07-26T12:43:12.180543+0000","last_unstale":"2024-07-26T12:46:34.652843+0000","last_undegraded":"2024-07-26T12:46:34.652843+0000","last_fullsized":"2024-07-26T12:46:34.652843+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:27.185082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.648877+0000","last_change":"2024-07-26T12:43:12.167235+0000","last_active":"2024-07-26T12:46:34.648877+0000","last_peered":"2024-07-26T12:46:34.648877+0000","last_clean":"2024-07-26T12:46:34.648877+0000","last_became_active":"2024-07-26T12:43:12.166489+0000","last_became_peered":"2024-07-26T12:43:12.166489+0000","last_unstale":"2024-07-26T12:46:34.648877+0000","last_undegraded":"2024-07-26T12:46:34.648877+0000","last_fullsized":"2024-07-26T12:46:34.648877+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:13.868870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668677+0000","last_change":"2024-07-26T12:43:12.176717+0000","last_active":"2024-07-26T12:46:34.668677+0000","last_peered":"2024-07-26T12:46:34.668677+0000","last_clean":"2024-07-26T12:46:34.668677+0000","last_became_active":"2024-07-26T12:43:12.172699+0000","last_became_peered":"2024-07-26T12:43:12.172699+0000","last_unstale":"2024-07-26T12:46:34.668677+0000","last_undegraded":"2024-07-26T12:46:34.668677+0000","last_fullsized":"2024-07-26T12:46:34.668677+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:23:36.853999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.667623+0000","last_change":"2024-07-26T12:43:12.177093+0000","last_active":"2024-07-26T12:46:34.667623+0000","last_peered":"2024-07-26T12:46:34.667623+0000","last_clean":"2024-07-26T12:46:34.667623+0000","last_became_active":"2024-07-26T12:43:12.176693+0000","last_became_peered":"2024-07-26T12:43:12.176693+0000","last_unstale":"2024-07-26T12:46:34.667623+0000","last_undegraded":"2024-07-26T12:46:34.667623+0000","last_fullsized":"2024-07-26T12:46:34.667623+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:56.875143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659723+0000","last_change":"2024-07-26T12:43:12.189531+0000","last_active":"2024-07-26T12:46:34.659723+0000","last_peered":"2024-07-26T12:46:34.659723+0000","last_clean":"2024-07-26T12:46:34.659723+0000","last_became_active":"2024-07-26T12:43:12.189162+0000","last_became_peered":"2024-07-26T12:43:12.189162+0000","last_unstale":"2024-07-26T12:46:34.659723+0000","last_undegraded":"2024-07-26T12:46:34.659723+0000","last_fullsized":"2024-07-26T12:46:34.659723+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:11:00.160614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71871,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1315,"num_read_kb":741,"num_write":260,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":215613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678075,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":536466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874305,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1744896,"data_stored":1187067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037828,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":539892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234059,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":536174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430288,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36964,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708188,"statfs":{"total":95995035648,"available":95957184512,"internally_reserved":0,"allocated":1667072,"data_stored":1132812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593814,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":531300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790044,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1128394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953569,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":535374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":531008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378731,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1130496,"data_stored":597931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542258,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1118208,"data_stored":596715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":533288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":67353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T12:46:50.141 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T12:46:50.141 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-07-26T12:46:50.141 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-07-26T12:46:50.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph health --format=json 2024-07-26T12:46:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:49 smithi112 ceph-mon[37501]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-07-26T12:46:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:49 smithi132 ceph-mon[37580]: pgmap v415: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2024-07-26T12:46:50.601 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:50 smithi107 ceph-mon[32659]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:46:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:50 smithi112 ceph-mon[37501]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:46:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:50 smithi132 ceph-mon[37580]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:46:52.023 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:51 smithi107 ceph-mon[32659]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T12:46:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:51 smithi112 ceph-mon[37501]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T12:46:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:51 smithi132 ceph-mon[37580]: pgmap v416: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T12:46:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.878+0000 7f0d7dc00640 1 -- 172.21.15.107:0/388963542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d78111700 msgr2=0x7f0d78111b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.878+0000 7f0d7dc00640 1 --2- 172.21.15.107:0/388963542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d78111700 0x7f0d78111b10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0d60009900 tx=0x7f0d6002a410 comp rx=0 tx=0).stop 2024-07-26T12:46:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.878+0000 7f0d7dc00640 1 -- 172.21.15.107:0/388963542 shutdown_connections 2024-07-26T12:46:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.878+0000 7f0d7dc00640 1 --2- 172.21.15.107:0/388963542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d78112050 0x7f0d78118910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.878+0000 7f0d7dc00640 1 --2- 172.21.15.107:0/388963542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d78111700 0x7f0d78111b10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.878+0000 7f0d7dc00640 1 --2- 172.21.15.107:0/388963542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d7810a250 0x7f0d7810a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.878+0000 7f0d7dc00640 1 -- 172.21.15.107:0/388963542 >> 172.21.15.107:0/388963542 conn(0x7f0d78074110 msgr2=0x7f0d78074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.878+0000 7f0d7dc00640 1 -- 172.21.15.107:0/388963542 shutdown_connections 2024-07-26T12:46:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.878+0000 7f0d7dc00640 1 -- 172.21.15.107:0/388963542 wait complete. 2024-07-26T12:46:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.880+0000 7f0d7dc00640 1 Processor -- start 2024-07-26T12:46:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.880+0000 7f0d7dc00640 1 -- start start 2024-07-26T12:46:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.880+0000 7f0d7dc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d7810a250 0x7f0d781a2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.880+0000 7f0d7dc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d78111700 0x7f0d781a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d7dc00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d78112050 0x7f0d781a7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d7dc00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7811bd50 con 0x7f0d78112050 2024-07-26T12:46:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d7dc00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7811bbd0 con 0x7f0d7810a250 2024-07-26T12:46:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d7dc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7811bed0 con 0x7f0d78111700 2024-07-26T12:46:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d76a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d7810a250 0x7f0d781a2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d76a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d7810a250 0x7f0d781a2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:50362/0 (socket says 172.21.15.107:50362) 2024-07-26T12:46:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d76a00640 1 -- 172.21.15.107:0/2561142723 learned_addr learned my addr 172.21.15.107:0/2561142723 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d76000640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d78111700 0x7f0d781a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d77400640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d78112050 0x7f0d781a7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d76a00640 1 -- 172.21.15.107:0/2561142723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d78111700 msgr2=0x7f0d781a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d76a00640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d78111700 0x7f0d781a2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d76a00640 1 -- 172.21.15.107:0/2561142723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d78112050 msgr2=0x7f0d781a7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d76a00640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d78112050 0x7f0d781a7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.881+0000 7f0d76a00640 1 -- 172.21.15.107:0/2561142723 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d781a8690 con 0x7f0d7810a250 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.882+0000 7f0d76a00640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d7810a250 0x7f0d781a2720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800e9f0 tx=0x7f0d6800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.882+0000 7f0d6f400640 1 -- 172.21.15.107:0/2561142723 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6800cd80 con 0x7f0d7810a250 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.882+0000 7f0d7dc00640 1 -- 172.21.15.107:0/2561142723 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d781a8980 con 0x7f0d7810a250 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.882+0000 7f0d6f400640 1 -- 172.21.15.107:0/2561142723 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d680108e0 con 0x7f0d7810a250 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.882+0000 7f0d6f400640 1 -- 172.21.15.107:0/2561142723 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68018980 con 0x7f0d7810a250 2024-07-26T12:46:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.882+0000 7f0d7dc00640 1 -- 172.21.15.107:0/2561142723 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d781af0e0 con 0x7f0d7810a250 2024-07-26T12:46:52.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.883+0000 7f0d7dc00640 1 -- 172.21.15.107:0/2561142723 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d7810a660 con 0x7f0d7810a250 2024-07-26T12:46:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.885+0000 7f0d6f400640 1 -- 172.21.15.107:0/2561142723 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f0d68010430 con 0x7f0d7810a250 2024-07-26T12:46:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.887+0000 7f0d6f400640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0d5c0b3c20 0x7f0d5c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.887+0000 7f0d6f400640 1 -- 172.21.15.107:0/2561142723 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0d680e4510 con 0x7f0d7810a250 2024-07-26T12:46:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.887+0000 7f0d76000640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0d5c0b3c20 0x7f0d5c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.888+0000 7f0d76000640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0d5c0b3c20 0x7f0d5c0b6130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0d6002aa00 tx=0x7f0d60005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:52.891+0000 7f0d6f400640 1 -- 172.21.15.107:0/2561142723 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0d680ab660 con 0x7f0d7810a250 2024-07-26T12:46:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:52 smithi107 ceph-mon[32659]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:53.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.131+0000 7f0d7dc00640 1 -- 172.21.15.107:0/2561142723 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f0d7802e0c0 con 0x7f0d7810a250 2024-07-26T12:46:53.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.132+0000 7f0d6f400640 1 -- 172.21.15.107:0/2561142723 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f0d680aaf60 con 0x7f0d7810a250 2024-07-26T12:46:53.133 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:46:53.133 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-07-26T12:46:53.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.135+0000 7f0d7dc00640 1 -- 172.21.15.107:0/2561142723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0d5c0b3c20 msgr2=0x7f0d5c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:53.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.135+0000 7f0d7dc00640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0d5c0b3c20 0x7f0d5c0b6130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0d6002aa00 tx=0x7f0d60005a20 comp rx=0 tx=0).stop 2024-07-26T12:46:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.135+0000 7f0d7dc00640 1 -- 172.21.15.107:0/2561142723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d7810a250 msgr2=0x7f0d781a2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.135+0000 7f0d7dc00640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d7810a250 0x7f0d781a2720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0d6800e9f0 tx=0x7f0d6800eeb0 comp rx=0 tx=0).stop 2024-07-26T12:46:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.135+0000 7f0d7dc00640 1 -- 172.21.15.107:0/2561142723 shutdown_connections 2024-07-26T12:46:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.135+0000 7f0d7dc00640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f0d5c0b3c20 0x7f0d5c0b6130 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.135+0000 7f0d7dc00640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0d78112050 0x7f0d781a7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.135+0000 7f0d7dc00640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0d78111700 0x7f0d781a2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.135+0000 7f0d7dc00640 1 --2- 172.21.15.107:0/2561142723 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f0d7810a250 0x7f0d781a2720 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.135+0000 7f0d7dc00640 1 -- 172.21.15.107:0/2561142723 >> 172.21.15.107:0/2561142723 conn(0x7f0d78074110 msgr2=0x7f0d7810f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.136+0000 7f0d7dc00640 1 -- 172.21.15.107:0/2561142723 shutdown_connections 2024-07-26T12:46:53.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:53.136+0000 7f0d7dc00640 1 -- 172.21.15.107:0/2561142723 wait complete. 2024-07-26T12:46:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:52 smithi112 ceph-mon[37501]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:52 smithi132 ceph-mon[37580]: pgmap v417: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:46:54.060 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-07-26T12:46:54.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:46:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:53 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2561142723' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T12:46:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:53 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2561142723' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T12:46:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:53 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2561142723' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T12:46:54.523 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:46:55.010 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:54 smithi107 ceph-mon[32659]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:46:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:54 smithi112 ceph-mon[37501]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:46:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:54 smithi132 ceph-mon[37580]: pgmap v418: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:46:56.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.819+0000 7f4bb7200640 1 -- 172.21.15.107:0/2290616654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4bb0104b40 msgr2=0x7f4bb010d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:56.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.819+0000 7f4bb7200640 1 --2- 172.21.15.107:0/2290616654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4bb0104b40 0x7f4bb010d090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0009900 tx=0x7f4ba002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:46:56.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.819+0000 7f4bb7200640 1 -- 172.21.15.107:0/2290616654 shutdown_connections 2024-07-26T12:46:56.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.819+0000 7f4bb7200640 1 --2- 172.21.15.107:0/2290616654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4bb010d650 0x7f4bb010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:56.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.819+0000 7f4bb7200640 1 --2- 172.21.15.107:0/2290616654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4bb0104b40 0x7f4bb010d090 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:56.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.819+0000 7f4bb7200640 1 --2- 172.21.15.107:0/2290616654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb01041f0 0x7f4bb0104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:56.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.819+0000 7f4bb7200640 1 -- 172.21.15.107:0/2290616654 >> 172.21.15.107:0/2290616654 conn(0x7f4bb00fdcf0 msgr2=0x7f4bb0100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:56.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.819+0000 7f4bb7200640 1 -- 172.21.15.107:0/2290616654 shutdown_connections 2024-07-26T12:46:56.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.819+0000 7f4bb7200640 1 -- 172.21.15.107:0/2290616654 wait complete. 2024-07-26T12:46:56.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.820+0000 7f4bb7200640 1 Processor -- start 2024-07-26T12:46:56.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.820+0000 7f4bb7200640 1 -- start start 2024-07-26T12:46:56.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb7200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4bb01041f0 0x7f4bb0101f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:56.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb7200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4bb0104b40 0x7f4bb0102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:56.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb7200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb010d650 0x7f4bb01029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:56.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb7200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb01132d0 con 0x7f4bb010d650 2024-07-26T12:46:56.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb7200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb0113150 con 0x7f4bb0104b40 2024-07-26T12:46:56.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb7200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb0113450 con 0x7f4bb01041f0 2024-07-26T12:46:56.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb5e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4bb01041f0 0x7f4bb0101f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:56.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb6800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb010d650 0x7f4bb01029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:56.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb5e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4bb01041f0 0x7f4bb0101f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:37328/0 (socket says 172.21.15.107:37328) 2024-07-26T12:46:56.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb5e00640 1 -- 172.21.15.107:0/3348615076 learned_addr learned my addr 172.21.15.107:0/3348615076 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:46:56.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb6800640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb010d650 0x7f4bb01029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:45188/0 (socket says 172.21.15.107:45188) 2024-07-26T12:46:56.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.821+0000 7f4bb5400640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4bb0104b40 0x7f4bb0102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:56.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.822+0000 7f4bb5e00640 1 -- 172.21.15.107:0/3348615076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4bb0104b40 msgr2=0x7f4bb0102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:56.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.822+0000 7f4bb5e00640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4bb0104b40 0x7f4bb0102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:56.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.822+0000 7f4bb5e00640 1 -- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb010d650 msgr2=0x7f4bb01029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:56.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.822+0000 7f4bb5e00640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb010d650 0x7f4bb01029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:56.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.822+0000 7f4bb5e00640 1 -- 172.21.15.107:0/3348615076 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb01a8020 con 0x7f4bb01041f0 2024-07-26T12:46:56.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.822+0000 7f4bb6800640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb010d650 0x7f4bb01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:56.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.822+0000 7f4bb5400640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4bb0104b40 0x7f4bb0102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:46:56.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.822+0000 7f4bb5e00640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4bb01041f0 0x7f4bb0101f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400e9f0 tx=0x7f4ba400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:56.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.823+0000 7f4baaa00640 1 -- 172.21.15.107:0/3348615076 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba400cd80 con 0x7f4bb01041f0 2024-07-26T12:46:56.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.823+0000 7f4baaa00640 1 -- 172.21.15.107:0/3348615076 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ba40108e0 con 0x7f4bb01041f0 2024-07-26T12:46:56.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.823+0000 7f4baaa00640 1 -- 172.21.15.107:0/3348615076 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4018980 con 0x7f4bb01041f0 2024-07-26T12:46:56.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.823+0000 7f4bb7200640 1 -- 172.21.15.107:0/3348615076 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb01a8310 con 0x7f4bb01041f0 2024-07-26T12:46:56.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.823+0000 7f4bb7200640 1 -- 172.21.15.107:0/3348615076 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb01a87d0 con 0x7f4bb01041f0 2024-07-26T12:46:56.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.824+0000 7f4bb7200640 1 -- 172.21.15.107:0/3348615076 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb0104600 con 0x7f4bb01041f0 2024-07-26T12:46:56.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.826+0000 7f4baaa00640 1 -- 172.21.15.107:0/3348615076 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f4ba4010430 con 0x7f4bb01041f0 2024-07-26T12:46:56.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.828+0000 7f4baaa00640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b980b3c20 0x7f4b980b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:46:56.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.828+0000 7f4baaa00640 1 -- 172.21.15.107:0/3348615076 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4ba40e4460 con 0x7f4bb01041f0 2024-07-26T12:46:56.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.828+0000 7f4bb5400640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b980b3c20 0x7f4b980b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:46:56.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.829+0000 7f4bb5400640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b980b3c20 0x7f4b980b6130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4ba002a9e0 tx=0x7f4ba0005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:46:56.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:56.832+0000 7f4baaa00640 1 -- 172.21.15.107:0/3348615076 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4ba40aaf60 con 0x7f4bb01041f0 2024-07-26T12:46:57.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.076+0000 7f4bb7200640 1 -- 172.21.15.107:0/3348615076 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bb01a8bd0 con 0x7f4bb01041f0 2024-07-26T12:46:57.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.077+0000 7f4baaa00640 1 -- 172.21.15.107:0/3348615076 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11993 (secure 0 0 0) 0x7f4ba40af4b0 con 0x7f4bb01041f0 2024-07-26T12:46:57.078 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:46:57.079 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-07-26T12:46:57.079 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"btime":"2024-07-26T12:44:44:796279+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:44:43.812461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24716,"mds_1":24715,"mds_2":24703,"mds_3":24694,"mds_4":24682},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24632":{"gid":24632,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24634":{"gid":24634,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24652":{"gid":24652,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24662":{"gid":24662,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":32,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24694":{"gid":24694,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":31,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":29,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":28,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24716,"qdb_cluster":[24682,24694,24703,24716,24715]},"id":1}]} 2024-07-26T12:46:57.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.079+0000 7f4bb7200640 1 -- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b980b3c20 msgr2=0x7f4b980b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:57.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.079+0000 7f4bb7200640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b980b3c20 0x7f4b980b6130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4ba002a9e0 tx=0x7f4ba0005a20 comp rx=0 tx=0).stop 2024-07-26T12:46:57.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.079+0000 7f4bb7200640 1 -- 172.21.15.107:0/3348615076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4bb01041f0 msgr2=0x7f4bb0101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:46:57.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.079+0000 7f4bb7200640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4bb01041f0 0x7f4bb0101f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400e9f0 tx=0x7f4ba400eeb0 comp rx=0 tx=0).stop 2024-07-26T12:46:57.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.080+0000 7f4bb7200640 1 -- 172.21.15.107:0/3348615076 shutdown_connections 2024-07-26T12:46:57.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.080+0000 7f4bb7200640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b980b3c20 0x7f4b980b6130 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:57.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.080+0000 7f4bb7200640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bb010d650 0x7f4bb01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:57.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.080+0000 7f4bb7200640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4bb0104b40 0x7f4bb0102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:57.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.080+0000 7f4bb7200640 1 --2- 172.21.15.107:0/3348615076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4bb01041f0 0x7f4bb0101f30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:46:57.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.080+0000 7f4bb7200640 1 -- 172.21.15.107:0/3348615076 >> 172.21.15.107:0/3348615076 conn(0x7f4bb00fdcf0 msgr2=0x7f4bb0106060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:46:57.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.080+0000 7f4bb7200640 1 -- 172.21.15.107:0/3348615076 shutdown_connections 2024-07-26T12:46:57.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:46:57.080+0000 7f4bb7200640 1 -- 172.21.15.107:0/3348615076 wait complete. 2024-07-26T12:46:57.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:46:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:57 smithi107 ceph-mon[32659]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:57 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3348615076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:46:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:57 smithi112 ceph-mon[37501]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:57 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3348615076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:46:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:57 smithi132 ceph-mon[37580]: pgmap v419: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:57 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3348615076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:46:58.311 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:46:59 smithi107 ceph-mon[32659]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:00.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:46:59 smithi132 ceph-mon[37580]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:46:59 smithi112 ceph-mon[37501]: pgmap v420: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:00.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.648+0000 7eff5f200640 1 -- 172.21.15.107:0/2618255707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff60077370 msgr2=0x7eff60077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:00.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.648+0000 7eff5f200640 1 --2- 172.21.15.107:0/2618255707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff60077370 0x7eff60077780 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7eff50009900 tx=0x7eff5002a3e0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.649+0000 7eff5f200640 1 -- 172.21.15.107:0/2618255707 shutdown_connections 2024-07-26T12:47:00.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.649+0000 7eff5f200640 1 --2- 172.21.15.107:0/2618255707 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff60114030 0x7eff60116450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.649+0000 7eff5f200640 1 --2- 172.21.15.107:0/2618255707 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff60077cc0 0x7eff60113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.649+0000 7eff5f200640 1 --2- 172.21.15.107:0/2618255707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff60077370 0x7eff60077780 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.649+0000 7eff5f200640 1 -- 172.21.15.107:0/2618255707 >> 172.21.15.107:0/2618255707 conn(0x7eff601010c0 msgr2=0x7eff60103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:00.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.649+0000 7eff5f200640 1 -- 172.21.15.107:0/2618255707 shutdown_connections 2024-07-26T12:47:00.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.649+0000 7eff5f200640 1 -- 172.21.15.107:0/2618255707 wait complete. 2024-07-26T12:47:00.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.650+0000 7eff5f200640 1 Processor -- start 2024-07-26T12:47:00.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff5f200640 1 -- start start 2024-07-26T12:47:00.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff5f200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff60077370 0x7eff601a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:00.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff5f200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff60077cc0 0x7eff601a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:00.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff5f200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff60114030 0x7eff601a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:00.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff5f200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7eff60119950 con 0x7eff60077cc0 2024-07-26T12:47:00.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff5f200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7eff601197d0 con 0x7eff60077370 2024-07-26T12:47:00.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff5f200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7eff60119ad0 con 0x7eff60114030 2024-07-26T12:47:00.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff55400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff60077cc0 0x7eff601a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:00.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff5de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff60077370 0x7eff601a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:00.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff5de00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff60077370 0x7eff601a0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:59454/0 (socket says 172.21.15.107:59454) 2024-07-26T12:47:00.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.651+0000 7eff5de00640 1 -- 172.21.15.107:0/3886738279 learned_addr learned my addr 172.21.15.107:0/3886738279 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:00.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.652+0000 7eff5e800640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff60114030 0x7eff601a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:00.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.652+0000 7eff55400640 1 -- 172.21.15.107:0/3886738279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff60114030 msgr2=0x7eff601a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:00.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.652+0000 7eff55400640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff60114030 0x7eff601a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.652+0000 7eff55400640 1 -- 172.21.15.107:0/3886738279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff60077370 msgr2=0x7eff601a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:00.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.652+0000 7eff55400640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff60077370 0x7eff601a0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.652+0000 7eff55400640 1 -- 172.21.15.107:0/3886738279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff601a62b0 con 0x7eff60077cc0 2024-07-26T12:47:00.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.652+0000 7eff55400640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff60077cc0 0x7eff601a08e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7eff6006b3c0 tx=0x7eff4800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:00.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.653+0000 7eff57400640 1 -- 172.21.15.107:0/3886738279 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4800cc40 con 0x7eff60077cc0 2024-07-26T12:47:00.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.653+0000 7eff57400640 1 -- 172.21.15.107:0/3886738279 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff4800cde0 con 0x7eff60077cc0 2024-07-26T12:47:00.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.653+0000 7eff57400640 1 -- 172.21.15.107:0/3886738279 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48004a00 con 0x7eff60077cc0 2024-07-26T12:47:00.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.653+0000 7eff5f200640 1 -- 172.21.15.107:0/3886738279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff601a65a0 con 0x7eff60077cc0 2024-07-26T12:47:00.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.653+0000 7eff5f200640 1 -- 172.21.15.107:0/3886738279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff601acd00 con 0x7eff60077cc0 2024-07-26T12:47:00.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.654+0000 7eff5f200640 1 -- 172.21.15.107:0/3886738279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff60077780 con 0x7eff60077cc0 2024-07-26T12:47:00.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.657+0000 7eff57400640 1 -- 172.21.15.107:0/3886738279 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7eff48004ba0 con 0x7eff60077cc0 2024-07-26T12:47:00.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.658+0000 7eff57400640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7eff440b3c20 0x7eff440b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:00.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.658+0000 7eff57400640 1 -- 172.21.15.107:0/3886738279 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7eff480e4180 con 0x7eff60077cc0 2024-07-26T12:47:00.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.658+0000 7eff5de00640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7eff440b3c20 0x7eff440b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:00.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.659+0000 7eff5de00640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7eff440b3c20 0x7eff440b6130 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7eff50031440 tx=0x7eff500058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:00.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.661+0000 7eff57400640 1 -- 172.21.15.107:0/3886738279 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7eff480ab360 con 0x7eff60077cc0 2024-07-26T12:47:00.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.895+0000 7eff5f200640 1 -- 172.21.15.107:0/3886738279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff601a1470 con 0x7eff60077cc0 2024-07-26T12:47:00.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.896+0000 7eff57400640 1 -- 172.21.15.107:0/3886738279 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11993 (secure 0 0 0) 0x7eff480aac60 con 0x7eff60077cc0 2024-07-26T12:47:00.899 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:47:00.899 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"btime":"2024-07-26T12:44:44:796279+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:44:43.812461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24716,"mds_1":24715,"mds_2":24703,"mds_3":24694,"mds_4":24682},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24632":{"gid":24632,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24634":{"gid":24634,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24652":{"gid":24652,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24662":{"gid":24662,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":32,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24694":{"gid":24694,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":31,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":29,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":28,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24716,"qdb_cluster":[24682,24694,24703,24716,24715]},"id":1}]} 2024-07-26T12:47:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-07-26T12:47:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.899+0000 7eff5f200640 1 -- 172.21.15.107:0/3886738279 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7eff440b3c20 msgr2=0x7eff440b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.899+0000 7eff5f200640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7eff440b3c20 0x7eff440b6130 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7eff50031440 tx=0x7eff500058d0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.899+0000 7eff5f200640 1 -- 172.21.15.107:0/3886738279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff60077cc0 msgr2=0x7eff601a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:00.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.899+0000 7eff5f200640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff60077cc0 0x7eff601a08e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7eff6006b3c0 tx=0x7eff4800ee50 comp rx=0 tx=0).stop 2024-07-26T12:47:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.900+0000 7eff5f200640 1 -- 172.21.15.107:0/3886738279 shutdown_connections 2024-07-26T12:47:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.900+0000 7eff5f200640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7eff440b3c20 0x7eff440b6130 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.900+0000 7eff5f200640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff60114030 0x7eff601a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.900+0000 7eff5f200640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff60077cc0 0x7eff601a08e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.900+0000 7eff5f200640 1 --2- 172.21.15.107:0/3886738279 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7eff60077370 0x7eff601a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.900+0000 7eff5f200640 1 -- 172.21.15.107:0/3886738279 >> 172.21.15.107:0/3886738279 conn(0x7eff601010c0 msgr2=0x7eff601160f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.900+0000 7eff5f200640 1 -- 172.21.15.107:0/3886738279 shutdown_connections 2024-07-26T12:47:00.902 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:00.900+0000 7eff5f200640 1 -- 172.21.15.107:0/3886738279 wait complete. 2024-07-26T12:47:01.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:47:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:01 smithi107 ceph-mon[32659]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:01 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3886738279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:47:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:01 smithi112 ceph-mon[37501]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:01 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3886738279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:47:02.226 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:01 smithi132 ceph-mon[37580]: pgmap v421: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:01 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3886738279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:47:04.103 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:03 smithi107 ceph-mon[32659]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:03 smithi112 ceph-mon[37501]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:03 smithi132 ceph-mon[37580]: pgmap v422: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:04.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.440+0000 7ff1cc800640 1 -- 172.21.15.107:0/807581835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c410ba70 msgr2=0x7ff1c41122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:04.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.440+0000 7ff1cc800640 1 --2- 172.21.15.107:0/807581835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c410ba70 0x7ff1c41122a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0009ca0 tx=0x7ff1c002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:47:04.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.441+0000 7ff1cc800640 1 -- 172.21.15.107:0/807581835 shutdown_connections 2024-07-26T12:47:04.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.441+0000 7ff1cc800640 1 --2- 172.21.15.107:0/807581835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c410ba70 0x7ff1c41122a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:04.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.441+0000 7ff1cc800640 1 --2- 172.21.15.107:0/807581835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1c410b120 0x7ff1c410b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:04.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.441+0000 7ff1cc800640 1 --2- 172.21.15.107:0/807581835 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1c41057e0 0x7ff1c4105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:04.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.441+0000 7ff1cc800640 1 -- 172.21.15.107:0/807581835 >> 172.21.15.107:0/807581835 conn(0x7ff1c41011f0 msgr2=0x7ff1c4103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:04.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.441+0000 7ff1cc800640 1 -- 172.21.15.107:0/807581835 shutdown_connections 2024-07-26T12:47:04.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.441+0000 7ff1cc800640 1 -- 172.21.15.107:0/807581835 wait complete. 2024-07-26T12:47:04.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.442+0000 7ff1cc800640 1 Processor -- start 2024-07-26T12:47:04.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.442+0000 7ff1cc800640 1 -- start start 2024-07-26T12:47:04.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.442+0000 7ff1cc800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c41057e0 0x7ff1c419c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:04.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.443+0000 7ff1cc800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1c410b120 0x7ff1c419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.443+0000 7ff1cc800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1c410ba70 0x7ff1c41a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.443+0000 7ff1cc800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c4115720 con 0x7ff1c41057e0 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.443+0000 7ff1cc800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c41155a0 con 0x7ff1c410b120 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.443+0000 7ff1cc800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c41158a0 con 0x7ff1c410ba70 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.443+0000 7ff1c9e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c41057e0 0x7ff1c419c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.443+0000 7ff1c9e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c41057e0 0x7ff1c419c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35948/0 (socket says 172.21.15.107:35948) 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.443+0000 7ff1c9400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1c410b120 0x7ff1c419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.443+0000 7ff1c9e00640 1 -- 172.21.15.107:0/325699825 learned_addr learned my addr 172.21.15.107:0/325699825 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.443+0000 7ff1ca800640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1c410ba70 0x7ff1c41a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1c9400640 1 -- 172.21.15.107:0/325699825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1c410ba70 msgr2=0x7ff1c41a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:04.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1c9400640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1c410ba70 0x7ff1c41a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:04.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1c9400640 1 -- 172.21.15.107:0/325699825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c41057e0 msgr2=0x7ff1c419c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:04.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1c9400640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c41057e0 0x7ff1c419c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:04.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1c9400640 1 -- 172.21.15.107:0/325699825 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c41a20a0 con 0x7ff1c410b120 2024-07-26T12:47:04.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1c9400640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1c410b120 0x7ff1c419c6d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff1c406c3a0 tx=0x7ff1b400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:04.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1bea00640 1 -- 172.21.15.107:0/325699825 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b400cd00 con 0x7ff1c410b120 2024-07-26T12:47:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1bea00640 1 -- 172.21.15.107:0/325699825 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1b40035f0 con 0x7ff1c410b120 2024-07-26T12:47:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1cc800640 1 -- 172.21.15.107:0/325699825 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1c41a2390 con 0x7ff1c410b120 2024-07-26T12:47:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1bea00640 1 -- 172.21.15.107:0/325699825 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4010690 con 0x7ff1c410b120 2024-07-26T12:47:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.444+0000 7ff1cc800640 1 -- 172.21.15.107:0/325699825 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1c41a8af0 con 0x7ff1c410b120 2024-07-26T12:47:04.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.446+0000 7ff1cc800640 1 -- 172.21.15.107:0/325699825 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1c4105bf0 con 0x7ff1c410b120 2024-07-26T12:47:04.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.448+0000 7ff1bea00640 1 -- 172.21.15.107:0/325699825 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7ff1b40037a0 con 0x7ff1c410b120 2024-07-26T12:47:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.450+0000 7ff1bea00640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1ac0b3ca0 0x7ff1ac0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.450+0000 7ff1bea00640 1 -- 172.21.15.107:0/325699825 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff1b40e4480 con 0x7ff1c410b120 2024-07-26T12:47:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.450+0000 7ff1c9e00640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1ac0b3ca0 0x7ff1ac0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:04.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.451+0000 7ff1c9e00640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1ac0b3ca0 0x7ff1ac0b61b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8009c90 tx=0x7ff1b8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:04.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.451+0000 7ff1bea00640 1 -- 172.21.15.107:0/325699825 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff1b4055600 con 0x7ff1c410b120 2024-07-26T12:47:04.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.654+0000 7ff1cc800640 1 -- 172.21.15.107:0/325699825 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff1c410d8c0 con 0x7ff1c410b120 2024-07-26T12:47:04.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.656+0000 7ff1bea00640 1 -- 172.21.15.107:0/325699825 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25203 (secure 0 0 0) 0x7ff1b40ab5d0 con 0x7ff1c410b120 2024-07-26T12:47:04.659 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:47:04.659 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:44:36.797948+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:47:04.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 -- 172.21.15.107:0/325699825 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1ac0b3ca0 msgr2=0x7ff1ac0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:04.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1ac0b3ca0 0x7ff1ac0b61b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8009c90 tx=0x7ff1b8005f90 comp rx=0 tx=0).stop 2024-07-26T12:47:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 -- 172.21.15.107:0/325699825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1c410b120 msgr2=0x7ff1c419c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1c410b120 0x7ff1c419c6d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff1c406c3a0 tx=0x7ff1b400ee50 comp rx=0 tx=0).stop 2024-07-26T12:47:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 -- 172.21.15.107:0/325699825 shutdown_connections 2024-07-26T12:47:04.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1ac0b3ca0 0x7ff1ac0b61b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1c410ba70 0x7ff1c41a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1c410b120 0x7ff1c419c6d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 --2- 172.21.15.107:0/325699825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1c41057e0 0x7ff1c419c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 -- 172.21.15.107:0/325699825 >> 172.21.15.107:0/325699825 conn(0x7ff1c41011f0 msgr2=0x7ff1c4109680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 -- 172.21.15.107:0/325699825 shutdown_connections 2024-07-26T12:47:04.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:04.660+0000 7ff1cc800640 1 -- 172.21.15.107:0/325699825 wait complete. 2024-07-26T12:47:05.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:04 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/325699825' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:47:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:04 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/325699825' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:47:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:04 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/325699825' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:47:05.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:47:06.026 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:06.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:05 smithi107 ceph-mon[32659]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:05 smithi112 ceph-mon[37501]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:05 smithi132 ceph-mon[37580]: pgmap v423: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:08.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:07 smithi107 ceph-mon[32659]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:07 smithi112 ceph-mon[37501]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:07 smithi132 ceph-mon[37580]: pgmap v424: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:08.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.443+0000 7f317ae00640 1 -- 172.21.15.107:0/2460793703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317410d650 msgr2=0x7f317410fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.443+0000 7f317ae00640 1 --2- 172.21.15.107:0/2460793703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317410d650 0x7f317410fb40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3164009ca0 tx=0x7f316402a8c0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.443+0000 7f317ae00640 1 -- 172.21.15.107:0/2460793703 shutdown_connections 2024-07-26T12:47:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.443+0000 7f317ae00640 1 --2- 172.21.15.107:0/2460793703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317410d650 0x7f317410fb40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.443+0000 7f317ae00640 1 --2- 172.21.15.107:0/2460793703 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3174104ad0 0x7f317410d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.443+0000 7f317ae00640 1 --2- 172.21.15.107:0/2460793703 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31741040a0 0x7f31741044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.443+0000 7f317ae00640 1 -- 172.21.15.107:0/2460793703 >> 172.21.15.107:0/2460793703 conn(0x7f31740fdcf0 msgr2=0x7f3174100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.443+0000 7f317ae00640 1 -- 172.21.15.107:0/2460793703 shutdown_connections 2024-07-26T12:47:08.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.443+0000 7f317ae00640 1 -- 172.21.15.107:0/2460793703 wait complete. 2024-07-26T12:47:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.445+0000 7f317ae00640 1 Processor -- start 2024-07-26T12:47:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.445+0000 7f317ae00640 1 -- start start 2024-07-26T12:47:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f317ae00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31741040a0 0x7f3174199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f317ae00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3174104ad0 0x7f317419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f317ae00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f317410d650 0x7f317419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f317ae00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f31741132b0 con 0x7f3174104ad0 2024-07-26T12:47:08.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f317ae00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3174113130 con 0x7f31741040a0 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f317ae00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3174113430 con 0x7f317410d650 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f3179000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3174104ad0 0x7f317419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f3179a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31741040a0 0x7f3174199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f3179a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31741040a0 0x7f3174199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:36676/0 (socket says 172.21.15.107:36676) 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f3179a00640 1 -- 172.21.15.107:0/3219433100 learned_addr learned my addr 172.21.15.107:0/3219433100 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f317a400640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f317410d650 0x7f317419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f3179a00640 1 -- 172.21.15.107:0/3219433100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f317410d650 msgr2=0x7f317419f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f3179a00640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f317410d650 0x7f317419f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f3179a00640 1 -- 172.21.15.107:0/3219433100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3174104ad0 msgr2=0x7f317419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f3179a00640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3174104ad0 0x7f317419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f3179a00640 1 -- 172.21.15.107:0/3219433100 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317419fbd0 con 0x7f31741040a0 2024-07-26T12:47:08.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.446+0000 7f317a400640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f317410d650 0x7f317419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:47:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.447+0000 7f3179a00640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31741040a0 0x7f3174199c60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f316c00e9f0 tx=0x7f316c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.447+0000 7f3172000640 1 -- 172.21.15.107:0/3219433100 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316c00cd80 con 0x7f31741040a0 2024-07-26T12:47:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.447+0000 7f3172000640 1 -- 172.21.15.107:0/3219433100 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f316c0054d0 con 0x7f31741040a0 2024-07-26T12:47:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.447+0000 7f317ae00640 1 -- 172.21.15.107:0/3219433100 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f317419fec0 con 0x7f31741040a0 2024-07-26T12:47:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.447+0000 7f3172000640 1 -- 172.21.15.107:0/3219433100 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316c010690 con 0x7f31741040a0 2024-07-26T12:47:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.447+0000 7f317ae00640 1 -- 172.21.15.107:0/3219433100 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31741a6620 con 0x7f31741040a0 2024-07-26T12:47:08.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.448+0000 7f317ae00640 1 -- 172.21.15.107:0/3219433100 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31741044b0 con 0x7f31741040a0 2024-07-26T12:47:08.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.451+0000 7f3172000640 1 -- 172.21.15.107:0/3219433100 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f316c004e30 con 0x7f31741040a0 2024-07-26T12:47:08.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.453+0000 7f3172000640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31580b3ca0 0x7f31580b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:08.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.453+0000 7f3172000640 1 -- 172.21.15.107:0/3219433100 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f316c0e4790 con 0x7f31741040a0 2024-07-26T12:47:08.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.453+0000 7f3179000640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31580b3ca0 0x7f31580b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:08.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.454+0000 7f3179000640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31580b3ca0 0x7f31580b61b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f31600095f0 tx=0x7f3160005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:08.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.454+0000 7f3172000640 1 -- 172.21.15.107:0/3219433100 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f316c055600 con 0x7f31741040a0 2024-07-26T12:47:08.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.688+0000 7f317ae00640 1 -- 172.21.15.107:0/3219433100 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f317419ad90 con 0x7f31741040a0 2024-07-26T12:47:08.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.689+0000 7f3172000640 1 -- 172.21.15.107:0/3219433100 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11993 (secure 0 0 0) 0x7f316c0ab8e0 con 0x7f31741040a0 2024-07-26T12:47:08.691 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:47:08.691 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"btime":"2024-07-26T12:44:44:796279+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:44:43.812461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24716,"mds_1":24715,"mds_2":24703,"mds_3":24694,"mds_4":24682},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24632":{"gid":24632,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24634":{"gid":24634,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24652":{"gid":24652,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24662":{"gid":24662,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":32,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24694":{"gid":24694,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":31,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":29,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":28,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24716,"qdb_cluster":[24682,24694,24703,24716,24715]},"id":1}]} 2024-07-26T12:47:08.692 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-07-26T12:47:08.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.693+0000 7f317ae00640 1 -- 172.21.15.107:0/3219433100 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31580b3ca0 msgr2=0x7f31580b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:08.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.693+0000 7f317ae00640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31580b3ca0 0x7f31580b61b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f31600095f0 tx=0x7f3160005f90 comp rx=0 tx=0).stop 2024-07-26T12:47:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.693+0000 7f317ae00640 1 -- 172.21.15.107:0/3219433100 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31741040a0 msgr2=0x7f3174199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.693+0000 7f317ae00640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31741040a0 0x7f3174199c60 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f316c00e9f0 tx=0x7f316c00edf0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.694+0000 7f317ae00640 1 -- 172.21.15.107:0/3219433100 shutdown_connections 2024-07-26T12:47:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.694+0000 7f317ae00640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31580b3ca0 0x7f31580b61b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.694+0000 7f317ae00640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f317410d650 0x7f317419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.694+0000 7f317ae00640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3174104ad0 0x7f317419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.694+0000 7f317ae00640 1 --2- 172.21.15.107:0/3219433100 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31741040a0 0x7f3174199c60 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:08.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.694+0000 7f317ae00640 1 -- 172.21.15.107:0/3219433100 >> 172.21.15.107:0/3219433100 conn(0x7f31740fdcf0 msgr2=0x7f31741008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:08.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.694+0000 7f317ae00640 1 -- 172.21.15.107:0/3219433100 shutdown_connections 2024-07-26T12:47:08.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:08.694+0000 7f317ae00640 1 -- 172.21.15.107:0/3219433100 wait complete. 2024-07-26T12:47:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:08 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3219433100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:47:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:08 smithi107 ceph-mon[32659]: pgmap v425: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:08 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3219433100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:47:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:08 smithi112 ceph-mon[37501]: pgmap v425: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:08 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3219433100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:47:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:08 smithi132 ceph-mon[37580]: pgmap v425: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:09.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T12:43:15.198289+0000', 'modified': '2024-07-26T12:44:43.812461+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24716, 'mds_1': 24715, 'mds_2': 24703, 'mds_3': 24694, 'mds_4': 24682}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24632': {'gid': 24632, 'name': 'k', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6833/2931369116', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6832', 'nonce': 2931369116}, {'type': 'v1', 'addr': '172.21.15.112:6833', 'nonce': 2931369116}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24634': {'gid': 24634, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.132:6833/277912155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6832', 'nonce': 277912155}, {'type': 'v1', 'addr': '172.21.15.132:6833', 'nonce': 277912155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24652': {'gid': 24652, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6835/3403405722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 3403405722}, {'type': 'v1', 'addr': '172.21.15.107:6835', 'nonce': 3403405722}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24662': {'gid': 24662, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6835/1038904649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6834', 'nonce': 1038904649}, {'type': 'v1', 'addr': '172.21.15.112:6835', 'nonce': 1038904649}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24680': {'gid': 24680, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6837/3739854241', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6836', 'nonce': 3739854241}, {'type': 'v1', 'addr': '172.21.15.112:6837', 'nonce': 3739854241}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24682': {'gid': 24682, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.132:6835/1242618859', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6834', 'nonce': 1242618859}, {'type': 'v1', 'addr': '172.21.15.132:6835', 'nonce': 1242618859}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24694': {'gid': 24694, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.132:6837/735918310', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6836', 'nonce': 735918310}, {'type': 'v1', 'addr': '172.21.15.132:6837', 'nonce': 735918310}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24703': {'gid': 24703, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 29, 'addr': '172.21.15.132:6839/3489644052', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 3489644052}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 3489644052}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24715': {'gid': 24715, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.107:6839/137526780', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 137526780}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 137526780}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24716': {'gid': 24716, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.112:6839/3779803434', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 3779803434}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 3779803434}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24716, 'qdb_cluster': [24682, 24694, 24703, 24716, 24715]} 2024-07-26T12:47:09.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T12:47:09.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.f status 2024-07-26T12:47:10.144 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:12.154 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:11 smithi107 ceph-mon[32659]: pgmap v426: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:11 smithi112 ceph-mon[37501]: pgmap v426: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:11 smithi132 ceph-mon[37580]: pgmap v426: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.538+0000 7fe237a00640 1 -- 172.21.15.107:0/978624260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2301035b0 msgr2=0x7fe2301039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.538+0000 7fe237a00640 1 --2- 172.21.15.107:0/978624260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2301035b0 0x7fe2301039c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe224009900 tx=0x7fe22402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.538+0000 7fe237a00640 1 -- 172.21.15.107:0/978624260 shutdown_connections 2024-07-26T12:47:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.538+0000 7fe237a00640 1 --2- 172.21.15.107:0/978624260 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe230109840 0x7fe230110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.538+0000 7fe237a00640 1 --2- 172.21.15.107:0/978624260 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe230108ef0 0x7fe230109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.538+0000 7fe237a00640 1 --2- 172.21.15.107:0/978624260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2301035b0 0x7fe2301039c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.538+0000 7fe237a00640 1 -- 172.21.15.107:0/978624260 >> 172.21.15.107:0/978624260 conn(0x7fe2300fefc0 msgr2=0x7fe230101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.539+0000 7fe237a00640 1 -- 172.21.15.107:0/978624260 shutdown_connections 2024-07-26T12:47:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.539+0000 7fe237a00640 1 -- 172.21.15.107:0/978624260 wait complete. 2024-07-26T12:47:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.540+0000 7fe237a00640 1 Processor -- start 2024-07-26T12:47:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.540+0000 7fe237a00640 1 -- start start 2024-07-26T12:47:12.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.540+0000 7fe237a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2301035b0 0x7fe230075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.540+0000 7fe237a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe230108ef0 0x7fe230075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.540+0000 7fe237a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe230109840 0x7fe230075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.540+0000 7fe237a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe230113560 con 0x7fe230108ef0 2024-07-26T12:47:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.540+0000 7fe237a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301133e0 con 0x7fe2301035b0 2024-07-26T12:47:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.540+0000 7fe237a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe2301136e0 con 0x7fe230109840 2024-07-26T12:47:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.541+0000 7fe235000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2301035b0 0x7fe230075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.541+0000 7fe22fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe230108ef0 0x7fe230075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.541+0000 7fe235a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe230109840 0x7fe230075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.541+0000 7fe235000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2301035b0 0x7fe230075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:36700/0 (socket says 172.21.15.107:36700) 2024-07-26T12:47:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.541+0000 7fe235000640 1 -- 172.21.15.107:0/1565779183 learned_addr learned my addr 172.21.15.107:0/1565779183 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.541+0000 7fe235a00640 1 -- 172.21.15.107:0/1565779183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2301035b0 msgr2=0x7fe230075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.541+0000 7fe235a00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2301035b0 0x7fe230075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.541+0000 7fe235a00640 1 -- 172.21.15.107:0/1565779183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe230108ef0 msgr2=0x7fe230075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.541+0000 7fe235a00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe230108ef0 0x7fe230075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.541+0000 7fe235a00640 1 -- 172.21.15.107:0/1565779183 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2301a8270 con 0x7fe230109840 2024-07-26T12:47:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.542+0000 7fe235a00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe230109840 0x7fe230075bc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe228009fd0 tx=0x7fe22800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:12.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.542+0000 7fe22d600640 1 -- 172.21.15.107:0/1565779183 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe22800ce20 con 0x7fe230109840 2024-07-26T12:47:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.542+0000 7fe237a00640 1 -- 172.21.15.107:0/1565779183 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2301a8560 con 0x7fe230109840 2024-07-26T12:47:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.542+0000 7fe22d600640 1 -- 172.21.15.107:0/1565779183 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe228005020 con 0x7fe230109840 2024-07-26T12:47:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.542+0000 7fe22d600640 1 -- 172.21.15.107:0/1565779183 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe228010690 con 0x7fe230109840 2024-07-26T12:47:12.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.542+0000 7fe237a00640 1 -- 172.21.15.107:0/1565779183 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2301a8af0 con 0x7fe230109840 2024-07-26T12:47:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.546+0000 7fe22d600640 1 -- 172.21.15.107:0/1565779183 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fe2280054d0 con 0x7fe230109840 2024-07-26T12:47:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.547+0000 7fe23e5d8740 1 Processor -- start 2024-07-26T12:47:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.547+0000 7fe23e5d8740 1 -- start start 2024-07-26T12:47:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe22d600640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe2180b3ca0 0x7fe2180b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 0x55fcff069550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff069a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 0x55fcff069fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fcfefdf950 con 0x55fcfeca3280 2024-07-26T12:47:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fcfefcee50 con 0x55fcfecc3f60 2024-07-26T12:47:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fcfefd2e90 con 0x55fcfec9b0a0 2024-07-26T12:47:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe22d600640 1 -- 172.21.15.107:0/1565779183 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe2280e4700 con 0x7fe230109840 2024-07-26T12:47:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe235000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 0x55fcff069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe22fe00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe2180b3ca0 0x7fe2180b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe235000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 0x55fcff069550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:36710/0 (socket says 172.21.15.107:36710) 2024-07-26T12:47:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe235000640 1 -- 172.21.15.107:0/3174626153 learned_addr learned my addr 172.21.15.107:0/3174626153 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe22fe00640 1 --2- 172.21.15.107:0/3174626153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff069a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.548+0000 7fe235a00640 1 --2- 172.21.15.107:0/3174626153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 0x55fcff069fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.549+0000 7fe235000640 1 -- 172.21.15.107:0/3174626153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 msgr2=0x55fcff069a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.549+0000 7fe235000640 1 --2- 172.21.15.107:0/3174626153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff069a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.549+0000 7fe235000640 1 -- 172.21.15.107:0/3174626153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 msgr2=0x55fcff069fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.549+0000 7fe235000640 1 --2- 172.21.15.107:0/3174626153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 0x55fcff069fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.549+0000 7fe235000640 1 -- 172.21.15.107:0/3174626153 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcfef760f0 con 0x55fcfecc3f60 2024-07-26T12:47:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.549+0000 7fe22fe00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe2180b3ca0 0x7fe2180b61b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe2301017c0 tx=0x7fe21c0098b0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.549+0000 7fe235000640 1 --2- 172.21.15.107:0/3174626153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 0x55fcff069550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe22402a9e0 tx=0x7fe224004100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4040b889b14f4cc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.550+0000 7fe222000640 1 -- 172.21.15.107:0/3174626153 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe224040070 con 0x55fcfecc3f60 2024-07-26T12:47:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.550+0000 7fe222000640 1 -- 172.21.15.107:0/3174626153 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2240054d0 con 0x55fcfecc3f60 2024-07-26T12:47:12.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.550+0000 7fe222000640 1 -- 172.21.15.107:0/3174626153 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe224035040 con 0x55fcfecc3f60 2024-07-26T12:47:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.551+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3174626153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 msgr2=0x55fcff069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.551+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/3174626153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 0x55fcff069550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe22402a9e0 tx=0x7fe224004100 comp rx=0 tx=0).stop 2024-07-26T12:47:12.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.551+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3174626153 shutdown_connections 2024-07-26T12:47:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.551+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/3174626153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 0x55fcff069fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.551+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/3174626153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff069a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.551+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/3174626153 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 0x55fcff069550 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.551+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3174626153 >> 172.21.15.107:0/3174626153 conn(0x55fcfecb8350 msgr2=0x55fcff067440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.551+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3174626153 shutdown_connections 2024-07-26T12:47:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.551+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3174626153 wait complete. 2024-07-26T12:47:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.552+0000 7fe23e5d8740 1 -- start start 2024-07-26T12:47:12.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.552+0000 7fe23e5d8740 1 Processor -- start 2024-07-26T12:47:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.552+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfec9b0a0 0x55fcff074d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.552+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 0x55fcff075350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.552+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfecc3f60 0x55fcff080ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.552+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fcfee325f0 con 0x55fcfec9b0a0 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.552+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fcfef6b520 con 0x55fcfeca3280 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.552+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fcff077cd0 con 0x55fcfecc3f60 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe23e5d8740 10 client.0 _ll_register_callbacks cb 0x55fcfef7aef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe235000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfec9b0a0 0x55fcff074d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe22fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 0x55fcff075350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe235000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfec9b0a0 0x55fcff074d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:35998/0 (socket says 172.21.15.107:35998) 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe235000640 1 -- 172.21.15.107:0/3549171821 learned_addr learned my addr 172.21.15.107:0/3549171821 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe235a00640 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfecc3f60 0x55fcff080ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe22fe00640 1 -- 172.21.15.107:0/3549171821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfecc3f60 msgr2=0x55fcff080ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe22fe00640 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfecc3f60 0x55fcff080ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe22fe00640 1 -- 172.21.15.107:0/3549171821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfec9b0a0 msgr2=0x55fcff074d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:12.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe22fe00640 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfec9b0a0 0x55fcff074d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe22fe00640 1 -- 172.21.15.107:0/3549171821 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcfed2f540 con 0x55fcfeca3280 2024-07-26T12:47:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.553+0000 7fe235000640 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfec9b0a0 0x55fcff074d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:47:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.554+0000 7fe22fe00640 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 0x55fcff075350 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe23006a230 tx=0x7fe21c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8885b31a614eacbe server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:12.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.554+0000 7fe216a00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:47:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.554+0000 7fe216a00640 1 -- 172.21.15.107:0/3549171821 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c00fe20 con 0x55fcfeca3280 2024-07-26T12:47:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.554+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3549171821 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2240054d0 con 0x55fcfeca3280 2024-07-26T12:47:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.554+0000 7fe216a00640 1 -- 172.21.15.107:0/3549171821 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe21c010880 con 0x55fcfeca3280 2024-07-26T12:47:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.554+0000 7fe216a00640 1 -- 172.21.15.107:0/3549171821 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe21c010a80 con 0x55fcfeca3280 2024-07-26T12:47:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.554+0000 7fe23e5d8740 10 client.25163 fetch_fsmap learned FSMap version 28 2024-07-26T12:47:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.555+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3549171821 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fcff083bf0 con 0x55fcfeca3280 2024-07-26T12:47:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.555+0000 7fe216a00640 1 -- 172.21.15.107:0/3549171821 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2240054d0 con 0x55fcfeca3280 2024-07-26T12:47:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.555+0000 7fe216a00640 1 -- 172.21.15.107:0/3549171821 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fe21c020050 con 0x55fcfeca3280 2024-07-26T12:47:12.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.555+0000 7fe23e5d8740 10 client.25163 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:47:12.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.555+0000 7fe23e5d8740 10 client.25163 resolve_mds: resolved name 'f' to daemon mds.f 2024-07-26T12:47:12.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.555+0000 7fe23e5d8740 20 client.25163 populate_metadata read hostname 'smithi107' 2024-07-26T12:47:12.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.555+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55fcfec828e0 0x55fcff087670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.555+0000 7fe23e5d8740 4 client.25163 mds_command: new command op to 24682 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:47:12.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.555+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3549171821 --> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcfefe18a0 con 0x55fcfec828e0 2024-07-26T12:47:12.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.556+0000 7fe235000640 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55fcfec828e0 0x55fcff087670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.557+0000 7fe216a00640 10 client.25163 ms_handle_connect on v2:172.21.15.132:6834/1242618859 2024-07-26T12:47:12.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.557+0000 7fe235000640 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55fcfec828e0 0x55fcff087670 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c24ebaee746d2a98 server_cookie=fdba848e49c31317 in_seq=0 out_seq=0 2024-07-26T12:47:12.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe216a00640 1 -- 172.21.15.107:0/3549171821 <== mds.4 v2:172.21.15.132:6834/1242618859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fcfefe18a0 con 0x55fcfec828e0 2024-07-26T12:47:12.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe216a00640 10 client.25163 handle_command_reply: tid=0 2024-07-26T12:47:12.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 client.25163 shutdown 2024-07-26T12:47:12.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3549171821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 msgr2=0x55fcff075350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:12.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 0x55fcff075350 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe23006a230 tx=0x7fe21c004730 comp rx=0 tx=0).stop 2024-07-26T12:47:12.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3549171821 shutdown_connections 2024-07-26T12:47:12.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55fcfec828e0 0x55fcff087670 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfecc3f60 0x55fcff080ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 0x55fcff075350 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/3549171821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfec9b0a0 0x55fcff074d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3549171821 >> 172.21.15.107:0/3549171821 conn(0x55fcfecb8350 msgr2=0x55fcff06c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:12.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe216a00640 0 client.25163 ms_handle_reset on v2:172.21.15.132:6834/1242618859 2024-07-26T12:47:12.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3549171821 shutdown_connections 2024-07-26T12:47:12.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.558+0000 7fe23e5d8740 1 -- 172.21.15.107:0/3549171821 wait complete. 2024-07-26T12:47:12.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.559+0000 7fe23e5d8740 20 client.25163 trim_cache size 0 max 16384 2024-07-26T12:47:12.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.578+0000 7fe23e5d8740 1 Processor -- start 2024-07-26T12:47:12.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.578+0000 7fe23e5d8740 1 -- start start 2024-07-26T12:47:12.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff086200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 0x55fcff06f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 0x55fcff080f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fcff0859b0 con 0x55fcfeca3280 2024-07-26T12:47:12.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fcff086080 con 0x55fcfecc3f60 2024-07-26T12:47:12.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fcff086750 con 0x55fcfec9b0a0 2024-07-26T12:47:12.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe22fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 0x55fcff06f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe235000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff086200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe22fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 0x55fcff06f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36006/0 (socket says 172.21.15.107:36006) 2024-07-26T12:47:12.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe22fe00640 1 -- 172.21.15.107:0/2239826617 learned_addr learned my addr 172.21.15.107:0/2239826617 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:12.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe235000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff086200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:49584/0 (socket says 172.21.15.107:49584) 2024-07-26T12:47:12.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.579+0000 7fe235a00640 1 --2- 172.21.15.107:0/2239826617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 0x55fcff080f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.580+0000 7fe235000640 1 -- 172.21.15.107:0/2239826617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 msgr2=0x55fcff080f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.580+0000 7fe235000640 1 --2- 172.21.15.107:0/2239826617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 0x55fcff080f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.580+0000 7fe235000640 1 -- 172.21.15.107:0/2239826617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 msgr2=0x55fcff06f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.580+0000 7fe235000640 1 --2- 172.21.15.107:0/2239826617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 0x55fcff06f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.580+0000 7fe235000640 1 -- 172.21.15.107:0/2239826617 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcff076050 con 0x55fcfec9b0a0 2024-07-26T12:47:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.580+0000 7fe235000640 1 --2- 172.21.15.107:0/2239826617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff086200 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe2240048c0 tx=0x7fe224009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8f29c841c3cdd4c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.581+0000 7fe221600640 1 -- 172.21.15.107:0/2239826617 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe224040040 con 0x55fcfec9b0a0 2024-07-26T12:47:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.581+0000 7fe221600640 1 -- 172.21.15.107:0/2239826617 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe224004370 con 0x55fcfec9b0a0 2024-07-26T12:47:12.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.581+0000 7fe221600640 1 -- 172.21.15.107:0/2239826617 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe224035040 con 0x55fcfec9b0a0 2024-07-26T12:47:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.582+0000 7fe23e5d8740 1 -- 172.21.15.107:0/2239826617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 msgr2=0x55fcff086200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.582+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/2239826617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff086200 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe2240048c0 tx=0x7fe224009260 comp rx=0 tx=0).stop 2024-07-26T12:47:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.582+0000 7fe23e5d8740 1 -- 172.21.15.107:0/2239826617 shutdown_connections 2024-07-26T12:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.582+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/2239826617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfecc3f60 0x55fcff080f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.582+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/2239826617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfeca3280 0x55fcff06f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.582+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/2239826617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff086200 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.582+0000 7fe23e5d8740 1 -- 172.21.15.107:0/2239826617 >> 172.21.15.107:0/2239826617 conn(0x55fcfecb8350 msgr2=0x55fcff074d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.582+0000 7fe23e5d8740 1 -- 172.21.15.107:0/2239826617 shutdown_connections 2024-07-26T12:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.582+0000 7fe23e5d8740 1 -- 172.21.15.107:0/2239826617 wait complete. 2024-07-26T12:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.582+0000 7fe23e5d8740 1 -- start start 2024-07-26T12:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.583+0000 7fe23e5d8740 1 Processor -- start 2024-07-26T12:47:12.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.583+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff088a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.583+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 0x55fcff089010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.583+0000 7fe23e5d8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfecc3f60 0x55fcff08f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.583+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fcff0874f0 con 0x55fcfecc3f60 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.583+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fcff086e20 con 0x55fcfeca3280 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.583+0000 7fe23e5d8740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fcff087bc0 con 0x55fcfec9b0a0 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.583+0000 7fe23e5d8740 10 client.0 _ll_register_callbacks cb 0x7fe21c009c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.584+0000 7fe235000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff088a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.584+0000 7fe235000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff088a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:49592/0 (socket says 172.21.15.107:49592) 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.584+0000 7fe235000640 1 -- 172.21.15.107:0/1142530939 learned_addr learned my addr 172.21.15.107:0/1142530939 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.584+0000 7fe22fe00640 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 0x55fcff089010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.584+0000 7fe235a00640 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfecc3f60 0x55fcff08f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.584+0000 7fe235000640 1 -- 172.21.15.107:0/1142530939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 msgr2=0x55fcff089010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:12.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.584+0000 7fe235000640 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 0x55fcff089010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.584+0000 7fe235000640 1 -- 172.21.15.107:0/1142530939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfecc3f60 msgr2=0x55fcff08f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:12.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.584+0000 7fe235000640 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfecc3f60 0x55fcff08f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.584+0000 7fe235000640 1 -- 172.21.15.107:0/1142530939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcff08ff10 con 0x55fcfec9b0a0 2024-07-26T12:47:12.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.585+0000 7fe235000640 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff088a70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe224034040 tx=0x7fe224005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4283ca3faa983fae server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:12.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.585+0000 7fe216a00640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:47:12.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.585+0000 7fe216a00640 1 -- 172.21.15.107:0/1142530939 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe224035040 con 0x55fcfec9b0a0 2024-07-26T12:47:12.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.585+0000 7fe23e5d8740 1 -- 172.21.15.107:0/1142530939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe224004370 con 0x55fcfec9b0a0 2024-07-26T12:47:12.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.585+0000 7fe216a00640 1 -- 172.21.15.107:0/1142530939 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe224004be0 con 0x55fcfec9b0a0 2024-07-26T12:47:12.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.585+0000 7fe216a00640 1 -- 172.21.15.107:0/1142530939 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe224040040 con 0x55fcfec9b0a0 2024-07-26T12:47:12.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.585+0000 7fe23e5d8740 10 client.25180 fetch_fsmap learned FSMap version 28 2024-07-26T12:47:12.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.585+0000 7fe23e5d8740 1 -- 172.21.15.107:0/1142530939 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fcff0f7b10 con 0x55fcfec9b0a0 2024-07-26T12:47:12.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.586+0000 7fe216a00640 1 -- 172.21.15.107:0/1142530939 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe224004370 con 0x55fcfec9b0a0 2024-07-26T12:47:12.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.586+0000 7fe216a00640 1 -- 172.21.15.107:0/1142530939 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fe2240509f0 con 0x55fcfec9b0a0 2024-07-26T12:47:12.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.586+0000 7fe23e5d8740 10 client.25180 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:47:12.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.586+0000 7fe23e5d8740 10 client.25180 resolve_mds: resolved name 'f' to daemon mds.f 2024-07-26T12:47:12.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.586+0000 7fe23e5d8740 20 client.25180 populate_metadata read hostname 'smithi107' 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.586+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55fcfec57c90 0x55fcff0fab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.586+0000 7fe23e5d8740 4 client.25180 mds_command: new command op to 24682 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.586+0000 7fe23e5d8740 1 -- 172.21.15.107:0/1142530939 --> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] -- command(tid 0: {"prefix": "status"}) -- 0x55fcfefdd340 con 0x55fcfec57c90 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.586+0000 7fe22fe00640 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55fcfec57c90 0x55fcff0fab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.587+0000 7fe216a00640 10 client.25180 ms_handle_connect on v2:172.21.15.132:6834/1242618859 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.587+0000 7fe22fe00640 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55fcfec57c90 0x55fcff0fab30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=48ab721b3cf41df1 server_cookie=69b1cc2b88e0fbbc in_seq=0 out_seq=0 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.587+0000 7fe216a00640 1 -- 172.21.15.107:0/1142530939 <== mds.4 v2:172.21.15.132:6834/1242618859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55fcfefdd340 con 0x55fcfec57c90 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.587+0000 7fe216a00640 10 client.25180 handle_command_reply: tid=0 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.587+0000 7fe23e5d8740 1 client.25180 shutdown 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 1 -- 172.21.15.107:0/1142530939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 msgr2=0x55fcff088a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff088a70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe224034040 tx=0x7fe224005fd0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 1 -- 172.21.15.107:0/1142530939 shutdown_connections 2024-07-26T12:47:12.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55fcfec57c90 0x55fcff0fab30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcfecc3f60 0x55fcff08f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcfeca3280 0x55fcff089010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 1 --2- 172.21.15.107:0/1142530939 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcfec9b0a0 0x55fcff088a70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe216a00640 0 client.25180 ms_handle_reset on v2:172.21.15.132:6834/1242618859 2024-07-26T12:47:12.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 1 -- 172.21.15.107:0/1142530939 >> 172.21.15.107:0/1142530939 conn(0x55fcfecb8350 msgr2=0x55fcff066c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:12.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 1 -- 172.21.15.107:0/1142530939 shutdown_connections 2024-07-26T12:47:12.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 1 -- 172.21.15.107:0/1142530939 wait complete. 2024-07-26T12:47:12.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.588+0000 7fe23e5d8740 20 client.25180 trim_cache size 0 max 16384 2024-07-26T12:47:12.592 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 4, 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24682, 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 153.72589549899999, 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 90, 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 275.581937891 2024-07-26T12:47:12.593 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:47:12.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 -- 172.21.15.107:0/1565779183 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe2180b3ca0 msgr2=0x7fe2180b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:12.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe2180b3ca0 0x7fe2180b61b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe2301017c0 tx=0x7fe21c0098b0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 -- 172.21.15.107:0/1565779183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe230109840 msgr2=0x7fe230075bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:12.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe230109840 0x7fe230075bc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe228009fd0 tx=0x7fe22800ef10 comp rx=0 tx=0).stop 2024-07-26T12:47:12.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 -- 172.21.15.107:0/1565779183 shutdown_connections 2024-07-26T12:47:12.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe2180b3ca0 0x7fe2180b61b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe230109840 0x7fe230075bc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe230108ef0 0x7fe230075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 --2- 172.21.15.107:0/1565779183 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe2301035b0 0x7fe230075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:12.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 -- 172.21.15.107:0/1565779183 >> 172.21.15.107:0/1565779183 conn(0x7fe2300fefc0 msgr2=0x7fe230102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:12.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 -- 172.21.15.107:0/1565779183 shutdown_connections 2024-07-26T12:47:12.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:12.589+0000 7fe237a00640 1 -- 172.21.15.107:0/1565779183 wait complete. 2024-07-26T12:47:13.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.c status 2024-07-26T12:47:14.078 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:13 smithi112 ceph-mon[37501]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:14.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:13 smithi107 ceph-mon[32659]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:13 smithi132 ceph-mon[37580]: pgmap v427: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:16.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:15 smithi107 ceph-mon[32659]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:15 smithi112 ceph-mon[37501]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:15 smithi132 ceph-mon[37580]: pgmap v428: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:16.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.391+0000 7f88b7e00640 1 -- 172.21.15.107:0/849910396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88b810b860 msgr2=0x7f88b8112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:16.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.391+0000 7f88b7e00640 1 --2- 172.21.15.107:0/849910396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88b810b860 0x7f88b8112090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f88b0009ca0 tx=0x7f88b002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.391+0000 7f88b7e00640 1 -- 172.21.15.107:0/849910396 shutdown_connections 2024-07-26T12:47:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.391+0000 7f88b7e00640 1 --2- 172.21.15.107:0/849910396 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88b810b860 0x7f88b8112090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.391+0000 7f88b7e00640 1 --2- 172.21.15.107:0/849910396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88b810af10 0x7f88b810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.391+0000 7f88b7e00640 1 --2- 172.21.15.107:0/849910396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88b81055d0 0x7f88b81059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.391+0000 7f88b7e00640 1 -- 172.21.15.107:0/849910396 >> 172.21.15.107:0/849910396 conn(0x7f88b8101040 msgr2=0x7f88b8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:16.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.392+0000 7f88b7e00640 1 -- 172.21.15.107:0/849910396 shutdown_connections 2024-07-26T12:47:16.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.392+0000 7f88b7e00640 1 -- 172.21.15.107:0/849910396 wait complete. 2024-07-26T12:47:16.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.393+0000 7f88b7e00640 1 Processor -- start 2024-07-26T12:47:16.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.393+0000 7f88b7e00640 1 -- start start 2024-07-26T12:47:16.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b7e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88b81055d0 0x7f88b819bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b7e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88b810af10 0x7f88b819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88b810b860 0x7f88b81a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b7e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8115580 con 0x7f88b810af10 2024-07-26T12:47:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b7e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8115400 con 0x7f88b81055d0 2024-07-26T12:47:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b7e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8115700 con 0x7f88b810b860 2024-07-26T12:47:16.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88ae000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88b810af10 0x7f88b819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b6a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88b81055d0 0x7f88b819bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b6a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88b81055d0 0x7f88b819bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:35578/0 (socket says 172.21.15.107:35578) 2024-07-26T12:47:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b6a00640 1 -- 172.21.15.107:0/3833133585 learned_addr learned my addr 172.21.15.107:0/3833133585 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88ae000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88b810af10 0x7f88b819c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47410/0 (socket says 172.21.15.107:47410) 2024-07-26T12:47:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b7400640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88b810b860 0x7f88b81a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b6a00640 1 -- 172.21.15.107:0/3833133585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88b810b860 msgr2=0x7f88b81a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b6a00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88b810b860 0x7f88b81a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b6a00640 1 -- 172.21.15.107:0/3833133585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88b810af10 msgr2=0x7f88b819c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b6a00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88b810af10 0x7f88b819c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b6a00640 1 -- 172.21.15.107:0/3833133585 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b81a1e40 con 0x7f88b81055d0 2024-07-26T12:47:16.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.394+0000 7f88b7400640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88b810b860 0x7f88b81a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:47:16.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.395+0000 7f88b6a00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88b81055d0 0x7f88b819bed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f88a800e9f0 tx=0x7f88a800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:16.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.395+0000 7f88bca00640 1 -- 172.21.15.107:0/3833133585 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a800cd80 con 0x7f88b81055d0 2024-07-26T12:47:16.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.395+0000 7f88bca00640 1 -- 172.21.15.107:0/3833133585 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a80108e0 con 0x7f88b81055d0 2024-07-26T12:47:16.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.395+0000 7f88b7e00640 1 -- 172.21.15.107:0/3833133585 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b81a2130 con 0x7f88b81055d0 2024-07-26T12:47:16.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.395+0000 7f88bca00640 1 -- 172.21.15.107:0/3833133585 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8018980 con 0x7f88b81055d0 2024-07-26T12:47:16.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.395+0000 7f88b7e00640 1 -- 172.21.15.107:0/3833133585 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b81a8890 con 0x7f88b81055d0 2024-07-26T12:47:16.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.399+0000 7f88c4a77740 1 Processor -- start 2024-07-26T12:47:16.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.399+0000 7f88c4a77740 1 -- start start 2024-07-26T12:47:16.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.399+0000 7f88bca00640 1 -- 172.21.15.107:0/3833133585 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f88a8010430 con 0x7f88b81055d0 2024-07-26T12:47:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.399+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f63414760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.399+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 0x559f63414ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.399+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 0x559f634151e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.399+0000 7f88c4a77740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559f6338a950 con 0x559f6306ef60 2024-07-26T12:47:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.399+0000 7f88c4a77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559f63379e50 con 0x559f630460a0 2024-07-26T12:47:16.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.399+0000 7f88c4a77740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559f6337de90 con 0x559f6304e280 2024-07-26T12:47:16.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88b7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f63414760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88b6a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 0x559f63414ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88b6a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 0x559f63414ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:35580/0 (socket says 172.21.15.107:35580) 2024-07-26T12:47:16.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88b7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f63414760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47424/0 (socket says 172.21.15.107:47424) 2024-07-26T12:47:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88b6a00640 1 -- 172.21.15.107:0/2141983646 learned_addr learned my addr 172.21.15.107:0/2141983646 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88ae000640 1 --2- 172.21.15.107:0/2141983646 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 0x559f634151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88b7400640 1 -- 172.21.15.107:0/2141983646 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 msgr2=0x559f634151e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88b7400640 1 --2- 172.21.15.107:0/2141983646 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 0x559f634151e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88b7400640 1 -- 172.21.15.107:0/2141983646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 msgr2=0x559f63414ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:16.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88b7400640 1 --2- 172.21.15.107:0/2141983646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 0x559f63414ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.400+0000 7f88b7400640 1 -- 172.21.15.107:0/2141983646 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f633210f0 con 0x559f6306ef60 2024-07-26T12:47:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.401+0000 7f88b7400640 1 --2- 172.21.15.107:0/2141983646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f63414760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f88b002ade0 tx=0x7f88b0009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933c71a4fda2ab44 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.401+0000 7f88a7e00640 1 -- 172.21.15.107:0/2141983646 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b0031cf0 con 0x559f6306ef60 2024-07-26T12:47:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.401+0000 7f88a7e00640 1 -- 172.21.15.107:0/2141983646 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88b0036040 con 0x559f6306ef60 2024-07-26T12:47:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.401+0000 7f88a7e00640 1 -- 172.21.15.107:0/2141983646 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b0004240 con 0x559f6306ef60 2024-07-26T12:47:16.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.401+0000 7f88bca00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f889c0b3ca0 0x7f889c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.401+0000 7f88bca00640 1 -- 172.21.15.107:0/3833133585 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f88a80e44a0 con 0x7f88b81055d0 2024-07-26T12:47:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.401+0000 7f88b6a00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f889c0b3ca0 0x7f889c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.402+0000 7f88c4a77740 1 -- 172.21.15.107:0/2141983646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 msgr2=0x559f63414760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.402+0000 7f88c4a77740 1 --2- 172.21.15.107:0/2141983646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f63414760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f88b002ade0 tx=0x7f88b0009870 comp rx=0 tx=0).stop 2024-07-26T12:47:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.402+0000 7f88c4a77740 1 -- 172.21.15.107:0/2141983646 shutdown_connections 2024-07-26T12:47:16.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.402+0000 7f88c4a77740 1 --2- 172.21.15.107:0/2141983646 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 0x559f634151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.402+0000 7f88c4a77740 1 --2- 172.21.15.107:0/2141983646 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 0x559f63414ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.402+0000 7f88c4a77740 1 --2- 172.21.15.107:0/2141983646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f63414760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.402+0000 7f88c4a77740 1 -- 172.21.15.107:0/2141983646 >> 172.21.15.107:0/2141983646 conn(0x559f63063350 msgr2=0x559f63412650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.402+0000 7f88c4a77740 1 -- 172.21.15.107:0/2141983646 shutdown_connections 2024-07-26T12:47:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.402+0000 7f88b6a00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f889c0b3ca0 0x7f889c0b61b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f88a80e1aa0 tx=0x7f88a80e7ee0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:16.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.402+0000 7f88c4a77740 1 -- 172.21.15.107:0/2141983646 wait complete. 2024-07-26T12:47:16.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.403+0000 7f88c4a77740 1 -- start start 2024-07-26T12:47:16.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.403+0000 7f88c4a77740 1 Processor -- start 2024-07-26T12:47:16.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 0x559f6341ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6304e280 0x559f634204d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6306ef60 0x559f6342c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88c4a77740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559f631dd5f0 con 0x559f6304e280 2024-07-26T12:47:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88c4a77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559f63316520 con 0x559f6306ef60 2024-07-26T12:47:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88c4a77740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559f63422ee0 con 0x559f630460a0 2024-07-26T12:47:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88c4a77740 10 client.0 _ll_register_callbacks cb 0x559f63325ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:47:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88ae000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6304e280 0x559f634204d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88b7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 0x559f6341ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88ae000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6304e280 0x559f634204d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47426/0 (socket says 172.21.15.107:47426) 2024-07-26T12:47:16.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88ae000640 1 -- 172.21.15.107:0/3089563389 learned_addr learned my addr 172.21.15.107:0/3089563389 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:16.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.404+0000 7f88b7400640 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6306ef60 0x559f6342c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88ae000640 1 -- 172.21.15.107:0/3089563389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 msgr2=0x559f6341ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:16.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88ae000640 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 0x559f6341ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88ae000640 1 -- 172.21.15.107:0/3089563389 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6306ef60 msgr2=0x559f6342c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:16.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88ae000640 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6306ef60 0x559f6342c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88ae000640 1 -- 172.21.15.107:0/3089563389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f630da540 con 0x559f6304e280 2024-07-26T12:47:16.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88b7400640 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 0x559f6341ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:47:16.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88ae000640 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6304e280 0x559f634204d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f88a00126e0 tx=0x7f88a0012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b4dea19638fb9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:16.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:47:16.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88a4c00640 1 -- 172.21.15.107:0/3089563389 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0012190 con 0x559f6304e280 2024-07-26T12:47:16.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88a4c00640 1 -- 172.21.15.107:0/3089563389 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f88a0004020 con 0x559f6304e280 2024-07-26T12:47:16.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.405+0000 7f88c4a77740 1 -- 172.21.15.107:0/3089563389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88b0036040 con 0x559f6304e280 2024-07-26T12:47:16.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.406+0000 7f88a4c00640 1 -- 172.21.15.107:0/3089563389 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f88a0025050 con 0x559f6304e280 2024-07-26T12:47:16.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.406+0000 7f88c4a77740 10 client.15045 fetch_fsmap learned FSMap version 28 2024-07-26T12:47:16.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.406+0000 7f88c4a77740 1 -- 172.21.15.107:0/3089563389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559f6342eda0 con 0x559f6304e280 2024-07-26T12:47:16.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.406+0000 7f88a4c00640 1 -- 172.21.15.107:0/3089563389 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88b0036040 con 0x559f6304e280 2024-07-26T12:47:16.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.406+0000 7f88a4c00640 1 -- 172.21.15.107:0/3089563389 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f88a0003680 con 0x559f6304e280 2024-07-26T12:47:16.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.406+0000 7f88c4a77740 10 client.15045 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:47:16.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.407+0000 7f88c4a77740 10 client.15045 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T12:47:16.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.407+0000 7f88c4a77740 20 client.15045 populate_metadata read hostname 'smithi107' 2024-07-26T12:47:16.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.407+0000 7f88c4a77740 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x559f6302d8e0 0x559f63432840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.407+0000 7f88c4a77740 4 client.15045 mds_command: new command op to 24694 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:47:16.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.407+0000 7f88c4a77740 1 -- 172.21.15.107:0/3089563389 --> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f6338c8a0 con 0x559f6302d8e0 2024-07-26T12:47:16.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.407+0000 7f88b7400640 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x559f6302d8e0 0x559f63432840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.408+0000 7f88b7400640 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x559f6302d8e0 0x559f63432840 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=83a5861041754334 server_cookie=e2b56c99df07ab8d in_seq=0 out_seq=0 2024-07-26T12:47:16.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.408+0000 7f88a4c00640 10 client.15045 ms_handle_connect on v2:172.21.15.132:6836/735918310 2024-07-26T12:47:16.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88a4c00640 1 -- 172.21.15.107:0/3089563389 <== mds.3 v2:172.21.15.132:6836/735918310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559f6338c8a0 con 0x559f6302d8e0 2024-07-26T12:47:16.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88a4c00640 10 client.15045 handle_command_reply: tid=0 2024-07-26T12:47:16.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88c4a77740 1 client.15045 shutdown 2024-07-26T12:47:16.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88c4a77740 1 -- 172.21.15.107:0/3089563389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6304e280 msgr2=0x559f634204d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88c4a77740 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6304e280 0x559f634204d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f88a00126e0 tx=0x7f88a0012ba0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88c4a77740 1 -- 172.21.15.107:0/3089563389 shutdown_connections 2024-07-26T12:47:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88c4a77740 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x559f6302d8e0 0x559f63432840 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88c4a77740 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6306ef60 0x559f6342c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88c4a77740 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6304e280 0x559f634204d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88c4a77740 1 --2- 172.21.15.107:0/3089563389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 0x559f6341ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88c4a77740 1 -- 172.21.15.107:0/3089563389 >> 172.21.15.107:0/3089563389 conn(0x559f63063350 msgr2=0x559f63417940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88a4c00640 0 client.15045 ms_handle_reset on v2:172.21.15.132:6836/735918310 2024-07-26T12:47:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.409+0000 7f88c4a77740 1 -- 172.21.15.107:0/3089563389 shutdown_connections 2024-07-26T12:47:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.410+0000 7f88c4a77740 1 -- 172.21.15.107:0/3089563389 wait complete. 2024-07-26T12:47:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.410+0000 7f88c4a77740 20 client.15045 trim_cache size 0 max 16384 2024-07-26T12:47:16.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.429+0000 7f88c4a77740 1 Processor -- start 2024-07-26T12:47:16.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.429+0000 7f88c4a77740 1 -- start start 2024-07-26T12:47:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.429+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 0x559f63430d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.429+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 0x559f63432170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.429+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f6342c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.429+0000 7f88c4a77740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559f63430b80 con 0x559f6306ef60 2024-07-26T12:47:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.429+0000 7f88c4a77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559f63431250 con 0x559f630460a0 2024-07-26T12:47:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.429+0000 7f88c4a77740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559f63431920 con 0x559f6304e280 2024-07-26T12:47:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.430+0000 7f88ae000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 0x559f63432170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.430+0000 7f88b7400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 0x559f63430d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.430+0000 7f88ae000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 0x559f63432170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53468/0 (socket says 172.21.15.107:53468) 2024-07-26T12:47:16.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.430+0000 7f88ae000640 1 -- 172.21.15.107:0/4171446613 learned_addr learned my addr 172.21.15.107:0/4171446613 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:16.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.430+0000 7f88b7400640 1 --2- 172.21.15.107:0/4171446613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f6342c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.430+0000 7f88ae000640 1 -- 172.21.15.107:0/4171446613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 msgr2=0x559f63430d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:16.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.430+0000 7f88ae000640 1 --2- 172.21.15.107:0/4171446613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 0x559f63430d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.430+0000 7f88ae000640 1 -- 172.21.15.107:0/4171446613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 msgr2=0x559f6342c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:16.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.430+0000 7f88ae000640 1 --2- 172.21.15.107:0/4171446613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f6342c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.430+0000 7f88ae000640 1 -- 172.21.15.107:0/4171446613 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f6340d7d0 con 0x559f6304e280 2024-07-26T12:47:16.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.431+0000 7f88ae000640 1 --2- 172.21.15.107:0/4171446613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 0x559f63432170 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f88a0001eb0 tx=0x7f88a00123a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4de794a60603407e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:16.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.431+0000 7f88a7400640 1 -- 172.21.15.107:0/4171446613 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0022040 con 0x559f6304e280 2024-07-26T12:47:16.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.431+0000 7f88a7400640 1 -- 172.21.15.107:0/4171446613 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a000fdd0 con 0x559f6304e280 2024-07-26T12:47:16.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.431+0000 7f88a7400640 1 -- 172.21.15.107:0/4171446613 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a001c470 con 0x559f6304e280 2024-07-26T12:47:16.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.432+0000 7f88c4a77740 1 -- 172.21.15.107:0/4171446613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 msgr2=0x559f63432170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:16.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.432+0000 7f88c4a77740 1 --2- 172.21.15.107:0/4171446613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 0x559f63432170 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f88a0001eb0 tx=0x7f88a00123a0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.432+0000 7f88c4a77740 1 -- 172.21.15.107:0/4171446613 shutdown_connections 2024-07-26T12:47:16.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.432+0000 7f88c4a77740 1 --2- 172.21.15.107:0/4171446613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f6342c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.432+0000 7f88c4a77740 1 --2- 172.21.15.107:0/4171446613 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f6304e280 0x559f63432170 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.432+0000 7f88c4a77740 1 --2- 172.21.15.107:0/4171446613 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f630460a0 0x559f63430d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.432+0000 7f88c4a77740 1 -- 172.21.15.107:0/4171446613 >> 172.21.15.107:0/4171446613 conn(0x559f63063350 msgr2=0x559f6341b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:16.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.432+0000 7f88c4a77740 1 -- 172.21.15.107:0/4171446613 shutdown_connections 2024-07-26T12:47:16.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.432+0000 7f88c4a77740 1 -- 172.21.15.107:0/4171446613 wait complete. 2024-07-26T12:47:16.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.433+0000 7f88c4a77740 1 -- start start 2024-07-26T12:47:16.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.433+0000 7f88c4a77740 1 Processor -- start 2024-07-26T12:47:16.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 0x559f6343add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6304e280 0x559f6343b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88c4a77740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f63439c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.449 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88c4a77740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559f634326c0 con 0x559f6306ef60 2024-07-26T12:47:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88c4a77740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559f63431ff0 con 0x559f6304e280 2024-07-26T12:47:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88c4a77740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559f63432d90 con 0x559f630460a0 2024-07-26T12:47:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88c4a77740 10 client.0 _ll_register_callbacks cb 0x7f88a0009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:47:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88ae000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6304e280 0x559f6343b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88ae000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6304e280 0x559f6343b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:35620/0 (socket says 172.21.15.107:35620) 2024-07-26T12:47:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88ae000640 1 -- 172.21.15.107:0/2175022541 learned_addr learned my addr 172.21.15.107:0/2175022541 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88b7400640 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f63439c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88b7400640 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 0x559f6343add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88ae000640 1 -- 172.21.15.107:0/2175022541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 msgr2=0x559f6343add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88ae000640 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 0x559f6343add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88ae000640 1 -- 172.21.15.107:0/2175022541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 msgr2=0x559f63439c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:16.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88ae000640 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f63439c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.434+0000 7f88ae000640 1 -- 172.21.15.107:0/2175022541 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f6343a400 con 0x559f6304e280 2024-07-26T12:47:16.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.435+0000 7f88ae000640 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6304e280 0x559f6343b390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f88a0005220 tx=0x7f88a000ff80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e8207c41b919650 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:16.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.435+0000 7f88a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:47:16.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.435+0000 7f88a4c00640 1 -- 172.21.15.107:0/2175022541 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a0003800 con 0x559f6304e280 2024-07-26T12:47:16.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.435+0000 7f88c4a77740 1 -- 172.21.15.107:0/2175022541 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88a000fdd0 con 0x559f6304e280 2024-07-26T12:47:16.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.435+0000 7f88a4c00640 1 -- 172.21.15.107:0/2175022541 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f88a002f070 con 0x559f6304e280 2024-07-26T12:47:16.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.435+0000 7f88a4c00640 1 -- 172.21.15.107:0/2175022541 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f88a003c050 con 0x559f6304e280 2024-07-26T12:47:16.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.435+0000 7f88c4a77740 10 client.25187 fetch_fsmap learned FSMap version 28 2024-07-26T12:47:16.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.435+0000 7f88c4a77740 1 -- 172.21.15.107:0/2175022541 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559f6349a610 con 0x559f6304e280 2024-07-26T12:47:16.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.436+0000 7f88a4c00640 1 -- 172.21.15.107:0/2175022541 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88a000fdd0 con 0x559f6304e280 2024-07-26T12:47:16.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.436+0000 7f88a4c00640 1 -- 172.21.15.107:0/2175022541 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f88a001c810 con 0x559f6304e280 2024-07-26T12:47:16.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.436+0000 7f88c4a77740 10 client.25187 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:47:16.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.436+0000 7f88c4a77740 10 client.25187 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T12:47:16.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.436+0000 7f88c4a77740 20 client.25187 populate_metadata read hostname 'smithi107' 2024-07-26T12:47:16.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.436+0000 7f88c4a77740 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x559f63002c90 0x559f6349db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:16.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.436+0000 7f88c4a77740 4 client.25187 mds_command: new command op to 24694 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:47:16.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.436+0000 7f88c4a77740 1 -- 172.21.15.107:0/2175022541 --> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] -- command(tid 0: {"prefix": "status"}) -- 0x559f63388340 con 0x559f63002c90 2024-07-26T12:47:16.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.436+0000 7f88b7400640 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x559f63002c90 0x559f6349db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:16.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.437+0000 7f88b7400640 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x559f63002c90 0x559f6349db00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=42402501f97c0322 server_cookie=81ba49c0d39923c1 in_seq=0 out_seq=0 2024-07-26T12:47:16.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.437+0000 7f88a4c00640 10 client.25187 ms_handle_connect on v2:172.21.15.132:6836/735918310 2024-07-26T12:47:16.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.437+0000 7f88a4c00640 1 -- 172.21.15.107:0/2175022541 <== mds.3 v2:172.21.15.132:6836/735918310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559f63388340 con 0x559f63002c90 2024-07-26T12:47:16.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.437+0000 7f88a4c00640 10 client.25187 handle_command_reply: tid=0 2024-07-26T12:47:16.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.437+0000 7f88c4a77740 1 client.25187 shutdown 2024-07-26T12:47:16.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.437+0000 7f88c4a77740 1 -- 172.21.15.107:0/2175022541 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6304e280 msgr2=0x559f6343b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:16.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.437+0000 7f88c4a77740 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6304e280 0x559f6343b390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f88a0005220 tx=0x7f88a000ff80 comp rx=0 tx=0).stop 2024-07-26T12:47:16.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.438+0000 7f88c4a77740 1 -- 172.21.15.107:0/2175022541 shutdown_connections 2024-07-26T12:47:16.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.438+0000 7f88c4a77740 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x559f63002c90 0x559f6349db00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.438+0000 7f88c4a77740 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f6306ef60 0x559f63439c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.438+0000 7f88c4a77740 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559f6304e280 0x559f6343b390 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.438+0000 7f88c4a77740 1 --2- 172.21.15.107:0/2175022541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559f630460a0 0x559f6343add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.438+0000 7f88a4c00640 0 client.25187 ms_handle_reset on v2:172.21.15.132:6836/735918310 2024-07-26T12:47:16.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.438+0000 7f88c4a77740 1 -- 172.21.15.107:0/2175022541 >> 172.21.15.107:0/2175022541 conn(0x559f63063350 msgr2=0x559f63411e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:16.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.438+0000 7f88c4a77740 1 -- 172.21.15.107:0/2175022541 shutdown_connections 2024-07-26T12:47:16.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.438+0000 7f88c4a77740 1 -- 172.21.15.107:0/2175022541 wait complete. 2024-07-26T12:47:16.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.438+0000 7f88c4a77740 20 client.25187 trim_cache size 0 max 16384 2024-07-26T12:47:16.458 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:47:16.458 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:47:16.458 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 3, 2024-07-26T12:47:16.458 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24694, 2024-07-26T12:47:16.459 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:47:16.459 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:47:16.459 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:47:16.459 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 158.584179842, 2024-07-26T12:47:16.459 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-07-26T12:47:16.459 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 90, 2024-07-26T12:47:16.459 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T12:47:16.459 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 276.74221350099998 2024-07-26T12:47:16.459 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:47:16.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 -- 172.21.15.107:0/3833133585 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f889c0b3ca0 msgr2=0x7f889c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:16.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f889c0b3ca0 0x7f889c0b61b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f88a80e1aa0 tx=0x7f88a80e7ee0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 -- 172.21.15.107:0/3833133585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88b81055d0 msgr2=0x7f88b819bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:16.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88b81055d0 0x7f88b819bed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f88a800e9f0 tx=0x7f88a800eeb0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 -- 172.21.15.107:0/3833133585 shutdown_connections 2024-07-26T12:47:16.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f889c0b3ca0 0x7f889c0b61b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f88b810b860 0x7f88b81a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f88b810af10 0x7f88b819c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 --2- 172.21.15.107:0/3833133585 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f88b81055d0 0x7f88b819bed0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:16.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 -- 172.21.15.107:0/3833133585 >> 172.21.15.107:0/3833133585 conn(0x7f88b8101040 msgr2=0x7f88b810a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:16.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 -- 172.21.15.107:0/3833133585 shutdown_connections 2024-07-26T12:47:16.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:16.439+0000 7f88b7e00640 1 -- 172.21.15.107:0/3833133585 wait complete. 2024-07-26T12:47:17.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.i status 2024-07-26T12:47:17.820 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:18.163 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:17 smithi107 ceph-mon[32659]: pgmap v429: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:17 smithi112 ceph-mon[37501]: pgmap v429: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:17 smithi132 ceph-mon[37580]: pgmap v429: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:19.969 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:19 smithi107 ceph-mon[32659]: pgmap v430: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:19 smithi112 ceph-mon[37501]: pgmap v430: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:19 smithi132 ceph-mon[37580]: pgmap v430: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:20.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.320+0000 7fa1f4200640 1 -- 172.21.15.107:0/3092452 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1ec10ba50 msgr2=0x7fa1ec112280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.320+0000 7fa1f4200640 1 --2- 172.21.15.107:0/3092452 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1ec10ba50 0x7fa1ec112280 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009be0 tx=0x7fa1e402a6c0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.320+0000 7fa1f4200640 1 -- 172.21.15.107:0/3092452 shutdown_connections 2024-07-26T12:47:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.320+0000 7fa1f4200640 1 --2- 172.21.15.107:0/3092452 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1ec10ba50 0x7fa1ec112280 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.320+0000 7fa1f4200640 1 --2- 172.21.15.107:0/3092452 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec10b100 0x7fa1ec10b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.320+0000 7fa1f4200640 1 --2- 172.21.15.107:0/3092452 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1ec1057c0 0x7fa1ec105bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.320+0000 7fa1f4200640 1 -- 172.21.15.107:0/3092452 >> 172.21.15.107:0/3092452 conn(0x7fa1ec101290 msgr2=0x7fa1ec1036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.320+0000 7fa1f4200640 1 -- 172.21.15.107:0/3092452 shutdown_connections 2024-07-26T12:47:20.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.320+0000 7fa1f4200640 1 -- 172.21.15.107:0/3092452 wait complete. 2024-07-26T12:47:20.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.322+0000 7fa1f4200640 1 Processor -- start 2024-07-26T12:47:20.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.322+0000 7fa1f4200640 1 -- start start 2024-07-26T12:47:20.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.323+0000 7fa1f4200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1ec1057c0 0x7fa1ec19c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.323+0000 7fa1f4200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1ec10b100 0x7fa1ec19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.323+0000 7fa1f4200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec10ba50 0x7fa1ec1a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.323+0000 7fa1f4200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec115700 con 0x7fa1ec10ba50 2024-07-26T12:47:20.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.323+0000 7fa1f4200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec115580 con 0x7fa1ec10b100 2024-07-26T12:47:20.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.323+0000 7fa1f4200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec115880 con 0x7fa1ec1057c0 2024-07-26T12:47:20.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.323+0000 7fa1f1800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1ec1057c0 0x7fa1ec19c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.323+0000 7fa1f1800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1ec1057c0 0x7fa1ec19c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53500/0 (socket says 172.21.15.107:53500) 2024-07-26T12:47:20.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.323+0000 7fa1f1800640 1 -- 172.21.15.107:0/3556664025 learned_addr learned my addr 172.21.15.107:0/3556664025 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.323+0000 7fa1f0e00640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1ec10b100 0x7fa1ec19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.324+0000 7fa1f2200640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec10ba50 0x7fa1ec1a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.324+0000 7fa1f1800640 1 -- 172.21.15.107:0/3556664025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1ec10b100 msgr2=0x7fa1ec19c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.324+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1ec10b100 0x7fa1ec19c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.324+0000 7fa1f1800640 1 -- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec10ba50 msgr2=0x7fa1ec1a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.324+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec10ba50 0x7fa1ec1a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.324+0000 7fa1f1800640 1 -- 172.21.15.107:0/3556664025 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec1a2000 con 0x7fa1ec1057c0 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.324+0000 7fa1f2200640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec10ba50 0x7fa1ec1a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.324+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1ec1057c0 0x7fa1ec19c090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000e9c0 tx=0x7fa1e000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.324+0000 7fa1ea000640 1 -- 172.21.15.107:0/3556664025 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e000cde0 con 0x7fa1ec1057c0 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.325+0000 7fa1f4200640 1 -- 172.21.15.107:0/3556664025 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ec1a22f0 con 0x7fa1ec1057c0 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.325+0000 7fa1ea000640 1 -- 172.21.15.107:0/3556664025 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1e00035f0 con 0x7fa1ec1057c0 2024-07-26T12:47:20.327 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.325+0000 7fa1ea000640 1 -- 172.21.15.107:0/3556664025 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0010690 con 0x7fa1ec1057c0 2024-07-26T12:47:20.328 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.325+0000 7fa1f4200640 1 -- 172.21.15.107:0/3556664025 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ec1a8a50 con 0x7fa1ec1057c0 2024-07-26T12:47:20.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.328+0000 7fa1fae01740 1 Processor -- start 2024-07-26T12:47:20.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.328+0000 7fa1ea000640 1 -- 172.21.15.107:0/3556664025 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fa1e0010830 con 0x7fa1ec1057c0 2024-07-26T12:47:20.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.328+0000 7fa1fae01740 1 -- start start 2024-07-26T12:47:20.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.329+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bf1df60 0x555a8c2c3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.329+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.329+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 0x555a8c2c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.329+0000 7fa1fae01740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555a8c239950 con 0x555a8befd280 2024-07-26T12:47:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.329+0000 7fa1fae01740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555a8c228e50 con 0x555a8bef50a0 2024-07-26T12:47:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.329+0000 7fa1fae01740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555a8c22ce90 con 0x555a8bf1df60 2024-07-26T12:47:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.329+0000 7fa1f2200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bf1df60 0x555a8c2c3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.329+0000 7fa1f1800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.329+0000 7fa1f1800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2c3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:35646/0 (socket says 172.21.15.107:35646) 2024-07-26T12:47:20.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.329+0000 7fa1f1800640 1 -- 172.21.15.107:0/3358851951 learned_addr learned my addr 172.21.15.107:0/3358851951 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:20.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.330+0000 7fa1f0e00640 1 --2- 172.21.15.107:0/3358851951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 0x555a8c2c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.330+0000 7fa1f1800640 1 -- 172.21.15.107:0/3358851951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bf1df60 msgr2=0x555a8c2c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:20.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.330+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3358851951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bf1df60 0x555a8c2c3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.330+0000 7fa1f1800640 1 -- 172.21.15.107:0/3358851951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 msgr2=0x555a8c2c3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:20.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.330+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3358851951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 0x555a8c2c3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.330+0000 7fa1f1800640 1 -- 172.21.15.107:0/3358851951 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a8c1d00f0 con 0x555a8bef50a0 2024-07-26T12:47:20.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.330+0000 7fa1f0e00640 1 --2- 172.21.15.107:0/3358851951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 0x555a8c2c3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:47:20.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.330+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3358851951 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2c3a90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00ec400 tx=0x7fa1e00ec8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e98afc8da1908ea server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:20.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.331+0000 7fa1dea00640 1 -- 172.21.15.107:0/3358851951 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00f8070 con 0x555a8bef50a0 2024-07-26T12:47:20.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.331+0000 7fa1dea00640 1 -- 172.21.15.107:0/3358851951 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1e00e6580 con 0x555a8bef50a0 2024-07-26T12:47:20.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.331+0000 7fa1dea00640 1 -- 172.21.15.107:0/3358851951 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00f0a30 con 0x555a8bef50a0 2024-07-26T12:47:20.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.331+0000 7fa1ea000640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa1d40b3ca0 0x7fa1d40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.332+0000 7fa1ea000640 1 -- 172.21.15.107:0/3556664025 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa1e00e4470 con 0x7fa1ec1057c0 2024-07-26T12:47:20.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.332+0000 7fa1f2200640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa1d40b3ca0 0x7fa1d40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.333+0000 7fa1fae01740 1 -- 172.21.15.107:0/3358851951 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 msgr2=0x555a8c2c3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:20.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.333+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3358851951 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2c3a90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00ec400 tx=0x7fa1e00ec8c0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.333+0000 7fa1f2200640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa1d40b3ca0 0x7fa1d40b61b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009420 tx=0x7fa1e4009350 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:20.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.333+0000 7fa1fae01740 1 -- 172.21.15.107:0/3358851951 shutdown_connections 2024-07-26T12:47:20.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.333+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3358851951 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 0x555a8c2c3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.333+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3358851951 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2c3a90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.333+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3358851951 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bf1df60 0x555a8c2c3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.333+0000 7fa1fae01740 1 -- 172.21.15.107:0/3358851951 >> 172.21.15.107:0/3358851951 conn(0x555a8bf12350 msgr2=0x555a8c2c1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:20.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.333+0000 7fa1fae01740 1 -- 172.21.15.107:0/3358851951 shutdown_connections 2024-07-26T12:47:20.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.333+0000 7fa1fae01740 1 -- 172.21.15.107:0/3358851951 wait complete. 2024-07-26T12:47:20.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.334+0000 7fa1fae01740 1 -- start start 2024-07-26T12:47:20.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.334+0000 7fa1fae01740 1 Processor -- start 2024-07-26T12:47:20.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.334+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.334+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8befd280 0x555a8c2cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.334+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8bf1df60 0x555a8c2db020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.334+0000 7fa1fae01740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555a8c08c5f0 con 0x555a8bf1df60 2024-07-26T12:47:20.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.334+0000 7fa1fae01740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555a8c1c5520 con 0x555a8bef50a0 2024-07-26T12:47:20.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.334+0000 7fa1fae01740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555a8c2d1d30 con 0x555a8befd280 2024-07-26T12:47:20.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1fae01740 10 client.0 _ll_register_callbacks cb 0x555a8c1d4ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:47:20.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f1800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f1800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:35650/0 (socket says 172.21.15.107:35650) 2024-07-26T12:47:20.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f1800640 1 -- 172.21.15.107:0/3330931403 learned_addr learned my addr 172.21.15.107:0/3330931403 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:20.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f0e00640 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8befd280 0x555a8c2cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f2200640 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8bf1df60 0x555a8c2db020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f1800640 1 -- 172.21.15.107:0/3330931403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8befd280 msgr2=0x555a8c2cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:20.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8befd280 0x555a8c2cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f1800640 1 -- 172.21.15.107:0/3330931403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8bf1df60 msgr2=0x555a8c2db020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:20.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8bf1df60 0x555a8c2db020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f1800640 1 -- 172.21.15.107:0/3330931403 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a8bf89540 con 0x555a8bef50a0 2024-07-26T12:47:20.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.335+0000 7fa1f0e00640 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8befd280 0x555a8c2cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:47:20.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.336+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2cf040 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00f6040 tx=0x7fa1e000c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=490186196211215b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:20.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.336+0000 7fa1d3400640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:47:20.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.336+0000 7fa1d3400640 1 -- 172.21.15.107:0/3330931403 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00aad00 con 0x555a8bef50a0 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.336+0000 7fa1d3400640 1 -- 172.21.15.107:0/3330931403 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa1e00094f0 con 0x555a8bef50a0 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.336+0000 7fa1fae01740 1 -- 172.21.15.107:0/3330931403 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1e00e6580 con 0x555a8bef50a0 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.336+0000 7fa1d3400640 1 -- 172.21.15.107:0/3330931403 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa1e00aa6a0 con 0x555a8bef50a0 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.336+0000 7fa1fae01740 10 client.25199 fetch_fsmap learned FSMap version 28 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.336+0000 7fa1fae01740 1 -- 172.21.15.107:0/3330931403 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555a8c2ddc70 con 0x555a8bef50a0 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.337+0000 7fa1d3400640 1 -- 172.21.15.107:0/3330931403 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1e00e6580 con 0x555a8bef50a0 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.337+0000 7fa1d3400640 1 -- 172.21.15.107:0/3330931403 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa1e00e1af0 con 0x555a8bef50a0 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.337+0000 7fa1fae01740 10 client.25199 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.337+0000 7fa1fae01740 10 client.25199 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.337+0000 7fa1fae01740 20 client.25199 populate_metadata read hostname 'smithi107' 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.337+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x555a8bedc8e0 0x555a8c2e1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.337+0000 7fa1fae01740 4 client.25199 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.337+0000 7fa1fae01740 1 -- 172.21.15.107:0/3330931403 --> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a8c23b8a0 con 0x555a8bedc8e0 2024-07-26T12:47:20.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.338+0000 7fa1f0e00640 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x555a8bedc8e0 0x555a8c2e1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.339+0000 7fa1f0e00640 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x555a8bedc8e0 0x555a8c2e1150 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cff928161d8c3e2d server_cookie=17d4dc325035d832 in_seq=0 out_seq=0 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.339+0000 7fa1d3400640 10 client.25199 ms_handle_connect on v2:172.21.15.132:6838/3489644052 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.339+0000 7fa1d3400640 1 -- 172.21.15.107:0/3330931403 <== mds.2 v2:172.21.15.132:6838/3489644052 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555a8c23b8a0 con 0x555a8bedc8e0 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.339+0000 7fa1d3400640 10 client.25199 handle_command_reply: tid=0 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 client.25199 shutdown 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 -- 172.21.15.107:0/3330931403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 msgr2=0x555a8c2cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2cf040 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00f6040 tx=0x7fa1e000c7d0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 -- 172.21.15.107:0/3330931403 shutdown_connections 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8bf1df60 0x555a8c2db020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8befd280 0x555a8c2cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x555a8bedc8e0 0x555a8c2e1150 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3330931403 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bef50a0 0x555a8c2cf040 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 -- 172.21.15.107:0/3330931403 >> 172.21.15.107:0/3330931403 conn(0x555a8bf12350 msgr2=0x555a8c2c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:20.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1d3400640 0 client.25199 ms_handle_reset on v2:172.21.15.132:6838/3489644052 2024-07-26T12:47:20.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 -- 172.21.15.107:0/3330931403 shutdown_connections 2024-07-26T12:47:20.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.340+0000 7fa1fae01740 1 -- 172.21.15.107:0/3330931403 wait complete. 2024-07-26T12:47:20.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.341+0000 7fa1fae01740 20 client.25199 trim_cache size 0 max 16384 2024-07-26T12:47:20.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.360+0000 7fa1fae01740 1 Processor -- start 2024-07-26T12:47:20.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.360+0000 7fa1fae01740 1 -- start start 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2c1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 0x555a8c2e1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bf1df60 0x555a8c2c9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1fae01740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555a8c2df490 con 0x555a8befd280 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1fae01740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555a8c2dfb60 con 0x555a8bf1df60 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1fae01740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555a8c2e0230 con 0x555a8bef50a0 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1f1800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2c1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1f0e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 0x555a8c2e1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1f1800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2c1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53530/0 (socket says 172.21.15.107:53530) 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1f1800640 1 -- 172.21.15.107:0/3986546243 learned_addr learned my addr 172.21.15.107:0/3986546243 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:20.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1f2200640 1 --2- 172.21.15.107:0/3986546243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bf1df60 0x555a8c2c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1f1800640 1 -- 172.21.15.107:0/3986546243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bf1df60 msgr2=0x555a8c2c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:20.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3986546243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bf1df60 0x555a8c2c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.361+0000 7fa1f1800640 1 -- 172.21.15.107:0/3986546243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 msgr2=0x555a8c2e1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:20.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.362+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3986546243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 0x555a8c2e1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.362+0000 7fa1f1800640 1 -- 172.21.15.107:0/3986546243 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a8c2c5840 con 0x555a8bef50a0 2024-07-26T12:47:20.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.362+0000 7fa1f1800640 1 --2- 172.21.15.107:0/3986546243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2c1600 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0004770 tx=0x7fa1e00ec8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c056eacc1349158 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:20.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.362+0000 7fa1de000640 1 -- 172.21.15.107:0/3986546243 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00aa880 con 0x555a8bef50a0 2024-07-26T12:47:20.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.362+0000 7fa1de000640 1 -- 172.21.15.107:0/3986546243 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1e00ecdc0 con 0x555a8bef50a0 2024-07-26T12:47:20.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.362+0000 7fa1de000640 1 -- 172.21.15.107:0/3986546243 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00ec250 con 0x555a8bef50a0 2024-07-26T12:47:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.363+0000 7fa1fae01740 1 -- 172.21.15.107:0/3986546243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 msgr2=0x555a8c2c1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.363+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3986546243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2c1600 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0004770 tx=0x7fa1e00ec8c0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.363+0000 7fa1fae01740 1 -- 172.21.15.107:0/3986546243 shutdown_connections 2024-07-26T12:47:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.363+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3986546243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8bf1df60 0x555a8c2c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.363+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3986546243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8befd280 0x555a8c2e1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.363+0000 7fa1fae01740 1 --2- 172.21.15.107:0/3986546243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2c1600 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.363+0000 7fa1fae01740 1 -- 172.21.15.107:0/3986546243 >> 172.21.15.107:0/3986546243 conn(0x555a8bf12350 msgr2=0x555a8c2c9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.363+0000 7fa1fae01740 1 -- 172.21.15.107:0/3986546243 shutdown_connections 2024-07-26T12:47:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.363+0000 7fa1fae01740 1 -- 172.21.15.107:0/3986546243 wait complete. 2024-07-26T12:47:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.364+0000 7fa1fae01740 1 -- start start 2024-07-26T12:47:20.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.364+0000 7fa1fae01740 1 Processor -- start 2024-07-26T12:47:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2e2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8befd280 0x555a8c2e2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1fae01740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8bf1df60 0x555a8c2e7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1fae01740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555a8c2e0fd0 con 0x555a8bf1df60 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1fae01740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x555a8c2e0900 con 0x555a8befd280 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1fae01740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x555a8c2e16a0 con 0x555a8bef50a0 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1fae01740 10 client.0 _ll_register_callbacks cb 0x7fa1e00046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1f2200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8bf1df60 0x555a8c2e7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1f0e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8befd280 0x555a8c2e2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1f1800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2e2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1f0e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8befd280 0x555a8c2e2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:35678/0 (socket says 172.21.15.107:35678) 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.365+0000 7fa1f0e00640 1 -- 172.21.15.107:0/2829454248 learned_addr learned my addr 172.21.15.107:0/2829454248 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1f0e00640 1 -- 172.21.15.107:0/2829454248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 msgr2=0x555a8c2e2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1f0e00640 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2e2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1f0e00640 1 -- 172.21.15.107:0/2829454248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8bf1df60 msgr2=0x555a8c2e7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:20.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1f0e00640 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8bf1df60 0x555a8c2e7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1f0e00640 1 -- 172.21.15.107:0/2829454248 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a8c2e8050 con 0x555a8befd280 2024-07-26T12:47:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1f1800640 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2e2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:47:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1f0e00640 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8befd280 0x555a8c2e2af0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x555a8c2e1a70 tx=0x7fa1d8010ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d79facbf09b500b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1d3400640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:47:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1d3400640 1 -- 172.21.15.107:0/2829454248 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8005020 con 0x555a8befd280 2024-07-26T12:47:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1fae01740 1 -- 172.21.15.107:0/2829454248 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1e00ecdc0 con 0x555a8befd280 2024-07-26T12:47:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.366+0000 7fa1d3400640 1 -- 172.21.15.107:0/2829454248 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa1d80051c0 con 0x555a8befd280 2024-07-26T12:47:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1d3400640 1 -- 172.21.15.107:0/2829454248 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa1d802f050 con 0x555a8befd280 2024-07-26T12:47:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1fae01740 10 client.25205 fetch_fsmap learned FSMap version 28 2024-07-26T12:47:20.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1fae01740 1 -- 172.21.15.107:0/2829454248 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555a8c2e94c0 con 0x555a8befd280 2024-07-26T12:47:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1d3400640 1 -- 172.21.15.107:0/2829454248 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1e00ecdc0 con 0x555a8befd280 2024-07-26T12:47:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1d3400640 1 -- 172.21.15.107:0/2829454248 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa1d80035f0 con 0x555a8befd280 2024-07-26T12:47:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1fae01740 10 client.25205 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:47:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1fae01740 10 client.25205 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T12:47:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1fae01740 20 client.25205 populate_metadata read hostname 'smithi107' 2024-07-26T12:47:20.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1fae01740 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x555a8beb1c90 0x555a8c34cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1fae01740 4 client.25205 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.367+0000 7fa1fae01740 1 -- 172.21.15.107:0/2829454248 --> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] -- command(tid 0: {"prefix": "status"}) -- 0x555a8c237340 con 0x555a8beb1c90 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.368+0000 7fa1f1800640 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x555a8beb1c90 0x555a8c34cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.368+0000 7fa1f1800640 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x555a8beb1c90 0x555a8c34cf50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8b2d9e26931fc4ef server_cookie=d1d7c2dc11184f54 in_seq=0 out_seq=0 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.368+0000 7fa1d3400640 10 client.25205 ms_handle_connect on v2:172.21.15.132:6838/3489644052 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1d3400640 1 -- 172.21.15.107:0/2829454248 <== mds.2 v2:172.21.15.132:6838/3489644052 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x555a8c237340 con 0x555a8beb1c90 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1d3400640 10 client.25205 handle_command_reply: tid=0 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 client.25205 shutdown 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 -- 172.21.15.107:0/2829454248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8befd280 msgr2=0x555a8c2e2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8befd280 0x555a8c2e2af0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x555a8c2e1a70 tx=0x7fa1d8010ab0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 -- 172.21.15.107:0/2829454248 shutdown_connections 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555a8bf1df60 0x555a8c2e7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x555a8beb1c90 0x555a8c34cf50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x555a8befd280 0x555a8c2e2af0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 --2- 172.21.15.107:0/2829454248 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x555a8bef50a0 0x555a8c2e2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1d3400640 0 client.25205 ms_handle_reset on v2:172.21.15.132:6838/3489644052 2024-07-26T12:47:20.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 -- 172.21.15.107:0/2829454248 >> 172.21.15.107:0/2829454248 conn(0x555a8bf12350 msgr2=0x555a8c2bf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:20.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 -- 172.21.15.107:0/2829454248 shutdown_connections 2024-07-26T12:47:20.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.369+0000 7fa1fae01740 1 -- 172.21.15.107:0/2829454248 wait complete. 2024-07-26T12:47:20.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.370+0000 7fa1fae01740 20 client.25205 trim_cache size 0 max 16384 2024-07-26T12:47:20.374 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:47:20.374 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 2, 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24703, 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 163.529444184, 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 90, 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 275.30718275100003 2024-07-26T12:47:20.375 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:47:20.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.370+0000 7fa1f4200640 1 -- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa1d40b3ca0 msgr2=0x7fa1d40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:20.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.370+0000 7fa1f4200640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa1d40b3ca0 0x7fa1d40b61b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009420 tx=0x7fa1e4009350 comp rx=0 tx=0).stop 2024-07-26T12:47:20.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.370+0000 7fa1f4200640 1 -- 172.21.15.107:0/3556664025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1ec1057c0 msgr2=0x7fa1ec19c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:20.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.370+0000 7fa1f4200640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1ec1057c0 0x7fa1ec19c090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000e9c0 tx=0x7fa1e000ee80 comp rx=0 tx=0).stop 2024-07-26T12:47:20.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.371+0000 7fa1f4200640 1 -- 172.21.15.107:0/3556664025 shutdown_connections 2024-07-26T12:47:20.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.371+0000 7fa1f4200640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa1d40b3ca0 0x7fa1d40b61b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.371+0000 7fa1f4200640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1ec10ba50 0x7fa1ec1a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.371+0000 7fa1f4200640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa1ec10b100 0x7fa1ec19c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.371+0000 7fa1f4200640 1 --2- 172.21.15.107:0/3556664025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1ec1057c0 0x7fa1ec19c090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:20.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.371+0000 7fa1f4200640 1 -- 172.21.15.107:0/3556664025 >> 172.21.15.107:0/3556664025 conn(0x7fa1ec101290 msgr2=0x7fa1ec1096e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:20.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.371+0000 7fa1f4200640 1 -- 172.21.15.107:0/3556664025 shutdown_connections 2024-07-26T12:47:20.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:20.371+0000 7fa1f4200640 1 -- 172.21.15.107:0/3556664025 wait complete. 2024-07-26T12:47:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:47:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:47:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:47:21.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.g status 2024-07-26T12:47:21.899 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:22.168 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:21 smithi107 ceph-mon[32659]: pgmap v431: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:21 smithi112 ceph-mon[37501]: pgmap v431: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:21 smithi132 ceph-mon[37580]: pgmap v431: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:23.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:22 smithi107 ceph-mon[32659]: pgmap v432: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:23.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:22 smithi112 ceph-mon[37501]: pgmap v432: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:22 smithi132 ceph-mon[37580]: pgmap v432: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:24.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.340+0000 7f1959000640 1 -- 172.21.15.107:0/1117687483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f195410b120 msgr2=0x7f195410b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:24.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.340+0000 7f1959000640 1 --2- 172.21.15.107:0/1117687483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f195410b120 0x7f195410b530 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1940009900 tx=0x7f194002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.340+0000 7f1959000640 1 -- 172.21.15.107:0/1117687483 shutdown_connections 2024-07-26T12:47:24.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.340+0000 7f1959000640 1 --2- 172.21.15.107:0/1117687483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f195410ba70 0x7f19541122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.340+0000 7f1959000640 1 --2- 172.21.15.107:0/1117687483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f195410b120 0x7f195410b530 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.340+0000 7f1959000640 1 --2- 172.21.15.107:0/1117687483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f19541057e0 0x7f1954105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.340+0000 7f1959000640 1 -- 172.21.15.107:0/1117687483 >> 172.21.15.107:0/1117687483 conn(0x7f19541011f0 msgr2=0x7f1954103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:24.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.340+0000 7f1959000640 1 -- 172.21.15.107:0/1117687483 shutdown_connections 2024-07-26T12:47:24.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.340+0000 7f1959000640 1 -- 172.21.15.107:0/1117687483 wait complete. 2024-07-26T12:47:24.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.342+0000 7f1959000640 1 Processor -- start 2024-07-26T12:47:24.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.342+0000 7f1959000640 1 -- start start 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.342+0000 7f1959000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541057e0 0x7f195419c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.342+0000 7f1959000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f195410b120 0x7f195419c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.342+0000 7f1959000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f195410ba70 0x7f19541a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1959000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1954115700 con 0x7f19541057e0 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1959000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1954115580 con 0x7f195410ba70 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1959000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1954115880 con 0x7f195410b120 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1951e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541057e0 0x7f195419c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1951e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541057e0 0x7f195419c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43278/0 (socket says 172.21.15.107:43278) 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1952800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f195410ba70 0x7f19541a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1951e00640 1 -- 172.21.15.107:0/369235749 learned_addr learned my addr 172.21.15.107:0/369235749 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:24.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1951400640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f195410b120 0x7f195419c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1952800640 1 -- 172.21.15.107:0/369235749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f195410b120 msgr2=0x7f195419c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1952800640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f195410b120 0x7f195419c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1952800640 1 -- 172.21.15.107:0/369235749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541057e0 msgr2=0x7f195419c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1952800640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541057e0 0x7f195419c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.343+0000 7f1952800640 1 -- 172.21.15.107:0/369235749 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19541a2120 con 0x7f195410ba70 2024-07-26T12:47:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.344+0000 7f1952800640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f195410ba70 0x7f19541a19f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f194c009fd0 tx=0x7f194c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.344+0000 7f194aa00640 1 -- 172.21.15.107:0/369235749 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c00ce20 con 0x7f195410ba70 2024-07-26T12:47:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.344+0000 7f1959000640 1 -- 172.21.15.107:0/369235749 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19541a2410 con 0x7f195410ba70 2024-07-26T12:47:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.344+0000 7f194aa00640 1 -- 172.21.15.107:0/369235749 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f194c005020 con 0x7f195410ba70 2024-07-26T12:47:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.344+0000 7f194aa00640 1 -- 172.21.15.107:0/369235749 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c010690 con 0x7f195410ba70 2024-07-26T12:47:24.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.344+0000 7f1959000640 1 -- 172.21.15.107:0/369235749 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19541a8b70 con 0x7f195410ba70 2024-07-26T12:47:24.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.348+0000 7f195fc1a740 1 Processor -- start 2024-07-26T12:47:24.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.349+0000 7f194aa00640 1 -- 172.21.15.107:0/369235749 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f194c0054d0 con 0x7f195410ba70 2024-07-26T12:47:24.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.349+0000 7f195fc1a740 1 -- start start 2024-07-26T12:47:24.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 0x558f7f3c5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7efff280 0x558f7f3c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f195fc1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558f7f33b970 con 0x558f7eff70a0 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f195fc1a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558f7f32ae70 con 0x558f7f01ff60 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f195fc1a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558f7f32eeb0 con 0x558f7efff280 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f1951e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f1951e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3c5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43294/0 (socket says 172.21.15.107:43294) 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f1951e00640 1 -- 172.21.15.107:0/2746188684 learned_addr learned my addr 172.21.15.107:0/2746188684 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f1951400640 1 --2- 172.21.15.107:0/2746188684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 0x558f7f3c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.350+0000 7f1951400640 1 --2- 172.21.15.107:0/2746188684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7efff280 0x558f7f3c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f1951e00640 1 -- 172.21.15.107:0/2746188684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7efff280 msgr2=0x558f7f3c5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f1951e00640 1 --2- 172.21.15.107:0/2746188684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7efff280 0x558f7f3c5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f1951e00640 1 -- 172.21.15.107:0/2746188684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 msgr2=0x558f7f3c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f1951e00640 1 --2- 172.21.15.107:0/2746188684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 0x558f7f3c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f1951e00640 1 -- 172.21.15.107:0/2746188684 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f7f2d2110 con 0x558f7eff70a0 2024-07-26T12:47:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f1951400640 1 --2- 172.21.15.107:0/2746188684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 0x558f7f3c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:47:24.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f1951e00640 1 --2- 172.21.15.107:0/2746188684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3c5ab0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1944009dc0 tx=0x7f1944011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406bd01dcaaa89b4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:24.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f194aa00640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f19380b3c20 0x7f19380b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f193f400640 1 -- 172.21.15.107:0/2746188684 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944005220 con 0x558f7eff70a0 2024-07-26T12:47:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f193f400640 1 -- 172.21.15.107:0/2746188684 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1944004c50 con 0x558f7eff70a0 2024-07-26T12:47:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.351+0000 7f193f400640 1 -- 172.21.15.107:0/2746188684 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19440197b0 con 0x558f7eff70a0 2024-07-26T12:47:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.352+0000 7f194aa00640 1 -- 172.21.15.107:0/369235749 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f194c0e4780 con 0x7f195410ba70 2024-07-26T12:47:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.352+0000 7f1951400640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f19380b3c20 0x7f19380b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f195fc1a740 1 -- 172.21.15.107:0/2746188684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 msgr2=0x558f7f3c5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f195fc1a740 1 --2- 172.21.15.107:0/2746188684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3c5ab0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1944009dc0 tx=0x7f1944011810 comp rx=0 tx=0).stop 2024-07-26T12:47:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f1951400640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f19380b3c20 0x7f19380b6130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1940009230 tx=0x7f1940000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f195fc1a740 1 -- 172.21.15.107:0/2746188684 shutdown_connections 2024-07-26T12:47:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f195fc1a740 1 --2- 172.21.15.107:0/2746188684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7efff280 0x558f7f3c5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f195fc1a740 1 --2- 172.21.15.107:0/2746188684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3c5ab0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f195fc1a740 1 --2- 172.21.15.107:0/2746188684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 0x558f7f3c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f195fc1a740 1 -- 172.21.15.107:0/2746188684 >> 172.21.15.107:0/2746188684 conn(0x558f7f014350 msgr2=0x558f7f3c3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f195fc1a740 1 -- 172.21.15.107:0/2746188684 shutdown_connections 2024-07-26T12:47:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f195fc1a740 1 -- 172.21.15.107:0/2746188684 wait complete. 2024-07-26T12:47:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.353+0000 7f195fc1a740 1 -- start start 2024-07-26T12:47:24.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.354+0000 7f195fc1a740 1 Processor -- start 2024-07-26T12:47:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.354+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.354+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7efff280 0x558f7f3d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.354+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 0x558f7f3dd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.354+0000 7f195fc1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558f7f18e5f0 con 0x558f7eff70a0 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.354+0000 7f195fc1a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558f7f2c7540 con 0x558f7f01ff60 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.354+0000 7f195fc1a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558f7f3d3d50 con 0x558f7efff280 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.354+0000 7f195fc1a740 10 client.0 _ll_register_callbacks cb 0x558f7f2d6f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1951e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1951e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43300/0 (socket says 172.21.15.107:43300) 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1951e00640 1 -- 172.21.15.107:0/3921565142 learned_addr learned my addr 172.21.15.107:0/3921565142 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1952800640 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 0x558f7f3dd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1951400640 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7efff280 0x558f7f3d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1952800640 1 -- 172.21.15.107:0/3921565142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7efff280 msgr2=0x558f7f3d1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1952800640 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7efff280 0x558f7f3d1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1952800640 1 -- 172.21.15.107:0/3921565142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 msgr2=0x558f7f3d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:24.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1952800640 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1952800640 1 -- 172.21.15.107:0/3921565142 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f7f08b540 con 0x558f7f01ff60 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1952800640 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 0x558f7f3dd0d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f194c0ab160 tx=0x7f194c0ab990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ac8b9dcf5222e02 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.355+0000 7f1933e00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.356+0000 7f1933e00640 1 -- 172.21.15.107:0/3921565142 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c0aafa0 con 0x558f7f01ff60 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.356+0000 7f1933e00640 1 -- 172.21.15.107:0/3921565142 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f194c0a77c0 con 0x558f7f01ff60 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.356+0000 7f195fc1a740 1 -- 172.21.15.107:0/3921565142 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1944004c50 con 0x558f7f01ff60 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.356+0000 7f1933e00640 1 -- 172.21.15.107:0/3921565142 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f194c0b4280 con 0x558f7f01ff60 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.356+0000 7f195fc1a740 10 client.25217 fetch_fsmap learned FSMap version 28 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.356+0000 7f195fc1a740 1 -- 172.21.15.107:0/3921565142 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558f7f3dfc20 con 0x558f7f01ff60 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.356+0000 7f1933e00640 1 -- 172.21.15.107:0/3921565142 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1944004c50 con 0x558f7f01ff60 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.356+0000 7f1933e00640 1 -- 172.21.15.107:0/3921565142 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f194c0a6b40 con 0x558f7f01ff60 2024-07-26T12:47:24.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.356+0000 7f195fc1a740 10 client.25217 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.357+0000 7f195fc1a740 10 client.25217 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.357+0000 7f195fc1a740 20 client.25217 populate_metadata read hostname 'smithi107' 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.357+0000 7f195fc1a740 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558f7efde8e0 0x558f7f3e3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.357+0000 7f195fc1a740 4 client.25217 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.357+0000 7f195fc1a740 1 -- 172.21.15.107:0/3921565142 --> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f7f33d8c0 con 0x558f7efde8e0 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.357+0000 7f1951e00640 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558f7efde8e0 0x558f7f3e3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.358+0000 7f1951e00640 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558f7efde8e0 0x558f7f3e3650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=844d7448e0e709b5 server_cookie=3e7067059137207 in_seq=0 out_seq=0 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.358+0000 7f1933e00640 10 client.25217 ms_handle_connect on v2:172.21.15.107:6838/137526780 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f1933e00640 1 -- 172.21.15.107:0/3921565142 <== mds.1 v2:172.21.15.107:6838/137526780 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558f7f33d8c0 con 0x558f7efde8e0 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f1933e00640 10 client.25217 handle_command_reply: tid=0 2024-07-26T12:47:24.363 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 client.25217 shutdown 2024-07-26T12:47:24.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 -- 172.21.15.107:0/3921565142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 msgr2=0x558f7f3dd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:24.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 0x558f7f3dd0d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f194c0ab160 tx=0x7f194c0ab990 comp rx=0 tx=0).stop 2024-07-26T12:47:24.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 -- 172.21.15.107:0/3921565142 shutdown_connections 2024-07-26T12:47:24.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558f7efde8e0 0x558f7f3e3650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7f01ff60 0x558f7f3dd0d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7efff280 0x558f7f3d1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 --2- 172.21.15.107:0/3921565142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7eff70a0 0x558f7f3d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f1933e00640 0 client.25217 ms_handle_reset on v2:172.21.15.107:6838/137526780 2024-07-26T12:47:24.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 -- 172.21.15.107:0/3921565142 >> 172.21.15.107:0/3921565142 conn(0x558f7f014350 msgr2=0x558f7f3c8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:24.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 -- 172.21.15.107:0/3921565142 shutdown_connections 2024-07-26T12:47:24.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 1 -- 172.21.15.107:0/3921565142 wait complete. 2024-07-26T12:47:24.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.359+0000 7f195fc1a740 20 client.25217 trim_cache size 0 max 16384 2024-07-26T12:47:24.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.379+0000 7f195fc1a740 1 Processor -- start 2024-07-26T12:47:24.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.379+0000 7f195fc1a740 1 -- start start 2024-07-26T12:47:24.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.379+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7eff70a0 0x558f7f3dd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.379+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7efff280 0x558f7f3cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7f01ff60 0x558f7f3cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f195fc1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558f7f3e1990 con 0x558f7f01ff60 2024-07-26T12:47:24.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f195fc1a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558f7f3e2060 con 0x558f7efff280 2024-07-26T12:47:24.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f195fc1a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558f7f3e2730 con 0x558f7eff70a0 2024-07-26T12:47:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f1952800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7f01ff60 0x558f7f3cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f1952800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7f01ff60 0x558f7f3cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43304/0 (socket says 172.21.15.107:43304) 2024-07-26T12:47:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f1952800640 1 -- 172.21.15.107:0/3223290057 learned_addr learned my addr 172.21.15.107:0/3223290057 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f1951400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7efff280 0x558f7f3cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f1951e00640 1 --2- 172.21.15.107:0/3223290057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7eff70a0 0x558f7f3dd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f1952800640 1 -- 172.21.15.107:0/3223290057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7eff70a0 msgr2=0x558f7f3dd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f1952800640 1 --2- 172.21.15.107:0/3223290057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7eff70a0 0x558f7f3dd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f1952800640 1 -- 172.21.15.107:0/3223290057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7efff280 msgr2=0x558f7f3cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f1952800640 1 --2- 172.21.15.107:0/3223290057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7efff280 0x558f7f3cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.380+0000 7f1952800640 1 -- 172.21.15.107:0/3223290057 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f7f3c83d0 con 0x558f7f01ff60 2024-07-26T12:47:24.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.381+0000 7f1952800640 1 --2- 172.21.15.107:0/3223290057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7f01ff60 0x558f7f3cfb10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f194c0b4440 tx=0x7f194c0053e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76262be12f7ed51f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.381+0000 7f193ea00640 1 -- 172.21.15.107:0/3223290057 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c0a9ea0 con 0x558f7f01ff60 2024-07-26T12:47:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.381+0000 7f193ea00640 1 -- 172.21.15.107:0/3223290057 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f194c0cc390 con 0x558f7f01ff60 2024-07-26T12:47:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.381+0000 7f193ea00640 1 -- 172.21.15.107:0/3223290057 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c0b3df0 con 0x558f7f01ff60 2024-07-26T12:47:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.382+0000 7f195fc1a740 1 -- 172.21.15.107:0/3223290057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7f01ff60 msgr2=0x558f7f3cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.382+0000 7f195fc1a740 1 --2- 172.21.15.107:0/3223290057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7f01ff60 0x558f7f3cfb10 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f194c0b4440 tx=0x7f194c0053e0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.382+0000 7f195fc1a740 1 -- 172.21.15.107:0/3223290057 shutdown_connections 2024-07-26T12:47:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.382+0000 7f195fc1a740 1 --2- 172.21.15.107:0/3223290057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7f01ff60 0x558f7f3cfb10 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.382+0000 7f195fc1a740 1 --2- 172.21.15.107:0/3223290057 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7efff280 0x558f7f3cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.382+0000 7f195fc1a740 1 --2- 172.21.15.107:0/3223290057 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7eff70a0 0x558f7f3dd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.382+0000 7f195fc1a740 1 -- 172.21.15.107:0/3223290057 >> 172.21.15.107:0/3223290057 conn(0x558f7f014350 msgr2=0x558f7f3cbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:24.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.382+0000 7f195fc1a740 1 -- 172.21.15.107:0/3223290057 shutdown_connections 2024-07-26T12:47:24.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.382+0000 7f195fc1a740 1 -- 172.21.15.107:0/3223290057 wait complete. 2024-07-26T12:47:24.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.383+0000 7f195fc1a740 1 -- start start 2024-07-26T12:47:24.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.383+0000 7f195fc1a740 1 Processor -- start 2024-07-26T12:47:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.383+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7eff70a0 0x558f7f3e4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.383+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7efff280 0x558f7f3e5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.383+0000 7f195fc1a740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7f01ff60 0x558f7f3e9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.383+0000 7f195fc1a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558f7f3e34d0 con 0x558f7efff280 2024-07-26T12:47:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.383+0000 7f195fc1a740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558f7f3e2e00 con 0x558f7eff70a0 2024-07-26T12:47:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.383+0000 7f195fc1a740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558f7f3e3ba0 con 0x558f7f01ff60 2024-07-26T12:47:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f195fc1a740 10 client.0 _ll_register_callbacks cb 0x7f194c0aabd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:47:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1951400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7efff280 0x558f7f3e5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1951400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7efff280 0x558f7f3e5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43316/0 (socket says 172.21.15.107:43316) 2024-07-26T12:47:24.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1951e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7eff70a0 0x558f7f3e4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1951400640 1 -- 172.21.15.107:0/4252593479 learned_addr learned my addr 172.21.15.107:0/4252593479 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1952800640 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7f01ff60 0x558f7f3e9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1951e00640 1 -- 172.21.15.107:0/4252593479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7f01ff60 msgr2=0x558f7f3e9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1951e00640 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7f01ff60 0x558f7f3e9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1951e00640 1 -- 172.21.15.107:0/4252593479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7efff280 msgr2=0x558f7f3e5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1951e00640 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7efff280 0x558f7f3e5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1951e00640 1 -- 172.21.15.107:0/4252593479 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f7f3ea460 con 0x558f7eff70a0 2024-07-26T12:47:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.384+0000 7f1951400640 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7efff280 0x558f7f3e5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:47:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.385+0000 7f1951e00640 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7eff70a0 0x558f7f3e4a50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f194401b040 tx=0x7f1944005760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dea2c9e5286241b3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.385+0000 7f1933e00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.385+0000 7f1933e00640 1 -- 172.21.15.107:0/4252593479 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1944004250 con 0x558f7eff70a0 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.385+0000 7f1933e00640 1 -- 172.21.15.107:0/4252593479 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f19440043f0 con 0x558f7eff70a0 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.385+0000 7f195fc1a740 1 -- 172.21.15.107:0/4252593479 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f194c0cc390 con 0x558f7eff70a0 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.385+0000 7f1933e00640 1 -- 172.21.15.107:0/4252593479 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1944037050 con 0x558f7eff70a0 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.385+0000 7f195fc1a740 10 client.25223 fetch_fsmap learned FSMap version 28 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.385+0000 7f195fc1a740 1 -- 172.21.15.107:0/4252593479 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558f7f3eb820 con 0x558f7eff70a0 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.386+0000 7f1933e00640 1 -- 172.21.15.107:0/4252593479 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f194c0cc390 con 0x558f7eff70a0 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.386+0000 7f1933e00640 1 -- 172.21.15.107:0/4252593479 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f19440194c0 con 0x558f7eff70a0 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.386+0000 7f195fc1a740 10 client.25223 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.386+0000 7f195fc1a740 10 client.25223 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.386+0000 7f195fc1a740 20 client.25223 populate_metadata read hostname 'smithi107' 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.386+0000 7f195fc1a740 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558f7efb3c90 0x558f7f44f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:24.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.386+0000 7f195fc1a740 4 client.25223 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:47:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.386+0000 7f195fc1a740 1 -- 172.21.15.107:0/4252593479 --> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] -- command(tid 0: {"prefix": "status"}) -- 0x558f7f339360 con 0x558f7efb3c90 2024-07-26T12:47:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.386+0000 7f1951400640 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558f7efb3c90 0x558f7f44f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.387+0000 7f1933e00640 10 client.25223 ms_handle_connect on v2:172.21.15.107:6838/137526780 2024-07-26T12:47:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.387+0000 7f1951400640 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558f7efb3c90 0x558f7f44f0f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=361b825db7aa83f9 server_cookie=d48669ba36f869e6 in_seq=0 out_seq=0 2024-07-26T12:47:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.387+0000 7f1933e00640 1 -- 172.21.15.107:0/4252593479 <== mds.1 v2:172.21.15.107:6838/137526780 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x558f7f339360 con 0x558f7efb3c90 2024-07-26T12:47:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.387+0000 7f1933e00640 10 client.25223 handle_command_reply: tid=0 2024-07-26T12:47:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.387+0000 7f195fc1a740 1 client.25223 shutdown 2024-07-26T12:47:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 1 -- 172.21.15.107:0/4252593479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7eff70a0 msgr2=0x558f7f3e4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7eff70a0 0x558f7f3e4a50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f194401b040 tx=0x7f1944005760 comp rx=0 tx=0).stop 2024-07-26T12:47:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 1 -- 172.21.15.107:0/4252593479 shutdown_connections 2024-07-26T12:47:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558f7efb3c90 0x558f7f44f0f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558f7f01ff60 0x558f7f3e9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558f7efff280 0x558f7f3e5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 1 --2- 172.21.15.107:0/4252593479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558f7eff70a0 0x558f7f3e4a50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f1933e00640 0 client.25223 ms_handle_reset on v2:172.21.15.107:6838/137526780 2024-07-26T12:47:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 1 -- 172.21.15.107:0/4252593479 >> 172.21.15.107:0/4252593479 conn(0x558f7f014350 msgr2=0x558f7f3c2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 1 -- 172.21.15.107:0/4252593479 shutdown_connections 2024-07-26T12:47:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 1 -- 172.21.15.107:0/4252593479 wait complete. 2024-07-26T12:47:24.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.388+0000 7f195fc1a740 20 client.25223 trim_cache size 0 max 16384 2024-07-26T12:47:24.394 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:47:24.394 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:47:24.394 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 1, 2024-07-26T12:47:24.394 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24715, 2024-07-26T12:47:24.395 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:47:24.395 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:47:24.395 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:47:24.395 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 168.56066455199999, 2024-07-26T12:47:24.395 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-07-26T12:47:24.395 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 90, 2024-07-26T12:47:24.395 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T12:47:24.395 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 275.26900376499998 2024-07-26T12:47:24.395 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:47:24.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 -- 172.21.15.107:0/369235749 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f19380b3c20 msgr2=0x7f19380b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:24.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f19380b3c20 0x7f19380b6130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1940009230 tx=0x7f1940000bc0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 -- 172.21.15.107:0/369235749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f195410ba70 msgr2=0x7f19541a19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:24.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f195410ba70 0x7f19541a19f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f194c009fd0 tx=0x7f194c00ef10 comp rx=0 tx=0).stop 2024-07-26T12:47:24.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 -- 172.21.15.107:0/369235749 shutdown_connections 2024-07-26T12:47:24.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f19380b3c20 0x7f19380b6130 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f195410ba70 0x7f19541a19f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f195410b120 0x7f195419c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 --2- 172.21.15.107:0/369235749 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541057e0 0x7f195419c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:24.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 -- 172.21.15.107:0/369235749 >> 172.21.15.107:0/369235749 conn(0x7f19541011f0 msgr2=0x7f19541098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:24.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 -- 172.21.15.107:0/369235749 shutdown_connections 2024-07-26T12:47:24.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:24.389+0000 7f1959000640 1 -- 172.21.15.107:0/369235749 wait complete. 2024-07-26T12:47:25.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T12:47:25.919 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:26.089 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:25 smithi107 ceph-mon[32659]: pgmap v433: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:25 smithi112 ceph-mon[37501]: pgmap v433: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:25 smithi132 ceph-mon[37580]: pgmap v433: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:28.124 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:27 smithi107 ceph-mon[32659]: pgmap v434: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:27 smithi112 ceph-mon[37501]: pgmap v434: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.221+0000 7f4b89800640 1 -- 172.21.15.107:0/4042989854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b8410f330 msgr2=0x7f4b8410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.221+0000 7f4b89800640 1 --2- 172.21.15.107:0/4042989854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b8410f330 0x7f4b8410f740 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009900 tx=0x7f4b6c02a410 comp rx=0 tx=0).stop 2024-07-26T12:47:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.221+0000 7f4b89800640 1 -- 172.21.15.107:0/4042989854 shutdown_connections 2024-07-26T12:47:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.221+0000 7f4b89800640 1 --2- 172.21.15.107:0/4042989854 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b8410fc80 0x7f4b841164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.221+0000 7f4b89800640 1 --2- 172.21.15.107:0/4042989854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b8410f330 0x7f4b8410f740 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.221+0000 7f4b89800640 1 --2- 172.21.15.107:0/4042989854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b841099f0 0x7f4b84109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.221+0000 7f4b89800640 1 -- 172.21.15.107:0/4042989854 >> 172.21.15.107:0/4042989854 conn(0x7f4b84078ff0 msgr2=0x7f4b84107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:28.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.221+0000 7f4b89800640 1 -- 172.21.15.107:0/4042989854 shutdown_connections 2024-07-26T12:47:28.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.221+0000 7f4b89800640 1 -- 172.21.15.107:0/4042989854 wait complete. 2024-07-26T12:47:28.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.223+0000 7f4b89800640 1 Processor -- start 2024-07-26T12:47:28.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.223+0000 7f4b89800640 1 -- start start 2024-07-26T12:47:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.223+0000 7f4b89800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b841099f0 0x7f4b841a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.223+0000 7f4b89800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b8410f330 0x7f4b841a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.223+0000 7f4b89800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b8410fc80 0x7f4b841a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.223+0000 7f4b89800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4b84119990 con 0x7f4b841099f0 2024-07-26T12:47:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.223+0000 7f4b89800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f4b84119810 con 0x7f4b8410f330 2024-07-26T12:47:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.223+0000 7f4b89800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4b84119b10 con 0x7f4b8410fc80 2024-07-26T12:47:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b7b400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b8410f330 0x7f4b841a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b83e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b841099f0 0x7f4b841a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b7b400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b8410f330 0x7f4b841a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:42498/0 (socket says 172.21.15.107:42498) 2024-07-26T12:47:28.225 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b7b400640 1 -- 172.21.15.107:0/2042819180 learned_addr learned my addr 172.21.15.107:0/2042819180 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b88e00640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b8410fc80 0x7f4b841a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b7b400640 1 -- 172.21.15.107:0/2042819180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b8410fc80 msgr2=0x7f4b841a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b7b400640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b8410fc80 0x7f4b841a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b7b400640 1 -- 172.21.15.107:0/2042819180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b841099f0 msgr2=0x7f4b841a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b7b400640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b841099f0 0x7f4b841a0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b7b400640 1 -- 172.21.15.107:0/2042819180 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b841a62b0 con 0x7f4b8410f330 2024-07-26T12:47:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.224+0000 7f4b7b400640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b8410f330 0x7f4b841a08e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c005ab0 tx=0x7f4b6c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.225+0000 7f4b81600640 1 -- 172.21.15.107:0/2042819180 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c03f070 con 0x7f4b8410f330 2024-07-26T12:47:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.225+0000 7f4b89800640 1 -- 172.21.15.107:0/2042819180 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b841a6540 con 0x7f4b8410f330 2024-07-26T12:47:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.225+0000 7f4b81600640 1 -- 172.21.15.107:0/2042819180 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b6c031b50 con 0x7f4b8410f330 2024-07-26T12:47:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.225+0000 7f4b81600640 1 -- 172.21.15.107:0/2042819180 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c035030 con 0x7f4b8410f330 2024-07-26T12:47:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.225+0000 7f4b89800640 1 -- 172.21.15.107:0/2042819180 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b841acd00 con 0x7f4b8410f330 2024-07-26T12:47:28.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.228+0000 7f4b919e7740 1 Processor -- start 2024-07-26T12:47:28.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.228+0000 7f4b919e7740 1 -- start start 2024-07-26T12:47:28.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.229+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4c3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.229+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4c3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.229+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0fd280 0x5580eb4c4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.229+0000 7f4b919e7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5580eb439970 con 0x5580eb11df60 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.229+0000 7f4b919e7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5580eb428e70 con 0x5580eb0f50a0 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.229+0000 7f4b919e7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5580eb42ceb0 con 0x5580eb0fd280 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.229+0000 7f4b88e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.229+0000 7f4b88e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4c3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43350/0 (socket says 172.21.15.107:43350) 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.229+0000 7f4b88e00640 1 -- 172.21.15.107:0/3264333318 learned_addr learned my addr 172.21.15.107:0/3264333318 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.229+0000 7f4b83e00640 1 --2- 172.21.15.107:0/3264333318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4c3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.230+0000 7f4b88e00640 1 -- 172.21.15.107:0/3264333318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0fd280 msgr2=0x5580eb4c4200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.230+0000 7f4b88e00640 1 --2- 172.21.15.107:0/3264333318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0fd280 0x5580eb4c4200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.230+0000 7f4b88e00640 1 -- 172.21.15.107:0/3264333318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 msgr2=0x5580eb4c3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.230+0000 7f4b88e00640 1 --2- 172.21.15.107:0/3264333318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4c3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.230+0000 7f4b88e00640 1 -- 172.21.15.107:0/3264333318 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580eb3d0110 con 0x5580eb11df60 2024-07-26T12:47:28.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.230+0000 7f4b81600640 1 -- 172.21.15.107:0/2042819180 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f4b6c042070 con 0x7f4b8410f330 2024-07-26T12:47:28.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.230+0000 7f4b88e00640 1 --2- 172.21.15.107:0/3264333318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4c3780 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009ca0 tx=0x7f4b7c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22c4875c9e46a20 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:28.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.230+0000 7f4b79600640 1 -- 172.21.15.107:0/3264333318 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c01e070 con 0x5580eb11df60 2024-07-26T12:47:28.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.230+0000 7f4b79600640 1 -- 172.21.15.107:0/3264333318 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b7c009e50 con 0x5580eb11df60 2024-07-26T12:47:28.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.230+0000 7f4b79600640 1 -- 172.21.15.107:0/3264333318 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c00ea10 con 0x5580eb11df60 2024-07-26T12:47:28.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.231+0000 7f4b919e7740 1 -- 172.21.15.107:0/3264333318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 msgr2=0x5580eb4c3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:28.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.231+0000 7f4b919e7740 1 --2- 172.21.15.107:0/3264333318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4c3780 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009ca0 tx=0x7f4b7c011810 comp rx=0 tx=0).stop 2024-07-26T12:47:28.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.231+0000 7f4b919e7740 1 -- 172.21.15.107:0/3264333318 shutdown_connections 2024-07-26T12:47:28.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.231+0000 7f4b919e7740 1 --2- 172.21.15.107:0/3264333318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0fd280 0x5580eb4c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.231+0000 7f4b919e7740 1 --2- 172.21.15.107:0/3264333318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4c3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.231+0000 7f4b919e7740 1 --2- 172.21.15.107:0/3264333318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4c3780 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.231+0000 7f4b81600640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b680b3c20 0x7f4b680b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.231+0000 7f4b919e7740 1 -- 172.21.15.107:0/3264333318 >> 172.21.15.107:0/3264333318 conn(0x5580eb112350 msgr2=0x5580eb4c1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:28.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.232+0000 7f4b919e7740 1 -- 172.21.15.107:0/3264333318 shutdown_connections 2024-07-26T12:47:28.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.232+0000 7f4b81600640 1 -- 172.21.15.107:0/2042819180 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4b6c104300 con 0x7f4b8410f330 2024-07-26T12:47:28.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.232+0000 7f4b88e00640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b680b3c20 0x7f4b680b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.232+0000 7f4b919e7740 1 -- 172.21.15.107:0/3264333318 wait complete. 2024-07-26T12:47:28.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.233+0000 7f4b919e7740 1 -- start start 2024-07-26T12:47:28.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.233+0000 7f4b88e00640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b680b3c20 0x7f4b680b6130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009ab0 tx=0x7f4b7c000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:28.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.233+0000 7f4b919e7740 1 Processor -- start 2024-07-26T12:47:28.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.233+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4cf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.233+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0fd280 0x5580eb4cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.233+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4db2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.233+0000 7f4b919e7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5580eb28c5f0 con 0x5580eb11df60 2024-07-26T12:47:28.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.233+0000 7f4b919e7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5580eb3c5540 con 0x5580eb0f50a0 2024-07-26T12:47:28.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.233+0000 7f4b919e7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5580eb4d1f60 con 0x5580eb0fd280 2024-07-26T12:47:28.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b919e7740 10 client.0 _ll_register_callbacks cb 0x5580eb3d4f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:47:28.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b83e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b88e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4db2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b7b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0fd280 0x5580eb4cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b83e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4cf030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:42514/0 (socket says 172.21.15.107:42514) 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b88e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4db2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43352/0 (socket says 172.21.15.107:43352) 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b83e00640 1 -- 172.21.15.107:0/4221394354 learned_addr learned my addr 172.21.15.107:0/4221394354 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b83e00640 1 -- 172.21.15.107:0/4221394354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0fd280 msgr2=0x5580eb4cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b83e00640 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0fd280 0x5580eb4cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b83e00640 1 -- 172.21.15.107:0/4221394354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 msgr2=0x5580eb4db2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b83e00640 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4db2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b83e00640 1 -- 172.21.15.107:0/4221394354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580eb189540 con 0x5580eb0f50a0 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b88e00640 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4db2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.234+0000 7f4b7b400640 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0fd280 0x5580eb4cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:47:28.238 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.235+0000 7f4b83e00640 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4cf030 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4b74011d10 tx=0x7f4b7400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cf7ab4ddc413062 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:28.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.235+0000 7f4b72000640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:47:28.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.235+0000 7f4b72000640 1 -- 172.21.15.107:0/4221394354 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74004b20 con 0x5580eb0f50a0 2024-07-26T12:47:28.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.235+0000 7f4b72000640 1 -- 172.21.15.107:0/4221394354 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4b74017050 con 0x5580eb0f50a0 2024-07-26T12:47:28.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.235+0000 7f4b919e7740 1 -- 172.21.15.107:0/4221394354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b7c009e50 con 0x5580eb0f50a0 2024-07-26T12:47:28.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.235+0000 7f4b72000640 1 -- 172.21.15.107:0/4221394354 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b74024050 con 0x5580eb0f50a0 2024-07-26T12:47:28.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.236+0000 7f4b919e7740 10 client.25235 fetch_fsmap learned FSMap version 28 2024-07-26T12:47:28.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.236+0000 7f4b919e7740 1 -- 172.21.15.107:0/4221394354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5580eb4ddec0 con 0x5580eb0f50a0 2024-07-26T12:47:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.236+0000 7f4b72000640 1 -- 172.21.15.107:0/4221394354 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b7c009e50 con 0x5580eb0f50a0 2024-07-26T12:47:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.236+0000 7f4b72000640 1 -- 172.21.15.107:0/4221394354 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f4b74019030 con 0x5580eb0f50a0 2024-07-26T12:47:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.236+0000 7f4b919e7740 10 client.25235 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:47:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.236+0000 7f4b919e7740 10 client.25235 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T12:47:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.236+0000 7f4b919e7740 20 client.25235 populate_metadata read hostname 'smithi107' 2024-07-26T12:47:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.237+0000 7f4b919e7740 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5580eb0dc8e0 0x5580eb4e19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.237+0000 7f4b919e7740 4 client.25235 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:47:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.237+0000 7f4b919e7740 1 -- 172.21.15.107:0/4221394354 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580eb43b8c0 con 0x5580eb0dc8e0 2024-07-26T12:47:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.237+0000 7f4b7b400640 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5580eb0dc8e0 0x5580eb4e19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.241 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.237+0000 7f4b7b400640 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5580eb0dc8e0 0x5580eb4e19c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=318171d50cab9ca6 server_cookie=311b8f65cc595b28 in_seq=0 out_seq=0 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.237+0000 7f4b72000640 10 client.25235 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.238+0000 7f4b72000640 1 -- 172.21.15.107:0/4221394354 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5580eb43b8c0 con 0x5580eb0dc8e0 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.238+0000 7f4b72000640 10 client.25235 handle_command_reply: tid=0 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.238+0000 7f4b919e7740 1 client.25235 shutdown 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.238+0000 7f4b919e7740 1 -- 172.21.15.107:0/4221394354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 msgr2=0x5580eb4cf030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.238+0000 7f4b919e7740 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4cf030 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4b74011d10 tx=0x7f4b7400f790 comp rx=0 tx=0).stop 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.239+0000 7f4b919e7740 1 -- 172.21.15.107:0/4221394354 shutdown_connections 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.239+0000 7f4b919e7740 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5580eb0dc8e0 0x5580eb4e19c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.239+0000 7f4b919e7740 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4db2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.239+0000 7f4b919e7740 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0fd280 0x5580eb4cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.239+0000 7f4b919e7740 1 --2- 172.21.15.107:0/4221394354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4cf030 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.239+0000 7f4b72000640 0 client.25235 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:47:28.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.239+0000 7f4b919e7740 1 -- 172.21.15.107:0/4221394354 >> 172.21.15.107:0/4221394354 conn(0x5580eb112350 msgr2=0x5580eb4c6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.239+0000 7f4b919e7740 1 -- 172.21.15.107:0/4221394354 shutdown_connections 2024-07-26T12:47:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.239+0000 7f4b919e7740 1 -- 172.21.15.107:0/4221394354 wait complete. 2024-07-26T12:47:28.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.239+0000 7f4b919e7740 20 client.25235 trim_cache size 0 max 16384 2024-07-26T12:47:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.258+0000 7f4b919e7740 1 Processor -- start 2024-07-26T12:47:28.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.258+0000 7f4b919e7740 1 -- start start 2024-07-26T12:47:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4e19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb0fd280 0x5580eb4ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb11df60 0x5580eb4c1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b919e7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5580eb4dfd00 con 0x5580eb0fd280 2024-07-26T12:47:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b919e7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5580eb4e03d0 con 0x5580eb0f50a0 2024-07-26T12:47:28.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b919e7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5580eb4e0aa0 con 0x5580eb11df60 2024-07-26T12:47:28.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b7b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb0fd280 0x5580eb4ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b88e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb11df60 0x5580eb4c1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b83e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4e19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b88e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb11df60 0x5580eb4c1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53528/0 (socket says 172.21.15.107:53528) 2024-07-26T12:47:28.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b88e00640 1 -- 172.21.15.107:0/4078501570 learned_addr learned my addr 172.21.15.107:0/4078501570 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:28.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.259+0000 7f4b7b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb0fd280 0x5580eb4ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43356/0 (socket says 172.21.15.107:43356) 2024-07-26T12:47:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.260+0000 7f4b7b400640 1 -- 172.21.15.107:0/4078501570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb11df60 msgr2=0x5580eb4c1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.260+0000 7f4b7b400640 1 --2- 172.21.15.107:0/4078501570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb11df60 0x5580eb4c1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.260+0000 7f4b7b400640 1 -- 172.21.15.107:0/4078501570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 msgr2=0x5580eb4e19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.260+0000 7f4b7b400640 1 --2- 172.21.15.107:0/4078501570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4e19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.260+0000 7f4b7b400640 1 -- 172.21.15.107:0/4078501570 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580eb4cdf90 con 0x5580eb0fd280 2024-07-26T12:47:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.260+0000 7f4b7b400640 1 --2- 172.21.15.107:0/4078501570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb0fd280 0x5580eb4ce520 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c101960 tx=0x7f4b6c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f145cb5275470f8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.260+0000 7f4b78c00640 1 -- 172.21.15.107:0/4078501570 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c049040 con 0x5580eb0fd280 2024-07-26T12:47:28.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.260+0000 7f4b78c00640 1 -- 172.21.15.107:0/4078501570 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b6c0cf3b0 con 0x5580eb0fd280 2024-07-26T12:47:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.260+0000 7f4b78c00640 1 -- 172.21.15.107:0/4078501570 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c101a70 con 0x5580eb0fd280 2024-07-26T12:47:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.261+0000 7f4b919e7740 1 -- 172.21.15.107:0/4078501570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb0fd280 msgr2=0x5580eb4ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:28.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.261+0000 7f4b919e7740 1 --2- 172.21.15.107:0/4078501570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb0fd280 0x5580eb4ce520 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c101960 tx=0x7f4b6c02a410 comp rx=0 tx=0).stop 2024-07-26T12:47:28.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.261+0000 7f4b919e7740 1 -- 172.21.15.107:0/4078501570 shutdown_connections 2024-07-26T12:47:28.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.261+0000 7f4b919e7740 1 --2- 172.21.15.107:0/4078501570 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb11df60 0x5580eb4c1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.261+0000 7f4b919e7740 1 --2- 172.21.15.107:0/4078501570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb0fd280 0x5580eb4ce520 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.261+0000 7f4b919e7740 1 --2- 172.21.15.107:0/4078501570 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0f50a0 0x5580eb4e19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.261+0000 7f4b919e7740 1 -- 172.21.15.107:0/4078501570 >> 172.21.15.107:0/4078501570 conn(0x5580eb112350 msgr2=0x5580eb4c0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:28.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.262+0000 7f4b919e7740 1 -- 172.21.15.107:0/4078501570 shutdown_connections 2024-07-26T12:47:28.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.262+0000 7f4b919e7740 1 -- 172.21.15.107:0/4078501570 wait complete. 2024-07-26T12:47:28.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.262+0000 7f4b919e7740 1 -- start start 2024-07-26T12:47:28.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.262+0000 7f4b919e7740 1 Processor -- start 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0f50a0 0x5580eb4ea3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0fd280 0x5580eb4e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b919e7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4e6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b919e7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5580eb4e1840 con 0x5580eb11df60 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b919e7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5580eb4e1170 con 0x5580eb0fd280 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b919e7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5580eb4e1f10 con 0x5580eb0f50a0 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b919e7740 10 client.0 _ll_register_callbacks cb 0x7f4b74011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b7b400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0fd280 0x5580eb4e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b7b400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0fd280 0x5580eb4e5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:42538/0 (socket says 172.21.15.107:42538) 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b83e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0f50a0 0x5580eb4ea3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b7b400640 1 -- 172.21.15.107:0/2779540397 learned_addr learned my addr 172.21.15.107:0/2779540397 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b88e00640 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4e6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b7b400640 1 -- 172.21.15.107:0/2779540397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0f50a0 msgr2=0x5580eb4ea3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.263+0000 7f4b7b400640 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0f50a0 0x5580eb4ea3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.264+0000 7f4b7b400640 1 -- 172.21.15.107:0/2779540397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 msgr2=0x5580eb4e6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.264+0000 7f4b7b400640 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4e6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.264+0000 7f4b7b400640 1 -- 172.21.15.107:0/2779540397 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580eb4e6ac0 con 0x5580eb0fd280 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.264+0000 7f4b83e00640 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0f50a0 0x5580eb4ea3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.264+0000 7f4b7b400640 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0fd280 0x5580eb4e5df0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0c6ee0 tx=0x7f4b6c0cafe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cef2ac9b007c9a92 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.264+0000 7f4b72000640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.264+0000 7f4b72000640 1 -- 172.21.15.107:0/2779540397 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c049030 con 0x5580eb0fd280 2024-07-26T12:47:28.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.264+0000 7f4b72000640 1 -- 172.21.15.107:0/2779540397 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4b6c03b460 con 0x5580eb0fd280 2024-07-26T12:47:28.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.264+0000 7f4b919e7740 1 -- 172.21.15.107:0/2779540397 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b6c0cf3b0 con 0x5580eb0fd280 2024-07-26T12:47:28.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.264+0000 7f4b72000640 1 -- 172.21.15.107:0/2779540397 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4b6c031d00 con 0x5580eb0fd280 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.265+0000 7f4b919e7740 10 client.25241 fetch_fsmap learned FSMap version 28 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.265+0000 7f4b919e7740 1 -- 172.21.15.107:0/2779540397 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5580eb549ba0 con 0x5580eb0fd280 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.265+0000 7f4b72000640 1 -- 172.21.15.107:0/2779540397 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b6c0cf3b0 con 0x5580eb0fd280 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.265+0000 7f4b72000640 1 -- 172.21.15.107:0/2779540397 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7f4b6c04c060 con 0x5580eb0fd280 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.265+0000 7f4b919e7740 10 client.25241 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.265+0000 7f4b919e7740 10 client.25241 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.265+0000 7f4b919e7740 20 client.25241 populate_metadata read hostname 'smithi107' 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.265+0000 7f4b919e7740 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5580eb0b1c90 0x5580eb54d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.265+0000 7f4b919e7740 4 client.25241 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.265+0000 7f4b919e7740 1 -- 172.21.15.107:0/2779540397 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "status"}) -- 0x5580eb437360 con 0x5580eb0b1c90 2024-07-26T12:47:28.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.266+0000 7f4b83e00640 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5580eb0b1c90 0x5580eb54d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.266+0000 7f4b83e00640 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5580eb0b1c90 0x5580eb54d2b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcac9272d8d5fcf7 server_cookie=6c71e590a16704c0 in_seq=0 out_seq=0 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.266+0000 7f4b72000640 10 client.25241 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.266+0000 7f4b72000640 1 -- 172.21.15.107:0/2779540397 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5580eb437360 con 0x5580eb0b1c90 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.266+0000 7f4b72000640 10 client.25241 handle_command_reply: tid=0 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.266+0000 7f4b919e7740 1 client.25241 shutdown 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 1 -- 172.21.15.107:0/2779540397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0fd280 msgr2=0x5580eb4e5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0fd280 0x5580eb4e5df0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0c6ee0 tx=0x7f4b6c0cafe0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 1 -- 172.21.15.107:0/2779540397 shutdown_connections 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5580eb0b1c90 0x5580eb54d2b0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580eb11df60 0x5580eb4e6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5580eb0fd280 0x5580eb4e5df0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 1 --2- 172.21.15.107:0/2779540397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5580eb0f50a0 0x5580eb4ea3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b72000640 0 client.25241 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:47:28.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 1 -- 172.21.15.107:0/2779540397 >> 172.21.15.107:0/2779540397 conn(0x5580eb112350 msgr2=0x5580eb4c0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:28.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 1 -- 172.21.15.107:0/2779540397 shutdown_connections 2024-07-26T12:47:28.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 1 -- 172.21.15.107:0/2779540397 wait complete. 2024-07-26T12:47:28.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.267+0000 7f4b919e7740 20 client.25241 trim_cache size 0 max 16384 2024-07-26T12:47:28.272 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:47:28.272 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:47:28.272 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2024-07-26T12:47:28.272 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24716, 2024-07-26T12:47:28.272 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:47:28.272 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:47:28.273 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:47:28.273 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 252.07048208800001, 2024-07-26T12:47:28.273 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-07-26T12:47:28.273 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 90, 2024-07-26T12:47:28.273 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T12:47:28.273 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 271.270208669 2024-07-26T12:47:28.273 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:47:28.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 -- 172.21.15.107:0/2042819180 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b680b3c20 msgr2=0x7f4b680b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:28.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b680b3c20 0x7f4b680b6130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009ab0 tx=0x7f4b7c000bc0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 -- 172.21.15.107:0/2042819180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b8410f330 msgr2=0x7f4b841a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:28.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b8410f330 0x7f4b841a08e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c005ab0 tx=0x7f4b6c003b90 comp rx=0 tx=0).stop 2024-07-26T12:47:28.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 -- 172.21.15.107:0/2042819180 shutdown_connections 2024-07-26T12:47:28.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f4b680b3c20 0x7f4b680b6130 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4b8410fc80 0x7f4b841a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4b8410f330 0x7f4b841a08e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 --2- 172.21.15.107:0/2042819180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b841099f0 0x7f4b841a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:28.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 -- 172.21.15.107:0/2042819180 >> 172.21.15.107:0/2042819180 conn(0x7f4b84078ff0 msgr2=0x7f4b840797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:28.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 -- 172.21.15.107:0/2042819180 shutdown_connections 2024-07-26T12:47:28.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:28.268+0000 7f4b89800640 1 -- 172.21.15.107:0/2042819180 wait complete. 2024-07-26T12:47:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:27 smithi132 ceph-mon[37580]: pgmap v434: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:47:29.265 INFO:tasks.cephadm:Setup complete, yielding 2024-07-26T12:47:29.266 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T12:47:29.276 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-07-26T12:47:29.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch status' 2024-07-26T12:47:29.714 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:30.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:29 smithi107 ceph-mon[32659]: pgmap v435: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:30.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:29 smithi132 ceph-mon[37580]: pgmap v435: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:29 smithi112 ceph-mon[37501]: pgmap v435: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:31 smithi107 ceph-mon[32659]: pgmap v436: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:32.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.083+0000 7f223ec00640 1 -- 172.21.15.107:0/1028303006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22381055d0 msgr2=0x7f22381059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:32.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.083+0000 7f223ec00640 1 --2- 172.21.15.107:0/1028303006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22381055d0 0x7f22381059e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f222c009900 tx=0x7f222c02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.083+0000 7f223ec00640 1 -- 172.21.15.107:0/1028303006 shutdown_connections 2024-07-26T12:47:32.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.083+0000 7f223ec00640 1 --2- 172.21.15.107:0/1028303006 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f223810b860 0x7f2238112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.083+0000 7f223ec00640 1 --2- 172.21.15.107:0/1028303006 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f223810af10 0x7f223810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.083+0000 7f223ec00640 1 --2- 172.21.15.107:0/1028303006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f22381055d0 0x7f22381059e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.083+0000 7f223ec00640 1 -- 172.21.15.107:0/1028303006 >> 172.21.15.107:0/1028303006 conn(0x7f2238101040 msgr2=0x7f2238103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:32.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.083+0000 7f223ec00640 1 -- 172.21.15.107:0/1028303006 shutdown_connections 2024-07-26T12:47:32.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.083+0000 7f223ec00640 1 -- 172.21.15.107:0/1028303006 wait complete. 2024-07-26T12:47:32.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.084+0000 7f223ec00640 1 Processor -- start 2024-07-26T12:47:32.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.084+0000 7f223ec00640 1 -- start start 2024-07-26T12:47:32.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223ec00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22381055d0 0x7f223819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:32.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223ec00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f223810af10 0x7f223819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:32.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223ec00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f223810b860 0x7f22381a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:32.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223ec00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2238115530 con 0x7f223810af10 2024-07-26T12:47:32.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223ec00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f22381153b0 con 0x7f223810b860 2024-07-26T12:47:32.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223ec00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f22381156b0 con 0x7f22381055d0 2024-07-26T12:47:32.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223ce00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f223810af10 0x7f223819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223ce00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f223810af10 0x7f223819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:43372/0 (socket says 172.21.15.107:43372) 2024-07-26T12:47:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223ce00640 1 -- 172.21.15.107:0/778040393 learned_addr learned my addr 172.21.15.107:0/778040393 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223d800640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22381055d0 0x7f223819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:32.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.085+0000 7f223e200640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f223810b860 0x7f22381a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f223ce00640 1 -- 172.21.15.107:0/778040393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22381055d0 msgr2=0x7f223819bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f223ce00640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22381055d0 0x7f223819bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f223ce00640 1 -- 172.21.15.107:0/778040393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f223810b860 msgr2=0x7f22381a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f223ce00640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f223810b860 0x7f22381a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f223ce00640 1 -- 172.21.15.107:0/778040393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22381a1e90 con 0x7f223810af10 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f223ce00640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f223810af10 0x7f223819c4c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f222400e9f0 tx=0x7f222400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f2236000640 1 -- 172.21.15.107:0/778040393 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222400ccc0 con 0x7f223810af10 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f2236000640 1 -- 172.21.15.107:0/778040393 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22240108e0 con 0x7f223810af10 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f2236000640 1 -- 172.21.15.107:0/778040393 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224018860 con 0x7f223810af10 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f223ec00640 1 -- 172.21.15.107:0/778040393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22381a2180 con 0x7f223810af10 2024-07-26T12:47:32.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.086+0000 7f223ec00640 1 -- 172.21.15.107:0/778040393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22381a88e0 con 0x7f223810af10 2024-07-26T12:47:32.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.088+0000 7f223ec00640 1 -- 172.21.15.107:0/778040393 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22381059e0 con 0x7f223810af10 2024-07-26T12:47:32.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.091+0000 7f2236000640 1 -- 172.21.15.107:0/778040393 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f2224010430 con 0x7f223810af10 2024-07-26T12:47:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.093+0000 7f2236000640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f22200b3ca0 0x7f22200b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:32.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.093+0000 7f2236000640 1 -- 172.21.15.107:0/778040393 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f22240e4860 con 0x7f223810af10 2024-07-26T12:47:32.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.093+0000 7f223d800640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f22200b3ca0 0x7f22200b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:32.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.093+0000 7f2236000640 1 -- 172.21.15.107:0/778040393 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f22240ab360 con 0x7f223810af10 2024-07-26T12:47:32.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.094+0000 7f223d800640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f22200b3ca0 0x7f22200b61b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f222c031440 tx=0x7f222c009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:31 smithi112 ceph-mon[37501]: pgmap v436: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:31 smithi132 ceph-mon[37580]: pgmap v436: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:32.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.316+0000 7f223ec00640 1 -- 172.21.15.107:0/778040393 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f2238063230 con 0x7f22200b3ca0 2024-07-26T12:47:32.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.318+0000 7f2236000640 1 -- 172.21.15.107:0/778040393 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f2238063230 con 0x7f22200b3ca0 2024-07-26T12:47:32.319 INFO:teuthology.orchestra.run.smithi107.stdout:Backend: cephadm 2024-07-26T12:47:32.319 INFO:teuthology.orchestra.run.smithi107.stdout:Available: Yes 2024-07-26T12:47:32.319 INFO:teuthology.orchestra.run.smithi107.stdout:Paused: No 2024-07-26T12:47:32.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.321+0000 7f223ec00640 1 -- 172.21.15.107:0/778040393 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f22200b3ca0 msgr2=0x7f22200b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:32.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.321+0000 7f223ec00640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f22200b3ca0 0x7f22200b61b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f222c031440 tx=0x7f222c009560 comp rx=0 tx=0).stop 2024-07-26T12:47:32.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.321+0000 7f223ec00640 1 -- 172.21.15.107:0/778040393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f223810af10 msgr2=0x7f223819c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:32.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.321+0000 7f223ec00640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f223810af10 0x7f223819c4c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f222400e9f0 tx=0x7f222400eeb0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.321+0000 7f223ec00640 1 -- 172.21.15.107:0/778040393 shutdown_connections 2024-07-26T12:47:32.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.321+0000 7f223ec00640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f22200b3ca0 0x7f22200b61b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.321+0000 7f223ec00640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f223810b860 0x7f22381a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.321+0000 7f223ec00640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f223810af10 0x7f223819c4c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.321+0000 7f223ec00640 1 --2- 172.21.15.107:0/778040393 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f22381055d0 0x7f223819bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:32.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.321+0000 7f223ec00640 1 -- 172.21.15.107:0/778040393 >> 172.21.15.107:0/778040393 conn(0x7f2238101040 msgr2=0x7f2238103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:32.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.322+0000 7f223ec00640 1 -- 172.21.15.107:0/778040393 shutdown_connections 2024-07-26T12:47:32.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:32.322+0000 7f223ec00640 1 -- 172.21.15.107:0/778040393 wait complete. 2024-07-26T12:47:33.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch ps' 2024-07-26T12:47:33.726 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:34.131 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:33 smithi107 ceph-mon[32659]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:34.132 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:33 smithi107 ceph-mon[32659]: pgmap v437: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:33 smithi112 ceph-mon[37501]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:33 smithi112 ceph-mon[37501]: pgmap v437: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:33 smithi132 ceph-mon[37580]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:33 smithi132 ceph-mon[37580]: pgmap v437: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.167+0000 7f01a7200640 1 -- 172.21.15.107:0/3915430295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01a01057b0 msgr2=0x7f01a0105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.167+0000 7f01a7200640 1 --2- 172.21.15.107:0/3915430295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01a01057b0 0x7f01a0105bc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0190009900 tx=0x7f019002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:47:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.168+0000 7f01a7200640 1 -- 172.21.15.107:0/3915430295 shutdown_connections 2024-07-26T12:47:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.168+0000 7f01a7200640 1 --2- 172.21.15.107:0/3915430295 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01a010ba40 0x7f01a0112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:36.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.168+0000 7f01a7200640 1 --2- 172.21.15.107:0/3915430295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a010b0f0 0x7f01a010b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.168+0000 7f01a7200640 1 --2- 172.21.15.107:0/3915430295 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01a01057b0 0x7f01a0105bc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.168+0000 7f01a7200640 1 -- 172.21.15.107:0/3915430295 >> 172.21.15.107:0/3915430295 conn(0x7f01a0101240 msgr2=0x7f01a0103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.168+0000 7f01a7200640 1 -- 172.21.15.107:0/3915430295 shutdown_connections 2024-07-26T12:47:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.168+0000 7f01a7200640 1 -- 172.21.15.107:0/3915430295 wait complete. 2024-07-26T12:47:36.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:35 smithi107 ceph-mon[32659]: pgmap v438: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.169+0000 7f01a7200640 1 Processor -- start 2024-07-26T12:47:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.169+0000 7f01a7200640 1 -- start start 2024-07-26T12:47:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f01a7200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01a01057b0 0x7f01a019c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f01a7200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01a010b0f0 0x7f01a019c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f01a7200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a010ba40 0x7f01a01a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f01a7200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f01a0115740 con 0x7f01a010ba40 2024-07-26T12:47:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f01a7200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f01a01155c0 con 0x7f01a01057b0 2024-07-26T12:47:36.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f01a7200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f01a01158c0 con 0x7f01a010b0f0 2024-07-26T12:47:36.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f019fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01a01057b0 0x7f01a019c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:36.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f01a5200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a010ba40 0x7f01a01a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:36.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f0197400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01a010b0f0 0x7f01a019c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:36.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f019fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01a01057b0 0x7f01a019c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:59392/0 (socket says 172.21.15.107:59392) 2024-07-26T12:47:36.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.170+0000 7f019fe00640 1 -- 172.21.15.107:0/3260750201 learned_addr learned my addr 172.21.15.107:0/3260750201 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.171+0000 7f0197400640 1 -- 172.21.15.107:0/3260750201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01a01057b0 msgr2=0x7f01a019c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.171+0000 7f0197400640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01a01057b0 0x7f01a019c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.171+0000 7f0197400640 1 -- 172.21.15.107:0/3260750201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a010ba40 msgr2=0x7f01a01a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.171+0000 7f0197400640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a010ba40 0x7f01a01a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.171+0000 7f0197400640 1 -- 172.21.15.107:0/3260750201 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a01a20e0 con 0x7f01a010b0f0 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.171+0000 7f019fe00640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01a01057b0 0x7f01a019c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.171+0000 7f0197400640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01a010b0f0 0x7f01a019c710 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f01a006c4d0 tx=0x7f018800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.172+0000 7f019d600640 1 -- 172.21.15.107:0/3260750201 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018800cd00 con 0x7f01a010b0f0 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.172+0000 7f019d600640 1 -- 172.21.15.107:0/3260750201 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01880035f0 con 0x7f01a010b0f0 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.172+0000 7f01a7200640 1 -- 172.21.15.107:0/3260750201 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a01a23d0 con 0x7f01a010b0f0 2024-07-26T12:47:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.172+0000 7f019d600640 1 -- 172.21.15.107:0/3260750201 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188010690 con 0x7f01a010b0f0 2024-07-26T12:47:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.172+0000 7f01a7200640 1 -- 172.21.15.107:0/3260750201 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a01a8b30 con 0x7f01a010b0f0 2024-07-26T12:47:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.173+0000 7f01a7200640 1 -- 172.21.15.107:0/3260750201 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a0105bc0 con 0x7f01a010b0f0 2024-07-26T12:47:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.176+0000 7f019d600640 1 -- 172.21.15.107:0/3260750201 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f01880037a0 con 0x7f01a010b0f0 2024-07-26T12:47:36.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.177+0000 7f019d600640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f01840b3c20 0x7f01840b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:36.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.177+0000 7f019d600640 1 -- 172.21.15.107:0/3260750201 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f01880e44e0 con 0x7f01a010b0f0 2024-07-26T12:47:36.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.178+0000 7f019fe00640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f01840b3c20 0x7f01840b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:36.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.178+0000 7f019fe00640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f01840b3c20 0x7f01840b6130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f01900096c0 tx=0x7f0190005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:36.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.180+0000 7f019d600640 1 -- 172.21.15.107:0/3260750201 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f0188055600 con 0x7f01a010b0f0 2024-07-26T12:47:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:35 smithi112 ceph-mon[37501]: pgmap v438: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:35 smithi132 ceph-mon[37580]: pgmap v438: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:36.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.391+0000 7f01a7200640 1 -- 172.21.15.107:0/3260750201 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f01a0063230 con 0x7f01840b3c20 2024-07-26T12:47:36.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.426+0000 7f019d600640 1 -- 172.21.15.107:0/3260750201 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f01a0063230 con 0x7f01840b3c20 2024-07-26T12:47:36.428 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-07-26T12:47:36.428 INFO:teuthology.orchestra.run.smithi107.stdout:mds.a smithi107 running (4m) 2m ago 4m 80.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa a91deb8200a2 2024-07-26T12:47:36.428 INFO:teuthology.orchestra.run.smithi107.stdout:mds.b smithi112 running (4m) 2m ago 4m 94.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 79e81ecd91c8 2024-07-26T12:47:36.428 INFO:teuthology.orchestra.run.smithi107.stdout:mds.c smithi132 running (4m) 2m ago 4m 85.9M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 1ee9887ee094 2024-07-26T12:47:36.428 INFO:teuthology.orchestra.run.smithi107.stdout:mds.d smithi107 running (5m) 2m ago 5m 80.9M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa dab1f4780889 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mds.e smithi112 running (5m) 2m ago 5m 84.2M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa cda99e4a7010 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mds.f smithi132 running (4m) 2m ago 4m 87.8M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 065e99edd8b0 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mds.g smithi107 running (4m) 2m ago 4m 84.2M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa f40a547edc8e 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mds.h smithi112 running (4m) 2m ago 4m 84.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 93b16afe7021 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mds.i smithi132 running (4m) 2m ago 4m 85.1M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 92dd05271193 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mds.j smithi107 running (5m) 2m ago 5m 85.1M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa c73824f5bc58 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mds.k smithi112 running (5m) 2m ago 5m 83.8M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa fe3a57bd22b0 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mds.l smithi132 running (5m) 2m ago 5m 83.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa f7e4d4a9d544 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.x smithi107 *:9283,8765 running (14m) 2m ago 14m 585M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa eccaf212cc64 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.y smithi112 *:8443,8765 running (11m) 2m ago 11m 524M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 5958d5a61995 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.z smithi132 *:8443,8765 running (11m) 2m ago 11m 526M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 4231fdaf7e41 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi107 running (14m) 2m ago 14m 117M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 4bc7dea60603 2024-07-26T12:47:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi112 running (12m) 2m ago 12m 112M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa a89a40fda6ea 2024-07-26T12:47:36.430 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi132 running (12m) 2m ago 12m 115M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa b4799022bd95 2024-07-26T12:47:36.430 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi107 running (10m) 2m ago 10m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 4adfa3a7d01f 2024-07-26T12:47:36.430 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi112 running (10m) 2m ago 10m 122M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa bfbc17730350 2024-07-26T12:47:36.430 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi132 running (9m) 2m ago 9m 118M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 7de6a5bb9547 2024-07-26T12:47:36.430 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi107 running (9m) 2m ago 9m 118M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 20cd420c65d7 2024-07-26T12:47:36.430 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi112 running (8m) 2m ago 8m 111M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 44d24862ebbd 2024-07-26T12:47:36.430 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi132 running (8m) 2m ago 8m 120M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa cdb94381a847 2024-07-26T12:47:36.430 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi107 running (7m) 2m ago 7m 109M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 719e69ddcb47 2024-07-26T12:47:36.430 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi112 running (7m) 2m ago 7m 120M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa e172de62524e 2024-07-26T12:47:36.430 INFO:teuthology.orchestra.run.smithi107.stdout:osd.8 smithi132 running (7m) 2m ago 7m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 05e7cbaca21e 2024-07-26T12:47:36.431 INFO:teuthology.orchestra.run.smithi107.stdout:osd.9 smithi107 running (6m) 2m ago 6m 111M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 254784b550fc 2024-07-26T12:47:36.431 INFO:teuthology.orchestra.run.smithi107.stdout:osd.10 smithi112 running (6m) 2m ago 6m 120M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 50dc5236e280 2024-07-26T12:47:36.431 INFO:teuthology.orchestra.run.smithi107.stdout:osd.11 smithi132 running (5m) 2m ago 5m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa c6c410d2d0b1 2024-07-26T12:47:36.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.430+0000 7f01a7200640 1 -- 172.21.15.107:0/3260750201 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f01840b3c20 msgr2=0x7f01840b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:36.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.430+0000 7f01a7200640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f01840b3c20 0x7f01840b6130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f01900096c0 tx=0x7f0190005a20 comp rx=0 tx=0).stop 2024-07-26T12:47:36.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.430+0000 7f01a7200640 1 -- 172.21.15.107:0/3260750201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01a010b0f0 msgr2=0x7f01a019c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:36.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.430+0000 7f01a7200640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01a010b0f0 0x7f01a019c710 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f01a006c4d0 tx=0x7f018800ee50 comp rx=0 tx=0).stop 2024-07-26T12:47:36.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.430+0000 7f01a7200640 1 -- 172.21.15.107:0/3260750201 shutdown_connections 2024-07-26T12:47:36.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.430+0000 7f01a7200640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f01840b3c20 0x7f01840b6130 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:36.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.430+0000 7f01a7200640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a010ba40 0x7f01a01a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:36.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.430+0000 7f01a7200640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01a010b0f0 0x7f01a019c710 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:36.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.430+0000 7f01a7200640 1 --2- 172.21.15.107:0/3260750201 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f01a01057b0 0x7f01a019c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:36.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.430+0000 7f01a7200640 1 -- 172.21.15.107:0/3260750201 >> 172.21.15.107:0/3260750201 conn(0x7f01a0101240 msgr2=0x7f01a0103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:36.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.431+0000 7f01a7200640 1 -- 172.21.15.107:0/3260750201 shutdown_connections 2024-07-26T12:47:36.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:36.431+0000 7f01a7200640 1 -- 172.21.15.107:0/3260750201 wait complete. 2024-07-26T12:47:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:36 smithi112 ceph-mon[37501]: from='client.25261 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:36 smithi112 ceph-mon[37501]: pgmap v439: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:37.213 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:36 smithi107 ceph-mon[32659]: from='client.25261 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:37.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:36 smithi107 ceph-mon[32659]: pgmap v439: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:37.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch ls' 2024-07-26T12:47:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:36 smithi132 ceph-mon[37580]: from='client.25261 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:36 smithi132 ceph-mon[37580]: pgmap v439: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:37.745 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:40.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:39 smithi107 ceph-mon[32659]: pgmap v440: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:40.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.152+0000 7fb133600640 1 -- 172.21.15.107:0/2997032240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb12c077cc0 msgr2=0x7fb12c113c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:40.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.152+0000 7fb133600640 1 --2- 172.21.15.107:0/2997032240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb12c077cc0 0x7fb12c113c60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb118009860 tx=0x7fb11802a340 comp rx=0 tx=0).stop 2024-07-26T12:47:40.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.153+0000 7fb133600640 1 -- 172.21.15.107:0/2997032240 shutdown_connections 2024-07-26T12:47:40.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.153+0000 7fb133600640 1 --2- 172.21.15.107:0/2997032240 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb12c1141a0 0x7fb12c1165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:40.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.153+0000 7fb133600640 1 --2- 172.21.15.107:0/2997032240 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb12c077cc0 0x7fb12c113c60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.153+0000 7fb133600640 1 --2- 172.21.15.107:0/2997032240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c077370 0x7fb12c077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.153+0000 7fb133600640 1 -- 172.21.15.107:0/2997032240 >> 172.21.15.107:0/2997032240 conn(0x7fb12c1011f0 msgr2=0x7fb12c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.153+0000 7fb133600640 1 -- 172.21.15.107:0/2997032240 shutdown_connections 2024-07-26T12:47:40.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.153+0000 7fb133600640 1 -- 172.21.15.107:0/2997032240 wait complete. 2024-07-26T12:47:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.154+0000 7fb133600640 1 Processor -- start 2024-07-26T12:47:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.154+0000 7fb133600640 1 -- start start 2024-07-26T12:47:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb133600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb12c077370 0x7fb12c1a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb133600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb12c077cc0 0x7fb12c1a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb133600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c1141a0 0x7fb12c1a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb133600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c119b00 con 0x7fb12c1141a0 2024-07-26T12:47:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb133600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c119980 con 0x7fb12c077370 2024-07-26T12:47:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb133600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c119c80 con 0x7fb12c077cc0 2024-07-26T12:47:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb130c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb12c077370 0x7fb12c1a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb131600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c1141a0 0x7fb12c1a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb12be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb12c077cc0 0x7fb12c1a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb130c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb12c077370 0x7fb12c1a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:59422/0 (socket says 172.21.15.107:59422) 2024-07-26T12:47:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.155+0000 7fb130c00640 1 -- 172.21.15.107:0/524881509 learned_addr learned my addr 172.21.15.107:0/524881509 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:40.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb130c00640 1 -- 172.21.15.107:0/524881509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb12c077cc0 msgr2=0x7fb12c1a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:40.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb130c00640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb12c077cc0 0x7fb12c1a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:40.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb130c00640 1 -- 172.21.15.107:0/524881509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c1141a0 msgr2=0x7fb12c1a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:40.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb130c00640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c1141a0 0x7fb12c1a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:40.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb130c00640 1 -- 172.21.15.107:0/524881509 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12c1a6440 con 0x7fb12c077370 2024-07-26T12:47:40.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb131600640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c1141a0 0x7fb12c1a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:47:40.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb130c00640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb12c077370 0x7fb12c1a04d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb12000e9c0 tx=0x7fb12000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:40.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb129600640 1 -- 172.21.15.107:0/524881509 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12000cde0 con 0x7fb12c077370 2024-07-26T12:47:40.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb133600640 1 -- 172.21.15.107:0/524881509 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12c1a6730 con 0x7fb12c077370 2024-07-26T12:47:40.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb129600640 1 -- 172.21.15.107:0/524881509 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1200035f0 con 0x7fb12c077370 2024-07-26T12:47:40.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb129600640 1 -- 172.21.15.107:0/524881509 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb120010690 con 0x7fb12c077370 2024-07-26T12:47:40.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.156+0000 7fb133600640 1 -- 172.21.15.107:0/524881509 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb12c1ace90 con 0x7fb12c077370 2024-07-26T12:47:40.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.158+0000 7fb133600640 1 -- 172.21.15.107:0/524881509 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb12c077780 con 0x7fb12c077370 2024-07-26T12:47:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.160+0000 7fb129600640 1 -- 172.21.15.107:0/524881509 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fb120004000 con 0x7fb12c077370 2024-07-26T12:47:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.161+0000 7fb129600640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1140b3ca0 0x7fb1140b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:40.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.162+0000 7fb129600640 1 -- 172.21.15.107:0/524881509 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb1200e44a0 con 0x7fb12c077370 2024-07-26T12:47:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.162+0000 7fb12be00640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1140b3ca0 0x7fb1140b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.163+0000 7fb12be00640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1140b3ca0 0x7fb1140b61b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb118005940 tx=0x7fb1180058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:40.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.163+0000 7fb129600640 1 -- 172.21.15.107:0/524881509 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb1200ab680 con 0x7fb12c077370 2024-07-26T12:47:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:39 smithi112 ceph-mon[37501]: pgmap v440: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:39 smithi132 ceph-mon[37580]: pgmap v440: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:40.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.371+0000 7fb133600640 1 -- 172.21.15.107:0/524881509 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fb12c063230 con 0x7fb1140b3ca0 2024-07-26T12:47:40.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.386+0000 7fb129600640 1 -- 172.21.15.107:0/524881509 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fb12c063230 con 0x7fb1140b3ca0 2024-07-26T12:47:40.387 INFO:teuthology.orchestra.run.smithi107.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-07-26T12:47:40.387 INFO:teuthology.orchestra.run.smithi107.stdout:mds.all 12/12 2m ago 5m smithi107=a;smithi107=d;smithi107=g;smithi107=j;smithi112=b;smithi112=e;smithi112=h;smithi112=k;smithi132=c;smithi132=f;smithi132=i;smithi132=l;count:12 2024-07-26T12:47:40.388 INFO:teuthology.orchestra.run.smithi107.stdout:mgr 3/3 2m ago 12m smithi107=x;smithi112=y;smithi132=z;count:3 2024-07-26T12:47:40.388 INFO:teuthology.orchestra.run.smithi107.stdout:mon 3/3 2m ago 12m smithi107:172.21.15.107=a;smithi112:172.21.15.112=b;smithi132:172.21.15.132=c;count:3 2024-07-26T12:47:40.388 INFO:teuthology.orchestra.run.smithi107.stdout:osd 12 2m ago - 2024-07-26T12:47:40.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.389+0000 7fb133600640 1 -- 172.21.15.107:0/524881509 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1140b3ca0 msgr2=0x7fb1140b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.389+0000 7fb133600640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1140b3ca0 0x7fb1140b61b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb118005940 tx=0x7fb1180058d0 comp rx=0 tx=0).stop 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.389+0000 7fb133600640 1 -- 172.21.15.107:0/524881509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb12c077370 msgr2=0x7fb12c1a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.389+0000 7fb133600640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb12c077370 0x7fb12c1a04d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb12000e9c0 tx=0x7fb12000ee80 comp rx=0 tx=0).stop 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.389+0000 7fb133600640 1 -- 172.21.15.107:0/524881509 shutdown_connections 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.389+0000 7fb133600640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb1140b3ca0 0x7fb1140b61b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.389+0000 7fb133600640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb12c1141a0 0x7fb12c1a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.389+0000 7fb133600640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb12c077cc0 0x7fb12c1a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.389+0000 7fb133600640 1 --2- 172.21.15.107:0/524881509 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb12c077370 0x7fb12c1a04d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.389+0000 7fb133600640 1 -- 172.21.15.107:0/524881509 >> 172.21.15.107:0/524881509 conn(0x7fb12c1011f0 msgr2=0x7fb12c103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.390+0000 7fb133600640 1 -- 172.21.15.107:0/524881509 shutdown_connections 2024-07-26T12:47:40.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:40.390+0000 7fb133600640 1 -- 172.21.15.107:0/524881509 wait complete. 2024-07-26T12:47:41.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch host ls' 2024-07-26T12:47:41.831 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:42.172 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:41 smithi107 ceph-mon[32659]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:42.172 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:41 smithi107 ceph-mon[32659]: pgmap v441: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:41 smithi112 ceph-mon[37501]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:41 smithi112 ceph-mon[37501]: pgmap v441: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:41 smithi132 ceph-mon[37580]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:41 smithi132 ceph-mon[37580]: pgmap v441: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:43.968 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:43 smithi107 ceph-mon[32659]: pgmap v442: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.166+0000 7f8e6ca00640 1 -- 172.21.15.107:0/1154610523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e6810b880 msgr2=0x7f8e681120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.166+0000 7f8e6ca00640 1 --2- 172.21.15.107:0/1154610523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e6810b880 0x7f8e681120b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009ca0 tx=0x7f8e6002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:47:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.166+0000 7f8e6ca00640 1 -- 172.21.15.107:0/1154610523 shutdown_connections 2024-07-26T12:47:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.166+0000 7f8e6ca00640 1 --2- 172.21.15.107:0/1154610523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e6810b880 0x7f8e681120b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.166+0000 7f8e6ca00640 1 --2- 172.21.15.107:0/1154610523 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e6810af30 0x7f8e6810b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.166+0000 7f8e6ca00640 1 --2- 172.21.15.107:0/1154610523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e681055f0 0x7f8e68105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.166+0000 7f8e6ca00640 1 -- 172.21.15.107:0/1154610523 >> 172.21.15.107:0/1154610523 conn(0x7f8e68101040 msgr2=0x7f8e68103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.166+0000 7f8e6ca00640 1 -- 172.21.15.107:0/1154610523 shutdown_connections 2024-07-26T12:47:44.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.166+0000 7f8e6ca00640 1 -- 172.21.15.107:0/1154610523 wait complete. 2024-07-26T12:47:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.167+0000 7f8e6ca00640 1 Processor -- start 2024-07-26T12:47:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.167+0000 7f8e6ca00640 1 -- start start 2024-07-26T12:47:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.167+0000 7f8e6ca00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e681055f0 0x7f8e6819bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.167+0000 7f8e6ca00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e6810af30 0x7f8e6819c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.167+0000 7f8e6ca00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e6810b880 0x7f8e681a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.167+0000 7f8e6ca00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8e681155a0 con 0x7f8e6810af30 2024-07-26T12:47:44.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.167+0000 7f8e6ca00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68115420 con 0x7f8e6810b880 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.167+0000 7f8e6ca00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68115720 con 0x7f8e681055f0 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.168+0000 7f8e5de00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e6810af30 0x7f8e6819c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.168+0000 7f8e5de00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e6810af30 0x7f8e6819c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58532/0 (socket says 172.21.15.107:58532) 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.168+0000 7f8e5de00640 1 -- 172.21.15.107:0/3093971876 learned_addr learned my addr 172.21.15.107:0/3093971876 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.168+0000 7f8e67200640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e6810b880 0x7f8e681a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.168+0000 7f8e66800640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e681055f0 0x7f8e6819bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.168+0000 7f8e5de00640 1 -- 172.21.15.107:0/3093971876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e681055f0 msgr2=0x7f8e6819bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.168+0000 7f8e5de00640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e681055f0 0x7f8e6819bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.168+0000 7f8e5de00640 1 -- 172.21.15.107:0/3093971876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e6810b880 msgr2=0x7f8e681a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.168+0000 7f8e5de00640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e6810b880 0x7f8e681a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.168+0000 7f8e5de00640 1 -- 172.21.15.107:0/3093971876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e681a1ec0 con 0x7f8e6810af30 2024-07-26T12:47:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.169+0000 7f8e5de00640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e6810af30 0x7f8e6819c4f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000e970 tx=0x7f8e5000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.169+0000 7f8e5fe00640 1 -- 172.21.15.107:0/3093971876 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5000cd80 con 0x7f8e6810af30 2024-07-26T12:47:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.169+0000 7f8e5fe00640 1 -- 172.21.15.107:0/3093971876 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e500035f0 con 0x7f8e6810af30 2024-07-26T12:47:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.169+0000 7f8e6ca00640 1 -- 172.21.15.107:0/3093971876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e681a21b0 con 0x7f8e6810af30 2024-07-26T12:47:44.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.169+0000 7f8e5fe00640 1 -- 172.21.15.107:0/3093971876 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e50010690 con 0x7f8e6810af30 2024-07-26T12:47:44.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.169+0000 7f8e6ca00640 1 -- 172.21.15.107:0/3093971876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e681a8910 con 0x7f8e6810af30 2024-07-26T12:47:44.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.170+0000 7f8e6ca00640 1 -- 172.21.15.107:0/3093971876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e68105a00 con 0x7f8e6810af30 2024-07-26T12:47:44.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.174+0000 7f8e5fe00640 1 -- 172.21.15.107:0/3093971876 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f8e50003aa0 con 0x7f8e6810af30 2024-07-26T12:47:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.176+0000 7f8e5fe00640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8e4c0b3c20 0x7f8e4c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.176+0000 7f8e5fe00640 1 -- 172.21.15.107:0/3093971876 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8e500e4400 con 0x7f8e6810af30 2024-07-26T12:47:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.176+0000 7f8e66800640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8e4c0b3c20 0x7f8e4c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.177+0000 7f8e66800640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8e4c0b3c20 0x7f8e4c0b6130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8e58009bd0 tx=0x7f8e58005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:44.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.178+0000 7f8e5fe00640 1 -- 172.21.15.107:0/3093971876 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8e500ab550 con 0x7f8e6810af30 2024-07-26T12:47:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:43 smithi112 ceph-mon[37501]: pgmap v442: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:43 smithi132 ceph-mon[37580]: pgmap v442: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:44.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.385+0000 7f8e6ca00640 1 -- 172.21.15.107:0/3093971876 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f8e68063230 con 0x7f8e4c0b3c20 2024-07-26T12:47:44.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.387+0000 7f8e5fe00640 1 -- 172.21.15.107:0/3093971876 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f8e68063230 con 0x7f8e4c0b3c20 2024-07-26T12:47:44.389 INFO:teuthology.orchestra.run.smithi107.stdout:HOST ADDR LABELS STATUS 2024-07-26T12:47:44.389 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 172.21.15.107 2024-07-26T12:47:44.389 INFO:teuthology.orchestra.run.smithi107.stdout:smithi112 172.21.15.112 2024-07-26T12:47:44.389 INFO:teuthology.orchestra.run.smithi107.stdout:smithi132 172.21.15.132 2024-07-26T12:47:44.389 INFO:teuthology.orchestra.run.smithi107.stdout:3 hosts in cluster 2024-07-26T12:47:44.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.390+0000 7f8e6ca00640 1 -- 172.21.15.107:0/3093971876 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8e4c0b3c20 msgr2=0x7f8e4c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:44.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.390+0000 7f8e6ca00640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8e4c0b3c20 0x7f8e4c0b6130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8e58009bd0 tx=0x7f8e58005f90 comp rx=0 tx=0).stop 2024-07-26T12:47:44.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.390+0000 7f8e6ca00640 1 -- 172.21.15.107:0/3093971876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e6810af30 msgr2=0x7f8e6819c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:44.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.390+0000 7f8e6ca00640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e6810af30 0x7f8e6819c4f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000e970 tx=0x7f8e5000ee30 comp rx=0 tx=0).stop 2024-07-26T12:47:44.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.391+0000 7f8e6ca00640 1 -- 172.21.15.107:0/3093971876 shutdown_connections 2024-07-26T12:47:44.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.391+0000 7f8e6ca00640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8e4c0b3c20 0x7f8e4c0b6130 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:44.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.391+0000 7f8e6ca00640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8e6810b880 0x7f8e681a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:44.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.391+0000 7f8e6ca00640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8e6810af30 0x7f8e6819c4f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:44.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.391+0000 7f8e6ca00640 1 --2- 172.21.15.107:0/3093971876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8e681055f0 0x7f8e6819bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:44.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.391+0000 7f8e6ca00640 1 -- 172.21.15.107:0/3093971876 >> 172.21.15.107:0/3093971876 conn(0x7f8e68101040 msgr2=0x7f8e681084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:44.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.391+0000 7f8e6ca00640 1 -- 172.21.15.107:0/3093971876 shutdown_connections 2024-07-26T12:47:44.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:44.391+0000 7f8e6ca00640 1 -- 172.21.15.107:0/3093971876 wait complete. 2024-07-26T12:47:45.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch device ls' 2024-07-26T12:47:45.956 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:45 smithi107 ceph-mon[32659]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:45 smithi107 ceph-mon[32659]: pgmap v443: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:45 smithi112 ceph-mon[37501]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:45 smithi112 ceph-mon[37501]: pgmap v443: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:45 smithi132 ceph-mon[37580]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:45 smithi132 ceph-mon[37580]: pgmap v443: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:48.169 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:47 smithi107 ceph-mon[32659]: pgmap v444: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:47 smithi112 ceph-mon[37501]: pgmap v444: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:47 smithi132 ceph-mon[37580]: pgmap v444: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:48.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.376+0000 7f4004e00640 1 -- 172.21.15.107:0/2342882838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4000114170 msgr2=0x7f4000116590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:48.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.376+0000 7f4004e00640 1 --2- 172.21.15.107:0/2342882838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4000114170 0x7f4000116590 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8009ca0 tx=0x7f3ff802a8c0 comp rx=0 tx=0).stop 2024-07-26T12:47:48.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.376+0000 7f4004e00640 1 -- 172.21.15.107:0/2342882838 shutdown_connections 2024-07-26T12:47:48.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.376+0000 7f4004e00640 1 --2- 172.21.15.107:0/2342882838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4000114170 0x7f4000116590 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:48.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.376+0000 7f4004e00640 1 --2- 172.21.15.107:0/2342882838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000077cc0 0x7f4000113c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:48.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.376+0000 7f4004e00640 1 --2- 172.21.15.107:0/2342882838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4000077370 0x7f4000077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:48.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.376+0000 7f4004e00640 1 -- 172.21.15.107:0/2342882838 >> 172.21.15.107:0/2342882838 conn(0x7f4000101240 msgr2=0x7f4000103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:48.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.376+0000 7f4004e00640 1 -- 172.21.15.107:0/2342882838 shutdown_connections 2024-07-26T12:47:48.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.376+0000 7f4004e00640 1 -- 172.21.15.107:0/2342882838 wait complete. 2024-07-26T12:47:48.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.378+0000 7f4004e00640 1 Processor -- start 2024-07-26T12:47:48.378 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.378+0000 7f4004e00640 1 -- start start 2024-07-26T12:47:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.378+0000 7f4004e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4000077370 0x7f40001a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.378+0000 7f4004e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4000077cc0 0x7f40001a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.378+0000 7f4004e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000114170 0x7f40001a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f4004e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4000119b40 con 0x7f4000077370 2024-07-26T12:47:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f4004e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f40001199c0 con 0x7f4000077cc0 2024-07-26T12:47:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f4004e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f4000119cc0 con 0x7f4000114170 2024-07-26T12:47:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f3ffde00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4000077370 0x7f40001a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f3ffd400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4000077cc0 0x7f40001a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f3ffde00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4000077370 0x7f40001a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58550/0 (socket says 172.21.15.107:58550) 2024-07-26T12:47:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f3ffde00640 1 -- 172.21.15.107:0/345730250 learned_addr learned my addr 172.21.15.107:0/345730250 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:48.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f3ffe800640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000114170 0x7f40001a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:48.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f3ffd400640 1 -- 172.21.15.107:0/345730250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000114170 msgr2=0x7f40001a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f3ffd400640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000114170 0x7f40001a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f3ffd400640 1 -- 172.21.15.107:0/345730250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4000077370 msgr2=0x7f40001a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f3ffd400640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4000077370 0x7f40001a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.379+0000 7f3ffd400640 1 -- 172.21.15.107:0/345730250 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40001a63d0 con 0x7f4000077cc0 2024-07-26T12:47:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.380+0000 7f3ffd400640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4000077cc0 0x7f40001a0a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f400006c4d0 tx=0x7f3fe800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.380+0000 7f3ff6a00640 1 -- 172.21.15.107:0/345730250 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe800cc40 con 0x7f4000077cc0 2024-07-26T12:47:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.380+0000 7f4004e00640 1 -- 172.21.15.107:0/345730250 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40001a66c0 con 0x7f4000077cc0 2024-07-26T12:47:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.380+0000 7f3ff6a00640 1 -- 172.21.15.107:0/345730250 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fe800cde0 con 0x7f4000077cc0 2024-07-26T12:47:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.380+0000 7f3ff6a00640 1 -- 172.21.15.107:0/345730250 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8004a00 con 0x7f4000077cc0 2024-07-26T12:47:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.380+0000 7f4004e00640 1 -- 172.21.15.107:0/345730250 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40001ace20 con 0x7f4000077cc0 2024-07-26T12:47:48.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.381+0000 7f4004e00640 1 -- 172.21.15.107:0/345730250 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4000077780 con 0x7f4000077cc0 2024-07-26T12:47:48.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.384+0000 7f3ff6a00640 1 -- 172.21.15.107:0/345730250 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f3fe8004ba0 con 0x7f4000077cc0 2024-07-26T12:47:48.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.385+0000 7f3ff6a00640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3fe40b3c20 0x7f3fe40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:48.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.386+0000 7f3ff6a00640 1 -- 172.21.15.107:0/345730250 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3fe80e4230 con 0x7f4000077cc0 2024-07-26T12:47:48.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.386+0000 7f3ffde00640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3fe40b3c20 0x7f3fe40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:48.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.386+0000 7f3ffde00640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3fe40b3c20 0x7f3fe40b6130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00095f0 tx=0x7f3ff0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:48.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.387+0000 7f3ff6a00640 1 -- 172.21.15.107:0/345730250 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3fe80ab410 con 0x7f4000077cc0 2024-07-26T12:47:48.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.592+0000 7f4004e00640 1 -- 172.21.15.107:0/345730250 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f4000063230 con 0x7f3fe40b3c20 2024-07-26T12:47:48.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.594+0000 7f3ff6a00640 1 -- 172.21.15.107:0/345730250 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f4000063230 con 0x7f3fe40b3c20 2024-07-26T12:47:48.595 INFO:teuthology.orchestra.run.smithi107.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-07-26T12:47:48.595 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NK400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-07-26T12:47:48.595 INFO:teuthology.orchestra.run.smithi107.stdout:smithi112 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NV400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-07-26T12:47:48.596 INFO:teuthology.orchestra.run.smithi107.stdout:smithi132 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232000B400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-07-26T12:47:48.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 -- 172.21.15.107:0/345730250 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3fe40b3c20 msgr2=0x7f3fe40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:48.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3fe40b3c20 0x7f3fe40b6130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00095f0 tx=0x7f3ff0005f90 comp rx=0 tx=0).stop 2024-07-26T12:47:48.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 -- 172.21.15.107:0/345730250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4000077cc0 msgr2=0x7f40001a0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:48.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4000077cc0 0x7f40001a0a00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f400006c4d0 tx=0x7f3fe800ee50 comp rx=0 tx=0).stop 2024-07-26T12:47:48.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 -- 172.21.15.107:0/345730250 shutdown_connections 2024-07-26T12:47:48.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3fe40b3c20 0x7f3fe40b6130 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:48.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4000114170 0x7f40001a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:48.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f4000077cc0 0x7f40001a0a00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:48.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 --2- 172.21.15.107:0/345730250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4000077370 0x7f40001a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:48.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 -- 172.21.15.107:0/345730250 >> 172.21.15.107:0/345730250 conn(0x7f4000101240 msgr2=0x7f4000116190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:48.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 -- 172.21.15.107:0/345730250 shutdown_connections 2024-07-26T12:47:48.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:48.597+0000 7f4004e00640 1 -- 172.21.15.107:0/345730250 wait complete. 2024-07-26T12:47:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:48 smithi112 ceph-mon[37501]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:48 smithi112 ceph-mon[37501]: pgmap v445: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:47:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:48 smithi107 ceph-mon[32659]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:48 smithi107 ceph-mon[32659]: pgmap v445: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:47:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:48 smithi132 ceph-mon[37580]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:47:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:48 smithi132 ceph-mon[37580]: pgmap v445: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:47:49.573 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T12:47:49.583 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-07-26T12:47:49.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- bash -c 'ceph fs dump' 2024-07-26T12:47:50.050 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:52.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:51 smithi107 ceph-mon[32659]: pgmap v446: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:51 smithi112 ceph-mon[37501]: pgmap v446: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:51 smithi132 ceph-mon[37580]: pgmap v446: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:47:52.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.500+0000 7f21e6000640 1 -- 172.21.15.107:0/2249252373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e0075b30 msgr2=0x7f21e010f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:52.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.500+0000 7f21e6000640 1 --2- 172.21.15.107:0/2249252373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e0075b30 0x7f21e010f400 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009860 tx=0x7f21c802a340 comp rx=0 tx=0).stop 2024-07-26T12:47:52.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.500+0000 7f21e6000640 1 -- 172.21.15.107:0/2249252373 shutdown_connections 2024-07-26T12:47:52.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.500+0000 7f21e6000640 1 --2- 172.21.15.107:0/2249252373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21e010f940 0x7f21e0111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:52.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.500+0000 7f21e6000640 1 --2- 172.21.15.107:0/2249252373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e0075b30 0x7f21e010f400 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:52.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.500+0000 7f21e6000640 1 --2- 172.21.15.107:0/2249252373 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21e00751e0 0x7f21e00755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:52.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.500+0000 7f21e6000640 1 -- 172.21.15.107:0/2249252373 >> 172.21.15.107:0/2249252373 conn(0x7f21e00fee70 msgr2=0x7f21e01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:52.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.500+0000 7f21e6000640 1 -- 172.21.15.107:0/2249252373 shutdown_connections 2024-07-26T12:47:52.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.500+0000 7f21e6000640 1 -- 172.21.15.107:0/2249252373 wait complete. 2024-07-26T12:47:52.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.501+0000 7f21e6000640 1 Processor -- start 2024-07-26T12:47:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.501+0000 7f21e6000640 1 -- start start 2024-07-26T12:47:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21e6000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e00751e0 0x7f21e019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21e6000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21e0075b30 0x7f21e019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21e6000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21e010f940 0x7f21e01a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21e6000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f21e01154e0 con 0x7f21e0075b30 2024-07-26T12:47:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21e6000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f21e0115360 con 0x7f21e010f940 2024-07-26T12:47:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21e6000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f21e0115660 con 0x7f21e00751e0 2024-07-26T12:47:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21dfe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21e0075b30 0x7f21e019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21dfe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21e0075b30 0x7f21e019c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:58572/0 (socket says 172.21.15.107:58572) 2024-07-26T12:47:52.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21dfe00640 1 -- 172.21.15.107:0/875473872 learned_addr learned my addr 172.21.15.107:0/875473872 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21e4c00640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e00751e0 0x7f21e019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.502+0000 7f21e5600640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21e010f940 0x7f21e01a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21dfe00640 1 -- 172.21.15.107:0/875473872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e00751e0 msgr2=0x7f21e019bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21dfe00640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e00751e0 0x7f21e019bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21dfe00640 1 -- 172.21.15.107:0/875473872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21e010f940 msgr2=0x7f21e01a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21dfe00640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21e010f940 0x7f21e01a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21dfe00640 1 -- 172.21.15.107:0/875473872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e01a1ec0 con 0x7f21e0075b30 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21dfe00640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21e0075b30 0x7f21e019c4f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f21c8000c00 tx=0x7f21c8032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21dd600640 1 -- 172.21.15.107:0/875473872 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8040020 con 0x7f21e0075b30 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21dd600640 1 -- 172.21.15.107:0/875473872 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21c8030e10 con 0x7f21e0075b30 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21e6000640 1 -- 172.21.15.107:0/875473872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e01a2150 con 0x7f21e0075b30 2024-07-26T12:47:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21dd600640 1 -- 172.21.15.107:0/875473872 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8037040 con 0x7f21e0075b30 2024-07-26T12:47:52.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.503+0000 7f21e6000640 1 -- 172.21.15.107:0/875473872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e01a8910 con 0x7f21e0075b30 2024-07-26T12:47:52.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.505+0000 7f21e6000640 1 -- 172.21.15.107:0/875473872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21e00766f0 con 0x7f21e0075b30 2024-07-26T12:47:52.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.507+0000 7f21dd600640 1 -- 172.21.15.107:0/875473872 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f21c8043020 con 0x7f21e0075b30 2024-07-26T12:47:52.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.508+0000 7f21dd600640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c40b3c20 0x7f21c40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:52.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.509+0000 7f21dd600640 1 -- 172.21.15.107:0/875473872 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f21c81045f0 con 0x7f21e0075b30 2024-07-26T12:47:52.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.509+0000 7f21e4c00640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c40b3c20 0x7f21c40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:52.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.510+0000 7f21e4c00640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c40b3c20 0x7f21c40b6130 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009bd0 tx=0x7f21d0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:52.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.511+0000 7f21dd600640 1 -- 172.21.15.107:0/875473872 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f21c804a030 con 0x7f21e0075b30 2024-07-26T12:47:52.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.749+0000 7f21e6000640 1 -- 172.21.15.107:0/875473872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f21e01a8ae0 con 0x7f21e0075b30 2024-07-26T12:47:52.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.750+0000 7f21dd600640 1 -- 172.21.15.107:0/875473872 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3142 (secure 0 0 0) 0x7f21c80cb7c0 con 0x7f21e0075b30 2024-07-26T12:47:52.752 INFO:teuthology.orchestra.run.smithi107.stdout:e28 2024-07-26T12:47:52.752 INFO:teuthology.orchestra.run.smithi107.stdout:btime 2024-07-26T12:44:44:796279+0000 2024-07-26T12:47:52.752 INFO:teuthology.orchestra.run.smithi107.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T12:47:52.752 INFO:teuthology.orchestra.run.smithi107.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T12:47:52.752 INFO:teuthology.orchestra.run.smithi107.stdout:legacy client fscid: 1 2024-07-26T12:47:52.752 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:47:52.752 INFO:teuthology.orchestra.run.smithi107.stdout:Filesystem 'cephfs' (1) 2024-07-26T12:47:52.752 INFO:teuthology.orchestra.run.smithi107.stdout:fs_name cephfs 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 28 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-07-26T12:43:15.198289+0000 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2024-07-26T12:44:43.812461+0000 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:tableserver 0 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:root 0 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:session_timeout 300 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:session_autoclose 300 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:max_file_size 1099511627776 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:max_xattr_size 65536 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:required_client_features {} 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:last_failure 0 2024-07-26T12:47:52.753 INFO:teuthology.orchestra.run.smithi107.stdout:last_failure_osd_epoch 0 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:max_mds 5 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:in 0,1,2,3,4 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:up {0=24716,1=24715,2=24703,3=24694,4=24682} 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:failed 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:damaged 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:stopped 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:data_pools [3,4] 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:metadata_pool 2 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:inline_data disabled 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:balancer 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:bal_rank_mask -1 2024-07-26T12:47:52.754 INFO:teuthology.orchestra.run.smithi107.stdout:standby_count_wanted 0 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:qdb_cluster leader: 24716 members: 24682,24694,24703,24716,24715 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.b{0:24716} state up:active seq 6 addr [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.h{0:24680} state up:standby-replay seq 1 addr [v2:172.21.15.112:6836/3739854241,v1:172.21.15.112:6837/3739854241] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.g{1:24715} state up:active seq 28 addr [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.e{1:24662} state up:standby-replay seq 1 addr [v2:172.21.15.112:6834/1038904649,v1:172.21.15.112:6835/1038904649] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.i{2:24703} state up:active seq 29 addr [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.j{2:24652} state up:standby-replay seq 1 addr [v2:172.21.15.107:6834/3403405722,v1:172.21.15.107:6835/3403405722] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.c{3:24694} state up:active seq 31 addr [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.l{3:24634} state up:standby-replay seq 1 addr [v2:172.21.15.132:6832/277912155,v1:172.21.15.132:6833/277912155] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.f{4:24682} state up:active seq 32 addr [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.k{4:24632} state up:standby-replay seq 1 addr [v2:172.21.15.112:6832/2931369116,v1:172.21.15.112:6833/2931369116] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:47:52.755 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:47:52.756 INFO:teuthology.orchestra.run.smithi107.stdout:Standby daemons: 2024-07-26T12:47:52.756 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:47:52.756 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.d{-1:14544} state up:standby seq 1 addr [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.756 INFO:teuthology.orchestra.run.smithi107.stdout:[mds.a{-1:14580} state up:standby seq 1 addr [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T12:47:52.756 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-07-26T12:47:52.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.753+0000 7f21e6000640 1 -- 172.21.15.107:0/875473872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c40b3c20 msgr2=0x7f21c40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.753+0000 7f21e6000640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c40b3c20 0x7f21c40b6130 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009bd0 tx=0x7f21d0005f90 comp rx=0 tx=0).stop 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.753+0000 7f21e6000640 1 -- 172.21.15.107:0/875473872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21e0075b30 msgr2=0x7f21e019c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.753+0000 7f21e6000640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21e0075b30 0x7f21e019c4f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f21c8000c00 tx=0x7f21c8032d40 comp rx=0 tx=0).stop 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.753+0000 7f21e6000640 1 -- 172.21.15.107:0/875473872 shutdown_connections 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.753+0000 7f21e6000640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21c40b3c20 0x7f21c40b6130 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.753+0000 7f21e6000640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f21e010f940 0x7f21e01a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.753+0000 7f21e6000640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21e0075b30 0x7f21e019c4f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.753+0000 7f21e6000640 1 --2- 172.21.15.107:0/875473872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f21e00751e0 0x7f21e019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.753+0000 7f21e6000640 1 -- 172.21.15.107:0/875473872 >> 172.21.15.107:0/875473872 conn(0x7f21e00fee70 msgr2=0x7f21e0109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.754+0000 7f21e6000640 1 -- 172.21.15.107:0/875473872 shutdown_connections 2024-07-26T12:47:52.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:52.754+0000 7f21e6000640 1 -- 172.21.15.107:0/875473872 wait complete. 2024-07-26T12:47:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:52 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/875473872' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T12:47:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:52 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/875473872' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T12:47:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:52 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/875473872' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T12:47:53.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- bash -c 'ceph osd dump' 2024-07-26T12:47:54.054 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:47:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:53 smithi112 ceph-mon[37501]: pgmap v447: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:54.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:53 smithi107 ceph-mon[32659]: pgmap v447: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:53 smithi132 ceph-mon[37580]: pgmap v447: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:47:56.087 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:55 smithi107 ceph-mon[32659]: pgmap v448: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:55 smithi112 ceph-mon[37501]: pgmap v448: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:55 smithi132 ceph-mon[37580]: pgmap v448: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:56.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.421+0000 7f7d1fe00640 1 -- 172.21.15.107:0/3064004465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d201099f0 msgr2=0x7f7d20109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.421+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/3064004465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d201099f0 0x7f7d20109e00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7d10009ca0 tx=0x7f7d1002a8b0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.421+0000 7f7d1fe00640 1 -- 172.21.15.107:0/3064004465 shutdown_connections 2024-07-26T12:47:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.421+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/3064004465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2010fc80 0x7f7d201164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.421+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/3064004465 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d2010f330 0x7f7d2010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.421+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/3064004465 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d201099f0 0x7f7d20109e00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.421+0000 7f7d1fe00640 1 -- 172.21.15.107:0/3064004465 >> 172.21.15.107:0/3064004465 conn(0x7f7d20078ff0 msgr2=0x7f7d20107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.421+0000 7f7d1fe00640 1 -- 172.21.15.107:0/3064004465 shutdown_connections 2024-07-26T12:47:56.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.421+0000 7f7d1fe00640 1 -- 172.21.15.107:0/3064004465 wait complete. 2024-07-26T12:47:56.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.423+0000 7f7d1fe00640 1 Processor -- start 2024-07-26T12:47:56.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.423+0000 7f7d1fe00640 1 -- start start 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.423+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d201099f0 0x7f7d201a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.423+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d2010f330 0x7f7d201a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.423+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2010fc80 0x7f7d201a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.423+0000 7f7d1fe00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d20119940 con 0x7f7d2010fc80 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.423+0000 7f7d1fe00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201197c0 con 0x7f7d2010f330 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.423+0000 7f7d1fe00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7d20119ac0 con 0x7f7d201099f0 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.424+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d201099f0 0x7f7d201a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.424+0000 7f7d1f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2010fc80 0x7f7d201a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.424+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d201099f0 0x7f7d201a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:60476/0 (socket says 172.21.15.107:60476) 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.424+0000 7f7d1ea00640 1 -- 172.21.15.107:0/4211052485 learned_addr learned my addr 172.21.15.107:0/4211052485 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:47:56.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.424+0000 7f7d1e000640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d2010f330 0x7f7d201a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.424+0000 7f7d1ea00640 1 -- 172.21.15.107:0/4211052485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d2010f330 msgr2=0x7f7d201a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.424+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d2010f330 0x7f7d201a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.424+0000 7f7d1ea00640 1 -- 172.21.15.107:0/4211052485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2010fc80 msgr2=0x7f7d201a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.424+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2010fc80 0x7f7d201a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.424+0000 7f7d1ea00640 1 -- 172.21.15.107:0/4211052485 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d201a62e0 con 0x7f7d201099f0 2024-07-26T12:47:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.425+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d201099f0 0x7f7d201a0370 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7d10006010 tx=0x7f7d10030cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.425+0000 7f7d24a00640 1 -- 172.21.15.107:0/4211052485 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10004040 con 0x7f7d201099f0 2024-07-26T12:47:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.425+0000 7f7d24a00640 1 -- 172.21.15.107:0/4211052485 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d10040070 con 0x7f7d201099f0 2024-07-26T12:47:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.425+0000 7f7d1fe00640 1 -- 172.21.15.107:0/4211052485 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d201a6570 con 0x7f7d201099f0 2024-07-26T12:47:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.425+0000 7f7d24a00640 1 -- 172.21.15.107:0/4211052485 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10045b50 con 0x7f7d201099f0 2024-07-26T12:47:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.425+0000 7f7d1fe00640 1 -- 172.21.15.107:0/4211052485 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d201acd30 con 0x7f7d201099f0 2024-07-26T12:47:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.426+0000 7f7d1fe00640 1 -- 172.21.15.107:0/4211052485 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d20109e00 con 0x7f7d201099f0 2024-07-26T12:47:56.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.429+0000 7f7d24a00640 1 -- 172.21.15.107:0/4211052485 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f7d10045cf0 con 0x7f7d201099f0 2024-07-26T12:47:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.430+0000 7f7d24a00640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3c20 0x7f7d040b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:47:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.430+0000 7f7d24a00640 1 -- 172.21.15.107:0/4211052485 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7d10105420 con 0x7f7d201099f0 2024-07-26T12:47:56.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.430+0000 7f7d1e000640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3c20 0x7f7d040b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:47:56.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.431+0000 7f7d1e000640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3c20 0x7f7d040b6130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7d2010fa80 tx=0x7f7d0c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:47:56.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.432+0000 7f7d24a00640 1 -- 172.21.15.107:0/4211052485 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7d100cc570 con 0x7f7d201099f0 2024-07-26T12:47:56.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.631+0000 7f7d1fe00640 1 -- 172.21.15.107:0/4211052485 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f7d20111a50 con 0x7f7d201099f0 2024-07-26T12:47:56.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.633+0000 7f7d24a00640 1 -- 172.21.15.107:0/4211052485 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+6112 (secure 0 0 0) 0x7f7d100cbe70 con 0x7f7d201099f0 2024-07-26T12:47:56.635 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 90 2024-07-26T12:47:56.635 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 2024-07-26T12:47:56.635 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-07-26T12:33:09.598433+0000 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2024-07-26T12:44:36.797948+0000 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:crush_version 27 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:full_ratio 0.95 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:backfillfull_ratio 0.9 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:nearfull_ratio 0.85 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:require_min_compat_client luminous 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:min_compat_client luminous 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:require_osd_release squid 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:stretch_mode_enabled false 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-07-26T12:47:56.636 INFO:teuthology.orchestra.run.smithi107.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:max_osd 12 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 up in weight 1 up_from 9 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6802/2354859122,v1:172.21.15.107:6803/2354859122] [v2:172.21.15.107:6804/2354859122,v1:172.21.15.107:6805/2354859122] exists,up 9f413e43-6b6b-4596-ab6e-c75cdf9c7c96 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 up in weight 1 up_from 14 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6800/3125120524,v1:172.21.15.112:6801/3125120524] [v2:172.21.15.112:6802/3125120524,v1:172.21.15.112:6803/3125120524] exists,up 44ac0320-aac0-4810-ad34-6798956f1969 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 up in weight 1 up_from 19 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6800/28260456,v1:172.21.15.132:6801/28260456] [v2:172.21.15.132:6802/28260456,v1:172.21.15.132:6803/28260456] exists,up 538267b7-07cf-4be0-95bf-bf9e8a4005f3 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 up in weight 1 up_from 26 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6810/1620179989,v1:172.21.15.107:6811/1620179989] [v2:172.21.15.107:6812/1620179989,v1:172.21.15.107:6813/1620179989] exists,up 914bea43-b9ac-4bed-817e-6af4d1d0aa2e 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 up in weight 1 up_from 32 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6808/95536584,v1:172.21.15.112:6809/95536584] [v2:172.21.15.112:6810/95536584,v1:172.21.15.112:6811/95536584] exists,up bc83673b-3617-4245-bfe1-b61c9cbe2ab4 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 up in weight 1 up_from 37 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6808/1054197844,v1:172.21.15.132:6809/1054197844] [v2:172.21.15.132:6810/1054197844,v1:172.21.15.132:6811/1054197844] exists,up 5d0394cf-c561-4a68-b674-92ea81d9ffe1 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 up in weight 1 up_from 43 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6818/2132739845,v1:172.21.15.107:6819/2132739845] [v2:172.21.15.107:6820/2132739845,v1:172.21.15.107:6821/2132739845] exists,up 17052119-63ed-49e0-91a7-1e888c82dee2 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 up in weight 1 up_from 48 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6816/3770813146,v1:172.21.15.112:6817/3770813146] [v2:172.21.15.112:6818/3770813146,v1:172.21.15.112:6819/3770813146] exists,up 4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53 2024-07-26T12:47:56.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.8 up in weight 1 up_from 54 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6816/2677294944,v1:172.21.15.132:6817/2677294944] [v2:172.21.15.132:6818/2677294944,v1:172.21.15.132:6819/2677294944] exists,up 4d3958e5-fe71-448b-a6de-6bcc423daca4 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.9 up in weight 1 up_from 60 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6826/3840896542,v1:172.21.15.107:6827/3840896542] [v2:172.21.15.107:6828/3840896542,v1:172.21.15.107:6829/3840896542] exists,up 6d641864-0787-4a2b-ae73-e590c1b5c0d0 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.10 up in weight 1 up_from 65 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.112:6824/3415013149,v1:172.21.15.112:6825/3415013149] [v2:172.21.15.112:6826/3415013149,v1:172.21.15.112:6827/3415013149] exists,up d5cdd3a2-311a-4c36-8553-3e86dad57015 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.11 up in weight 1 up_from 71 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.132:6824/3905824310,v1:172.21.15.132:6825/3905824310] [v2:172.21.15.132:6826/3905824310,v1:172.21.15.132:6827/3905824310] exists,up d738c87c-2b11-4ba4-b38d-a58e2fd9a235 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.b [8,11] 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 2.13 [2,11] 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.0 [4,7] 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.5 [3,9] 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.14 [4,6] 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.17 [4,6] 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.1b [4,8] 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.21 [5,9] 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.2a [3,8] 2024-07-26T12:47:56.638 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 3.33 [5,9] 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.6 [9,11] 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.f [9,8] 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.11 [5,10] 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.1f [9,8] 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/3311144690 expires 2024-07-27T12:35:34.485438+0000 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/81377212 expires 2024-07-27T12:35:34.485438+0000 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/634100778 expires 2024-07-27T12:33:46.111763+0000 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/4257991258 expires 2024-07-27T12:35:34.485438+0000 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/2773605049 expires 2024-07-27T12:35:34.485438+0000 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/312154060 expires 2024-07-27T12:33:46.111763+0000 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/1797109211 expires 2024-07-27T12:33:46.111763+0000 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/670053235 expires 2024-07-27T12:34:14.657397+0000 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/453757265 expires 2024-07-27T12:33:46.111763+0000 2024-07-26T12:47:56.639 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/2773605049 expires 2024-07-27T12:35:34.485438+0000 2024-07-26T12:47:56.640 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/425164471 expires 2024-07-27T12:34:14.657397+0000 2024-07-26T12:47:56.640 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/4171514587 expires 2024-07-27T12:34:14.657397+0000 2024-07-26T12:47:56.640 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/634100778 expires 2024-07-27T12:33:46.111763+0000 2024-07-26T12:47:56.640 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/425164471 expires 2024-07-27T12:34:14.657397+0000 2024-07-26T12:47:56.640 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/2624080873 expires 2024-07-27T12:34:14.657397+0000 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.637+0000 7f7d1fe00640 1 -- 172.21.15.107:0/4211052485 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3c20 msgr2=0x7f7d040b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.637+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3c20 0x7f7d040b6130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7d2010fa80 tx=0x7f7d0c005f90 comp rx=0 tx=0).stop 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.637+0000 7f7d1fe00640 1 -- 172.21.15.107:0/4211052485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d201099f0 msgr2=0x7f7d201a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.637+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d201099f0 0x7f7d201a0370 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7d10006010 tx=0x7f7d10030cf0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.637+0000 7f7d1fe00640 1 -- 172.21.15.107:0/4211052485 shutdown_connections 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.637+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3c20 0x7f7d040b6130 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.637+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d2010fc80 0x7f7d201a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.637+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d2010f330 0x7f7d201a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.637+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/4211052485 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d201099f0 0x7f7d201a0370 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.637+0000 7f7d1fe00640 1 -- 172.21.15.107:0/4211052485 >> 172.21.15.107:0/4211052485 conn(0x7f7d20078ff0 msgr2=0x7f7d20107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.638+0000 7f7d1fe00640 1 -- 172.21.15.107:0/4211052485 shutdown_connections 2024-07-26T12:47:56.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:47:56.638+0000 7f7d1fe00640 1 -- 172.21.15.107:0/4211052485 wait complete. 2024-07-26T12:47:57.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:56 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/4211052485' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T12:47:57.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:56 smithi107 ceph-mon[32659]: pgmap v449: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:56 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/4211052485' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T12:47:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:56 smithi112 ceph-mon[37501]: pgmap v449: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:56 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/4211052485' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T12:47:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:56 smithi132 ceph-mon[37580]: pgmap v449: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:47:57.656 INFO:teuthology.run_tasks:Running task fs.ready... 2024-07-26T12:47:57.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:47:58.119 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:00.115 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:47:59 smithi107 ceph-mon[32659]: pgmap v450: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:47:59 smithi112 ceph-mon[37501]: pgmap v450: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:47:59 smithi132 ceph-mon[37580]: pgmap v450: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.502+0000 7f597ec00640 1 -- 172.21.15.107:0/2798950513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5978104490 msgr2=0x7f59781048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.502+0000 7f597ec00640 1 --2- 172.21.15.107:0/2798950513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5978104490 0x7f59781048a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5968009900 tx=0x7f596802a410 comp rx=0 tx=0).stop 2024-07-26T12:48:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.503+0000 7f597ec00640 1 -- 172.21.15.107:0/2798950513 shutdown_connections 2024-07-26T12:48:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.503+0000 7f597ec00640 1 --2- 172.21.15.107:0/2798950513 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f597810d8f0 0x7f597810fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.503+0000 7f597ec00640 1 --2- 172.21.15.107:0/2798950513 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5978104de0 0x7f597810d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.503+0000 7f597ec00640 1 --2- 172.21.15.107:0/2798950513 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5978104490 0x7f59781048a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.503+0000 7f597ec00640 1 -- 172.21.15.107:0/2798950513 >> 172.21.15.107:0/2798950513 conn(0x7f5978079100 msgr2=0x7f5978079510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:00.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.503+0000 7f597ec00640 1 -- 172.21.15.107:0/2798950513 shutdown_connections 2024-07-26T12:48:00.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.503+0000 7f597ec00640 1 -- 172.21.15.107:0/2798950513 wait complete. 2024-07-26T12:48:00.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.504+0000 7f597ec00640 1 Processor -- start 2024-07-26T12:48:00.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.504+0000 7f597ec00640 1 -- start start 2024-07-26T12:48:00.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.504+0000 7f597ec00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5978104490 0x7f597819c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:00.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.504+0000 7f597ec00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5978104de0 0x7f597819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:00.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.504+0000 7f597ec00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f597810d8f0 0x7f59781a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:00.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.504+0000 7f597ec00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5978113580 con 0x7f5978104490 2024-07-26T12:48:00.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.504+0000 7f597ec00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f5978113400 con 0x7f5978104de0 2024-07-26T12:48:00.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.504+0000 7f597ec00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f5978113700 con 0x7f597810d8f0 2024-07-26T12:48:00.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f5977e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5978104490 0x7f597819c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:00.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f5977400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5978104de0 0x7f597819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:00.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f5977e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5978104490 0x7f597819c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:47462/0 (socket says 172.21.15.107:47462) 2024-07-26T12:48:00.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f5977e00640 1 -- 172.21.15.107:0/3971209889 learned_addr learned my addr 172.21.15.107:0/3971209889 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:00.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f597cc00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f597810d8f0 0x7f59781a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:00.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f5977400640 1 -- 172.21.15.107:0/3971209889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f597810d8f0 msgr2=0x7f59781a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f5977400640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f597810d8f0 0x7f59781a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f5977400640 1 -- 172.21.15.107:0/3971209889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5978104490 msgr2=0x7f597819c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f5977400640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5978104490 0x7f597819c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f5977400640 1 -- 172.21.15.107:0/3971209889 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59781a20a0 con 0x7f5978104de0 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.505+0000 7f5977e00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5978104490 0x7f597819c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.506+0000 7f5977400640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5978104de0 0x7f597819c6d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f597806b420 tx=0x7f596400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.506+0000 7f5974c00640 1 -- 172.21.15.107:0/3971209889 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596400cc40 con 0x7f5978104de0 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.506+0000 7f597ec00640 1 -- 172.21.15.107:0/3971209889 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59781a2390 con 0x7f5978104de0 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.506+0000 7f5974c00640 1 -- 172.21.15.107:0/3971209889 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f596400cde0 con 0x7f5978104de0 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.506+0000 7f5974c00640 1 -- 172.21.15.107:0/3971209889 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964018760 con 0x7f5978104de0 2024-07-26T12:48:00.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.506+0000 7f597ec00640 1 -- 172.21.15.107:0/3971209889 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59781a8af0 con 0x7f5978104de0 2024-07-26T12:48:00.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.507+0000 7f597ec00640 1 -- 172.21.15.107:0/3971209889 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59781048a0 con 0x7f5978104de0 2024-07-26T12:48:00.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.511+0000 7f5974c00640 1 -- 172.21.15.107:0/3971209889 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f5964010a90 con 0x7f5978104de0 2024-07-26T12:48:00.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.512+0000 7f5974c00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f595c0b3c20 0x7f595c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:00.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.513+0000 7f5974c00640 1 -- 172.21.15.107:0/3971209889 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f59640e48e0 con 0x7f5978104de0 2024-07-26T12:48:00.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.513+0000 7f5977e00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f595c0b3c20 0x7f595c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:00.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.513+0000 7f5974c00640 1 -- 172.21.15.107:0/3971209889 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f59640ab3e0 con 0x7f5978104de0 2024-07-26T12:48:00.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.513+0000 7f5977e00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f595c0b3c20 0x7f595c0b6130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5968031440 tx=0x7f59680058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:00.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.748+0000 7f597ec00640 1 -- 172.21.15.107:0/3971209889 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59781a8cc0 con 0x7f5978104de0 2024-07-26T12:48:00.752 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:48:00.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.749+0000 7f5974c00640 1 -- 172.21.15.107:0/3971209889 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11993 (secure 0 0 0) 0x7f59640af930 con 0x7f5978104de0 2024-07-26T12:48:00.752 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"btime":"2024-07-26T12:44:44:796279+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:44:43.812461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24716,"mds_1":24715,"mds_2":24703,"mds_3":24694,"mds_4":24682},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24632":{"gid":24632,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24634":{"gid":24634,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24652":{"gid":24652,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24662":{"gid":24662,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":32,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24694":{"gid":24694,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":31,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":29,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":28,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24716,"qdb_cluster":[24682,24694,24703,24716,24715]},"id":1}]} 2024-07-26T12:48:00.753 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-07-26T12:48:00.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.752+0000 7f597ec00640 1 -- 172.21.15.107:0/3971209889 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f595c0b3c20 msgr2=0x7f595c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:00.754 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.752+0000 7f597ec00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f595c0b3c20 0x7f595c0b6130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5968031440 tx=0x7f59680058d0 comp rx=0 tx=0).stop 2024-07-26T12:48:00.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.752+0000 7f597ec00640 1 -- 172.21.15.107:0/3971209889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5978104de0 msgr2=0x7f597819c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:00.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.752+0000 7f597ec00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5978104de0 0x7f597819c6d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f597806b420 tx=0x7f596400ee50 comp rx=0 tx=0).stop 2024-07-26T12:48:00.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.753+0000 7f597ec00640 1 -- 172.21.15.107:0/3971209889 shutdown_connections 2024-07-26T12:48:00.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.753+0000 7f597ec00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f595c0b3c20 0x7f595c0b6130 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:00.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.753+0000 7f597ec00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f597810d8f0 0x7f59781a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:00.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.753+0000 7f597ec00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f5978104de0 0x7f597819c6d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:00.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.753+0000 7f597ec00640 1 --2- 172.21.15.107:0/3971209889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5978104490 0x7f597819c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:00.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.753+0000 7f597ec00640 1 -- 172.21.15.107:0/3971209889 >> 172.21.15.107:0/3971209889 conn(0x7f5978079100 msgr2=0x7f5978103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:00.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.753+0000 7f597ec00640 1 -- 172.21.15.107:0/3971209889 shutdown_connections 2024-07-26T12:48:00.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:00.753+0000 7f597ec00640 1 -- 172.21.15.107:0/3971209889 wait complete. 2024-07-26T12:48:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:00 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3971209889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:00 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3971209889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:00 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3971209889' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:01.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:48:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:01 smithi107 ceph-mon[32659]: pgmap v451: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:02.181 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:01 smithi112 ceph-mon[37501]: pgmap v451: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:01 smithi132 ceph-mon[37580]: pgmap v451: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:02 smithi107 ceph-mon[32659]: pgmap v452: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:48:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:02 smithi112 ceph-mon[37501]: pgmap v452: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:48:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:02 smithi132 ceph-mon[37580]: pgmap v452: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:48:04.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.549+0000 7f2df6000640 1 -- 172.21.15.107:0/2021271245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df00726a0 msgr2=0x7f2df0072ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:04.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.549+0000 7f2df6000640 1 --2- 172.21.15.107:0/2021271245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df00726a0 0x7f2df0072ab0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009900 tx=0x7f2dd802a480 comp rx=0 tx=0).stop 2024-07-26T12:48:04.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.549+0000 7f2df6000640 1 -- 172.21.15.107:0/2021271245 shutdown_connections 2024-07-26T12:48:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.549+0000 7f2df6000640 1 --2- 172.21.15.107:0/2021271245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df0118470 0x7f2df011a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.549+0000 7f2df6000640 1 --2- 172.21.15.107:0/2021271245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df00726a0 0x7f2df0072ab0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.549+0000 7f2df6000640 1 --2- 172.21.15.107:0/2021271245 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2df010c8d0 0x7f2df0072160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.549+0000 7f2df6000640 1 -- 172.21.15.107:0/2021271245 >> 172.21.15.107:0/2021271245 conn(0x7f2df0074a40 msgr2=0x7f2df006eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.549+0000 7f2df6000640 1 -- 172.21.15.107:0/2021271245 shutdown_connections 2024-07-26T12:48:04.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.549+0000 7f2df6000640 1 -- 172.21.15.107:0/2021271245 wait complete. 2024-07-26T12:48:04.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.550+0000 7f2df6000640 1 Processor -- start 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.550+0000 7f2df6000640 1 -- start start 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2df6000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df00726a0 0x7f2df01a48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2df6000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df010c8d0 0x7f2df01a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2df6000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2df0118470 0x7f2df01aa120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2df6000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2df011deb0 con 0x7f2df010c8d0 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2df6000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2df011dd30 con 0x7f2df0118470 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2df6000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2df011e030 con 0x7f2df00726a0 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2def400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df00726a0 0x7f2df01a48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2def400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df00726a0 0x7f2df01a48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:52956/0 (socket says 172.21.15.107:52956) 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2def400640 1 -- 172.21.15.107:0/3743722072 learned_addr learned my addr 172.21.15.107:0/3743722072 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2de7e00640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df010c8d0 0x7f2df01a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:04.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.551+0000 7f2defe00640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2df0118470 0x7f2df01aa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2def400640 1 -- 172.21.15.107:0/3743722072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2df0118470 msgr2=0x7f2df01aa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2def400640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2df0118470 0x7f2df01aa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2def400640 1 -- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df010c8d0 msgr2=0x7f2df01a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2def400640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df010c8d0 0x7f2df01a4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2def400640 1 -- 172.21.15.107:0/3743722072 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df01aa850 con 0x7f2df00726a0 2024-07-26T12:48:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2de7e00640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df010c8d0 0x7f2df01a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:48:04.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2def400640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df00726a0 0x7f2df01a48e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2de000e9f0 tx=0x7f2de000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2decc00640 1 -- 172.21.15.107:0/3743722072 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de000cd80 con 0x7f2df00726a0 2024-07-26T12:48:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2df6000640 1 -- 172.21.15.107:0/3743722072 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2df01aab40 con 0x7f2df00726a0 2024-07-26T12:48:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2decc00640 1 -- 172.21.15.107:0/3743722072 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2de00108e0 con 0x7f2df00726a0 2024-07-26T12:48:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.552+0000 7f2decc00640 1 -- 172.21.15.107:0/3743722072 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de0018980 con 0x7f2df00726a0 2024-07-26T12:48:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.553+0000 7f2df6000640 1 -- 172.21.15.107:0/3743722072 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2df01b12a0 con 0x7f2df00726a0 2024-07-26T12:48:04.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.554+0000 7f2df6000640 1 -- 172.21.15.107:0/3743722072 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2df0072ab0 con 0x7f2df00726a0 2024-07-26T12:48:04.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.556+0000 7f2decc00640 1 -- 172.21.15.107:0/3743722072 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f2de0010430 con 0x7f2df00726a0 2024-07-26T12:48:04.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.558+0000 7f2decc00640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2dd40b3c20 0x7f2dd40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.558+0000 7f2decc00640 1 -- 172.21.15.107:0/3743722072 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2de00e4a90 con 0x7f2df00726a0 2024-07-26T12:48:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.558+0000 7f2de7e00640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2dd40b3c20 0x7f2dd40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.559+0000 7f2de7e00640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2dd40b3c20 0x7f2dd40b6130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8031440 tx=0x7f2dd8009680 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:04.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.560+0000 7f2decc00640 1 -- 172.21.15.107:0/3743722072 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2de00abbe0 con 0x7f2df00726a0 2024-07-26T12:48:04.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.802+0000 7f2df6000640 1 -- 172.21.15.107:0/3743722072 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2df01b1540 con 0x7f2df00726a0 2024-07-26T12:48:04.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.803+0000 7f2decc00640 1 -- 172.21.15.107:0/3743722072 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11993 (secure 0 0 0) 0x7f2de00ab4e0 con 0x7f2df00726a0 2024-07-26T12:48:04.806 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:48:04.806 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 28 2024-07-26T12:48:04.806 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":28,"btime":"2024-07-26T12:44:44:796279+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:44:43.812461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24716,"mds_1":24715,"mds_2":24703,"mds_3":24694,"mds_4":24682},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24632":{"gid":24632,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24634":{"gid":24634,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24652":{"gid":24652,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24662":{"gid":24662,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":32,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24694":{"gid":24694,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":31,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":29,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":28,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24716,"qdb_cluster":[24682,24694,24703,24716,24715]},"id":1}]} 2024-07-26T12:48:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.806+0000 7f2df6000640 1 -- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2dd40b3c20 msgr2=0x7f2dd40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.806+0000 7f2df6000640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2dd40b3c20 0x7f2dd40b6130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8031440 tx=0x7f2dd8009680 comp rx=0 tx=0).stop 2024-07-26T12:48:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.806+0000 7f2df6000640 1 -- 172.21.15.107:0/3743722072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df00726a0 msgr2=0x7f2df01a48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.806+0000 7f2df6000640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df00726a0 0x7f2df01a48e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2de000e9f0 tx=0x7f2de000eeb0 comp rx=0 tx=0).stop 2024-07-26T12:48:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.807+0000 7f2df6000640 1 -- 172.21.15.107:0/3743722072 shutdown_connections 2024-07-26T12:48:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.807+0000 7f2df6000640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f2dd40b3c20 0x7f2dd40b6130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.807+0000 7f2df6000640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2df0118470 0x7f2df01aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.807+0000 7f2df6000640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2df010c8d0 0x7f2df01a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.807+0000 7f2df6000640 1 --2- 172.21.15.107:0/3743722072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2df00726a0 0x7f2df01a48e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:04.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.807+0000 7f2df6000640 1 -- 172.21.15.107:0/3743722072 >> 172.21.15.107:0/3743722072 conn(0x7f2df0074a40 msgr2=0x7f2df0118880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:04.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.807+0000 7f2df6000640 1 -- 172.21.15.107:0/3743722072 shutdown_connections 2024-07-26T12:48:04.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:04.807+0000 7f2df6000640 1 -- 172.21.15.107:0/3743722072 wait complete. 2024-07-26T12:48:05.197 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:04 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3743722072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:04 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3743722072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:04 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3743722072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:05.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:48:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:05 smithi107 ceph-mon[32659]: pgmap v453: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:06.195 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:05 smithi112 ceph-mon[37501]: pgmap v453: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:05 smithi132 ceph-mon[37580]: pgmap v453: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:08.015 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:07 smithi107 ceph-mon[32659]: pgmap v454: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:48:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:07 smithi112 ceph-mon[37501]: pgmap v454: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:48:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:07 smithi132 ceph-mon[37580]: pgmap v454: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:48:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.607+0000 7f8c66200640 1 -- 172.21.15.107:0/1444630507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c601099f0 msgr2=0x7f8c60109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.607+0000 7f8c66200640 1 --2- 172.21.15.107:0/1444630507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c601099f0 0x7f8c60109e00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009900 tx=0x7f8c5402a410 comp rx=0 tx=0).stop 2024-07-26T12:48:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.608+0000 7f8c66200640 1 -- 172.21.15.107:0/1444630507 shutdown_connections 2024-07-26T12:48:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.608+0000 7f8c66200640 1 --2- 172.21.15.107:0/1444630507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c6010fc80 0x7f8c601164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.608+0000 7f8c66200640 1 --2- 172.21.15.107:0/1444630507 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c6010f330 0x7f8c6010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.608+0000 7f8c66200640 1 --2- 172.21.15.107:0/1444630507 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c601099f0 0x7f8c60109e00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.608+0000 7f8c66200640 1 -- 172.21.15.107:0/1444630507 >> 172.21.15.107:0/1444630507 conn(0x7f8c60078ff0 msgr2=0x7f8c60107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.608+0000 7f8c66200640 1 -- 172.21.15.107:0/1444630507 shutdown_connections 2024-07-26T12:48:08.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.608+0000 7f8c66200640 1 -- 172.21.15.107:0/1444630507 wait complete. 2024-07-26T12:48:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.609+0000 7f8c66200640 1 Processor -- start 2024-07-26T12:48:08.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.609+0000 7f8c66200640 1 -- start start 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.609+0000 7f8c66200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c601099f0 0x7f8c601a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.609+0000 7f8c66200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c6010f330 0x7f8c601a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.609+0000 7f8c66200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c6010fc80 0x7f8c601a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.609+0000 7f8c66200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8c60119940 con 0x7f8c6010fc80 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.609+0000 7f8c66200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8c601197c0 con 0x7f8c6010f330 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.609+0000 7f8c66200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8c60119ac0 con 0x7f8c601099f0 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c65800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c6010fc80 0x7f8c601a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c65800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c6010fc80 0x7f8c601a5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50694/0 (socket says 172.21.15.107:50694) 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c65800640 1 -- 172.21.15.107:0/2444815607 learned_addr learned my addr 172.21.15.107:0/2444815607 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c64e00640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c601099f0 0x7f8c601a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c5fe00640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c6010f330 0x7f8c601a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c65800640 1 -- 172.21.15.107:0/2444815607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c601099f0 msgr2=0x7f8c601a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:08.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c65800640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c601099f0 0x7f8c601a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c65800640 1 -- 172.21.15.107:0/2444815607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c6010f330 msgr2=0x7f8c601a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c65800640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c6010f330 0x7f8c601a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c65800640 1 -- 172.21.15.107:0/2444815607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c601a62e0 con 0x7f8c6010fc80 2024-07-26T12:48:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.610+0000 7f8c5fe00640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c6010f330 0x7f8c601a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:08.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.611+0000 7f8c65800640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c6010fc80 0x7f8c601a5bb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8c5800eb10 tx=0x7f8c5800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:08.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.611+0000 7f8c5d600640 1 -- 172.21.15.107:0/2444815607 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c58004b20 con 0x7f8c6010fc80 2024-07-26T12:48:08.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.611+0000 7f8c5d600640 1 -- 172.21.15.107:0/2444815607 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c58004cc0 con 0x7f8c6010fc80 2024-07-26T12:48:08.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.611+0000 7f8c66200640 1 -- 172.21.15.107:0/2444815607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c601a65d0 con 0x7f8c6010fc80 2024-07-26T12:48:08.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.611+0000 7f8c5d600640 1 -- 172.21.15.107:0/2444815607 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c58010670 con 0x7f8c6010fc80 2024-07-26T12:48:08.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.611+0000 7f8c66200640 1 -- 172.21.15.107:0/2444815607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c601acd30 con 0x7f8c6010fc80 2024-07-26T12:48:08.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.612+0000 7f8c66200640 1 -- 172.21.15.107:0/2444815607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c60109e00 con 0x7f8c6010fc80 2024-07-26T12:48:08.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.615+0000 7f8c5d600640 1 -- 172.21.15.107:0/2444815607 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f8c58005020 con 0x7f8c6010fc80 2024-07-26T12:48:08.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.616+0000 7f8c5d600640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8c480b3ca0 0x7f8c480b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:08.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.617+0000 7f8c5d600640 1 -- 172.21.15.107:0/2444815607 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8c580e4670 con 0x7f8c6010fc80 2024-07-26T12:48:08.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.617+0000 7f8c64e00640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8c480b3ca0 0x7f8c480b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:08.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.617+0000 7f8c64e00640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8c480b3ca0 0x7f8c480b61b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8c54031440 tx=0x7f8c54009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:08.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.621+0000 7f8c5d600640 1 -- 172.21.15.107:0/2444815607 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8c580ab7c0 con 0x7f8c6010fc80 2024-07-26T12:48:08.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.822+0000 7f8c66200640 1 -- 172.21.15.107:0/2444815607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c601114c0 con 0x7f8c6010fc80 2024-07-26T12:48:08.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.825+0000 7f8c5d600640 1 -- 172.21.15.107:0/2444815607 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25203 (secure 0 0 0) 0x7f8c580ab0c0 con 0x7f8c6010fc80 2024-07-26T12:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:48:08.828 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:44:36.797948+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:48:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 -- 172.21.15.107:0/2444815607 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8c480b3ca0 msgr2=0x7f8c480b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8c480b3ca0 0x7f8c480b61b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8c54031440 tx=0x7f8c54009560 comp rx=0 tx=0).stop 2024-07-26T12:48:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 -- 172.21.15.107:0/2444815607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c6010fc80 msgr2=0x7f8c601a5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c6010fc80 0x7f8c601a5bb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8c5800eb10 tx=0x7f8c5800c2d0 comp rx=0 tx=0).stop 2024-07-26T12:48:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 -- 172.21.15.107:0/2444815607 shutdown_connections 2024-07-26T12:48:08.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f8c480b3ca0 0x7f8c480b61b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8c6010fc80 0x7f8c601a5bb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8c6010f330 0x7f8c601a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 --2- 172.21.15.107:0/2444815607 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8c601099f0 0x7f8c601a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 -- 172.21.15.107:0/2444815607 >> 172.21.15.107:0/2444815607 conn(0x7f8c60078ff0 msgr2=0x7f8c60107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 -- 172.21.15.107:0/2444815607 shutdown_connections 2024-07-26T12:48:08.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:08.829+0000 7f8c66200640 1 -- 172.21.15.107:0/2444815607 wait complete. 2024-07-26T12:48:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:08 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2444815607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:48:09.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:08 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2444815607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:48:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:08 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2444815607' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:48:09.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T12:43:15.198289+0000', 'modified': '2024-07-26T12:44:43.812461+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24716, 'mds_1': 24715, 'mds_2': 24703, 'mds_3': 24694, 'mds_4': 24682}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24632': {'gid': 24632, 'name': 'k', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6833/2931369116', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6832', 'nonce': 2931369116}, {'type': 'v1', 'addr': '172.21.15.112:6833', 'nonce': 2931369116}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24634': {'gid': 24634, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.132:6833/277912155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6832', 'nonce': 277912155}, {'type': 'v1', 'addr': '172.21.15.132:6833', 'nonce': 277912155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24652': {'gid': 24652, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6835/3403405722', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 3403405722}, {'type': 'v1', 'addr': '172.21.15.107:6835', 'nonce': 3403405722}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24662': {'gid': 24662, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6835/1038904649', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6834', 'nonce': 1038904649}, {'type': 'v1', 'addr': '172.21.15.112:6835', 'nonce': 1038904649}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24680': {'gid': 24680, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.112:6837/3739854241', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6836', 'nonce': 3739854241}, {'type': 'v1', 'addr': '172.21.15.112:6837', 'nonce': 3739854241}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24682': {'gid': 24682, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.132:6835/1242618859', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6834', 'nonce': 1242618859}, {'type': 'v1', 'addr': '172.21.15.132:6835', 'nonce': 1242618859}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24694': {'gid': 24694, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.132:6837/735918310', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6836', 'nonce': 735918310}, {'type': 'v1', 'addr': '172.21.15.132:6837', 'nonce': 735918310}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24703': {'gid': 24703, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 29, 'addr': '172.21.15.132:6839/3489644052', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.132:6838', 'nonce': 3489644052}, {'type': 'v1', 'addr': '172.21.15.132:6839', 'nonce': 3489644052}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24715': {'gid': 24715, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.107:6839/137526780', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 137526780}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 137526780}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24716': {'gid': 24716, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.112:6839/3779803434', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.112:6838', 'nonce': 3779803434}, {'type': 'v1', 'addr': '172.21.15.112:6839', 'nonce': 3779803434}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24716, 'qdb_cluster': [24682, 24694, 24703, 24716, 24715]} 2024-07-26T12:48:09.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T12:48:09.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.f status 2024-07-26T12:48:10.168 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:09 smithi107 ceph-mon[32659]: pgmap v455: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:48:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:09 smithi112 ceph-mon[37501]: pgmap v455: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:48:10.267 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:09 smithi132 ceph-mon[37580]: pgmap v455: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T12:48:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:10 smithi107 ceph-mon[32659]: pgmap v456: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T12:48:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:10 smithi112 ceph-mon[37501]: pgmap v456: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T12:48:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:10 smithi132 ceph-mon[37580]: pgmap v456: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T12:48:12.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.682+0000 7fa92c800640 1 -- 172.21.15.107:0/3156597324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa924075ad0 msgr2=0x7fa92410f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.682+0000 7fa92c800640 1 --2- 172.21.15.107:0/3156597324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa924075ad0 0x7fa92410f640 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa914009900 tx=0x7fa91402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.682+0000 7fa92c800640 1 -- 172.21.15.107:0/3156597324 shutdown_connections 2024-07-26T12:48:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.682+0000 7fa92c800640 1 --2- 172.21.15.107:0/3156597324 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa92410fb80 0x7fa924111fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.682+0000 7fa92c800640 1 --2- 172.21.15.107:0/3156597324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa924075ad0 0x7fa92410f640 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.682+0000 7fa92c800640 1 --2- 172.21.15.107:0/3156597324 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa924075180 0x7fa924075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.682+0000 7fa92c800640 1 -- 172.21.15.107:0/3156597324 >> 172.21.15.107:0/3156597324 conn(0x7fa9240ff010 msgr2=0x7fa924101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:12.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.682+0000 7fa92c800640 1 -- 172.21.15.107:0/3156597324 shutdown_connections 2024-07-26T12:48:12.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.682+0000 7fa92c800640 1 -- 172.21.15.107:0/3156597324 wait complete. 2024-07-26T12:48:12.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.684+0000 7fa92c800640 1 Processor -- start 2024-07-26T12:48:12.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.684+0000 7fa92c800640 1 -- start start 2024-07-26T12:48:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa92c800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa924075180 0x7fa924106550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa92c800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa924075ad0 0x7fa924106a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa92c800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa92410fb80 0x7fa924106fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa92c800640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa924115710 con 0x7fa924075ad0 2024-07-26T12:48:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa92c800640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa924115590 con 0x7fa92410fb80 2024-07-26T12:48:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa92c800640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa924115890 con 0x7fa924075180 2024-07-26T12:48:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa929400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa924075ad0 0x7fa924106a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa929e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa924075180 0x7fa924106550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa929e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa924075180 0x7fa924106550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53002/0 (socket says 172.21.15.107:53002) 2024-07-26T12:48:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa929e00640 1 -- 172.21.15.107:0/2091081775 learned_addr learned my addr 172.21.15.107:0/2091081775 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:12.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa929400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa924075ad0 0x7fa924106a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50710/0 (socket says 172.21.15.107:50710) 2024-07-26T12:48:12.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.685+0000 7fa92a800640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa92410fb80 0x7fa924106fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.686+0000 7fa929e00640 1 -- 172.21.15.107:0/2091081775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa92410fb80 msgr2=0x7fa924106fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:12.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.686+0000 7fa929e00640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa92410fb80 0x7fa924106fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.686+0000 7fa929e00640 1 -- 172.21.15.107:0/2091081775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa924075ad0 msgr2=0x7fa924106a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:12.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.686+0000 7fa929e00640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa924075ad0 0x7fa924106a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.686+0000 7fa929e00640 1 -- 172.21.15.107:0/2091081775 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9241ac550 con 0x7fa924075180 2024-07-26T12:48:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.686+0000 7fa929e00640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa924075180 0x7fa924106550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa91800ea90 tx=0x7fa91800ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.686+0000 7fa91ea00640 1 -- 172.21.15.107:0/2091081775 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa91800ce40 con 0x7fa924075180 2024-07-26T12:48:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.686+0000 7fa91ea00640 1 -- 172.21.15.107:0/2091081775 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9180035f0 con 0x7fa924075180 2024-07-26T12:48:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.686+0000 7fa91ea00640 1 -- 172.21.15.107:0/2091081775 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918010670 con 0x7fa924075180 2024-07-26T12:48:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.687+0000 7fa92c800640 1 -- 172.21.15.107:0/2091081775 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9241ac840 con 0x7fa924075180 2024-07-26T12:48:12.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.687+0000 7fa92c800640 1 -- 172.21.15.107:0/2091081775 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9241acd00 con 0x7fa924075180 2024-07-26T12:48:12.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.690+0000 7fa9333da740 1 Processor -- start 2024-07-26T12:48:12.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.691+0000 7fa9333da740 1 -- start start 2024-07-26T12:48:12.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.691+0000 7fa9333da740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b280570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.691+0000 7fa9333da740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeb20a0 0x56298b280ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.691+0000 7fa9333da740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeba280 0x56298b280ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.691+0000 7fa9333da740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56298b1f6970 con 0x56298aeb20a0 2024-07-26T12:48:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.691+0000 7fa9333da740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56298b1e5e70 con 0x56298aedaf60 2024-07-26T12:48:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.691+0000 7fa9333da740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56298b1e9eb0 con 0x56298aeba280 2024-07-26T12:48:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.691+0000 7fa91ea00640 1 -- 172.21.15.107:0/2091081775 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fa918003aa0 con 0x7fa924075180 2024-07-26T12:48:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.691+0000 7fa929e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeb20a0 0x56298b280ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.691+0000 7fa92a800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b280570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.692+0000 7fa92a800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b280570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:48104/0 (socket says 172.21.15.107:48104) 2024-07-26T12:48:12.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.692+0000 7fa92a800640 1 -- 172.21.15.107:0/2713137494 learned_addr learned my addr 172.21.15.107:0/2713137494 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.692+0000 7fa929400640 1 --2- 172.21.15.107:0/2713137494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeba280 0x56298b280ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.692+0000 7fa92a800640 1 -- 172.21.15.107:0/2713137494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeba280 msgr2=0x56298b280ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.692+0000 7fa92a800640 1 --2- 172.21.15.107:0/2713137494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeba280 0x56298b280ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.692+0000 7fa92a800640 1 -- 172.21.15.107:0/2713137494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeb20a0 msgr2=0x56298b280ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.692+0000 7fa92a800640 1 --2- 172.21.15.107:0/2713137494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeb20a0 0x56298b280ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.692+0000 7fa92a800640 1 -- 172.21.15.107:0/2713137494 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56298b18d110 con 0x56298aedaf60 2024-07-26T12:48:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.692+0000 7fa92a800640 1 --2- 172.21.15.107:0/2713137494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b280570 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa920009fc0 tx=0x7fa920010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f73d9b1d7e32bec5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.693+0000 7fa913000640 1 -- 172.21.15.107:0/2713137494 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa920003b70 con 0x56298aedaf60 2024-07-26T12:48:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.693+0000 7fa913000640 1 -- 172.21.15.107:0/2713137494 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa920003d10 con 0x56298aedaf60 2024-07-26T12:48:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.693+0000 7fa913000640 1 -- 172.21.15.107:0/2713137494 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa920011a40 con 0x56298aedaf60 2024-07-26T12:48:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.694+0000 7fa9333da740 1 -- 172.21.15.107:0/2713137494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 msgr2=0x56298b280570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.694+0000 7fa9333da740 1 --2- 172.21.15.107:0/2713137494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b280570 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa920009fc0 tx=0x7fa920010860 comp rx=0 tx=0).stop 2024-07-26T12:48:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.694+0000 7fa9333da740 1 -- 172.21.15.107:0/2713137494 shutdown_connections 2024-07-26T12:48:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.694+0000 7fa9333da740 1 --2- 172.21.15.107:0/2713137494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeba280 0x56298b280ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.694+0000 7fa9333da740 1 --2- 172.21.15.107:0/2713137494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeb20a0 0x56298b280ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.694+0000 7fa9333da740 1 --2- 172.21.15.107:0/2713137494 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b280570 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.694+0000 7fa9333da740 1 -- 172.21.15.107:0/2713137494 >> 172.21.15.107:0/2713137494 conn(0x56298aecf350 msgr2=0x56298b27e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.694+0000 7fa9333da740 1 -- 172.21.15.107:0/2713137494 shutdown_connections 2024-07-26T12:48:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.694+0000 7fa9333da740 1 -- 172.21.15.107:0/2713137494 wait complete. 2024-07-26T12:48:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.694+0000 7fa91ea00640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa90c0b3b30 0x7fa90c0b6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.695+0000 7fa91ea00640 1 -- 172.21.15.107:0/2091081775 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa9180e4610 con 0x7fa924075180 2024-07-26T12:48:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.695+0000 7fa929400640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa90c0b3b30 0x7fa90c0b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.695+0000 7fa9333da740 1 -- start start 2024-07-26T12:48:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.695+0000 7fa9333da740 1 Processor -- start 2024-07-26T12:48:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa929400640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa90c0b3b30 0x7fa90c0b6040 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa91402a9e0 tx=0x7fa914005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa9333da740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b28be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa9333da740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 0x56298b28c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa9333da740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b2980a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa9333da740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56298b0495f0 con 0x56298aeba280 2024-07-26T12:48:12.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa9333da740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56298b182540 con 0x56298aedaf60 2024-07-26T12:48:12.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa9333da740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56298b28ed50 con 0x56298aeb20a0 2024-07-26T12:48:12.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa9333da740 10 client.0 _ll_register_callbacks cb 0x56298b191f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:12.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa929400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 0x56298b28c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa929e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b28be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa929e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b28be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53016/0 (socket says 172.21.15.107:53016) 2024-07-26T12:48:12.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa92a800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b2980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa929e00640 1 -- 172.21.15.107:0/2297986767 learned_addr learned my addr 172.21.15.107:0/2297986767 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:12.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.696+0000 7fa929400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 0x56298b28c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50718/0 (socket says 172.21.15.107:50718) 2024-07-26T12:48:12.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.697+0000 7fa929e00640 1 -- 172.21.15.107:0/2297986767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 msgr2=0x56298b2980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:12.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.697+0000 7fa929e00640 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b2980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.697+0000 7fa929e00640 1 -- 172.21.15.107:0/2297986767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 msgr2=0x56298b28c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:12.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.697+0000 7fa929e00640 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 0x56298b28c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.697+0000 7fa929e00640 1 -- 172.21.15.107:0/2297986767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56298af46540 con 0x56298aeb20a0 2024-07-26T12:48:12.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.697+0000 7fa929e00640 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b28be20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa9180a70e0 tx=0x7fa9180095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60c24055df1404b5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:12.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.697+0000 7fa907400640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:48:12.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.697+0000 7fa907400640 1 -- 172.21.15.107:0/2297986767 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9180aab70 con 0x56298aeb20a0 2024-07-26T12:48:12.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.697+0000 7fa907400640 1 -- 172.21.15.107:0/2297986767 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa9180a72f0 con 0x56298aeb20a0 2024-07-26T12:48:12.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.697+0000 7fa9333da740 1 -- 172.21.15.107:0/2297986767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa920003d10 con 0x56298aeb20a0 2024-07-26T12:48:12.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.698+0000 7fa907400640 1 -- 172.21.15.107:0/2297986767 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa9180ab760 con 0x56298aeb20a0 2024-07-26T12:48:12.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.698+0000 7fa9333da740 10 client.25321 fetch_fsmap learned FSMap version 28 2024-07-26T12:48:12.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.698+0000 7fa9333da740 1 -- 172.21.15.107:0/2297986767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56298b29ac50 con 0x56298aeb20a0 2024-07-26T12:48:12.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.698+0000 7fa907400640 1 -- 172.21.15.107:0/2297986767 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa920003d10 con 0x56298aeb20a0 2024-07-26T12:48:12.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.698+0000 7fa907400640 1 -- 172.21.15.107:0/2297986767 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa9180e4260 con 0x56298aeb20a0 2024-07-26T12:48:12.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.698+0000 7fa9333da740 10 client.25321 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:48:12.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.698+0000 7fa9333da740 10 client.25321 resolve_mds: resolved name 'f' to daemon mds.f 2024-07-26T12:48:12.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.698+0000 7fa9333da740 20 client.25321 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:12.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.699+0000 7fa9333da740 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x56298ae998e0 0x56298b29e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.699+0000 7fa9333da740 4 client.25321 mds_command: new command op to 24682 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:48:12.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.699+0000 7fa9333da740 1 -- 172.21.15.107:0/2297986767 --> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56298b1f88c0 con 0x56298ae998e0 2024-07-26T12:48:12.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.699+0000 7fa929400640 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x56298ae998e0 0x56298b29e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.700+0000 7fa929400640 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x56298ae998e0 0x56298b29e630 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=65ee12a64a5dc53 server_cookie=91a3ff0884eb72e4 in_seq=0 out_seq=0 2024-07-26T12:48:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.700+0000 7fa907400640 10 client.25321 ms_handle_connect on v2:172.21.15.132:6834/1242618859 2024-07-26T12:48:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.700+0000 7fa907400640 1 -- 172.21.15.107:0/2297986767 <== mds.4 v2:172.21.15.132:6834/1242618859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56298b1f88c0 con 0x56298ae998e0 2024-07-26T12:48:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.700+0000 7fa907400640 10 client.25321 handle_command_reply: tid=0 2024-07-26T12:48:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 client.25321 shutdown 2024-07-26T12:48:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 -- 172.21.15.107:0/2297986767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 msgr2=0x56298b28be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:12.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b28be20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa9180a70e0 tx=0x7fa9180095b0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 -- 172.21.15.107:0/2297986767 shutdown_connections 2024-07-26T12:48:12.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x56298ae998e0 0x56298b29e630 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b2980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 0x56298b28c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 --2- 172.21.15.107:0/2297986767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b28be20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 -- 172.21.15.107:0/2297986767 >> 172.21.15.107:0/2297986767 conn(0x56298aecf350 msgr2=0x56298b283700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:12.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa907400640 0 client.25321 ms_handle_reset on v2:172.21.15.132:6834/1242618859 2024-07-26T12:48:12.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 -- 172.21.15.107:0/2297986767 shutdown_connections 2024-07-26T12:48:12.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 1 -- 172.21.15.107:0/2297986767 wait complete. 2024-07-26T12:48:12.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.701+0000 7fa9333da740 20 client.25321 trim_cache size 0 max 16384 2024-07-26T12:48:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.721+0000 7fa9333da740 1 Processor -- start 2024-07-26T12:48:12.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.721+0000 7fa9333da740 1 -- start start 2024-07-26T12:48:12.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.721+0000 7fa9333da740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aeb20a0 0x56298b286690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.721+0000 7fa9333da740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeba280 0x56298b286ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.721+0000 7fa9333da740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aedaf60 0x56298b298010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.721+0000 7fa9333da740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56298b29c970 con 0x56298aedaf60 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.721+0000 7fa9333da740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56298b29d040 con 0x56298aeb20a0 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.721+0000 7fa9333da740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56298b29d710 con 0x56298aeba280 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.722+0000 7fa929e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aeb20a0 0x56298b286690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.722+0000 7fa929e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aeb20a0 0x56298b286690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:48122/0 (socket says 172.21.15.107:48122) 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.722+0000 7fa929e00640 1 -- 172.21.15.107:0/4081256691 learned_addr learned my addr 172.21.15.107:0/4081256691 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.722+0000 7fa92a800640 1 --2- 172.21.15.107:0/4081256691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aedaf60 0x56298b298010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.722+0000 7fa929400640 1 --2- 172.21.15.107:0/4081256691 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeba280 0x56298b286ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.722+0000 7fa929e00640 1 -- 172.21.15.107:0/4081256691 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeba280 msgr2=0x56298b286ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.722+0000 7fa929e00640 1 --2- 172.21.15.107:0/4081256691 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeba280 0x56298b286ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.722+0000 7fa929e00640 1 -- 172.21.15.107:0/4081256691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aedaf60 msgr2=0x56298b298010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.722+0000 7fa929e00640 1 --2- 172.21.15.107:0/4081256691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aedaf60 0x56298b298010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.722+0000 7fa929e00640 1 -- 172.21.15.107:0/4081256691 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56298b28d0e0 con 0x56298aeb20a0 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.723+0000 7fa929e00640 1 --2- 172.21.15.107:0/4081256691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aeb20a0 0x56298b286690 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa9180aad60 tx=0x7fa918004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9cce14b0c854c74 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.723+0000 7fa912600640 1 -- 172.21.15.107:0/4081256691 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9180aab70 con 0x56298aeb20a0 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.723+0000 7fa912600640 1 -- 172.21.15.107:0/4081256691 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9180ab060 con 0x56298aeb20a0 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.723+0000 7fa912600640 1 -- 172.21.15.107:0/4081256691 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9180e41d0 con 0x56298aeb20a0 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.724+0000 7fa9333da740 1 -- 172.21.15.107:0/4081256691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aeb20a0 msgr2=0x56298b286690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.724+0000 7fa9333da740 1 --2- 172.21.15.107:0/4081256691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aeb20a0 0x56298b286690 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa9180aad60 tx=0x7fa918004360 comp rx=0 tx=0).stop 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.724+0000 7fa9333da740 1 -- 172.21.15.107:0/4081256691 shutdown_connections 2024-07-26T12:48:12.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.724+0000 7fa9333da740 1 --2- 172.21.15.107:0/4081256691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aedaf60 0x56298b298010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.724+0000 7fa9333da740 1 --2- 172.21.15.107:0/4081256691 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeba280 0x56298b286ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.724+0000 7fa9333da740 1 --2- 172.21.15.107:0/4081256691 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aeb20a0 0x56298b286690 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.724+0000 7fa9333da740 1 -- 172.21.15.107:0/4081256691 >> 172.21.15.107:0/4081256691 conn(0x56298aecf350 msgr2=0x56298b29c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.724+0000 7fa9333da740 1 -- 172.21.15.107:0/4081256691 shutdown_connections 2024-07-26T12:48:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.724+0000 7fa9333da740 1 -- 172.21.15.107:0/4081256691 wait complete. 2024-07-26T12:48:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.725+0000 7fa9333da740 1 -- start start 2024-07-26T12:48:12.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.725+0000 7fa9333da740 1 Processor -- start 2024-07-26T12:48:12.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.725+0000 7fa9333da740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b29fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.725+0000 7fa9333da740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 0x56298b2a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.725+0000 7fa9333da740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b3058c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.725+0000 7fa9333da740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56298b29e4b0 con 0x56298aeba280 2024-07-26T12:48:12.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.725+0000 7fa9333da740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56298b29dde0 con 0x56298aedaf60 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.725+0000 7fa9333da740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56298b29eb80 con 0x56298aeb20a0 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.725+0000 7fa9333da740 10 client.0 _ll_register_callbacks cb 0x7fa9180e7d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa929e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b29fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa929400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 0x56298b2a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa929e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b29fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:53032/0 (socket says 172.21.15.107:53032) 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa92a800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b3058c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa929e00640 1 -- 172.21.15.107:0/3499561555 learned_addr learned my addr 172.21.15.107:0/3499561555 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa929e00640 1 -- 172.21.15.107:0/3499561555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 msgr2=0x56298b3058c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa929e00640 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b3058c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa929e00640 1 -- 172.21.15.107:0/3499561555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 msgr2=0x56298b2a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa929e00640 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 0x56298b2a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa929e00640 1 -- 172.21.15.107:0/3499561555 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56298b2a0a70 con 0x56298aeb20a0 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.726+0000 7fa92a800640 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b3058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.727+0000 7fa929e00640 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b29fc60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa9180a82d0 tx=0x7fa918004130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e10071c9445e7808 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.727+0000 7fa907400640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.727+0000 7fa907400640 1 -- 172.21.15.107:0/3499561555 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9180aab70 con 0x56298aeb20a0 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.727+0000 7fa9333da740 1 -- 172.21.15.107:0/3499561555 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9180ab060 con 0x56298aeb20a0 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.727+0000 7fa907400640 1 -- 172.21.15.107:0/3499561555 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa9180c5ba0 con 0x56298aeb20a0 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.727+0000 7fa907400640 1 -- 172.21.15.107:0/3499561555 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa9180aa9b0 con 0x56298aeb20a0 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.727+0000 7fa9333da740 10 client.25327 fetch_fsmap learned FSMap version 28 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.727+0000 7fa9333da740 1 -- 172.21.15.107:0/3499561555 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56298b306e60 con 0x56298aeb20a0 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.727+0000 7fa907400640 1 -- 172.21.15.107:0/3499561555 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9180ab060 con 0x56298aeb20a0 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.728+0000 7fa907400640 1 -- 172.21.15.107:0/3499561555 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fa9180a69d0 con 0x56298aeb20a0 2024-07-26T12:48:12.734 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.728+0000 7fa9333da740 10 client.25327 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.728+0000 7fa9333da740 10 client.25327 resolve_mds: resolved name 'f' to daemon mds.f 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.728+0000 7fa9333da740 20 client.25327 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.728+0000 7fa9333da740 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x56298ae6ec90 0x56298b30a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.728+0000 7fa9333da740 4 client.25327 mds_command: new command op to 24682 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.728+0000 7fa9333da740 1 -- 172.21.15.107:0/3499561555 --> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] -- command(tid 0: {"prefix": "status"}) -- 0x56298b1f4360 con 0x56298ae6ec90 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.728+0000 7fa929400640 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x56298ae6ec90 0x56298b30a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.729+0000 7fa907400640 10 client.25327 ms_handle_connect on v2:172.21.15.132:6834/1242618859 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.729+0000 7fa929400640 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x56298ae6ec90 0x56298b30a0f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dc9f5f4e53da2afc server_cookie=22ff3d96ba4c7833 in_seq=0 out_seq=0 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.729+0000 7fa907400640 1 -- 172.21.15.107:0/3499561555 <== mds.4 v2:172.21.15.132:6834/1242618859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56298b1f4360 con 0x56298ae6ec90 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.729+0000 7fa907400640 10 client.25327 handle_command_reply: tid=0 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.729+0000 7fa9333da740 1 client.25327 shutdown 2024-07-26T12:48:12.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 1 -- 172.21.15.107:0/3499561555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 msgr2=0x56298b29fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b29fc60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa9180a82d0 tx=0x7fa918004130 comp rx=0 tx=0).stop 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 1 -- 172.21.15.107:0/3499561555 shutdown_connections 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x56298ae6ec90 0x56298b30a0f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56298aedaf60 0x56298b3058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56298aeba280 0x56298b2a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa907400640 0 client.25327 ms_handle_reset on v2:172.21.15.132:6834/1242618859 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 1 --2- 172.21.15.107:0/3499561555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56298aeb20a0 0x56298b29fc60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 1 -- 172.21.15.107:0/3499561555 >> 172.21.15.107:0/3499561555 conn(0x56298aecf350 msgr2=0x56298b29c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 1 -- 172.21.15.107:0/3499561555 shutdown_connections 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 1 -- 172.21.15.107:0/3499561555 wait complete. 2024-07-26T12:48:12.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.730+0000 7fa9333da740 20 client.25327 trim_cache size 0 max 16384 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 4, 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24682, 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 213.867901112, 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 90, 2024-07-26T12:48:12.737 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T12:48:12.738 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 335.72394603599997 2024-07-26T12:48:12.738 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:48:12.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 -- 172.21.15.107:0/2091081775 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa90c0b3b30 msgr2=0x7fa90c0b6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:12.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa90c0b3b30 0x7fa90c0b6040 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa91402a9e0 tx=0x7fa914005a20 comp rx=0 tx=0).stop 2024-07-26T12:48:12.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 -- 172.21.15.107:0/2091081775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa924075180 msgr2=0x7fa924106550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:12.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa924075180 0x7fa924106550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa91800ea90 tx=0x7fa91800ef50 comp rx=0 tx=0).stop 2024-07-26T12:48:12.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 -- 172.21.15.107:0/2091081775 shutdown_connections 2024-07-26T12:48:12.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa90c0b3b30 0x7fa90c0b6040 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa92410fb80 0x7fa924106fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa924075ad0 0x7fa924106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 --2- 172.21.15.107:0/2091081775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa924075180 0x7fa924106550 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:12.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 -- 172.21.15.107:0/2091081775 >> 172.21.15.107:0/2091081775 conn(0x7fa9240ff010 msgr2=0x7fa924109690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:12.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 -- 172.21.15.107:0/2091081775 shutdown_connections 2024-07-26T12:48:12.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:12.731+0000 7fa92c800640 1 -- 172.21.15.107:0/2091081775 wait complete. 2024-07-26T12:48:13.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.c status 2024-07-26T12:48:13.988 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:13 smithi107 ceph-mon[32659]: pgmap v457: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-07-26T12:48:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:13 smithi112 ceph-mon[37501]: pgmap v457: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-07-26T12:48:14.216 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:13 smithi132 ceph-mon[37580]: pgmap v457: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-07-26T12:48:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:15 smithi112 ceph-mon[37501]: pgmap v458: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:48:16.224 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:15 smithi107 ceph-mon[32659]: pgmap v458: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:48:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:15 smithi132 ceph-mon[37580]: pgmap v458: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:48:16.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.660+0000 7fed0a200640 1 -- 172.21.15.107:0/2908930547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed041055d0 msgr2=0x7fed041059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:16.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.660+0000 7fed0a200640 1 --2- 172.21.15.107:0/2908930547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed041055d0 0x7fed041059e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fecec009900 tx=0x7fecec02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.660+0000 7fed0a200640 1 -- 172.21.15.107:0/2908930547 shutdown_connections 2024-07-26T12:48:16.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.660+0000 7fed0a200640 1 --2- 172.21.15.107:0/2908930547 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed0410b860 0x7fed04112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.660+0000 7fed0a200640 1 --2- 172.21.15.107:0/2908930547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed0410af10 0x7fed0410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.660+0000 7fed0a200640 1 --2- 172.21.15.107:0/2908930547 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed041055d0 0x7fed041059e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.660+0000 7fed0a200640 1 -- 172.21.15.107:0/2908930547 >> 172.21.15.107:0/2908930547 conn(0x7fed04101040 msgr2=0x7fed04103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:16.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.660+0000 7fed0a200640 1 -- 172.21.15.107:0/2908930547 shutdown_connections 2024-07-26T12:48:16.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.661+0000 7fed0a200640 1 -- 172.21.15.107:0/2908930547 wait complete. 2024-07-26T12:48:16.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.662+0000 7fed0a200640 1 Processor -- start 2024-07-26T12:48:16.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.662+0000 7fed0a200640 1 -- start start 2024-07-26T12:48:16.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fed0a200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed041055d0 0x7fed0419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fed0a200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0410af10 0x7fed0419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fed0a200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed0410b860 0x7fed041a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fed0a200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fed04115530 con 0x7fed0410b860 2024-07-26T12:48:16.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fed0a200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fed041153b0 con 0x7fed0410af10 2024-07-26T12:48:16.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fed0a200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fed041156b0 con 0x7fed041055d0 2024-07-26T12:48:16.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fed08e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed041055d0 0x7fed0419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fed08e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed041055d0 0x7fed0419bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:48208/0 (socket says 172.21.15.107:48208) 2024-07-26T12:48:16.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fed08e00640 1 -- 172.21.15.107:0/4074314861 learned_addr learned my addr 172.21.15.107:0/4074314861 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:16.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fed09800640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed0410b860 0x7fed041a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.663+0000 7fecfbe00640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0410af10 0x7fed0419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.664+0000 7fed08e00640 1 -- 172.21.15.107:0/4074314861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0410af10 msgr2=0x7fed0419c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.664+0000 7fed08e00640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0410af10 0x7fed0419c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.664+0000 7fed08e00640 1 -- 172.21.15.107:0/4074314861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed0410b860 msgr2=0x7fed041a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.664+0000 7fed08e00640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed0410b860 0x7fed041a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.664+0000 7fed08e00640 1 -- 172.21.15.107:0/4074314861 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed041a1e70 con 0x7fed041055d0 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.664+0000 7fed08e00640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed041055d0 0x7fed0419bf00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fecec005940 tx=0x7fecec0040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.664+0000 7fed02000640 1 -- 172.21.15.107:0/4074314861 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec03f030 con 0x7fed041055d0 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.665+0000 7fed0a200640 1 -- 172.21.15.107:0/4074314861 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed041a2100 con 0x7fed041055d0 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.665+0000 7fed02000640 1 -- 172.21.15.107:0/4074314861 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecec0044c0 con 0x7fed041055d0 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.665+0000 7fed02000640 1 -- 172.21.15.107:0/4074314861 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecec035070 con 0x7fed041055d0 2024-07-26T12:48:16.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.665+0000 7fed0a200640 1 -- 172.21.15.107:0/4074314861 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed041a88c0 con 0x7fed041055d0 2024-07-26T12:48:16.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.669+0000 7fed124cc740 1 Processor -- start 2024-07-26T12:48:16.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.669+0000 7fed124cc740 1 -- start start 2024-07-26T12:48:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.669+0000 7fed124cc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a60c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed124cc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a23e0a0 0x56301a60ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed124cc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a60d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed124cc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56301a582970 con 0x56301a266f60 2024-07-26T12:48:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed124cc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56301a571e70 con 0x56301a246280 2024-07-26T12:48:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed124cc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56301a575eb0 con 0x56301a23e0a0 2024-07-26T12:48:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed09800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a60c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed09800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a60c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49556/0 (socket says 172.21.15.107:49556) 2024-07-26T12:48:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed02000640 1 -- 172.21.15.107:0/4074314861 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fecec042030 con 0x7fed041055d0 2024-07-26T12:48:16.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed09800640 1 -- 172.21.15.107:0/1963705867 learned_addr learned my addr 172.21.15.107:0/1963705867 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fecfbe00640 1 --2- 172.21.15.107:0/1963705867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a60d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed08e00640 1 --2- 172.21.15.107:0/1963705867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a23e0a0 0x56301a60ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed09800640 1 -- 172.21.15.107:0/1963705867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a23e0a0 msgr2=0x56301a60ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.670+0000 7fed09800640 1 --2- 172.21.15.107:0/1963705867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a23e0a0 0x56301a60ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.671+0000 7fed09800640 1 -- 172.21.15.107:0/1963705867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 msgr2=0x56301a60d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.671+0000 7fed09800640 1 --2- 172.21.15.107:0/1963705867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a60d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.671+0000 7fed09800640 1 -- 172.21.15.107:0/1963705867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56301a519110 con 0x56301a266f60 2024-07-26T12:48:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.671+0000 7fed09800640 1 --2- 172.21.15.107:0/1963705867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a60c780 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009fd0 tx=0x7fecfc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a27b73482587b8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.671+0000 7fecfa000640 1 -- 172.21.15.107:0/1963705867 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc004d20 con 0x56301a266f60 2024-07-26T12:48:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.671+0000 7fecfa000640 1 -- 172.21.15.107:0/1963705867 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecfc004050 con 0x56301a266f60 2024-07-26T12:48:16.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.671+0000 7fecfa000640 1 -- 172.21.15.107:0/1963705867 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc005240 con 0x56301a266f60 2024-07-26T12:48:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.672+0000 7fed02000640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fece40b3c20 0x7fece40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.672+0000 7fed124cc740 1 -- 172.21.15.107:0/1963705867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 msgr2=0x56301a60c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:16.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.672+0000 7fed124cc740 1 --2- 172.21.15.107:0/1963705867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a60c780 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009fd0 tx=0x7fecfc0118d0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.672+0000 7fed08e00640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fece40b3c20 0x7fece40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.672+0000 7fed02000640 1 -- 172.21.15.107:0/4074314861 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fecec1044a0 con 0x7fed041055d0 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.673+0000 7fed124cc740 1 -- 172.21.15.107:0/1963705867 shutdown_connections 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.673+0000 7fed124cc740 1 --2- 172.21.15.107:0/1963705867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a60d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.673+0000 7fed124cc740 1 --2- 172.21.15.107:0/1963705867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a23e0a0 0x56301a60ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.673+0000 7fed124cc740 1 --2- 172.21.15.107:0/1963705867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a60c780 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.673+0000 7fed124cc740 1 -- 172.21.15.107:0/1963705867 >> 172.21.15.107:0/1963705867 conn(0x56301a25b350 msgr2=0x56301a60a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.673+0000 7fed124cc740 1 -- 172.21.15.107:0/1963705867 shutdown_connections 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.673+0000 7fed124cc740 1 -- 172.21.15.107:0/1963705867 wait complete. 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.673+0000 7fed08e00640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fece40b3c20 0x7fece40b6130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fecec0caca0 tx=0x7fecec0b2040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.674+0000 7fed124cc740 1 -- start start 2024-07-26T12:48:16.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.674+0000 7fed124cc740 1 Processor -- start 2024-07-26T12:48:16.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.674+0000 7fed124cc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a23e0a0 0x56301a617fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.674+0000 7fed124cc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a618550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.674+0000 7fed124cc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a624280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.674+0000 7fed124cc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56301a3d55f0 con 0x56301a266f60 2024-07-26T12:48:16.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.674+0000 7fed124cc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56301a50e540 con 0x56301a246280 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.674+0000 7fed124cc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56301a61af60 con 0x56301a23e0a0 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.674+0000 7fed124cc740 10 client.0 _ll_register_callbacks cb 0x56301a51df10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.675+0000 7fed09800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a23e0a0 0x56301a617fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.675+0000 7fed09800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a624280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.675+0000 7fecfbe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a618550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.675+0000 7fed09800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a624280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49562/0 (socket says 172.21.15.107:49562) 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.675+0000 7fed09800640 1 -- 172.21.15.107:0/1711931315 learned_addr learned my addr 172.21.15.107:0/1711931315 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.675+0000 7fecfbe00640 1 -- 172.21.15.107:0/1711931315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a23e0a0 msgr2=0x56301a617fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.675+0000 7fecfbe00640 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a23e0a0 0x56301a617fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.675+0000 7fecfbe00640 1 -- 172.21.15.107:0/1711931315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 msgr2=0x56301a624280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.675+0000 7fecfbe00640 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a624280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.675+0000 7fecfbe00640 1 -- 172.21.15.107:0/1711931315 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56301a2d2540 con 0x56301a246280 2024-07-26T12:48:16.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.676+0000 7fecfbe00640 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a618550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fed0406bfe0 tx=0x7fecf4012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e449db4dc899605 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.676+0000 7fecf2a00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.676+0000 7fecf2a00640 1 -- 172.21.15.107:0/1711931315 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf400f8f0 con 0x56301a246280 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.676+0000 7fed124cc740 1 -- 172.21.15.107:0/1711931315 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecfc004050 con 0x56301a246280 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.676+0000 7fecf2a00640 1 -- 172.21.15.107:0/1711931315 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fecf400fa90 con 0x56301a246280 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.676+0000 7fecf2a00640 1 -- 172.21.15.107:0/1711931315 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fecf400fc90 con 0x56301a246280 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.676+0000 7fed124cc740 10 client.25331 fetch_fsmap learned FSMap version 28 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.676+0000 7fed124cc740 1 -- 172.21.15.107:0/1711931315 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56301a626ea0 con 0x56301a246280 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.677+0000 7fecf2a00640 1 -- 172.21.15.107:0/1711931315 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecfc004050 con 0x56301a246280 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.677+0000 7fecf2a00640 1 -- 172.21.15.107:0/1711931315 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fecf401c420 con 0x56301a246280 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.677+0000 7fed124cc740 10 client.25331 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.677+0000 7fed124cc740 10 client.25331 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.677+0000 7fed124cc740 20 client.25331 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:16.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.677+0000 7fed124cc740 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x56301a2258e0 0x56301a62a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.677+0000 7fed124cc740 4 client.25331 mds_command: new command op to 24694 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:48:16.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.677+0000 7fed124cc740 1 -- 172.21.15.107:0/1711931315 --> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56301a5848c0 con 0x56301a2258e0 2024-07-26T12:48:16.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.677+0000 7fed09800640 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x56301a2258e0 0x56301a62a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.678+0000 7fed09800640 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x56301a2258e0 0x56301a62a970 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7811ef9c62ce1c25 server_cookie=7d9a15f7a7ca0ac5 in_seq=0 out_seq=0 2024-07-26T12:48:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.678+0000 7fecf2a00640 10 client.25331 ms_handle_connect on v2:172.21.15.132:6836/735918310 2024-07-26T12:48:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fecf2a00640 1 -- 172.21.15.107:0/1711931315 <== mds.3 v2:172.21.15.132:6836/735918310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56301a5848c0 con 0x56301a2258e0 2024-07-26T12:48:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fecf2a00640 10 client.25331 handle_command_reply: tid=0 2024-07-26T12:48:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fed124cc740 1 client.25331 shutdown 2024-07-26T12:48:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fed124cc740 1 -- 172.21.15.107:0/1711931315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 msgr2=0x56301a618550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fed124cc740 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a618550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fed0406bfe0 tx=0x7fecf4012b60 comp rx=0 tx=0).stop 2024-07-26T12:48:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fed124cc740 1 -- 172.21.15.107:0/1711931315 shutdown_connections 2024-07-26T12:48:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fed124cc740 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x56301a2258e0 0x56301a62a970 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fed124cc740 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a266f60 0x56301a624280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fed124cc740 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a618550 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fecf2a00640 0 client.25331 ms_handle_reset on v2:172.21.15.132:6836/735918310 2024-07-26T12:48:16.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fed124cc740 1 --2- 172.21.15.107:0/1711931315 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a23e0a0 0x56301a617fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.679+0000 7fed124cc740 1 -- 172.21.15.107:0/1711931315 >> 172.21.15.107:0/1711931315 conn(0x56301a25b350 msgr2=0x56301a60f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:16.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.680+0000 7fed124cc740 1 -- 172.21.15.107:0/1711931315 shutdown_connections 2024-07-26T12:48:16.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.680+0000 7fed124cc740 1 -- 172.21.15.107:0/1711931315 wait complete. 2024-07-26T12:48:16.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.680+0000 7fed124cc740 20 client.25331 trim_cache size 0 max 16384 2024-07-26T12:48:16.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.699+0000 7fed124cc740 1 Processor -- start 2024-07-26T12:48:16.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.699+0000 7fed124cc740 1 -- start start 2024-07-26T12:48:16.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.700+0000 7fed124cc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a60a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.700+0000 7fed124cc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a6174c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.700+0000 7fed124cc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a266f60 0x56301a6241f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.700+0000 7fed124cc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56301a628cb0 con 0x56301a23e0a0 2024-07-26T12:48:16.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.700+0000 7fed124cc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56301a629380 con 0x56301a246280 2024-07-26T12:48:16.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.700+0000 7fed124cc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56301a629a50 con 0x56301a266f60 2024-07-26T12:48:16.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.700+0000 7fed09800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a60a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.700+0000 7fed09800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a60a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49576/0 (socket says 172.21.15.107:49576) 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.700+0000 7fed09800640 1 -- 172.21.15.107:0/3247558114 learned_addr learned my addr 172.21.15.107:0/3247558114 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.700+0000 7fecfbe00640 1 --2- 172.21.15.107:0/3247558114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a6174c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fed09800640 1 --2- 172.21.15.107:0/3247558114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a266f60 0x56301a6241f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fecfbe00640 1 -- 172.21.15.107:0/3247558114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a266f60 msgr2=0x56301a6241f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fecfbe00640 1 --2- 172.21.15.107:0/3247558114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a266f60 0x56301a6241f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fecfbe00640 1 -- 172.21.15.107:0/3247558114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 msgr2=0x56301a60a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fecfbe00640 1 --2- 172.21.15.107:0/3247558114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a60a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fecfbe00640 1 -- 172.21.15.107:0/3247558114 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56301a613860 con 0x56301a246280 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fed09800640 1 --2- 172.21.15.107:0/3247558114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a60a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fecfbe00640 1 --2- 172.21.15.107:0/3247558114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a6174c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fecf401ca50 tx=0x7fecf400f810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2879ea34164ee34 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fecf9600640 1 -- 172.21.15.107:0/3247558114 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf400fe30 con 0x56301a246280 2024-07-26T12:48:16.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fecf9600640 1 -- 172.21.15.107:0/3247558114 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf400f8f0 con 0x56301a246280 2024-07-26T12:48:16.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.701+0000 7fecf9600640 1 -- 172.21.15.107:0/3247558114 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf4005500 con 0x56301a246280 2024-07-26T12:48:16.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.702+0000 7fed124cc740 1 -- 172.21.15.107:0/3247558114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 msgr2=0x56301a6174c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.702+0000 7fed124cc740 1 --2- 172.21.15.107:0/3247558114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a6174c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fecf401ca50 tx=0x7fecf400f810 comp rx=0 tx=0).stop 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.702+0000 7fed124cc740 1 -- 172.21.15.107:0/3247558114 shutdown_connections 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.702+0000 7fed124cc740 1 --2- 172.21.15.107:0/3247558114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a266f60 0x56301a6241f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.702+0000 7fed124cc740 1 --2- 172.21.15.107:0/3247558114 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a246280 0x56301a6174c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.702+0000 7fed124cc740 1 --2- 172.21.15.107:0/3247558114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a60a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.702+0000 7fed124cc740 1 -- 172.21.15.107:0/3247558114 >> 172.21.15.107:0/3247558114 conn(0x56301a25b350 msgr2=0x56301a609e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.702+0000 7fed124cc740 1 -- 172.21.15.107:0/3247558114 shutdown_connections 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.703+0000 7fed124cc740 1 -- 172.21.15.107:0/3247558114 wait complete. 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.703+0000 7fed124cc740 1 -- start start 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.703+0000 7fed124cc740 1 Processor -- start 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fed124cc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a62bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fed124cc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a246280 0x56301a62c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fed124cc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a266f60 0x56301a631190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fed124cc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56301a62a7f0 con 0x56301a23e0a0 2024-07-26T12:48:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fed124cc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56301a62a120 con 0x56301a266f60 2024-07-26T12:48:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fed124cc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56301a62aec0 con 0x56301a246280 2024-07-26T12:48:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fed124cc740 10 client.0 _ll_register_callbacks cb 0x7fecf4009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fecfbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a246280 0x56301a62c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fed09800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a62bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fecfbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a246280 0x56301a62c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:48234/0 (socket says 172.21.15.107:48234) 2024-07-26T12:48:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fecfbe00640 1 -- 172.21.15.107:0/3216049902 learned_addr learned my addr 172.21.15.107:0/3216049902 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:16.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fed09800640 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a62bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49592/0 (socket says 172.21.15.107:49592) 2024-07-26T12:48:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.704+0000 7fed09800640 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a266f60 0x56301a631190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fecfbe00640 1 -- 172.21.15.107:0/3216049902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a266f60 msgr2=0x56301a631190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fecfbe00640 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a266f60 0x56301a631190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fecfbe00640 1 -- 172.21.15.107:0/3216049902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 msgr2=0x56301a62bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fecfbe00640 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a62bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fecfbe00640 1 -- 172.21.15.107:0/3216049902 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56301a631800 con 0x56301a246280 2024-07-26T12:48:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fed09800640 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a266f60 0x56301a631190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:48:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fecfbe00640 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a246280 0x56301a62c330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fecf401c980 tx=0x7fecf4005050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb1b618638d5238f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:16.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fecf2a00640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:48:16.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fecf2a00640 1 -- 172.21.15.107:0/3216049902 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf400faa0 con 0x56301a246280 2024-07-26T12:48:16.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fed124cc740 1 -- 172.21.15.107:0/3216049902 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecf400f8f0 con 0x56301a246280 2024-07-26T12:48:16.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.705+0000 7fecf2a00640 1 -- 172.21.15.107:0/3216049902 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fecf4018070 con 0x56301a246280 2024-07-26T12:48:16.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.706+0000 7fecf2a00640 1 -- 172.21.15.107:0/3216049902 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fecf403d050 con 0x56301a246280 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.706+0000 7fed124cc740 10 client.25345 fetch_fsmap learned FSMap version 28 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.706+0000 7fed124cc740 1 -- 172.21.15.107:0/3216049902 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56301a632bc0 con 0x56301a246280 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.706+0000 7fecf2a00640 1 -- 172.21.15.107:0/3216049902 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecf400f8f0 con 0x56301a246280 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.706+0000 7fecf2a00640 1 -- 172.21.15.107:0/3216049902 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fecf4003b90 con 0x56301a246280 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.706+0000 7fed124cc740 10 client.25345 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.706+0000 7fed124cc740 10 client.25345 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.706+0000 7fed124cc740 20 client.25345 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.707+0000 7fed124cc740 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x56301a1fac90 0x56301a696270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.707+0000 7fed124cc740 4 client.25345 mds_command: new command op to 24694 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.707+0000 7fed124cc740 1 -- 172.21.15.107:0/3216049902 --> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] -- command(tid 0: {"prefix": "status"}) -- 0x56301a580360 con 0x56301a1fac90 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.707+0000 7fed09800640 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x56301a1fac90 0x56301a696270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:16.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.707+0000 7fed09800640 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x56301a1fac90 0x56301a696270 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=672d12fb63bfe147 server_cookie=6fd25f6d79a65919 in_seq=0 out_seq=0 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.707+0000 7fecf2a00640 10 client.25345 ms_handle_connect on v2:172.21.15.132:6836/735918310 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fecf2a00640 1 -- 172.21.15.107:0/3216049902 <== mds.3 v2:172.21.15.132:6836/735918310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56301a580360 con 0x56301a1fac90 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fecf2a00640 10 client.25345 handle_command_reply: tid=0 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 client.25345 shutdown 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 -- 172.21.15.107:0/3216049902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a246280 msgr2=0x56301a62c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a246280 0x56301a62c330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fecf401c980 tx=0x7fecf4005050 comp rx=0 tx=0).stop 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 -- 172.21.15.107:0/3216049902 shutdown_connections 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x56301a1fac90 0x56301a696270 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56301a266f60 0x56301a631190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56301a246280 0x56301a62c330 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fecf2a00640 0 client.25345 ms_handle_reset on v2:172.21.15.132:6836/735918310 2024-07-26T12:48:16.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 --2- 172.21.15.107:0/3216049902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56301a23e0a0 0x56301a62bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 -- 172.21.15.107:0/3216049902 >> 172.21.15.107:0/3216049902 conn(0x56301a25b350 msgr2=0x56301a608c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 -- 172.21.15.107:0/3216049902 shutdown_connections 2024-07-26T12:48:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 1 -- 172.21.15.107:0/3216049902 wait complete. 2024-07-26T12:48:16.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.708+0000 7fed124cc740 20 client.25345 trim_cache size 0 max 16384 2024-07-26T12:48:16.714 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:48:16.714 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:48:16.714 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 3, 2024-07-26T12:48:16.714 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24694, 2024-07-26T12:48:16.714 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:48:16.715 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:48:16.715 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:48:16.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 218.85463937599999, 2024-07-26T12:48:16.715 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-07-26T12:48:16.715 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 90, 2024-07-26T12:48:16.715 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T12:48:16.715 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 337.01267399900001 2024-07-26T12:48:16.715 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:48:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.709+0000 7fed0a200640 1 -- 172.21.15.107:0/4074314861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fece40b3c20 msgr2=0x7fece40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.709+0000 7fed0a200640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fece40b3c20 0x7fece40b6130 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fecec0caca0 tx=0x7fecec0b2040 comp rx=0 tx=0).stop 2024-07-26T12:48:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.709+0000 7fed0a200640 1 -- 172.21.15.107:0/4074314861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed041055d0 msgr2=0x7fed0419bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.709+0000 7fed0a200640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed041055d0 0x7fed0419bf00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fecec005940 tx=0x7fecec0040c0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.709+0000 7fed0a200640 1 -- 172.21.15.107:0/4074314861 shutdown_connections 2024-07-26T12:48:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.709+0000 7fed0a200640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fece40b3c20 0x7fece40b6130 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.709+0000 7fed0a200640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fed0410b860 0x7fed041a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.709+0000 7fed0a200640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fed0410af10 0x7fed0419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.709+0000 7fed0a200640 1 --2- 172.21.15.107:0/4074314861 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fed041055d0 0x7fed0419bf00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.709+0000 7fed0a200640 1 -- 172.21.15.107:0/4074314861 >> 172.21.15.107:0/4074314861 conn(0x7fed04101040 msgr2=0x7fed04103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.710+0000 7fed0a200640 1 -- 172.21.15.107:0/4074314861 shutdown_connections 2024-07-26T12:48:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:16.710+0000 7fed0a200640 1 -- 172.21.15.107:0/4074314861 wait complete. 2024-07-26T12:48:17.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.i status 2024-07-26T12:48:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:17 smithi107 ceph-mon[32659]: pgmap v459: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:48:18.170 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:17 smithi112 ceph-mon[37501]: pgmap v459: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:48:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:17 smithi132 ceph-mon[37580]: pgmap v459: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T12:48:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:18 smithi112 ceph-mon[37501]: pgmap v460: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:19.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:18 smithi107 ceph-mon[32659]: pgmap v460: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:18 smithi132 ceph-mon[37580]: pgmap v460: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:20.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.625+0000 7fd9f2e00640 1 -- 172.21.15.107:0/602497385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec109650 msgr2=0x7fd9ec10fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:20.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.625+0000 7fd9f2e00640 1 --2- 172.21.15.107:0/602497385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec109650 0x7fd9ec10fe80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009ca0 tx=0x7fd9e402a8c0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.626+0000 7fd9f2e00640 1 -- 172.21.15.107:0/602497385 shutdown_connections 2024-07-26T12:48:20.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.626+0000 7fd9f2e00640 1 --2- 172.21.15.107:0/602497385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec109650 0x7fd9ec10fe80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.626+0000 7fd9f2e00640 1 --2- 172.21.15.107:0/602497385 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9ec108d00 0x7fd9ec109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.626+0000 7fd9f2e00640 1 --2- 172.21.15.107:0/602497385 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec1033c0 0x7fd9ec1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.626+0000 7fd9f2e00640 1 -- 172.21.15.107:0/602497385 >> 172.21.15.107:0/602497385 conn(0x7fd9ec0fee10 msgr2=0x7fd9ec101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:20.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.626+0000 7fd9f2e00640 1 -- 172.21.15.107:0/602497385 shutdown_connections 2024-07-26T12:48:20.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.626+0000 7fd9f2e00640 1 -- 172.21.15.107:0/602497385 wait complete. 2024-07-26T12:48:20.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.627+0000 7fd9f2e00640 1 Processor -- start 2024-07-26T12:48:20.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.627+0000 7fd9f2e00640 1 -- start start 2024-07-26T12:48:20.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.627+0000 7fd9f2e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1033c0 0x7fd9ec19bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.627+0000 7fd9f2e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec108d00 0x7fd9ec19c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.627+0000 7fd9f2e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9ec109650 0x7fd9ec1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f2e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ec113320 con 0x7fd9ec1033c0 2024-07-26T12:48:20.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f2e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ec1131a0 con 0x7fd9ec108d00 2024-07-26T12:48:20.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f2e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ec1134a0 con 0x7fd9ec109650 2024-07-26T12:48:20.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f1a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1033c0 0x7fd9ec19bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f1a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1033c0 0x7fd9ec19bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49612/0 (socket says 172.21.15.107:49612) 2024-07-26T12:48:20.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f1a00640 1 -- 172.21.15.107:0/57625038 learned_addr learned my addr 172.21.15.107:0/57625038 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:20.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f1000640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec108d00 0x7fd9ec19c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f2400640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9ec109650 0x7fd9ec1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f1000640 1 -- 172.21.15.107:0/57625038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9ec109650 msgr2=0x7fd9ec1a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:20.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f1000640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9ec109650 0x7fd9ec1a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f1000640 1 -- 172.21.15.107:0/57625038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1033c0 msgr2=0x7fd9ec19bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:20.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f1000640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1033c0 0x7fd9ec19bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f1000640 1 -- 172.21.15.107:0/57625038 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ec1a1e40 con 0x7fd9ec108d00 2024-07-26T12:48:20.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.628+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1033c0 0x7fd9ec19bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:20.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.629+0000 7fd9f1000640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec108d00 0x7fd9ec19c470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec06c260 tx=0x7fd9d800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:20.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.629+0000 7fd9ea000640 1 -- 172.21.15.107:0/57625038 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d800cc40 con 0x7fd9ec108d00 2024-07-26T12:48:20.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.629+0000 7fd9f2e00640 1 -- 172.21.15.107:0/57625038 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9ec1a2130 con 0x7fd9ec108d00 2024-07-26T12:48:20.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.629+0000 7fd9ea000640 1 -- 172.21.15.107:0/57625038 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9d800cde0 con 0x7fd9ec108d00 2024-07-26T12:48:20.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.629+0000 7fd9ea000640 1 -- 172.21.15.107:0/57625038 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8004a00 con 0x7fd9ec108d00 2024-07-26T12:48:20.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.629+0000 7fd9f2e00640 1 -- 172.21.15.107:0/57625038 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9ec1a8890 con 0x7fd9ec108d00 2024-07-26T12:48:20.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.633+0000 7fd9fb059740 1 Processor -- start 2024-07-26T12:48:20.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.633+0000 7fd9fb059740 1 -- start start 2024-07-26T12:48:20.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.633+0000 7fd9ea000640 1 -- 172.21.15.107:0/57625038 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fd9d8004ba0 con 0x7fd9ec108d00 2024-07-26T12:48:20.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.633+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c027bf60 0x5569c0621760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.633+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c0621ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.633+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c06221e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.633+0000 7fd9fb059740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5569c0597950 con 0x5569c025b280 2024-07-26T12:48:20.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.633+0000 7fd9fb059740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5569c0586e50 con 0x5569c027bf60 2024-07-26T12:48:20.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.633+0000 7fd9fb059740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5569c058ae90 con 0x5569c02530a0 2024-07-26T12:48:20.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.634+0000 7fd9f1000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c06221e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.634+0000 7fd9f1a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c0621ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.634+0000 7fd9f1000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c06221e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49624/0 (socket says 172.21.15.107:49624) 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.634+0000 7fd9f2400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c027bf60 0x5569c0621760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.634+0000 7fd9f1000640 1 -- 172.21.15.107:0/555898227 learned_addr learned my addr 172.21.15.107:0/555898227 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.634+0000 7fd9f1a00640 1 -- 172.21.15.107:0/555898227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c027bf60 msgr2=0x5569c0621760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.634+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/555898227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c027bf60 0x5569c0621760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.634+0000 7fd9f1a00640 1 -- 172.21.15.107:0/555898227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 msgr2=0x5569c06221e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.634+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/555898227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c06221e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.634+0000 7fd9f1a00640 1 -- 172.21.15.107:0/555898227 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569c052e0f0 con 0x5569c02530a0 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.635+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/555898227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c0621ca0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0011510 tx=0x7fd9e00119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2498a562b0ec2ed server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.635+0000 7fd9dea00640 1 -- 172.21.15.107:0/555898227 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e001e070 con 0x5569c02530a0 2024-07-26T12:48:20.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.635+0000 7fd9dea00640 1 -- 172.21.15.107:0/555898227 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e000fb30 con 0x5569c02530a0 2024-07-26T12:48:20.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.635+0000 7fd9dea00640 1 -- 172.21.15.107:0/555898227 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0019b20 con 0x5569c02530a0 2024-07-26T12:48:20.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.635+0000 7fd9ea000640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd9d40b3c20 0x7fd9d40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9ea000640 1 -- 172.21.15.107:0/57625038 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd9d80e40a0 con 0x7fd9ec108d00 2024-07-26T12:48:20.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9f2400640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd9d40b3c20 0x7fd9d40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9fb059740 1 -- 172.21.15.107:0/555898227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 msgr2=0x5569c0621ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:20.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9fb059740 1 --2- 172.21.15.107:0/555898227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c0621ca0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0011510 tx=0x7fd9e00119d0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9fb059740 1 -- 172.21.15.107:0/555898227 shutdown_connections 2024-07-26T12:48:20.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9fb059740 1 --2- 172.21.15.107:0/555898227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c06221e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9fb059740 1 --2- 172.21.15.107:0/555898227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c0621ca0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9fb059740 1 --2- 172.21.15.107:0/555898227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c027bf60 0x5569c0621760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9fb059740 1 -- 172.21.15.107:0/555898227 >> 172.21.15.107:0/555898227 conn(0x5569c0270350 msgr2=0x5569c061f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:20.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9fb059740 1 -- 172.21.15.107:0/555898227 shutdown_connections 2024-07-26T12:48:20.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.636+0000 7fd9fb059740 1 -- 172.21.15.107:0/555898227 wait complete. 2024-07-26T12:48:20.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.637+0000 7fd9f2400640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd9d40b3c20 0x7fd9d40b6130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd9e402f500 tx=0x7fd9e402f310 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:20.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.637+0000 7fd9fb059740 1 -- start start 2024-07-26T12:48:20.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.637+0000 7fd9fb059740 1 Processor -- start 2024-07-26T12:48:20.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c062cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c025b280 0x5569c062d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 0x5569c0639230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9fb059740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5569c03ea5f0 con 0x5569c027bf60 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9fb059740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5569c0523520 con 0x5569c025b280 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9fb059740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5569c062ff40 con 0x5569c02530a0 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9fb059740 10 client.0 _ll_register_callbacks cb 0x5569c0532ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9f1a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c062cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9f2400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 0x5569c0639230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9f1a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c062cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:48256/0 (socket says 172.21.15.107:48256) 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9f1a00640 1 -- 172.21.15.107:0/2356033376 learned_addr learned my addr 172.21.15.107:0/2356033376 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.638+0000 7fd9f1000640 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c025b280 0x5569c062d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.639+0000 7fd9f1a00640 1 -- 172.21.15.107:0/2356033376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c025b280 msgr2=0x5569c062d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.639+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c025b280 0x5569c062d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.639+0000 7fd9f1a00640 1 -- 172.21.15.107:0/2356033376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 msgr2=0x5569c0639230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:20.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.639+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 0x5569c0639230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.639+0000 7fd9f1a00640 1 -- 172.21.15.107:0/2356033376 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569c02e7540 con 0x5569c02530a0 2024-07-26T12:48:20.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.639+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c062cfa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0019700 tx=0x7fd9e000f8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a66a5cbd08363d86 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:20.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.639+0000 7fd9d3400640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:48:20.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.639+0000 7fd9d3400640 1 -- 172.21.15.107:0/2356033376 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e001e070 con 0x5569c02530a0 2024-07-26T12:48:20.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.639+0000 7fd9fb059740 1 -- 172.21.15.107:0/2356033376 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9e000fb30 con 0x5569c02530a0 2024-07-26T12:48:20.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.640+0000 7fd9d3400640 1 -- 172.21.15.107:0/2356033376 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd9e0004250 con 0x5569c02530a0 2024-07-26T12:48:20.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.640+0000 7fd9d3400640 1 -- 172.21.15.107:0/2356033376 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd9e0030050 con 0x5569c02530a0 2024-07-26T12:48:20.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.640+0000 7fd9fb059740 10 client.25357 fetch_fsmap learned FSMap version 28 2024-07-26T12:48:20.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.640+0000 7fd9fb059740 1 -- 172.21.15.107:0/2356033376 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5569c063bde0 con 0x5569c02530a0 2024-07-26T12:48:20.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.640+0000 7fd9d3400640 1 -- 172.21.15.107:0/2356033376 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e000fb30 con 0x5569c02530a0 2024-07-26T12:48:20.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.640+0000 7fd9d3400640 1 -- 172.21.15.107:0/2356033376 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd9e002b030 con 0x5569c02530a0 2024-07-26T12:48:20.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.640+0000 7fd9fb059740 10 client.25357 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:48:20.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.641+0000 7fd9fb059740 10 client.25357 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T12:48:20.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.641+0000 7fd9fb059740 20 client.25357 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:20.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.641+0000 7fd9fb059740 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x5569c023a8e0 0x5569c063f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.641+0000 7fd9fb059740 4 client.25357 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:48:20.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.641+0000 7fd9fb059740 1 -- 172.21.15.107:0/2356033376 --> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569c05998a0 con 0x5569c023a8e0 2024-07-26T12:48:20.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.641+0000 7fd9f1000640 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x5569c023a8e0 0x5569c063f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.642+0000 7fd9f1000640 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x5569c023a8e0 0x5569c063f610 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8eb79a4bb5099a42 server_cookie=24394cb84ad2fda4 in_seq=0 out_seq=0 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.642+0000 7fd9d3400640 10 client.25357 ms_handle_connect on v2:172.21.15.132:6838/3489644052 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9d3400640 1 -- 172.21.15.107:0/2356033376 <== mds.2 v2:172.21.15.132:6838/3489644052 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5569c05998a0 con 0x5569c023a8e0 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9d3400640 10 client.25357 handle_command_reply: tid=0 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 client.25357 shutdown 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 -- 172.21.15.107:0/2356033376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 msgr2=0x5569c062cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c062cfa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0019700 tx=0x7fd9e000f8b0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 -- 172.21.15.107:0/2356033376 shutdown_connections 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 0x5569c0639230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x5569c023a8e0 0x5569c063f610 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c025b280 0x5569c062d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9d3400640 0 client.25357 ms_handle_reset on v2:172.21.15.132:6838/3489644052 2024-07-26T12:48:20.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 --2- 172.21.15.107:0/2356033376 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c062cfa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 -- 172.21.15.107:0/2356033376 >> 172.21.15.107:0/2356033376 conn(0x5569c0270350 msgr2=0x5569c0624940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:20.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 -- 172.21.15.107:0/2356033376 shutdown_connections 2024-07-26T12:48:20.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.643+0000 7fd9fb059740 1 -- 172.21.15.107:0/2356033376 wait complete. 2024-07-26T12:48:20.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.644+0000 7fd9fb059740 20 client.25357 trim_cache size 0 max 16384 2024-07-26T12:48:20.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.663+0000 7fd9fb059740 1 Processor -- start 2024-07-26T12:48:20.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.663+0000 7fd9fb059740 1 -- start start 2024-07-26T12:48:20.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c02530a0 0x5569c063ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c025b280 0x5569c06391a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 0x5569c06396e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9fb059740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5569c063d950 con 0x5569c027bf60 2024-07-26T12:48:20.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9fb059740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5569c063e020 con 0x5569c02530a0 2024-07-26T12:48:20.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9fb059740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5569c063e6f0 con 0x5569c025b280 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9f2400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 0x5569c06396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9f2400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 0x5569c06396e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49644/0 (socket says 172.21.15.107:49644) 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9f2400640 1 -- 172.21.15.107:0/1328602281 learned_addr learned my addr 172.21.15.107:0/1328602281 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c02530a0 0x5569c063ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.664+0000 7fd9f1000640 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c025b280 0x5569c06391a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9f2400640 1 -- 172.21.15.107:0/1328602281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c025b280 msgr2=0x5569c06391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9f2400640 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c025b280 0x5569c06391a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9f2400640 1 -- 172.21.15.107:0/1328602281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c02530a0 msgr2=0x5569c063ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9f2400640 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c02530a0 0x5569c063ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9f2400640 1 -- 172.21.15.107:0/1328602281 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569c061a7d0 con 0x5569c027bf60 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9f1000640 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c025b280 0x5569c06391a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:20.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c02530a0 0x5569c063ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:20.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9f2400640 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 0x5569c06396e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4035000 tx=0x7fd9e4044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290d9e2dd5e1af0f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:20.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9de000640 1 -- 172.21.15.107:0/1328602281 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4003970 con 0x5569c027bf60 2024-07-26T12:48:20.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9de000640 1 -- 172.21.15.107:0/1328602281 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e4036050 con 0x5569c027bf60 2024-07-26T12:48:20.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.665+0000 7fd9de000640 1 -- 172.21.15.107:0/1328602281 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e402f8c0 con 0x5569c027bf60 2024-07-26T12:48:20.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.666+0000 7fd9fb059740 1 -- 172.21.15.107:0/1328602281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 msgr2=0x5569c06396e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:20.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.666+0000 7fd9fb059740 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 0x5569c06396e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4035000 tx=0x7fd9e4044000 comp rx=0 tx=0).stop 2024-07-26T12:48:20.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.666+0000 7fd9fb059740 1 -- 172.21.15.107:0/1328602281 shutdown_connections 2024-07-26T12:48:20.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.666+0000 7fd9fb059740 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c027bf60 0x5569c06396e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.666+0000 7fd9fb059740 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c025b280 0x5569c06391a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.666+0000 7fd9fb059740 1 --2- 172.21.15.107:0/1328602281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c02530a0 0x5569c063ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.666+0000 7fd9fb059740 1 -- 172.21.15.107:0/1328602281 >> 172.21.15.107:0/1328602281 conn(0x5569c0270350 msgr2=0x5569c062b820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.666+0000 7fd9fb059740 1 -- 172.21.15.107:0/1328602281 shutdown_connections 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.666+0000 7fd9fb059740 1 -- 172.21.15.107:0/1328602281 wait complete. 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.667+0000 7fd9fb059740 1 -- start start 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.667+0000 7fd9fb059740 1 Processor -- start 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.667+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c0645760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.667+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c0645d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9fb059740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c027bf60 0x5569c0640f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9fb059740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5569c063f490 con 0x5569c025b280 2024-07-26T12:48:20.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9fb059740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5569c063edc0 con 0x5569c027bf60 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9fb059740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5569c063fb60 con 0x5569c02530a0 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9fb059740 10 client.0 _ll_register_callbacks cb 0x7fd9e0009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f1000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c0645d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f1000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c0645d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49652/0 (socket says 172.21.15.107:49652) 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f1000640 1 -- 172.21.15.107:0/1677596809 learned_addr learned my addr 172.21.15.107:0/1677596809 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c0645760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f2400640 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c027bf60 0x5569c0640f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f1000640 1 -- 172.21.15.107:0/1677596809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 msgr2=0x5569c0645760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f1000640 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c0645760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f1000640 1 -- 172.21.15.107:0/1677596809 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c027bf60 msgr2=0x5569c0640f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:20.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f1000640 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c027bf60 0x5569c0640f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f1000640 1 -- 172.21.15.107:0/1677596809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569c06257e0 con 0x5569c025b280 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.668+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c0645760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.669+0000 7fd9f1000640 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c0645d00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd9d80a6b70 tx=0x7fd9d80053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992801a7be6bdb35 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.669+0000 7fd9d3400640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.669+0000 7fd9d3400640 1 -- 172.21.15.107:0/1677596809 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d801c040 con 0x5569c025b280 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.669+0000 7fd9d3400640 1 -- 172.21.15.107:0/1677596809 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd9d80920d0 con 0x5569c025b280 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.669+0000 7fd9fb059740 1 -- 172.21.15.107:0/1677596809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9e4036050 con 0x5569c025b280 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.669+0000 7fd9d3400640 1 -- 172.21.15.107:0/1677596809 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd9d80aa600 con 0x5569c025b280 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.669+0000 7fd9fb059740 10 client.15216 fetch_fsmap learned FSMap version 28 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.669+0000 7fd9fb059740 1 -- 172.21.15.107:0/1677596809 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5569c0646d30 con 0x5569c025b280 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.670+0000 7fd9d3400640 1 -- 172.21.15.107:0/1677596809 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9e4036050 con 0x5569c025b280 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.670+0000 7fd9d3400640 1 -- 172.21.15.107:0/1677596809 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd9d80a6d80 con 0x5569c025b280 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.670+0000 7fd9fb059740 10 client.15216 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:48:20.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.670+0000 7fd9fb059740 10 client.15216 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T12:48:20.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.670+0000 7fd9fb059740 20 client.15216 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:20.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.670+0000 7fd9fb059740 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x5569c020fc90 0x5569c06ab920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:20.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.670+0000 7fd9fb059740 4 client.15216 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:48:20.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.670+0000 7fd9fb059740 1 -- 172.21.15.107:0/1677596809 --> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] -- command(tid 0: {"prefix": "status"}) -- 0x5569c0595340 con 0x5569c020fc90 2024-07-26T12:48:20.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.670+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x5569c020fc90 0x5569c06ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:20.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.671+0000 7fd9f1a00640 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x5569c020fc90 0x5569c06ab920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a71c572eced0b78b server_cookie=7bc493195e1c665 in_seq=0 out_seq=0 2024-07-26T12:48:20.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.671+0000 7fd9d3400640 10 client.15216 ms_handle_connect on v2:172.21.15.132:6838/3489644052 2024-07-26T12:48:20.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.671+0000 7fd9d3400640 1 -- 172.21.15.107:0/1677596809 <== mds.2 v2:172.21.15.132:6838/3489644052 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5569c0595340 con 0x5569c020fc90 2024-07-26T12:48:20.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.671+0000 7fd9d3400640 10 client.15216 handle_command_reply: tid=0 2024-07-26T12:48:20.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.671+0000 7fd9fb059740 1 client.15216 shutdown 2024-07-26T12:48:20.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.671+0000 7fd9fb059740 1 -- 172.21.15.107:0/1677596809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 msgr2=0x5569c0645d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.671+0000 7fd9fb059740 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c0645d00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd9d80a6b70 tx=0x7fd9d80053f0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.672+0000 7fd9fb059740 1 -- 172.21.15.107:0/1677596809 shutdown_connections 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.672+0000 7fd9fb059740 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x5569c020fc90 0x5569c06ab920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.672+0000 7fd9fb059740 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5569c027bf60 0x5569c0640f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.672+0000 7fd9fb059740 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569c025b280 0x5569c0645d00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.672+0000 7fd9fb059740 1 --2- 172.21.15.107:0/1677596809 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5569c02530a0 0x5569c0645760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.672+0000 7fd9fb059740 1 -- 172.21.15.107:0/1677596809 >> 172.21.15.107:0/1677596809 conn(0x5569c0270350 msgr2=0x5569c062cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.672+0000 7fd9d3400640 0 client.15216 ms_handle_reset on v2:172.21.15.132:6838/3489644052 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.672+0000 7fd9fb059740 1 -- 172.21.15.107:0/1677596809 shutdown_connections 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.672+0000 7fd9fb059740 1 -- 172.21.15.107:0/1677596809 wait complete. 2024-07-26T12:48:20.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.672+0000 7fd9fb059740 20 client.15216 trim_cache size 0 max 16384 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 2, 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24703, 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 223.83197958400001, 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 90, 2024-07-26T12:48:20.681 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T12:48:20.682 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 335.60971938500001 2024-07-26T12:48:20.682 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:48:20.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 -- 172.21.15.107:0/57625038 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd9d40b3c20 msgr2=0x7fd9d40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:20.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd9d40b3c20 0x7fd9d40b6130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd9e402f500 tx=0x7fd9e402f310 comp rx=0 tx=0).stop 2024-07-26T12:48:20.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 -- 172.21.15.107:0/57625038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec108d00 msgr2=0x7fd9ec19c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:20.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec108d00 0x7fd9ec19c470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec06c260 tx=0x7fd9d800ee50 comp rx=0 tx=0).stop 2024-07-26T12:48:20.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 -- 172.21.15.107:0/57625038 shutdown_connections 2024-07-26T12:48:20.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd9d40b3c20 0x7fd9d40b6130 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd9ec109650 0x7fd9ec1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd9ec108d00 0x7fd9ec19c470 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 --2- 172.21.15.107:0/57625038 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9ec1033c0 0x7fd9ec19bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:20.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 -- 172.21.15.107:0/57625038 >> 172.21.15.107:0/57625038 conn(0x7fd9ec0fee10 msgr2=0x7fd9ec108350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:20.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 -- 172.21.15.107:0/57625038 shutdown_connections 2024-07-26T12:48:20.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:20.673+0000 7fd9f2e00640 1 -- 172.21.15.107:0/57625038 wait complete. 2024-07-26T12:48:21.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.g status 2024-07-26T12:48:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:21 smithi107 ceph-mon[32659]: pgmap v461: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:22.138 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:21 smithi112 ceph-mon[37501]: pgmap v461: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:21 smithi132 ceph-mon[37580]: pgmap v461: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:48:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:22 smithi107 ceph-mon[32659]: pgmap v462: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:48:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:48:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:22 smithi112 ceph-mon[37501]: pgmap v462: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:48:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:48:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:22 smithi132 ceph-mon[37580]: pgmap v462: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:48:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.542+0000 7fab60c00640 1 -- 172.21.15.107:0/3417010029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5c10f330 msgr2=0x7fab5c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.542+0000 7fab60c00640 1 --2- 172.21.15.107:0/3417010029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5c10f330 0x7fab5c10f740 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fab44009900 tx=0x7fab4402a400 comp rx=0 tx=0).stop 2024-07-26T12:48:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.542+0000 7fab60c00640 1 -- 172.21.15.107:0/3417010029 shutdown_connections 2024-07-26T12:48:24.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.542+0000 7fab60c00640 1 --2- 172.21.15.107:0/3417010029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab5c10fc80 0x7fab5c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.542+0000 7fab60c00640 1 --2- 172.21.15.107:0/3417010029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5c10f330 0x7fab5c10f740 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.542+0000 7fab60c00640 1 --2- 172.21.15.107:0/3417010029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c1099f0 0x7fab5c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.542+0000 7fab60c00640 1 -- 172.21.15.107:0/3417010029 >> 172.21.15.107:0/3417010029 conn(0x7fab5c078ff0 msgr2=0x7fab5c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.542+0000 7fab60c00640 1 -- 172.21.15.107:0/3417010029 shutdown_connections 2024-07-26T12:48:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.542+0000 7fab60c00640 1 -- 172.21.15.107:0/3417010029 wait complete. 2024-07-26T12:48:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.543+0000 7fab60c00640 1 Processor -- start 2024-07-26T12:48:24.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.544+0000 7fab60c00640 1 -- start start 2024-07-26T12:48:24.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.544+0000 7fab60c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c1099f0 0x7fab5c1a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.544+0000 7fab60c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5c10f330 0x7fab5c1a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.544+0000 7fab60c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab5c10fc80 0x7fab5c1a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.544+0000 7fab60c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c119940 con 0x7fab5c1099f0 2024-07-26T12:48:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.544+0000 7fab60c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c1197c0 con 0x7fab5c10f330 2024-07-26T12:48:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.544+0000 7fab60c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c119ac0 con 0x7fab5c10fc80 2024-07-26T12:48:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab5b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c1099f0 0x7fab5c1a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab53e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5c10f330 0x7fab5c1a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab53e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5c10f330 0x7fab5c1a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44128/0 (socket says 172.21.15.107:44128) 2024-07-26T12:48:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab53e00640 1 -- 172.21.15.107:0/1753462661 learned_addr learned my addr 172.21.15.107:0/1753462661 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:24.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab5be00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab5c10fc80 0x7fab5c1a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab53e00640 1 -- 172.21.15.107:0/1753462661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab5c10fc80 msgr2=0x7fab5c1a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:24.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab53e00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab5c10fc80 0x7fab5c1a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab53e00640 1 -- 172.21.15.107:0/1753462661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c1099f0 msgr2=0x7fab5c1a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:24.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab53e00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c1099f0 0x7fab5c1a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab53e00640 1 -- 172.21.15.107:0/1753462661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c1a62e0 con 0x7fab5c10f330 2024-07-26T12:48:24.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab5be00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab5c10fc80 0x7fab5c1a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:48:24.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.545+0000 7fab53e00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5c10f330 0x7fab5c1a0910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fab4402a920 tx=0x7fab44031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:24.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.546+0000 7fab58c00640 1 -- 172.21.15.107:0/1753462661 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4403f030 con 0x7fab5c10f330 2024-07-26T12:48:24.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.546+0000 7fab58c00640 1 -- 172.21.15.107:0/1753462661 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab44004490 con 0x7fab5c10f330 2024-07-26T12:48:24.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.546+0000 7fab60c00640 1 -- 172.21.15.107:0/1753462661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5c1a6570 con 0x7fab5c10f330 2024-07-26T12:48:24.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.546+0000 7fab58c00640 1 -- 172.21.15.107:0/1753462661 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab44035070 con 0x7fab5c10f330 2024-07-26T12:48:24.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.546+0000 7fab60c00640 1 -- 172.21.15.107:0/1753462661 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5c1acd30 con 0x7fab5c10f330 2024-07-26T12:48:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.550+0000 7fab69034740 1 Processor -- start 2024-07-26T12:48:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.551+0000 7fab69034740 1 -- start start 2024-07-26T12:48:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.551+0000 7fab69034740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 0x564af7614760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.551+0000 7fab69034740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af72460a0 0x564af7614ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.551+0000 7fab69034740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af76151e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.551+0000 7fab69034740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564af758a950 con 0x564af72460a0 2024-07-26T12:48:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.551+0000 7fab69034740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564af7579e50 con 0x564af724e280 2024-07-26T12:48:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.551+0000 7fab69034740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564af757de90 con 0x564af726ef60 2024-07-26T12:48:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.551+0000 7fab58c00640 1 -- 172.21.15.107:0/1753462661 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fab44042030 con 0x7fab5c10f330 2024-07-26T12:48:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.551+0000 7fab5b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af72460a0 0x564af7614ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.551+0000 7fab5be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 0x564af7614760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.552+0000 7fab5be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 0x564af7614760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:54900/0 (socket says 172.21.15.107:54900) 2024-07-26T12:48:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.552+0000 7fab5be00640 1 -- 172.21.15.107:0/3947058399 learned_addr learned my addr 172.21.15.107:0/3947058399 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.552+0000 7fab53e00640 1 --2- 172.21.15.107:0/3947058399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af76151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.552+0000 7fab5be00640 1 -- 172.21.15.107:0/3947058399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 msgr2=0x564af76151e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.552+0000 7fab5be00640 1 --2- 172.21.15.107:0/3947058399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af76151e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.552+0000 7fab5be00640 1 -- 172.21.15.107:0/3947058399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af72460a0 msgr2=0x564af7614ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.552+0000 7fab5be00640 1 --2- 172.21.15.107:0/3947058399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af72460a0 0x564af7614ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.552+0000 7fab5be00640 1 -- 172.21.15.107:0/3947058399 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564af75210f0 con 0x564af726ef60 2024-07-26T12:48:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.552+0000 7fab53e00640 1 --2- 172.21.15.107:0/3947058399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af76151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:48:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.553+0000 7fab5be00640 1 --2- 172.21.15.107:0/3947058399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 0x564af7614760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fab54009fd0 tx=0x7fab54010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f283ba4f826e5460 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.553+0000 7fab4be00640 1 -- 172.21.15.107:0/3947058399 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab54011700 con 0x564af726ef60 2024-07-26T12:48:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.553+0000 7fab4be00640 1 -- 172.21.15.107:0/3947058399 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab54011d50 con 0x564af726ef60 2024-07-26T12:48:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.553+0000 7fab4be00640 1 -- 172.21.15.107:0/3947058399 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5400e3c0 con 0x564af726ef60 2024-07-26T12:48:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab58c00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab3c0b3c20 0x7fab3c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab69034740 1 -- 172.21.15.107:0/3947058399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 msgr2=0x564af7614760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:24.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab69034740 1 --2- 172.21.15.107:0/3947058399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 0x564af7614760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fab54009fd0 tx=0x7fab54010860 comp rx=0 tx=0).stop 2024-07-26T12:48:24.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab58c00640 1 -- 172.21.15.107:0/1753462661 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fab44104180 con 0x7fab5c10f330 2024-07-26T12:48:24.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab69034740 1 -- 172.21.15.107:0/3947058399 shutdown_connections 2024-07-26T12:48:24.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab69034740 1 --2- 172.21.15.107:0/3947058399 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af76151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab69034740 1 --2- 172.21.15.107:0/3947058399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af72460a0 0x564af7614ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab69034740 1 --2- 172.21.15.107:0/3947058399 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 0x564af7614760 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab69034740 1 -- 172.21.15.107:0/3947058399 >> 172.21.15.107:0/3947058399 conn(0x564af7263350 msgr2=0x564af7612650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab5b400640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab3c0b3c20 0x7fab3c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab69034740 1 -- 172.21.15.107:0/3947058399 shutdown_connections 2024-07-26T12:48:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.554+0000 7fab69034740 1 -- 172.21.15.107:0/3947058399 wait complete. 2024-07-26T12:48:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.555+0000 7fab5b400640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab3c0b3c20 0x7fab3c0b6130 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x564af76133e0 tx=0x7fab4c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.555+0000 7fab69034740 1 -- start start 2024-07-26T12:48:24.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.555+0000 7fab69034740 1 Processor -- start 2024-07-26T12:48:24.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab69034740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af72460a0 0x564af7620210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab69034740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af724e280 0x564af7620750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab69034740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af726ef60 0x564af762c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab69034740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564af73dd5f0 con 0x564af724e280 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab69034740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564af7516520 con 0x564af726ef60 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab69034740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564af7622f40 con 0x564af72460a0 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab69034740 10 client.0 _ll_register_callbacks cb 0x564af7525ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab53e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af724e280 0x564af7620750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab53e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af724e280 0x564af7620750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34138/0 (socket says 172.21.15.107:34138) 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab53e00640 1 -- 172.21.15.107:0/2216613405 learned_addr learned my addr 172.21.15.107:0/2216613405 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab5be00640 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af726ef60 0x564af762c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.556+0000 7fab5b400640 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af72460a0 0x564af7620210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab5be00640 1 -- 172.21.15.107:0/2216613405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af72460a0 msgr2=0x564af7620210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab5be00640 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af72460a0 0x564af7620210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab5be00640 1 -- 172.21.15.107:0/2216613405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af724e280 msgr2=0x564af7620750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab5be00640 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af724e280 0x564af7620750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab5be00640 1 -- 172.21.15.107:0/2216613405 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564af72da540 con 0x564af726ef60 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab53e00640 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af724e280 0x564af7620750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab5be00640 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af726ef60 0x564af762c230 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fab54005bf0 tx=0x7fab5401b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42adc1d8b06d1d13 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab48c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab48c00640 1 -- 172.21.15.107:0/2216613405 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab54004c40 con 0x564af726ef60 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab69034740 1 -- 172.21.15.107:0/2216613405 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab54011d50 con 0x564af726ef60 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.557+0000 7fab48c00640 1 -- 172.21.15.107:0/2216613405 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fab54004de0 con 0x564af726ef60 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.558+0000 7fab48c00640 1 -- 172.21.15.107:0/2216613405 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fab54033050 con 0x564af726ef60 2024-07-26T12:48:24.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.558+0000 7fab69034740 10 client.25367 fetch_fsmap learned FSMap version 28 2024-07-26T12:48:24.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.558+0000 7fab69034740 1 -- 172.21.15.107:0/2216613405 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564af762ee40 con 0x564af726ef60 2024-07-26T12:48:24.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.558+0000 7fab48c00640 1 -- 172.21.15.107:0/2216613405 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab54011d50 con 0x564af726ef60 2024-07-26T12:48:24.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.559+0000 7fab48c00640 1 -- 172.21.15.107:0/2216613405 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fab540054d0 con 0x564af726ef60 2024-07-26T12:48:24.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.559+0000 7fab69034740 10 client.25367 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.559+0000 7fab69034740 10 client.25367 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.559+0000 7fab69034740 20 client.25367 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.559+0000 7fab69034740 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x564af722d8e0 0x564af76324f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.559+0000 7fab69034740 4 client.25367 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.559+0000 7fab69034740 1 -- 172.21.15.107:0/2216613405 --> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564af758c8a0 con 0x564af722d8e0 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.559+0000 7fab5b400640 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x564af722d8e0 0x564af76324f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.560+0000 7fab5b400640 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x564af722d8e0 0x564af76324f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b10c22ef63656a87 server_cookie=3d7693caf1e85c06 in_seq=0 out_seq=0 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.560+0000 7fab48c00640 10 client.25367 ms_handle_connect on v2:172.21.15.107:6838/137526780 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.560+0000 7fab48c00640 1 -- 172.21.15.107:0/2216613405 <== mds.1 v2:172.21.15.107:6838/137526780 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564af758c8a0 con 0x564af722d8e0 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.560+0000 7fab48c00640 10 client.25367 handle_command_reply: tid=0 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 client.25367 shutdown 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 -- 172.21.15.107:0/2216613405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af726ef60 msgr2=0x564af762c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:24.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af726ef60 0x564af762c230 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fab54005bf0 tx=0x7fab5401b780 comp rx=0 tx=0).stop 2024-07-26T12:48:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 -- 172.21.15.107:0/2216613405 shutdown_connections 2024-07-26T12:48:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x564af722d8e0 0x564af76324f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af726ef60 0x564af762c230 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af724e280 0x564af7620750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 --2- 172.21.15.107:0/2216613405 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af72460a0 0x564af7620210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab48c00640 0 client.25367 ms_handle_reset on v2:172.21.15.107:6838/137526780 2024-07-26T12:48:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 -- 172.21.15.107:0/2216613405 >> 172.21.15.107:0/2216613405 conn(0x564af7263350 msgr2=0x564af76178f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 -- 172.21.15.107:0/2216613405 shutdown_connections 2024-07-26T12:48:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 1 -- 172.21.15.107:0/2216613405 wait complete. 2024-07-26T12:48:24.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.561+0000 7fab69034740 20 client.25367 trim_cache size 0 max 16384 2024-07-26T12:48:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.581+0000 7fab69034740 1 Processor -- start 2024-07-26T12:48:24.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.581+0000 7fab69034740 1 -- start start 2024-07-26T12:48:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.581+0000 7fab69034740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af72460a0 0x564af7631750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.581+0000 7fab69034740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af761f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.581+0000 7fab69034740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 0x564af762c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.581+0000 7fab69034740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564af7630830 con 0x564af72460a0 2024-07-26T12:48:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.581+0000 7fab69034740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564af7630f00 con 0x564af724e280 2024-07-26T12:48:24.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.581+0000 7fab69034740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564af76315d0 con 0x564af726ef60 2024-07-26T12:48:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab53e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af761f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab53e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af761f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44148/0 (socket says 172.21.15.107:44148) 2024-07-26T12:48:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab53e00640 1 -- 172.21.15.107:0/1669920700 learned_addr learned my addr 172.21.15.107:0/1669920700 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:24.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab5be00640 1 --2- 172.21.15.107:0/1669920700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 0x564af762c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab5b400640 1 --2- 172.21.15.107:0/1669920700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af72460a0 0x564af7631750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab53e00640 1 -- 172.21.15.107:0/1669920700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 msgr2=0x564af762c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab53e00640 1 --2- 172.21.15.107:0/1669920700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 0x564af762c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab53e00640 1 -- 172.21.15.107:0/1669920700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af72460a0 msgr2=0x564af7631750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab53e00640 1 --2- 172.21.15.107:0/1669920700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af72460a0 0x564af7631750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab53e00640 1 -- 172.21.15.107:0/1669920700 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564af760d7d0 con 0x564af724e280 2024-07-26T12:48:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab53e00640 1 --2- 172.21.15.107:0/1669920700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af761f500 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fab440c7ea0 tx=0x7fab4402a400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44fdc137cb3a8de8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.582+0000 7fab4b400640 1 -- 172.21.15.107:0/1669920700 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab44049040 con 0x564af724e280 2024-07-26T12:48:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.583+0000 7fab4b400640 1 -- 172.21.15.107:0/1669920700 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab440ca740 con 0x564af724e280 2024-07-26T12:48:24.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.583+0000 7fab4b400640 1 -- 172.21.15.107:0/1669920700 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab440d3ce0 con 0x564af724e280 2024-07-26T12:48:24.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.583+0000 7fab69034740 1 -- 172.21.15.107:0/1669920700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 msgr2=0x564af761f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:24.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.583+0000 7fab69034740 1 --2- 172.21.15.107:0/1669920700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af761f500 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fab440c7ea0 tx=0x7fab4402a400 comp rx=0 tx=0).stop 2024-07-26T12:48:24.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.583+0000 7fab69034740 1 -- 172.21.15.107:0/1669920700 shutdown_connections 2024-07-26T12:48:24.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.584+0000 7fab69034740 1 --2- 172.21.15.107:0/1669920700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af726ef60 0x564af762c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.584+0000 7fab69034740 1 --2- 172.21.15.107:0/1669920700 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af724e280 0x564af761f500 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.584+0000 7fab69034740 1 --2- 172.21.15.107:0/1669920700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af72460a0 0x564af7631750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.584+0000 7fab69034740 1 -- 172.21.15.107:0/1669920700 >> 172.21.15.107:0/1669920700 conn(0x564af7263350 msgr2=0x564af761e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.584+0000 7fab69034740 1 -- 172.21.15.107:0/1669920700 shutdown_connections 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.584+0000 7fab69034740 1 -- 172.21.15.107:0/1669920700 wait complete. 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.584+0000 7fab69034740 1 -- start start 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.584+0000 7fab69034740 1 Processor -- start 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab69034740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af72460a0 0x564af763ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab69034740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af724e280 0x564af7635200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab69034740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af726ef60 0x564af7635740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab69034740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564af7632370 con 0x564af726ef60 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab69034740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564af7631ca0 con 0x564af72460a0 2024-07-26T12:48:24.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab69034740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564af7632a40 con 0x564af724e280 2024-07-26T12:48:24.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab69034740 10 client.0 _ll_register_callbacks cb 0x7fab54011f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:24.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab5b400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af72460a0 0x564af763ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab5b400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af72460a0 0x564af763ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44164/0 (socket says 172.21.15.107:44164) 2024-07-26T12:48:24.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab5b400640 1 -- 172.21.15.107:0/2211958615 learned_addr learned my addr 172.21.15.107:0/2211958615 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:24.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab5be00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af726ef60 0x564af7635740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.585+0000 7fab53e00640 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af724e280 0x564af7635200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.586+0000 7fab5b400640 1 -- 172.21.15.107:0/2211958615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af724e280 msgr2=0x564af7635200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:24.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.586+0000 7fab5b400640 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af724e280 0x564af7635200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.586+0000 7fab5b400640 1 -- 172.21.15.107:0/2211958615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af726ef60 msgr2=0x564af7635740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.586+0000 7fab5b400640 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af726ef60 0x564af7635740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.586+0000 7fab5b400640 1 -- 172.21.15.107:0/2211958615 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564af7635d50 con 0x564af72460a0 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.586+0000 7fab5b400640 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af72460a0 0x564af763ad10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x564af7632e40 tx=0x7fab4c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41b7cc0e0c8d700e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.586+0000 7fab48c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.586+0000 7fab48c00640 1 -- 172.21.15.107:0/2211958615 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4c02b3b0 con 0x564af72460a0 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.586+0000 7fab69034740 1 -- 172.21.15.107:0/2211958615 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab440ca740 con 0x564af72460a0 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.587+0000 7fab48c00640 1 -- 172.21.15.107:0/2211958615 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fab4c02b550 con 0x564af72460a0 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.587+0000 7fab48c00640 1 -- 172.21.15.107:0/2211958615 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fab4c02b750 con 0x564af72460a0 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.587+0000 7fab69034740 10 client.25373 fetch_fsmap learned FSMap version 28 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.587+0000 7fab69034740 1 -- 172.21.15.107:0/2211958615 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564af769a5b0 con 0x564af72460a0 2024-07-26T12:48:24.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.587+0000 7fab48c00640 1 -- 172.21.15.107:0/2211958615 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab440ca740 con 0x564af72460a0 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.587+0000 7fab48c00640 1 -- 172.21.15.107:0/2211958615 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fab4c01f9e0 con 0x564af72460a0 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.587+0000 7fab69034740 10 client.25373 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.588+0000 7fab69034740 10 client.25373 resolve_mds: resolved name 'g' to daemon mds.g 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.588+0000 7fab69034740 20 client.25373 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.588+0000 7fab69034740 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x564af7202c90 0x564af769de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.588+0000 7fab69034740 4 client.25373 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.588+0000 7fab69034740 1 -- 172.21.15.107:0/2211958615 --> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] -- command(tid 0: {"prefix": "status"}) -- 0x564af7588340 con 0x564af7202c90 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.588+0000 7fab53e00640 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x564af7202c90 0x564af769de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab53e00640 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x564af7202c90 0x564af769de70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d3e1ffebe5622f4 server_cookie=5a15794987470ebf in_seq=0 out_seq=0 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab48c00640 10 client.25373 ms_handle_connect on v2:172.21.15.107:6838/137526780 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab48c00640 1 -- 172.21.15.107:0/2211958615 <== mds.1 v2:172.21.15.107:6838/137526780 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564af7588340 con 0x564af7202c90 2024-07-26T12:48:24.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab48c00640 10 client.25373 handle_command_reply: tid=0 2024-07-26T12:48:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab69034740 1 client.25373 shutdown 2024-07-26T12:48:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab69034740 1 -- 172.21.15.107:0/2211958615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af72460a0 msgr2=0x564af763ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:24.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab69034740 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af72460a0 0x564af763ad10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x564af7632e40 tx=0x7fab4c0046c0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab69034740 1 -- 172.21.15.107:0/2211958615 shutdown_connections 2024-07-26T12:48:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab69034740 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x564af7202c90 0x564af769de70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab69034740 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564af726ef60 0x564af7635740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab69034740 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564af724e280 0x564af7635200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab69034740 1 --2- 172.21.15.107:0/2211958615 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564af72460a0 0x564af763ad10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab48c00640 0 client.25373 ms_handle_reset on v2:172.21.15.107:6838/137526780 2024-07-26T12:48:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.589+0000 7fab69034740 1 -- 172.21.15.107:0/2211958615 >> 172.21.15.107:0/2211958615 conn(0x564af7263350 msgr2=0x564af7617860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.590+0000 7fab69034740 1 -- 172.21.15.107:0/2211958615 shutdown_connections 2024-07-26T12:48:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.590+0000 7fab69034740 1 -- 172.21.15.107:0/2211958615 wait complete. 2024-07-26T12:48:24.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.590+0000 7fab69034740 20 client.25373 trim_cache size 0 max 16384 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 1, 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24715, 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 228.76212908599999, 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 90, 2024-07-26T12:48:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T12:48:24.596 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 335.470470412 2024-07-26T12:48:24.596 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:48:24.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 -- 172.21.15.107:0/1753462661 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab3c0b3c20 msgr2=0x7fab3c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:24.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab3c0b3c20 0x7fab3c0b6130 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x564af76133e0 tx=0x7fab4c005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:24.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 -- 172.21.15.107:0/1753462661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5c10f330 msgr2=0x7fab5c1a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5c10f330 0x7fab5c1a0910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fab4402a920 tx=0x7fab44031a20 comp rx=0 tx=0).stop 2024-07-26T12:48:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 -- 172.21.15.107:0/1753462661 shutdown_connections 2024-07-26T12:48:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fab3c0b3c20 0x7fab3c0b6130 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fab5c10fc80 0x7fab5c1a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fab5c10f330 0x7fab5c1a0910 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 --2- 172.21.15.107:0/1753462661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c1099f0 0x7fab5c1a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 -- 172.21.15.107:0/1753462661 >> 172.21.15.107:0/1753462661 conn(0x7fab5c078ff0 msgr2=0x7fab5c07a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 -- 172.21.15.107:0/1753462661 shutdown_connections 2024-07-26T12:48:24.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:24.591+0000 7fab60c00640 1 -- 172.21.15.107:0/1753462661 wait complete. 2024-07-26T12:48:25.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T12:48:25.982 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:26.134 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:25 smithi107 ceph-mon[32659]: pgmap v463: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:25 smithi112 ceph-mon[37501]: pgmap v463: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:25 smithi132 ceph-mon[37580]: pgmap v463: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:28.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:27 smithi107 ceph-mon[32659]: pgmap v464: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:27 smithi112 ceph-mon[37501]: pgmap v464: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:27 smithi132 ceph-mon[37580]: pgmap v464: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.431+0000 7fd533e00640 1 -- 172.21.15.107:0/3980046185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd534075b30 msgr2=0x7fd53410f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.431+0000 7fd533e00640 1 --2- 172.21.15.107:0/3980046185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd534075b30 0x7fd53410f400 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd51c009900 tx=0x7fd51c02a410 comp rx=0 tx=0).stop 2024-07-26T12:48:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.431+0000 7fd533e00640 1 -- 172.21.15.107:0/3980046185 shutdown_connections 2024-07-26T12:48:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.431+0000 7fd533e00640 1 --2- 172.21.15.107:0/3980046185 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd53410f940 0x7fd534111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.431+0000 7fd533e00640 1 --2- 172.21.15.107:0/3980046185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd534075b30 0x7fd53410f400 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.431+0000 7fd533e00640 1 --2- 172.21.15.107:0/3980046185 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5340751e0 0x7fd5340755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.431+0000 7fd533e00640 1 -- 172.21.15.107:0/3980046185 >> 172.21.15.107:0/3980046185 conn(0x7fd5340fee70 msgr2=0x7fd5341012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.431+0000 7fd533e00640 1 -- 172.21.15.107:0/3980046185 shutdown_connections 2024-07-26T12:48:28.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.431+0000 7fd533e00640 1 -- 172.21.15.107:0/3980046185 wait complete. 2024-07-26T12:48:28.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.433+0000 7fd533e00640 1 Processor -- start 2024-07-26T12:48:28.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.433+0000 7fd533e00640 1 -- start start 2024-07-26T12:48:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.434+0000 7fd533e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5340751e0 0x7fd53419bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.434+0000 7fd533e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd534075b30 0x7fd53419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.434+0000 7fd533e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd53410f940 0x7fd5341a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.434+0000 7fd533e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd5341154e0 con 0x7fd5340751e0 2024-07-26T12:48:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.434+0000 7fd533e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd534115360 con 0x7fd534075b30 2024-07-26T12:48:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.434+0000 7fd533e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd534115660 con 0x7fd53410f940 2024-07-26T12:48:28.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd532a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5340751e0 0x7fd53419bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd52a000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd534075b30 0x7fd53419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd532a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5340751e0 0x7fd53419bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34178/0 (socket says 172.21.15.107:34178) 2024-07-26T12:48:28.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd532a00640 1 -- 172.21.15.107:0/4130030352 learned_addr learned my addr 172.21.15.107:0/4130030352 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:28.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd533400640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd53410f940 0x7fd5341a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd533400640 1 -- 172.21.15.107:0/4130030352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd534075b30 msgr2=0x7fd53419c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd533400640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd534075b30 0x7fd53419c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd533400640 1 -- 172.21.15.107:0/4130030352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5340751e0 msgr2=0x7fd53419bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd533400640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5340751e0 0x7fd53419bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd533400640 1 -- 172.21.15.107:0/4130030352 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5341a1e80 con 0x7fd53410f940 2024-07-26T12:48:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.435+0000 7fd532a00640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5340751e0 0x7fd53419bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.436+0000 7fd533400640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd53410f940 0x7fd5341a1750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00e970 tx=0x7fd52c00ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.436+0000 7fd52be00640 1 -- 172.21.15.107:0/4130030352 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c00ccc0 con 0x7fd53410f940 2024-07-26T12:48:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.436+0000 7fd533e00640 1 -- 172.21.15.107:0/4130030352 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5341a2170 con 0x7fd53410f940 2024-07-26T12:48:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.436+0000 7fd52be00640 1 -- 172.21.15.107:0/4130030352 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd52c005020 con 0x7fd53410f940 2024-07-26T12:48:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.436+0000 7fd52be00640 1 -- 172.21.15.107:0/4130030352 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c004a00 con 0x7fd53410f940 2024-07-26T12:48:28.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.436+0000 7fd533e00640 1 -- 172.21.15.107:0/4130030352 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5341a88d0 con 0x7fd53410f940 2024-07-26T12:48:28.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.439+0000 7fd540810740 1 Processor -- start 2024-07-26T12:48:28.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.440+0000 7fd52be00640 1 -- 172.21.15.107:0/4130030352 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fd52c004ba0 con 0x7fd53410f940 2024-07-26T12:48:28.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.440+0000 7fd540810740 1 -- start start 2024-07-26T12:48:28.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.441+0000 7fd540810740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45be760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.441+0000 7fd540810740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f00a0 0x55f1d45beca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.441+0000 7fd540810740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 0x55f1d45bf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.441+0000 7fd540810740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4534950 con 0x55f1d41f8280 2024-07-26T12:48:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.441+0000 7fd540810740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4523e50 con 0x55f1d4218f60 2024-07-26T12:48:28.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.441+0000 7fd540810740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f1d4527e90 con 0x55f1d41f00a0 2024-07-26T12:48:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.441+0000 7fd52a000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45be760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.441+0000 7fd52a000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45be760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44178/0 (socket says 172.21.15.107:44178) 2024-07-26T12:48:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.441+0000 7fd52a000640 1 -- 172.21.15.107:0/2975645662 learned_addr learned my addr 172.21.15.107:0/2975645662 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.441+0000 7fd52a000640 1 --2- 172.21.15.107:0/2975645662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 0x55f1d45bf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.442+0000 7fd532a00640 1 --2- 172.21.15.107:0/2975645662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f00a0 0x55f1d45beca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.442+0000 7fd532a00640 1 -- 172.21.15.107:0/2975645662 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 msgr2=0x55f1d45be760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.442+0000 7fd532a00640 1 --2- 172.21.15.107:0/2975645662 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45be760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.442+0000 7fd532a00640 1 -- 172.21.15.107:0/2975645662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 msgr2=0x55f1d45bf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.442+0000 7fd532a00640 1 --2- 172.21.15.107:0/2975645662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 0x55f1d45bf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.443 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.442+0000 7fd532a00640 1 -- 172.21.15.107:0/2975645662 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d44cb0f0 con 0x55f1d41f00a0 2024-07-26T12:48:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.442+0000 7fd52a000640 1 --2- 172.21.15.107:0/2975645662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 0x55f1d45bf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.442+0000 7fd52be00640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5180b3c20 0x7fd5180b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.443+0000 7fd532a00640 1 --2- 172.21.15.107:0/2975645662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f00a0 0x55f1d45beca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd524011860 tx=0x7fd524011d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1e5ce3e84fb63d4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.443+0000 7fd52be00640 1 -- 172.21.15.107:0/4130030352 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd52c0e4110 con 0x7fd53410f940 2024-07-26T12:48:28.444 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.443+0000 7fd52a000640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5180b3c20 0x7fd5180b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.443+0000 7fd523e00640 1 -- 172.21.15.107:0/2975645662 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5240112f0 con 0x55f1d41f00a0 2024-07-26T12:48:28.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.443+0000 7fd523e00640 1 -- 172.21.15.107:0/2975645662 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd52400f9a0 con 0x55f1d41f00a0 2024-07-26T12:48:28.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.443+0000 7fd523e00640 1 -- 172.21.15.107:0/2975645662 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524005660 con 0x55f1d41f00a0 2024-07-26T12:48:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.444+0000 7fd52a000640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5180b3c20 0x7fd5180b6130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0096c0 tx=0x7fd51c031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.444+0000 7fd540810740 1 -- 172.21.15.107:0/2975645662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f00a0 msgr2=0x55f1d45beca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.444+0000 7fd540810740 1 --2- 172.21.15.107:0/2975645662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f00a0 0x55f1d45beca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd524011860 tx=0x7fd524011d20 comp rx=0 tx=0).stop 2024-07-26T12:48:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.445+0000 7fd540810740 1 -- 172.21.15.107:0/2975645662 shutdown_connections 2024-07-26T12:48:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.445+0000 7fd540810740 1 --2- 172.21.15.107:0/2975645662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 0x55f1d45bf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.445+0000 7fd540810740 1 --2- 172.21.15.107:0/2975645662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f00a0 0x55f1d45beca0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.445+0000 7fd540810740 1 --2- 172.21.15.107:0/2975645662 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45be760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.445+0000 7fd540810740 1 -- 172.21.15.107:0/2975645662 >> 172.21.15.107:0/2975645662 conn(0x55f1d420d350 msgr2=0x55f1d45bc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.445+0000 7fd540810740 1 -- 172.21.15.107:0/2975645662 shutdown_connections 2024-07-26T12:48:28.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.445+0000 7fd540810740 1 -- 172.21.15.107:0/2975645662 wait complete. 2024-07-26T12:48:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.445+0000 7fd540810740 1 -- start start 2024-07-26T12:48:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.446+0000 7fd540810740 1 Processor -- start 2024-07-26T12:48:28.447 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.446+0000 7fd540810740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.446+0000 7fd540810740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 0x55f1d45ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.446+0000 7fd540810740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45d6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.446+0000 7fd540810740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f1d43875f0 con 0x55f1d41f00a0 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.446+0000 7fd540810740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f1d44c0520 con 0x55f1d4218f60 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.446+0000 7fd540810740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f1d45ccf40 con 0x55f1d41f8280 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.446+0000 7fd540810740 10 client.0 _ll_register_callbacks cb 0x55f1d44cfef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.446+0000 7fd532a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.446+0000 7fd532a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45c9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34194/0 (socket says 172.21.15.107:34194) 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.447+0000 7fd532a00640 1 -- 172.21.15.107:0/1374670872 learned_addr learned my addr 172.21.15.107:0/1374670872 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.447+0000 7fd52a000640 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 0x55f1d45ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.447+0000 7fd533400640 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45d6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.447+0000 7fd532a00640 1 -- 172.21.15.107:0/1374670872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 msgr2=0x55f1d45ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.447+0000 7fd532a00640 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 0x55f1d45ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.447+0000 7fd532a00640 1 -- 172.21.15.107:0/1374670872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 msgr2=0x55f1d45d6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.447+0000 7fd532a00640 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45d6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.447+0000 7fd532a00640 1 -- 172.21.15.107:0/1374670872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d4284540 con 0x55f1d41f00a0 2024-07-26T12:48:28.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.447+0000 7fd532a00640 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45c9f70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd524009a70 tx=0x7fd5240115b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7be46735b10f616 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.447+0000 7fd520c00640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.448+0000 7fd520c00640 1 -- 172.21.15.107:0/1374670872 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524028020 con 0x55f1d41f00a0 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.448+0000 7fd520c00640 1 -- 172.21.15.107:0/1374670872 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd524004c40 con 0x55f1d41f00a0 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.448+0000 7fd540810740 1 -- 172.21.15.107:0/1374670872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd52400f9a0 con 0x55f1d41f00a0 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.448+0000 7fd520c00640 1 -- 172.21.15.107:0/1374670872 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd524033050 con 0x55f1d41f00a0 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.448+0000 7fd540810740 10 client.15246 fetch_fsmap learned FSMap version 28 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.448+0000 7fd540810740 1 -- 172.21.15.107:0/1374670872 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f1d45d8da0 con 0x55f1d41f00a0 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.449+0000 7fd520c00640 1 -- 172.21.15.107:0/1374670872 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd52400f9a0 con 0x55f1d41f00a0 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.449+0000 7fd520c00640 1 -- 172.21.15.107:0/1374670872 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd524018070 con 0x55f1d41f00a0 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.449+0000 7fd540810740 10 client.15246 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:48:28.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.449+0000 7fd540810740 10 client.15246 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T12:48:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.449+0000 7fd540810740 20 client.15246 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.449+0000 7fd540810740 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55f1d41d78e0 0x55f1d45dc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.449+0000 7fd540810740 4 client.15246 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:48:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.449+0000 7fd540810740 1 -- 172.21.15.107:0/1374670872 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1d45368a0 con 0x55f1d41d78e0 2024-07-26T12:48:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.450+0000 7fd52a000640 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55f1d41d78e0 0x55f1d45dc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.450+0000 7fd52a000640 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55f1d41d78e0 0x55f1d45dc650 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9779e62b2a3f196 server_cookie=d06710e0fb7ec1 in_seq=0 out_seq=0 2024-07-26T12:48:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.450+0000 7fd520c00640 10 client.15246 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:48:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.451+0000 7fd520c00640 1 -- 172.21.15.107:0/1374670872 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f1d45368a0 con 0x55f1d41d78e0 2024-07-26T12:48:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.451+0000 7fd520c00640 10 client.15246 handle_command_reply: tid=0 2024-07-26T12:48:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.451+0000 7fd540810740 1 client.15246 shutdown 2024-07-26T12:48:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.451+0000 7fd540810740 1 -- 172.21.15.107:0/1374670872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 msgr2=0x55f1d45c9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:28.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.451+0000 7fd540810740 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45c9f70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd524009a70 tx=0x7fd5240115b0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.451+0000 7fd540810740 1 -- 172.21.15.107:0/1374670872 shutdown_connections 2024-07-26T12:48:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.451+0000 7fd540810740 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55f1d41d78e0 0x55f1d45dc650 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.452+0000 7fd540810740 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45d6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.452+0000 7fd540810740 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 0x55f1d45ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.452+0000 7fd540810740 1 --2- 172.21.15.107:0/1374670872 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45c9f70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.452+0000 7fd540810740 1 -- 172.21.15.107:0/1374670872 >> 172.21.15.107:0/1374670872 conn(0x55f1d420d350 msgr2=0x55f1d45c1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.452+0000 7fd520c00640 0 client.15246 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:48:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.452+0000 7fd540810740 1 -- 172.21.15.107:0/1374670872 shutdown_connections 2024-07-26T12:48:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.452+0000 7fd540810740 1 -- 172.21.15.107:0/1374670872 wait complete. 2024-07-26T12:48:28.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.452+0000 7fd540810740 20 client.15246 trim_cache size 0 max 16384 2024-07-26T12:48:28.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.472+0000 7fd540810740 1 Processor -- start 2024-07-26T12:48:28.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.472+0000 7fd540810740 1 -- start start 2024-07-26T12:48:28.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.472+0000 7fd540810740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d41f00a0 0x55f1d45dbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.472+0000 7fd540810740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 0x55f1d45ca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.472+0000 7fd540810740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d4218f60 0x55f1d45d61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.472+0000 7fd540810740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f1d45da990 con 0x55f1d41f8280 2024-07-26T12:48:28.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.472+0000 7fd540810740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f1d45db060 con 0x55f1d41f00a0 2024-07-26T12:48:28.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.472+0000 7fd540810740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f1d45db730 con 0x55f1d4218f60 2024-07-26T12:48:28.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd532a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d41f00a0 0x55f1d45dbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd52a000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 0x55f1d45ca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd532a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d41f00a0 0x55f1d45dbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44204/0 (socket says 172.21.15.107:44204) 2024-07-26T12:48:28.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd532a00640 1 -- 172.21.15.107:0/1371118138 learned_addr learned my addr 172.21.15.107:0/1371118138 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:28.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd52a000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 0x55f1d45ca430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34202/0 (socket says 172.21.15.107:34202) 2024-07-26T12:48:28.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd533400640 1 --2- 172.21.15.107:0/1371118138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d4218f60 0x55f1d45d61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd532a00640 1 -- 172.21.15.107:0/1371118138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d4218f60 msgr2=0x55f1d45d61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:28.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd532a00640 1 --2- 172.21.15.107:0/1371118138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d4218f60 0x55f1d45d61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd532a00640 1 -- 172.21.15.107:0/1371118138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 msgr2=0x55f1d45ca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:28.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd532a00640 1 --2- 172.21.15.107:0/1371118138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 0x55f1d45ca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd532a00640 1 -- 172.21.15.107:0/1371118138 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1d45bcb90 con 0x55f1d41f00a0 2024-07-26T12:48:28.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.473+0000 7fd532a00640 1 --2- 172.21.15.107:0/1371118138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d41f00a0 0x55f1d45dbf80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd5240247a0 tx=0x7fd524011790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc3ab38d32dcc1cb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:28.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.474+0000 7fd523400640 1 -- 172.21.15.107:0/1371118138 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52403b020 con 0x55f1d41f00a0 2024-07-26T12:48:28.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.474+0000 7fd523400640 1 -- 172.21.15.107:0/1371118138 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd52401f040 con 0x55f1d41f00a0 2024-07-26T12:48:28.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.474+0000 7fd523400640 1 -- 172.21.15.107:0/1371118138 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524028030 con 0x55f1d41f00a0 2024-07-26T12:48:28.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.475+0000 7fd540810740 1 -- 172.21.15.107:0/1371118138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d41f00a0 msgr2=0x55f1d45dbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:28.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.475+0000 7fd540810740 1 --2- 172.21.15.107:0/1371118138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d41f00a0 0x55f1d45dbf80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd5240247a0 tx=0x7fd524011790 comp rx=0 tx=0).stop 2024-07-26T12:48:28.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.476+0000 7fd540810740 1 -- 172.21.15.107:0/1371118138 shutdown_connections 2024-07-26T12:48:28.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.476+0000 7fd540810740 1 --2- 172.21.15.107:0/1371118138 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d4218f60 0x55f1d45d61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.476+0000 7fd540810740 1 --2- 172.21.15.107:0/1371118138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f8280 0x55f1d45ca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.476+0000 7fd540810740 1 --2- 172.21.15.107:0/1371118138 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d41f00a0 0x55f1d45dbf80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.476+0000 7fd540810740 1 -- 172.21.15.107:0/1371118138 >> 172.21.15.107:0/1371118138 conn(0x55f1d420d350 msgr2=0x55f1d45c9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:28.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.476+0000 7fd540810740 1 -- 172.21.15.107:0/1371118138 shutdown_connections 2024-07-26T12:48:28.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.476+0000 7fd540810740 1 -- 172.21.15.107:0/1371118138 wait complete. 2024-07-26T12:48:28.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.476+0000 7fd540810740 1 -- start start 2024-07-26T12:48:28.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.477+0000 7fd540810740 1 Processor -- start 2024-07-26T12:48:28.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.477+0000 7fd540810740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45dda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.477+0000 7fd540810740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 0x55f1d45de010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.477+0000 7fd540810740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45e2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd540810740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f1d45dc4d0 con 0x55f1d41f00a0 2024-07-26T12:48:28.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd540810740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55f1d45dbe00 con 0x55f1d4218f60 2024-07-26T12:48:28.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd540810740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55f1d45dcba0 con 0x55f1d41f8280 2024-07-26T12:48:28.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd540810740 10 client.0 _ll_register_callbacks cb 0x7fd5240093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:28.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd532a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45dda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd532a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45dda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34216/0 (socket says 172.21.15.107:34216) 2024-07-26T12:48:28.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd532a00640 1 -- 172.21.15.107:0/352462345 learned_addr learned my addr 172.21.15.107:0/352462345 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:28.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd52a000640 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 0x55f1d45de010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd533400640 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45e2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd533400640 1 -- 172.21.15.107:0/352462345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 msgr2=0x55f1d45de010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:28.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd533400640 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 0x55f1d45de010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd533400640 1 -- 172.21.15.107:0/352462345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 msgr2=0x55f1d45dda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:28.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd533400640 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45dda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd533400640 1 -- 172.21.15.107:0/352462345 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1d45ca970 con 0x55f1d4218f60 2024-07-26T12:48:28.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd52a000640 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 0x55f1d45de010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:28.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.478+0000 7fd532a00640 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45dda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:28.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.479+0000 7fd533400640 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45e2ee0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00e940 tx=0x7fd52c0a63b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5841ed6b3fc0898 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:28.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.479+0000 7fd520c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:48:28.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.479+0000 7fd520c00640 1 -- 172.21.15.107:0/352462345 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c01c040 con 0x55f1d4218f60 2024-07-26T12:48:28.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.479+0000 7fd520c00640 1 -- 172.21.15.107:0/352462345 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd52c092070 con 0x55f1d4218f60 2024-07-26T12:48:28.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.479+0000 7fd540810740 1 -- 172.21.15.107:0/352462345 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd52401f040 con 0x55f1d4218f60 2024-07-26T12:48:28.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.479+0000 7fd520c00640 1 -- 172.21.15.107:0/352462345 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd52c092270 con 0x55f1d4218f60 2024-07-26T12:48:28.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.479+0000 7fd540810740 10 client.25391 fetch_fsmap learned FSMap version 28 2024-07-26T12:48:28.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.480+0000 7fd540810740 1 -- 172.21.15.107:0/352462345 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f1d45b77d0 con 0x55f1d4218f60 2024-07-26T12:48:28.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.480+0000 7fd520c00640 1 -- 172.21.15.107:0/352462345 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd52401f040 con 0x55f1d4218f60 2024-07-26T12:48:28.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.480+0000 7fd520c00640 1 -- 172.21.15.107:0/352462345 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 28) ==== 8370+0+0 (secure 0 0 0) 0x7fd52c0b3780 con 0x55f1d4218f60 2024-07-26T12:48:28.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.480+0000 7fd540810740 10 client.25391 fetch_fsmap finished waiting for FSMap version 28 2024-07-26T12:48:28.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.480+0000 7fd540810740 10 client.25391 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T12:48:28.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.480+0000 7fd540810740 20 client.25391 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:28.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.480+0000 7fd540810740 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55f1d41acc90 0x55f1d4648000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:28.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.480+0000 7fd540810740 4 client.25391 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T12:48:28.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.480+0000 7fd540810740 1 -- 172.21.15.107:0/352462345 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "status"}) -- 0x55f1d4532340 con 0x55f1d41acc90 2024-07-26T12:48:28.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.481+0000 7fd532a00640 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55f1d41acc90 0x55f1d4648000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:28.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.481+0000 7fd532a00640 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55f1d41acc90 0x55f1d4648000 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=366eade75efc768d server_cookie=a2318550398b2caf in_seq=0 out_seq=0 2024-07-26T12:48:28.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.481+0000 7fd520c00640 10 client.25391 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:48:28.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd520c00640 1 -- 172.21.15.107:0/352462345 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55f1d4532340 con 0x55f1d41acc90 2024-07-26T12:48:28.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd520c00640 10 client.25391 handle_command_reply: tid=0 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 client.25391 shutdown 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 -- 172.21.15.107:0/352462345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 msgr2=0x55f1d45e2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45e2ee0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00e940 tx=0x7fd52c0a63b0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 -- 172.21.15.107:0/352462345 shutdown_connections 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55f1d41acc90 0x55f1d4648000 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55f1d4218f60 0x55f1d45e2ee0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55f1d41f8280 0x55f1d45de010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd520c00640 0 client.25391 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 --2- 172.21.15.107:0/352462345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1d41f00a0 0x55f1d45dda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 -- 172.21.15.107:0/352462345 >> 172.21.15.107:0/352462345 conn(0x55f1d420d350 msgr2=0x55f1d45bf790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 -- 172.21.15.107:0/352462345 shutdown_connections 2024-07-26T12:48:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.482+0000 7fd540810740 1 -- 172.21.15.107:0/352462345 wait complete. 2024-07-26T12:48:28.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.483+0000 7fd540810740 20 client.25391 trim_cache size 0 max 16384 2024-07-26T12:48:28.490 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:48:28.490 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_fsid": "14de5e40-4b4b-11ef-bcb7-c7b262605968", 2024-07-26T12:48:28.490 INFO:teuthology.orchestra.run.smithi107.stdout: "whoami": 0, 2024-07-26T12:48:28.490 INFO:teuthology.orchestra.run.smithi107.stdout: "id": 24716, 2024-07-26T12:48:28.490 INFO:teuthology.orchestra.run.smithi107.stdout: "want_state": "up:active", 2024-07-26T12:48:28.490 INFO:teuthology.orchestra.run.smithi107.stdout: "state": "up:active", 2024-07-26T12:48:28.490 INFO:teuthology.orchestra.run.smithi107.stdout: "fs_name": "cephfs", 2024-07-26T12:48:28.491 INFO:teuthology.orchestra.run.smithi107.stdout: "rank_uptime": 312.28587846300002, 2024-07-26T12:48:28.491 INFO:teuthology.orchestra.run.smithi107.stdout: "mdsmap_epoch": 28, 2024-07-26T12:48:28.491 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch": 90, 2024-07-26T12:48:28.491 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T12:48:28.491 INFO:teuthology.orchestra.run.smithi107.stdout: "uptime": 331.48560557399998 2024-07-26T12:48:28.491 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.483+0000 7fd533e00640 1 -- 172.21.15.107:0/4130030352 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5180b3c20 msgr2=0x7fd5180b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.483+0000 7fd533e00640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5180b3c20 0x7fd5180b6130 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd51c0096c0 tx=0x7fd51c031400 comp rx=0 tx=0).stop 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.483+0000 7fd533e00640 1 -- 172.21.15.107:0/4130030352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd53410f940 msgr2=0x7fd5341a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.483+0000 7fd533e00640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd53410f940 0x7fd5341a1750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd52c00e970 tx=0x7fd52c00ee30 comp rx=0 tx=0).stop 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.483+0000 7fd533e00640 1 -- 172.21.15.107:0/4130030352 shutdown_connections 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.484+0000 7fd533e00640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd5180b3c20 0x7fd5180b6130 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.484+0000 7fd533e00640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd53410f940 0x7fd5341a1750 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.484+0000 7fd533e00640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd534075b30 0x7fd53419c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.484+0000 7fd533e00640 1 --2- 172.21.15.107:0/4130030352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5340751e0 0x7fd53419bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.484+0000 7fd533e00640 1 -- 172.21.15.107:0/4130030352 >> 172.21.15.107:0/4130030352 conn(0x7fd5340fee70 msgr2=0x7fd5341094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:28.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.484+0000 7fd533e00640 1 -- 172.21.15.107:0/4130030352 shutdown_connections 2024-07-26T12:48:28.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:28.484+0000 7fd533e00640 1 -- 172.21.15.107:0/4130030352 wait complete. 2024-07-26T12:48:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:28 smithi112 ceph-mon[37501]: pgmap v465: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:28 smithi107 ceph-mon[32659]: pgmap v465: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:28 smithi132 ceph-mon[37580]: pgmap v465: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:29.523 INFO:teuthology.run_tasks:Running task sequential... 2024-07-26T12:48:29.533 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-07-26T12:48:29.533 INFO:teuthology.task.sequential:In sequential, running task print... 2024-07-26T12:48:29.534 INFO:teuthology.task.print:Enabling mgr modules 2024-07-26T12:48:29.534 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-07-26T12:48:29.535 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T12:48:29.535 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-07-26T12:48:29.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-07-26T12:48:29.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.790+0000 7f9435400640 1 -- 172.21.15.107:0/2787259262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9430104ae0 msgr2=0x7f943010d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:29.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.790+0000 7f9435400640 1 --2- 172.21.15.107:0/2787259262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9430104ae0 0x7f943010d020 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f941c009900 tx=0x7f941c02a410 comp rx=0 tx=0).stop 2024-07-26T12:48:29.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.790+0000 7f9435400640 1 -- 172.21.15.107:0/2787259262 shutdown_connections 2024-07-26T12:48:29.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.790+0000 7f9435400640 1 --2- 172.21.15.107:0/2787259262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f943010d5f0 0x7f943010fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:29.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.791+0000 7f9435400640 1 --2- 172.21.15.107:0/2787259262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9430104ae0 0x7f943010d020 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:29.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.791+0000 7f9435400640 1 --2- 172.21.15.107:0/2787259262 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9430104190 0x7f94301045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:29.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.791+0000 7f9435400640 1 -- 172.21.15.107:0/2787259262 >> 172.21.15.107:0/2787259262 conn(0x7f94300fdc60 msgr2=0x7f94301000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:29.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.791+0000 7f9435400640 1 -- 172.21.15.107:0/2787259262 shutdown_connections 2024-07-26T12:48:29.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.791+0000 7f9435400640 1 -- 172.21.15.107:0/2787259262 wait complete. 2024-07-26T12:48:29.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.792+0000 7f9435400640 1 Processor -- start 2024-07-26T12:48:29.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.792+0000 7f9435400640 1 -- start start 2024-07-26T12:48:29.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.792+0000 7f9435400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9430104190 0x7f9430199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:29.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.792+0000 7f9435400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9430104ae0 0x7f943019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:29.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.792+0000 7f9435400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f943010d5f0 0x7f943019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:29.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.792+0000 7f9435400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9430113240 con 0x7f9430104190 2024-07-26T12:48:29.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.792+0000 7f9435400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f94301130c0 con 0x7f943010d5f0 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.792+0000 7f9435400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f94301133c0 con 0x7f9430104ae0 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f942fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9430104190 0x7f9430199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f9434a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f943010d5f0 0x7f943019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f942fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9430104190 0x7f9430199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34228/0 (socket says 172.21.15.107:34228) 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f942fe00640 1 -- 172.21.15.107:0/3087618851 learned_addr learned my addr 172.21.15.107:0/3087618851 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f942f400640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9430104ae0 0x7f943019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f942fe00640 1 -- 172.21.15.107:0/3087618851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9430104ae0 msgr2=0x7f943019a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f942fe00640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9430104ae0 0x7f943019a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f942fe00640 1 -- 172.21.15.107:0/3087618851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f943010d5f0 msgr2=0x7f943019f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f942fe00640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f943010d5f0 0x7f943019f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f942fe00640 1 -- 172.21.15.107:0/3087618851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943019fcb0 con 0x7f9430104190 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f9434a00640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f943010d5f0 0x7f943019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:29.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.793+0000 7f942fe00640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9430104190 0x7f9430199d80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f942000e9f0 tx=0x7f942000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:29.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.794+0000 7f942cc00640 1 -- 172.21.15.107:0/3087618851 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942000cd80 con 0x7f9430104190 2024-07-26T12:48:29.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.794+0000 7f942cc00640 1 -- 172.21.15.107:0/3087618851 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94200108e0 con 0x7f9430104190 2024-07-26T12:48:29.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.794+0000 7f942cc00640 1 -- 172.21.15.107:0/3087618851 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420018980 con 0x7f9430104190 2024-07-26T12:48:29.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.794+0000 7f9435400640 1 -- 172.21.15.107:0/3087618851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f943019ffa0 con 0x7f9430104190 2024-07-26T12:48:29.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.794+0000 7f9435400640 1 -- 172.21.15.107:0/3087618851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94301a6730 con 0x7f9430104190 2024-07-26T12:48:29.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.795+0000 7f9435400640 1 -- 172.21.15.107:0/3087618851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94301045a0 con 0x7f9430104190 2024-07-26T12:48:29.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.799+0000 7f942cc00640 1 -- 172.21.15.107:0/3087618851 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f9420010430 con 0x7f9430104190 2024-07-26T12:48:29.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.801+0000 7f942cc00640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f94140b3c20 0x7f94140b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:29.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.801+0000 7f942cc00640 1 -- 172.21.15.107:0/3087618851 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f94200e4930 con 0x7f9430104190 2024-07-26T12:48:29.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.801+0000 7f942f400640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f94140b3c20 0x7f94140b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:29.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.802+0000 7f942f400640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f94140b3c20 0x7f94140b6130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f941c009230 tx=0x7f941c031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:29.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:29.803+0000 7f942cc00640 1 -- 172.21.15.107:0/3087618851 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f94200ab430 con 0x7f9430104190 2024-07-26T12:48:30.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.047+0000 7f9435400640 1 -- 172.21.15.107:0/3087618851 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f94301a6aa0 con 0x7f9430104190 2024-07-26T12:48:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.105+0000 7f942cc00640 1 -- 172.21.15.107:0/3087618851 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 module 'snap_schedule' is already enabled v23) ==== 133+0+0 (secure 0 0 0) 0x7f94200af980 con 0x7f9430104190 2024-07-26T12:48:30.106 INFO:teuthology.orchestra.run.smithi107.stderr:module 'snap_schedule' is already enabled 2024-07-26T12:48:30.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.108+0000 7f9435400640 1 -- 172.21.15.107:0/3087618851 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f94140b3c20 msgr2=0x7f94140b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:30.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.108+0000 7f9435400640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f94140b3c20 0x7f94140b6130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f941c009230 tx=0x7f941c031400 comp rx=0 tx=0).stop 2024-07-26T12:48:30.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.108+0000 7f9435400640 1 -- 172.21.15.107:0/3087618851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9430104190 msgr2=0x7f9430199d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:30.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.108+0000 7f9435400640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9430104190 0x7f9430199d80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f942000e9f0 tx=0x7f942000eeb0 comp rx=0 tx=0).stop 2024-07-26T12:48:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.109+0000 7f9435400640 1 -- 172.21.15.107:0/3087618851 shutdown_connections 2024-07-26T12:48:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.109+0000 7f9435400640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f94140b3c20 0x7f94140b6130 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.109+0000 7f9435400640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f943010d5f0 0x7f943019f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.109+0000 7f9435400640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9430104ae0 0x7f943019a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.109+0000 7f9435400640 1 --2- 172.21.15.107:0/3087618851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9430104190 0x7f9430199d80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.109+0000 7f9435400640 1 -- 172.21.15.107:0/3087618851 >> 172.21.15.107:0/3087618851 conn(0x7f94300fdc60 msgr2=0x7f9430105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:30.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.109+0000 7f9435400640 1 -- 172.21.15.107:0/3087618851 shutdown_connections 2024-07-26T12:48:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:30.109+0000 7f9435400640 1 -- 172.21.15.107:0/3087618851 wait complete. 2024-07-26T12:48:30.146 INFO:teuthology.run_tasks:Running task kclient... 2024-07-26T12:48:30.161 INFO:tasks.kclient:Mounting kernel clients... 2024-07-26T12:48:30.161 INFO:tasks.kclient:config is {'client.0': {}} 2024-07-26T12:48:30.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:48:30.162 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-07-26T12:48:30.212 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-07-26T12:48:30.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:48:30.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T12:48:30.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:48:30.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link delete ceph-brx 2024-07-26T12:48:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:30 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3087618851' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T12:48:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:30 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3087618851' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T12:48:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:30 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3087618851' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T12:48:30.653 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-07-26T12:48:30.653 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T12:48:30.653 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T12:48:30.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs ls 2024-07-26T12:48:31.089 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:31 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3087618851' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T12:48:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:31 smithi112 ceph-mon[37501]: mgrmap e23: x(active, since 12m), standbys: z, y 2024-07-26T12:48:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:31 smithi112 ceph-mon[37501]: pgmap v466: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:31.482 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:31 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3087618851' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T12:48:31.482 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:31 smithi107 ceph-mon[32659]: mgrmap e23: x(active, since 12m), standbys: z, y 2024-07-26T12:48:31.482 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:31 smithi107 ceph-mon[32659]: pgmap v466: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:31 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3087618851' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T12:48:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:31 smithi132 ceph-mon[37580]: mgrmap e23: x(active, since 12m), standbys: z, y 2024-07-26T12:48:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:31 smithi132 ceph-mon[37580]: pgmap v466: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:33.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.575+0000 7f96ed600640 1 -- 172.21.15.107:0/3281850178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96e8108d00 msgr2=0x7f96e8109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:33.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.575+0000 7f96ed600640 1 --2- 172.21.15.107:0/3281850178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96e8108d00 0x7f96e8109110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009900 tx=0x7f96d002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:48:33.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.575+0000 7f96ed600640 1 -- 172.21.15.107:0/3281850178 shutdown_connections 2024-07-26T12:48:33.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.575+0000 7f96ed600640 1 --2- 172.21.15.107:0/3281850178 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f96e8109650 0x7f96e810fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:33.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.575+0000 7f96ed600640 1 --2- 172.21.15.107:0/3281850178 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96e8108d00 0x7f96e8109110 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:33.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.575+0000 7f96ed600640 1 --2- 172.21.15.107:0/3281850178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e81033c0 0x7f96e81037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:33.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.575+0000 7f96ed600640 1 -- 172.21.15.107:0/3281850178 >> 172.21.15.107:0/3281850178 conn(0x7f96e80fee10 msgr2=0x7f96e8101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:33.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.575+0000 7f96ed600640 1 -- 172.21.15.107:0/3281850178 shutdown_connections 2024-07-26T12:48:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.575+0000 7f96ed600640 1 -- 172.21.15.107:0/3281850178 wait complete. 2024-07-26T12:48:33.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.576+0000 7f96ed600640 1 Processor -- start 2024-07-26T12:48:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96ed600640 1 -- start start 2024-07-26T12:48:33.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96ed600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e81033c0 0x7f96e819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96ed600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f96e8108d00 0x7f96e819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96ed600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96e8109650 0x7f96e81a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96ed600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f96e8113310 con 0x7f96e81033c0 2024-07-26T12:48:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96ed600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f96e8113190 con 0x7f96e8109650 2024-07-26T12:48:33.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96ed600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f96e8113490 con 0x7f96e8108d00 2024-07-26T12:48:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96e7e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e81033c0 0x7f96e819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96e7e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e81033c0 0x7f96e819bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:34246/0 (socket says 172.21.15.107:34246) 2024-07-26T12:48:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96e7e00640 1 -- 172.21.15.107:0/4011845945 learned_addr learned my addr 172.21.15.107:0/4011845945 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.577+0000 7f96df400640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f96e8108d00 0x7f96e819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:33.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.578+0000 7f96ecc00640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96e8109650 0x7f96e81a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.578+0000 7f96e7e00640 1 -- 172.21.15.107:0/4011845945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f96e8108d00 msgr2=0x7f96e819c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.578+0000 7f96e7e00640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f96e8108d00 0x7f96e819c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.578+0000 7f96e7e00640 1 -- 172.21.15.107:0/4011845945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96e8109650 msgr2=0x7f96e81a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.578+0000 7f96e7e00640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96e8109650 0x7f96e81a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:33.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.578+0000 7f96e7e00640 1 -- 172.21.15.107:0/4011845945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e81a1e90 con 0x7f96e81033c0 2024-07-26T12:48:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.578+0000 7f96e7e00640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e81033c0 0x7f96e819bf20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f96d800ea60 tx=0x7f96d800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.578+0000 7f96e5600640 1 -- 172.21.15.107:0/4011845945 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d800cde0 con 0x7f96e81033c0 2024-07-26T12:48:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.579+0000 7f96e5600640 1 -- 172.21.15.107:0/4011845945 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96d80035f0 con 0x7f96e81033c0 2024-07-26T12:48:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.579+0000 7f96ed600640 1 -- 172.21.15.107:0/4011845945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e81a2180 con 0x7f96e81033c0 2024-07-26T12:48:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.579+0000 7f96e5600640 1 -- 172.21.15.107:0/4011845945 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d8010690 con 0x7f96e81033c0 2024-07-26T12:48:33.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.579+0000 7f96ed600640 1 -- 172.21.15.107:0/4011845945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e81a88e0 con 0x7f96e81033c0 2024-07-26T12:48:33.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.580+0000 7f96ed600640 1 -- 172.21.15.107:0/4011845945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e81037d0 con 0x7f96e81033c0 2024-07-26T12:48:33.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.583+0000 7f96e5600640 1 -- 172.21.15.107:0/4011845945 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f96d80037a0 con 0x7f96e81033c0 2024-07-26T12:48:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.584+0000 7f96e5600640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96cc0b3c20 0x7f96cc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.584+0000 7f96e5600640 1 -- 172.21.15.107:0/4011845945 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f96d80e4920 con 0x7f96e81033c0 2024-07-26T12:48:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.584+0000 7f96df400640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96cc0b3c20 0x7f96cc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.585+0000 7f96df400640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96cc0b3c20 0x7f96cc0b6130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f96d002a9e0 tx=0x7f96d0005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:33.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.586+0000 7f96e5600640 1 -- 172.21.15.107:0/4011845945 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f96d80aba70 con 0x7f96e81033c0 2024-07-26T12:48:33.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.828+0000 7f96ed600640 1 -- 172.21.15.107:0/4011845945 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f96e819d050 con 0x7f96e81033c0 2024-07-26T12:48:33.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.829+0000 7f96e5600640 1 -- 172.21.15.107:0/4011845945 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7f96d80ab370 con 0x7f96e81033c0 2024-07-26T12:48:33.829 INFO:teuthology.orchestra.run.smithi107.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-07-26T12:48:33.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.831+0000 7f96ed600640 1 -- 172.21.15.107:0/4011845945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96cc0b3c20 msgr2=0x7f96cc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:33.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.831+0000 7f96ed600640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96cc0b3c20 0x7f96cc0b6130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f96d002a9e0 tx=0x7f96d0005a20 comp rx=0 tx=0).stop 2024-07-26T12:48:33.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.831+0000 7f96ed600640 1 -- 172.21.15.107:0/4011845945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e81033c0 msgr2=0x7f96e819bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:33.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.831+0000 7f96ed600640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e81033c0 0x7f96e819bf20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f96d800ea60 tx=0x7f96d800ef20 comp rx=0 tx=0).stop 2024-07-26T12:48:33.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.832+0000 7f96ed600640 1 -- 172.21.15.107:0/4011845945 shutdown_connections 2024-07-26T12:48:33.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.832+0000 7f96ed600640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f96cc0b3c20 0x7f96cc0b6130 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:33.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.832+0000 7f96ed600640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f96e8109650 0x7f96e81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:33.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.832+0000 7f96ed600640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f96e8108d00 0x7f96e819c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:33.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.832+0000 7f96ed600640 1 --2- 172.21.15.107:0/4011845945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f96e81033c0 0x7f96e819bf20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:33.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.832+0000 7f96ed600640 1 -- 172.21.15.107:0/4011845945 >> 172.21.15.107:0/4011845945 conn(0x7f96e80fee10 msgr2=0x7f96e81083b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:33.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.832+0000 7f96ed600640 1 -- 172.21.15.107:0/4011845945 shutdown_connections 2024-07-26T12:48:33.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:33.832+0000 7f96ed600640 1 -- 172.21.15.107:0/4011845945 wait complete. 2024-07-26T12:48:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:33 smithi112 ceph-mon[37501]: pgmap v467: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:48:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:33 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/4011845945' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T12:48:34.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:33 smithi107 ceph-mon[32659]: pgmap v467: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:48:34.248 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:33 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/4011845945' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T12:48:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:33 smithi132 ceph-mon[37580]: pgmap v467: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-07-26T12:48:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:33 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/4011845945' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T12:48:34.794 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-26T12:48:34.794 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T12:48:34.794 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi107.front.sepia.ceph.com 2024-07-26T12:48:34.794 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T12:48:34.794 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T12:48:34.794 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-26T12:48:34.795 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T12:48:34.795 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T12:48:34.795 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T12:48:34.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:48:34.795 DEBUG:teuthology.orchestra.run.smithi107:> ip addr 2024-07-26T12:48:34.848 INFO:teuthology.orchestra.run.smithi107.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T12:48:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T12:48:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T12:48:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-07-26T12:48:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 ::1/128 scope host 2024-07-26T12:48:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-07-26T12:48:34.848 INFO:teuthology.orchestra.run.smithi107.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T12:48:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0a brd ff:ff:ff:ff:ff:ff 2024-07-26T12:48:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f0 2024-07-26T12:48:34.849 INFO:teuthology.orchestra.run.smithi107.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T12:48:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:d8:a2:0b brd ff:ff:ff:ff:ff:ff 2024-07-26T12:48:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: altname enp7s0f1 2024-07-26T12:48:34.849 INFO:teuthology.orchestra.run.smithi107.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T12:48:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:ea brd ff:ff:ff:ff:ff:ff 2024-07-26T12:48:34.849 INFO:teuthology.orchestra.run.smithi107.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T12:48:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: link/ether 0c:c4:7a:8f:47:eb brd ff:ff:ff:ff:ff:ff 2024-07-26T12:48:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: inet 172.21.15.107/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T12:48:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft 41697sec preferred_lft 41697sec 2024-07-26T12:48:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2024-07-26T12:48:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: valid_lft forever preferred_lft forever 2024-07-26T12:48:34.851 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T12:48:34.851 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T12:48:34.851 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-07-26T12:48:34.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add name ceph-brx type bridge 2024-07-26T12:48:34.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr flush dev ceph-brx 2024-07-26T12:48:34.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set ceph-brx up 2024-07-26T12:48:34.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T12:48:34.851 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-07-26T12:48:34.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:48:35.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:48:35.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:48:35.096 DEBUG:teuthology.orchestra.run.smithi107:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T12:48:35.164 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-07-26T12:48:35.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:48:35.169 DEBUG:teuthology.orchestra.run.smithi107:> route 2024-07-26T12:48:35.218 INFO:teuthology.orchestra.run.smithi107.stdout:Kernel IP routing table 2024-07-26T12:48:35.218 INFO:teuthology.orchestra.run.smithi107.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T12:48:35.219 INFO:teuthology.orchestra.run.smithi107.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T12:48:35.219 INFO:teuthology.orchestra.run.smithi107.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T12:48:35.219 INFO:teuthology.orchestra.run.smithi107.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T12:48:35.219 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T12:48:35.220 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-07-26T12:48:35.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T12:48:35.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T12:48:35.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T12:48:35.220 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-07-26T12:48:35.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:48:35.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:48:35.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:48:35.430 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list 2024-07-26T12:48:35.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:48:35.479 DEBUG:teuthology.orchestra.run.smithi107:> ip netns list-id 2024-07-26T12:48:35.575 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T12:48:35.576 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-07-26T12:48:35.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T12:48:35.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-07-26T12:48:35.576 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-07-26T12:48:35.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:48:35.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:48:35.761 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-07-26T12:48:35.761 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T12:48:35.761 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-07-26T12:48:35.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-07-26T12:48:35.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T12:48:35.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-07-26T12:48:35.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-07-26T12:48:35.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-07-26T12:48:35.762 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-07-26T12:48:35.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:48:35.957 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:35 smithi107 ceph-mon[32659]: pgmap v468: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:36.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:48:36.027 DEBUG:teuthology.orchestra.run.smithi107:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T12:48:36.027 DEBUG:teuthology.orchestra.run.smithi107:> set -e 2024-07-26T12:48:36.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set brx.0 up 2024-07-26T12:48:36.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T12:48:36.028 DEBUG:teuthology.orchestra.run.smithi107:> ') 2024-07-26T12:48:36.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:48:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:35 smithi112 ceph-mon[37501]: pgmap v468: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:36.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:48:36.223 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T12:48:36.224 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T12:48:36.270 INFO:teuthology.orchestra.run.smithi107.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T12:48:36.270 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T12:48:36.270 DEBUG:teuthology.orchestra.run.smithi107:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T12:48:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:35 smithi132 ceph-mon[37580]: pgmap v468: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:36.363 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-07-26T12:48:36.363 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-07-26T12:48:36.363 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/ff915441-bb90-467d-b025-a122c34b612f 2024-07-26T12:48:36.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:48:36.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/ff915441-bb90-467d-b025-a122c34b612f /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-07-26T12:48:36.560 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-07-26T12:48:36.560 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-07-26T12:48:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:36 smithi107 ceph-mon[32659]: pgmap v469: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:36 smithi112 ceph-mon[37501]: pgmap v469: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:36 smithi132 ceph-mon[37580]: pgmap v469: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:48:37.365 INFO:teuthology.orchestra.run.smithi107.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-07-26T12:48:37.365 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-07-26T12:48:37.365 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: resolved to: "172.21.15.107:3300,172.21.15.112:3300,172.21.15.132:3300" 2024-07-26T12:48:37.365 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: trying mount with new device syntax: 0@14de5e40-4b4b-11ef-bcb7-c7b262605968.cephfs=/volumes/qa/sv_1/ff915441-bb90-467d-b025-a122c34b612f 2024-07-26T12:48:37.365 INFO:teuthology.orchestra.run.smithi107.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.107:3300/172.21.15.112:3300/172.21.15.132:3300" will pass to kernel 2024-07-26T12:48:37.369 INFO:teuthology.orchestra.run.smithi107.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T12:48:37.369 INFO:teuthology.orchestra.run.smithi107.stdout: You just mounted a file system that supports labels which does not 2024-07-26T12:48:37.369 INFO:teuthology.orchestra.run.smithi107.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T12:48:37.369 INFO:teuthology.orchestra.run.smithi107.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T12:48:37.369 INFO:teuthology.orchestra.run.smithi107.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T12:48:37.374 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T12:48:37.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T12:48:37.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T12:48:37.462 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T12:48:37.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T12:48:37.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:48:37.585 INFO:teuthology.orchestra.run.smithi107.stdout:client15267 2024-07-26T12:48:37.585 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:48:37.585 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T12:48:37.633 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T12:48:37.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T12:48:37.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:48:37.800 INFO:teuthology.orchestra.run.smithi107.stdout:client15267 2024-07-26T12:48:37.800 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T12:48:37.901 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T12:48:37.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:48:37.915 INFO:teuthology.orchestra.run.smithi107.stdout:client15267 2024-07-26T12:48:37.916 DEBUG:teuthology.orchestra.run.smithi107:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968.client15267/status') 2024-07-26T12:48:38.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T12:48:38.052 INFO:teuthology.orchestra.run.smithi107.stdout:instance: client.15267 (3)192.168.0.1:0/1025124677 2024-07-26T12:48:38.053 INFO:teuthology.orchestra.run.smithi107.stdout:blocklisted: false 2024-07-26T12:48:38.053 INFO:teuthology.orchestra.run.smithi107.stderr:0+1 records in 2024-07-26T12:48:38.053 INFO:teuthology.orchestra.run.smithi107.stderr:0+1 records out 2024-07-26T12:48:38.053 INFO:teuthology.orchestra.run.smithi107.stderr:70 bytes copied, 7.7241e-05 s, 906 kB/s 2024-07-26T12:48:38.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T12:48:38.066 INFO:teuthology.run_tasks:Running task exec... 2024-07-26T12:48:38.075 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T12:48:38.076 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-07-26T12:48:38.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-07-26T12:48:38.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.338+0000 7f174ca00640 1 -- 172.21.15.107:0/670888910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174810b7d0 msgr2=0x7f1748112000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:38.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.338+0000 7f174ca00640 1 --2- 172.21.15.107:0/670888910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174810b7d0 0x7f1748112000 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1740009ca0 tx=0x7f174002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.339+0000 7f174ca00640 1 -- 172.21.15.107:0/670888910 shutdown_connections 2024-07-26T12:48:38.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.339+0000 7f174ca00640 1 --2- 172.21.15.107:0/670888910 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174810b7d0 0x7f1748112000 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.339+0000 7f174ca00640 1 --2- 172.21.15.107:0/670888910 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f174810ae80 0x7f174810b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.339+0000 7f174ca00640 1 --2- 172.21.15.107:0/670888910 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1748105550 0x7f1748105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.339+0000 7f174ca00640 1 -- 172.21.15.107:0/670888910 >> 172.21.15.107:0/670888910 conn(0x7f1748100f90 msgr2=0x7f17481033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:38.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.339+0000 7f174ca00640 1 -- 172.21.15.107:0/670888910 shutdown_connections 2024-07-26T12:48:38.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.339+0000 7f174ca00640 1 -- 172.21.15.107:0/670888910 wait complete. 2024-07-26T12:48:38.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.340+0000 7f174ca00640 1 Processor -- start 2024-07-26T12:48:38.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.340+0000 7f174ca00640 1 -- start start 2024-07-26T12:48:38.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f174ca00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1748105550 0x7f174819be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:38.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f174ca00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f174810ae80 0x7f174819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:38.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f174ca00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f174810b7d0 0x7f17481a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:38.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f174ca00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17481154a0 con 0x7f1748105550 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f174ca00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f1748115320 con 0x7f174810ae80 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f174ca00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f1748115620 con 0x7f174810b7d0 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f1746800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1748105550 0x7f174819be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f1746800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1748105550 0x7f174819be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55734/0 (socket says 172.21.15.107:55734) 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f1746800640 1 -- 172.21.15.107:0/3434569329 learned_addr learned my addr 172.21.15.107:0/3434569329 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f1745e00640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f174810ae80 0x7f174819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f1747200640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f174810b7d0 0x7f17481a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f1746800640 1 -- 172.21.15.107:0/3434569329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f174810b7d0 msgr2=0x7f17481a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f1746800640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f174810b7d0 0x7f17481a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f1746800640 1 -- 172.21.15.107:0/3434569329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f174810ae80 msgr2=0x7f174819c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f1746800640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f174810ae80 0x7f174819c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.341+0000 7f1746800640 1 -- 172.21.15.107:0/3434569329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17481a1db0 con 0x7f1748105550 2024-07-26T12:48:38.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.342+0000 7f1746800640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1748105550 0x7f174819be80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f173800e9f0 tx=0x7f173800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:38.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.342+0000 7f173f400640 1 -- 172.21.15.107:0/3434569329 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173800cd80 con 0x7f1748105550 2024-07-26T12:48:38.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.342+0000 7f174ca00640 1 -- 172.21.15.107:0/3434569329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17481a20a0 con 0x7f1748105550 2024-07-26T12:48:38.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.342+0000 7f173f400640 1 -- 172.21.15.107:0/3434569329 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17380054d0 con 0x7f1748105550 2024-07-26T12:48:38.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.342+0000 7f173f400640 1 -- 172.21.15.107:0/3434569329 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738010690 con 0x7f1748105550 2024-07-26T12:48:38.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.342+0000 7f174ca00640 1 -- 172.21.15.107:0/3434569329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17481a8830 con 0x7f1748105550 2024-07-26T12:48:38.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.343+0000 7f174ca00640 1 -- 172.21.15.107:0/3434569329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1748105960 con 0x7f1748105550 2024-07-26T12:48:38.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.346+0000 7f173f400640 1 -- 172.21.15.107:0/3434569329 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f1738004e50 con 0x7f1748105550 2024-07-26T12:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.347+0000 7f173f400640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f172c0b3c20 0x7f172c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:38.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.348+0000 7f173f400640 1 -- 172.21.15.107:0/3434569329 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f17380e4740 con 0x7f1748105550 2024-07-26T12:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.348+0000 7f1745e00640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f172c0b3c20 0x7f172c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:38.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.349+0000 7f1745e00640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f172c0b3c20 0x7f172c0b6130 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f17481088b0 tx=0x7f1734005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:38.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.349+0000 7f173f400640 1 -- 172.21.15.107:0/3434569329 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1738055600 con 0x7f1748105550 2024-07-26T12:48:38.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.604+0000 7f174ca00640 1 -- 172.21.15.107:0/3434569329 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f174802e0c0 con 0x7f1748105550 2024-07-26T12:48:38.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.659+0000 7f173f400640 1 -- 172.21.15.107:0/3434569329 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) ==== 117+0+0 (secure 0 0 0) 0x7f17380ab890 con 0x7f1748105550 2024-07-26T12:48:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.663+0000 7f174ca00640 1 -- 172.21.15.107:0/3434569329 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f172c0b3c20 msgr2=0x7f172c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.663+0000 7f174ca00640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f172c0b3c20 0x7f172c0b6130 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f17481088b0 tx=0x7f1734005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.663+0000 7f174ca00640 1 -- 172.21.15.107:0/3434569329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1748105550 msgr2=0x7f174819be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.663+0000 7f174ca00640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1748105550 0x7f174819be80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f173800e9f0 tx=0x7f173800edf0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.663+0000 7f174ca00640 1 -- 172.21.15.107:0/3434569329 shutdown_connections 2024-07-26T12:48:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.663+0000 7f174ca00640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f172c0b3c20 0x7f172c0b6130 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.663+0000 7f174ca00640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f174810b7d0 0x7f17481a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.663+0000 7f174ca00640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f174810ae80 0x7f174819c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.663+0000 7f174ca00640 1 --2- 172.21.15.107:0/3434569329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1748105550 0x7f174819be80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.663+0000 7f174ca00640 1 -- 172.21.15.107:0/3434569329 >> 172.21.15.107:0/3434569329 conn(0x7f1748100f90 msgr2=0x7f17481033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.664+0000 7f174ca00640 1 -- 172.21.15.107:0/3434569329 shutdown_connections 2024-07-26T12:48:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.664+0000 7f174ca00640 1 -- 172.21.15.107:0/3434569329 wait complete. 2024-07-26T12:48:38.694 INFO:teuthology.run_tasks:Running task check-counter... 2024-07-26T12:48:38.705 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-07-26T12:48:38.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T12:48:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:38 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3434569329' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T12:48:38.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.995+0000 7f3439200640 1 -- 172.21.15.107:0/3429053967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3434076dc0 msgr2=0x7f34340751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:38.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.995+0000 7f3439200640 1 --2- 172.21.15.107:0/3429053967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3434076dc0 0x7f34340751e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3424009900 tx=0x7f342402a410 comp rx=0 tx=0).stop 2024-07-26T12:48:38.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.996+0000 7f3439200640 1 -- 172.21.15.107:0/3429053967 shutdown_connections 2024-07-26T12:48:38.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.996+0000 7f3439200640 1 --2- 172.21.15.107:0/3429053967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3434113be0 0x7f34341160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.996+0000 7f3439200640 1 --2- 172.21.15.107:0/3429053967 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3434075800 0x7f3434075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.996+0000 7f3439200640 1 --2- 172.21.15.107:0/3429053967 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3434076dc0 0x7f34340751e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:38.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.996+0000 7f3439200640 1 -- 172.21.15.107:0/3429053967 >> 172.21.15.107:0/3429053967 conn(0x7f34340fed80 msgr2=0x7f34341011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:38.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.996+0000 7f3439200640 1 -- 172.21.15.107:0/3429053967 shutdown_connections 2024-07-26T12:48:38.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.996+0000 7f3439200640 1 -- 172.21.15.107:0/3429053967 wait complete. 2024-07-26T12:48:38.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.997+0000 7f3439200640 1 Processor -- start 2024-07-26T12:48:38.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.997+0000 7f3439200640 1 -- start start 2024-07-26T12:48:38.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.997+0000 7f3439200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3434075800 0x7f34341a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:38.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.997+0000 7f3439200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3434076dc0 0x7f34341a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:38.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.997+0000 7f3439200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3434113be0 0x7f34341a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:38.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.997+0000 7f3439200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f34341197e0 con 0x7f3434076dc0 2024-07-26T12:48:38.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.997+0000 7f3439200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3434119660 con 0x7f3434075800 2024-07-26T12:48:38.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.997+0000 7f3439200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3434119960 con 0x7f3434113be0 2024-07-26T12:48:38.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3432a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3434076dc0 0x7f34341a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:39.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3432a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3434076dc0 0x7f34341a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55748/0 (socket says 172.21.15.107:55748) 2024-07-26T12:48:39.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3433400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3434075800 0x7f34341a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:39.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3432a00640 1 -- 172.21.15.107:0/2727206699 learned_addr learned my addr 172.21.15.107:0/2727206699 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:39.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3433e00640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3434113be0 0x7f34341a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:39.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3432a00640 1 -- 172.21.15.107:0/2727206699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3434113be0 msgr2=0x7f34341a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:39.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3432a00640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3434113be0 0x7f34341a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3432a00640 1 -- 172.21.15.107:0/2727206699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3434075800 msgr2=0x7f34341a0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:39.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3432a00640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3434075800 0x7f34341a0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3432a00640 1 -- 172.21.15.107:0/2727206699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34341a6130 con 0x7f3434076dc0 2024-07-26T12:48:39.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.998+0000 7f3433400640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3434075800 0x7f34341a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:39.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.999+0000 7f3432a00640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3434076dc0 0x7f34341a07a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f341c00d950 tx=0x7f341c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:39.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.999+0000 7f342be00640 1 -- 172.21.15.107:0/2727206699 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c017070 con 0x7f3434076dc0 2024-07-26T12:48:39.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.999+0000 7f3439200640 1 -- 172.21.15.107:0/2727206699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34341a6420 con 0x7f3434076dc0 2024-07-26T12:48:39.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.999+0000 7f342be00640 1 -- 172.21.15.107:0/2727206699 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f341c009c30 con 0x7f3434076dc0 2024-07-26T12:48:39.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.999+0000 7f342be00640 1 -- 172.21.15.107:0/2727206699 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c00f620 con 0x7f3434076dc0 2024-07-26T12:48:39.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:38.999+0000 7f3439200640 1 -- 172.21.15.107:0/2727206699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34341acbb0 con 0x7f3434076dc0 2024-07-26T12:48:39.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.000+0000 7f3439200640 1 -- 172.21.15.107:0/2727206699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3434075c10 con 0x7f3434076dc0 2024-07-26T12:48:39.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.003+0000 7f342be00640 1 -- 172.21.15.107:0/2727206699 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f341c010460 con 0x7f3434076dc0 2024-07-26T12:48:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.005+0000 7f342be00640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34180b3ca0 0x7f34180b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.006+0000 7f342be00640 1 -- 172.21.15.107:0/2727206699 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f341c0e3bf0 con 0x7f3434076dc0 2024-07-26T12:48:39.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.006+0000 7f3433400640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34180b3ca0 0x7f34180b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:39.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.007+0000 7f3433400640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34180b3ca0 0x7f34180b61b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3424031440 tx=0x7f3424032ed0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:39.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.008+0000 7f342be00640 1 -- 172.21.15.107:0/2727206699 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f341c009de0 con 0x7f3434076dc0 2024-07-26T12:48:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:38 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3434569329' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T12:48:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:38 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3434569329' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T12:48:39.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.218+0000 7f3439200640 1 -- 172.21.15.107:0/2727206699 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f34341acef0 con 0x7f3434076dc0 2024-07-26T12:48:39.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.222+0000 7f342be00640 1 -- 172.21.15.107:0/2727206699 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25203 (secure 0 0 0) 0x7f341c0b30f0 con 0x7f3434076dc0 2024-07-26T12:48:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:48:39.224 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:44:36.797948+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:48:39.226 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.225+0000 7f3439200640 1 -- 172.21.15.107:0/2727206699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34180b3ca0 msgr2=0x7f34180b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.225+0000 7f3439200640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34180b3ca0 0x7f34180b61b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3424031440 tx=0x7f3424032ed0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.225+0000 7f3439200640 1 -- 172.21.15.107:0/2727206699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3434076dc0 msgr2=0x7f34341a07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.225+0000 7f3439200640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3434076dc0 0x7f34341a07a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f341c00d950 tx=0x7f341c00de10 comp rx=0 tx=0).stop 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.226+0000 7f3439200640 1 -- 172.21.15.107:0/2727206699 shutdown_connections 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.226+0000 7f3439200640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34180b3ca0 0x7f34180b61b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.226+0000 7f3439200640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3434113be0 0x7f34341a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.226+0000 7f3439200640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3434076dc0 0x7f34341a07a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.226+0000 7f3439200640 1 --2- 172.21.15.107:0/2727206699 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3434075800 0x7f34341a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.226+0000 7f3439200640 1 -- 172.21.15.107:0/2727206699 >> 172.21.15.107:0/2727206699 conn(0x7f34340fed80 msgr2=0x7f3434101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.226+0000 7f3439200640 1 -- 172.21.15.107:0/2727206699 shutdown_connections 2024-07-26T12:48:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.226+0000 7f3439200640 1 -- 172.21.15.107:0/2727206699 wait complete. 2024-07-26T12:48:39.254 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T12:37:58.631844+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-07-26T12:43:07.012156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-07-26T12:43:10.809864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-07-26T12:43:24.750177+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-07-26T12:48:39.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-26T12:48:39.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.529+0000 7ff68e400640 1 -- 172.21.15.107:0/2353432206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff688104330 msgr2=0x7ff688104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:39.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.529+0000 7ff68e400640 1 --2- 172.21.15.107:0/2353432206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff688104330 0x7ff688104740 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff680009be0 tx=0x7ff68002a6d0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.529+0000 7ff68e400640 1 -- 172.21.15.107:0/2353432206 shutdown_connections 2024-07-26T12:48:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.529+0000 7ff68e400640 1 --2- 172.21.15.107:0/2353432206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff68810d790 0x7ff68810fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.529+0000 7ff68e400640 1 --2- 172.21.15.107:0/2353432206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688104c80 0x7ff68810d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.529+0000 7ff68e400640 1 --2- 172.21.15.107:0/2353432206 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff688104330 0x7ff688104740 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.529+0000 7ff68e400640 1 -- 172.21.15.107:0/2353432206 >> 172.21.15.107:0/2353432206 conn(0x7ff6880fde00 msgr2=0x7ff688100250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.529+0000 7ff68e400640 1 -- 172.21.15.107:0/2353432206 shutdown_connections 2024-07-26T12:48:39.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.530+0000 7ff68e400640 1 -- 172.21.15.107:0/2353432206 wait complete. 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.531+0000 7ff68e400640 1 Processor -- start 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.531+0000 7ff68e400640 1 -- start start 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.531+0000 7ff68e400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688104c80 0x7ff688102060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.531+0000 7ff68e400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff68810d790 0x7ff6881025a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.531+0000 7ff68e400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff688102ae0 0x7ff6881a80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.531+0000 7ff68e400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff688113400 con 0x7ff688104c80 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.531+0000 7ff68e400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff688113280 con 0x7ff68810d790 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.531+0000 7ff68e400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff688113580 con 0x7ff688102ae0 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff687400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688104c80 0x7ff688102060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff686a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff68810d790 0x7ff6881025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff687400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688104c80 0x7ff688102060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55772/0 (socket says 172.21.15.107:55772) 2024-07-26T12:48:39.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff686a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff68810d790 0x7ff6881025a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:48928/0 (socket says 172.21.15.107:48928) 2024-07-26T12:48:39.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff687400640 1 -- 172.21.15.107:0/3642944435 learned_addr learned my addr 172.21.15.107:0/3642944435 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:39.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff687e00640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff688102ae0 0x7ff6881a80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:39.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff686a00640 1 -- 172.21.15.107:0/3642944435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff688102ae0 msgr2=0x7ff6881a80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:39.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff686a00640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff688102ae0 0x7ff6881a80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff686a00640 1 -- 172.21.15.107:0/3642944435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688104c80 msgr2=0x7ff688102060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:39.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff686a00640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688104c80 0x7ff688102060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff686a00640 1 -- 172.21.15.107:0/3642944435 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6881a86b0 con 0x7ff68810d790 2024-07-26T12:48:39.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff687400640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688104c80 0x7ff688102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:39.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.532+0000 7ff686a00640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff68810d790 0x7ff6881025a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff68806aee0 tx=0x7ff67400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:39.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.533+0000 7ff67fe00640 1 -- 172.21.15.107:0/3642944435 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67400ccd0 con 0x7ff68810d790 2024-07-26T12:48:39.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.533+0000 7ff67fe00640 1 -- 172.21.15.107:0/3642944435 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff67400ce70 con 0x7ff68810d790 2024-07-26T12:48:39.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.533+0000 7ff68e400640 1 -- 172.21.15.107:0/3642944435 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6881a89a0 con 0x7ff68810d790 2024-07-26T12:48:39.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.533+0000 7ff67fe00640 1 -- 172.21.15.107:0/3642944435 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674010430 con 0x7ff68810d790 2024-07-26T12:48:39.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.533+0000 7ff68e400640 1 -- 172.21.15.107:0/3642944435 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6881a8f40 con 0x7ff68810d790 2024-07-26T12:48:39.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.534+0000 7ff68e400640 1 -- 172.21.15.107:0/3642944435 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff68806b9b0 con 0x7ff68810d790 2024-07-26T12:48:39.538 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.538+0000 7ff67fe00640 1 -- 172.21.15.107:0/3642944435 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7ff6740035f0 con 0x7ff68810d790 2024-07-26T12:48:39.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.539+0000 7ff67fe00640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff66c0b3ca0 0x7ff66c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:39.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.539+0000 7ff67fe00640 1 -- 172.21.15.107:0/3642944435 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff6740e42f0 con 0x7ff68810d790 2024-07-26T12:48:39.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.539+0000 7ff687400640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff66c0b3ca0 0x7ff66c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:39.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.540+0000 7ff67fe00640 1 -- 172.21.15.107:0/3642944435 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff6740ab4d0 con 0x7ff68810d790 2024-07-26T12:48:39.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.540+0000 7ff687400640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff66c0b3ca0 0x7ff66c0b61b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff688103ab0 tx=0x7ff680004a40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.750+0000 7ff68e400640 1 -- 172.21.15.107:0/3642944435 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff688103360 con 0x7ff68810d790 2024-07-26T12:48:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.751+0000 7ff67fe00640 1 -- 172.21.15.107:0/3642944435 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7ff688103360 con 0x7ff68810d790 2024-07-26T12:48:39.752 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-07-26T12:48:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 -- 172.21.15.107:0/3642944435 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff66c0b3ca0 msgr2=0x7ff66c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff66c0b3ca0 0x7ff66c0b61b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff688103ab0 tx=0x7ff680004a40 comp rx=0 tx=0).stop 2024-07-26T12:48:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 -- 172.21.15.107:0/3642944435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff68810d790 msgr2=0x7ff6881025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff68810d790 0x7ff6881025a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff68806aee0 tx=0x7ff67400ee50 comp rx=0 tx=0).stop 2024-07-26T12:48:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 -- 172.21.15.107:0/3642944435 shutdown_connections 2024-07-26T12:48:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff66c0b3ca0 0x7ff66c0b61b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff688102ae0 0x7ff6881a80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff68810d790 0x7ff6881025a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 --2- 172.21.15.107:0/3642944435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff688104c80 0x7ff688102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:39.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 -- 172.21.15.107:0/3642944435 >> 172.21.15.107:0/3642944435 conn(0x7ff6880fde00 msgr2=0x7ff6880fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:39.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 -- 172.21.15.107:0/3642944435 shutdown_connections 2024-07-26T12:48:39.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:39.755+0000 7ff68e400640 1 -- 172.21.15.107:0/3642944435 wait complete. 2024-07-26T12:48:39.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-07-26T12:48:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:39 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3434569329' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T12:48:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:39 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:48:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:39 smithi112 ceph-mon[37501]: pgmap v470: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:39 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2727206699' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:48:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:39 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3434569329' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T12:48:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:39 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:48:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:39 smithi107 ceph-mon[32659]: pgmap v470: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:39 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2727206699' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:48:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:39 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3434569329' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T12:48:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:39 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:48:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:39 smithi132 ceph-mon[37580]: pgmap v470: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-07-26T12:48:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:39 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2727206699' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:48:40.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.059+0000 7f87cee00640 1 -- 172.21.15.107:0/26883721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87c810b7b0 msgr2=0x7f87c8111fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:40.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.059+0000 7f87cee00640 1 --2- 172.21.15.107:0/26883721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87c810b7b0 0x7f87c8111fe0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f87c0009ca0 tx=0x7f87c002a8b0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.059+0000 7f87cee00640 1 -- 172.21.15.107:0/26883721 shutdown_connections 2024-07-26T12:48:40.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.059+0000 7f87cee00640 1 --2- 172.21.15.107:0/26883721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87c810b7b0 0x7f87c8111fe0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.059+0000 7f87cee00640 1 --2- 172.21.15.107:0/26883721 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87c810ae60 0x7f87c810b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.059+0000 7f87cee00640 1 --2- 172.21.15.107:0/26883721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c8105530 0x7f87c8105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.059+0000 7f87cee00640 1 -- 172.21.15.107:0/26883721 >> 172.21.15.107:0/26883721 conn(0x7f87c8100f90 msgr2=0x7f87c81033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:40.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.059+0000 7f87cee00640 1 -- 172.21.15.107:0/26883721 shutdown_connections 2024-07-26T12:48:40.061 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.060+0000 7f87cee00640 1 -- 172.21.15.107:0/26883721 wait complete. 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.061+0000 7f87cee00640 1 Processor -- start 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.061+0000 7f87cee00640 1 -- start start 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.061+0000 7f87cee00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87c8105530 0x7f87c819bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.061+0000 7f87cee00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87c810ae60 0x7f87c819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.061+0000 7f87cee00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c810b7b0 0x7f87c81a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.061+0000 7f87cee00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f87c8115460 con 0x7f87c810ae60 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.061+0000 7f87cee00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f87c81152e0 con 0x7f87c8105530 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.061+0000 7f87cee00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f87c81155e0 con 0x7f87c810b7b0 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87cda00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87c8105530 0x7f87c819bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87cda00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87c8105530 0x7f87c819bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:48948/0 (socket says 172.21.15.107:48948) 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87cda00640 1 -- 172.21.15.107:0/1959366367 learned_addr learned my addr 172.21.15.107:0/1959366367 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87cd000640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87c810ae60 0x7f87c819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87ce400640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c810b7b0 0x7f87c81a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87cd000640 1 -- 172.21.15.107:0/1959366367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c810b7b0 msgr2=0x7f87c81a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87cd000640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c810b7b0 0x7f87c81a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87cd000640 1 -- 172.21.15.107:0/1959366367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87c8105530 msgr2=0x7f87c819bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87cd000640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87c8105530 0x7f87c819bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87cd000640 1 -- 172.21.15.107:0/1959366367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c81a1dd0 con 0x7f87c810ae60 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.062+0000 7f87cda00640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87c8105530 0x7f87c819bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.063+0000 7f87cd000640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87c810ae60 0x7f87c819c440 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f87c806c260 tx=0x7f87b400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.063+0000 7f87c6000640 1 -- 172.21.15.107:0/1959366367 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b400cd00 con 0x7f87c810ae60 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.063+0000 7f87c6000640 1 -- 172.21.15.107:0/1959366367 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87b40035f0 con 0x7f87c810ae60 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.063+0000 7f87cee00640 1 -- 172.21.15.107:0/1959366367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c81a20c0 con 0x7f87c810ae60 2024-07-26T12:48:40.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.063+0000 7f87c6000640 1 -- 172.21.15.107:0/1959366367 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4010690 con 0x7f87c810ae60 2024-07-26T12:48:40.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.063+0000 7f87cee00640 1 -- 172.21.15.107:0/1959366367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c81a8850 con 0x7f87c810ae60 2024-07-26T12:48:40.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.064+0000 7f87cee00640 1 -- 172.21.15.107:0/1959366367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87c8105940 con 0x7f87c810ae60 2024-07-26T12:48:40.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.066+0000 7f87c6000640 1 -- 172.21.15.107:0/1959366367 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f87b40037a0 con 0x7f87c810ae60 2024-07-26T12:48:40.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.069+0000 7f87c6000640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87b00b3c20 0x7f87b00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:40.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.069+0000 7f87c6000640 1 -- 172.21.15.107:0/1959366367 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f87b40e4360 con 0x7f87c810ae60 2024-07-26T12:48:40.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.069+0000 7f87cda00640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87b00b3c20 0x7f87b00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:40.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.070+0000 7f87cda00640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87b00b3c20 0x7f87b00b6130 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f87bc011550 tx=0x7f87bc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:40.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.074+0000 7f87c6000640 1 -- 172.21.15.107:0/1959366367 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f87b40ab4b0 con 0x7f87c810ae60 2024-07-26T12:48:40.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.288+0000 7f87cee00640 1 -- 172.21.15.107:0/1959366367 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f87c802e0c0 con 0x7f87c810ae60 2024-07-26T12:48:40.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.289+0000 7f87c6000640 1 -- 172.21.15.107:0/1959366367 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f87b40aadb0 con 0x7f87c810ae60 2024-07-26T12:48:40.290 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-07-26T12:48:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 -- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87b00b3c20 msgr2=0x7f87b00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87b00b3c20 0x7f87b00b6130 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f87bc011550 tx=0x7f87bc005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 -- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87c810ae60 msgr2=0x7f87c819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87c810ae60 0x7f87c819c440 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f87c806c260 tx=0x7f87b400ee50 comp rx=0 tx=0).stop 2024-07-26T12:48:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 -- 172.21.15.107:0/1959366367 shutdown_connections 2024-07-26T12:48:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f87b00b3c20 0x7f87b00b6130 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c810b7b0 0x7f87c81a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87c810ae60 0x7f87c819c440 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 --2- 172.21.15.107:0/1959366367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f87c8105530 0x7f87c819bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 -- 172.21.15.107:0/1959366367 >> 172.21.15.107:0/1959366367 conn(0x7f87c8100f90 msgr2=0x7f87c81033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:40.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 -- 172.21.15.107:0/1959366367 shutdown_connections 2024-07-26T12:48:40.294 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.292+0000 7f87cee00640 1 -- 172.21.15.107:0/1959366367 wait complete. 2024-07-26T12:48:40.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-07-26T12:48:40.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.592+0000 7f63d2e00640 1 -- 172.21.15.107:0/4210811499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63cc10f870 msgr2=0x7f63cc111cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:40.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.592+0000 7f63d2e00640 1 --2- 172.21.15.107:0/4210811499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63cc10f870 0x7f63cc111cd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009ca0 tx=0x7f63c402a8c0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.592+0000 7f63d2e00640 1 -- 172.21.15.107:0/4210811499 shutdown_connections 2024-07-26T12:48:40.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.592+0000 7f63d2e00640 1 --2- 172.21.15.107:0/4210811499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63cc10f870 0x7f63cc111cd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.592+0000 7f63d2e00640 1 --2- 172.21.15.107:0/4210811499 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63cc075b30 0x7f63cc10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.592+0000 7f63d2e00640 1 --2- 172.21.15.107:0/4210811499 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f63cc0751e0 0x7f63cc0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.592+0000 7f63d2e00640 1 -- 172.21.15.107:0/4210811499 >> 172.21.15.107:0/4210811499 conn(0x7f63cc0fed60 msgr2=0x7f63cc1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:40.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.592+0000 7f63d2e00640 1 -- 172.21.15.107:0/4210811499 shutdown_connections 2024-07-26T12:48:40.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.592+0000 7f63d2e00640 1 -- 172.21.15.107:0/4210811499 wait complete. 2024-07-26T12:48:40.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.593+0000 7f63d2e00640 1 Processor -- start 2024-07-26T12:48:40.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.593+0000 7f63d2e00640 1 -- start start 2024-07-26T12:48:40.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d2e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63cc0751e0 0x7f63cc19be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:40.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d2e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63cc075b30 0x7f63cc19c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d2e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f63cc10f870 0x7f63cc1a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d2e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc1154a0 con 0x7f63cc0751e0 2024-07-26T12:48:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d2e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc115320 con 0x7f63cc10f870 2024-07-26T12:48:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d2e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc115620 con 0x7f63cc075b30 2024-07-26T12:48:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d1a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63cc0751e0 0x7f63cc19be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d1a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63cc0751e0 0x7f63cc19be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55794/0 (socket says 172.21.15.107:55794) 2024-07-26T12:48:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d1a00640 1 -- 172.21.15.107:0/295557898 learned_addr learned my addr 172.21.15.107:0/295557898 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d1000640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63cc075b30 0x7f63cc19c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.594+0000 7f63d2400640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f63cc10f870 0x7f63cc1a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:40.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63d1a00640 1 -- 172.21.15.107:0/295557898 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63cc075b30 msgr2=0x7f63cc19c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63d1a00640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63cc075b30 0x7f63cc19c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63d1a00640 1 -- 172.21.15.107:0/295557898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f63cc10f870 msgr2=0x7f63cc1a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63d1a00640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f63cc10f870 0x7f63cc1a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63d1a00640 1 -- 172.21.15.107:0/295557898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63cc1a1d50 con 0x7f63cc0751e0 2024-07-26T12:48:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63d1000640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63cc075b30 0x7f63cc19c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63d1a00640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63cc0751e0 0x7f63cc19be20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f63c000ea60 tx=0x7f63c000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63ca000640 1 -- 172.21.15.107:0/295557898 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c000cde0 con 0x7f63cc0751e0 2024-07-26T12:48:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63ca000640 1 -- 172.21.15.107:0/295557898 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63c00035f0 con 0x7f63cc0751e0 2024-07-26T12:48:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63d2e00640 1 -- 172.21.15.107:0/295557898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63cc1a2040 con 0x7f63cc0751e0 2024-07-26T12:48:40.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63ca000640 1 -- 172.21.15.107:0/295557898 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0010690 con 0x7f63cc0751e0 2024-07-26T12:48:40.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.595+0000 7f63d2e00640 1 -- 172.21.15.107:0/295557898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63cc1a87d0 con 0x7f63cc0751e0 2024-07-26T12:48:40.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.596+0000 7f63d2e00640 1 -- 172.21.15.107:0/295557898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63cc0766f0 con 0x7f63cc0751e0 2024-07-26T12:48:40.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.599+0000 7f63ca000640 1 -- 172.21.15.107:0/295557898 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f63c00037a0 con 0x7f63cc0751e0 2024-07-26T12:48:40.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.600+0000 7f63ca000640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f63b40b3ca0 0x7f63b40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:40.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.600+0000 7f63ca000640 1 -- 172.21.15.107:0/295557898 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f63c00e47c0 con 0x7f63cc0751e0 2024-07-26T12:48:40.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.601+0000 7f63d1000640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f63b40b3ca0 0x7f63b40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:40.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.601+0000 7f63d1000640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f63b40b3ca0 0x7f63b40b61b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0ffb70 tx=0x7f63b8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:40.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.602+0000 7f63ca000640 1 -- 172.21.15.107:0/295557898 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f63c00ab910 con 0x7f63cc0751e0 2024-07-26T12:48:40.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.811+0000 7f63d2e00640 1 -- 172.21.15.107:0/295557898 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f63cc0755f0 con 0x7f63cc0751e0 2024-07-26T12:48:40.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.812+0000 7f63ca000640 1 -- 172.21.15.107:0/295557898 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f63cc0755f0 con 0x7f63cc0751e0 2024-07-26T12:48:40.813 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 64 2024-07-26T12:48:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.814+0000 7f63d2e00640 1 -- 172.21.15.107:0/295557898 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f63b40b3ca0 msgr2=0x7f63b40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.814+0000 7f63d2e00640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f63b40b3ca0 0x7f63b40b61b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0ffb70 tx=0x7f63b8005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.814+0000 7f63d2e00640 1 -- 172.21.15.107:0/295557898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63cc0751e0 msgr2=0x7f63cc19be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:40.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.814+0000 7f63d2e00640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63cc0751e0 0x7f63cc19be20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f63c000ea60 tx=0x7f63c000ef20 comp rx=0 tx=0).stop 2024-07-26T12:48:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.815+0000 7f63d2e00640 1 -- 172.21.15.107:0/295557898 shutdown_connections 2024-07-26T12:48:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.815+0000 7f63d2e00640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f63b40b3ca0 0x7f63b40b61b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.815+0000 7f63d2e00640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f63cc10f870 0x7f63cc1a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.815+0000 7f63d2e00640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f63cc075b30 0x7f63cc19c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.815+0000 7f63d2e00640 1 --2- 172.21.15.107:0/295557898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63cc0751e0 0x7f63cc19be20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.815+0000 7f63d2e00640 1 -- 172.21.15.107:0/295557898 >> 172.21.15.107:0/295557898 conn(0x7f63cc0fed60 msgr2=0x7f63cc109310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.815+0000 7f63d2e00640 1 -- 172.21.15.107:0/295557898 shutdown_connections 2024-07-26T12:48:40.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:40.815+0000 7f63d2e00640 1 -- 172.21.15.107:0/295557898 wait complete. 2024-07-26T12:48:40.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-07-26T12:48:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:40 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3642944435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T12:48:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:40 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/1959366367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T12:48:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:40 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3642944435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T12:48:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:40 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/1959366367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T12:48:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:40 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3642944435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T12:48:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:40 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/1959366367' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T12:48:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.113+0000 7fd356400640 1 -- 172.21.15.107:0/4196486246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd350105700 msgr2=0x7fd350105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:41.114 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.113+0000 7fd356400640 1 --2- 172.21.15.107:0/4196486246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd350105700 0x7fd350105b10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd340009860 tx=0x7fd34002a340 comp rx=0 tx=0).stop 2024-07-26T12:48:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.114+0000 7fd356400640 1 -- 172.21.15.107:0/4196486246 shutdown_connections 2024-07-26T12:48:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.114+0000 7fd356400640 1 --2- 172.21.15.107:0/4196486246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd35010b980 0x7fd3501121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.114+0000 7fd356400640 1 --2- 172.21.15.107:0/4196486246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd35010b030 0x7fd35010b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.114+0000 7fd356400640 1 --2- 172.21.15.107:0/4196486246 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd350105700 0x7fd350105b10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.114+0000 7fd356400640 1 -- 172.21.15.107:0/4196486246 >> 172.21.15.107:0/4196486246 conn(0x7fd350101120 msgr2=0x7fd350103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.114+0000 7fd356400640 1 -- 172.21.15.107:0/4196486246 shutdown_connections 2024-07-26T12:48:41.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.114+0000 7fd356400640 1 -- 172.21.15.107:0/4196486246 wait complete. 2024-07-26T12:48:41.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.115+0000 7fd356400640 1 Processor -- start 2024-07-26T12:48:41.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.115+0000 7fd356400640 1 -- start start 2024-07-26T12:48:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd356400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd350105700 0x7fd35019c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd356400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd35010b030 0x7fd35019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd356400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd35010b980 0x7fd3501a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd356400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd350115650 con 0x7fd35010b030 2024-07-26T12:48:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd356400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501154d0 con 0x7fd350105700 2024-07-26T12:48:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd356400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fd3501157d0 con 0x7fd35010b980 2024-07-26T12:48:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd34f400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd350105700 0x7fd35019c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd34ea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd35010b030 0x7fd35019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd34f400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd350105700 0x7fd35019c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:48998/0 (socket says 172.21.15.107:48998) 2024-07-26T12:48:41.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd34f400640 1 -- 172.21.15.107:0/3616785362 learned_addr learned my addr 172.21.15.107:0/3616785362 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd34fe00640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd35010b980 0x7fd3501a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:41.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.116+0000 7fd34ea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd35010b030 0x7fd35019c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:55810/0 (socket says 172.21.15.107:55810) 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd34f400640 1 -- 172.21.15.107:0/3616785362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd35010b980 msgr2=0x7fd3501a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd34f400640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd35010b980 0x7fd3501a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd34f400640 1 -- 172.21.15.107:0/3616785362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd35010b030 msgr2=0x7fd35019c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd34f400640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd35010b030 0x7fd35019c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd34f400640 1 -- 172.21.15.107:0/3616785362 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3501a2020 con 0x7fd350105700 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd34fe00640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd35010b980 0x7fd3501a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd34f400640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd350105700 0x7fd35019c0f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd3400303f0 tx=0x7fd340032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd347e00640 1 -- 172.21.15.107:0/3616785362 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340040020 con 0x7fd350105700 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd356400640 1 -- 172.21.15.107:0/3616785362 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3501a22b0 con 0x7fd350105700 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd347e00640 1 -- 172.21.15.107:0/3616785362 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd340030dd0 con 0x7fd350105700 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd347e00640 1 -- 172.21.15.107:0/3616785362 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340037040 con 0x7fd350105700 2024-07-26T12:48:41.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.117+0000 7fd356400640 1 -- 172.21.15.107:0/3616785362 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3501a8aa0 con 0x7fd350105700 2024-07-26T12:48:41.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.118+0000 7fd356400640 1 -- 172.21.15.107:0/3616785362 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd350105b10 con 0x7fd350105700 2024-07-26T12:48:41.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.121+0000 7fd347e00640 1 -- 172.21.15.107:0/3616785362 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fd340043020 con 0x7fd350105700 2024-07-26T12:48:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.122+0000 7fd347e00640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3340b3ca0 0x7fd3340b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.122+0000 7fd347e00640 1 -- 172.21.15.107:0/3616785362 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd340104190 con 0x7fd350105700 2024-07-26T12:48:41.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.123+0000 7fd34ea00640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3340b3ca0 0x7fd3340b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.123+0000 7fd34ea00640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3340b3ca0 0x7fd3340b61b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd350069b80 tx=0x7fd33c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.124+0000 7fd347e00640 1 -- 172.21.15.107:0/3616785362 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd34004a020 con 0x7fd350105700 2024-07-26T12:48:41.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.332+0000 7fd356400640 1 -- 172.21.15.107:0/3616785362 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd35010d170 con 0x7fd350105700 2024-07-26T12:48:41.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.333+0000 7fd347e00640 1 -- 172.21.15.107:0/3616785362 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7fd3400328c0 con 0x7fd350105700 2024-07-26T12:48:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 32 2024-07-26T12:48:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 -- 172.21.15.107:0/3616785362 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3340b3ca0 msgr2=0x7fd3340b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3340b3ca0 0x7fd3340b61b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd350069b80 tx=0x7fd33c009260 comp rx=0 tx=0).stop 2024-07-26T12:48:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 -- 172.21.15.107:0/3616785362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd350105700 msgr2=0x7fd35019c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd350105700 0x7fd35019c0f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd3400303f0 tx=0x7fd340032da0 comp rx=0 tx=0).stop 2024-07-26T12:48:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 -- 172.21.15.107:0/3616785362 shutdown_connections 2024-07-26T12:48:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fd3340b3ca0 0x7fd3340b61b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd35010b980 0x7fd3501a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd35010b030 0x7fd35019c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 --2- 172.21.15.107:0/3616785362 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fd350105700 0x7fd35019c0f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:41.337 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 -- 172.21.15.107:0/3616785362 >> 172.21.15.107:0/3616785362 conn(0x7fd350101120 msgr2=0x7fd350103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:41.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 -- 172.21.15.107:0/3616785362 shutdown_connections 2024-07-26T12:48:41.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:41.336+0000 7fd356400640 1 -- 172.21.15.107:0/3616785362 wait complete. 2024-07-26T12:48:41.365 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-07-26T12:48:41.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:48:41.819 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:41 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/295557898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T12:48:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:41 smithi112 ceph-mon[37501]: pgmap v471: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:41 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3616785362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T12:48:41.967 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:41 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/295557898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T12:48:41.967 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:41 smithi107 ceph-mon[32659]: pgmap v471: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:41.968 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:41 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3616785362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T12:48:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:41 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/295557898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T12:48:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:41 smithi132 ceph-mon[37580]: pgmap v471: 129 pgs: 129 active+clean; 647 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:48:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:41 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3616785362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T12:48:44.186 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:43 smithi107 ceph-mon[32659]: pgmap v472: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T12:48:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:43 smithi112 ceph-mon[37501]: pgmap v472: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T12:48:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:43 smithi132 ceph-mon[37580]: pgmap v472: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-07-26T12:48:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.548+0000 7f760f200640 1 -- 172.21.15.107:0/3090434604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7610108d20 msgr2=0x7f7610109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.548+0000 7f760f200640 1 --2- 172.21.15.107:0/3090434604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7610108d20 0x7f7610109130 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009900 tx=0x7f75f802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:48:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.548+0000 7f760f200640 1 -- 172.21.15.107:0/3090434604 shutdown_connections 2024-07-26T12:48:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.548+0000 7f760f200640 1 --2- 172.21.15.107:0/3090434604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7610109670 0x7f761010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.548+0000 7f760f200640 1 --2- 172.21.15.107:0/3090434604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7610108d20 0x7f7610109130 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.548+0000 7f760f200640 1 --2- 172.21.15.107:0/3090434604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f76101033e0 0x7f76101037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.548+0000 7f760f200640 1 -- 172.21.15.107:0/3090434604 >> 172.21.15.107:0/3090434604 conn(0x7f76100fee70 msgr2=0x7f76101012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.548+0000 7f760f200640 1 -- 172.21.15.107:0/3090434604 shutdown_connections 2024-07-26T12:48:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.548+0000 7f760f200640 1 -- 172.21.15.107:0/3090434604 wait complete. 2024-07-26T12:48:44.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.550+0000 7f760f200640 1 Processor -- start 2024-07-26T12:48:44.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.550+0000 7f760f200640 1 -- start start 2024-07-26T12:48:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.550+0000 7f760f200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76101033e0 0x7f7610199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.551+0000 7f760f200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7610108d20 0x7f761019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.551+0000 7f760f200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7610109670 0x7f761019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.551+0000 7f760f200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7610113350 con 0x7f76101033e0 2024-07-26T12:48:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.551+0000 7f760f200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f76101131d0 con 0x7f7610109670 2024-07-26T12:48:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.551+0000 7f760f200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f76101134d0 con 0x7f7610108d20 2024-07-26T12:48:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.551+0000 7f760de00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76101033e0 0x7f7610199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.551+0000 7f760de00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76101033e0 0x7f7610199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33402/0 (socket says 172.21.15.107:33402) 2024-07-26T12:48:44.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.551+0000 7f760de00640 1 -- 172.21.15.107:0/2110642117 learned_addr learned my addr 172.21.15.107:0/2110642117 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:44.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.551+0000 7f7605400640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7610108d20 0x7f761019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:44.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.551+0000 7f760e800640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7610109670 0x7f761019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:44.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f760e800640 1 -- 172.21.15.107:0/2110642117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7610108d20 msgr2=0x7f761019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f760e800640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7610108d20 0x7f761019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f760e800640 1 -- 172.21.15.107:0/2110642117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76101033e0 msgr2=0x7f7610199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f760e800640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76101033e0 0x7f7610199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f760e800640 1 -- 172.21.15.107:0/2110642117 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761019fcd0 con 0x7f7610109670 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f760de00640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76101033e0 0x7f7610199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f760e800640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7610109670 0x7f761019f5a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7608009fd0 tx=0x7f760800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f7607400640 1 -- 172.21.15.107:0/2110642117 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760800ce20 con 0x7f7610109670 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f7607400640 1 -- 172.21.15.107:0/2110642117 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7608005020 con 0x7f7610109670 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f760f200640 1 -- 172.21.15.107:0/2110642117 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f761019ffc0 con 0x7f7610109670 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.552+0000 7f7607400640 1 -- 172.21.15.107:0/2110642117 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7608010690 con 0x7f7610109670 2024-07-26T12:48:44.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.553+0000 7f760f200640 1 -- 172.21.15.107:0/2110642117 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7610075020 con 0x7f7610109670 2024-07-26T12:48:44.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.554+0000 7f760f200640 1 -- 172.21.15.107:0/2110642117 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76101037f0 con 0x7f7610109670 2024-07-26T12:48:44.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.556+0000 7f7607400640 1 -- 172.21.15.107:0/2110642117 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f76080054d0 con 0x7f7610109670 2024-07-26T12:48:44.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.558+0000 7f7607400640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f75f40b3c20 0x7f75f40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.558+0000 7f7607400640 1 -- 172.21.15.107:0/2110642117 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f76080e47d0 con 0x7f7610109670 2024-07-26T12:48:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.558+0000 7f760de00640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f75f40b3c20 0x7f75f40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:44.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.559+0000 7f760de00640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f75f40b3c20 0x7f75f40b6130 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f76000095f0 tx=0x7f7600005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:44.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.559+0000 7f7607400640 1 -- 172.21.15.107:0/2110642117 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f76080ab920 con 0x7f7610109670 2024-07-26T12:48:44.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.798+0000 7f760f200640 1 -- 172.21.15.107:0/2110642117 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f761010b4b0 con 0x7f7610109670 2024-07-26T12:48:44.800 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.799+0000 7f7607400640 1 -- 172.21.15.107:0/2110642117 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11993 (secure 0 0 0) 0x7f761010b4b0 con 0x7f7610109670 2024-07-26T12:48:44.800 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-07-26T12:48:44.801 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:48:44.801 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"btime":"2024-07-26T12:48:38:656386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:48:38.606034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24716,"mds_1":24715,"mds_2":24703,"mds_3":24694,"mds_4":24682},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24632":{"gid":24632,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24634":{"gid":24634,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24652":{"gid":24652,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24662":{"gid":24662,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":32,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24694":{"gid":24694,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":31,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":29,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":28,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24716,"qdb_cluster":[24715,24716,24703,24694,24682]},"id":1}]} 2024-07-26T12:48:44.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.802+0000 7f760f200640 1 -- 172.21.15.107:0/2110642117 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f75f40b3c20 msgr2=0x7f75f40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:44.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.802+0000 7f760f200640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f75f40b3c20 0x7f75f40b6130 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f76000095f0 tx=0x7f7600005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:44.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.802+0000 7f760f200640 1 -- 172.21.15.107:0/2110642117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7610109670 msgr2=0x7f761019f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:44.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.802+0000 7f760f200640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7610109670 0x7f761019f5a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7608009fd0 tx=0x7f760800ef10 comp rx=0 tx=0).stop 2024-07-26T12:48:44.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.803+0000 7f760f200640 1 -- 172.21.15.107:0/2110642117 shutdown_connections 2024-07-26T12:48:44.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.803+0000 7f760f200640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f75f40b3c20 0x7f75f40b6130 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:44.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.803+0000 7f760f200640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7610109670 0x7f761019f5a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:44.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.803+0000 7f760f200640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7610108d20 0x7f761019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:44.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.803+0000 7f760f200640 1 --2- 172.21.15.107:0/2110642117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76101033e0 0x7f7610199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:44.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.803+0000 7f760f200640 1 -- 172.21.15.107:0/2110642117 >> 172.21.15.107:0/2110642117 conn(0x7f76100fee70 msgr2=0x7f7610101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:44.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.803+0000 7f760f200640 1 -- 172.21.15.107:0/2110642117 shutdown_connections 2024-07-26T12:48:44.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:44.803+0000 7f760f200640 1 -- 172.21.15.107:0/2110642117 wait complete. 2024-07-26T12:48:45.119 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:44 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/2110642117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:45.119 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:44 smithi107 ceph-mon[32659]: pgmap v473: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-07-26T12:48:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:44 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/2110642117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:44 smithi112 ceph-mon[37501]: pgmap v473: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-07-26T12:48:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:44 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/2110642117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:44 smithi132 ceph-mon[37580]: pgmap v473: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-07-26T12:48:45.771 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-07-26T12:48:45.771 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-07-26T12:48:45.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T12:48:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.026+0000 7f094be00640 1 -- 172.21.15.107:0/905936543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f094c102790 msgr2=0x7f094c10f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:46.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.026+0000 7f094be00640 1 --2- 172.21.15.107:0/905936543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f094c102790 0x7f094c10f4b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0934009860 tx=0x7f093402a3d0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.026+0000 7f094be00640 1 -- 172.21.15.107:0/905936543 shutdown_connections 2024-07-26T12:48:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.026+0000 7f094be00640 1 --2- 172.21.15.107:0/905936543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f094c10f9f0 0x7f094c111e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.026+0000 7f094be00640 1 --2- 172.21.15.107:0/905936543 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f094c102790 0x7f094c10f4b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.026+0000 7f094be00640 1 --2- 172.21.15.107:0/905936543 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f094c101e40 0x7f094c102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.026+0000 7f094be00640 1 -- 172.21.15.107:0/905936543 >> 172.21.15.107:0/905936543 conn(0x7f094c079f10 msgr2=0x7f094c07a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.026+0000 7f094be00640 1 -- 172.21.15.107:0/905936543 shutdown_connections 2024-07-26T12:48:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.026+0000 7f094be00640 1 -- 172.21.15.107:0/905936543 wait complete. 2024-07-26T12:48:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.027+0000 7f094be00640 1 Processor -- start 2024-07-26T12:48:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094be00640 1 -- start start 2024-07-26T12:48:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f094c101e40 0x7f094c19be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f094c102790 0x7f094c19c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094be00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f094c10f9f0 0x7f094c1a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094be00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f094c115390 con 0x7f094c10f9f0 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094be00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f094c115210 con 0x7f094c102790 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f094c115510 con 0x7f094c101e40 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094aa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f094c101e40 0x7f094c19be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094aa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f094c101e40 0x7f094c19be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:36294/0 (socket says 172.21.15.107:36294) 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094aa00640 1 -- 172.21.15.107:0/3615810482 learned_addr learned my addr 172.21.15.107:0/3615810482 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f094b400640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f094c10f9f0 0x7f094c1a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.028+0000 7f0942000640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f094c102790 0x7f094c19c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.029+0000 7f094aa00640 1 -- 172.21.15.107:0/3615810482 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f094c102790 msgr2=0x7f094c19c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.029+0000 7f094aa00640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f094c102790 0x7f094c19c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.029+0000 7f094aa00640 1 -- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f094c10f9f0 msgr2=0x7f094c1a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.029+0000 7f094aa00640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f094c10f9f0 0x7f094c1a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.029+0000 7f094aa00640 1 -- 172.21.15.107:0/3615810482 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094c1a1d40 con 0x7f094c101e40 2024-07-26T12:48:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.029+0000 7f094b400640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f094c10f9f0 0x7f094c1a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.029+0000 7f094aa00640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f094c101e40 0x7f094c19be10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f093c00e9f0 tx=0x7f093c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.029+0000 7f0943e00640 1 -- 172.21.15.107:0/3615810482 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c00cd80 con 0x7f094c101e40 2024-07-26T12:48:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.030+0000 7f094be00640 1 -- 172.21.15.107:0/3615810482 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f094c1a2030 con 0x7f094c101e40 2024-07-26T12:48:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.030+0000 7f0943e00640 1 -- 172.21.15.107:0/3615810482 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f093c0108e0 con 0x7f094c101e40 2024-07-26T12:48:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.030+0000 7f0943e00640 1 -- 172.21.15.107:0/3615810482 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c018980 con 0x7f094c101e40 2024-07-26T12:48:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.030+0000 7f094be00640 1 -- 172.21.15.107:0/3615810482 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f094c1a87c0 con 0x7f094c101e40 2024-07-26T12:48:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.031+0000 7f094be00640 1 -- 172.21.15.107:0/3615810482 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f094c102250 con 0x7f094c101e40 2024-07-26T12:48:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.033+0000 7f0943e00640 1 -- 172.21.15.107:0/3615810482 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f093c010430 con 0x7f094c101e40 2024-07-26T12:48:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.035+0000 7f0943e00640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f09300b3c20 0x7f09300b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.035+0000 7f0943e00640 1 -- 172.21.15.107:0/3615810482 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f093c0e4930 con 0x7f094c101e40 2024-07-26T12:48:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.035+0000 7f0942000640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f09300b3c20 0x7f09300b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.036+0000 7f0942000640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f09300b3c20 0x7f09300b6130 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f094c101950 tx=0x7f0934009490 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.036+0000 7f0943e00640 1 -- 172.21.15.107:0/3615810482 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f093c0ab430 con 0x7f094c101e40 2024-07-26T12:48:46.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.247+0000 7f094be00640 1 -- 172.21.15.107:0/3615810482 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f094c063230 con 0x7f09300b3c20 2024-07-26T12:48:46.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.256+0000 7f0943e00640 1 -- 172.21.15.107:0/3615810482 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340543 (secure 0 0 0) 0x7f094c063230 con 0x7f09300b3c20 2024-07-26T12:48:46.262 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:48:46.264 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-07-26T12:48:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.263+0000 7f094be00640 1 -- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f09300b3c20 msgr2=0x7f09300b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.263+0000 7f094be00640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f09300b3c20 0x7f09300b6130 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f094c101950 tx=0x7f0934009490 comp rx=0 tx=0).stop 2024-07-26T12:48:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.263+0000 7f094be00640 1 -- 172.21.15.107:0/3615810482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f094c101e40 msgr2=0x7f094c19be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.263+0000 7f094be00640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f094c101e40 0x7f094c19be10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f093c00e9f0 tx=0x7f093c00eeb0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.263+0000 7f094be00640 1 -- 172.21.15.107:0/3615810482 shutdown_connections 2024-07-26T12:48:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.263+0000 7f094be00640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f09300b3c20 0x7f09300b6130 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.263+0000 7f094be00640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f094c10f9f0 0x7f094c1a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.263+0000 7f094be00640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f094c102790 0x7f094c19c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.263+0000 7f094be00640 1 --2- 172.21.15.107:0/3615810482 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f094c101e40 0x7f094c19be10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.263+0000 7f094be00640 1 -- 172.21.15.107:0/3615810482 >> 172.21.15.107:0/3615810482 conn(0x7f094c079f10 msgr2=0x7f094c110e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:46.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.264+0000 7f094be00640 1 -- 172.21.15.107:0/3615810482 shutdown_connections 2024-07-26T12:48:46.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.264+0000 7f094be00640 1 -- 172.21.15.107:0/3615810482 wait complete. 2024-07-26T12:48:46.289 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-07-26T12:48:44.817705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668153,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2556,"num_read_kb":1391,"num_write":390,"num_write_kb":2312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2951874,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":822,"ondisk_log_size":822,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376068,"kb_used_data":14436,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565756,"statfs":{"total":1151940427776,"available":1151555334144,"internally_reserved":0,"allocated":14782464,"data_stored":8392715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":116,"num_read_kb":62,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006243"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":302,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:39.370594+0000","last_change":"2024-07-26T12:43:08.129417+0000","last_active":"2024-07-26T12:48:39.370594+0000","last_peered":"2024-07-26T12:48:39.370594+0000","last_clean":"2024-07-26T12:48:39.370594+0000","last_became_active":"2024-07-26T12:43:08.128919+0000","last_became_peered":"2024-07-26T12:43:08.128919+0000","last_unstale":"2024-07-26T12:48:39.370594+0000","last_undegraded":"2024-07-26T12:48:39.370594+0000","last_fullsized":"2024-07-26T12:48:39.370594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:35:55.414185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.658083+0000","last_change":"2024-07-26T12:43:12.172058+0000","last_active":"2024-07-26T12:46:34.658083+0000","last_peered":"2024-07-26T12:46:34.658083+0000","last_clean":"2024-07-26T12:46:34.658083+0000","last_became_active":"2024-07-26T12:43:12.171631+0000","last_became_peered":"2024-07-26T12:43:12.171631+0000","last_unstale":"2024-07-26T12:46:34.658083+0000","last_undegraded":"2024-07-26T12:46:34.658083+0000","last_fullsized":"2024-07-26T12:46:34.658083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:51.972962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:54.250635+0000","last_change":"2024-07-26T12:43:26.773739+0000","last_active":"2024-07-26T12:45:54.250635+0000","last_peered":"2024-07-26T12:45:54.250635+0000","last_clean":"2024-07-26T12:45:54.250635+0000","last_became_active":"2024-07-26T12:43:26.773254+0000","last_became_peered":"2024-07-26T12:43:26.773254+0000","last_unstale":"2024-07-26T12:45:54.250635+0000","last_undegraded":"2024-07-26T12:45:54.250635+0000","last_fullsized":"2024-07-26T12:45:54.250635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:03.125497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:11.254691+0000","last_change":"2024-07-26T12:43:08.227329+0000","last_active":"2024-07-26T12:45:11.254691+0000","last_peered":"2024-07-26T12:45:11.254691+0000","last_clean":"2024-07-26T12:45:11.254691+0000","last_became_active":"2024-07-26T12:43:08.226674+0000","last_became_peered":"2024-07-26T12:43:08.226674+0000","last_unstale":"2024-07-26T12:45:11.254691+0000","last_undegraded":"2024-07-26T12:45:11.254691+0000","last_fullsized":"2024-07-26T12:45:11.254691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.222684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674670+0000","last_change":"2024-07-26T12:43:12.190616+0000","last_active":"2024-07-26T12:46:34.674670+0000","last_peered":"2024-07-26T12:46:34.674670+0000","last_clean":"2024-07-26T12:46:34.674670+0000","last_became_active":"2024-07-26T12:43:12.189717+0000","last_became_peered":"2024-07-26T12:43:12.189717+0000","last_unstale":"2024-07-26T12:46:34.674670+0000","last_undegraded":"2024-07-26T12:46:34.674670+0000","last_fullsized":"2024-07-26T12:46:34.674670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:03.509592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:51.766027+0000","last_change":"2024-07-26T12:43:26.774486+0000","last_active":"2024-07-26T12:45:51.766027+0000","last_peered":"2024-07-26T12:45:51.766027+0000","last_clean":"2024-07-26T12:45:51.766027+0000","last_became_active":"2024-07-26T12:43:26.774046+0000","last_became_peered":"2024-07-26T12:43:26.774046+0000","last_unstale":"2024-07-26T12:45:51.766027+0000","last_undegraded":"2024-07-26T12:45:51.766027+0000","last_fullsized":"2024-07-26T12:45:51.766027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.847072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:37.304088+0000","last_change":"2024-07-26T12:43:08.123084+0000","last_active":"2024-07-26T12:45:37.304088+0000","last_peered":"2024-07-26T12:45:37.304088+0000","last_clean":"2024-07-26T12:45:37.304088+0000","last_became_active":"2024-07-26T12:43:08.122371+0000","last_became_peered":"2024-07-26T12:43:08.122371+0000","last_unstale":"2024-07-26T12:45:37.304088+0000","last_undegraded":"2024-07-26T12:45:37.304088+0000","last_fullsized":"2024-07-26T12:45:37.304088+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:27:09.665660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672192+0000","last_change":"2024-07-26T12:43:12.170290+0000","last_active":"2024-07-26T12:46:34.672192+0000","last_peered":"2024-07-26T12:46:34.672192+0000","last_clean":"2024-07-26T12:46:34.672192+0000","last_became_active":"2024-07-26T12:43:12.169849+0000","last_became_peered":"2024-07-26T12:43:12.169849+0000","last_unstale":"2024-07-26T12:46:34.672192+0000","last_undegraded":"2024-07-26T12:46:34.672192+0000","last_fullsized":"2024-07-26T12:46:34.672192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:46.044921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:56.766737+0000","last_change":"2024-07-26T12:43:26.764804+0000","last_active":"2024-07-26T12:45:56.766737+0000","last_peered":"2024-07-26T12:45:56.766737+0000","last_clean":"2024-07-26T12:45:56.766737+0000","last_became_active":"2024-07-26T12:43:26.764065+0000","last_became_peered":"2024-07-26T12:43:26.764065+0000","last_unstale":"2024-07-26T12:45:56.766737+0000","last_undegraded":"2024-07-26T12:45:56.766737+0000","last_fullsized":"2024-07-26T12:45:56.766737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:16:59.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:37.304036+0000","last_change":"2024-07-26T12:43:08.121496+0000","last_active":"2024-07-26T12:45:37.304036+0000","last_peered":"2024-07-26T12:45:37.304036+0000","last_clean":"2024-07-26T12:45:37.304036+0000","last_became_active":"2024-07-26T12:43:08.120962+0000","last_became_peered":"2024-07-26T12:43:08.120962+0000","last_unstale":"2024-07-26T12:45:37.304036+0000","last_undegraded":"2024-07-26T12:45:37.304036+0000","last_fullsized":"2024-07-26T12:45:37.304036+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:42:02.998023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655232+0000","last_change":"2024-07-26T12:43:12.168707+0000","last_active":"2024-07-26T12:46:34.655232+0000","last_peered":"2024-07-26T12:46:34.655232+0000","last_clean":"2024-07-26T12:46:34.655232+0000","last_became_active":"2024-07-26T12:43:12.168338+0000","last_became_peered":"2024-07-26T12:43:12.168338+0000","last_unstale":"2024-07-26T12:46:34.655232+0000","last_undegraded":"2024-07-26T12:46:34.655232+0000","last_fullsized":"2024-07-26T12:46:34.655232+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:39:44.724864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:56.766714+0000","last_change":"2024-07-26T12:43:26.770863+0000","last_active":"2024-07-26T12:45:56.766714+0000","last_peered":"2024-07-26T12:45:56.766714+0000","last_clean":"2024-07-26T12:45:56.766714+0000","last_became_active":"2024-07-26T12:43:26.770480+0000","last_became_peered":"2024-07-26T12:43:26.770480+0000","last_unstale":"2024-07-26T12:45:56.766714+0000","last_undegraded":"2024-07-26T12:45:56.766714+0000","last_fullsized":"2024-07-26T12:45:56.766714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:09.128364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672691+0000","last_change":"2024-07-26T12:43:12.181659+0000","last_active":"2024-07-26T12:46:34.672691+0000","last_peered":"2024-07-26T12:46:34.672691+0000","last_clean":"2024-07-26T12:46:34.672691+0000","last_became_active":"2024-07-26T12:43:12.181268+0000","last_became_peered":"2024-07-26T12:43:12.181268+0000","last_unstale":"2024-07-26T12:46:34.672691+0000","last_undegraded":"2024-07-26T12:46:34.672691+0000","last_fullsized":"2024-07-26T12:46:34.672691+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:38:28.445981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:51.540084+0000","last_change":"2024-07-26T12:43:35.856273+0000","last_active":"2024-07-26T12:45:51.540084+0000","last_peered":"2024-07-26T12:45:51.540084+0000","last_clean":"2024-07-26T12:45:51.540084+0000","last_became_active":"2024-07-26T12:43:35.849625+0000","last_became_peered":"2024-07-26T12:43:35.849625+0000","last_unstale":"2024-07-26T12:45:51.540084+0000","last_undegraded":"2024-07-26T12:45:51.540084+0000","last_fullsized":"2024-07-26T12:45:51.540084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:34:45.261474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:41.764778+0000","last_change":"2024-07-26T12:43:26.793431+0000","last_active":"2024-07-26T12:45:41.764778+0000","last_peered":"2024-07-26T12:45:41.764778+0000","last_clean":"2024-07-26T12:45:41.764778+0000","last_became_active":"2024-07-26T12:43:26.792931+0000","last_became_peered":"2024-07-26T12:43:26.792931+0000","last_unstale":"2024-07-26T12:45:41.764778+0000","last_undegraded":"2024-07-26T12:45:41.764778+0000","last_fullsized":"2024-07-26T12:45:41.764778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:31.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669969+0000","last_change":"2024-07-26T12:43:12.183784+0000","last_active":"2024-07-26T12:46:34.669969+0000","last_peered":"2024-07-26T12:46:34.669969+0000","last_clean":"2024-07-26T12:46:34.669969+0000","last_became_active":"2024-07-26T12:43:12.182696+0000","last_became_peered":"2024-07-26T12:43:12.182696+0000","last_unstale":"2024-07-26T12:46:34.669969+0000","last_undegraded":"2024-07-26T12:46:34.669969+0000","last_fullsized":"2024-07-26T12:46:34.669969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:33:19.918180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:19.245986+0000","last_change":"2024-07-26T12:43:08.131069+0000","last_active":"2024-07-26T12:45:19.245986+0000","last_peered":"2024-07-26T12:45:19.245986+0000","last_clean":"2024-07-26T12:45:19.245986+0000","last_became_active":"2024-07-26T12:43:08.130637+0000","last_became_peered":"2024-07-26T12:43:08.130637+0000","last_unstale":"2024-07-26T12:45:19.245986+0000","last_undegraded":"2024-07-26T12:45:19.245986+0000","last_fullsized":"2024-07-26T12:45:19.245986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:04:16.461900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:41.764748+0000","last_change":"2024-07-26T12:43:26.763388+0000","last_active":"2024-07-26T12:45:41.764748+0000","last_peered":"2024-07-26T12:45:41.764748+0000","last_clean":"2024-07-26T12:45:41.764748+0000","last_became_active":"2024-07-26T12:43:26.762464+0000","last_became_peered":"2024-07-26T12:43:26.762464+0000","last_unstale":"2024-07-26T12:45:41.764748+0000","last_undegraded":"2024-07-26T12:45:41.764748+0000","last_fullsized":"2024-07-26T12:45:41.764748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:10:17.598266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668869+0000","last_change":"2024-07-26T12:43:12.191640+0000","last_active":"2024-07-26T12:46:34.668869+0000","last_peered":"2024-07-26T12:46:34.668869+0000","last_clean":"2024-07-26T12:46:34.668869+0000","last_became_active":"2024-07-26T12:43:12.191260+0000","last_became_peered":"2024-07-26T12:43:12.191260+0000","last_unstale":"2024-07-26T12:46:34.668869+0000","last_undegraded":"2024-07-26T12:46:34.668869+0000","last_fullsized":"2024-07-26T12:46:34.668869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:47:18.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.257126+0000","last_change":"2024-07-26T12:43:08.135035+0000","last_active":"2024-07-26T12:45:21.257126+0000","last_peered":"2024-07-26T12:45:21.257126+0000","last_clean":"2024-07-26T12:45:21.257126+0000","last_became_active":"2024-07-26T12:43:08.134204+0000","last_became_peered":"2024-07-26T12:43:08.134204+0000","last_unstale":"2024-07-26T12:45:21.257126+0000","last_undegraded":"2024-07-26T12:45:21.257126+0000","last_fullsized":"2024-07-26T12:45:21.257126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:38.954460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:52.306230+0000","last_change":"2024-07-26T12:43:35.853244+0000","last_active":"2024-07-26T12:45:52.306230+0000","last_peered":"2024-07-26T12:45:52.306230+0000","last_clean":"2024-07-26T12:45:52.306230+0000","last_became_active":"2024-07-26T12:43:35.852728+0000","last_became_peered":"2024-07-26T12:43:35.852728+0000","last_unstale":"2024-07-26T12:45:52.306230+0000","last_undegraded":"2024-07-26T12:45:52.306230+0000","last_fullsized":"2024-07-26T12:45:52.306230+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:20:39.080000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.635080+0000","last_change":"2024-07-26T12:43:12.172457+0000","last_active":"2024-07-26T12:46:34.635080+0000","last_peered":"2024-07-26T12:46:34.635080+0000","last_clean":"2024-07-26T12:46:34.635080+0000","last_became_active":"2024-07-26T12:43:12.172034+0000","last_became_peered":"2024-07-26T12:43:12.172034+0000","last_unstale":"2024-07-26T12:46:34.635080+0000","last_undegraded":"2024-07-26T12:46:34.635080+0000","last_fullsized":"2024-07-26T12:46:34.635080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:37:02.237852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.698668+0000","last_change":"2024-07-26T12:43:08.227152+0000","last_active":"2024-07-26T12:45:21.698668+0000","last_peered":"2024-07-26T12:45:21.698668+0000","last_clean":"2024-07-26T12:45:21.698668+0000","last_became_active":"2024-07-26T12:43:08.226722+0000","last_became_peered":"2024-07-26T12:43:08.226722+0000","last_unstale":"2024-07-26T12:45:21.698668+0000","last_undegraded":"2024-07-26T12:45:21.698668+0000","last_fullsized":"2024-07-26T12:45:21.698668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:16.293211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:41.764791+0000","last_change":"2024-07-26T12:43:26.792999+0000","last_active":"2024-07-26T12:45:41.764791+0000","last_peered":"2024-07-26T12:45:41.764791+0000","last_clean":"2024-07-26T12:45:41.764791+0000","last_became_active":"2024-07-26T12:43:26.792515+0000","last_became_peered":"2024-07-26T12:43:26.792515+0000","last_unstale":"2024-07-26T12:45:41.764791+0000","last_undegraded":"2024-07-26T12:45:41.764791+0000","last_fullsized":"2024-07-26T12:45:41.764791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:49:35.104600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.673277+0000","last_change":"2024-07-26T12:43:12.185381+0000","last_active":"2024-07-26T12:46:34.673277+0000","last_peered":"2024-07-26T12:46:34.673277+0000","last_clean":"2024-07-26T12:46:34.673277+0000","last_became_active":"2024-07-26T12:43:12.184892+0000","last_became_peered":"2024-07-26T12:43:12.184892+0000","last_unstale":"2024-07-26T12:46:34.673277+0000","last_undegraded":"2024-07-26T12:46:34.673277+0000","last_fullsized":"2024-07-26T12:46:34.673277+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:38:53.992586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.762689+0000","last_change":"2024-07-26T12:43:08.895079+0000","last_active":"2024-07-26T12:45:21.762689+0000","last_peered":"2024-07-26T12:45:21.762689+0000","last_clean":"2024-07-26T12:45:21.762689+0000","last_became_active":"2024-07-26T12:43:08.894653+0000","last_became_peered":"2024-07-26T12:43:08.894653+0000","last_unstale":"2024-07-26T12:45:21.762689+0000","last_undegraded":"2024-07-26T12:45:21.762689+0000","last_fullsized":"2024-07-26T12:45:21.762689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:02:53.137905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:34.247883+0000","last_change":"2024-07-26T12:43:26.762802+0000","last_active":"2024-07-26T12:45:34.247883+0000","last_peered":"2024-07-26T12:45:34.247883+0000","last_clean":"2024-07-26T12:45:34.247883+0000","last_became_active":"2024-07-26T12:43:26.762211+0000","last_became_peered":"2024-07-26T12:43:26.762211+0000","last_unstale":"2024-07-26T12:45:34.247883+0000","last_undegraded":"2024-07-26T12:45:34.247883+0000","last_fullsized":"2024-07-26T12:45:34.247883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:35:19.623168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662791+0000","last_change":"2024-07-26T12:43:12.176062+0000","last_active":"2024-07-26T12:46:34.662791+0000","last_peered":"2024-07-26T12:46:34.662791+0000","last_clean":"2024-07-26T12:46:34.662791+0000","last_became_active":"2024-07-26T12:43:12.175671+0000","last_became_peered":"2024-07-26T12:43:12.175671+0000","last_unstale":"2024-07-26T12:46:34.662791+0000","last_undegraded":"2024-07-26T12:46:34.662791+0000","last_fullsized":"2024-07-26T12:46:34.662791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:48.374964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'18","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:47:31.284996+0000","last_change":"2024-07-26T12:43:08.131453+0000","last_active":"2024-07-26T12:47:31.284996+0000","last_peered":"2024-07-26T12:47:31.284996+0000","last_clean":"2024-07-26T12:47:31.284996+0000","last_became_active":"2024-07-26T12:43:08.130928+0000","last_became_peered":"2024-07-26T12:43:08.130928+0000","last_unstale":"2024-07-26T12:47:31.284996+0000","last_undegraded":"2024-07-26T12:47:31.284996+0000","last_fullsized":"2024-07-26T12:47:31.284996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:43.190860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:26.258119+0000","last_change":"2024-07-26T12:43:26.792942+0000","last_active":"2024-07-26T12:45:26.258119+0000","last_peered":"2024-07-26T12:45:26.258119+0000","last_clean":"2024-07-26T12:45:26.258119+0000","last_became_active":"2024-07-26T12:43:26.792045+0000","last_became_peered":"2024-07-26T12:43:26.792045+0000","last_unstale":"2024-07-26T12:45:26.258119+0000","last_undegraded":"2024-07-26T12:45:26.258119+0000","last_fullsized":"2024-07-26T12:45:26.258119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:56.005295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662936+0000","last_change":"2024-07-26T12:43:12.187601+0000","last_active":"2024-07-26T12:46:34.662936+0000","last_peered":"2024-07-26T12:46:34.662936+0000","last_clean":"2024-07-26T12:46:34.662936+0000","last_became_active":"2024-07-26T12:43:12.187175+0000","last_became_peered":"2024-07-26T12:43:12.187175+0000","last_unstale":"2024-07-26T12:46:34.662936+0000","last_undegraded":"2024-07-26T12:46:34.662936+0000","last_fullsized":"2024-07-26T12:46:34.662936+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:06.992854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:12.301049+0000","last_change":"2024-07-26T12:43:08.227279+0000","last_active":"2024-07-26T12:45:12.301049+0000","last_peered":"2024-07-26T12:45:12.301049+0000","last_clean":"2024-07-26T12:45:12.301049+0000","last_became_active":"2024-07-26T12:43:08.226634+0000","last_became_peered":"2024-07-26T12:43:08.226634+0000","last_unstale":"2024-07-26T12:45:12.301049+0000","last_undegraded":"2024-07-26T12:45:12.301049+0000","last_fullsized":"2024-07-26T12:45:12.301049+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:19.845664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:31.537561+0000","last_change":"2024-07-26T12:43:26.774537+0000","last_active":"2024-07-26T12:45:31.537561+0000","last_peered":"2024-07-26T12:45:31.537561+0000","last_clean":"2024-07-26T12:45:31.537561+0000","last_became_active":"2024-07-26T12:43:26.774036+0000","last_became_peered":"2024-07-26T12:43:26.774036+0000","last_unstale":"2024-07-26T12:45:31.537561+0000","last_undegraded":"2024-07-26T12:45:31.537561+0000","last_fullsized":"2024-07-26T12:45:31.537561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:21.774498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662838+0000","last_change":"2024-07-26T12:44:36.857443+0000","last_active":"2024-07-26T12:46:34.662838+0000","last_peered":"2024-07-26T12:46:34.662838+0000","last_clean":"2024-07-26T12:46:34.662838+0000","last_became_active":"2024-07-26T12:44:36.857001+0000","last_became_peered":"2024-07-26T12:44:36.857001+0000","last_unstale":"2024-07-26T12:46:34.662838+0000","last_undegraded":"2024-07-26T12:46:34.662838+0000","last_fullsized":"2024-07-26T12:46:34.662838+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:08.236404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:16.261130+0000","last_change":"2024-07-26T12:43:08.227432+0000","last_active":"2024-07-26T12:45:16.261130+0000","last_peered":"2024-07-26T12:45:16.261130+0000","last_clean":"2024-07-26T12:45:16.261130+0000","last_became_active":"2024-07-26T12:43:08.226645+0000","last_became_peered":"2024-07-26T12:43:08.226645+0000","last_unstale":"2024-07-26T12:45:16.261130+0000","last_undegraded":"2024-07-26T12:45:16.261130+0000","last_fullsized":"2024-07-26T12:45:16.261130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:23.498855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:32.303272+0000","last_change":"2024-07-26T12:43:26.792874+0000","last_active":"2024-07-26T12:45:32.303272+0000","last_peered":"2024-07-26T12:45:32.303272+0000","last_clean":"2024-07-26T12:45:32.303272+0000","last_became_active":"2024-07-26T12:43:26.792371+0000","last_became_peered":"2024-07-26T12:43:26.792371+0000","last_unstale":"2024-07-26T12:45:32.303272+0000","last_undegraded":"2024-07-26T12:45:32.303272+0000","last_fullsized":"2024-07-26T12:45:32.303272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:25.753111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.654086+0000","last_change":"2024-07-26T12:43:12.185733+0000","last_active":"2024-07-26T12:46:34.654086+0000","last_peered":"2024-07-26T12:46:34.654086+0000","last_clean":"2024-07-26T12:46:34.654086+0000","last_became_active":"2024-07-26T12:43:12.185305+0000","last_became_peered":"2024-07-26T12:43:12.185305+0000","last_unstale":"2024-07-26T12:46:34.654086+0000","last_undegraded":"2024-07-26T12:46:34.654086+0000","last_fullsized":"2024-07-26T12:46:34.654086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:44.634607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":296,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:41.374820+0000","last_change":"2024-07-26T12:43:08.118508+0000","last_active":"2024-07-26T12:48:41.374820+0000","last_peered":"2024-07-26T12:48:41.374820+0000","last_clean":"2024-07-26T12:48:41.374820+0000","last_became_active":"2024-07-26T12:43:08.118088+0000","last_became_peered":"2024-07-26T12:43:08.118088+0000","last_unstale":"2024-07-26T12:48:41.374820+0000","last_undegraded":"2024-07-26T12:48:41.374820+0000","last_fullsized":"2024-07-26T12:48:41.374820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:23:46.474298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:33.076402+0000","last_change":"2024-07-26T12:43:26.779328+0000","last_active":"2024-07-26T12:45:33.076402+0000","last_peered":"2024-07-26T12:45:33.076402+0000","last_clean":"2024-07-26T12:45:33.076402+0000","last_became_active":"2024-07-26T12:43:26.778912+0000","last_became_peered":"2024-07-26T12:43:26.778912+0000","last_unstale":"2024-07-26T12:45:33.076402+0000","last_undegraded":"2024-07-26T12:45:33.076402+0000","last_fullsized":"2024-07-26T12:45:33.076402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:58.364775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650476+0000","last_change":"2024-07-26T12:43:35.854997+0000","last_active":"2024-07-26T12:46:34.650476+0000","last_peered":"2024-07-26T12:46:34.650476+0000","last_clean":"2024-07-26T12:46:34.650476+0000","last_became_active":"2024-07-26T12:43:35.854387+0000","last_became_peered":"2024-07-26T12:43:35.854387+0000","last_unstale":"2024-07-26T12:46:34.650476+0000","last_undegraded":"2024-07-26T12:46:34.650476+0000","last_fullsized":"2024-07-26T12:46:34.650476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:04:20.489388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":296,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:41.504837+0000","last_change":"2024-07-26T12:43:08.227628+0000","last_active":"2024-07-26T12:48:41.504837+0000","last_peered":"2024-07-26T12:48:41.504837+0000","last_clean":"2024-07-26T12:48:41.504837+0000","last_became_active":"2024-07-26T12:43:08.227133+0000","last_became_peered":"2024-07-26T12:43:08.227133+0000","last_unstale":"2024-07-26T12:48:41.504837+0000","last_undegraded":"2024-07-26T12:48:41.504837+0000","last_fullsized":"2024-07-26T12:48:41.504837+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:28:28.484315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:38.077398+0000","last_change":"2024-07-26T12:43:26.794016+0000","last_active":"2024-07-26T12:45:38.077398+0000","last_peered":"2024-07-26T12:45:38.077398+0000","last_clean":"2024-07-26T12:45:38.077398+0000","last_became_active":"2024-07-26T12:43:26.792562+0000","last_became_peered":"2024-07-26T12:43:26.792562+0000","last_unstale":"2024-07-26T12:45:38.077398+0000","last_undegraded":"2024-07-26T12:45:38.077398+0000","last_fullsized":"2024-07-26T12:45:38.077398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:26.162840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.653115+0000","last_change":"2024-07-26T12:43:12.173014+0000","last_active":"2024-07-26T12:46:34.653115+0000","last_peered":"2024-07-26T12:46:34.653115+0000","last_clean":"2024-07-26T12:46:34.653115+0000","last_became_active":"2024-07-26T12:43:12.172600+0000","last_became_peered":"2024-07-26T12:43:12.172600+0000","last_unstale":"2024-07-26T12:46:34.653115+0000","last_undegraded":"2024-07-26T12:46:34.653115+0000","last_fullsized":"2024-07-26T12:46:34.653115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:59.842744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:07.570667+0000","last_change":"2024-07-26T12:43:08.119000+0000","last_active":"2024-07-26T12:45:07.570667+0000","last_peered":"2024-07-26T12:45:07.570667+0000","last_clean":"2024-07-26T12:45:07.570667+0000","last_became_active":"2024-07-26T12:43:08.118608+0000","last_became_peered":"2024-07-26T12:43:08.118608+0000","last_unstale":"2024-07-26T12:45:07.570667+0000","last_undegraded":"2024-07-26T12:45:07.570667+0000","last_fullsized":"2024-07-26T12:45:07.570667+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:29:47.663449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:39.012097+0000","last_change":"2024-07-26T12:43:35.914716+0000","last_active":"2024-07-26T12:45:39.012097+0000","last_peered":"2024-07-26T12:45:39.012097+0000","last_clean":"2024-07-26T12:45:39.012097+0000","last_became_active":"2024-07-26T12:43:35.914354+0000","last_became_peered":"2024-07-26T12:43:35.914354+0000","last_unstale":"2024-07-26T12:45:39.012097+0000","last_undegraded":"2024-07-26T12:45:39.012097+0000","last_fullsized":"2024-07-26T12:45:39.012097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:09:10.994141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662686+0000","last_change":"2024-07-26T12:43:12.169997+0000","last_active":"2024-07-26T12:46:34.662686+0000","last_peered":"2024-07-26T12:46:34.662686+0000","last_clean":"2024-07-26T12:46:34.662686+0000","last_became_active":"2024-07-26T12:43:12.169044+0000","last_became_peered":"2024-07-26T12:43:12.169044+0000","last_unstale":"2024-07-26T12:46:34.662686+0000","last_undegraded":"2024-07-26T12:46:34.662686+0000","last_fullsized":"2024-07-26T12:46:34.662686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:41.791434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:16.762094+0000","last_change":"2024-07-26T12:43:08.129025+0000","last_active":"2024-07-26T12:45:16.762094+0000","last_peered":"2024-07-26T12:45:16.762094+0000","last_clean":"2024-07-26T12:45:16.762094+0000","last_became_active":"2024-07-26T12:43:08.128608+0000","last_became_peered":"2024-07-26T12:43:08.128608+0000","last_unstale":"2024-07-26T12:45:16.762094+0000","last_undegraded":"2024-07-26T12:45:16.762094+0000","last_fullsized":"2024-07-26T12:45:16.762094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:43.749473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":123,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:22.580061+0000","last_change":"2024-07-26T12:41:42.411595+0000","last_active":"2024-07-26T12:46:22.580061+0000","last_peered":"2024-07-26T12:46:22.580061+0000","last_clean":"2024-07-26T12:46:22.580061+0000","last_became_active":"2024-07-26T12:41:42.407119+0000","last_became_peered":"2024-07-26T12:41:42.407119+0000","last_unstale":"2024-07-26T12:46:22.580061+0000","last_undegraded":"2024-07-26T12:46:22.580061+0000","last_fullsized":"2024-07-26T12:46:22.580061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_clean_scrub_stamp":"2024-07-26T12:37:59.605754+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:21.376418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:32.573794+0000","last_change":"2024-07-26T12:43:26.763955+0000","last_active":"2024-07-26T12:45:32.573794+0000","last_peered":"2024-07-26T12:45:32.573794+0000","last_clean":"2024-07-26T12:45:32.573794+0000","last_became_active":"2024-07-26T12:43:26.763368+0000","last_became_peered":"2024-07-26T12:43:26.763368+0000","last_unstale":"2024-07-26T12:45:32.573794+0000","last_undegraded":"2024-07-26T12:45:32.573794+0000","last_fullsized":"2024-07-26T12:45:32.573794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:48:14.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.663123+0000","last_change":"2024-07-26T12:43:12.174780+0000","last_active":"2024-07-26T12:46:34.663123+0000","last_peered":"2024-07-26T12:46:34.663123+0000","last_clean":"2024-07-26T12:46:34.663123+0000","last_became_active":"2024-07-26T12:43:12.174379+0000","last_became_peered":"2024-07-26T12:43:12.174379+0000","last_unstale":"2024-07-26T12:46:34.663123+0000","last_undegraded":"2024-07-26T12:46:34.663123+0000","last_fullsized":"2024-07-26T12:46:34.663123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:42:22.783259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'7","reported_seq":302,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.278004+0000","last_change":"2024-07-26T12:43:08.894655+0000","last_active":"2024-07-26T12:48:42.278004+0000","last_peered":"2024-07-26T12:48:42.278004+0000","last_clean":"2024-07-26T12:48:42.278004+0000","last_became_active":"2024-07-26T12:43:08.894270+0000","last_became_peered":"2024-07-26T12:43:08.894270+0000","last_unstale":"2024-07-26T12:48:42.278004+0000","last_undegraded":"2024-07-26T12:48:42.278004+0000","last_fullsized":"2024-07-26T12:48:42.278004+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:34.431204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":248,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:36.700412+0000","last_change":"2024-07-26T12:43:26.769716+0000","last_active":"2024-07-26T12:45:36.700412+0000","last_peered":"2024-07-26T12:45:36.700412+0000","last_clean":"2024-07-26T12:45:36.700412+0000","last_became_active":"2024-07-26T12:43:26.769154+0000","last_became_peered":"2024-07-26T12:43:26.769154+0000","last_unstale":"2024-07-26T12:45:36.700412+0000","last_undegraded":"2024-07-26T12:45:36.700412+0000","last_fullsized":"2024-07-26T12:45:36.700412+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:43.102498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664039+0000","last_change":"2024-07-26T12:43:12.179488+0000","last_active":"2024-07-26T12:46:34.664039+0000","last_peered":"2024-07-26T12:46:34.664039+0000","last_clean":"2024-07-26T12:46:34.664039+0000","last_became_active":"2024-07-26T12:43:12.179171+0000","last_became_peered":"2024-07-26T12:43:12.179171+0000","last_unstale":"2024-07-26T12:46:34.664039+0000","last_undegraded":"2024-07-26T12:46:34.664039+0000","last_fullsized":"2024-07-26T12:46:34.664039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:43.695582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:24.009763+0000","last_change":"2024-07-26T12:43:08.130838+0000","last_active":"2024-07-26T12:45:24.009763+0000","last_peered":"2024-07-26T12:45:24.009763+0000","last_clean":"2024-07-26T12:45:24.009763+0000","last_became_active":"2024-07-26T12:43:08.130446+0000","last_became_peered":"2024-07-26T12:43:08.130446+0000","last_unstale":"2024-07-26T12:45:24.009763+0000","last_undegraded":"2024-07-26T12:45:24.009763+0000","last_fullsized":"2024-07-26T12:45:24.009763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:33:55.271038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:42.787196+0000","last_change":"2024-07-26T12:43:26.781982+0000","last_active":"2024-07-26T12:45:42.787196+0000","last_peered":"2024-07-26T12:45:42.787196+0000","last_clean":"2024-07-26T12:45:42.787196+0000","last_became_active":"2024-07-26T12:43:26.777102+0000","last_became_peered":"2024-07-26T12:43:26.777102+0000","last_unstale":"2024-07-26T12:45:42.787196+0000","last_undegraded":"2024-07-26T12:45:42.787196+0000","last_fullsized":"2024-07-26T12:45:42.787196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:49.553124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659860+0000","last_change":"2024-07-26T12:43:12.177043+0000","last_active":"2024-07-26T12:46:34.659860+0000","last_peered":"2024-07-26T12:46:34.659860+0000","last_clean":"2024-07-26T12:46:34.659860+0000","last_became_active":"2024-07-26T12:43:12.176634+0000","last_became_peered":"2024-07-26T12:43:12.176634+0000","last_unstale":"2024-07-26T12:46:34.659860+0000","last_undegraded":"2024-07-26T12:46:34.659860+0000","last_fullsized":"2024-07-26T12:46:34.659860+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:20.210176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:23.074306+0000","last_change":"2024-07-26T12:43:08.131279+0000","last_active":"2024-07-26T12:45:23.074306+0000","last_peered":"2024-07-26T12:45:23.074306+0000","last_clean":"2024-07-26T12:45:23.074306+0000","last_became_active":"2024-07-26T12:43:08.130755+0000","last_became_peered":"2024-07-26T12:43:08.130755+0000","last_unstale":"2024-07-26T12:45:23.074306+0000","last_undegraded":"2024-07-26T12:45:23.074306+0000","last_fullsized":"2024-07-26T12:45:23.074306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:03.234905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:39.012012+0000","last_change":"2024-07-26T12:43:26.764407+0000","last_active":"2024-07-26T12:45:39.012012+0000","last_peered":"2024-07-26T12:45:39.012012+0000","last_clean":"2024-07-26T12:45:39.012012+0000","last_became_active":"2024-07-26T12:43:26.762408+0000","last_became_peered":"2024-07-26T12:43:26.762408+0000","last_unstale":"2024-07-26T12:45:39.012012+0000","last_undegraded":"2024-07-26T12:45:39.012012+0000","last_fullsized":"2024-07-26T12:45:39.012012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:35:29.139476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.648078+0000","last_change":"2024-07-26T12:43:12.174587+0000","last_active":"2024-07-26T12:46:34.648078+0000","last_peered":"2024-07-26T12:46:34.648078+0000","last_clean":"2024-07-26T12:46:34.648078+0000","last_became_active":"2024-07-26T12:43:12.173929+0000","last_became_peered":"2024-07-26T12:43:12.173929+0000","last_unstale":"2024-07-26T12:46:34.648078+0000","last_undegraded":"2024-07-26T12:46:34.648078+0000","last_fullsized":"2024-07-26T12:46:34.648078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:16.775198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:22.302322+0000","last_change":"2024-07-26T12:43:08.119825+0000","last_active":"2024-07-26T12:45:22.302322+0000","last_peered":"2024-07-26T12:45:22.302322+0000","last_clean":"2024-07-26T12:45:22.302322+0000","last_became_active":"2024-07-26T12:43:08.119273+0000","last_became_peered":"2024-07-26T12:43:08.119273+0000","last_unstale":"2024-07-26T12:45:22.302322+0000","last_undegraded":"2024-07-26T12:45:22.302322+0000","last_fullsized":"2024-07-26T12:45:22.302322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:39.973020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:39.012041+0000","last_change":"2024-07-26T12:43:26.773226+0000","last_active":"2024-07-26T12:45:39.012041+0000","last_peered":"2024-07-26T12:45:39.012041+0000","last_clean":"2024-07-26T12:45:39.012041+0000","last_became_active":"2024-07-26T12:43:26.772745+0000","last_became_peered":"2024-07-26T12:43:26.772745+0000","last_unstale":"2024-07-26T12:45:39.012041+0000","last_undegraded":"2024-07-26T12:45:39.012041+0000","last_fullsized":"2024-07-26T12:45:39.012041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:35.407138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.648325+0000","last_change":"2024-07-26T12:43:12.176178+0000","last_active":"2024-07-26T12:46:34.648325+0000","last_peered":"2024-07-26T12:46:34.648325+0000","last_clean":"2024-07-26T12:46:34.648325+0000","last_became_active":"2024-07-26T12:43:12.175791+0000","last_became_peered":"2024-07-26T12:43:12.175791+0000","last_unstale":"2024-07-26T12:46:34.648325+0000","last_undegraded":"2024-07-26T12:46:34.648325+0000","last_fullsized":"2024-07-26T12:46:34.648325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:48.885831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:27.302835+0000","last_change":"2024-07-26T12:43:08.226345+0000","last_active":"2024-07-26T12:45:27.302835+0000","last_peered":"2024-07-26T12:45:27.302835+0000","last_clean":"2024-07-26T12:45:27.302835+0000","last_became_active":"2024-07-26T12:43:08.225897+0000","last_became_peered":"2024-07-26T12:43:08.225897+0000","last_unstale":"2024-07-26T12:45:27.302835+0000","last_undegraded":"2024-07-26T12:45:27.302835+0000","last_fullsized":"2024-07-26T12:45:27.302835+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:59.351076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:53.079873+0000","last_change":"2024-07-26T12:43:26.774800+0000","last_active":"2024-07-26T12:45:53.079873+0000","last_peered":"2024-07-26T12:45:53.079873+0000","last_clean":"2024-07-26T12:45:53.079873+0000","last_became_active":"2024-07-26T12:43:26.774192+0000","last_became_peered":"2024-07-26T12:43:26.774192+0000","last_unstale":"2024-07-26T12:45:53.079873+0000","last_undegraded":"2024-07-26T12:45:53.079873+0000","last_fullsized":"2024-07-26T12:45:53.079873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:18.379619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659204+0000","last_change":"2024-07-26T12:43:12.171720+0000","last_active":"2024-07-26T12:46:34.659204+0000","last_peered":"2024-07-26T12:46:34.659204+0000","last_clean":"2024-07-26T12:46:34.659204+0000","last_became_active":"2024-07-26T12:43:12.171334+0000","last_became_peered":"2024-07-26T12:43:12.171334+0000","last_unstale":"2024-07-26T12:46:34.659204+0000","last_undegraded":"2024-07-26T12:46:34.659204+0000","last_fullsized":"2024-07-26T12:46:34.659204+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:35.156430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":296,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:40.503197+0000","last_change":"2024-07-26T12:43:08.120325+0000","last_active":"2024-07-26T12:48:40.503197+0000","last_peered":"2024-07-26T12:48:40.503197+0000","last_clean":"2024-07-26T12:48:40.503197+0000","last_became_active":"2024-07-26T12:43:08.119896+0000","last_became_peered":"2024-07-26T12:43:08.119896+0000","last_unstale":"2024-07-26T12:48:40.503197+0000","last_undegraded":"2024-07-26T12:48:40.503197+0000","last_fullsized":"2024-07-26T12:48:40.503197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:23:19.470491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":245,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:51.266909+0000","last_change":"2024-07-26T12:43:26.778688+0000","last_active":"2024-07-26T12:45:51.266909+0000","last_peered":"2024-07-26T12:45:51.266909+0000","last_clean":"2024-07-26T12:45:51.266909+0000","last_became_active":"2024-07-26T12:43:26.778239+0000","last_became_peered":"2024-07-26T12:43:26.778239+0000","last_unstale":"2024-07-26T12:45:51.266909+0000","last_undegraded":"2024-07-26T12:45:51.266909+0000","last_fullsized":"2024-07-26T12:45:51.266909+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:50:48.409636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.676557+0000","last_change":"2024-07-26T12:43:12.182323+0000","last_active":"2024-07-26T12:46:34.676557+0000","last_peered":"2024-07-26T12:46:34.676557+0000","last_clean":"2024-07-26T12:46:34.676557+0000","last_became_active":"2024-07-26T12:43:12.181860+0000","last_became_peered":"2024-07-26T12:43:12.181860+0000","last_unstale":"2024-07-26T12:46:34.676557+0000","last_undegraded":"2024-07-26T12:46:34.676557+0000","last_fullsized":"2024-07-26T12:46:34.676557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:08.764979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:29.010371+0000","last_change":"2024-07-26T12:43:08.130823+0000","last_active":"2024-07-26T12:45:29.010371+0000","last_peered":"2024-07-26T12:45:29.010371+0000","last_clean":"2024-07-26T12:45:29.010371+0000","last_became_active":"2024-07-26T12:43:08.130424+0000","last_became_peered":"2024-07-26T12:43:08.130424+0000","last_unstale":"2024-07-26T12:45:29.010371+0000","last_undegraded":"2024-07-26T12:45:29.010371+0000","last_fullsized":"2024-07-26T12:45:29.010371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:49:35.829624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.539348+0000","last_change":"2024-07-26T12:43:26.770070+0000","last_active":"2024-07-26T12:45:46.539348+0000","last_peered":"2024-07-26T12:45:46.539348+0000","last_clean":"2024-07-26T12:45:46.539348+0000","last_became_active":"2024-07-26T12:43:26.769594+0000","last_became_peered":"2024-07-26T12:43:26.769594+0000","last_unstale":"2024-07-26T12:45:46.539348+0000","last_undegraded":"2024-07-26T12:45:46.539348+0000","last_fullsized":"2024-07-26T12:45:46.539348+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:18.342739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669284+0000","last_change":"2024-07-26T12:43:12.183134+0000","last_active":"2024-07-26T12:46:34.669284+0000","last_peered":"2024-07-26T12:46:34.669284+0000","last_clean":"2024-07-26T12:46:34.669284+0000","last_became_active":"2024-07-26T12:43:12.182406+0000","last_became_peered":"2024-07-26T12:43:12.182406+0000","last_unstale":"2024-07-26T12:46:34.669284+0000","last_undegraded":"2024-07-26T12:46:34.669284+0000","last_fullsized":"2024-07-26T12:46:34.669284+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:54:24.038971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:56.540854+0000","last_change":"2024-07-26T12:43:35.862005+0000","last_active":"2024-07-26T12:45:56.540854+0000","last_peered":"2024-07-26T12:45:56.540854+0000","last_clean":"2024-07-26T12:45:56.540854+0000","last_became_active":"2024-07-26T12:43:35.850911+0000","last_became_peered":"2024-07-26T12:43:35.850911+0000","last_unstale":"2024-07-26T12:45:56.540854+0000","last_undegraded":"2024-07-26T12:45:56.540854+0000","last_fullsized":"2024-07-26T12:45:56.540854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:18:16.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:54.014118+0000","last_change":"2024-07-26T12:43:26.774599+0000","last_active":"2024-07-26T12:45:54.014118+0000","last_peered":"2024-07-26T12:45:54.014118+0000","last_clean":"2024-07-26T12:45:54.014118+0000","last_became_active":"2024-07-26T12:43:26.774186+0000","last_became_peered":"2024-07-26T12:43:26.774186+0000","last_unstale":"2024-07-26T12:45:54.014118+0000","last_undegraded":"2024-07-26T12:45:54.014118+0000","last_fullsized":"2024-07-26T12:45:54.014118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:02:42.252093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.646564+0000","last_change":"2024-07-26T12:43:12.174676+0000","last_active":"2024-07-26T12:46:34.646564+0000","last_peered":"2024-07-26T12:46:34.646564+0000","last_clean":"2024-07-26T12:46:34.646564+0000","last_became_active":"2024-07-26T12:43:12.174310+0000","last_became_peered":"2024-07-26T12:43:12.174310+0000","last_unstale":"2024-07-26T12:46:34.646564+0000","last_undegraded":"2024-07-26T12:46:34.646564+0000","last_fullsized":"2024-07-26T12:46:34.646564+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:44.381289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'42","reported_seq":587,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.277571+0000","last_change":"2024-07-26T12:43:08.122675+0000","last_active":"2024-07-26T12:48:42.277571+0000","last_peered":"2024-07-26T12:48:42.277571+0000","last_clean":"2024-07-26T12:48:42.277571+0000","last_became_active":"2024-07-26T12:43:08.122174+0000","last_became_peered":"2024-07-26T12:43:08.122174+0000","last_unstale":"2024-07-26T12:48:42.277571+0000","last_undegraded":"2024-07-26T12:48:42.277571+0000","last_fullsized":"2024-07-26T12:48:42.277571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:54.513306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70065,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":498,"num_read_kb":318,"num_write":44,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.265915+0000","last_change":"2024-07-26T12:43:26.764283+0000","last_active":"2024-07-26T12:45:46.265915+0000","last_peered":"2024-07-26T12:45:46.265915+0000","last_clean":"2024-07-26T12:45:46.265915+0000","last_became_active":"2024-07-26T12:43:26.763821+0000","last_became_peered":"2024-07-26T12:43:26.763821+0000","last_unstale":"2024-07-26T12:45:46.265915+0000","last_undegraded":"2024-07-26T12:45:46.265915+0000","last_fullsized":"2024-07-26T12:45:46.265915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:56.753016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672277+0000","last_change":"2024-07-26T12:43:35.915277+0000","last_active":"2024-07-26T12:46:34.672277+0000","last_peered":"2024-07-26T12:46:34.672277+0000","last_clean":"2024-07-26T12:46:34.672277+0000","last_became_active":"2024-07-26T12:43:35.914848+0000","last_became_peered":"2024-07-26T12:43:35.914848+0000","last_unstale":"2024-07-26T12:46:34.672277+0000","last_undegraded":"2024-07-26T12:46:34.672277+0000","last_fullsized":"2024-07-26T12:46:34.672277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:56:35.639076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:34.011224+0000","last_change":"2024-07-26T12:43:08.227352+0000","last_active":"2024-07-26T12:45:34.011224+0000","last_peered":"2024-07-26T12:45:34.011224+0000","last_clean":"2024-07-26T12:45:34.011224+0000","last_became_active":"2024-07-26T12:43:08.226857+0000","last_became_peered":"2024-07-26T12:43:08.226857+0000","last_unstale":"2024-07-26T12:45:34.011224+0000","last_undegraded":"2024-07-26T12:45:34.011224+0000","last_fullsized":"2024-07-26T12:45:34.011224+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:20.156503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.765392+0000","last_change":"2024-07-26T12:43:26.762943+0000","last_active":"2024-07-26T12:45:46.765392+0000","last_peered":"2024-07-26T12:45:46.765392+0000","last_clean":"2024-07-26T12:45:46.765392+0000","last_became_active":"2024-07-26T12:43:26.762436+0000","last_became_peered":"2024-07-26T12:43:26.762436+0000","last_unstale":"2024-07-26T12:45:46.765392+0000","last_undegraded":"2024-07-26T12:45:46.765392+0000","last_fullsized":"2024-07-26T12:45:46.765392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:05:23.924397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.673242+0000","last_change":"2024-07-26T12:44:36.854705+0000","last_active":"2024-07-26T12:46:34.673242+0000","last_peered":"2024-07-26T12:46:34.673242+0000","last_clean":"2024-07-26T12:46:34.673242+0000","last_became_active":"2024-07-26T12:44:36.854268+0000","last_became_peered":"2024-07-26T12:44:36.854268+0000","last_unstale":"2024-07-26T12:46:34.673242+0000","last_undegraded":"2024-07-26T12:46:34.673242+0000","last_fullsized":"2024-07-26T12:46:34.673242+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:24:30.768905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:32.573818+0000","last_change":"2024-07-26T12:43:08.118597+0000","last_active":"2024-07-26T12:45:32.573818+0000","last_peered":"2024-07-26T12:45:32.573818+0000","last_clean":"2024-07-26T12:45:32.573818+0000","last_became_active":"2024-07-26T12:43:08.118145+0000","last_became_peered":"2024-07-26T12:43:08.118145+0000","last_unstale":"2024-07-26T12:45:32.573818+0000","last_undegraded":"2024-07-26T12:45:32.573818+0000","last_fullsized":"2024-07-26T12:45:32.573818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:52:45.511693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.265831+0000","last_change":"2024-07-26T12:43:26.771861+0000","last_active":"2024-07-26T12:45:46.265831+0000","last_peered":"2024-07-26T12:45:46.265831+0000","last_clean":"2024-07-26T12:45:46.265831+0000","last_became_active":"2024-07-26T12:43:26.771323+0000","last_became_peered":"2024-07-26T12:43:26.771323+0000","last_unstale":"2024-07-26T12:45:46.265831+0000","last_undegraded":"2024-07-26T12:45:46.265831+0000","last_fullsized":"2024-07-26T12:45:46.265831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:34.207514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656491+0000","last_change":"2024-07-26T12:43:12.169437+0000","last_active":"2024-07-26T12:46:34.656491+0000","last_peered":"2024-07-26T12:46:34.656491+0000","last_clean":"2024-07-26T12:46:34.656491+0000","last_became_active":"2024-07-26T12:43:12.168387+0000","last_became_peered":"2024-07-26T12:43:12.168387+0000","last_unstale":"2024-07-26T12:46:34.656491+0000","last_undegraded":"2024-07-26T12:46:34.656491+0000","last_fullsized":"2024-07-26T12:46:34.656491+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:57:06.148965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'6","reported_seq":539,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.277078+0000","last_change":"2024-07-26T12:43:08.129618+0000","last_active":"2024-07-26T12:48:42.277078+0000","last_peered":"2024-07-26T12:48:42.277078+0000","last_clean":"2024-07-26T12:48:42.277078+0000","last_became_active":"2024-07-26T12:43:08.129064+0000","last_became_peered":"2024-07-26T12:43:08.129064+0000","last_unstale":"2024-07-26T12:48:42.277078+0000","last_undegraded":"2024-07-26T12:48:42.277078+0000","last_fullsized":"2024-07-26T12:48:42.277078+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:26:22.406075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":486,"num_read_kb":486,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.261842+0000","last_change":"2024-07-26T12:43:35.854710+0000","last_active":"2024-07-26T12:45:46.261842+0000","last_peered":"2024-07-26T12:45:46.261842+0000","last_clean":"2024-07-26T12:45:46.261842+0000","last_became_active":"2024-07-26T12:43:35.852772+0000","last_became_peered":"2024-07-26T12:43:35.852772+0000","last_unstale":"2024-07-26T12:45:46.261842+0000","last_undegraded":"2024-07-26T12:45:46.261842+0000","last_fullsized":"2024-07-26T12:45:46.261842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:24:54.015879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:27.784210+0000","last_change":"2024-07-26T12:43:26.783182+0000","last_active":"2024-07-26T12:45:27.784210+0000","last_peered":"2024-07-26T12:45:27.784210+0000","last_clean":"2024-07-26T12:45:27.784210+0000","last_became_active":"2024-07-26T12:43:26.782697+0000","last_became_peered":"2024-07-26T12:43:26.782697+0000","last_unstale":"2024-07-26T12:45:27.784210+0000","last_undegraded":"2024-07-26T12:45:27.784210+0000","last_fullsized":"2024-07-26T12:45:27.784210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:57.169222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671751+0000","last_change":"2024-07-26T12:43:12.166163+0000","last_active":"2024-07-26T12:46:34.671751+0000","last_peered":"2024-07-26T12:46:34.671751+0000","last_clean":"2024-07-26T12:46:34.671751+0000","last_became_active":"2024-07-26T12:43:12.164427+0000","last_became_peered":"2024-07-26T12:43:12.164427+0000","last_unstale":"2024-07-26T12:46:34.671751+0000","last_undegraded":"2024-07-26T12:46:34.671751+0000","last_fullsized":"2024-07-26T12:46:34.671751+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:25.335508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:34.011158+0000","last_change":"2024-07-26T12:43:08.118224+0000","last_active":"2024-07-26T12:45:34.011158+0000","last_peered":"2024-07-26T12:45:34.011158+0000","last_clean":"2024-07-26T12:45:34.011158+0000","last_became_active":"2024-07-26T12:43:08.117821+0000","last_became_peered":"2024-07-26T12:43:08.117821+0000","last_unstale":"2024-07-26T12:45:34.011158+0000","last_undegraded":"2024-07-26T12:45:34.011158+0000","last_fullsized":"2024-07-26T12:45:34.011158+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:56:14.380931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:31.537591+0000","last_change":"2024-07-26T12:43:35.907740+0000","last_active":"2024-07-26T12:45:31.537591+0000","last_peered":"2024-07-26T12:45:31.537591+0000","last_clean":"2024-07-26T12:45:31.537591+0000","last_became_active":"2024-07-26T12:43:35.907115+0000","last_became_peered":"2024-07-26T12:43:35.907115+0000","last_unstale":"2024-07-26T12:45:31.537591+0000","last_undegraded":"2024-07-26T12:45:31.537591+0000","last_fullsized":"2024-07-26T12:45:31.537591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:53:24.783872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.670573+0000","last_change":"2024-07-26T12:43:12.177579+0000","last_active":"2024-07-26T12:46:34.670573+0000","last_peered":"2024-07-26T12:46:34.670573+0000","last_clean":"2024-07-26T12:46:34.670573+0000","last_became_active":"2024-07-26T12:43:12.171433+0000","last_became_peered":"2024-07-26T12:43:12.171433+0000","last_unstale":"2024-07-26T12:46:34.670573+0000","last_undegraded":"2024-07-26T12:46:34.670573+0000","last_fullsized":"2024-07-26T12:46:34.670573+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:50.972151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:33.076329+0000","last_change":"2024-07-26T12:43:08.133446+0000","last_active":"2024-07-26T12:45:33.076329+0000","last_peered":"2024-07-26T12:45:33.076329+0000","last_clean":"2024-07-26T12:45:33.076329+0000","last_became_active":"2024-07-26T12:43:08.133008+0000","last_became_peered":"2024-07-26T12:43:08.133008+0000","last_unstale":"2024-07-26T12:45:33.076329+0000","last_undegraded":"2024-07-26T12:45:33.076329+0000","last_fullsized":"2024-07-26T12:45:33.076329+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:05:21.283738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:37.266863+0000","last_change":"2024-07-26T12:43:08.124976+0000","last_active":"2024-07-26T12:45:37.266863+0000","last_peered":"2024-07-26T12:45:37.266863+0000","last_clean":"2024-07-26T12:45:37.266863+0000","last_became_active":"2024-07-26T12:43:08.124664+0000","last_became_peered":"2024-07-26T12:43:08.124664+0000","last_unstale":"2024-07-26T12:45:37.266863+0000","last_undegraded":"2024-07-26T12:45:37.266863+0000","last_fullsized":"2024-07-26T12:45:37.266863+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:51.798657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664310+0000","last_change":"2024-07-26T12:43:35.856266+0000","last_active":"2024-07-26T12:46:34.664310+0000","last_peered":"2024-07-26T12:46:34.664310+0000","last_clean":"2024-07-26T12:46:34.664310+0000","last_became_active":"2024-07-26T12:43:35.855802+0000","last_became_peered":"2024-07-26T12:43:35.855802+0000","last_unstale":"2024-07-26T12:46:34.664310+0000","last_undegraded":"2024-07-26T12:46:34.664310+0000","last_fullsized":"2024-07-26T12:46:34.664310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:16.432983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:59.251187+0000","last_change":"2024-07-26T12:43:26.778626+0000","last_active":"2024-07-26T12:45:59.251187+0000","last_peered":"2024-07-26T12:45:59.251187+0000","last_clean":"2024-07-26T12:45:59.251187+0000","last_became_active":"2024-07-26T12:43:26.778165+0000","last_became_peered":"2024-07-26T12:43:26.778165+0000","last_unstale":"2024-07-26T12:45:59.251187+0000","last_undegraded":"2024-07-26T12:45:59.251187+0000","last_fullsized":"2024-07-26T12:45:59.251187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:06:08.766495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":302,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.277799+0000","last_change":"2024-07-26T12:43:08.119702+0000","last_active":"2024-07-26T12:48:42.277799+0000","last_peered":"2024-07-26T12:48:42.277799+0000","last_clean":"2024-07-26T12:48:42.277799+0000","last_became_active":"2024-07-26T12:43:08.119225+0000","last_became_peered":"2024-07-26T12:43:08.119225+0000","last_unstale":"2024-07-26T12:48:42.277799+0000","last_undegraded":"2024-07-26T12:48:42.277799+0000","last_fullsized":"2024-07-26T12:48:42.277799+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:27.616873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":248,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.640998+0000","last_change":"2024-07-26T12:43:12.166239+0000","last_active":"2024-07-26T12:46:34.640998+0000","last_peered":"2024-07-26T12:46:34.640998+0000","last_clean":"2024-07-26T12:46:34.640998+0000","last_became_active":"2024-07-26T12:43:12.165770+0000","last_became_peered":"2024-07-26T12:43:12.165770+0000","last_unstale":"2024-07-26T12:46:34.640998+0000","last_undegraded":"2024-07-26T12:46:34.640998+0000","last_fullsized":"2024-07-26T12:46:34.640998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:11:15.290898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:59.251224+0000","last_change":"2024-07-26T12:43:26.765905+0000","last_active":"2024-07-26T12:45:59.251224+0000","last_peered":"2024-07-26T12:45:59.251224+0000","last_clean":"2024-07-26T12:45:59.251224+0000","last_became_active":"2024-07-26T12:43:26.765425+0000","last_became_peered":"2024-07-26T12:43:26.765425+0000","last_unstale":"2024-07-26T12:45:59.251224+0000","last_undegraded":"2024-07-26T12:45:59.251224+0000","last_fullsized":"2024-07-26T12:45:59.251224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:16.786604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.645058+0000","last_change":"2024-07-26T12:43:12.171400+0000","last_active":"2024-07-26T12:46:34.645058+0000","last_peered":"2024-07-26T12:46:34.645058+0000","last_clean":"2024-07-26T12:46:34.645058+0000","last_became_active":"2024-07-26T12:43:12.170998+0000","last_became_peered":"2024-07-26T12:43:12.170998+0000","last_unstale":"2024-07-26T12:46:34.645058+0000","last_undegraded":"2024-07-26T12:46:34.645058+0000","last_fullsized":"2024-07-26T12:46:34.645058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:43.251975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656729+0000","last_change":"2024-07-26T12:43:35.912729+0000","last_active":"2024-07-26T12:46:34.656729+0000","last_peered":"2024-07-26T12:46:34.656729+0000","last_clean":"2024-07-26T12:46:34.656729+0000","last_became_active":"2024-07-26T12:43:35.912318+0000","last_became_peered":"2024-07-26T12:43:35.912318+0000","last_unstale":"2024-07-26T12:46:34.656729+0000","last_undegraded":"2024-07-26T12:46:34.656729+0000","last_fullsized":"2024-07-26T12:46:34.656729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:14.473536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661325+0000","last_change":"2024-07-26T12:43:12.182933+0000","last_active":"2024-07-26T12:46:34.661325+0000","last_peered":"2024-07-26T12:46:34.661325+0000","last_clean":"2024-07-26T12:46:34.661325+0000","last_became_active":"2024-07-26T12:43:12.182656+0000","last_became_peered":"2024-07-26T12:43:12.182656+0000","last_unstale":"2024-07-26T12:46:34.661325+0000","last_undegraded":"2024-07-26T12:46:34.661325+0000","last_fullsized":"2024-07-26T12:46:34.661325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:05:59.717662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.675425+0000","last_change":"2024-07-26T12:43:12.181196+0000","last_active":"2024-07-26T12:46:34.675425+0000","last_peered":"2024-07-26T12:46:34.675425+0000","last_clean":"2024-07-26T12:46:34.675425+0000","last_became_active":"2024-07-26T12:43:12.180765+0000","last_became_peered":"2024-07-26T12:43:12.180765+0000","last_unstale":"2024-07-26T12:46:34.675425+0000","last_undegraded":"2024-07-26T12:46:34.675425+0000","last_fullsized":"2024-07-26T12:46:34.675425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:20.590023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671094+0000","last_change":"2024-07-26T12:43:12.185408+0000","last_active":"2024-07-26T12:46:34.671094+0000","last_peered":"2024-07-26T12:46:34.671094+0000","last_clean":"2024-07-26T12:46:34.671094+0000","last_became_active":"2024-07-26T12:43:12.185038+0000","last_became_peered":"2024-07-26T12:43:12.185038+0000","last_unstale":"2024-07-26T12:46:34.671094+0000","last_undegraded":"2024-07-26T12:46:34.671094+0000","last_fullsized":"2024-07-26T12:46:34.671094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:55:30.604995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644437+0000","last_change":"2024-07-26T12:43:12.169301+0000","last_active":"2024-07-26T12:46:34.644437+0000","last_peered":"2024-07-26T12:46:34.644437+0000","last_clean":"2024-07-26T12:46:34.644437+0000","last_became_active":"2024-07-26T12:43:12.168923+0000","last_became_peered":"2024-07-26T12:43:12.168923+0000","last_unstale":"2024-07-26T12:46:34.644437+0000","last_undegraded":"2024-07-26T12:46:34.644437+0000","last_fullsized":"2024-07-26T12:46:34.644437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:04.506485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.647538+0000","last_change":"2024-07-26T12:43:12.166663+0000","last_active":"2024-07-26T12:46:34.647538+0000","last_peered":"2024-07-26T12:46:34.647538+0000","last_clean":"2024-07-26T12:46:34.647538+0000","last_became_active":"2024-07-26T12:43:12.166200+0000","last_became_peered":"2024-07-26T12:43:12.166200+0000","last_unstale":"2024-07-26T12:46:34.647538+0000","last_undegraded":"2024-07-26T12:46:34.647538+0000","last_fullsized":"2024-07-26T12:46:34.647538+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:14.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664657+0000","last_change":"2024-07-26T12:43:12.171273+0000","last_active":"2024-07-26T12:46:34.664657+0000","last_peered":"2024-07-26T12:46:34.664657+0000","last_clean":"2024-07-26T12:46:34.664657+0000","last_became_active":"2024-07-26T12:43:12.167680+0000","last_became_peered":"2024-07-26T12:43:12.167680+0000","last_unstale":"2024-07-26T12:46:34.664657+0000","last_undegraded":"2024-07-26T12:46:34.664657+0000","last_fullsized":"2024-07-26T12:46:34.664657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:44.797644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671048+0000","last_change":"2024-07-26T12:43:12.179891+0000","last_active":"2024-07-26T12:46:34.671048+0000","last_peered":"2024-07-26T12:46:34.671048+0000","last_clean":"2024-07-26T12:46:34.671048+0000","last_became_active":"2024-07-26T12:43:12.179518+0000","last_became_peered":"2024-07-26T12:43:12.179518+0000","last_unstale":"2024-07-26T12:46:34.671048+0000","last_undegraded":"2024-07-26T12:46:34.671048+0000","last_fullsized":"2024-07-26T12:46:34.671048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:24.246023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671055+0000","last_change":"2024-07-26T12:43:12.168521+0000","last_active":"2024-07-26T12:46:34.671055+0000","last_peered":"2024-07-26T12:46:34.671055+0000","last_clean":"2024-07-26T12:46:34.671055+0000","last_became_active":"2024-07-26T12:43:12.168126+0000","last_became_peered":"2024-07-26T12:43:12.168126+0000","last_unstale":"2024-07-26T12:46:34.671055+0000","last_undegraded":"2024-07-26T12:46:34.671055+0000","last_fullsized":"2024-07-26T12:46:34.671055+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:00.135090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661599+0000","last_change":"2024-07-26T12:44:36.843322+0000","last_active":"2024-07-26T12:46:34.661599+0000","last_peered":"2024-07-26T12:46:34.661599+0000","last_clean":"2024-07-26T12:46:34.661599+0000","last_became_active":"2024-07-26T12:44:36.842821+0000","last_became_peered":"2024-07-26T12:44:36.842821+0000","last_unstale":"2024-07-26T12:46:34.661599+0000","last_undegraded":"2024-07-26T12:46:34.661599+0000","last_fullsized":"2024-07-26T12:46:34.661599+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:05.609298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.675045+0000","last_change":"2024-07-26T12:43:12.185565+0000","last_active":"2024-07-26T12:46:34.675045+0000","last_peered":"2024-07-26T12:46:34.675045+0000","last_clean":"2024-07-26T12:46:34.675045+0000","last_became_active":"2024-07-26T12:43:12.185057+0000","last_became_peered":"2024-07-26T12:43:12.185057+0000","last_unstale":"2024-07-26T12:46:34.675045+0000","last_undegraded":"2024-07-26T12:46:34.675045+0000","last_fullsized":"2024-07-26T12:46:34.675045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:58.108311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644690+0000","last_change":"2024-07-26T12:43:12.173301+0000","last_active":"2024-07-26T12:46:34.644690+0000","last_peered":"2024-07-26T12:46:34.644690+0000","last_clean":"2024-07-26T12:46:34.644690+0000","last_became_active":"2024-07-26T12:43:12.172955+0000","last_became_peered":"2024-07-26T12:43:12.172955+0000","last_unstale":"2024-07-26T12:46:34.644690+0000","last_undegraded":"2024-07-26T12:46:34.644690+0000","last_fullsized":"2024-07-26T12:46:34.644690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:41.731141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661490+0000","last_change":"2024-07-26T12:43:12.176169+0000","last_active":"2024-07-26T12:46:34.661490+0000","last_peered":"2024-07-26T12:46:34.661490+0000","last_clean":"2024-07-26T12:46:34.661490+0000","last_became_active":"2024-07-26T12:43:12.175745+0000","last_became_peered":"2024-07-26T12:43:12.175745+0000","last_unstale":"2024-07-26T12:46:34.661490+0000","last_undegraded":"2024-07-26T12:46:34.661490+0000","last_fullsized":"2024-07-26T12:46:34.661490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:20.393207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.654907+0000","last_change":"2024-07-26T12:43:12.168182+0000","last_active":"2024-07-26T12:46:34.654907+0000","last_peered":"2024-07-26T12:46:34.654907+0000","last_clean":"2024-07-26T12:46:34.654907+0000","last_became_active":"2024-07-26T12:43:12.163771+0000","last_became_peered":"2024-07-26T12:43:12.163771+0000","last_unstale":"2024-07-26T12:46:34.654907+0000","last_undegraded":"2024-07-26T12:46:34.654907+0000","last_fullsized":"2024-07-26T12:46:34.654907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:40:30.960153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.660143+0000","last_change":"2024-07-26T12:43:12.189524+0000","last_active":"2024-07-26T12:46:34.660143+0000","last_peered":"2024-07-26T12:46:34.660143+0000","last_clean":"2024-07-26T12:46:34.660143+0000","last_became_active":"2024-07-26T12:43:12.189165+0000","last_became_peered":"2024-07-26T12:43:12.189165+0000","last_unstale":"2024-07-26T12:46:34.660143+0000","last_undegraded":"2024-07-26T12:46:34.660143+0000","last_fullsized":"2024-07-26T12:46:34.660143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:11:08.479058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.665412+0000","last_change":"2024-07-26T12:43:12.180603+0000","last_active":"2024-07-26T12:46:34.665412+0000","last_peered":"2024-07-26T12:46:34.665412+0000","last_clean":"2024-07-26T12:46:34.665412+0000","last_became_active":"2024-07-26T12:43:12.180204+0000","last_became_peered":"2024-07-26T12:43:12.180204+0000","last_unstale":"2024-07-26T12:46:34.665412+0000","last_undegraded":"2024-07-26T12:46:34.665412+0000","last_fullsized":"2024-07-26T12:46:34.665412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:46.768213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668524+0000","last_change":"2024-07-26T12:43:12.185347+0000","last_active":"2024-07-26T12:46:34.668524+0000","last_peered":"2024-07-26T12:46:34.668524+0000","last_clean":"2024-07-26T12:46:34.668524+0000","last_became_active":"2024-07-26T12:43:12.180190+0000","last_became_peered":"2024-07-26T12:43:12.180190+0000","last_unstale":"2024-07-26T12:46:34.668524+0000","last_undegraded":"2024-07-26T12:46:34.668524+0000","last_fullsized":"2024-07-26T12:46:34.668524+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:07.654967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.667916+0000","last_change":"2024-07-26T12:43:12.177258+0000","last_active":"2024-07-26T12:46:34.667916+0000","last_peered":"2024-07-26T12:46:34.667916+0000","last_clean":"2024-07-26T12:46:34.667916+0000","last_became_active":"2024-07-26T12:43:12.176879+0000","last_became_peered":"2024-07-26T12:43:12.176879+0000","last_unstale":"2024-07-26T12:46:34.667916+0000","last_undegraded":"2024-07-26T12:46:34.667916+0000","last_fullsized":"2024-07-26T12:46:34.667916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:24.244759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.675740+0000","last_change":"2024-07-26T12:44:36.857403+0000","last_active":"2024-07-26T12:46:34.675740+0000","last_peered":"2024-07-26T12:46:34.675740+0000","last_clean":"2024-07-26T12:46:34.675740+0000","last_became_active":"2024-07-26T12:44:36.856980+0000","last_became_peered":"2024-07-26T12:44:36.856980+0000","last_unstale":"2024-07-26T12:46:34.675740+0000","last_undegraded":"2024-07-26T12:46:34.675740+0000","last_fullsized":"2024-07-26T12:46:34.675740+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.166383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.653669+0000","last_change":"2024-07-26T12:43:12.166827+0000","last_active":"2024-07-26T12:46:34.653669+0000","last_peered":"2024-07-26T12:46:34.653669+0000","last_clean":"2024-07-26T12:46:34.653669+0000","last_became_active":"2024-07-26T12:43:12.165222+0000","last_became_peered":"2024-07-26T12:43:12.165222+0000","last_unstale":"2024-07-26T12:46:34.653669+0000","last_undegraded":"2024-07-26T12:46:34.653669+0000","last_fullsized":"2024-07-26T12:46:34.653669+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:37:42.219844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.663059+0000","last_change":"2024-07-26T12:43:12.176961+0000","last_active":"2024-07-26T12:46:34.663059+0000","last_peered":"2024-07-26T12:46:34.663059+0000","last_clean":"2024-07-26T12:46:34.663059+0000","last_became_active":"2024-07-26T12:43:12.176550+0000","last_became_peered":"2024-07-26T12:43:12.176550+0000","last_unstale":"2024-07-26T12:46:34.663059+0000","last_undegraded":"2024-07-26T12:46:34.663059+0000","last_fullsized":"2024-07-26T12:46:34.663059+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:36:19.720801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650547+0000","last_change":"2024-07-26T12:43:12.173560+0000","last_active":"2024-07-26T12:46:34.650547+0000","last_peered":"2024-07-26T12:46:34.650547+0000","last_clean":"2024-07-26T12:46:34.650547+0000","last_became_active":"2024-07-26T12:43:12.173187+0000","last_became_peered":"2024-07-26T12:43:12.173187+0000","last_unstale":"2024-07-26T12:46:34.650547+0000","last_undegraded":"2024-07-26T12:46:34.650547+0000","last_fullsized":"2024-07-26T12:46:34.650547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:53.929210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655516+0000","last_change":"2024-07-26T12:43:12.169829+0000","last_active":"2024-07-26T12:46:34.655516+0000","last_peered":"2024-07-26T12:46:34.655516+0000","last_clean":"2024-07-26T12:46:34.655516+0000","last_became_active":"2024-07-26T12:43:12.169444+0000","last_became_peered":"2024-07-26T12:43:12.169444+0000","last_unstale":"2024-07-26T12:46:34.655516+0000","last_undegraded":"2024-07-26T12:46:34.655516+0000","last_fullsized":"2024-07-26T12:46:34.655516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:29.111483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671995+0000","last_change":"2024-07-26T12:43:12.171857+0000","last_active":"2024-07-26T12:46:34.671995+0000","last_peered":"2024-07-26T12:46:34.671995+0000","last_clean":"2024-07-26T12:46:34.671995+0000","last_became_active":"2024-07-26T12:43:12.170752+0000","last_became_peered":"2024-07-26T12:43:12.170752+0000","last_unstale":"2024-07-26T12:46:34.671995+0000","last_undegraded":"2024-07-26T12:46:34.671995+0000","last_fullsized":"2024-07-26T12:46:34.671995+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:22:09.442916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.657139+0000","last_change":"2024-07-26T12:43:12.178646+0000","last_active":"2024-07-26T12:46:34.657139+0000","last_peered":"2024-07-26T12:46:34.657139+0000","last_clean":"2024-07-26T12:46:34.657139+0000","last_became_active":"2024-07-26T12:43:12.178259+0000","last_became_peered":"2024-07-26T12:43:12.178259+0000","last_unstale":"2024-07-26T12:46:34.657139+0000","last_undegraded":"2024-07-26T12:46:34.657139+0000","last_fullsized":"2024-07-26T12:46:34.657139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:30:59.417843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655755+0000","last_change":"2024-07-26T12:43:12.164579+0000","last_active":"2024-07-26T12:46:34.655755+0000","last_peered":"2024-07-26T12:46:34.655755+0000","last_clean":"2024-07-26T12:46:34.655755+0000","last_became_active":"2024-07-26T12:43:12.164099+0000","last_became_peered":"2024-07-26T12:43:12.164099+0000","last_unstale":"2024-07-26T12:46:34.655755+0000","last_undegraded":"2024-07-26T12:46:34.655755+0000","last_fullsized":"2024-07-26T12:46:34.655755+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:51:45.293190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664174+0000","last_change":"2024-07-26T12:43:12.180838+0000","last_active":"2024-07-26T12:46:34.664174+0000","last_peered":"2024-07-26T12:46:34.664174+0000","last_clean":"2024-07-26T12:46:34.664174+0000","last_became_active":"2024-07-26T12:43:12.180543+0000","last_became_peered":"2024-07-26T12:43:12.180543+0000","last_unstale":"2024-07-26T12:46:34.664174+0000","last_undegraded":"2024-07-26T12:46:34.664174+0000","last_fullsized":"2024-07-26T12:46:34.664174+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:27.185082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650139+0000","last_change":"2024-07-26T12:43:12.167235+0000","last_active":"2024-07-26T12:46:34.650139+0000","last_peered":"2024-07-26T12:46:34.650139+0000","last_clean":"2024-07-26T12:46:34.650139+0000","last_became_active":"2024-07-26T12:43:12.166489+0000","last_became_peered":"2024-07-26T12:43:12.166489+0000","last_unstale":"2024-07-26T12:46:34.650139+0000","last_undegraded":"2024-07-26T12:46:34.650139+0000","last_fullsized":"2024-07-26T12:46:34.650139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:13.868870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668862+0000","last_change":"2024-07-26T12:43:12.176717+0000","last_active":"2024-07-26T12:46:34.668862+0000","last_peered":"2024-07-26T12:46:34.668862+0000","last_clean":"2024-07-26T12:46:34.668862+0000","last_became_active":"2024-07-26T12:43:12.172699+0000","last_became_peered":"2024-07-26T12:43:12.172699+0000","last_unstale":"2024-07-26T12:46:34.668862+0000","last_undegraded":"2024-07-26T12:46:34.668862+0000","last_fullsized":"2024-07-26T12:46:34.668862+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:23:36.853999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669127+0000","last_change":"2024-07-26T12:43:12.177093+0000","last_active":"2024-07-26T12:46:34.669127+0000","last_peered":"2024-07-26T12:46:34.669127+0000","last_clean":"2024-07-26T12:46:34.669127+0000","last_became_active":"2024-07-26T12:43:12.176693+0000","last_became_peered":"2024-07-26T12:43:12.176693+0000","last_unstale":"2024-07-26T12:46:34.669127+0000","last_undegraded":"2024-07-26T12:46:34.669127+0000","last_fullsized":"2024-07-26T12:46:34.669127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:56.875143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.660186+0000","last_change":"2024-07-26T12:43:12.189531+0000","last_active":"2024-07-26T12:46:34.660186+0000","last_peered":"2024-07-26T12:46:34.660186+0000","last_clean":"2024-07-26T12:46:34.660186+0000","last_became_active":"2024-07-26T12:43:12.189162+0000","last_became_peered":"2024-07-26T12:43:12.189162+0000","last_unstale":"2024-07-26T12:46:34.660186+0000","last_undegraded":"2024-07-26T12:46:34.660186+0000","last_fullsized":"2024-07-26T12:46:34.660186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:11:00.160614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77551,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2475,"num_read_kb":1322,"num_write":265,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":227280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678098,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":536466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874328,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1744896,"data_stored":1187374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037851,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":539892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234082,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":536174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430311,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36964,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708188,"statfs":{"total":95995035648,"available":95957184512,"internally_reserved":0,"allocated":1667072,"data_stored":1132812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593837,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":531300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790068,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1128394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953592,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":535374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":531008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378755,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":603611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542281,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1118208,"data_stored":597022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":533288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":73033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":71801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T12:48:46.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T12:48:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.593+0000 7ffbb2200640 1 -- 172.21.15.107:0/3385810710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffbac10c980 msgr2=0x7ffbac1163f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.593+0000 7ffbb2200640 1 --2- 172.21.15.107:0/3385810710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffbac10c980 0x7ffbac1163f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffba4009ca0 tx=0x7ffba402a8c0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.593+0000 7ffbb2200640 1 -- 172.21.15.107:0/3385810710 shutdown_connections 2024-07-26T12:48:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.593+0000 7ffbb2200640 1 --2- 172.21.15.107:0/3385810710 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffbac10c980 0x7ffbac1163f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.593+0000 7ffbb2200640 1 --2- 172.21.15.107:0/3385810710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbac10c030 0x7ffbac10c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.593+0000 7ffbb2200640 1 --2- 172.21.15.107:0/3385810710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac109990 0x7ffbac109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.593+0000 7ffbb2200640 1 -- 172.21.15.107:0/3385810710 >> 172.21.15.107:0/3385810710 conn(0x7ffbac0790b0 msgr2=0x7ffbac107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.593+0000 7ffbb2200640 1 -- 172.21.15.107:0/3385810710 shutdown_connections 2024-07-26T12:48:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.593+0000 7ffbb2200640 1 -- 172.21.15.107:0/3385810710 wait complete. 2024-07-26T12:48:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.594+0000 7ffbb2200640 1 Processor -- start 2024-07-26T12:48:46.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbb2200640 1 -- start start 2024-07-26T12:48:46.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbb2200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac109990 0x7ffbac1a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbb2200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbac10c030 0x7ffbac1a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbb2200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffbac10c980 0x7ffbac1a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbb2200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac119880 con 0x7ffbac109990 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbb2200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac119700 con 0x7ffbac10c980 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbb2200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac119a00 con 0x7ffbac10c030 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbb0e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac109990 0x7ffbac1a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbb0e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac109990 0x7ffbac1a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33430/0 (socket says 172.21.15.107:33430) 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbb0e00640 1 -- 172.21.15.107:0/3003305105 learned_addr learned my addr 172.21.15.107:0/3003305105 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.595+0000 7ffbabe00640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbac10c030 0x7ffbac1a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffbb1800640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffbac10c980 0x7ffbac1a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffbb0e00640 1 -- 172.21.15.107:0/3003305105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbac10c030 msgr2=0x7ffbac1a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:46.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffbb0e00640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbac10c030 0x7ffbac1a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffbb0e00640 1 -- 172.21.15.107:0/3003305105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffbac10c980 msgr2=0x7ffbac1a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:46.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffbb0e00640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffbac10c980 0x7ffbac1a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffbb0e00640 1 -- 172.21.15.107:0/3003305105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac1a60f0 con 0x7ffbac109990 2024-07-26T12:48:46.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffbabe00640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbac10c030 0x7ffbac1a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:48:46.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffbb0e00640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac109990 0x7ffbac1a01c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffba000e9f0 tx=0x7ffba000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffba9600640 1 -- 172.21.15.107:0/3003305105 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba000cd80 con 0x7ffbac109990 2024-07-26T12:48:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffba9600640 1 -- 172.21.15.107:0/3003305105 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffba00108e0 con 0x7ffbac109990 2024-07-26T12:48:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffbb2200640 1 -- 172.21.15.107:0/3003305105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbac1a63e0 con 0x7ffbac109990 2024-07-26T12:48:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.596+0000 7ffba9600640 1 -- 172.21.15.107:0/3003305105 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba0018980 con 0x7ffbac109990 2024-07-26T12:48:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.597+0000 7ffbb2200640 1 -- 172.21.15.107:0/3003305105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbac1acb70 con 0x7ffbac109990 2024-07-26T12:48:46.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.598+0000 7ffbb2200640 1 -- 172.21.15.107:0/3003305105 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbac109da0 con 0x7ffbac109990 2024-07-26T12:48:46.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.600+0000 7ffba9600640 1 -- 172.21.15.107:0/3003305105 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7ffba0010430 con 0x7ffbac109990 2024-07-26T12:48:46.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.602+0000 7ffba9600640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ffb940b3c20 0x7ffb940b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:46.602 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.602+0000 7ffbabe00640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ffb940b3c20 0x7ffb940b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:46.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.602+0000 7ffba9600640 1 -- 172.21.15.107:0/3003305105 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ffba00e49e0 con 0x7ffbac109990 2024-07-26T12:48:46.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.603+0000 7ffbabe00640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ffb940b3c20 0x7ffb940b6130 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffbac108f70 tx=0x7ffb98005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:46.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.603+0000 7ffba9600640 1 -- 172.21.15.107:0/3003305105 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ffba00abb30 con 0x7ffbac109990 2024-07-26T12:48:46.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.814+0000 7ffbb2200640 1 -- 172.21.15.107:0/3003305105 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffbac063230 con 0x7ffb940b3c20 2024-07-26T12:48:46.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.830+0000 7ffba9600640 1 -- 172.21.15.107:0/3003305105 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340543 (secure 0 0 0) 0x7ffbac063230 con 0x7ffb940b3c20 2024-07-26T12:48:46.833 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:48:46.836 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-07-26T12:48:46.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 -- 172.21.15.107:0/3003305105 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ffb940b3c20 msgr2=0x7ffb940b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:46.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ffb940b3c20 0x7ffb940b6130 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffbac108f70 tx=0x7ffb98005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:46.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 -- 172.21.15.107:0/3003305105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac109990 msgr2=0x7ffbac1a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:46.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac109990 0x7ffbac1a01c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ffba000e9f0 tx=0x7ffba000eeb0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 -- 172.21.15.107:0/3003305105 shutdown_connections 2024-07-26T12:48:46.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ffb940b3c20 0x7ffb940b6130 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ffbac10c980 0x7ffbac1a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbac10c030 0x7ffbac1a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 --2- 172.21.15.107:0/3003305105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ffbac109990 0x7ffbac1a01c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:46.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 -- 172.21.15.107:0/3003305105 >> 172.21.15.107:0/3003305105 conn(0x7ffbac0790b0 msgr2=0x7ffbac10d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:46.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 -- 172.21.15.107:0/3003305105 shutdown_connections 2024-07-26T12:48:46.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:46.837+0000 7ffbb2200640 1 -- 172.21.15.107:0/3003305105 wait complete. 2024-07-26T12:48:46.868 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-07-26T12:48:44.817705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668153,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2556,"num_read_kb":1391,"num_write":390,"num_write_kb":2312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2951874,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":822,"ondisk_log_size":822,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376068,"kb_used_data":14436,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565756,"statfs":{"total":1151940427776,"available":1151555334144,"internally_reserved":0,"allocated":14782464,"data_stored":8392715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":116,"num_read_kb":62,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006243"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":302,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:39.370594+0000","last_change":"2024-07-26T12:43:08.129417+0000","last_active":"2024-07-26T12:48:39.370594+0000","last_peered":"2024-07-26T12:48:39.370594+0000","last_clean":"2024-07-26T12:48:39.370594+0000","last_became_active":"2024-07-26T12:43:08.128919+0000","last_became_peered":"2024-07-26T12:43:08.128919+0000","last_unstale":"2024-07-26T12:48:39.370594+0000","last_undegraded":"2024-07-26T12:48:39.370594+0000","last_fullsized":"2024-07-26T12:48:39.370594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:35:55.414185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.658083+0000","last_change":"2024-07-26T12:43:12.172058+0000","last_active":"2024-07-26T12:46:34.658083+0000","last_peered":"2024-07-26T12:46:34.658083+0000","last_clean":"2024-07-26T12:46:34.658083+0000","last_became_active":"2024-07-26T12:43:12.171631+0000","last_became_peered":"2024-07-26T12:43:12.171631+0000","last_unstale":"2024-07-26T12:46:34.658083+0000","last_undegraded":"2024-07-26T12:46:34.658083+0000","last_fullsized":"2024-07-26T12:46:34.658083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:51.972962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:54.250635+0000","last_change":"2024-07-26T12:43:26.773739+0000","last_active":"2024-07-26T12:45:54.250635+0000","last_peered":"2024-07-26T12:45:54.250635+0000","last_clean":"2024-07-26T12:45:54.250635+0000","last_became_active":"2024-07-26T12:43:26.773254+0000","last_became_peered":"2024-07-26T12:43:26.773254+0000","last_unstale":"2024-07-26T12:45:54.250635+0000","last_undegraded":"2024-07-26T12:45:54.250635+0000","last_fullsized":"2024-07-26T12:45:54.250635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:03.125497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:11.254691+0000","last_change":"2024-07-26T12:43:08.227329+0000","last_active":"2024-07-26T12:45:11.254691+0000","last_peered":"2024-07-26T12:45:11.254691+0000","last_clean":"2024-07-26T12:45:11.254691+0000","last_became_active":"2024-07-26T12:43:08.226674+0000","last_became_peered":"2024-07-26T12:43:08.226674+0000","last_unstale":"2024-07-26T12:45:11.254691+0000","last_undegraded":"2024-07-26T12:45:11.254691+0000","last_fullsized":"2024-07-26T12:45:11.254691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.222684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674670+0000","last_change":"2024-07-26T12:43:12.190616+0000","last_active":"2024-07-26T12:46:34.674670+0000","last_peered":"2024-07-26T12:46:34.674670+0000","last_clean":"2024-07-26T12:46:34.674670+0000","last_became_active":"2024-07-26T12:43:12.189717+0000","last_became_peered":"2024-07-26T12:43:12.189717+0000","last_unstale":"2024-07-26T12:46:34.674670+0000","last_undegraded":"2024-07-26T12:46:34.674670+0000","last_fullsized":"2024-07-26T12:46:34.674670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:03.509592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:51.766027+0000","last_change":"2024-07-26T12:43:26.774486+0000","last_active":"2024-07-26T12:45:51.766027+0000","last_peered":"2024-07-26T12:45:51.766027+0000","last_clean":"2024-07-26T12:45:51.766027+0000","last_became_active":"2024-07-26T12:43:26.774046+0000","last_became_peered":"2024-07-26T12:43:26.774046+0000","last_unstale":"2024-07-26T12:45:51.766027+0000","last_undegraded":"2024-07-26T12:45:51.766027+0000","last_fullsized":"2024-07-26T12:45:51.766027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.847072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:37.304088+0000","last_change":"2024-07-26T12:43:08.123084+0000","last_active":"2024-07-26T12:45:37.304088+0000","last_peered":"2024-07-26T12:45:37.304088+0000","last_clean":"2024-07-26T12:45:37.304088+0000","last_became_active":"2024-07-26T12:43:08.122371+0000","last_became_peered":"2024-07-26T12:43:08.122371+0000","last_unstale":"2024-07-26T12:45:37.304088+0000","last_undegraded":"2024-07-26T12:45:37.304088+0000","last_fullsized":"2024-07-26T12:45:37.304088+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:27:09.665660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672192+0000","last_change":"2024-07-26T12:43:12.170290+0000","last_active":"2024-07-26T12:46:34.672192+0000","last_peered":"2024-07-26T12:46:34.672192+0000","last_clean":"2024-07-26T12:46:34.672192+0000","last_became_active":"2024-07-26T12:43:12.169849+0000","last_became_peered":"2024-07-26T12:43:12.169849+0000","last_unstale":"2024-07-26T12:46:34.672192+0000","last_undegraded":"2024-07-26T12:46:34.672192+0000","last_fullsized":"2024-07-26T12:46:34.672192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:46.044921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:56.766737+0000","last_change":"2024-07-26T12:43:26.764804+0000","last_active":"2024-07-26T12:45:56.766737+0000","last_peered":"2024-07-26T12:45:56.766737+0000","last_clean":"2024-07-26T12:45:56.766737+0000","last_became_active":"2024-07-26T12:43:26.764065+0000","last_became_peered":"2024-07-26T12:43:26.764065+0000","last_unstale":"2024-07-26T12:45:56.766737+0000","last_undegraded":"2024-07-26T12:45:56.766737+0000","last_fullsized":"2024-07-26T12:45:56.766737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:16:59.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:37.304036+0000","last_change":"2024-07-26T12:43:08.121496+0000","last_active":"2024-07-26T12:45:37.304036+0000","last_peered":"2024-07-26T12:45:37.304036+0000","last_clean":"2024-07-26T12:45:37.304036+0000","last_became_active":"2024-07-26T12:43:08.120962+0000","last_became_peered":"2024-07-26T12:43:08.120962+0000","last_unstale":"2024-07-26T12:45:37.304036+0000","last_undegraded":"2024-07-26T12:45:37.304036+0000","last_fullsized":"2024-07-26T12:45:37.304036+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:42:02.998023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655232+0000","last_change":"2024-07-26T12:43:12.168707+0000","last_active":"2024-07-26T12:46:34.655232+0000","last_peered":"2024-07-26T12:46:34.655232+0000","last_clean":"2024-07-26T12:46:34.655232+0000","last_became_active":"2024-07-26T12:43:12.168338+0000","last_became_peered":"2024-07-26T12:43:12.168338+0000","last_unstale":"2024-07-26T12:46:34.655232+0000","last_undegraded":"2024-07-26T12:46:34.655232+0000","last_fullsized":"2024-07-26T12:46:34.655232+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:39:44.724864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:56.766714+0000","last_change":"2024-07-26T12:43:26.770863+0000","last_active":"2024-07-26T12:45:56.766714+0000","last_peered":"2024-07-26T12:45:56.766714+0000","last_clean":"2024-07-26T12:45:56.766714+0000","last_became_active":"2024-07-26T12:43:26.770480+0000","last_became_peered":"2024-07-26T12:43:26.770480+0000","last_unstale":"2024-07-26T12:45:56.766714+0000","last_undegraded":"2024-07-26T12:45:56.766714+0000","last_fullsized":"2024-07-26T12:45:56.766714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:09.128364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672691+0000","last_change":"2024-07-26T12:43:12.181659+0000","last_active":"2024-07-26T12:46:34.672691+0000","last_peered":"2024-07-26T12:46:34.672691+0000","last_clean":"2024-07-26T12:46:34.672691+0000","last_became_active":"2024-07-26T12:43:12.181268+0000","last_became_peered":"2024-07-26T12:43:12.181268+0000","last_unstale":"2024-07-26T12:46:34.672691+0000","last_undegraded":"2024-07-26T12:46:34.672691+0000","last_fullsized":"2024-07-26T12:46:34.672691+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:38:28.445981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:51.540084+0000","last_change":"2024-07-26T12:43:35.856273+0000","last_active":"2024-07-26T12:45:51.540084+0000","last_peered":"2024-07-26T12:45:51.540084+0000","last_clean":"2024-07-26T12:45:51.540084+0000","last_became_active":"2024-07-26T12:43:35.849625+0000","last_became_peered":"2024-07-26T12:43:35.849625+0000","last_unstale":"2024-07-26T12:45:51.540084+0000","last_undegraded":"2024-07-26T12:45:51.540084+0000","last_fullsized":"2024-07-26T12:45:51.540084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:34:45.261474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:41.764778+0000","last_change":"2024-07-26T12:43:26.793431+0000","last_active":"2024-07-26T12:45:41.764778+0000","last_peered":"2024-07-26T12:45:41.764778+0000","last_clean":"2024-07-26T12:45:41.764778+0000","last_became_active":"2024-07-26T12:43:26.792931+0000","last_became_peered":"2024-07-26T12:43:26.792931+0000","last_unstale":"2024-07-26T12:45:41.764778+0000","last_undegraded":"2024-07-26T12:45:41.764778+0000","last_fullsized":"2024-07-26T12:45:41.764778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:31.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669969+0000","last_change":"2024-07-26T12:43:12.183784+0000","last_active":"2024-07-26T12:46:34.669969+0000","last_peered":"2024-07-26T12:46:34.669969+0000","last_clean":"2024-07-26T12:46:34.669969+0000","last_became_active":"2024-07-26T12:43:12.182696+0000","last_became_peered":"2024-07-26T12:43:12.182696+0000","last_unstale":"2024-07-26T12:46:34.669969+0000","last_undegraded":"2024-07-26T12:46:34.669969+0000","last_fullsized":"2024-07-26T12:46:34.669969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:33:19.918180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:19.245986+0000","last_change":"2024-07-26T12:43:08.131069+0000","last_active":"2024-07-26T12:45:19.245986+0000","last_peered":"2024-07-26T12:45:19.245986+0000","last_clean":"2024-07-26T12:45:19.245986+0000","last_became_active":"2024-07-26T12:43:08.130637+0000","last_became_peered":"2024-07-26T12:43:08.130637+0000","last_unstale":"2024-07-26T12:45:19.245986+0000","last_undegraded":"2024-07-26T12:45:19.245986+0000","last_fullsized":"2024-07-26T12:45:19.245986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:04:16.461900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:41.764748+0000","last_change":"2024-07-26T12:43:26.763388+0000","last_active":"2024-07-26T12:45:41.764748+0000","last_peered":"2024-07-26T12:45:41.764748+0000","last_clean":"2024-07-26T12:45:41.764748+0000","last_became_active":"2024-07-26T12:43:26.762464+0000","last_became_peered":"2024-07-26T12:43:26.762464+0000","last_unstale":"2024-07-26T12:45:41.764748+0000","last_undegraded":"2024-07-26T12:45:41.764748+0000","last_fullsized":"2024-07-26T12:45:41.764748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:10:17.598266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668869+0000","last_change":"2024-07-26T12:43:12.191640+0000","last_active":"2024-07-26T12:46:34.668869+0000","last_peered":"2024-07-26T12:46:34.668869+0000","last_clean":"2024-07-26T12:46:34.668869+0000","last_became_active":"2024-07-26T12:43:12.191260+0000","last_became_peered":"2024-07-26T12:43:12.191260+0000","last_unstale":"2024-07-26T12:46:34.668869+0000","last_undegraded":"2024-07-26T12:46:34.668869+0000","last_fullsized":"2024-07-26T12:46:34.668869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:47:18.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.257126+0000","last_change":"2024-07-26T12:43:08.135035+0000","last_active":"2024-07-26T12:45:21.257126+0000","last_peered":"2024-07-26T12:45:21.257126+0000","last_clean":"2024-07-26T12:45:21.257126+0000","last_became_active":"2024-07-26T12:43:08.134204+0000","last_became_peered":"2024-07-26T12:43:08.134204+0000","last_unstale":"2024-07-26T12:45:21.257126+0000","last_undegraded":"2024-07-26T12:45:21.257126+0000","last_fullsized":"2024-07-26T12:45:21.257126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:38.954460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:52.306230+0000","last_change":"2024-07-26T12:43:35.853244+0000","last_active":"2024-07-26T12:45:52.306230+0000","last_peered":"2024-07-26T12:45:52.306230+0000","last_clean":"2024-07-26T12:45:52.306230+0000","last_became_active":"2024-07-26T12:43:35.852728+0000","last_became_peered":"2024-07-26T12:43:35.852728+0000","last_unstale":"2024-07-26T12:45:52.306230+0000","last_undegraded":"2024-07-26T12:45:52.306230+0000","last_fullsized":"2024-07-26T12:45:52.306230+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:20:39.080000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.635080+0000","last_change":"2024-07-26T12:43:12.172457+0000","last_active":"2024-07-26T12:46:34.635080+0000","last_peered":"2024-07-26T12:46:34.635080+0000","last_clean":"2024-07-26T12:46:34.635080+0000","last_became_active":"2024-07-26T12:43:12.172034+0000","last_became_peered":"2024-07-26T12:43:12.172034+0000","last_unstale":"2024-07-26T12:46:34.635080+0000","last_undegraded":"2024-07-26T12:46:34.635080+0000","last_fullsized":"2024-07-26T12:46:34.635080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:37:02.237852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.698668+0000","last_change":"2024-07-26T12:43:08.227152+0000","last_active":"2024-07-26T12:45:21.698668+0000","last_peered":"2024-07-26T12:45:21.698668+0000","last_clean":"2024-07-26T12:45:21.698668+0000","last_became_active":"2024-07-26T12:43:08.226722+0000","last_became_peered":"2024-07-26T12:43:08.226722+0000","last_unstale":"2024-07-26T12:45:21.698668+0000","last_undegraded":"2024-07-26T12:45:21.698668+0000","last_fullsized":"2024-07-26T12:45:21.698668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:16.293211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:41.764791+0000","last_change":"2024-07-26T12:43:26.792999+0000","last_active":"2024-07-26T12:45:41.764791+0000","last_peered":"2024-07-26T12:45:41.764791+0000","last_clean":"2024-07-26T12:45:41.764791+0000","last_became_active":"2024-07-26T12:43:26.792515+0000","last_became_peered":"2024-07-26T12:43:26.792515+0000","last_unstale":"2024-07-26T12:45:41.764791+0000","last_undegraded":"2024-07-26T12:45:41.764791+0000","last_fullsized":"2024-07-26T12:45:41.764791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:49:35.104600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.673277+0000","last_change":"2024-07-26T12:43:12.185381+0000","last_active":"2024-07-26T12:46:34.673277+0000","last_peered":"2024-07-26T12:46:34.673277+0000","last_clean":"2024-07-26T12:46:34.673277+0000","last_became_active":"2024-07-26T12:43:12.184892+0000","last_became_peered":"2024-07-26T12:43:12.184892+0000","last_unstale":"2024-07-26T12:46:34.673277+0000","last_undegraded":"2024-07-26T12:46:34.673277+0000","last_fullsized":"2024-07-26T12:46:34.673277+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:38:53.992586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.762689+0000","last_change":"2024-07-26T12:43:08.895079+0000","last_active":"2024-07-26T12:45:21.762689+0000","last_peered":"2024-07-26T12:45:21.762689+0000","last_clean":"2024-07-26T12:45:21.762689+0000","last_became_active":"2024-07-26T12:43:08.894653+0000","last_became_peered":"2024-07-26T12:43:08.894653+0000","last_unstale":"2024-07-26T12:45:21.762689+0000","last_undegraded":"2024-07-26T12:45:21.762689+0000","last_fullsized":"2024-07-26T12:45:21.762689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:02:53.137905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:34.247883+0000","last_change":"2024-07-26T12:43:26.762802+0000","last_active":"2024-07-26T12:45:34.247883+0000","last_peered":"2024-07-26T12:45:34.247883+0000","last_clean":"2024-07-26T12:45:34.247883+0000","last_became_active":"2024-07-26T12:43:26.762211+0000","last_became_peered":"2024-07-26T12:43:26.762211+0000","last_unstale":"2024-07-26T12:45:34.247883+0000","last_undegraded":"2024-07-26T12:45:34.247883+0000","last_fullsized":"2024-07-26T12:45:34.247883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:35:19.623168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662791+0000","last_change":"2024-07-26T12:43:12.176062+0000","last_active":"2024-07-26T12:46:34.662791+0000","last_peered":"2024-07-26T12:46:34.662791+0000","last_clean":"2024-07-26T12:46:34.662791+0000","last_became_active":"2024-07-26T12:43:12.175671+0000","last_became_peered":"2024-07-26T12:43:12.175671+0000","last_unstale":"2024-07-26T12:46:34.662791+0000","last_undegraded":"2024-07-26T12:46:34.662791+0000","last_fullsized":"2024-07-26T12:46:34.662791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:48.374964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'18","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:47:31.284996+0000","last_change":"2024-07-26T12:43:08.131453+0000","last_active":"2024-07-26T12:47:31.284996+0000","last_peered":"2024-07-26T12:47:31.284996+0000","last_clean":"2024-07-26T12:47:31.284996+0000","last_became_active":"2024-07-26T12:43:08.130928+0000","last_became_peered":"2024-07-26T12:43:08.130928+0000","last_unstale":"2024-07-26T12:47:31.284996+0000","last_undegraded":"2024-07-26T12:47:31.284996+0000","last_fullsized":"2024-07-26T12:47:31.284996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:43.190860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:26.258119+0000","last_change":"2024-07-26T12:43:26.792942+0000","last_active":"2024-07-26T12:45:26.258119+0000","last_peered":"2024-07-26T12:45:26.258119+0000","last_clean":"2024-07-26T12:45:26.258119+0000","last_became_active":"2024-07-26T12:43:26.792045+0000","last_became_peered":"2024-07-26T12:43:26.792045+0000","last_unstale":"2024-07-26T12:45:26.258119+0000","last_undegraded":"2024-07-26T12:45:26.258119+0000","last_fullsized":"2024-07-26T12:45:26.258119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:56.005295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662936+0000","last_change":"2024-07-26T12:43:12.187601+0000","last_active":"2024-07-26T12:46:34.662936+0000","last_peered":"2024-07-26T12:46:34.662936+0000","last_clean":"2024-07-26T12:46:34.662936+0000","last_became_active":"2024-07-26T12:43:12.187175+0000","last_became_peered":"2024-07-26T12:43:12.187175+0000","last_unstale":"2024-07-26T12:46:34.662936+0000","last_undegraded":"2024-07-26T12:46:34.662936+0000","last_fullsized":"2024-07-26T12:46:34.662936+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:06.992854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:12.301049+0000","last_change":"2024-07-26T12:43:08.227279+0000","last_active":"2024-07-26T12:45:12.301049+0000","last_peered":"2024-07-26T12:45:12.301049+0000","last_clean":"2024-07-26T12:45:12.301049+0000","last_became_active":"2024-07-26T12:43:08.226634+0000","last_became_peered":"2024-07-26T12:43:08.226634+0000","last_unstale":"2024-07-26T12:45:12.301049+0000","last_undegraded":"2024-07-26T12:45:12.301049+0000","last_fullsized":"2024-07-26T12:45:12.301049+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:19.845664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:31.537561+0000","last_change":"2024-07-26T12:43:26.774537+0000","last_active":"2024-07-26T12:45:31.537561+0000","last_peered":"2024-07-26T12:45:31.537561+0000","last_clean":"2024-07-26T12:45:31.537561+0000","last_became_active":"2024-07-26T12:43:26.774036+0000","last_became_peered":"2024-07-26T12:43:26.774036+0000","last_unstale":"2024-07-26T12:45:31.537561+0000","last_undegraded":"2024-07-26T12:45:31.537561+0000","last_fullsized":"2024-07-26T12:45:31.537561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:21.774498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662838+0000","last_change":"2024-07-26T12:44:36.857443+0000","last_active":"2024-07-26T12:46:34.662838+0000","last_peered":"2024-07-26T12:46:34.662838+0000","last_clean":"2024-07-26T12:46:34.662838+0000","last_became_active":"2024-07-26T12:44:36.857001+0000","last_became_peered":"2024-07-26T12:44:36.857001+0000","last_unstale":"2024-07-26T12:46:34.662838+0000","last_undegraded":"2024-07-26T12:46:34.662838+0000","last_fullsized":"2024-07-26T12:46:34.662838+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:08.236404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:16.261130+0000","last_change":"2024-07-26T12:43:08.227432+0000","last_active":"2024-07-26T12:45:16.261130+0000","last_peered":"2024-07-26T12:45:16.261130+0000","last_clean":"2024-07-26T12:45:16.261130+0000","last_became_active":"2024-07-26T12:43:08.226645+0000","last_became_peered":"2024-07-26T12:43:08.226645+0000","last_unstale":"2024-07-26T12:45:16.261130+0000","last_undegraded":"2024-07-26T12:45:16.261130+0000","last_fullsized":"2024-07-26T12:45:16.261130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:23.498855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:32.303272+0000","last_change":"2024-07-26T12:43:26.792874+0000","last_active":"2024-07-26T12:45:32.303272+0000","last_peered":"2024-07-26T12:45:32.303272+0000","last_clean":"2024-07-26T12:45:32.303272+0000","last_became_active":"2024-07-26T12:43:26.792371+0000","last_became_peered":"2024-07-26T12:43:26.792371+0000","last_unstale":"2024-07-26T12:45:32.303272+0000","last_undegraded":"2024-07-26T12:45:32.303272+0000","last_fullsized":"2024-07-26T12:45:32.303272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:25.753111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.654086+0000","last_change":"2024-07-26T12:43:12.185733+0000","last_active":"2024-07-26T12:46:34.654086+0000","last_peered":"2024-07-26T12:46:34.654086+0000","last_clean":"2024-07-26T12:46:34.654086+0000","last_became_active":"2024-07-26T12:43:12.185305+0000","last_became_peered":"2024-07-26T12:43:12.185305+0000","last_unstale":"2024-07-26T12:46:34.654086+0000","last_undegraded":"2024-07-26T12:46:34.654086+0000","last_fullsized":"2024-07-26T12:46:34.654086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:44.634607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":296,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:41.374820+0000","last_change":"2024-07-26T12:43:08.118508+0000","last_active":"2024-07-26T12:48:41.374820+0000","last_peered":"2024-07-26T12:48:41.374820+0000","last_clean":"2024-07-26T12:48:41.374820+0000","last_became_active":"2024-07-26T12:43:08.118088+0000","last_became_peered":"2024-07-26T12:43:08.118088+0000","last_unstale":"2024-07-26T12:48:41.374820+0000","last_undegraded":"2024-07-26T12:48:41.374820+0000","last_fullsized":"2024-07-26T12:48:41.374820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:23:46.474298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:33.076402+0000","last_change":"2024-07-26T12:43:26.779328+0000","last_active":"2024-07-26T12:45:33.076402+0000","last_peered":"2024-07-26T12:45:33.076402+0000","last_clean":"2024-07-26T12:45:33.076402+0000","last_became_active":"2024-07-26T12:43:26.778912+0000","last_became_peered":"2024-07-26T12:43:26.778912+0000","last_unstale":"2024-07-26T12:45:33.076402+0000","last_undegraded":"2024-07-26T12:45:33.076402+0000","last_fullsized":"2024-07-26T12:45:33.076402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:58.364775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650476+0000","last_change":"2024-07-26T12:43:35.854997+0000","last_active":"2024-07-26T12:46:34.650476+0000","last_peered":"2024-07-26T12:46:34.650476+0000","last_clean":"2024-07-26T12:46:34.650476+0000","last_became_active":"2024-07-26T12:43:35.854387+0000","last_became_peered":"2024-07-26T12:43:35.854387+0000","last_unstale":"2024-07-26T12:46:34.650476+0000","last_undegraded":"2024-07-26T12:46:34.650476+0000","last_fullsized":"2024-07-26T12:46:34.650476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:04:20.489388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":296,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:41.504837+0000","last_change":"2024-07-26T12:43:08.227628+0000","last_active":"2024-07-26T12:48:41.504837+0000","last_peered":"2024-07-26T12:48:41.504837+0000","last_clean":"2024-07-26T12:48:41.504837+0000","last_became_active":"2024-07-26T12:43:08.227133+0000","last_became_peered":"2024-07-26T12:43:08.227133+0000","last_unstale":"2024-07-26T12:48:41.504837+0000","last_undegraded":"2024-07-26T12:48:41.504837+0000","last_fullsized":"2024-07-26T12:48:41.504837+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:28:28.484315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:38.077398+0000","last_change":"2024-07-26T12:43:26.794016+0000","last_active":"2024-07-26T12:45:38.077398+0000","last_peered":"2024-07-26T12:45:38.077398+0000","last_clean":"2024-07-26T12:45:38.077398+0000","last_became_active":"2024-07-26T12:43:26.792562+0000","last_became_peered":"2024-07-26T12:43:26.792562+0000","last_unstale":"2024-07-26T12:45:38.077398+0000","last_undegraded":"2024-07-26T12:45:38.077398+0000","last_fullsized":"2024-07-26T12:45:38.077398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:26.162840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.653115+0000","last_change":"2024-07-26T12:43:12.173014+0000","last_active":"2024-07-26T12:46:34.653115+0000","last_peered":"2024-07-26T12:46:34.653115+0000","last_clean":"2024-07-26T12:46:34.653115+0000","last_became_active":"2024-07-26T12:43:12.172600+0000","last_became_peered":"2024-07-26T12:43:12.172600+0000","last_unstale":"2024-07-26T12:46:34.653115+0000","last_undegraded":"2024-07-26T12:46:34.653115+0000","last_fullsized":"2024-07-26T12:46:34.653115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:59.842744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:07.570667+0000","last_change":"2024-07-26T12:43:08.119000+0000","last_active":"2024-07-26T12:45:07.570667+0000","last_peered":"2024-07-26T12:45:07.570667+0000","last_clean":"2024-07-26T12:45:07.570667+0000","last_became_active":"2024-07-26T12:43:08.118608+0000","last_became_peered":"2024-07-26T12:43:08.118608+0000","last_unstale":"2024-07-26T12:45:07.570667+0000","last_undegraded":"2024-07-26T12:45:07.570667+0000","last_fullsized":"2024-07-26T12:45:07.570667+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:29:47.663449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:39.012097+0000","last_change":"2024-07-26T12:43:35.914716+0000","last_active":"2024-07-26T12:45:39.012097+0000","last_peered":"2024-07-26T12:45:39.012097+0000","last_clean":"2024-07-26T12:45:39.012097+0000","last_became_active":"2024-07-26T12:43:35.914354+0000","last_became_peered":"2024-07-26T12:43:35.914354+0000","last_unstale":"2024-07-26T12:45:39.012097+0000","last_undegraded":"2024-07-26T12:45:39.012097+0000","last_fullsized":"2024-07-26T12:45:39.012097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:09:10.994141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662686+0000","last_change":"2024-07-26T12:43:12.169997+0000","last_active":"2024-07-26T12:46:34.662686+0000","last_peered":"2024-07-26T12:46:34.662686+0000","last_clean":"2024-07-26T12:46:34.662686+0000","last_became_active":"2024-07-26T12:43:12.169044+0000","last_became_peered":"2024-07-26T12:43:12.169044+0000","last_unstale":"2024-07-26T12:46:34.662686+0000","last_undegraded":"2024-07-26T12:46:34.662686+0000","last_fullsized":"2024-07-26T12:46:34.662686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:41.791434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:16.762094+0000","last_change":"2024-07-26T12:43:08.129025+0000","last_active":"2024-07-26T12:45:16.762094+0000","last_peered":"2024-07-26T12:45:16.762094+0000","last_clean":"2024-07-26T12:45:16.762094+0000","last_became_active":"2024-07-26T12:43:08.128608+0000","last_became_peered":"2024-07-26T12:43:08.128608+0000","last_unstale":"2024-07-26T12:45:16.762094+0000","last_undegraded":"2024-07-26T12:45:16.762094+0000","last_fullsized":"2024-07-26T12:45:16.762094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:43.749473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":123,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:22.580061+0000","last_change":"2024-07-26T12:41:42.411595+0000","last_active":"2024-07-26T12:46:22.580061+0000","last_peered":"2024-07-26T12:46:22.580061+0000","last_clean":"2024-07-26T12:46:22.580061+0000","last_became_active":"2024-07-26T12:41:42.407119+0000","last_became_peered":"2024-07-26T12:41:42.407119+0000","last_unstale":"2024-07-26T12:46:22.580061+0000","last_undegraded":"2024-07-26T12:46:22.580061+0000","last_fullsized":"2024-07-26T12:46:22.580061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_clean_scrub_stamp":"2024-07-26T12:37:59.605754+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:21.376418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:32.573794+0000","last_change":"2024-07-26T12:43:26.763955+0000","last_active":"2024-07-26T12:45:32.573794+0000","last_peered":"2024-07-26T12:45:32.573794+0000","last_clean":"2024-07-26T12:45:32.573794+0000","last_became_active":"2024-07-26T12:43:26.763368+0000","last_became_peered":"2024-07-26T12:43:26.763368+0000","last_unstale":"2024-07-26T12:45:32.573794+0000","last_undegraded":"2024-07-26T12:45:32.573794+0000","last_fullsized":"2024-07-26T12:45:32.573794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:48:14.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.663123+0000","last_change":"2024-07-26T12:43:12.174780+0000","last_active":"2024-07-26T12:46:34.663123+0000","last_peered":"2024-07-26T12:46:34.663123+0000","last_clean":"2024-07-26T12:46:34.663123+0000","last_became_active":"2024-07-26T12:43:12.174379+0000","last_became_peered":"2024-07-26T12:43:12.174379+0000","last_unstale":"2024-07-26T12:46:34.663123+0000","last_undegraded":"2024-07-26T12:46:34.663123+0000","last_fullsized":"2024-07-26T12:46:34.663123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:42:22.783259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'7","reported_seq":302,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.278004+0000","last_change":"2024-07-26T12:43:08.894655+0000","last_active":"2024-07-26T12:48:42.278004+0000","last_peered":"2024-07-26T12:48:42.278004+0000","last_clean":"2024-07-26T12:48:42.278004+0000","last_became_active":"2024-07-26T12:43:08.894270+0000","last_became_peered":"2024-07-26T12:43:08.894270+0000","last_unstale":"2024-07-26T12:48:42.278004+0000","last_undegraded":"2024-07-26T12:48:42.278004+0000","last_fullsized":"2024-07-26T12:48:42.278004+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:34.431204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":248,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:36.700412+0000","last_change":"2024-07-26T12:43:26.769716+0000","last_active":"2024-07-26T12:45:36.700412+0000","last_peered":"2024-07-26T12:45:36.700412+0000","last_clean":"2024-07-26T12:45:36.700412+0000","last_became_active":"2024-07-26T12:43:26.769154+0000","last_became_peered":"2024-07-26T12:43:26.769154+0000","last_unstale":"2024-07-26T12:45:36.700412+0000","last_undegraded":"2024-07-26T12:45:36.700412+0000","last_fullsized":"2024-07-26T12:45:36.700412+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:43.102498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664039+0000","last_change":"2024-07-26T12:43:12.179488+0000","last_active":"2024-07-26T12:46:34.664039+0000","last_peered":"2024-07-26T12:46:34.664039+0000","last_clean":"2024-07-26T12:46:34.664039+0000","last_became_active":"2024-07-26T12:43:12.179171+0000","last_became_peered":"2024-07-26T12:43:12.179171+0000","last_unstale":"2024-07-26T12:46:34.664039+0000","last_undegraded":"2024-07-26T12:46:34.664039+0000","last_fullsized":"2024-07-26T12:46:34.664039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:43.695582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:24.009763+0000","last_change":"2024-07-26T12:43:08.130838+0000","last_active":"2024-07-26T12:45:24.009763+0000","last_peered":"2024-07-26T12:45:24.009763+0000","last_clean":"2024-07-26T12:45:24.009763+0000","last_became_active":"2024-07-26T12:43:08.130446+0000","last_became_peered":"2024-07-26T12:43:08.130446+0000","last_unstale":"2024-07-26T12:45:24.009763+0000","last_undegraded":"2024-07-26T12:45:24.009763+0000","last_fullsized":"2024-07-26T12:45:24.009763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:33:55.271038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:42.787196+0000","last_change":"2024-07-26T12:43:26.781982+0000","last_active":"2024-07-26T12:45:42.787196+0000","last_peered":"2024-07-26T12:45:42.787196+0000","last_clean":"2024-07-26T12:45:42.787196+0000","last_became_active":"2024-07-26T12:43:26.777102+0000","last_became_peered":"2024-07-26T12:43:26.777102+0000","last_unstale":"2024-07-26T12:45:42.787196+0000","last_undegraded":"2024-07-26T12:45:42.787196+0000","last_fullsized":"2024-07-26T12:45:42.787196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:49.553124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659860+0000","last_change":"2024-07-26T12:43:12.177043+0000","last_active":"2024-07-26T12:46:34.659860+0000","last_peered":"2024-07-26T12:46:34.659860+0000","last_clean":"2024-07-26T12:46:34.659860+0000","last_became_active":"2024-07-26T12:43:12.176634+0000","last_became_peered":"2024-07-26T12:43:12.176634+0000","last_unstale":"2024-07-26T12:46:34.659860+0000","last_undegraded":"2024-07-26T12:46:34.659860+0000","last_fullsized":"2024-07-26T12:46:34.659860+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:20.210176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:23.074306+0000","last_change":"2024-07-26T12:43:08.131279+0000","last_active":"2024-07-26T12:45:23.074306+0000","last_peered":"2024-07-26T12:45:23.074306+0000","last_clean":"2024-07-26T12:45:23.074306+0000","last_became_active":"2024-07-26T12:43:08.130755+0000","last_became_peered":"2024-07-26T12:43:08.130755+0000","last_unstale":"2024-07-26T12:45:23.074306+0000","last_undegraded":"2024-07-26T12:45:23.074306+0000","last_fullsized":"2024-07-26T12:45:23.074306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:03.234905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:39.012012+0000","last_change":"2024-07-26T12:43:26.764407+0000","last_active":"2024-07-26T12:45:39.012012+0000","last_peered":"2024-07-26T12:45:39.012012+0000","last_clean":"2024-07-26T12:45:39.012012+0000","last_became_active":"2024-07-26T12:43:26.762408+0000","last_became_peered":"2024-07-26T12:43:26.762408+0000","last_unstale":"2024-07-26T12:45:39.012012+0000","last_undegraded":"2024-07-26T12:45:39.012012+0000","last_fullsized":"2024-07-26T12:45:39.012012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:35:29.139476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.648078+0000","last_change":"2024-07-26T12:43:12.174587+0000","last_active":"2024-07-26T12:46:34.648078+0000","last_peered":"2024-07-26T12:46:34.648078+0000","last_clean":"2024-07-26T12:46:34.648078+0000","last_became_active":"2024-07-26T12:43:12.173929+0000","last_became_peered":"2024-07-26T12:43:12.173929+0000","last_unstale":"2024-07-26T12:46:34.648078+0000","last_undegraded":"2024-07-26T12:46:34.648078+0000","last_fullsized":"2024-07-26T12:46:34.648078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:16.775198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:22.302322+0000","last_change":"2024-07-26T12:43:08.119825+0000","last_active":"2024-07-26T12:45:22.302322+0000","last_peered":"2024-07-26T12:45:22.302322+0000","last_clean":"2024-07-26T12:45:22.302322+0000","last_became_active":"2024-07-26T12:43:08.119273+0000","last_became_peered":"2024-07-26T12:43:08.119273+0000","last_unstale":"2024-07-26T12:45:22.302322+0000","last_undegraded":"2024-07-26T12:45:22.302322+0000","last_fullsized":"2024-07-26T12:45:22.302322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:39.973020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:39.012041+0000","last_change":"2024-07-26T12:43:26.773226+0000","last_active":"2024-07-26T12:45:39.012041+0000","last_peered":"2024-07-26T12:45:39.012041+0000","last_clean":"2024-07-26T12:45:39.012041+0000","last_became_active":"2024-07-26T12:43:26.772745+0000","last_became_peered":"2024-07-26T12:43:26.772745+0000","last_unstale":"2024-07-26T12:45:39.012041+0000","last_undegraded":"2024-07-26T12:45:39.012041+0000","last_fullsized":"2024-07-26T12:45:39.012041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:35.407138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.648325+0000","last_change":"2024-07-26T12:43:12.176178+0000","last_active":"2024-07-26T12:46:34.648325+0000","last_peered":"2024-07-26T12:46:34.648325+0000","last_clean":"2024-07-26T12:46:34.648325+0000","last_became_active":"2024-07-26T12:43:12.175791+0000","last_became_peered":"2024-07-26T12:43:12.175791+0000","last_unstale":"2024-07-26T12:46:34.648325+0000","last_undegraded":"2024-07-26T12:46:34.648325+0000","last_fullsized":"2024-07-26T12:46:34.648325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:48.885831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:27.302835+0000","last_change":"2024-07-26T12:43:08.226345+0000","last_active":"2024-07-26T12:45:27.302835+0000","last_peered":"2024-07-26T12:45:27.302835+0000","last_clean":"2024-07-26T12:45:27.302835+0000","last_became_active":"2024-07-26T12:43:08.225897+0000","last_became_peered":"2024-07-26T12:43:08.225897+0000","last_unstale":"2024-07-26T12:45:27.302835+0000","last_undegraded":"2024-07-26T12:45:27.302835+0000","last_fullsized":"2024-07-26T12:45:27.302835+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:59.351076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:53.079873+0000","last_change":"2024-07-26T12:43:26.774800+0000","last_active":"2024-07-26T12:45:53.079873+0000","last_peered":"2024-07-26T12:45:53.079873+0000","last_clean":"2024-07-26T12:45:53.079873+0000","last_became_active":"2024-07-26T12:43:26.774192+0000","last_became_peered":"2024-07-26T12:43:26.774192+0000","last_unstale":"2024-07-26T12:45:53.079873+0000","last_undegraded":"2024-07-26T12:45:53.079873+0000","last_fullsized":"2024-07-26T12:45:53.079873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:18.379619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659204+0000","last_change":"2024-07-26T12:43:12.171720+0000","last_active":"2024-07-26T12:46:34.659204+0000","last_peered":"2024-07-26T12:46:34.659204+0000","last_clean":"2024-07-26T12:46:34.659204+0000","last_became_active":"2024-07-26T12:43:12.171334+0000","last_became_peered":"2024-07-26T12:43:12.171334+0000","last_unstale":"2024-07-26T12:46:34.659204+0000","last_undegraded":"2024-07-26T12:46:34.659204+0000","last_fullsized":"2024-07-26T12:46:34.659204+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:35.156430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":296,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:40.503197+0000","last_change":"2024-07-26T12:43:08.120325+0000","last_active":"2024-07-26T12:48:40.503197+0000","last_peered":"2024-07-26T12:48:40.503197+0000","last_clean":"2024-07-26T12:48:40.503197+0000","last_became_active":"2024-07-26T12:43:08.119896+0000","last_became_peered":"2024-07-26T12:43:08.119896+0000","last_unstale":"2024-07-26T12:48:40.503197+0000","last_undegraded":"2024-07-26T12:48:40.503197+0000","last_fullsized":"2024-07-26T12:48:40.503197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:23:19.470491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":245,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:51.266909+0000","last_change":"2024-07-26T12:43:26.778688+0000","last_active":"2024-07-26T12:45:51.266909+0000","last_peered":"2024-07-26T12:45:51.266909+0000","last_clean":"2024-07-26T12:45:51.266909+0000","last_became_active":"2024-07-26T12:43:26.778239+0000","last_became_peered":"2024-07-26T12:43:26.778239+0000","last_unstale":"2024-07-26T12:45:51.266909+0000","last_undegraded":"2024-07-26T12:45:51.266909+0000","last_fullsized":"2024-07-26T12:45:51.266909+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:50:48.409636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.676557+0000","last_change":"2024-07-26T12:43:12.182323+0000","last_active":"2024-07-26T12:46:34.676557+0000","last_peered":"2024-07-26T12:46:34.676557+0000","last_clean":"2024-07-26T12:46:34.676557+0000","last_became_active":"2024-07-26T12:43:12.181860+0000","last_became_peered":"2024-07-26T12:43:12.181860+0000","last_unstale":"2024-07-26T12:46:34.676557+0000","last_undegraded":"2024-07-26T12:46:34.676557+0000","last_fullsized":"2024-07-26T12:46:34.676557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:08.764979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:29.010371+0000","last_change":"2024-07-26T12:43:08.130823+0000","last_active":"2024-07-26T12:45:29.010371+0000","last_peered":"2024-07-26T12:45:29.010371+0000","last_clean":"2024-07-26T12:45:29.010371+0000","last_became_active":"2024-07-26T12:43:08.130424+0000","last_became_peered":"2024-07-26T12:43:08.130424+0000","last_unstale":"2024-07-26T12:45:29.010371+0000","last_undegraded":"2024-07-26T12:45:29.010371+0000","last_fullsized":"2024-07-26T12:45:29.010371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:49:35.829624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.539348+0000","last_change":"2024-07-26T12:43:26.770070+0000","last_active":"2024-07-26T12:45:46.539348+0000","last_peered":"2024-07-26T12:45:46.539348+0000","last_clean":"2024-07-26T12:45:46.539348+0000","last_became_active":"2024-07-26T12:43:26.769594+0000","last_became_peered":"2024-07-26T12:43:26.769594+0000","last_unstale":"2024-07-26T12:45:46.539348+0000","last_undegraded":"2024-07-26T12:45:46.539348+0000","last_fullsized":"2024-07-26T12:45:46.539348+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:18.342739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669284+0000","last_change":"2024-07-26T12:43:12.183134+0000","last_active":"2024-07-26T12:46:34.669284+0000","last_peered":"2024-07-26T12:46:34.669284+0000","last_clean":"2024-07-26T12:46:34.669284+0000","last_became_active":"2024-07-26T12:43:12.182406+0000","last_became_peered":"2024-07-26T12:43:12.182406+0000","last_unstale":"2024-07-26T12:46:34.669284+0000","last_undegraded":"2024-07-26T12:46:34.669284+0000","last_fullsized":"2024-07-26T12:46:34.669284+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:54:24.038971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:56.540854+0000","last_change":"2024-07-26T12:43:35.862005+0000","last_active":"2024-07-26T12:45:56.540854+0000","last_peered":"2024-07-26T12:45:56.540854+0000","last_clean":"2024-07-26T12:45:56.540854+0000","last_became_active":"2024-07-26T12:43:35.850911+0000","last_became_peered":"2024-07-26T12:43:35.850911+0000","last_unstale":"2024-07-26T12:45:56.540854+0000","last_undegraded":"2024-07-26T12:45:56.540854+0000","last_fullsized":"2024-07-26T12:45:56.540854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:18:16.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:54.014118+0000","last_change":"2024-07-26T12:43:26.774599+0000","last_active":"2024-07-26T12:45:54.014118+0000","last_peered":"2024-07-26T12:45:54.014118+0000","last_clean":"2024-07-26T12:45:54.014118+0000","last_became_active":"2024-07-26T12:43:26.774186+0000","last_became_peered":"2024-07-26T12:43:26.774186+0000","last_unstale":"2024-07-26T12:45:54.014118+0000","last_undegraded":"2024-07-26T12:45:54.014118+0000","last_fullsized":"2024-07-26T12:45:54.014118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:02:42.252093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.646564+0000","last_change":"2024-07-26T12:43:12.174676+0000","last_active":"2024-07-26T12:46:34.646564+0000","last_peered":"2024-07-26T12:46:34.646564+0000","last_clean":"2024-07-26T12:46:34.646564+0000","last_became_active":"2024-07-26T12:43:12.174310+0000","last_became_peered":"2024-07-26T12:43:12.174310+0000","last_unstale":"2024-07-26T12:46:34.646564+0000","last_undegraded":"2024-07-26T12:46:34.646564+0000","last_fullsized":"2024-07-26T12:46:34.646564+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:44.381289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'42","reported_seq":587,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.277571+0000","last_change":"2024-07-26T12:43:08.122675+0000","last_active":"2024-07-26T12:48:42.277571+0000","last_peered":"2024-07-26T12:48:42.277571+0000","last_clean":"2024-07-26T12:48:42.277571+0000","last_became_active":"2024-07-26T12:43:08.122174+0000","last_became_peered":"2024-07-26T12:43:08.122174+0000","last_unstale":"2024-07-26T12:48:42.277571+0000","last_undegraded":"2024-07-26T12:48:42.277571+0000","last_fullsized":"2024-07-26T12:48:42.277571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:54.513306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70065,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":498,"num_read_kb":318,"num_write":44,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.265915+0000","last_change":"2024-07-26T12:43:26.764283+0000","last_active":"2024-07-26T12:45:46.265915+0000","last_peered":"2024-07-26T12:45:46.265915+0000","last_clean":"2024-07-26T12:45:46.265915+0000","last_became_active":"2024-07-26T12:43:26.763821+0000","last_became_peered":"2024-07-26T12:43:26.763821+0000","last_unstale":"2024-07-26T12:45:46.265915+0000","last_undegraded":"2024-07-26T12:45:46.265915+0000","last_fullsized":"2024-07-26T12:45:46.265915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:56.753016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672277+0000","last_change":"2024-07-26T12:43:35.915277+0000","last_active":"2024-07-26T12:46:34.672277+0000","last_peered":"2024-07-26T12:46:34.672277+0000","last_clean":"2024-07-26T12:46:34.672277+0000","last_became_active":"2024-07-26T12:43:35.914848+0000","last_became_peered":"2024-07-26T12:43:35.914848+0000","last_unstale":"2024-07-26T12:46:34.672277+0000","last_undegraded":"2024-07-26T12:46:34.672277+0000","last_fullsized":"2024-07-26T12:46:34.672277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:56:35.639076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:34.011224+0000","last_change":"2024-07-26T12:43:08.227352+0000","last_active":"2024-07-26T12:45:34.011224+0000","last_peered":"2024-07-26T12:45:34.011224+0000","last_clean":"2024-07-26T12:45:34.011224+0000","last_became_active":"2024-07-26T12:43:08.226857+0000","last_became_peered":"2024-07-26T12:43:08.226857+0000","last_unstale":"2024-07-26T12:45:34.011224+0000","last_undegraded":"2024-07-26T12:45:34.011224+0000","last_fullsized":"2024-07-26T12:45:34.011224+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:20.156503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.765392+0000","last_change":"2024-07-26T12:43:26.762943+0000","last_active":"2024-07-26T12:45:46.765392+0000","last_peered":"2024-07-26T12:45:46.765392+0000","last_clean":"2024-07-26T12:45:46.765392+0000","last_became_active":"2024-07-26T12:43:26.762436+0000","last_became_peered":"2024-07-26T12:43:26.762436+0000","last_unstale":"2024-07-26T12:45:46.765392+0000","last_undegraded":"2024-07-26T12:45:46.765392+0000","last_fullsized":"2024-07-26T12:45:46.765392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:05:23.924397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.673242+0000","last_change":"2024-07-26T12:44:36.854705+0000","last_active":"2024-07-26T12:46:34.673242+0000","last_peered":"2024-07-26T12:46:34.673242+0000","last_clean":"2024-07-26T12:46:34.673242+0000","last_became_active":"2024-07-26T12:44:36.854268+0000","last_became_peered":"2024-07-26T12:44:36.854268+0000","last_unstale":"2024-07-26T12:46:34.673242+0000","last_undegraded":"2024-07-26T12:46:34.673242+0000","last_fullsized":"2024-07-26T12:46:34.673242+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:24:30.768905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:32.573818+0000","last_change":"2024-07-26T12:43:08.118597+0000","last_active":"2024-07-26T12:45:32.573818+0000","last_peered":"2024-07-26T12:45:32.573818+0000","last_clean":"2024-07-26T12:45:32.573818+0000","last_became_active":"2024-07-26T12:43:08.118145+0000","last_became_peered":"2024-07-26T12:43:08.118145+0000","last_unstale":"2024-07-26T12:45:32.573818+0000","last_undegraded":"2024-07-26T12:45:32.573818+0000","last_fullsized":"2024-07-26T12:45:32.573818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:52:45.511693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.265831+0000","last_change":"2024-07-26T12:43:26.771861+0000","last_active":"2024-07-26T12:45:46.265831+0000","last_peered":"2024-07-26T12:45:46.265831+0000","last_clean":"2024-07-26T12:45:46.265831+0000","last_became_active":"2024-07-26T12:43:26.771323+0000","last_became_peered":"2024-07-26T12:43:26.771323+0000","last_unstale":"2024-07-26T12:45:46.265831+0000","last_undegraded":"2024-07-26T12:45:46.265831+0000","last_fullsized":"2024-07-26T12:45:46.265831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:34.207514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656491+0000","last_change":"2024-07-26T12:43:12.169437+0000","last_active":"2024-07-26T12:46:34.656491+0000","last_peered":"2024-07-26T12:46:34.656491+0000","last_clean":"2024-07-26T12:46:34.656491+0000","last_became_active":"2024-07-26T12:43:12.168387+0000","last_became_peered":"2024-07-26T12:43:12.168387+0000","last_unstale":"2024-07-26T12:46:34.656491+0000","last_undegraded":"2024-07-26T12:46:34.656491+0000","last_fullsized":"2024-07-26T12:46:34.656491+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:57:06.148965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'6","reported_seq":539,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.277078+0000","last_change":"2024-07-26T12:43:08.129618+0000","last_active":"2024-07-26T12:48:42.277078+0000","last_peered":"2024-07-26T12:48:42.277078+0000","last_clean":"2024-07-26T12:48:42.277078+0000","last_became_active":"2024-07-26T12:43:08.129064+0000","last_became_peered":"2024-07-26T12:43:08.129064+0000","last_unstale":"2024-07-26T12:48:42.277078+0000","last_undegraded":"2024-07-26T12:48:42.277078+0000","last_fullsized":"2024-07-26T12:48:42.277078+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:26:22.406075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":486,"num_read_kb":486,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.261842+0000","last_change":"2024-07-26T12:43:35.854710+0000","last_active":"2024-07-26T12:45:46.261842+0000","last_peered":"2024-07-26T12:45:46.261842+0000","last_clean":"2024-07-26T12:45:46.261842+0000","last_became_active":"2024-07-26T12:43:35.852772+0000","last_became_peered":"2024-07-26T12:43:35.852772+0000","last_unstale":"2024-07-26T12:45:46.261842+0000","last_undegraded":"2024-07-26T12:45:46.261842+0000","last_fullsized":"2024-07-26T12:45:46.261842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:24:54.015879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:27.784210+0000","last_change":"2024-07-26T12:43:26.783182+0000","last_active":"2024-07-26T12:45:27.784210+0000","last_peered":"2024-07-26T12:45:27.784210+0000","last_clean":"2024-07-26T12:45:27.784210+0000","last_became_active":"2024-07-26T12:43:26.782697+0000","last_became_peered":"2024-07-26T12:43:26.782697+0000","last_unstale":"2024-07-26T12:45:27.784210+0000","last_undegraded":"2024-07-26T12:45:27.784210+0000","last_fullsized":"2024-07-26T12:45:27.784210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:57.169222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671751+0000","last_change":"2024-07-26T12:43:12.166163+0000","last_active":"2024-07-26T12:46:34.671751+0000","last_peered":"2024-07-26T12:46:34.671751+0000","last_clean":"2024-07-26T12:46:34.671751+0000","last_became_active":"2024-07-26T12:43:12.164427+0000","last_became_peered":"2024-07-26T12:43:12.164427+0000","last_unstale":"2024-07-26T12:46:34.671751+0000","last_undegraded":"2024-07-26T12:46:34.671751+0000","last_fullsized":"2024-07-26T12:46:34.671751+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:25.335508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:34.011158+0000","last_change":"2024-07-26T12:43:08.118224+0000","last_active":"2024-07-26T12:45:34.011158+0000","last_peered":"2024-07-26T12:45:34.011158+0000","last_clean":"2024-07-26T12:45:34.011158+0000","last_became_active":"2024-07-26T12:43:08.117821+0000","last_became_peered":"2024-07-26T12:43:08.117821+0000","last_unstale":"2024-07-26T12:45:34.011158+0000","last_undegraded":"2024-07-26T12:45:34.011158+0000","last_fullsized":"2024-07-26T12:45:34.011158+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:56:14.380931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:31.537591+0000","last_change":"2024-07-26T12:43:35.907740+0000","last_active":"2024-07-26T12:45:31.537591+0000","last_peered":"2024-07-26T12:45:31.537591+0000","last_clean":"2024-07-26T12:45:31.537591+0000","last_became_active":"2024-07-26T12:43:35.907115+0000","last_became_peered":"2024-07-26T12:43:35.907115+0000","last_unstale":"2024-07-26T12:45:31.537591+0000","last_undegraded":"2024-07-26T12:45:31.537591+0000","last_fullsized":"2024-07-26T12:45:31.537591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:53:24.783872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.670573+0000","last_change":"2024-07-26T12:43:12.177579+0000","last_active":"2024-07-26T12:46:34.670573+0000","last_peered":"2024-07-26T12:46:34.670573+0000","last_clean":"2024-07-26T12:46:34.670573+0000","last_became_active":"2024-07-26T12:43:12.171433+0000","last_became_peered":"2024-07-26T12:43:12.171433+0000","last_unstale":"2024-07-26T12:46:34.670573+0000","last_undegraded":"2024-07-26T12:46:34.670573+0000","last_fullsized":"2024-07-26T12:46:34.670573+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:50.972151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:33.076329+0000","last_change":"2024-07-26T12:43:08.133446+0000","last_active":"2024-07-26T12:45:33.076329+0000","last_peered":"2024-07-26T12:45:33.076329+0000","last_clean":"2024-07-26T12:45:33.076329+0000","last_became_active":"2024-07-26T12:43:08.133008+0000","last_became_peered":"2024-07-26T12:43:08.133008+0000","last_unstale":"2024-07-26T12:45:33.076329+0000","last_undegraded":"2024-07-26T12:45:33.076329+0000","last_fullsized":"2024-07-26T12:45:33.076329+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:05:21.283738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:37.266863+0000","last_change":"2024-07-26T12:43:08.124976+0000","last_active":"2024-07-26T12:45:37.266863+0000","last_peered":"2024-07-26T12:45:37.266863+0000","last_clean":"2024-07-26T12:45:37.266863+0000","last_became_active":"2024-07-26T12:43:08.124664+0000","last_became_peered":"2024-07-26T12:43:08.124664+0000","last_unstale":"2024-07-26T12:45:37.266863+0000","last_undegraded":"2024-07-26T12:45:37.266863+0000","last_fullsized":"2024-07-26T12:45:37.266863+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:51.798657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664310+0000","last_change":"2024-07-26T12:43:35.856266+0000","last_active":"2024-07-26T12:46:34.664310+0000","last_peered":"2024-07-26T12:46:34.664310+0000","last_clean":"2024-07-26T12:46:34.664310+0000","last_became_active":"2024-07-26T12:43:35.855802+0000","last_became_peered":"2024-07-26T12:43:35.855802+0000","last_unstale":"2024-07-26T12:46:34.664310+0000","last_undegraded":"2024-07-26T12:46:34.664310+0000","last_fullsized":"2024-07-26T12:46:34.664310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:16.432983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:59.251187+0000","last_change":"2024-07-26T12:43:26.778626+0000","last_active":"2024-07-26T12:45:59.251187+0000","last_peered":"2024-07-26T12:45:59.251187+0000","last_clean":"2024-07-26T12:45:59.251187+0000","last_became_active":"2024-07-26T12:43:26.778165+0000","last_became_peered":"2024-07-26T12:43:26.778165+0000","last_unstale":"2024-07-26T12:45:59.251187+0000","last_undegraded":"2024-07-26T12:45:59.251187+0000","last_fullsized":"2024-07-26T12:45:59.251187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:06:08.766495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":302,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.277799+0000","last_change":"2024-07-26T12:43:08.119702+0000","last_active":"2024-07-26T12:48:42.277799+0000","last_peered":"2024-07-26T12:48:42.277799+0000","last_clean":"2024-07-26T12:48:42.277799+0000","last_became_active":"2024-07-26T12:43:08.119225+0000","last_became_peered":"2024-07-26T12:43:08.119225+0000","last_unstale":"2024-07-26T12:48:42.277799+0000","last_undegraded":"2024-07-26T12:48:42.277799+0000","last_fullsized":"2024-07-26T12:48:42.277799+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:27.616873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":248,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.640998+0000","last_change":"2024-07-26T12:43:12.166239+0000","last_active":"2024-07-26T12:46:34.640998+0000","last_peered":"2024-07-26T12:46:34.640998+0000","last_clean":"2024-07-26T12:46:34.640998+0000","last_became_active":"2024-07-26T12:43:12.165770+0000","last_became_peered":"2024-07-26T12:43:12.165770+0000","last_unstale":"2024-07-26T12:46:34.640998+0000","last_undegraded":"2024-07-26T12:46:34.640998+0000","last_fullsized":"2024-07-26T12:46:34.640998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:11:15.290898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:59.251224+0000","last_change":"2024-07-26T12:43:26.765905+0000","last_active":"2024-07-26T12:45:59.251224+0000","last_peered":"2024-07-26T12:45:59.251224+0000","last_clean":"2024-07-26T12:45:59.251224+0000","last_became_active":"2024-07-26T12:43:26.765425+0000","last_became_peered":"2024-07-26T12:43:26.765425+0000","last_unstale":"2024-07-26T12:45:59.251224+0000","last_undegraded":"2024-07-26T12:45:59.251224+0000","last_fullsized":"2024-07-26T12:45:59.251224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:16.786604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.645058+0000","last_change":"2024-07-26T12:43:12.171400+0000","last_active":"2024-07-26T12:46:34.645058+0000","last_peered":"2024-07-26T12:46:34.645058+0000","last_clean":"2024-07-26T12:46:34.645058+0000","last_became_active":"2024-07-26T12:43:12.170998+0000","last_became_peered":"2024-07-26T12:43:12.170998+0000","last_unstale":"2024-07-26T12:46:34.645058+0000","last_undegraded":"2024-07-26T12:46:34.645058+0000","last_fullsized":"2024-07-26T12:46:34.645058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:43.251975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656729+0000","last_change":"2024-07-26T12:43:35.912729+0000","last_active":"2024-07-26T12:46:34.656729+0000","last_peered":"2024-07-26T12:46:34.656729+0000","last_clean":"2024-07-26T12:46:34.656729+0000","last_became_active":"2024-07-26T12:43:35.912318+0000","last_became_peered":"2024-07-26T12:43:35.912318+0000","last_unstale":"2024-07-26T12:46:34.656729+0000","last_undegraded":"2024-07-26T12:46:34.656729+0000","last_fullsized":"2024-07-26T12:46:34.656729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:14.473536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661325+0000","last_change":"2024-07-26T12:43:12.182933+0000","last_active":"2024-07-26T12:46:34.661325+0000","last_peered":"2024-07-26T12:46:34.661325+0000","last_clean":"2024-07-26T12:46:34.661325+0000","last_became_active":"2024-07-26T12:43:12.182656+0000","last_became_peered":"2024-07-26T12:43:12.182656+0000","last_unstale":"2024-07-26T12:46:34.661325+0000","last_undegraded":"2024-07-26T12:46:34.661325+0000","last_fullsized":"2024-07-26T12:46:34.661325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:05:59.717662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.675425+0000","last_change":"2024-07-26T12:43:12.181196+0000","last_active":"2024-07-26T12:46:34.675425+0000","last_peered":"2024-07-26T12:46:34.675425+0000","last_clean":"2024-07-26T12:46:34.675425+0000","last_became_active":"2024-07-26T12:43:12.180765+0000","last_became_peered":"2024-07-26T12:43:12.180765+0000","last_unstale":"2024-07-26T12:46:34.675425+0000","last_undegraded":"2024-07-26T12:46:34.675425+0000","last_fullsized":"2024-07-26T12:46:34.675425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:20.590023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671094+0000","last_change":"2024-07-26T12:43:12.185408+0000","last_active":"2024-07-26T12:46:34.671094+0000","last_peered":"2024-07-26T12:46:34.671094+0000","last_clean":"2024-07-26T12:46:34.671094+0000","last_became_active":"2024-07-26T12:43:12.185038+0000","last_became_peered":"2024-07-26T12:43:12.185038+0000","last_unstale":"2024-07-26T12:46:34.671094+0000","last_undegraded":"2024-07-26T12:46:34.671094+0000","last_fullsized":"2024-07-26T12:46:34.671094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:55:30.604995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644437+0000","last_change":"2024-07-26T12:43:12.169301+0000","last_active":"2024-07-26T12:46:34.644437+0000","last_peered":"2024-07-26T12:46:34.644437+0000","last_clean":"2024-07-26T12:46:34.644437+0000","last_became_active":"2024-07-26T12:43:12.168923+0000","last_became_peered":"2024-07-26T12:43:12.168923+0000","last_unstale":"2024-07-26T12:46:34.644437+0000","last_undegraded":"2024-07-26T12:46:34.644437+0000","last_fullsized":"2024-07-26T12:46:34.644437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:04.506485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.647538+0000","last_change":"2024-07-26T12:43:12.166663+0000","last_active":"2024-07-26T12:46:34.647538+0000","last_peered":"2024-07-26T12:46:34.647538+0000","last_clean":"2024-07-26T12:46:34.647538+0000","last_became_active":"2024-07-26T12:43:12.166200+0000","last_became_peered":"2024-07-26T12:43:12.166200+0000","last_unstale":"2024-07-26T12:46:34.647538+0000","last_undegraded":"2024-07-26T12:46:34.647538+0000","last_fullsized":"2024-07-26T12:46:34.647538+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:14.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664657+0000","last_change":"2024-07-26T12:43:12.171273+0000","last_active":"2024-07-26T12:46:34.664657+0000","last_peered":"2024-07-26T12:46:34.664657+0000","last_clean":"2024-07-26T12:46:34.664657+0000","last_became_active":"2024-07-26T12:43:12.167680+0000","last_became_peered":"2024-07-26T12:43:12.167680+0000","last_unstale":"2024-07-26T12:46:34.664657+0000","last_undegraded":"2024-07-26T12:46:34.664657+0000","last_fullsized":"2024-07-26T12:46:34.664657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:44.797644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671048+0000","last_change":"2024-07-26T12:43:12.179891+0000","last_active":"2024-07-26T12:46:34.671048+0000","last_peered":"2024-07-26T12:46:34.671048+0000","last_clean":"2024-07-26T12:46:34.671048+0000","last_became_active":"2024-07-26T12:43:12.179518+0000","last_became_peered":"2024-07-26T12:43:12.179518+0000","last_unstale":"2024-07-26T12:46:34.671048+0000","last_undegraded":"2024-07-26T12:46:34.671048+0000","last_fullsized":"2024-07-26T12:46:34.671048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:24.246023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671055+0000","last_change":"2024-07-26T12:43:12.168521+0000","last_active":"2024-07-26T12:46:34.671055+0000","last_peered":"2024-07-26T12:46:34.671055+0000","last_clean":"2024-07-26T12:46:34.671055+0000","last_became_active":"2024-07-26T12:43:12.168126+0000","last_became_peered":"2024-07-26T12:43:12.168126+0000","last_unstale":"2024-07-26T12:46:34.671055+0000","last_undegraded":"2024-07-26T12:46:34.671055+0000","last_fullsized":"2024-07-26T12:46:34.671055+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:00.135090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661599+0000","last_change":"2024-07-26T12:44:36.843322+0000","last_active":"2024-07-26T12:46:34.661599+0000","last_peered":"2024-07-26T12:46:34.661599+0000","last_clean":"2024-07-26T12:46:34.661599+0000","last_became_active":"2024-07-26T12:44:36.842821+0000","last_became_peered":"2024-07-26T12:44:36.842821+0000","last_unstale":"2024-07-26T12:46:34.661599+0000","last_undegraded":"2024-07-26T12:46:34.661599+0000","last_fullsized":"2024-07-26T12:46:34.661599+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:05.609298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.675045+0000","last_change":"2024-07-26T12:43:12.185565+0000","last_active":"2024-07-26T12:46:34.675045+0000","last_peered":"2024-07-26T12:46:34.675045+0000","last_clean":"2024-07-26T12:46:34.675045+0000","last_became_active":"2024-07-26T12:43:12.185057+0000","last_became_peered":"2024-07-26T12:43:12.185057+0000","last_unstale":"2024-07-26T12:46:34.675045+0000","last_undegraded":"2024-07-26T12:46:34.675045+0000","last_fullsized":"2024-07-26T12:46:34.675045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:58.108311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644690+0000","last_change":"2024-07-26T12:43:12.173301+0000","last_active":"2024-07-26T12:46:34.644690+0000","last_peered":"2024-07-26T12:46:34.644690+0000","last_clean":"2024-07-26T12:46:34.644690+0000","last_became_active":"2024-07-26T12:43:12.172955+0000","last_became_peered":"2024-07-26T12:43:12.172955+0000","last_unstale":"2024-07-26T12:46:34.644690+0000","last_undegraded":"2024-07-26T12:46:34.644690+0000","last_fullsized":"2024-07-26T12:46:34.644690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:41.731141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661490+0000","last_change":"2024-07-26T12:43:12.176169+0000","last_active":"2024-07-26T12:46:34.661490+0000","last_peered":"2024-07-26T12:46:34.661490+0000","last_clean":"2024-07-26T12:46:34.661490+0000","last_became_active":"2024-07-26T12:43:12.175745+0000","last_became_peered":"2024-07-26T12:43:12.175745+0000","last_unstale":"2024-07-26T12:46:34.661490+0000","last_undegraded":"2024-07-26T12:46:34.661490+0000","last_fullsized":"2024-07-26T12:46:34.661490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:20.393207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.654907+0000","last_change":"2024-07-26T12:43:12.168182+0000","last_active":"2024-07-26T12:46:34.654907+0000","last_peered":"2024-07-26T12:46:34.654907+0000","last_clean":"2024-07-26T12:46:34.654907+0000","last_became_active":"2024-07-26T12:43:12.163771+0000","last_became_peered":"2024-07-26T12:43:12.163771+0000","last_unstale":"2024-07-26T12:46:34.654907+0000","last_undegraded":"2024-07-26T12:46:34.654907+0000","last_fullsized":"2024-07-26T12:46:34.654907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:40:30.960153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.660143+0000","last_change":"2024-07-26T12:43:12.189524+0000","last_active":"2024-07-26T12:46:34.660143+0000","last_peered":"2024-07-26T12:46:34.660143+0000","last_clean":"2024-07-26T12:46:34.660143+0000","last_became_active":"2024-07-26T12:43:12.189165+0000","last_became_peered":"2024-07-26T12:43:12.189165+0000","last_unstale":"2024-07-26T12:46:34.660143+0000","last_undegraded":"2024-07-26T12:46:34.660143+0000","last_fullsized":"2024-07-26T12:46:34.660143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:11:08.479058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.665412+0000","last_change":"2024-07-26T12:43:12.180603+0000","last_active":"2024-07-26T12:46:34.665412+0000","last_peered":"2024-07-26T12:46:34.665412+0000","last_clean":"2024-07-26T12:46:34.665412+0000","last_became_active":"2024-07-26T12:43:12.180204+0000","last_became_peered":"2024-07-26T12:43:12.180204+0000","last_unstale":"2024-07-26T12:46:34.665412+0000","last_undegraded":"2024-07-26T12:46:34.665412+0000","last_fullsized":"2024-07-26T12:46:34.665412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:46.768213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668524+0000","last_change":"2024-07-26T12:43:12.185347+0000","last_active":"2024-07-26T12:46:34.668524+0000","last_peered":"2024-07-26T12:46:34.668524+0000","last_clean":"2024-07-26T12:46:34.668524+0000","last_became_active":"2024-07-26T12:43:12.180190+0000","last_became_peered":"2024-07-26T12:43:12.180190+0000","last_unstale":"2024-07-26T12:46:34.668524+0000","last_undegraded":"2024-07-26T12:46:34.668524+0000","last_fullsized":"2024-07-26T12:46:34.668524+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:07.654967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.667916+0000","last_change":"2024-07-26T12:43:12.177258+0000","last_active":"2024-07-26T12:46:34.667916+0000","last_peered":"2024-07-26T12:46:34.667916+0000","last_clean":"2024-07-26T12:46:34.667916+0000","last_became_active":"2024-07-26T12:43:12.176879+0000","last_became_peered":"2024-07-26T12:43:12.176879+0000","last_unstale":"2024-07-26T12:46:34.667916+0000","last_undegraded":"2024-07-26T12:46:34.667916+0000","last_fullsized":"2024-07-26T12:46:34.667916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:24.244759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.675740+0000","last_change":"2024-07-26T12:44:36.857403+0000","last_active":"2024-07-26T12:46:34.675740+0000","last_peered":"2024-07-26T12:46:34.675740+0000","last_clean":"2024-07-26T12:46:34.675740+0000","last_became_active":"2024-07-26T12:44:36.856980+0000","last_became_peered":"2024-07-26T12:44:36.856980+0000","last_unstale":"2024-07-26T12:46:34.675740+0000","last_undegraded":"2024-07-26T12:46:34.675740+0000","last_fullsized":"2024-07-26T12:46:34.675740+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.166383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.653669+0000","last_change":"2024-07-26T12:43:12.166827+0000","last_active":"2024-07-26T12:46:34.653669+0000","last_peered":"2024-07-26T12:46:34.653669+0000","last_clean":"2024-07-26T12:46:34.653669+0000","last_became_active":"2024-07-26T12:43:12.165222+0000","last_became_peered":"2024-07-26T12:43:12.165222+0000","last_unstale":"2024-07-26T12:46:34.653669+0000","last_undegraded":"2024-07-26T12:46:34.653669+0000","last_fullsized":"2024-07-26T12:46:34.653669+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:37:42.219844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.663059+0000","last_change":"2024-07-26T12:43:12.176961+0000","last_active":"2024-07-26T12:46:34.663059+0000","last_peered":"2024-07-26T12:46:34.663059+0000","last_clean":"2024-07-26T12:46:34.663059+0000","last_became_active":"2024-07-26T12:43:12.176550+0000","last_became_peered":"2024-07-26T12:43:12.176550+0000","last_unstale":"2024-07-26T12:46:34.663059+0000","last_undegraded":"2024-07-26T12:46:34.663059+0000","last_fullsized":"2024-07-26T12:46:34.663059+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:36:19.720801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650547+0000","last_change":"2024-07-26T12:43:12.173560+0000","last_active":"2024-07-26T12:46:34.650547+0000","last_peered":"2024-07-26T12:46:34.650547+0000","last_clean":"2024-07-26T12:46:34.650547+0000","last_became_active":"2024-07-26T12:43:12.173187+0000","last_became_peered":"2024-07-26T12:43:12.173187+0000","last_unstale":"2024-07-26T12:46:34.650547+0000","last_undegraded":"2024-07-26T12:46:34.650547+0000","last_fullsized":"2024-07-26T12:46:34.650547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:53.929210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655516+0000","last_change":"2024-07-26T12:43:12.169829+0000","last_active":"2024-07-26T12:46:34.655516+0000","last_peered":"2024-07-26T12:46:34.655516+0000","last_clean":"2024-07-26T12:46:34.655516+0000","last_became_active":"2024-07-26T12:43:12.169444+0000","last_became_peered":"2024-07-26T12:43:12.169444+0000","last_unstale":"2024-07-26T12:46:34.655516+0000","last_undegraded":"2024-07-26T12:46:34.655516+0000","last_fullsized":"2024-07-26T12:46:34.655516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:29.111483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671995+0000","last_change":"2024-07-26T12:43:12.171857+0000","last_active":"2024-07-26T12:46:34.671995+0000","last_peered":"2024-07-26T12:46:34.671995+0000","last_clean":"2024-07-26T12:46:34.671995+0000","last_became_active":"2024-07-26T12:43:12.170752+0000","last_became_peered":"2024-07-26T12:43:12.170752+0000","last_unstale":"2024-07-26T12:46:34.671995+0000","last_undegraded":"2024-07-26T12:46:34.671995+0000","last_fullsized":"2024-07-26T12:46:34.671995+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:22:09.442916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.657139+0000","last_change":"2024-07-26T12:43:12.178646+0000","last_active":"2024-07-26T12:46:34.657139+0000","last_peered":"2024-07-26T12:46:34.657139+0000","last_clean":"2024-07-26T12:46:34.657139+0000","last_became_active":"2024-07-26T12:43:12.178259+0000","last_became_peered":"2024-07-26T12:43:12.178259+0000","last_unstale":"2024-07-26T12:46:34.657139+0000","last_undegraded":"2024-07-26T12:46:34.657139+0000","last_fullsized":"2024-07-26T12:46:34.657139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:30:59.417843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655755+0000","last_change":"2024-07-26T12:43:12.164579+0000","last_active":"2024-07-26T12:46:34.655755+0000","last_peered":"2024-07-26T12:46:34.655755+0000","last_clean":"2024-07-26T12:46:34.655755+0000","last_became_active":"2024-07-26T12:43:12.164099+0000","last_became_peered":"2024-07-26T12:43:12.164099+0000","last_unstale":"2024-07-26T12:46:34.655755+0000","last_undegraded":"2024-07-26T12:46:34.655755+0000","last_fullsized":"2024-07-26T12:46:34.655755+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:51:45.293190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664174+0000","last_change":"2024-07-26T12:43:12.180838+0000","last_active":"2024-07-26T12:46:34.664174+0000","last_peered":"2024-07-26T12:46:34.664174+0000","last_clean":"2024-07-26T12:46:34.664174+0000","last_became_active":"2024-07-26T12:43:12.180543+0000","last_became_peered":"2024-07-26T12:43:12.180543+0000","last_unstale":"2024-07-26T12:46:34.664174+0000","last_undegraded":"2024-07-26T12:46:34.664174+0000","last_fullsized":"2024-07-26T12:46:34.664174+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:27.185082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650139+0000","last_change":"2024-07-26T12:43:12.167235+0000","last_active":"2024-07-26T12:46:34.650139+0000","last_peered":"2024-07-26T12:46:34.650139+0000","last_clean":"2024-07-26T12:46:34.650139+0000","last_became_active":"2024-07-26T12:43:12.166489+0000","last_became_peered":"2024-07-26T12:43:12.166489+0000","last_unstale":"2024-07-26T12:46:34.650139+0000","last_undegraded":"2024-07-26T12:46:34.650139+0000","last_fullsized":"2024-07-26T12:46:34.650139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:13.868870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668862+0000","last_change":"2024-07-26T12:43:12.176717+0000","last_active":"2024-07-26T12:46:34.668862+0000","last_peered":"2024-07-26T12:46:34.668862+0000","last_clean":"2024-07-26T12:46:34.668862+0000","last_became_active":"2024-07-26T12:43:12.172699+0000","last_became_peered":"2024-07-26T12:43:12.172699+0000","last_unstale":"2024-07-26T12:46:34.668862+0000","last_undegraded":"2024-07-26T12:46:34.668862+0000","last_fullsized":"2024-07-26T12:46:34.668862+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:23:36.853999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669127+0000","last_change":"2024-07-26T12:43:12.177093+0000","last_active":"2024-07-26T12:46:34.669127+0000","last_peered":"2024-07-26T12:46:34.669127+0000","last_clean":"2024-07-26T12:46:34.669127+0000","last_became_active":"2024-07-26T12:43:12.176693+0000","last_became_peered":"2024-07-26T12:43:12.176693+0000","last_unstale":"2024-07-26T12:46:34.669127+0000","last_undegraded":"2024-07-26T12:46:34.669127+0000","last_fullsized":"2024-07-26T12:46:34.669127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:56.875143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.660186+0000","last_change":"2024-07-26T12:43:12.189531+0000","last_active":"2024-07-26T12:46:34.660186+0000","last_peered":"2024-07-26T12:46:34.660186+0000","last_clean":"2024-07-26T12:46:34.660186+0000","last_became_active":"2024-07-26T12:43:12.189162+0000","last_became_peered":"2024-07-26T12:43:12.189162+0000","last_unstale":"2024-07-26T12:46:34.660186+0000","last_undegraded":"2024-07-26T12:46:34.660186+0000","last_fullsized":"2024-07-26T12:46:34.660186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:11:00.160614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77551,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2475,"num_read_kb":1322,"num_write":265,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585728,"data_stored":227280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678098,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":536466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874328,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1744896,"data_stored":1187374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037851,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":539892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234082,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":536174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430311,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36964,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708188,"statfs":{"total":95995035648,"available":95957184512,"internally_reserved":0,"allocated":1667072,"data_stored":1132812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593837,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":531300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790068,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1128394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953592,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":535374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":531008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378755,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":603611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542281,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36428,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708724,"statfs":{"total":95995035648,"available":95957733376,"internally_reserved":0,"allocated":1118208,"data_stored":597022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":533288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":66444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":73033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":71801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T12:48:46.873 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-07-26T12:48:46.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T12:48:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.180+0000 7fc95b200640 1 -- 172.21.15.107:0/2066148067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc95c105530 msgr2=0x7fc95c105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.180+0000 7fc95b200640 1 --2- 172.21.15.107:0/2066148067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc95c105530 0x7fc95c105940 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc94c009900 tx=0x7fc94c02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:48:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.180+0000 7fc95b200640 1 -- 172.21.15.107:0/2066148067 shutdown_connections 2024-07-26T12:48:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.180+0000 7fc95b200640 1 --2- 172.21.15.107:0/2066148067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc95c10b7b0 0x7fc95c111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.180+0000 7fc95b200640 1 --2- 172.21.15.107:0/2066148067 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc95c10ae60 0x7fc95c10b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.180+0000 7fc95b200640 1 --2- 172.21.15.107:0/2066148067 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc95c105530 0x7fc95c105940 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.180+0000 7fc95b200640 1 -- 172.21.15.107:0/2066148067 >> 172.21.15.107:0/2066148067 conn(0x7fc95c100f90 msgr2=0x7fc95c1033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:47.181 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.180+0000 7fc95b200640 1 -- 172.21.15.107:0/2066148067 shutdown_connections 2024-07-26T12:48:47.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.180+0000 7fc95b200640 1 -- 172.21.15.107:0/2066148067 wait complete. 2024-07-26T12:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.181+0000 7fc95b200640 1 Processor -- start 2024-07-26T12:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.181+0000 7fc95b200640 1 -- start start 2024-07-26T12:48:47.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc95b200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc95c105530 0x7fc95c19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc95b200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc95c10ae60 0x7fc95c19c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc95b200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc95c10b7b0 0x7fc95c1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc95b200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c1154b0 con 0x7fc95c105530 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc95b200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c115330 con 0x7fc95c10b7b0 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc95b200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c115630 con 0x7fc95c10ae60 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc959e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc95c105530 0x7fc95c19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc959e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc95c105530 0x7fc95c19bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33442/0 (socket says 172.21.15.107:33442) 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc959e00640 1 -- 172.21.15.107:0/718436717 learned_addr learned my addr 172.21.15.107:0/718436717 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc95a800640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc95c10b7b0 0x7fc95c1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.182+0000 7fc951400640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc95c10ae60 0x7fc95c19c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc959e00640 1 -- 172.21.15.107:0/718436717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc95c10ae60 msgr2=0x7fc95c19c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc959e00640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc95c10ae60 0x7fc95c19c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:47.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc959e00640 1 -- 172.21.15.107:0/718436717 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc95c10b7b0 msgr2=0x7fc95c1a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc959e00640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc95c10b7b0 0x7fc95c1a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc959e00640 1 -- 172.21.15.107:0/718436717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc95c1a1e20 con 0x7fc95c105530 2024-07-26T12:48:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc951400640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc95c10ae60 0x7fc95c19c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:48:47.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc959e00640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc95c105530 0x7fc95c19bef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc94c005ab0 tx=0x7fc94c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc953400640 1 -- 172.21.15.107:0/718436717 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c03f070 con 0x7fc95c105530 2024-07-26T12:48:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc953400640 1 -- 172.21.15.107:0/718436717 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc94c031b60 con 0x7fc95c105530 2024-07-26T12:48:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc95b200640 1 -- 172.21.15.107:0/718436717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc95c1a20b0 con 0x7fc95c105530 2024-07-26T12:48:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc953400640 1 -- 172.21.15.107:0/718436717 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c035030 con 0x7fc95c105530 2024-07-26T12:48:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.183+0000 7fc95b200640 1 -- 172.21.15.107:0/718436717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc95c1a88a0 con 0x7fc95c105530 2024-07-26T12:48:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.184+0000 7fc95b200640 1 -- 172.21.15.107:0/718436717 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc95c105940 con 0x7fc95c105530 2024-07-26T12:48:47.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.187+0000 7fc953400640 1 -- 172.21.15.107:0/718436717 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fc94c042070 con 0x7fc95c105530 2024-07-26T12:48:47.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.189+0000 7fc953400640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9400b3c20 0x7fc9400b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:47.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.189+0000 7fc953400640 1 -- 172.21.15.107:0/718436717 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc94c104f90 con 0x7fc95c105530 2024-07-26T12:48:47.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.189+0000 7fc951400640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9400b3c20 0x7fc9400b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:47.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.190+0000 7fc951400640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9400b3c20 0x7fc9400b6130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc95c10aa10 tx=0x7fc944005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:47.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.191+0000 7fc953400640 1 -- 172.21.15.107:0/718436717 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc94c0cc0e0 con 0x7fc95c105530 2024-07-26T12:48:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:46 smithi112 ceph-mon[37501]: from='client.25459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:48:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:46 smithi112 ceph-mon[37501]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:48:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:46 smithi112 ceph-mon[37501]: pgmap v474: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 766 B/s wr, 9 op/s 2024-07-26T12:48:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:46 smithi107 ceph-mon[32659]: from='client.25459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:48:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:46 smithi107 ceph-mon[32659]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:48:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:46 smithi107 ceph-mon[32659]: pgmap v474: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 766 B/s wr, 9 op/s 2024-07-26T12:48:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:46 smithi132 ceph-mon[37580]: from='client.25459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:48:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:46 smithi132 ceph-mon[37580]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:48:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:46 smithi132 ceph-mon[37580]: pgmap v474: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 766 B/s wr, 9 op/s 2024-07-26T12:48:47.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.398+0000 7fc95b200640 1 -- 172.21.15.107:0/718436717 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc95c10d510 con 0x7fc9400b3c20 2024-07-26T12:48:47.416 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.414+0000 7fc953400640 1 -- 172.21.15.107:0/718436717 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340519 (secure 0 0 0) 0x7fc95c10d510 con 0x7fc9400b3c20 2024-07-26T12:48:47.417 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:48:47.422 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-07-26T12:48:47.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.421+0000 7fc95b200640 1 -- 172.21.15.107:0/718436717 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9400b3c20 msgr2=0x7fc9400b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.421+0000 7fc95b200640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9400b3c20 0x7fc9400b6130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc95c10aa10 tx=0x7fc944005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.421+0000 7fc95b200640 1 -- 172.21.15.107:0/718436717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc95c105530 msgr2=0x7fc95c19bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.421+0000 7fc95b200640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc95c105530 0x7fc95c19bef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fc94c005ab0 tx=0x7fc94c003c10 comp rx=0 tx=0).stop 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.421+0000 7fc95b200640 1 -- 172.21.15.107:0/718436717 shutdown_connections 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.421+0000 7fc95b200640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc9400b3c20 0x7fc9400b6130 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.421+0000 7fc95b200640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc95c10b7b0 0x7fc95c1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.421+0000 7fc95b200640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc95c10ae60 0x7fc95c19c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.421+0000 7fc95b200640 1 --2- 172.21.15.107:0/718436717 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc95c105530 0x7fc95c19bef0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.421+0000 7fc95b200640 1 -- 172.21.15.107:0/718436717 >> 172.21.15.107:0/718436717 conn(0x7fc95c100f90 msgr2=0x7fc95c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.422+0000 7fc95b200640 1 -- 172.21.15.107:0/718436717 shutdown_connections 2024-07-26T12:48:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:47.422+0000 7fc95b200640 1 -- 172.21.15.107:0/718436717 wait complete. 2024-07-26T12:48:47.452 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-07-26T12:48:46.829844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":668153,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2566,"num_read_kb":1396,"num_write":390,"num_write_kb":2312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2951874,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":822,"ondisk_log_size":822,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":376072,"kb_used_data":14440,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565752,"statfs":{"total":1151940427776,"available":1151555330048,"internally_reserved":0,"allocated":14786560,"data_stored":8398088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":370193721},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5680,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":62,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.017072"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":307,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:44.380039+0000","last_change":"2024-07-26T12:43:08.129417+0000","last_active":"2024-07-26T12:48:44.380039+0000","last_peered":"2024-07-26T12:48:44.380039+0000","last_clean":"2024-07-26T12:48:44.380039+0000","last_became_active":"2024-07-26T12:43:08.128919+0000","last_became_peered":"2024-07-26T12:43:08.128919+0000","last_unstale":"2024-07-26T12:48:44.380039+0000","last_undegraded":"2024-07-26T12:48:44.380039+0000","last_fullsized":"2024-07-26T12:48:44.380039+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:35:55.414185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":247,"num_read_kb":247,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.658083+0000","last_change":"2024-07-26T12:43:12.172058+0000","last_active":"2024-07-26T12:46:34.658083+0000","last_peered":"2024-07-26T12:46:34.658083+0000","last_clean":"2024-07-26T12:46:34.658083+0000","last_became_active":"2024-07-26T12:43:12.171631+0000","last_became_peered":"2024-07-26T12:43:12.171631+0000","last_unstale":"2024-07-26T12:46:34.658083+0000","last_undegraded":"2024-07-26T12:46:34.658083+0000","last_fullsized":"2024-07-26T12:46:34.658083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:51.972962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:54.250635+0000","last_change":"2024-07-26T12:43:26.773739+0000","last_active":"2024-07-26T12:45:54.250635+0000","last_peered":"2024-07-26T12:45:54.250635+0000","last_clean":"2024-07-26T12:45:54.250635+0000","last_became_active":"2024-07-26T12:43:26.773254+0000","last_became_peered":"2024-07-26T12:43:26.773254+0000","last_unstale":"2024-07-26T12:45:54.250635+0000","last_undegraded":"2024-07-26T12:45:54.250635+0000","last_fullsized":"2024-07-26T12:45:54.250635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:03.125497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:11.254691+0000","last_change":"2024-07-26T12:43:08.227329+0000","last_active":"2024-07-26T12:45:11.254691+0000","last_peered":"2024-07-26T12:45:11.254691+0000","last_clean":"2024-07-26T12:45:11.254691+0000","last_became_active":"2024-07-26T12:43:08.226674+0000","last_became_peered":"2024-07-26T12:43:08.226674+0000","last_unstale":"2024-07-26T12:45:11.254691+0000","last_undegraded":"2024-07-26T12:45:11.254691+0000","last_fullsized":"2024-07-26T12:45:11.254691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.222684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.674670+0000","last_change":"2024-07-26T12:43:12.190616+0000","last_active":"2024-07-26T12:46:34.674670+0000","last_peered":"2024-07-26T12:46:34.674670+0000","last_clean":"2024-07-26T12:46:34.674670+0000","last_became_active":"2024-07-26T12:43:12.189717+0000","last_became_peered":"2024-07-26T12:43:12.189717+0000","last_unstale":"2024-07-26T12:46:34.674670+0000","last_undegraded":"2024-07-26T12:46:34.674670+0000","last_fullsized":"2024-07-26T12:46:34.674670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:03.509592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:51.766027+0000","last_change":"2024-07-26T12:43:26.774486+0000","last_active":"2024-07-26T12:45:51.766027+0000","last_peered":"2024-07-26T12:45:51.766027+0000","last_clean":"2024-07-26T12:45:51.766027+0000","last_became_active":"2024-07-26T12:43:26.774046+0000","last_became_peered":"2024-07-26T12:43:26.774046+0000","last_unstale":"2024-07-26T12:45:51.766027+0000","last_undegraded":"2024-07-26T12:45:51.766027+0000","last_fullsized":"2024-07-26T12:45:51.766027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.847072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:37.304088+0000","last_change":"2024-07-26T12:43:08.123084+0000","last_active":"2024-07-26T12:45:37.304088+0000","last_peered":"2024-07-26T12:45:37.304088+0000","last_clean":"2024-07-26T12:45:37.304088+0000","last_became_active":"2024-07-26T12:43:08.122371+0000","last_became_peered":"2024-07-26T12:43:08.122371+0000","last_unstale":"2024-07-26T12:45:37.304088+0000","last_undegraded":"2024-07-26T12:45:37.304088+0000","last_fullsized":"2024-07-26T12:45:37.304088+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:27:09.665660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672192+0000","last_change":"2024-07-26T12:43:12.170290+0000","last_active":"2024-07-26T12:46:34.672192+0000","last_peered":"2024-07-26T12:46:34.672192+0000","last_clean":"2024-07-26T12:46:34.672192+0000","last_became_active":"2024-07-26T12:43:12.169849+0000","last_became_peered":"2024-07-26T12:43:12.169849+0000","last_unstale":"2024-07-26T12:46:34.672192+0000","last_undegraded":"2024-07-26T12:46:34.672192+0000","last_fullsized":"2024-07-26T12:46:34.672192+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:46.044921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:56.766737+0000","last_change":"2024-07-26T12:43:26.764804+0000","last_active":"2024-07-26T12:45:56.766737+0000","last_peered":"2024-07-26T12:45:56.766737+0000","last_clean":"2024-07-26T12:45:56.766737+0000","last_became_active":"2024-07-26T12:43:26.764065+0000","last_became_peered":"2024-07-26T12:43:26.764065+0000","last_unstale":"2024-07-26T12:45:56.766737+0000","last_undegraded":"2024-07-26T12:45:56.766737+0000","last_fullsized":"2024-07-26T12:45:56.766737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:16:59.060110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:37.304036+0000","last_change":"2024-07-26T12:43:08.121496+0000","last_active":"2024-07-26T12:45:37.304036+0000","last_peered":"2024-07-26T12:45:37.304036+0000","last_clean":"2024-07-26T12:45:37.304036+0000","last_became_active":"2024-07-26T12:43:08.120962+0000","last_became_peered":"2024-07-26T12:43:08.120962+0000","last_unstale":"2024-07-26T12:45:37.304036+0000","last_undegraded":"2024-07-26T12:45:37.304036+0000","last_fullsized":"2024-07-26T12:45:37.304036+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:42:02.998023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655232+0000","last_change":"2024-07-26T12:43:12.168707+0000","last_active":"2024-07-26T12:46:34.655232+0000","last_peered":"2024-07-26T12:46:34.655232+0000","last_clean":"2024-07-26T12:46:34.655232+0000","last_became_active":"2024-07-26T12:43:12.168338+0000","last_became_peered":"2024-07-26T12:43:12.168338+0000","last_unstale":"2024-07-26T12:46:34.655232+0000","last_undegraded":"2024-07-26T12:46:34.655232+0000","last_fullsized":"2024-07-26T12:46:34.655232+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:39:44.724864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:56.766714+0000","last_change":"2024-07-26T12:43:26.770863+0000","last_active":"2024-07-26T12:45:56.766714+0000","last_peered":"2024-07-26T12:45:56.766714+0000","last_clean":"2024-07-26T12:45:56.766714+0000","last_became_active":"2024-07-26T12:43:26.770480+0000","last_became_peered":"2024-07-26T12:43:26.770480+0000","last_unstale":"2024-07-26T12:45:56.766714+0000","last_undegraded":"2024-07-26T12:45:56.766714+0000","last_fullsized":"2024-07-26T12:45:56.766714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:09.128364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672691+0000","last_change":"2024-07-26T12:43:12.181659+0000","last_active":"2024-07-26T12:46:34.672691+0000","last_peered":"2024-07-26T12:46:34.672691+0000","last_clean":"2024-07-26T12:46:34.672691+0000","last_became_active":"2024-07-26T12:43:12.181268+0000","last_became_peered":"2024-07-26T12:43:12.181268+0000","last_unstale":"2024-07-26T12:46:34.672691+0000","last_undegraded":"2024-07-26T12:46:34.672691+0000","last_fullsized":"2024-07-26T12:46:34.672691+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:38:28.445981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:51.540084+0000","last_change":"2024-07-26T12:43:35.856273+0000","last_active":"2024-07-26T12:45:51.540084+0000","last_peered":"2024-07-26T12:45:51.540084+0000","last_clean":"2024-07-26T12:45:51.540084+0000","last_became_active":"2024-07-26T12:43:35.849625+0000","last_became_peered":"2024-07-26T12:43:35.849625+0000","last_unstale":"2024-07-26T12:45:51.540084+0000","last_undegraded":"2024-07-26T12:45:51.540084+0000","last_fullsized":"2024-07-26T12:45:51.540084+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:34:45.261474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:41.764778+0000","last_change":"2024-07-26T12:43:26.793431+0000","last_active":"2024-07-26T12:45:41.764778+0000","last_peered":"2024-07-26T12:45:41.764778+0000","last_clean":"2024-07-26T12:45:41.764778+0000","last_became_active":"2024-07-26T12:43:26.792931+0000","last_became_peered":"2024-07-26T12:43:26.792931+0000","last_unstale":"2024-07-26T12:45:41.764778+0000","last_undegraded":"2024-07-26T12:45:41.764778+0000","last_fullsized":"2024-07-26T12:45:41.764778+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:31.121847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669969+0000","last_change":"2024-07-26T12:43:12.183784+0000","last_active":"2024-07-26T12:46:34.669969+0000","last_peered":"2024-07-26T12:46:34.669969+0000","last_clean":"2024-07-26T12:46:34.669969+0000","last_became_active":"2024-07-26T12:43:12.182696+0000","last_became_peered":"2024-07-26T12:43:12.182696+0000","last_unstale":"2024-07-26T12:46:34.669969+0000","last_undegraded":"2024-07-26T12:46:34.669969+0000","last_fullsized":"2024-07-26T12:46:34.669969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:33:19.918180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:19.245986+0000","last_change":"2024-07-26T12:43:08.131069+0000","last_active":"2024-07-26T12:45:19.245986+0000","last_peered":"2024-07-26T12:45:19.245986+0000","last_clean":"2024-07-26T12:45:19.245986+0000","last_became_active":"2024-07-26T12:43:08.130637+0000","last_became_peered":"2024-07-26T12:43:08.130637+0000","last_unstale":"2024-07-26T12:45:19.245986+0000","last_undegraded":"2024-07-26T12:45:19.245986+0000","last_fullsized":"2024-07-26T12:45:19.245986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:04:16.461900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:41.764748+0000","last_change":"2024-07-26T12:43:26.763388+0000","last_active":"2024-07-26T12:45:41.764748+0000","last_peered":"2024-07-26T12:45:41.764748+0000","last_clean":"2024-07-26T12:45:41.764748+0000","last_became_active":"2024-07-26T12:43:26.762464+0000","last_became_peered":"2024-07-26T12:43:26.762464+0000","last_unstale":"2024-07-26T12:45:41.764748+0000","last_undegraded":"2024-07-26T12:45:41.764748+0000","last_fullsized":"2024-07-26T12:45:41.764748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:10:17.598266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668869+0000","last_change":"2024-07-26T12:43:12.191640+0000","last_active":"2024-07-26T12:46:34.668869+0000","last_peered":"2024-07-26T12:46:34.668869+0000","last_clean":"2024-07-26T12:46:34.668869+0000","last_became_active":"2024-07-26T12:43:12.191260+0000","last_became_peered":"2024-07-26T12:43:12.191260+0000","last_unstale":"2024-07-26T12:46:34.668869+0000","last_undegraded":"2024-07-26T12:46:34.668869+0000","last_fullsized":"2024-07-26T12:46:34.668869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:47:18.929690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.257126+0000","last_change":"2024-07-26T12:43:08.135035+0000","last_active":"2024-07-26T12:45:21.257126+0000","last_peered":"2024-07-26T12:45:21.257126+0000","last_clean":"2024-07-26T12:45:21.257126+0000","last_became_active":"2024-07-26T12:43:08.134204+0000","last_became_peered":"2024-07-26T12:43:08.134204+0000","last_unstale":"2024-07-26T12:45:21.257126+0000","last_undegraded":"2024-07-26T12:45:21.257126+0000","last_fullsized":"2024-07-26T12:45:21.257126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:49:38.954460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:52.306230+0000","last_change":"2024-07-26T12:43:35.853244+0000","last_active":"2024-07-26T12:45:52.306230+0000","last_peered":"2024-07-26T12:45:52.306230+0000","last_clean":"2024-07-26T12:45:52.306230+0000","last_became_active":"2024-07-26T12:43:35.852728+0000","last_became_peered":"2024-07-26T12:43:35.852728+0000","last_unstale":"2024-07-26T12:45:52.306230+0000","last_undegraded":"2024-07-26T12:45:52.306230+0000","last_fullsized":"2024-07-26T12:45:52.306230+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:20:39.080000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.635080+0000","last_change":"2024-07-26T12:43:12.172457+0000","last_active":"2024-07-26T12:46:34.635080+0000","last_peered":"2024-07-26T12:46:34.635080+0000","last_clean":"2024-07-26T12:46:34.635080+0000","last_became_active":"2024-07-26T12:43:12.172034+0000","last_became_peered":"2024-07-26T12:43:12.172034+0000","last_unstale":"2024-07-26T12:46:34.635080+0000","last_undegraded":"2024-07-26T12:46:34.635080+0000","last_fullsized":"2024-07-26T12:46:34.635080+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:37:02.237852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.698668+0000","last_change":"2024-07-26T12:43:08.227152+0000","last_active":"2024-07-26T12:45:21.698668+0000","last_peered":"2024-07-26T12:45:21.698668+0000","last_clean":"2024-07-26T12:45:21.698668+0000","last_became_active":"2024-07-26T12:43:08.226722+0000","last_became_peered":"2024-07-26T12:43:08.226722+0000","last_unstale":"2024-07-26T12:45:21.698668+0000","last_undegraded":"2024-07-26T12:45:21.698668+0000","last_fullsized":"2024-07-26T12:45:21.698668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:16.293211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:41.764791+0000","last_change":"2024-07-26T12:43:26.792999+0000","last_active":"2024-07-26T12:45:41.764791+0000","last_peered":"2024-07-26T12:45:41.764791+0000","last_clean":"2024-07-26T12:45:41.764791+0000","last_became_active":"2024-07-26T12:43:26.792515+0000","last_became_peered":"2024-07-26T12:43:26.792515+0000","last_unstale":"2024-07-26T12:45:41.764791+0000","last_undegraded":"2024-07-26T12:45:41.764791+0000","last_fullsized":"2024-07-26T12:45:41.764791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:49:35.104600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.673277+0000","last_change":"2024-07-26T12:43:12.185381+0000","last_active":"2024-07-26T12:46:34.673277+0000","last_peered":"2024-07-26T12:46:34.673277+0000","last_clean":"2024-07-26T12:46:34.673277+0000","last_became_active":"2024-07-26T12:43:12.184892+0000","last_became_peered":"2024-07-26T12:43:12.184892+0000","last_unstale":"2024-07-26T12:46:34.673277+0000","last_undegraded":"2024-07-26T12:46:34.673277+0000","last_fullsized":"2024-07-26T12:46:34.673277+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:38:53.992586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:21.762689+0000","last_change":"2024-07-26T12:43:08.895079+0000","last_active":"2024-07-26T12:45:21.762689+0000","last_peered":"2024-07-26T12:45:21.762689+0000","last_clean":"2024-07-26T12:45:21.762689+0000","last_became_active":"2024-07-26T12:43:08.894653+0000","last_became_peered":"2024-07-26T12:43:08.894653+0000","last_unstale":"2024-07-26T12:45:21.762689+0000","last_undegraded":"2024-07-26T12:45:21.762689+0000","last_fullsized":"2024-07-26T12:45:21.762689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:02:53.137905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:34.247883+0000","last_change":"2024-07-26T12:43:26.762802+0000","last_active":"2024-07-26T12:45:34.247883+0000","last_peered":"2024-07-26T12:45:34.247883+0000","last_clean":"2024-07-26T12:45:34.247883+0000","last_became_active":"2024-07-26T12:43:26.762211+0000","last_became_peered":"2024-07-26T12:43:26.762211+0000","last_unstale":"2024-07-26T12:45:34.247883+0000","last_undegraded":"2024-07-26T12:45:34.247883+0000","last_fullsized":"2024-07-26T12:45:34.247883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:35:19.623168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662791+0000","last_change":"2024-07-26T12:43:12.176062+0000","last_active":"2024-07-26T12:46:34.662791+0000","last_peered":"2024-07-26T12:46:34.662791+0000","last_clean":"2024-07-26T12:46:34.662791+0000","last_became_active":"2024-07-26T12:43:12.175671+0000","last_became_peered":"2024-07-26T12:43:12.175671+0000","last_unstale":"2024-07-26T12:46:34.662791+0000","last_undegraded":"2024-07-26T12:46:34.662791+0000","last_fullsized":"2024-07-26T12:46:34.662791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:47:48.374964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'18","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:47:31.284996+0000","last_change":"2024-07-26T12:43:08.131453+0000","last_active":"2024-07-26T12:47:31.284996+0000","last_peered":"2024-07-26T12:47:31.284996+0000","last_clean":"2024-07-26T12:47:31.284996+0000","last_became_active":"2024-07-26T12:43:08.130928+0000","last_became_peered":"2024-07-26T12:43:08.130928+0000","last_unstale":"2024-07-26T12:47:31.284996+0000","last_undegraded":"2024-07-26T12:47:31.284996+0000","last_fullsized":"2024-07-26T12:47:31.284996+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:43.190860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:26.258119+0000","last_change":"2024-07-26T12:43:26.792942+0000","last_active":"2024-07-26T12:45:26.258119+0000","last_peered":"2024-07-26T12:45:26.258119+0000","last_clean":"2024-07-26T12:45:26.258119+0000","last_became_active":"2024-07-26T12:43:26.792045+0000","last_became_peered":"2024-07-26T12:43:26.792045+0000","last_unstale":"2024-07-26T12:45:26.258119+0000","last_undegraded":"2024-07-26T12:45:26.258119+0000","last_fullsized":"2024-07-26T12:45:26.258119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:31:56.005295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662936+0000","last_change":"2024-07-26T12:43:12.187601+0000","last_active":"2024-07-26T12:46:34.662936+0000","last_peered":"2024-07-26T12:46:34.662936+0000","last_clean":"2024-07-26T12:46:34.662936+0000","last_became_active":"2024-07-26T12:43:12.187175+0000","last_became_peered":"2024-07-26T12:43:12.187175+0000","last_unstale":"2024-07-26T12:46:34.662936+0000","last_undegraded":"2024-07-26T12:46:34.662936+0000","last_fullsized":"2024-07-26T12:46:34.662936+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:06.992854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:12.301049+0000","last_change":"2024-07-26T12:43:08.227279+0000","last_active":"2024-07-26T12:45:12.301049+0000","last_peered":"2024-07-26T12:45:12.301049+0000","last_clean":"2024-07-26T12:45:12.301049+0000","last_became_active":"2024-07-26T12:43:08.226634+0000","last_became_peered":"2024-07-26T12:43:08.226634+0000","last_unstale":"2024-07-26T12:45:12.301049+0000","last_undegraded":"2024-07-26T12:45:12.301049+0000","last_fullsized":"2024-07-26T12:45:12.301049+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:23:19.845664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:31.537561+0000","last_change":"2024-07-26T12:43:26.774537+0000","last_active":"2024-07-26T12:45:31.537561+0000","last_peered":"2024-07-26T12:45:31.537561+0000","last_clean":"2024-07-26T12:45:31.537561+0000","last_became_active":"2024-07-26T12:43:26.774036+0000","last_became_peered":"2024-07-26T12:43:26.774036+0000","last_unstale":"2024-07-26T12:45:31.537561+0000","last_undegraded":"2024-07-26T12:45:31.537561+0000","last_fullsized":"2024-07-26T12:45:31.537561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:21.774498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662838+0000","last_change":"2024-07-26T12:44:36.857443+0000","last_active":"2024-07-26T12:46:34.662838+0000","last_peered":"2024-07-26T12:46:34.662838+0000","last_clean":"2024-07-26T12:46:34.662838+0000","last_became_active":"2024-07-26T12:44:36.857001+0000","last_became_peered":"2024-07-26T12:44:36.857001+0000","last_unstale":"2024-07-26T12:46:34.662838+0000","last_undegraded":"2024-07-26T12:46:34.662838+0000","last_fullsized":"2024-07-26T12:46:34.662838+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:08.236404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:16.261130+0000","last_change":"2024-07-26T12:43:08.227432+0000","last_active":"2024-07-26T12:45:16.261130+0000","last_peered":"2024-07-26T12:45:16.261130+0000","last_clean":"2024-07-26T12:45:16.261130+0000","last_became_active":"2024-07-26T12:43:08.226645+0000","last_became_peered":"2024-07-26T12:43:08.226645+0000","last_unstale":"2024-07-26T12:45:16.261130+0000","last_undegraded":"2024-07-26T12:45:16.261130+0000","last_fullsized":"2024-07-26T12:45:16.261130+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:23.498855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:32.303272+0000","last_change":"2024-07-26T12:43:26.792874+0000","last_active":"2024-07-26T12:45:32.303272+0000","last_peered":"2024-07-26T12:45:32.303272+0000","last_clean":"2024-07-26T12:45:32.303272+0000","last_became_active":"2024-07-26T12:43:26.792371+0000","last_became_peered":"2024-07-26T12:43:26.792371+0000","last_unstale":"2024-07-26T12:45:32.303272+0000","last_undegraded":"2024-07-26T12:45:32.303272+0000","last_fullsized":"2024-07-26T12:45:32.303272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:28:25.753111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.654086+0000","last_change":"2024-07-26T12:43:12.185733+0000","last_active":"2024-07-26T12:46:34.654086+0000","last_peered":"2024-07-26T12:46:34.654086+0000","last_clean":"2024-07-26T12:46:34.654086+0000","last_became_active":"2024-07-26T12:43:12.185305+0000","last_became_peered":"2024-07-26T12:43:12.185305+0000","last_unstale":"2024-07-26T12:46:34.654086+0000","last_undegraded":"2024-07-26T12:46:34.654086+0000","last_fullsized":"2024-07-26T12:46:34.654086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:16:44.634607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":296,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:41.374820+0000","last_change":"2024-07-26T12:43:08.118508+0000","last_active":"2024-07-26T12:48:41.374820+0000","last_peered":"2024-07-26T12:48:41.374820+0000","last_clean":"2024-07-26T12:48:41.374820+0000","last_became_active":"2024-07-26T12:43:08.118088+0000","last_became_peered":"2024-07-26T12:43:08.118088+0000","last_unstale":"2024-07-26T12:48:41.374820+0000","last_undegraded":"2024-07-26T12:48:41.374820+0000","last_fullsized":"2024-07-26T12:48:41.374820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:23:46.474298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:33.076402+0000","last_change":"2024-07-26T12:43:26.779328+0000","last_active":"2024-07-26T12:45:33.076402+0000","last_peered":"2024-07-26T12:45:33.076402+0000","last_clean":"2024-07-26T12:45:33.076402+0000","last_became_active":"2024-07-26T12:43:26.778912+0000","last_became_peered":"2024-07-26T12:43:26.778912+0000","last_unstale":"2024-07-26T12:45:33.076402+0000","last_undegraded":"2024-07-26T12:45:33.076402+0000","last_fullsized":"2024-07-26T12:45:33.076402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:07:58.364775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650476+0000","last_change":"2024-07-26T12:43:35.854997+0000","last_active":"2024-07-26T12:46:34.650476+0000","last_peered":"2024-07-26T12:46:34.650476+0000","last_clean":"2024-07-26T12:46:34.650476+0000","last_became_active":"2024-07-26T12:43:35.854387+0000","last_became_peered":"2024-07-26T12:43:35.854387+0000","last_unstale":"2024-07-26T12:46:34.650476+0000","last_undegraded":"2024-07-26T12:46:34.650476+0000","last_fullsized":"2024-07-26T12:46:34.650476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:04:20.489388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":296,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:41.504837+0000","last_change":"2024-07-26T12:43:08.227628+0000","last_active":"2024-07-26T12:48:41.504837+0000","last_peered":"2024-07-26T12:48:41.504837+0000","last_clean":"2024-07-26T12:48:41.504837+0000","last_became_active":"2024-07-26T12:43:08.227133+0000","last_became_peered":"2024-07-26T12:43:08.227133+0000","last_unstale":"2024-07-26T12:48:41.504837+0000","last_undegraded":"2024-07-26T12:48:41.504837+0000","last_fullsized":"2024-07-26T12:48:41.504837+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:28:28.484315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:38.077398+0000","last_change":"2024-07-26T12:43:26.794016+0000","last_active":"2024-07-26T12:45:38.077398+0000","last_peered":"2024-07-26T12:45:38.077398+0000","last_clean":"2024-07-26T12:45:38.077398+0000","last_became_active":"2024-07-26T12:43:26.792562+0000","last_became_peered":"2024-07-26T12:43:26.792562+0000","last_unstale":"2024-07-26T12:45:38.077398+0000","last_undegraded":"2024-07-26T12:45:38.077398+0000","last_fullsized":"2024-07-26T12:45:38.077398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:26.162840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.653115+0000","last_change":"2024-07-26T12:43:12.173014+0000","last_active":"2024-07-26T12:46:34.653115+0000","last_peered":"2024-07-26T12:46:34.653115+0000","last_clean":"2024-07-26T12:46:34.653115+0000","last_became_active":"2024-07-26T12:43:12.172600+0000","last_became_peered":"2024-07-26T12:43:12.172600+0000","last_unstale":"2024-07-26T12:46:34.653115+0000","last_undegraded":"2024-07-26T12:46:34.653115+0000","last_fullsized":"2024-07-26T12:46:34.653115+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:18:59.842744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:07.570667+0000","last_change":"2024-07-26T12:43:08.119000+0000","last_active":"2024-07-26T12:45:07.570667+0000","last_peered":"2024-07-26T12:45:07.570667+0000","last_clean":"2024-07-26T12:45:07.570667+0000","last_became_active":"2024-07-26T12:43:08.118608+0000","last_became_peered":"2024-07-26T12:43:08.118608+0000","last_unstale":"2024-07-26T12:45:07.570667+0000","last_undegraded":"2024-07-26T12:45:07.570667+0000","last_fullsized":"2024-07-26T12:45:07.570667+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:29:47.663449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:39.012097+0000","last_change":"2024-07-26T12:43:35.914716+0000","last_active":"2024-07-26T12:45:39.012097+0000","last_peered":"2024-07-26T12:45:39.012097+0000","last_clean":"2024-07-26T12:45:39.012097+0000","last_became_active":"2024-07-26T12:43:35.914354+0000","last_became_peered":"2024-07-26T12:43:35.914354+0000","last_unstale":"2024-07-26T12:45:39.012097+0000","last_undegraded":"2024-07-26T12:45:39.012097+0000","last_fullsized":"2024-07-26T12:45:39.012097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:09:10.994141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.662686+0000","last_change":"2024-07-26T12:43:12.169997+0000","last_active":"2024-07-26T12:46:34.662686+0000","last_peered":"2024-07-26T12:46:34.662686+0000","last_clean":"2024-07-26T12:46:34.662686+0000","last_became_active":"2024-07-26T12:43:12.169044+0000","last_became_peered":"2024-07-26T12:43:12.169044+0000","last_unstale":"2024-07-26T12:46:34.662686+0000","last_undegraded":"2024-07-26T12:46:34.662686+0000","last_fullsized":"2024-07-26T12:46:34.662686+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:41.791434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:16.762094+0000","last_change":"2024-07-26T12:43:08.129025+0000","last_active":"2024-07-26T12:45:16.762094+0000","last_peered":"2024-07-26T12:45:16.762094+0000","last_clean":"2024-07-26T12:45:16.762094+0000","last_became_active":"2024-07-26T12:43:08.128608+0000","last_became_peered":"2024-07-26T12:43:08.128608+0000","last_unstale":"2024-07-26T12:45:16.762094+0000","last_undegraded":"2024-07-26T12:45:16.762094+0000","last_fullsized":"2024-07-26T12:45:16.762094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:43.749473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":123,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:22.580061+0000","last_change":"2024-07-26T12:41:42.411595+0000","last_active":"2024-07-26T12:46:22.580061+0000","last_peered":"2024-07-26T12:46:22.580061+0000","last_clean":"2024-07-26T12:46:22.580061+0000","last_became_active":"2024-07-26T12:41:42.407119+0000","last_became_peered":"2024-07-26T12:41:42.407119+0000","last_unstale":"2024-07-26T12:46:22.580061+0000","last_undegraded":"2024-07-26T12:46:22.580061+0000","last_fullsized":"2024-07-26T12:46:22.580061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:37:59.605754+0000","last_clean_scrub_stamp":"2024-07-26T12:37:59.605754+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:07:21.376418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:32.573794+0000","last_change":"2024-07-26T12:43:26.763955+0000","last_active":"2024-07-26T12:45:32.573794+0000","last_peered":"2024-07-26T12:45:32.573794+0000","last_clean":"2024-07-26T12:45:32.573794+0000","last_became_active":"2024-07-26T12:43:26.763368+0000","last_became_peered":"2024-07-26T12:43:26.763368+0000","last_unstale":"2024-07-26T12:45:32.573794+0000","last_undegraded":"2024-07-26T12:45:32.573794+0000","last_fullsized":"2024-07-26T12:45:32.573794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:48:14.389451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.663123+0000","last_change":"2024-07-26T12:43:12.174780+0000","last_active":"2024-07-26T12:46:34.663123+0000","last_peered":"2024-07-26T12:46:34.663123+0000","last_clean":"2024-07-26T12:46:34.663123+0000","last_became_active":"2024-07-26T12:43:12.174379+0000","last_became_peered":"2024-07-26T12:43:12.174379+0000","last_unstale":"2024-07-26T12:46:34.663123+0000","last_undegraded":"2024-07-26T12:46:34.663123+0000","last_fullsized":"2024-07-26T12:46:34.663123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:42:22.783259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'7","reported_seq":302,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.278004+0000","last_change":"2024-07-26T12:43:08.894655+0000","last_active":"2024-07-26T12:48:42.278004+0000","last_peered":"2024-07-26T12:48:42.278004+0000","last_clean":"2024-07-26T12:48:42.278004+0000","last_became_active":"2024-07-26T12:43:08.894270+0000","last_became_peered":"2024-07-26T12:43:08.894270+0000","last_unstale":"2024-07-26T12:48:42.278004+0000","last_undegraded":"2024-07-26T12:48:42.278004+0000","last_fullsized":"2024-07-26T12:48:42.278004+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:42:34.431204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":248,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:36.700412+0000","last_change":"2024-07-26T12:43:26.769716+0000","last_active":"2024-07-26T12:45:36.700412+0000","last_peered":"2024-07-26T12:45:36.700412+0000","last_clean":"2024-07-26T12:45:36.700412+0000","last_became_active":"2024-07-26T12:43:26.769154+0000","last_became_peered":"2024-07-26T12:43:26.769154+0000","last_unstale":"2024-07-26T12:45:36.700412+0000","last_undegraded":"2024-07-26T12:45:36.700412+0000","last_fullsized":"2024-07-26T12:45:36.700412+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:43.102498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664039+0000","last_change":"2024-07-26T12:43:12.179488+0000","last_active":"2024-07-26T12:46:34.664039+0000","last_peered":"2024-07-26T12:46:34.664039+0000","last_clean":"2024-07-26T12:46:34.664039+0000","last_became_active":"2024-07-26T12:43:12.179171+0000","last_became_peered":"2024-07-26T12:43:12.179171+0000","last_unstale":"2024-07-26T12:46:34.664039+0000","last_undegraded":"2024-07-26T12:46:34.664039+0000","last_fullsized":"2024-07-26T12:46:34.664039+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:43.695582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:24.009763+0000","last_change":"2024-07-26T12:43:08.130838+0000","last_active":"2024-07-26T12:45:24.009763+0000","last_peered":"2024-07-26T12:45:24.009763+0000","last_clean":"2024-07-26T12:45:24.009763+0000","last_became_active":"2024-07-26T12:43:08.130446+0000","last_became_peered":"2024-07-26T12:43:08.130446+0000","last_unstale":"2024-07-26T12:45:24.009763+0000","last_undegraded":"2024-07-26T12:45:24.009763+0000","last_fullsized":"2024-07-26T12:45:24.009763+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:33:55.271038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:42.787196+0000","last_change":"2024-07-26T12:43:26.781982+0000","last_active":"2024-07-26T12:45:42.787196+0000","last_peered":"2024-07-26T12:45:42.787196+0000","last_clean":"2024-07-26T12:45:42.787196+0000","last_became_active":"2024-07-26T12:43:26.777102+0000","last_became_peered":"2024-07-26T12:43:26.777102+0000","last_unstale":"2024-07-26T12:45:42.787196+0000","last_undegraded":"2024-07-26T12:45:42.787196+0000","last_fullsized":"2024-07-26T12:45:42.787196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:49.553124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659860+0000","last_change":"2024-07-26T12:43:12.177043+0000","last_active":"2024-07-26T12:46:34.659860+0000","last_peered":"2024-07-26T12:46:34.659860+0000","last_clean":"2024-07-26T12:46:34.659860+0000","last_became_active":"2024-07-26T12:43:12.176634+0000","last_became_peered":"2024-07-26T12:43:12.176634+0000","last_unstale":"2024-07-26T12:46:34.659860+0000","last_undegraded":"2024-07-26T12:46:34.659860+0000","last_fullsized":"2024-07-26T12:46:34.659860+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:40:20.210176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:23.074306+0000","last_change":"2024-07-26T12:43:08.131279+0000","last_active":"2024-07-26T12:45:23.074306+0000","last_peered":"2024-07-26T12:45:23.074306+0000","last_clean":"2024-07-26T12:45:23.074306+0000","last_became_active":"2024-07-26T12:43:08.130755+0000","last_became_peered":"2024-07-26T12:43:08.130755+0000","last_unstale":"2024-07-26T12:45:23.074306+0000","last_undegraded":"2024-07-26T12:45:23.074306+0000","last_fullsized":"2024-07-26T12:45:23.074306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:03.234905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:39.012012+0000","last_change":"2024-07-26T12:43:26.764407+0000","last_active":"2024-07-26T12:45:39.012012+0000","last_peered":"2024-07-26T12:45:39.012012+0000","last_clean":"2024-07-26T12:45:39.012012+0000","last_became_active":"2024-07-26T12:43:26.762408+0000","last_became_peered":"2024-07-26T12:43:26.762408+0000","last_unstale":"2024-07-26T12:45:39.012012+0000","last_undegraded":"2024-07-26T12:45:39.012012+0000","last_fullsized":"2024-07-26T12:45:39.012012+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:35:29.139476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.648078+0000","last_change":"2024-07-26T12:43:12.174587+0000","last_active":"2024-07-26T12:46:34.648078+0000","last_peered":"2024-07-26T12:46:34.648078+0000","last_clean":"2024-07-26T12:46:34.648078+0000","last_became_active":"2024-07-26T12:43:12.173929+0000","last_became_peered":"2024-07-26T12:43:12.173929+0000","last_unstale":"2024-07-26T12:46:34.648078+0000","last_undegraded":"2024-07-26T12:46:34.648078+0000","last_fullsized":"2024-07-26T12:46:34.648078+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:16.775198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:22.302322+0000","last_change":"2024-07-26T12:43:08.119825+0000","last_active":"2024-07-26T12:45:22.302322+0000","last_peered":"2024-07-26T12:45:22.302322+0000","last_clean":"2024-07-26T12:45:22.302322+0000","last_became_active":"2024-07-26T12:43:08.119273+0000","last_became_peered":"2024-07-26T12:43:08.119273+0000","last_unstale":"2024-07-26T12:45:22.302322+0000","last_undegraded":"2024-07-26T12:45:22.302322+0000","last_fullsized":"2024-07-26T12:45:22.302322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:35:39.973020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:39.012041+0000","last_change":"2024-07-26T12:43:26.773226+0000","last_active":"2024-07-26T12:45:39.012041+0000","last_peered":"2024-07-26T12:45:39.012041+0000","last_clean":"2024-07-26T12:45:39.012041+0000","last_became_active":"2024-07-26T12:43:26.772745+0000","last_became_peered":"2024-07-26T12:43:26.772745+0000","last_unstale":"2024-07-26T12:45:39.012041+0000","last_undegraded":"2024-07-26T12:45:39.012041+0000","last_fullsized":"2024-07-26T12:45:39.012041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:35.407138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.648325+0000","last_change":"2024-07-26T12:43:12.176178+0000","last_active":"2024-07-26T12:46:34.648325+0000","last_peered":"2024-07-26T12:46:34.648325+0000","last_clean":"2024-07-26T12:46:34.648325+0000","last_became_active":"2024-07-26T12:43:12.175791+0000","last_became_peered":"2024-07-26T12:43:12.175791+0000","last_unstale":"2024-07-26T12:46:34.648325+0000","last_undegraded":"2024-07-26T12:46:34.648325+0000","last_fullsized":"2024-07-26T12:46:34.648325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:00:48.885831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:27.302835+0000","last_change":"2024-07-26T12:43:08.226345+0000","last_active":"2024-07-26T12:45:27.302835+0000","last_peered":"2024-07-26T12:45:27.302835+0000","last_clean":"2024-07-26T12:45:27.302835+0000","last_became_active":"2024-07-26T12:43:08.225897+0000","last_became_peered":"2024-07-26T12:43:08.225897+0000","last_unstale":"2024-07-26T12:45:27.302835+0000","last_undegraded":"2024-07-26T12:45:27.302835+0000","last_fullsized":"2024-07-26T12:45:27.302835+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:36:59.351076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:53.079873+0000","last_change":"2024-07-26T12:43:26.774800+0000","last_active":"2024-07-26T12:45:53.079873+0000","last_peered":"2024-07-26T12:45:53.079873+0000","last_clean":"2024-07-26T12:45:53.079873+0000","last_became_active":"2024-07-26T12:43:26.774192+0000","last_became_peered":"2024-07-26T12:43:26.774192+0000","last_unstale":"2024-07-26T12:45:53.079873+0000","last_undegraded":"2024-07-26T12:45:53.079873+0000","last_fullsized":"2024-07-26T12:45:53.079873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:31:18.379619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.659204+0000","last_change":"2024-07-26T12:43:12.171720+0000","last_active":"2024-07-26T12:46:34.659204+0000","last_peered":"2024-07-26T12:46:34.659204+0000","last_clean":"2024-07-26T12:46:34.659204+0000","last_became_active":"2024-07-26T12:43:12.171334+0000","last_became_peered":"2024-07-26T12:43:12.171334+0000","last_unstale":"2024-07-26T12:46:34.659204+0000","last_undegraded":"2024-07-26T12:46:34.659204+0000","last_fullsized":"2024-07-26T12:46:34.659204+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:35.156430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":301,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:45.515825+0000","last_change":"2024-07-26T12:43:08.120325+0000","last_active":"2024-07-26T12:48:45.515825+0000","last_peered":"2024-07-26T12:48:45.515825+0000","last_clean":"2024-07-26T12:48:45.515825+0000","last_became_active":"2024-07-26T12:43:08.119896+0000","last_became_peered":"2024-07-26T12:43:08.119896+0000","last_unstale":"2024-07-26T12:48:45.515825+0000","last_undegraded":"2024-07-26T12:48:45.515825+0000","last_fullsized":"2024-07-26T12:48:45.515825+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:23:19.470491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":250,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:51.266909+0000","last_change":"2024-07-26T12:43:26.778688+0000","last_active":"2024-07-26T12:45:51.266909+0000","last_peered":"2024-07-26T12:45:51.266909+0000","last_clean":"2024-07-26T12:45:51.266909+0000","last_became_active":"2024-07-26T12:43:26.778239+0000","last_became_peered":"2024-07-26T12:43:26.778239+0000","last_unstale":"2024-07-26T12:45:51.266909+0000","last_undegraded":"2024-07-26T12:45:51.266909+0000","last_fullsized":"2024-07-26T12:45:51.266909+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:50:48.409636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.676557+0000","last_change":"2024-07-26T12:43:12.182323+0000","last_active":"2024-07-26T12:46:34.676557+0000","last_peered":"2024-07-26T12:46:34.676557+0000","last_clean":"2024-07-26T12:46:34.676557+0000","last_became_active":"2024-07-26T12:43:12.181860+0000","last_became_peered":"2024-07-26T12:43:12.181860+0000","last_unstale":"2024-07-26T12:46:34.676557+0000","last_undegraded":"2024-07-26T12:46:34.676557+0000","last_fullsized":"2024-07-26T12:46:34.676557+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:08.764979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:29.010371+0000","last_change":"2024-07-26T12:43:08.130823+0000","last_active":"2024-07-26T12:45:29.010371+0000","last_peered":"2024-07-26T12:45:29.010371+0000","last_clean":"2024-07-26T12:45:29.010371+0000","last_became_active":"2024-07-26T12:43:08.130424+0000","last_became_peered":"2024-07-26T12:43:08.130424+0000","last_unstale":"2024-07-26T12:45:29.010371+0000","last_undegraded":"2024-07-26T12:45:29.010371+0000","last_fullsized":"2024-07-26T12:45:29.010371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:49:35.829624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.539348+0000","last_change":"2024-07-26T12:43:26.770070+0000","last_active":"2024-07-26T12:45:46.539348+0000","last_peered":"2024-07-26T12:45:46.539348+0000","last_clean":"2024-07-26T12:45:46.539348+0000","last_became_active":"2024-07-26T12:43:26.769594+0000","last_became_peered":"2024-07-26T12:43:26.769594+0000","last_unstale":"2024-07-26T12:45:46.539348+0000","last_undegraded":"2024-07-26T12:45:46.539348+0000","last_fullsized":"2024-07-26T12:45:46.539348+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:18.342739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669284+0000","last_change":"2024-07-26T12:43:12.183134+0000","last_active":"2024-07-26T12:46:34.669284+0000","last_peered":"2024-07-26T12:46:34.669284+0000","last_clean":"2024-07-26T12:46:34.669284+0000","last_became_active":"2024-07-26T12:43:12.182406+0000","last_became_peered":"2024-07-26T12:43:12.182406+0000","last_unstale":"2024-07-26T12:46:34.669284+0000","last_undegraded":"2024-07-26T12:46:34.669284+0000","last_fullsized":"2024-07-26T12:46:34.669284+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:54:24.038971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:56.540854+0000","last_change":"2024-07-26T12:43:35.862005+0000","last_active":"2024-07-26T12:45:56.540854+0000","last_peered":"2024-07-26T12:45:56.540854+0000","last_clean":"2024-07-26T12:45:56.540854+0000","last_became_active":"2024-07-26T12:43:35.850911+0000","last_became_peered":"2024-07-26T12:43:35.850911+0000","last_unstale":"2024-07-26T12:45:56.540854+0000","last_undegraded":"2024-07-26T12:45:56.540854+0000","last_fullsized":"2024-07-26T12:45:56.540854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:18:16.120983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:54.014118+0000","last_change":"2024-07-26T12:43:26.774599+0000","last_active":"2024-07-26T12:45:54.014118+0000","last_peered":"2024-07-26T12:45:54.014118+0000","last_clean":"2024-07-26T12:45:54.014118+0000","last_became_active":"2024-07-26T12:43:26.774186+0000","last_became_peered":"2024-07-26T12:43:26.774186+0000","last_unstale":"2024-07-26T12:45:54.014118+0000","last_undegraded":"2024-07-26T12:45:54.014118+0000","last_fullsized":"2024-07-26T12:45:54.014118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:02:42.252093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.646564+0000","last_change":"2024-07-26T12:43:12.174676+0000","last_active":"2024-07-26T12:46:34.646564+0000","last_peered":"2024-07-26T12:46:34.646564+0000","last_clean":"2024-07-26T12:46:34.646564+0000","last_became_active":"2024-07-26T12:43:12.174310+0000","last_became_peered":"2024-07-26T12:43:12.174310+0000","last_unstale":"2024-07-26T12:46:34.646564+0000","last_undegraded":"2024-07-26T12:46:34.646564+0000","last_fullsized":"2024-07-26T12:46:34.646564+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:18:44.381289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'42","reported_seq":587,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.277571+0000","last_change":"2024-07-26T12:43:08.122675+0000","last_active":"2024-07-26T12:48:42.277571+0000","last_peered":"2024-07-26T12:48:42.277571+0000","last_clean":"2024-07-26T12:48:42.277571+0000","last_became_active":"2024-07-26T12:43:08.122174+0000","last_became_peered":"2024-07-26T12:43:08.122174+0000","last_unstale":"2024-07-26T12:48:42.277571+0000","last_undegraded":"2024-07-26T12:48:42.277571+0000","last_fullsized":"2024-07-26T12:48:42.277571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:54.513306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70065,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":498,"num_read_kb":318,"num_write":44,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.265915+0000","last_change":"2024-07-26T12:43:26.764283+0000","last_active":"2024-07-26T12:45:46.265915+0000","last_peered":"2024-07-26T12:45:46.265915+0000","last_clean":"2024-07-26T12:45:46.265915+0000","last_became_active":"2024-07-26T12:43:26.763821+0000","last_became_peered":"2024-07-26T12:43:26.763821+0000","last_unstale":"2024-07-26T12:45:46.265915+0000","last_undegraded":"2024-07-26T12:45:46.265915+0000","last_fullsized":"2024-07-26T12:45:46.265915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:43:56.753016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.672277+0000","last_change":"2024-07-26T12:43:35.915277+0000","last_active":"2024-07-26T12:46:34.672277+0000","last_peered":"2024-07-26T12:46:34.672277+0000","last_clean":"2024-07-26T12:46:34.672277+0000","last_became_active":"2024-07-26T12:43:35.914848+0000","last_became_peered":"2024-07-26T12:43:35.914848+0000","last_unstale":"2024-07-26T12:46:34.672277+0000","last_undegraded":"2024-07-26T12:46:34.672277+0000","last_fullsized":"2024-07-26T12:46:34.672277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:56:35.639076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:34.011224+0000","last_change":"2024-07-26T12:43:08.227352+0000","last_active":"2024-07-26T12:45:34.011224+0000","last_peered":"2024-07-26T12:45:34.011224+0000","last_clean":"2024-07-26T12:45:34.011224+0000","last_became_active":"2024-07-26T12:43:08.226857+0000","last_became_peered":"2024-07-26T12:43:08.226857+0000","last_unstale":"2024-07-26T12:45:34.011224+0000","last_undegraded":"2024-07-26T12:45:34.011224+0000","last_fullsized":"2024-07-26T12:45:34.011224+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:00:20.156503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.765392+0000","last_change":"2024-07-26T12:43:26.762943+0000","last_active":"2024-07-26T12:45:46.765392+0000","last_peered":"2024-07-26T12:45:46.765392+0000","last_clean":"2024-07-26T12:45:46.765392+0000","last_became_active":"2024-07-26T12:43:26.762436+0000","last_became_peered":"2024-07-26T12:43:26.762436+0000","last_unstale":"2024-07-26T12:45:46.765392+0000","last_undegraded":"2024-07-26T12:45:46.765392+0000","last_fullsized":"2024-07-26T12:45:46.765392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:05:23.924397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.673242+0000","last_change":"2024-07-26T12:44:36.854705+0000","last_active":"2024-07-26T12:46:34.673242+0000","last_peered":"2024-07-26T12:46:34.673242+0000","last_clean":"2024-07-26T12:46:34.673242+0000","last_became_active":"2024-07-26T12:44:36.854268+0000","last_became_peered":"2024-07-26T12:44:36.854268+0000","last_unstale":"2024-07-26T12:46:34.673242+0000","last_undegraded":"2024-07-26T12:46:34.673242+0000","last_fullsized":"2024-07-26T12:46:34.673242+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:24:30.768905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:32.573818+0000","last_change":"2024-07-26T12:43:08.118597+0000","last_active":"2024-07-26T12:45:32.573818+0000","last_peered":"2024-07-26T12:45:32.573818+0000","last_clean":"2024-07-26T12:45:32.573818+0000","last_became_active":"2024-07-26T12:43:08.118145+0000","last_became_peered":"2024-07-26T12:43:08.118145+0000","last_unstale":"2024-07-26T12:45:32.573818+0000","last_undegraded":"2024-07-26T12:45:32.573818+0000","last_fullsized":"2024-07-26T12:45:32.573818+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:52:45.511693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.265831+0000","last_change":"2024-07-26T12:43:26.771861+0000","last_active":"2024-07-26T12:45:46.265831+0000","last_peered":"2024-07-26T12:45:46.265831+0000","last_clean":"2024-07-26T12:45:46.265831+0000","last_became_active":"2024-07-26T12:43:26.771323+0000","last_became_peered":"2024-07-26T12:43:26.771323+0000","last_unstale":"2024-07-26T12:45:46.265831+0000","last_undegraded":"2024-07-26T12:45:46.265831+0000","last_fullsized":"2024-07-26T12:45:46.265831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:34.207514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656491+0000","last_change":"2024-07-26T12:43:12.169437+0000","last_active":"2024-07-26T12:46:34.656491+0000","last_peered":"2024-07-26T12:46:34.656491+0000","last_clean":"2024-07-26T12:46:34.656491+0000","last_became_active":"2024-07-26T12:43:12.168387+0000","last_became_peered":"2024-07-26T12:43:12.168387+0000","last_unstale":"2024-07-26T12:46:34.656491+0000","last_undegraded":"2024-07-26T12:46:34.656491+0000","last_fullsized":"2024-07-26T12:46:34.656491+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:57:06.148965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'6","reported_seq":539,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.277078+0000","last_change":"2024-07-26T12:43:08.129618+0000","last_active":"2024-07-26T12:48:42.277078+0000","last_peered":"2024-07-26T12:48:42.277078+0000","last_clean":"2024-07-26T12:48:42.277078+0000","last_became_active":"2024-07-26T12:43:08.129064+0000","last_became_peered":"2024-07-26T12:43:08.129064+0000","last_unstale":"2024-07-26T12:48:42.277078+0000","last_undegraded":"2024-07-26T12:48:42.277078+0000","last_fullsized":"2024-07-26T12:48:42.277078+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:26:22.406075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":486,"num_read_kb":486,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:46.261842+0000","last_change":"2024-07-26T12:43:35.854710+0000","last_active":"2024-07-26T12:45:46.261842+0000","last_peered":"2024-07-26T12:45:46.261842+0000","last_clean":"2024-07-26T12:45:46.261842+0000","last_became_active":"2024-07-26T12:43:35.852772+0000","last_became_peered":"2024-07-26T12:43:35.852772+0000","last_unstale":"2024-07-26T12:45:46.261842+0000","last_undegraded":"2024-07-26T12:45:46.261842+0000","last_fullsized":"2024-07-26T12:45:46.261842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:24:54.015879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,10],"acting":[3,7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:27.784210+0000","last_change":"2024-07-26T12:43:26.783182+0000","last_active":"2024-07-26T12:45:27.784210+0000","last_peered":"2024-07-26T12:45:27.784210+0000","last_clean":"2024-07-26T12:45:27.784210+0000","last_became_active":"2024-07-26T12:43:26.782697+0000","last_became_peered":"2024-07-26T12:43:26.782697+0000","last_unstale":"2024-07-26T12:45:27.784210+0000","last_undegraded":"2024-07-26T12:45:27.784210+0000","last_fullsized":"2024-07-26T12:45:27.784210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:09:57.169222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671751+0000","last_change":"2024-07-26T12:43:12.166163+0000","last_active":"2024-07-26T12:46:34.671751+0000","last_peered":"2024-07-26T12:46:34.671751+0000","last_clean":"2024-07-26T12:46:34.671751+0000","last_became_active":"2024-07-26T12:43:12.164427+0000","last_became_peered":"2024-07-26T12:43:12.164427+0000","last_unstale":"2024-07-26T12:46:34.671751+0000","last_undegraded":"2024-07-26T12:46:34.671751+0000","last_fullsized":"2024-07-26T12:46:34.671751+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:25.335508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:34.011158+0000","last_change":"2024-07-26T12:43:08.118224+0000","last_active":"2024-07-26T12:45:34.011158+0000","last_peered":"2024-07-26T12:45:34.011158+0000","last_clean":"2024-07-26T12:45:34.011158+0000","last_became_active":"2024-07-26T12:43:08.117821+0000","last_became_peered":"2024-07-26T12:43:08.117821+0000","last_unstale":"2024-07-26T12:45:34.011158+0000","last_undegraded":"2024-07-26T12:45:34.011158+0000","last_fullsized":"2024-07-26T12:45:34.011158+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:56:14.380931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:31.537591+0000","last_change":"2024-07-26T12:43:35.907740+0000","last_active":"2024-07-26T12:45:31.537591+0000","last_peered":"2024-07-26T12:45:31.537591+0000","last_clean":"2024-07-26T12:45:31.537591+0000","last_became_active":"2024-07-26T12:43:35.907115+0000","last_became_peered":"2024-07-26T12:43:35.907115+0000","last_unstale":"2024-07-26T12:45:31.537591+0000","last_undegraded":"2024-07-26T12:45:31.537591+0000","last_fullsized":"2024-07-26T12:45:31.537591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:53:24.783872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.670573+0000","last_change":"2024-07-26T12:43:12.177579+0000","last_active":"2024-07-26T12:46:34.670573+0000","last_peered":"2024-07-26T12:46:34.670573+0000","last_clean":"2024-07-26T12:46:34.670573+0000","last_became_active":"2024-07-26T12:43:12.171433+0000","last_became_peered":"2024-07-26T12:43:12.171433+0000","last_unstale":"2024-07-26T12:46:34.670573+0000","last_undegraded":"2024-07-26T12:46:34.670573+0000","last_fullsized":"2024-07-26T12:46:34.670573+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:50.972151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:33.076329+0000","last_change":"2024-07-26T12:43:08.133446+0000","last_active":"2024-07-26T12:45:33.076329+0000","last_peered":"2024-07-26T12:45:33.076329+0000","last_clean":"2024-07-26T12:45:33.076329+0000","last_became_active":"2024-07-26T12:43:08.133008+0000","last_became_peered":"2024-07-26T12:43:08.133008+0000","last_unstale":"2024-07-26T12:45:33.076329+0000","last_undegraded":"2024-07-26T12:45:33.076329+0000","last_fullsized":"2024-07-26T12:45:33.076329+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:05:21.283738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:37.266863+0000","last_change":"2024-07-26T12:43:08.124976+0000","last_active":"2024-07-26T12:45:37.266863+0000","last_peered":"2024-07-26T12:45:37.266863+0000","last_clean":"2024-07-26T12:45:37.266863+0000","last_became_active":"2024-07-26T12:43:08.124664+0000","last_became_peered":"2024-07-26T12:43:08.124664+0000","last_unstale":"2024-07-26T12:45:37.266863+0000","last_undegraded":"2024-07-26T12:45:37.266863+0000","last_fullsized":"2024-07-26T12:45:37.266863+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:51.798657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664310+0000","last_change":"2024-07-26T12:43:35.856266+0000","last_active":"2024-07-26T12:46:34.664310+0000","last_peered":"2024-07-26T12:46:34.664310+0000","last_clean":"2024-07-26T12:46:34.664310+0000","last_became_active":"2024-07-26T12:43:35.855802+0000","last_became_peered":"2024-07-26T12:43:35.855802+0000","last_unstale":"2024-07-26T12:46:34.664310+0000","last_undegraded":"2024-07-26T12:46:34.664310+0000","last_fullsized":"2024-07-26T12:46:34.664310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:43:16.432983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:59.251187+0000","last_change":"2024-07-26T12:43:26.778626+0000","last_active":"2024-07-26T12:45:59.251187+0000","last_peered":"2024-07-26T12:45:59.251187+0000","last_clean":"2024-07-26T12:45:59.251187+0000","last_became_active":"2024-07-26T12:43:26.778165+0000","last_became_peered":"2024-07-26T12:43:26.778165+0000","last_unstale":"2024-07-26T12:45:59.251187+0000","last_undegraded":"2024-07-26T12:45:59.251187+0000","last_fullsized":"2024-07-26T12:45:59.251187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:06:08.766495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":302,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:48:42.277799+0000","last_change":"2024-07-26T12:43:08.119702+0000","last_active":"2024-07-26T12:48:42.277799+0000","last_peered":"2024-07-26T12:48:42.277799+0000","last_clean":"2024-07-26T12:48:42.277799+0000","last_became_active":"2024-07-26T12:43:08.119225+0000","last_became_peered":"2024-07-26T12:43:08.119225+0000","last_unstale":"2024-07-26T12:48:42.277799+0000","last_undegraded":"2024-07-26T12:48:42.277799+0000","last_fullsized":"2024-07-26T12:48:42.277799+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:07.087597+0000","last_clean_scrub_stamp":"2024-07-26T12:43:07.087597+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:59:27.616873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":248,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.640998+0000","last_change":"2024-07-26T12:43:12.166239+0000","last_active":"2024-07-26T12:46:34.640998+0000","last_peered":"2024-07-26T12:46:34.640998+0000","last_clean":"2024-07-26T12:46:34.640998+0000","last_became_active":"2024-07-26T12:43:12.165770+0000","last_became_peered":"2024-07-26T12:43:12.165770+0000","last_unstale":"2024-07-26T12:46:34.640998+0000","last_undegraded":"2024-07-26T12:46:34.640998+0000","last_fullsized":"2024-07-26T12:46:34.640998+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:11:15.290898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:45:59.251224+0000","last_change":"2024-07-26T12:43:26.765905+0000","last_active":"2024-07-26T12:45:59.251224+0000","last_peered":"2024-07-26T12:45:59.251224+0000","last_clean":"2024-07-26T12:45:59.251224+0000","last_became_active":"2024-07-26T12:43:26.765425+0000","last_became_peered":"2024-07-26T12:43:26.765425+0000","last_unstale":"2024-07-26T12:45:59.251224+0000","last_undegraded":"2024-07-26T12:45:59.251224+0000","last_fullsized":"2024-07-26T12:45:59.251224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:25.720109+0000","last_clean_scrub_stamp":"2024-07-26T12:43:25.720109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:01:16.786604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.645058+0000","last_change":"2024-07-26T12:43:12.171400+0000","last_active":"2024-07-26T12:46:34.645058+0000","last_peered":"2024-07-26T12:46:34.645058+0000","last_clean":"2024-07-26T12:46:34.645058+0000","last_became_active":"2024-07-26T12:43:12.170998+0000","last_became_peered":"2024-07-26T12:43:12.170998+0000","last_unstale":"2024-07-26T12:46:34.645058+0000","last_undegraded":"2024-07-26T12:46:34.645058+0000","last_fullsized":"2024-07-26T12:46:34.645058+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:43.251975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.656729+0000","last_change":"2024-07-26T12:43:35.912729+0000","last_active":"2024-07-26T12:46:34.656729+0000","last_peered":"2024-07-26T12:46:34.656729+0000","last_clean":"2024-07-26T12:46:34.656729+0000","last_became_active":"2024-07-26T12:43:35.912318+0000","last_became_peered":"2024-07-26T12:43:35.912318+0000","last_unstale":"2024-07-26T12:46:34.656729+0000","last_undegraded":"2024-07-26T12:46:34.656729+0000","last_fullsized":"2024-07-26T12:46:34.656729+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:14.473536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661325+0000","last_change":"2024-07-26T12:43:12.182933+0000","last_active":"2024-07-26T12:46:34.661325+0000","last_peered":"2024-07-26T12:46:34.661325+0000","last_clean":"2024-07-26T12:46:34.661325+0000","last_became_active":"2024-07-26T12:43:12.182656+0000","last_became_peered":"2024-07-26T12:43:12.182656+0000","last_unstale":"2024-07-26T12:46:34.661325+0000","last_undegraded":"2024-07-26T12:46:34.661325+0000","last_fullsized":"2024-07-26T12:46:34.661325+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:05:59.717662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.675425+0000","last_change":"2024-07-26T12:43:12.181196+0000","last_active":"2024-07-26T12:46:34.675425+0000","last_peered":"2024-07-26T12:46:34.675425+0000","last_clean":"2024-07-26T12:46:34.675425+0000","last_became_active":"2024-07-26T12:43:12.180765+0000","last_became_peered":"2024-07-26T12:43:12.180765+0000","last_unstale":"2024-07-26T12:46:34.675425+0000","last_undegraded":"2024-07-26T12:46:34.675425+0000","last_fullsized":"2024-07-26T12:46:34.675425+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:20.590023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671094+0000","last_change":"2024-07-26T12:43:12.185408+0000","last_active":"2024-07-26T12:46:34.671094+0000","last_peered":"2024-07-26T12:46:34.671094+0000","last_clean":"2024-07-26T12:46:34.671094+0000","last_became_active":"2024-07-26T12:43:12.185038+0000","last_became_peered":"2024-07-26T12:43:12.185038+0000","last_unstale":"2024-07-26T12:46:34.671094+0000","last_undegraded":"2024-07-26T12:46:34.671094+0000","last_fullsized":"2024-07-26T12:46:34.671094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:55:30.604995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644437+0000","last_change":"2024-07-26T12:43:12.169301+0000","last_active":"2024-07-26T12:46:34.644437+0000","last_peered":"2024-07-26T12:46:34.644437+0000","last_clean":"2024-07-26T12:46:34.644437+0000","last_became_active":"2024-07-26T12:43:12.168923+0000","last_became_peered":"2024-07-26T12:43:12.168923+0000","last_unstale":"2024-07-26T12:46:34.644437+0000","last_undegraded":"2024-07-26T12:46:34.644437+0000","last_fullsized":"2024-07-26T12:46:34.644437+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:04.506485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.647538+0000","last_change":"2024-07-26T12:43:12.166663+0000","last_active":"2024-07-26T12:46:34.647538+0000","last_peered":"2024-07-26T12:46:34.647538+0000","last_clean":"2024-07-26T12:46:34.647538+0000","last_became_active":"2024-07-26T12:43:12.166200+0000","last_became_peered":"2024-07-26T12:43:12.166200+0000","last_unstale":"2024-07-26T12:46:34.647538+0000","last_undegraded":"2024-07-26T12:46:34.647538+0000","last_fullsized":"2024-07-26T12:46:34.647538+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:32:14.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664657+0000","last_change":"2024-07-26T12:43:12.171273+0000","last_active":"2024-07-26T12:46:34.664657+0000","last_peered":"2024-07-26T12:46:34.664657+0000","last_clean":"2024-07-26T12:46:34.664657+0000","last_became_active":"2024-07-26T12:43:12.167680+0000","last_became_peered":"2024-07-26T12:43:12.167680+0000","last_unstale":"2024-07-26T12:46:34.664657+0000","last_undegraded":"2024-07-26T12:46:34.664657+0000","last_fullsized":"2024-07-26T12:46:34.664657+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:44.797644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671048+0000","last_change":"2024-07-26T12:43:12.179891+0000","last_active":"2024-07-26T12:46:34.671048+0000","last_peered":"2024-07-26T12:46:34.671048+0000","last_clean":"2024-07-26T12:46:34.671048+0000","last_became_active":"2024-07-26T12:43:12.179518+0000","last_became_peered":"2024-07-26T12:43:12.179518+0000","last_unstale":"2024-07-26T12:46:34.671048+0000","last_undegraded":"2024-07-26T12:46:34.671048+0000","last_fullsized":"2024-07-26T12:46:34.671048+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:24.246023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671055+0000","last_change":"2024-07-26T12:43:12.168521+0000","last_active":"2024-07-26T12:46:34.671055+0000","last_peered":"2024-07-26T12:46:34.671055+0000","last_clean":"2024-07-26T12:46:34.671055+0000","last_became_active":"2024-07-26T12:43:12.168126+0000","last_became_peered":"2024-07-26T12:43:12.168126+0000","last_unstale":"2024-07-26T12:46:34.671055+0000","last_undegraded":"2024-07-26T12:46:34.671055+0000","last_fullsized":"2024-07-26T12:46:34.671055+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:00.135090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661599+0000","last_change":"2024-07-26T12:44:36.843322+0000","last_active":"2024-07-26T12:46:34.661599+0000","last_peered":"2024-07-26T12:46:34.661599+0000","last_clean":"2024-07-26T12:46:34.661599+0000","last_became_active":"2024-07-26T12:44:36.842821+0000","last_became_peered":"2024-07-26T12:44:36.842821+0000","last_unstale":"2024-07-26T12:46:34.661599+0000","last_undegraded":"2024-07-26T12:46:34.661599+0000","last_fullsized":"2024-07-26T12:46:34.661599+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:38:05.609298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,8],"acting":[1,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.675045+0000","last_change":"2024-07-26T12:43:12.185565+0000","last_active":"2024-07-26T12:46:34.675045+0000","last_peered":"2024-07-26T12:46:34.675045+0000","last_clean":"2024-07-26T12:46:34.675045+0000","last_became_active":"2024-07-26T12:43:12.185057+0000","last_became_peered":"2024-07-26T12:43:12.185057+0000","last_unstale":"2024-07-26T12:46:34.675045+0000","last_undegraded":"2024-07-26T12:46:34.675045+0000","last_fullsized":"2024-07-26T12:46:34.675045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:46:58.108311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.644690+0000","last_change":"2024-07-26T12:43:12.173301+0000","last_active":"2024-07-26T12:46:34.644690+0000","last_peered":"2024-07-26T12:46:34.644690+0000","last_clean":"2024-07-26T12:46:34.644690+0000","last_became_active":"2024-07-26T12:43:12.172955+0000","last_became_peered":"2024-07-26T12:43:12.172955+0000","last_unstale":"2024-07-26T12:46:34.644690+0000","last_undegraded":"2024-07-26T12:46:34.644690+0000","last_fullsized":"2024-07-26T12:46:34.644690+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:09:41.731141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.661490+0000","last_change":"2024-07-26T12:43:12.176169+0000","last_active":"2024-07-26T12:46:34.661490+0000","last_peered":"2024-07-26T12:46:34.661490+0000","last_clean":"2024-07-26T12:46:34.661490+0000","last_became_active":"2024-07-26T12:43:12.175745+0000","last_became_peered":"2024-07-26T12:43:12.175745+0000","last_unstale":"2024-07-26T12:46:34.661490+0000","last_undegraded":"2024-07-26T12:46:34.661490+0000","last_fullsized":"2024-07-26T12:46:34.661490+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:31:20.393207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.654907+0000","last_change":"2024-07-26T12:43:12.168182+0000","last_active":"2024-07-26T12:46:34.654907+0000","last_peered":"2024-07-26T12:46:34.654907+0000","last_clean":"2024-07-26T12:46:34.654907+0000","last_became_active":"2024-07-26T12:43:12.163771+0000","last_became_peered":"2024-07-26T12:43:12.163771+0000","last_unstale":"2024-07-26T12:46:34.654907+0000","last_undegraded":"2024-07-26T12:46:34.654907+0000","last_fullsized":"2024-07-26T12:46:34.654907+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:40:30.960153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.660143+0000","last_change":"2024-07-26T12:43:12.189524+0000","last_active":"2024-07-26T12:46:34.660143+0000","last_peered":"2024-07-26T12:46:34.660143+0000","last_clean":"2024-07-26T12:46:34.660143+0000","last_became_active":"2024-07-26T12:43:12.189165+0000","last_became_peered":"2024-07-26T12:43:12.189165+0000","last_unstale":"2024-07-26T12:46:34.660143+0000","last_undegraded":"2024-07-26T12:46:34.660143+0000","last_fullsized":"2024-07-26T12:46:34.660143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-28T00:11:08.479058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.665412+0000","last_change":"2024-07-26T12:43:12.180603+0000","last_active":"2024-07-26T12:46:34.665412+0000","last_peered":"2024-07-26T12:46:34.665412+0000","last_clean":"2024-07-26T12:46:34.665412+0000","last_became_active":"2024-07-26T12:43:12.180204+0000","last_became_peered":"2024-07-26T12:43:12.180204+0000","last_unstale":"2024-07-26T12:46:34.665412+0000","last_undegraded":"2024-07-26T12:46:34.665412+0000","last_fullsized":"2024-07-26T12:46:34.665412+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:46.768213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668524+0000","last_change":"2024-07-26T12:43:12.185347+0000","last_active":"2024-07-26T12:46:34.668524+0000","last_peered":"2024-07-26T12:46:34.668524+0000","last_clean":"2024-07-26T12:46:34.668524+0000","last_became_active":"2024-07-26T12:43:12.180190+0000","last_became_peered":"2024-07-26T12:43:12.180190+0000","last_unstale":"2024-07-26T12:46:34.668524+0000","last_undegraded":"2024-07-26T12:46:34.668524+0000","last_fullsized":"2024-07-26T12:46:34.668524+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:49:07.654967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.667916+0000","last_change":"2024-07-26T12:43:12.177258+0000","last_active":"2024-07-26T12:46:34.667916+0000","last_peered":"2024-07-26T12:46:34.667916+0000","last_clean":"2024-07-26T12:46:34.667916+0000","last_became_active":"2024-07-26T12:43:12.176879+0000","last_became_peered":"2024-07-26T12:43:12.176879+0000","last_unstale":"2024-07-26T12:46:34.667916+0000","last_undegraded":"2024-07-26T12:46:34.667916+0000","last_fullsized":"2024-07-26T12:46:34.667916+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:44:24.244759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.675740+0000","last_change":"2024-07-26T12:44:36.857403+0000","last_active":"2024-07-26T12:46:34.675740+0000","last_peered":"2024-07-26T12:46:34.675740+0000","last_clean":"2024-07-26T12:46:34.675740+0000","last_became_active":"2024-07-26T12:44:36.856980+0000","last_became_peered":"2024-07-26T12:44:36.856980+0000","last_unstale":"2024-07-26T12:46:34.675740+0000","last_undegraded":"2024-07-26T12:46:34.675740+0000","last_fullsized":"2024-07-26T12:46:34.675740+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:11:20.166383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.653669+0000","last_change":"2024-07-26T12:43:12.166827+0000","last_active":"2024-07-26T12:46:34.653669+0000","last_peered":"2024-07-26T12:46:34.653669+0000","last_clean":"2024-07-26T12:46:34.653669+0000","last_became_active":"2024-07-26T12:43:12.165222+0000","last_became_peered":"2024-07-26T12:43:12.165222+0000","last_unstale":"2024-07-26T12:46:34.653669+0000","last_undegraded":"2024-07-26T12:46:34.653669+0000","last_fullsized":"2024-07-26T12:46:34.653669+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:37:42.219844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.663059+0000","last_change":"2024-07-26T12:43:12.176961+0000","last_active":"2024-07-26T12:46:34.663059+0000","last_peered":"2024-07-26T12:46:34.663059+0000","last_clean":"2024-07-26T12:46:34.663059+0000","last_became_active":"2024-07-26T12:43:12.176550+0000","last_became_peered":"2024-07-26T12:43:12.176550+0000","last_unstale":"2024-07-26T12:46:34.663059+0000","last_undegraded":"2024-07-26T12:46:34.663059+0000","last_fullsized":"2024-07-26T12:46:34.663059+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:36:19.720801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650547+0000","last_change":"2024-07-26T12:43:12.173560+0000","last_active":"2024-07-26T12:46:34.650547+0000","last_peered":"2024-07-26T12:46:34.650547+0000","last_clean":"2024-07-26T12:46:34.650547+0000","last_became_active":"2024-07-26T12:43:12.173187+0000","last_became_peered":"2024-07-26T12:43:12.173187+0000","last_unstale":"2024-07-26T12:46:34.650547+0000","last_undegraded":"2024-07-26T12:46:34.650547+0000","last_fullsized":"2024-07-26T12:46:34.650547+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:37:53.929210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655516+0000","last_change":"2024-07-26T12:43:12.169829+0000","last_active":"2024-07-26T12:46:34.655516+0000","last_peered":"2024-07-26T12:46:34.655516+0000","last_clean":"2024-07-26T12:46:34.655516+0000","last_became_active":"2024-07-26T12:43:12.169444+0000","last_became_peered":"2024-07-26T12:43:12.169444+0000","last_unstale":"2024-07-26T12:46:34.655516+0000","last_undegraded":"2024-07-26T12:46:34.655516+0000","last_fullsized":"2024-07-26T12:46:34.655516+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:29.111483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.671995+0000","last_change":"2024-07-26T12:43:12.171857+0000","last_active":"2024-07-26T12:46:34.671995+0000","last_peered":"2024-07-26T12:46:34.671995+0000","last_clean":"2024-07-26T12:46:34.671995+0000","last_became_active":"2024-07-26T12:43:12.170752+0000","last_became_peered":"2024-07-26T12:43:12.170752+0000","last_unstale":"2024-07-26T12:46:34.671995+0000","last_undegraded":"2024-07-26T12:46:34.671995+0000","last_fullsized":"2024-07-26T12:46:34.671995+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:22:09.442916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.657139+0000","last_change":"2024-07-26T12:43:12.178646+0000","last_active":"2024-07-26T12:46:34.657139+0000","last_peered":"2024-07-26T12:46:34.657139+0000","last_clean":"2024-07-26T12:46:34.657139+0000","last_became_active":"2024-07-26T12:43:12.178259+0000","last_became_peered":"2024-07-26T12:43:12.178259+0000","last_unstale":"2024-07-26T12:46:34.657139+0000","last_undegraded":"2024-07-26T12:46:34.657139+0000","last_fullsized":"2024-07-26T12:46:34.657139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:30:59.417843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.655755+0000","last_change":"2024-07-26T12:43:12.164579+0000","last_active":"2024-07-26T12:46:34.655755+0000","last_peered":"2024-07-26T12:46:34.655755+0000","last_clean":"2024-07-26T12:46:34.655755+0000","last_became_active":"2024-07-26T12:43:12.164099+0000","last_became_peered":"2024-07-26T12:43:12.164099+0000","last_unstale":"2024-07-26T12:46:34.655755+0000","last_undegraded":"2024-07-26T12:46:34.655755+0000","last_fullsized":"2024-07-26T12:46:34.655755+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:51:45.293190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.664174+0000","last_change":"2024-07-26T12:43:12.180838+0000","last_active":"2024-07-26T12:46:34.664174+0000","last_peered":"2024-07-26T12:46:34.664174+0000","last_clean":"2024-07-26T12:46:34.664174+0000","last_became_active":"2024-07-26T12:43:12.180543+0000","last_became_peered":"2024-07-26T12:43:12.180543+0000","last_unstale":"2024-07-26T12:46:34.664174+0000","last_undegraded":"2024-07-26T12:46:34.664174+0000","last_fullsized":"2024-07-26T12:46:34.664174+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:53:27.185082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.650139+0000","last_change":"2024-07-26T12:43:12.167235+0000","last_active":"2024-07-26T12:46:34.650139+0000","last_peered":"2024-07-26T12:46:34.650139+0000","last_clean":"2024-07-26T12:46:34.650139+0000","last_became_active":"2024-07-26T12:43:12.166489+0000","last_became_peered":"2024-07-26T12:43:12.166489+0000","last_unstale":"2024-07-26T12:46:34.650139+0000","last_undegraded":"2024-07-26T12:46:34.650139+0000","last_fullsized":"2024-07-26T12:46:34.650139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:13.868870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.668862+0000","last_change":"2024-07-26T12:43:12.176717+0000","last_active":"2024-07-26T12:46:34.668862+0000","last_peered":"2024-07-26T12:46:34.668862+0000","last_clean":"2024-07-26T12:46:34.668862+0000","last_became_active":"2024-07-26T12:43:12.172699+0000","last_became_peered":"2024-07-26T12:43:12.172699+0000","last_unstale":"2024-07-26T12:46:34.668862+0000","last_undegraded":"2024-07-26T12:46:34.668862+0000","last_fullsized":"2024-07-26T12:46:34.668862+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:23:36.853999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.669127+0000","last_change":"2024-07-26T12:43:12.177093+0000","last_active":"2024-07-26T12:46:34.669127+0000","last_peered":"2024-07-26T12:46:34.669127+0000","last_clean":"2024-07-26T12:46:34.669127+0000","last_became_active":"2024-07-26T12:43:12.176693+0000","last_became_peered":"2024-07-26T12:43:12.176693+0000","last_unstale":"2024-07-26T12:46:34.669127+0000","last_undegraded":"2024-07-26T12:46:34.669127+0000","last_fullsized":"2024-07-26T12:46:34.669127+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:19:56.875143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T12:46:34.660186+0000","last_change":"2024-07-26T12:43:12.189531+0000","last_active":"2024-07-26T12:46:34.660186+0000","last_peered":"2024-07-26T12:46:34.660186+0000","last_clean":"2024-07-26T12:46:34.660186+0000","last_became_active":"2024-07-26T12:43:12.189162+0000","last_became_peered":"2024-07-26T12:43:12.189162+0000","last_unstale":"2024-07-26T12:46:34.660186+0000","last_undegraded":"2024-07-26T12:46:34.660186+0000","last_fullsized":"2024-07-26T12:46:34.660186+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T12:43:11.133406+0000","last_clean_scrub_stamp":"2024-07-26T12:43:11.133406+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:11:00.160614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":77551,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2485,"num_read_kb":1327,"num_write":265,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":232653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678099,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":536466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874328,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37040,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708112,"statfs":{"total":95995035648,"available":95957106688,"internally_reserved":0,"allocated":1744896,"data_stored":1187374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037852,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":539892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234082,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":536174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430311,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36964,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708188,"statfs":{"total":95995035648,"available":95957184512,"internally_reserved":0,"allocated":1667072,"data_stored":1132812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593837,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":531300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790068,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1675264,"data_stored":1128394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953593,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":535374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":531008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378755,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1134592,"data_stored":603611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542282,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":602395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":533288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":73033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":71801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T12:48:47.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:48:47.967 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:47 smithi112 ceph-mon[37501]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:48:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:47 smithi107 ceph-mon[32659]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:48:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:47 smithi132 ceph-mon[37580]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T12:48:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:48 smithi112 ceph-mon[37501]: pgmap v475: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 766 B/s wr, 11 op/s 2024-07-26T12:48:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:48 smithi107 ceph-mon[32659]: pgmap v475: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 766 B/s wr, 11 op/s 2024-07-26T12:48:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:48 smithi132 ceph-mon[37580]: pgmap v475: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 766 B/s wr, 11 op/s 2024-07-26T12:48:50.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.474+0000 7fea14e00640 1 -- 172.21.15.107:0/3896562422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea1010fc80 msgr2=0x7fea101164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:50.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.474+0000 7fea14e00640 1 --2- 172.21.15.107:0/3896562422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea1010fc80 0x7fea101164b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fea08009ca0 tx=0x7fea0802a8c0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.475+0000 7fea14e00640 1 -- 172.21.15.107:0/3896562422 shutdown_connections 2024-07-26T12:48:50.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.475+0000 7fea14e00640 1 --2- 172.21.15.107:0/3896562422 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea1010fc80 0x7fea101164b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.475+0000 7fea14e00640 1 --2- 172.21.15.107:0/3896562422 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea1010f330 0x7fea1010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.475+0000 7fea14e00640 1 --2- 172.21.15.107:0/3896562422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea101099f0 0x7fea10109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.475+0000 7fea14e00640 1 -- 172.21.15.107:0/3896562422 >> 172.21.15.107:0/3896562422 conn(0x7fea10078ff0 msgr2=0x7fea10107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:50.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.475+0000 7fea14e00640 1 -- 172.21.15.107:0/3896562422 shutdown_connections 2024-07-26T12:48:50.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.475+0000 7fea14e00640 1 -- 172.21.15.107:0/3896562422 wait complete. 2024-07-26T12:48:50.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.476+0000 7fea14e00640 1 Processor -- start 2024-07-26T12:48:50.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.476+0000 7fea14e00640 1 -- start start 2024-07-26T12:48:50.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.476+0000 7fea14e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea101099f0 0x7fea101a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:50.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.476+0000 7fea14e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea1010f330 0x7fea101a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.476+0000 7fea14e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea1010fc80 0x7fea101a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.476+0000 7fea14e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea10119990 con 0x7fea101099f0 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.476+0000 7fea14e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fea10119810 con 0x7fea1010fc80 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.476+0000 7fea14e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fea10119b10 con 0x7fea1010f330 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea0fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea1010fc80 0x7fea101a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea0fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea1010fc80 0x7fea101a5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:38976/0 (socket says 172.21.15.107:38976) 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea0f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea101099f0 0x7fea101a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea0fe00640 1 -- 172.21.15.107:0/512569716 learned_addr learned my addr 172.21.15.107:0/512569716 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea07e00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea1010f330 0x7fea101a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea0f400640 1 -- 172.21.15.107:0/512569716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea1010f330 msgr2=0x7fea101a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:50.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea0f400640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea1010f330 0x7fea101a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea0f400640 1 -- 172.21.15.107:0/512569716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea1010fc80 msgr2=0x7fea101a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:50.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea0f400640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea1010fc80 0x7fea101a5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea0f400640 1 -- 172.21.15.107:0/512569716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea101a6210 con 0x7fea101099f0 2024-07-26T12:48:50.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea07e00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea1010f330 0x7fea101a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:50.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.477+0000 7fea0fe00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea1010fc80 0x7fea101a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:50.480 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.478+0000 7fea0f400640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea101099f0 0x7fea101a02a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fea0000e9f0 tx=0x7fea0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.478+0000 7fea0cc00640 1 -- 172.21.15.107:0/512569716 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0000ccc0 con 0x7fea101099f0 2024-07-26T12:48:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.478+0000 7fea14e00640 1 -- 172.21.15.107:0/512569716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea101a6500 con 0x7fea101099f0 2024-07-26T12:48:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.478+0000 7fea0cc00640 1 -- 172.21.15.107:0/512569716 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea00005020 con 0x7fea101099f0 2024-07-26T12:48:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.478+0000 7fea0cc00640 1 -- 172.21.15.107:0/512569716 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00004a00 con 0x7fea101099f0 2024-07-26T12:48:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.478+0000 7fea14e00640 1 -- 172.21.15.107:0/512569716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea101acc60 con 0x7fea101099f0 2024-07-26T12:48:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.479+0000 7fea14e00640 1 -- 172.21.15.107:0/512569716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea10109e00 con 0x7fea101099f0 2024-07-26T12:48:50.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.482+0000 7fea0cc00640 1 -- 172.21.15.107:0/512569716 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fea00004ba0 con 0x7fea101099f0 2024-07-26T12:48:50.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.483+0000 7fea0cc00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe9f40b3c20 0x7fe9f40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:50.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.484+0000 7fea0cc00640 1 -- 172.21.15.107:0/512569716 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fea000e40d0 con 0x7fea101099f0 2024-07-26T12:48:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.484+0000 7fea07e00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe9f40b3c20 0x7fe9f40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:50.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.485+0000 7fea07e00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe9f40b3c20 0x7fe9f40b6130 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fea1010fac0 tx=0x7fe9f8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:50.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.488+0000 7fea0cc00640 1 -- 172.21.15.107:0/512569716 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fea000ab220 con 0x7fea101099f0 2024-07-26T12:48:50.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.726+0000 7fea14e00640 1 -- 172.21.15.107:0/512569716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea1002e0c0 con 0x7fea101099f0 2024-07-26T12:48:50.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.727+0000 7fea0cc00640 1 -- 172.21.15.107:0/512569716 <== mon.0 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11993 (secure 0 0 0) 0x7fea000aab20 con 0x7fea101099f0 2024-07-26T12:48:50.729 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:48:50.729 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"btime":"2024-07-26T12:48:38:656386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:48:38.606034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24716,"mds_1":24715,"mds_2":24703,"mds_3":24694,"mds_4":24682},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24632":{"gid":24632,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24634":{"gid":24634,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24652":{"gid":24652,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24662":{"gid":24662,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":32,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24694":{"gid":24694,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":31,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":29,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":28,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24716,"qdb_cluster":[24715,24716,24703,24694,24682]},"id":1}]} 2024-07-26T12:48:50.729 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-07-26T12:48:50.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 -- 172.21.15.107:0/512569716 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe9f40b3c20 msgr2=0x7fe9f40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:50.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe9f40b3c20 0x7fe9f40b6130 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fea1010fac0 tx=0x7fe9f8005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:50.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 -- 172.21.15.107:0/512569716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea101099f0 msgr2=0x7fea101a02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:50.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea101099f0 0x7fea101a02a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fea0000e9f0 tx=0x7fea0000eeb0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.732 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 -- 172.21.15.107:0/512569716 shutdown_connections 2024-07-26T12:48:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe9f40b3c20 0x7fe9f40b6130 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fea1010fc80 0x7fea101a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea1010f330 0x7fea101a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 --2- 172.21.15.107:0/512569716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea101099f0 0x7fea101a02a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 -- 172.21.15.107:0/512569716 >> 172.21.15.107:0/512569716 conn(0x7fea10078ff0 msgr2=0x7fea10079650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 -- 172.21.15.107:0/512569716 shutdown_connections 2024-07-26T12:48:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:50.731+0000 7fea14e00640 1 -- 172.21.15.107:0/512569716 wait complete. 2024-07-26T12:48:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:50 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/512569716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:50 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/512569716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:50 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/512569716' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:48:51.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T12:48:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:51 smithi107 ceph-mon[32659]: pgmap v476: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 766 B/s wr, 9 op/s 2024-07-26T12:48:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:51 smithi132 ceph-mon[37580]: pgmap v476: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 766 B/s wr, 9 op/s 2024-07-26T12:48:52.171 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:51 smithi112 ceph-mon[37501]: pgmap v476: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 766 B/s wr, 9 op/s 2024-07-26T12:48:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:52 smithi107 ceph-mon[32659]: pgmap v477: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 766 B/s wr, 11 op/s 2024-07-26T12:48:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:52 smithi112 ceph-mon[37501]: pgmap v477: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 766 B/s wr, 11 op/s 2024-07-26T12:48:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:52 smithi132 ceph-mon[37580]: pgmap v477: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 766 B/s wr, 11 op/s 2024-07-26T12:48:54.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.616+0000 7fe073e00640 1 -- 172.21.15.107:0/976823606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe074109670 msgr2=0x7fe07410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:54.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.616+0000 7fe073e00640 1 --2- 172.21.15.107:0/976823606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe074109670 0x7fe07410fea0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe06c009ca0 tx=0x7fe06c02a8c0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.616+0000 7fe073e00640 1 -- 172.21.15.107:0/976823606 shutdown_connections 2024-07-26T12:48:54.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.616+0000 7fe073e00640 1 --2- 172.21.15.107:0/976823606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe074109670 0x7fe07410fea0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.616+0000 7fe073e00640 1 --2- 172.21.15.107:0/976823606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe074108d20 0x7fe074109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.616+0000 7fe073e00640 1 --2- 172.21.15.107:0/976823606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe0741033e0 0x7fe0741037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.616+0000 7fe073e00640 1 -- 172.21.15.107:0/976823606 >> 172.21.15.107:0/976823606 conn(0x7fe0740fee70 msgr2=0x7fe0741012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:54.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.616+0000 7fe073e00640 1 -- 172.21.15.107:0/976823606 shutdown_connections 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.616+0000 7fe073e00640 1 -- 172.21.15.107:0/976823606 wait complete. 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.617+0000 7fe073e00640 1 Processor -- start 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.617+0000 7fe073e00640 1 -- start start 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe073e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0741033e0 0x7fe074199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe073e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe074108d20 0x7fe07419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe073e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe074109670 0x7fe07419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe073e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe074113320 con 0x7fe074108d20 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe073e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fe0741131a0 con 0x7fe074109670 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe073e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fe0741134a0 con 0x7fe0741033e0 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe072a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0741033e0 0x7fe074199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe072000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe074108d20 0x7fe07419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe072a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0741033e0 0x7fe074199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:57942/0 (socket says 172.21.15.107:57942) 2024-07-26T12:48:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe072a00640 1 -- 172.21.15.107:0/4210935579 learned_addr learned my addr 172.21.15.107:0/4210935579 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:54.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.618+0000 7fe073400640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe074109670 0x7fe07419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:54.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.619+0000 7fe072a00640 1 -- 172.21.15.107:0/4210935579 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe074109670 msgr2=0x7fe07419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:54.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.619+0000 7fe072a00640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe074109670 0x7fe07419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.619+0000 7fe072a00640 1 -- 172.21.15.107:0/4210935579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe074108d20 msgr2=0x7fe07419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:54.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.619+0000 7fe072a00640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe074108d20 0x7fe07419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.619+0000 7fe072a00640 1 -- 172.21.15.107:0/4210935579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe07419fc20 con 0x7fe0741033e0 2024-07-26T12:48:54.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.619+0000 7fe072a00640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0741033e0 0x7fe074199cb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00e9f0 tx=0x7fe05c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:54.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.619+0000 7fe06b400640 1 -- 172.21.15.107:0/4210935579 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c00cd80 con 0x7fe0741033e0 2024-07-26T12:48:54.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.619+0000 7fe06b400640 1 -- 172.21.15.107:0/4210935579 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe05c0108e0 con 0x7fe0741033e0 2024-07-26T12:48:54.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.619+0000 7fe073e00640 1 -- 172.21.15.107:0/4210935579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe07419ff10 con 0x7fe0741033e0 2024-07-26T12:48:54.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.619+0000 7fe06b400640 1 -- 172.21.15.107:0/4210935579 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe05c018980 con 0x7fe0741033e0 2024-07-26T12:48:54.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.620+0000 7fe073e00640 1 -- 172.21.15.107:0/4210935579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0741a6670 con 0x7fe0741033e0 2024-07-26T12:48:54.623 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.621+0000 7fe073e00640 1 -- 172.21.15.107:0/4210935579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0741037f0 con 0x7fe0741033e0 2024-07-26T12:48:54.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.623+0000 7fe06b400640 1 -- 172.21.15.107:0/4210935579 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fe05c010430 con 0x7fe0741033e0 2024-07-26T12:48:54.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.625+0000 7fe06b400640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe0580b3c20 0x7fe0580b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:54.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.625+0000 7fe06b400640 1 -- 172.21.15.107:0/4210935579 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe05c0e49e0 con 0x7fe0741033e0 2024-07-26T12:48:54.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.625+0000 7fe072000640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe0580b3c20 0x7fe0580b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:54.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.626+0000 7fe072000640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe0580b3c20 0x7fe0580b6130 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe074108820 tx=0x7fe064005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:54.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.626+0000 7fe06b400640 1 -- 172.21.15.107:0/4210935579 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe05c0abb30 con 0x7fe0741033e0 2024-07-26T12:48:54.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.832+0000 7fe073e00640 1 -- 172.21.15.107:0/4210935579 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe07402e0c0 con 0x7fe0741033e0 2024-07-26T12:48:54.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.834+0000 7fe06b400640 1 -- 172.21.15.107:0/4210935579 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25203 (secure 0 0 0) 0x7fe05c0ab430 con 0x7fe0741033e0 2024-07-26T12:48:54.836 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:48:54.836 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":90,"fsid":"14de5e40-4b4b-11ef-bcb7-c7b262605968","created":"2024-07-26T12:33:09.598433+0000","modified":"2024-07-26T12:44:36.797948+0000","last_up_change":"2024-07-26T12:42:05.518580+0000","last_in_change":"2024-07-26T12:41:46.421114+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T12:37:58.631844+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T12:43:07.012156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T12:43:10.809864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T12:43:24.750177+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9f413e43-6b6b-4596-ab6e-c75cdf9c7c96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6803","nonce":2354859122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6805","nonce":2354859122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6809","nonce":2354859122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2354859122},{"type":"v1","addr":"172.21.15.107:6807","nonce":2354859122}]},"public_addr":"172.21.15.107:6803/2354859122","cluster_addr":"172.21.15.107:6805/2354859122","heartbeat_back_addr":"172.21.15.107:6809/2354859122","heartbeat_front_addr":"172.21.15.107:6807/2354859122","state":["exists","up"]},{"osd":1,"uuid":"44ac0320-aac0-4810-ad34-6798956f1969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6801","nonce":3125120524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6803","nonce":3125120524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6807","nonce":3125120524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":3125120524},{"type":"v1","addr":"172.21.15.112:6805","nonce":3125120524}]},"public_addr":"172.21.15.112:6801/3125120524","cluster_addr":"172.21.15.112:6803/3125120524","heartbeat_back_addr":"172.21.15.112:6807/3125120524","heartbeat_front_addr":"172.21.15.112:6805/3125120524","state":["exists","up"]},{"osd":2,"uuid":"538267b7-07cf-4be0-95bf-bf9e8a4005f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6800","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6801","nonce":28260456}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6802","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6803","nonce":28260456}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6806","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6807","nonce":28260456}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6804","nonce":28260456},{"type":"v1","addr":"172.21.15.132:6805","nonce":28260456}]},"public_addr":"172.21.15.132:6801/28260456","cluster_addr":"172.21.15.132:6803/28260456","heartbeat_back_addr":"172.21.15.132:6807/28260456","heartbeat_front_addr":"172.21.15.132:6805/28260456","state":["exists","up"]},{"osd":3,"uuid":"914bea43-b9ac-4bed-817e-6af4d1d0aa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6811","nonce":1620179989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6813","nonce":1620179989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6817","nonce":1620179989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1620179989},{"type":"v1","addr":"172.21.15.107:6815","nonce":1620179989}]},"public_addr":"172.21.15.107:6811/1620179989","cluster_addr":"172.21.15.107:6813/1620179989","heartbeat_back_addr":"172.21.15.107:6817/1620179989","heartbeat_front_addr":"172.21.15.107:6815/1620179989","state":["exists","up"]},{"osd":4,"uuid":"bc83673b-3617-4245-bfe1-b61c9cbe2ab4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6809","nonce":95536584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6811","nonce":95536584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6815","nonce":95536584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":95536584},{"type":"v1","addr":"172.21.15.112:6813","nonce":95536584}]},"public_addr":"172.21.15.112:6809/95536584","cluster_addr":"172.21.15.112:6811/95536584","heartbeat_back_addr":"172.21.15.112:6815/95536584","heartbeat_front_addr":"172.21.15.112:6813/95536584","state":["exists","up"]},{"osd":5,"uuid":"5d0394cf-c561-4a68-b674-92ea81d9ffe1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6808","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6809","nonce":1054197844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6810","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6811","nonce":1054197844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6814","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6815","nonce":1054197844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6812","nonce":1054197844},{"type":"v1","addr":"172.21.15.132:6813","nonce":1054197844}]},"public_addr":"172.21.15.132:6809/1054197844","cluster_addr":"172.21.15.132:6811/1054197844","heartbeat_back_addr":"172.21.15.132:6815/1054197844","heartbeat_front_addr":"172.21.15.132:6813/1054197844","state":["exists","up"]},{"osd":6,"uuid":"17052119-63ed-49e0-91a7-1e888c82dee2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6819","nonce":2132739845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6821","nonce":2132739845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6825","nonce":2132739845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2132739845},{"type":"v1","addr":"172.21.15.107:6823","nonce":2132739845}]},"public_addr":"172.21.15.107:6819/2132739845","cluster_addr":"172.21.15.107:6821/2132739845","heartbeat_back_addr":"172.21.15.107:6825/2132739845","heartbeat_front_addr":"172.21.15.107:6823/2132739845","state":["exists","up"]},{"osd":7,"uuid":"4a8f4e83-a1d2-40e2-b0f9-5b2b776b6f53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6817","nonce":3770813146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6819","nonce":3770813146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6823","nonce":3770813146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":3770813146},{"type":"v1","addr":"172.21.15.112:6821","nonce":3770813146}]},"public_addr":"172.21.15.112:6817/3770813146","cluster_addr":"172.21.15.112:6819/3770813146","heartbeat_back_addr":"172.21.15.112:6823/3770813146","heartbeat_front_addr":"172.21.15.112:6821/3770813146","state":["exists","up"]},{"osd":8,"uuid":"4d3958e5-fe71-448b-a6de-6bcc423daca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6816","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6817","nonce":2677294944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6818","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6819","nonce":2677294944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6822","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6823","nonce":2677294944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6820","nonce":2677294944},{"type":"v1","addr":"172.21.15.132:6821","nonce":2677294944}]},"public_addr":"172.21.15.132:6817/2677294944","cluster_addr":"172.21.15.132:6819/2677294944","heartbeat_back_addr":"172.21.15.132:6823/2677294944","heartbeat_front_addr":"172.21.15.132:6821/2677294944","state":["exists","up"]},{"osd":9,"uuid":"6d641864-0787-4a2b-ae73-e590c1b5c0d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6827","nonce":3840896542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6829","nonce":3840896542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6833","nonce":3840896542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3840896542},{"type":"v1","addr":"172.21.15.107:6831","nonce":3840896542}]},"public_addr":"172.21.15.107:6827/3840896542","cluster_addr":"172.21.15.107:6829/3840896542","heartbeat_back_addr":"172.21.15.107:6833/3840896542","heartbeat_front_addr":"172.21.15.107:6831/3840896542","state":["exists","up"]},{"osd":10,"uuid":"d5cdd3a2-311a-4c36-8553-3e86dad57015","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6825","nonce":3415013149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6826","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6827","nonce":3415013149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6830","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6831","nonce":3415013149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6828","nonce":3415013149},{"type":"v1","addr":"172.21.15.112:6829","nonce":3415013149}]},"public_addr":"172.21.15.112:6825/3415013149","cluster_addr":"172.21.15.112:6827/3415013149","heartbeat_back_addr":"172.21.15.112:6831/3415013149","heartbeat_front_addr":"172.21.15.112:6829/3415013149","state":["exists","up"]},{"osd":11,"uuid":"d738c87c-2b11-4ba4-b38d-a58e2fd9a235","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6824","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6825","nonce":3905824310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6826","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6827","nonce":3905824310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6830","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6831","nonce":3905824310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6828","nonce":3905824310},{"type":"v1","addr":"172.21.15.132:6829","nonce":3905824310}]},"public_addr":"172.21.15.132:6825/3905824310","cluster_addr":"172.21.15.132:6827/3905824310","heartbeat_back_addr":"172.21.15.132:6831/3905824310","heartbeat_front_addr":"172.21.15.132:6829/3905824310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:10.010602+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:33.867192+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:37:56.733771+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:29.561351+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:38:54.398545+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:17.735988+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:39:50.540612+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:15.991827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:40:40.263297+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:14.053735+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:41:39.419029+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T12:42:03.949027+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":7}]},{"pgid":"3.5","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.2a","mappings":[{"from":3,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.11","mappings":[{"from":5,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3311144690":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/81377212":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/4257991258":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6801/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:0/312154060":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/1797109211":"2024-07-27T12:33:46.111763+0000","172.21.15.107:0/670053235":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/453757265":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6800/2773605049":"2024-07-27T12:35:34.485438+0000","172.21.15.107:6800/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/4171514587":"2024-07-27T12:34:14.657397+0000","172.21.15.107:6800/634100778":"2024-07-27T12:33:46.111763+0000","172.21.15.107:6801/425164471":"2024-07-27T12:34:14.657397+0000","172.21.15.107:0/2624080873":"2024-07-27T12:34:14.657397+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T12:48:54.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.838+0000 7fe073e00640 1 -- 172.21.15.107:0/4210935579 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe0580b3c20 msgr2=0x7fe0580b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:54.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.838+0000 7fe073e00640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe0580b3c20 0x7fe0580b6130 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe074108820 tx=0x7fe064005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.838+0000 7fe073e00640 1 -- 172.21.15.107:0/4210935579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0741033e0 msgr2=0x7fe074199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.838+0000 7fe073e00640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0741033e0 0x7fe074199cb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00e9f0 tx=0x7fe05c00eeb0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.838+0000 7fe073e00640 1 -- 172.21.15.107:0/4210935579 shutdown_connections 2024-07-26T12:48:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.838+0000 7fe073e00640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fe0580b3c20 0x7fe0580b6130 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.838+0000 7fe073e00640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fe074109670 0x7fe07419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.838+0000 7fe073e00640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe074108d20 0x7fe07419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.838+0000 7fe073e00640 1 --2- 172.21.15.107:0/4210935579 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0741033e0 0x7fe074199cb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.838+0000 7fe073e00640 1 -- 172.21.15.107:0/4210935579 >> 172.21.15.107:0/4210935579 conn(0x7fe0740fee70 msgr2=0x7fe074101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.839+0000 7fe073e00640 1 -- 172.21.15.107:0/4210935579 shutdown_connections 2024-07-26T12:48:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:54.839+0000 7fe073e00640 1 -- 172.21.15.107:0/4210935579 wait complete. 2024-07-26T12:48:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:54 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/4210935579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:48:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:54 smithi112 ceph-mon[37501]: pgmap v478: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T12:48:55.246 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:54 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/4210935579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:48:55.246 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:54 smithi107 ceph-mon[32659]: pgmap v478: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T12:48:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:54 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/4210935579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T12:48:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:54 smithi132 ceph-mon[37580]: pgmap v478: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T12:48:55.786 DEBUG:tasks.fwd_scrub:Yielding 2024-07-26T12:48:55.786 INFO:teuthology.run_tasks:Running task exec... 2024-07-26T12:48:55.799 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T12:48:55.799 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-07-26T12:48:55.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-07-26T12:48:55.799 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-07-26T12:48:55.799 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T12:48:55.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T12:48:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.064+0000 7fa3abc00640 1 -- 172.21.15.107:0/2281745684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3a4104350 msgr2=0x7fa3a4104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:56.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.064+0000 7fa3abc00640 1 --2- 172.21.15.107:0/2281745684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3a4104350 0x7fa3a4104760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa398009900 tx=0x7fa39802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.064+0000 7fa3abc00640 1 -- 172.21.15.107:0/2281745684 shutdown_connections 2024-07-26T12:48:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.064+0000 7fa3abc00640 1 --2- 172.21.15.107:0/2281745684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a410d7a0 0x7fa3a410fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.064+0000 7fa3abc00640 1 --2- 172.21.15.107:0/2281745684 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3a4104ca0 0x7fa3a410d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.064+0000 7fa3abc00640 1 --2- 172.21.15.107:0/2281745684 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3a4104350 0x7fa3a4104760 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.064+0000 7fa3abc00640 1 -- 172.21.15.107:0/2281745684 >> 172.21.15.107:0/2281745684 conn(0x7fa3a407a120 msgr2=0x7fa3a407a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.064+0000 7fa3abc00640 1 -- 172.21.15.107:0/2281745684 shutdown_connections 2024-07-26T12:48:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.064+0000 7fa3abc00640 1 -- 172.21.15.107:0/2281745684 wait complete. 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3abc00640 1 Processor -- start 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3abc00640 1 -- start start 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3abc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3a4104350 0x7fa3a419c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3abc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3a4104ca0 0x7fa3a419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3abc00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a410d7a0 0x7fa3a41a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3abc00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a4113460 con 0x7fa3a410d7a0 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3abc00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a41132e0 con 0x7fa3a4104350 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3abc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a41135e0 con 0x7fa3a4104ca0 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3a3e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3a4104ca0 0x7fa3a419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3a9c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a410d7a0 0x7fa3a41a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3a9200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3a4104350 0x7fa3a419c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.066+0000 7fa3a9200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3a4104350 0x7fa3a419c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:36150/0 (socket says 172.21.15.107:36150) 2024-07-26T12:48:56.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.067+0000 7fa3a9200640 1 -- 172.21.15.107:0/2608625328 learned_addr learned my addr 172.21.15.107:0/2608625328 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.067+0000 7fa3a9200640 1 -- 172.21.15.107:0/2608625328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3a4104ca0 msgr2=0x7fa3a419c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.067+0000 7fa3a9200640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3a4104ca0 0x7fa3a419c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.067+0000 7fa3a9200640 1 -- 172.21.15.107:0/2608625328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a410d7a0 msgr2=0x7fa3a41a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.067+0000 7fa3a9200640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a410d7a0 0x7fa3a41a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.067+0000 7fa3a9200640 1 -- 172.21.15.107:0/2608625328 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a41a1f70 con 0x7fa3a4104350 2024-07-26T12:48:56.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.067+0000 7fa3a9200640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3a4104350 0x7fa3a419c040 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa398005ab0 tx=0x7fa398003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:56.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.068+0000 7fa3a1600640 1 -- 172.21.15.107:0/2608625328 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39803f070 con 0x7fa3a4104350 2024-07-26T12:48:56.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.068+0000 7fa3abc00640 1 -- 172.21.15.107:0/2608625328 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a41a2200 con 0x7fa3a4104350 2024-07-26T12:48:56.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.068+0000 7fa3abc00640 1 -- 172.21.15.107:0/2608625328 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a41a89f0 con 0x7fa3a4104350 2024-07-26T12:48:56.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.068+0000 7fa3a1600640 1 -- 172.21.15.107:0/2608625328 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa398031d60 con 0x7fa3a4104350 2024-07-26T12:48:56.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.068+0000 7fa3a1600640 1 -- 172.21.15.107:0/2608625328 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398049060 con 0x7fa3a4104350 2024-07-26T12:48:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.069+0000 7fa3abc00640 1 -- 172.21.15.107:0/2608625328 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3840052c0 con 0x7fa3a4104350 2024-07-26T12:48:56.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.071+0000 7fa3a1600640 1 -- 172.21.15.107:0/2608625328 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fa398042070 con 0x7fa3a4104350 2024-07-26T12:48:56.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.073+0000 7fa3a1600640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa38c0b3c20 0x7fa38c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:56.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.073+0000 7fa3a1600640 1 -- 172.21.15.107:0/2608625328 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa3981045c0 con 0x7fa3a4104350 2024-07-26T12:48:56.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.073+0000 7fa3a3e00640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa38c0b3c20 0x7fa38c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:56.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.074+0000 7fa3a3e00640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa38c0b3c20 0x7fa38c0b6130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa3a410f5a0 tx=0x7fa390005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:56.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.075+0000 7fa3a1600640 1 -- 172.21.15.107:0/2608625328 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa3980cb0c0 con 0x7fa3a4104350 2024-07-26T12:48:56.256 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:48:56.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.334+0000 7fa3abc00640 1 -- 172.21.15.107:0/2608625328 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fa384000bf0 con 0x7fa38c0b3c20 2024-07-26T12:48:56.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.644+0000 7fa3a1600640 1 -- 172.21.15.107:0/2608625328 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa384000bf0 con 0x7fa38c0b3c20 2024-07-26T12:48:56.645 INFO:teuthology.orchestra.run.smithi107.stdout:Schedule set for path / 2024-07-26T12:48:56.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.647+0000 7fa3abc00640 1 -- 172.21.15.107:0/2608625328 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa38c0b3c20 msgr2=0x7fa38c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:56.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.647+0000 7fa3abc00640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa38c0b3c20 0x7fa38c0b6130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa3a410f5a0 tx=0x7fa390005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:56.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.647+0000 7fa3abc00640 1 -- 172.21.15.107:0/2608625328 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3a4104350 msgr2=0x7fa3a419c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:56.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.647+0000 7fa3abc00640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3a4104350 0x7fa3a419c040 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa398005ab0 tx=0x7fa398003c80 comp rx=0 tx=0).stop 2024-07-26T12:48:56.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.647+0000 7fa3abc00640 1 -- 172.21.15.107:0/2608625328 shutdown_connections 2024-07-26T12:48:56.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.647+0000 7fa3abc00640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa38c0b3c20 0x7fa38c0b6130 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.647+0000 7fa3abc00640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3a410d7a0 0x7fa3a41a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.647+0000 7fa3abc00640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa3a4104ca0 0x7fa3a419c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.647+0000 7fa3abc00640 1 --2- 172.21.15.107:0/2608625328 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa3a4104350 0x7fa3a419c040 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.648+0000 7fa3abc00640 1 -- 172.21.15.107:0/2608625328 >> 172.21.15.107:0/2608625328 conn(0x7fa3a407a120 msgr2=0x7fa3a41005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:56.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.648+0000 7fa3abc00640 1 -- 172.21.15.107:0/2608625328 shutdown_connections 2024-07-26T12:48:56.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.648+0000 7fa3abc00640 1 -- 172.21.15.107:0/2608625328 wait complete. 2024-07-26T12:48:56.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-07-26T12:48:56.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:56 smithi107 ceph-mon[32659]: from='client.25481 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:56.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.943+0000 7fbc36200640 1 -- 172.21.15.107:0/857555174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc3010ae70 msgr2=0x7fbc3010b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:56.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.943+0000 7fbc36200640 1 --2- 172.21.15.107:0/857555174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc3010ae70 0x7fbc3010b280 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009900 tx=0x7fbc1c02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.943+0000 7fbc36200640 1 -- 172.21.15.107:0/857555174 shutdown_connections 2024-07-26T12:48:56.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.943+0000 7fbc36200640 1 --2- 172.21.15.107:0/857555174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc3010b7c0 0x7fbc30111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.943+0000 7fbc36200640 1 --2- 172.21.15.107:0/857555174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc3010ae70 0x7fbc3010b280 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.943+0000 7fbc36200640 1 --2- 172.21.15.107:0/857555174 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc30105540 0x7fbc30105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.943+0000 7fbc36200640 1 -- 172.21.15.107:0/857555174 >> 172.21.15.107:0/857555174 conn(0x7fbc30100f20 msgr2=0x7fbc30103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:56.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.943+0000 7fbc36200640 1 -- 172.21.15.107:0/857555174 shutdown_connections 2024-07-26T12:48:56.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.943+0000 7fbc36200640 1 -- 172.21.15.107:0/857555174 wait complete. 2024-07-26T12:48:56.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.944+0000 7fbc36200640 1 Processor -- start 2024-07-26T12:48:56.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.944+0000 7fbc36200640 1 -- start start 2024-07-26T12:48:56.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc36200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc30105540 0x7fbc3019bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:56.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc36200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc3010ae70 0x7fbc3019c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:56.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc36200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc3010b7c0 0x7fbc301a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:56.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc36200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbc30115430 con 0x7fbc3010ae70 2024-07-26T12:48:56.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc36200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbc301152b0 con 0x7fbc3010b7c0 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc36200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbc301155b0 con 0x7fbc30105540 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc34e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc30105540 0x7fbc3019bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc34e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc30105540 0x7fbc3019bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:57998/0 (socket says 172.21.15.107:57998) 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc34e00640 1 -- 172.21.15.107:0/3854051244 learned_addr learned my addr 172.21.15.107:0/3854051244 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc2fe00640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc3010ae70 0x7fbc3019c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.945+0000 7fbc35800640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc3010b7c0 0x7fbc301a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.946+0000 7fbc34e00640 1 -- 172.21.15.107:0/3854051244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc3010b7c0 msgr2=0x7fbc301a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.946+0000 7fbc34e00640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc3010b7c0 0x7fbc301a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.946+0000 7fbc34e00640 1 -- 172.21.15.107:0/3854051244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc3010ae70 msgr2=0x7fbc3019c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.946+0000 7fbc34e00640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc3010ae70 0x7fbc3019c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:56.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.946+0000 7fbc34e00640 1 -- 172.21.15.107:0/3854051244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc301a1e00 con 0x7fbc30105540 2024-07-26T12:48:56.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.946+0000 7fbc34e00640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc30105540 0x7fbc3019bed0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400e9f0 tx=0x7fbc2400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:56.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.946+0000 7fbc2d600640 1 -- 172.21.15.107:0/3854051244 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2400cd80 con 0x7fbc30105540 2024-07-26T12:48:56.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.946+0000 7fbc2d600640 1 -- 172.21.15.107:0/3854051244 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc240108e0 con 0x7fbc30105540 2024-07-26T12:48:56.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.946+0000 7fbc36200640 1 -- 172.21.15.107:0/3854051244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc301a20f0 con 0x7fbc30105540 2024-07-26T12:48:56.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.946+0000 7fbc2d600640 1 -- 172.21.15.107:0/3854051244 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24018980 con 0x7fbc30105540 2024-07-26T12:48:56.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.947+0000 7fbc36200640 1 -- 172.21.15.107:0/3854051244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc301a8880 con 0x7fbc30105540 2024-07-26T12:48:56.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.948+0000 7fbc36200640 1 -- 172.21.15.107:0/3854051244 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc30105950 con 0x7fbc30105540 2024-07-26T12:48:56.950 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.950+0000 7fbc2d600640 1 -- 172.21.15.107:0/3854051244 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fbc24010430 con 0x7fbc30105540 2024-07-26T12:48:56.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.951+0000 7fbc2d600640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbc180b3ca0 0x7fbc180b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:56.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.951+0000 7fbc2d600640 1 -- 172.21.15.107:0/3854051244 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbc240e43f0 con 0x7fbc30105540 2024-07-26T12:48:56.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.951+0000 7fbc2fe00640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbc180b3ca0 0x7fbc180b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:56.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.952+0000 7fbc2fe00640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbc180b3ca0 0x7fbc180b61b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009230 tx=0x7fbc1c031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:56.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:56.955+0000 7fbc2d600640 1 -- 172.21.15.107:0/3854051244 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbc240aaef0 con 0x7fbc30105540 2024-07-26T12:48:57.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.195+0000 7fbc36200640 1 -- 172.21.15.107:0/3854051244 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fbc30063230 con 0x7fbc180b3ca0 2024-07-26T12:48:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:56 smithi112 ceph-mon[37501]: from='client.25481 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:56 smithi112 ceph-mon[37501]: pgmap v479: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:48:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:56 smithi107 ceph-mon[32659]: pgmap v479: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:48:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:56 smithi132 ceph-mon[37580]: from='client.25481 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:56 smithi132 ceph-mon[37580]: pgmap v479: 129 pgs: 129 active+clean; 652 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:48:57.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.346+0000 7fbc2d600640 1 -- 172.21.15.107:0/3854051244 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fbc30063230 con 0x7fbc180b3ca0 2024-07-26T12:48:57.347 INFO:teuthology.orchestra.run.smithi107.stdout:Retention added to path / 2024-07-26T12:48:57.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 -- 172.21.15.107:0/3854051244 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbc180b3ca0 msgr2=0x7fbc180b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:57.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbc180b3ca0 0x7fbc180b61b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009230 tx=0x7fbc1c031400 comp rx=0 tx=0).stop 2024-07-26T12:48:57.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 -- 172.21.15.107:0/3854051244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc30105540 msgr2=0x7fbc3019bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:57.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc30105540 0x7fbc3019bed0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400e9f0 tx=0x7fbc2400eeb0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 -- 172.21.15.107:0/3854051244 shutdown_connections 2024-07-26T12:48:57.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbc180b3ca0 0x7fbc180b61b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbc3010b7c0 0x7fbc301a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc3010ae70 0x7fbc3019c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 --2- 172.21.15.107:0/3854051244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc30105540 0x7fbc3019bed0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 -- 172.21.15.107:0/3854051244 >> 172.21.15.107:0/3854051244 conn(0x7fbc30100f20 msgr2=0x7fbc30103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:57.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 -- 172.21.15.107:0/3854051244 shutdown_connections 2024-07-26T12:48:57.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.350+0000 7fbc36200640 1 -- 172.21.15.107:0/3854051244 wait complete. 2024-07-26T12:48:57.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-07-26T12:48:57.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.634+0000 7f2206600640 1 -- 172.21.15.107:0/158319936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220010b7d0 msgr2=0x7f2200112000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:57.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.634+0000 7f2206600640 1 --2- 172.21.15.107:0/158319936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220010b7d0 0x7f2200112000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f21f8009ca0 tx=0x7f21f802a8c0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.634+0000 7f2206600640 1 -- 172.21.15.107:0/158319936 shutdown_connections 2024-07-26T12:48:57.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.634+0000 7f2206600640 1 --2- 172.21.15.107:0/158319936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220010b7d0 0x7f2200112000 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.634+0000 7f2206600640 1 --2- 172.21.15.107:0/158319936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f220010ae80 0x7f220010b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.634+0000 7f2206600640 1 --2- 172.21.15.107:0/158319936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f2200105550 0x7f2200105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.634+0000 7f2206600640 1 -- 172.21.15.107:0/158319936 >> 172.21.15.107:0/158319936 conn(0x7f2200100f90 msgr2=0x7f22001033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:57.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.634+0000 7f2206600640 1 -- 172.21.15.107:0/158319936 shutdown_connections 2024-07-26T12:48:57.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.634+0000 7f2206600640 1 -- 172.21.15.107:0/158319936 wait complete. 2024-07-26T12:48:57.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.635+0000 7f2206600640 1 Processor -- start 2024-07-26T12:48:57.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f2206600640 1 -- start start 2024-07-26T12:48:57.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f2206600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2200105550 0x7f220019beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:57.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f2206600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220010ae80 0x7f220019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:57.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f2206600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f220010b7d0 0x7f22001a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f2206600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f22001154f0 con 0x7f2200105550 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f2206600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f2200115370 con 0x7f220010b7d0 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f2206600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f2200115670 con 0x7f220010ae80 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f2205200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2200105550 0x7f220019beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f21ffe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220010ae80 0x7f220019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f21ffe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220010ae80 0x7f220019c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:58012/0 (socket says 172.21.15.107:58012) 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f21ffe00640 1 -- 172.21.15.107:0/3640250686 learned_addr learned my addr 172.21.15.107:0/3640250686 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.636+0000 7f2205c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f220010b7d0 0x7f22001a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.637+0000 7f2205200640 1 -- 172.21.15.107:0/3640250686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220010ae80 msgr2=0x7f220019c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.637+0000 7f2205200640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220010ae80 0x7f220019c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.637+0000 7f2205200640 1 -- 172.21.15.107:0/3640250686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f220010b7d0 msgr2=0x7f22001a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.637+0000 7f2205200640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f220010b7d0 0x7f22001a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.637+0000 7f2205200640 1 -- 172.21.15.107:0/3640250686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22001a1de0 con 0x7f2200105550 2024-07-26T12:48:57.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.637+0000 7f2205c00640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f220010b7d0 0x7f22001a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:48:57.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.637+0000 7f2205200640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2200105550 0x7f220019beb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f21f400e9f0 tx=0x7f21f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:57.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.637+0000 7f21fd600640 1 -- 172.21.15.107:0/3640250686 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f400ccc0 con 0x7f2200105550 2024-07-26T12:48:57.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.637+0000 7f21fd600640 1 -- 172.21.15.107:0/3640250686 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21f4005020 con 0x7f2200105550 2024-07-26T12:48:57.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.637+0000 7f2206600640 1 -- 172.21.15.107:0/3640250686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22001a20d0 con 0x7f2200105550 2024-07-26T12:48:57.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.638+0000 7f21fd600640 1 -- 172.21.15.107:0/3640250686 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4004a00 con 0x7f2200105550 2024-07-26T12:48:57.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.638+0000 7f2206600640 1 -- 172.21.15.107:0/3640250686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22001a8860 con 0x7f2200105550 2024-07-26T12:48:57.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.639+0000 7f2206600640 1 -- 172.21.15.107:0/3640250686 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2200105960 con 0x7f2200105550 2024-07-26T12:48:57.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.641+0000 7f21fd600640 1 -- 172.21.15.107:0/3640250686 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f21f4004ba0 con 0x7f2200105550 2024-07-26T12:48:57.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.643+0000 7f21fd600640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21e80b3c20 0x7f21e80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:57.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.644+0000 7f21fd600640 1 -- 172.21.15.107:0/3640250686 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f21f40e4140 con 0x7f2200105550 2024-07-26T12:48:57.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.644+0000 7f21ffe00640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21e80b3c20 0x7f21e80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:57.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.644+0000 7f21ffe00640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21e80b3c20 0x7f21e80b6130 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f22001047e0 tx=0x7f21ec005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:57.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.649+0000 7f21fd600640 1 -- 172.21.15.107:0/3640250686 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f21f40ab290 con 0x7f2200105550 2024-07-26T12:48:57.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.892+0000 7f2206600640 1 -- 172.21.15.107:0/3640250686 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f2200063230 con 0x7f21e80b3c20 2024-07-26T12:48:57.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.908+0000 7f21fd600640 1 -- 172.21.15.107:0/3640250686 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f2200063230 con 0x7f21e80b3c20 2024-07-26T12:48:57.909 INFO:teuthology.orchestra.run.smithi107.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-07-26T00:00:00", "created": "2024-07-26T12:48:56", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-07-26T12:48:57.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 -- 172.21.15.107:0/3640250686 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21e80b3c20 msgr2=0x7f21e80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:57.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21e80b3c20 0x7f21e80b6130 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f22001047e0 tx=0x7f21ec005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:57.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 -- 172.21.15.107:0/3640250686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2200105550 msgr2=0x7f220019beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:57.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2200105550 0x7f220019beb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f21f400e9f0 tx=0x7f21f400eeb0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 -- 172.21.15.107:0/3640250686 shutdown_connections 2024-07-26T12:48:57.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f21e80b3c20 0x7f21e80b6130 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f220010b7d0 0x7f22001a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f220010ae80 0x7f220019c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 --2- 172.21.15.107:0/3640250686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2200105550 0x7f220019beb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:57.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 -- 172.21.15.107:0/3640250686 >> 172.21.15.107:0/3640250686 conn(0x7f2200100f90 msgr2=0x7f2200108450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:57.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 -- 172.21.15.107:0/3640250686 shutdown_connections 2024-07-26T12:48:57.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:57.911+0000 7f2206600640 1 -- 172.21.15.107:0/3640250686 wait complete. 2024-07-26T12:48:57.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-07-26T12:48:58.018 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:57 smithi107 ceph-mon[32659]: from='client.25501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.198+0000 7fbb5be00640 1 -- 172.21.15.107:0/978423916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb5c113c00 msgr2=0x7fbb5c1160d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.198+0000 7fbb5be00640 1 --2- 172.21.15.107:0/978423916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb5c113c00 0x7fbb5c1160d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbb54009ca0 tx=0x7fbb5402a8c0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.198+0000 7fbb5be00640 1 -- 172.21.15.107:0/978423916 shutdown_connections 2024-07-26T12:48:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.198+0000 7fbb5be00640 1 --2- 172.21.15.107:0/978423916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb5c113c00 0x7fbb5c1160d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.198+0000 7fbb5be00640 1 --2- 172.21.15.107:0/978423916 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb5c075800 0x7fbb5c075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.198+0000 7fbb5be00640 1 --2- 172.21.15.107:0/978423916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb5c076dc0 0x7fbb5c0751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.198+0000 7fbb5be00640 1 -- 172.21.15.107:0/978423916 >> 172.21.15.107:0/978423916 conn(0x7fbb5c0fed60 msgr2=0x7fbb5c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.198+0000 7fbb5be00640 1 -- 172.21.15.107:0/978423916 shutdown_connections 2024-07-26T12:48:58.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.198+0000 7fbb5be00640 1 -- 172.21.15.107:0/978423916 wait complete. 2024-07-26T12:48:58.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.199+0000 7fbb5be00640 1 Processor -- start 2024-07-26T12:48:58.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.199+0000 7fbb5be00640 1 -- start start 2024-07-26T12:48:58.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5be00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb5c075800 0x7fbb5c1a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb5c076dc0 0x7fbb5c1a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb5c113c00 0x7fbb5c1a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5be00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5c119840 con 0x7fbb5c075800 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5be00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5c1196c0 con 0x7fbb5c076dc0 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5be00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5c1199c0 con 0x7fbb5c113c00 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5aa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb5c075800 0x7fbb5c1a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5aa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb5c075800 0x7fbb5c1a0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46302/0 (socket says 172.21.15.107:46302) 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5aa00640 1 -- 172.21.15.107:0/794657459 learned_addr learned my addr 172.21.15.107:0/794657459 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5a000640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb5c076dc0 0x7fbb5c1a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.200+0000 7fbb5b400640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb5c113c00 0x7fbb5c1a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb5aa00640 1 -- 172.21.15.107:0/794657459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb5c113c00 msgr2=0x7fbb5c1a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb5aa00640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb5c113c00 0x7fbb5c1a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb5aa00640 1 -- 172.21.15.107:0/794657459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb5c076dc0 msgr2=0x7fbb5c1a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb5aa00640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb5c076dc0 0x7fbb5c1a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb5aa00640 1 -- 172.21.15.107:0/794657459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5c1a6130 con 0x7fbb5c075800 2024-07-26T12:48:58.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb5aa00640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb5c075800 0x7fbb5c1a0200 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800ea60 tx=0x7fbb4800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb60a00640 1 -- 172.21.15.107:0/794657459 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4800cde0 con 0x7fbb5c075800 2024-07-26T12:48:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb60a00640 1 -- 172.21.15.107:0/794657459 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb480035f0 con 0x7fbb5c075800 2024-07-26T12:48:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb5be00640 1 -- 172.21.15.107:0/794657459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5c1a6420 con 0x7fbb5c075800 2024-07-26T12:48:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb60a00640 1 -- 172.21.15.107:0/794657459 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb48010690 con 0x7fbb5c075800 2024-07-26T12:48:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.201+0000 7fbb5be00640 1 -- 172.21.15.107:0/794657459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb5c1acbb0 con 0x7fbb5c075800 2024-07-26T12:48:58.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.202+0000 7fbb5be00640 1 -- 172.21.15.107:0/794657459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb5c075c10 con 0x7fbb5c075800 2024-07-26T12:48:58.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.206+0000 7fbb60a00640 1 -- 172.21.15.107:0/794657459 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7fbb480037a0 con 0x7fbb5c075800 2024-07-26T12:48:58.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.207+0000 7fbb60a00640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbb400b3cb0 0x7fbb400b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.207+0000 7fbb60a00640 1 -- 172.21.15.107:0/794657459 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbb480e47b0 con 0x7fbb5c075800 2024-07-26T12:48:58.208 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.208+0000 7fbb5a000640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbb400b3cb0 0x7fbb400b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.208+0000 7fbb5a000640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbb400b3cb0 0x7fbb400b61c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c0095f0 tx=0x7fbb4c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:58.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.213+0000 7fbb60a00640 1 -- 172.21.15.107:0/794657459 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbb48055600 con 0x7fbb5c075800 2024-07-26T12:48:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:57 smithi132 ceph-mon[37580]: from='client.25501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:57 smithi112 ceph-mon[37501]: from='client.25501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.464+0000 7fbb5be00640 1 -- 172.21.15.107:0/794657459 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fbb5c063230 con 0x7fbb400b3cb0 2024-07-26T12:48:58.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.474+0000 7fbb60a00640 1 -- 172.21.15.107:0/794657459 <== mgr.14211 v2:172.21.15.107:6800/3577384363 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fbb5c063230 con 0x7fbb400b3cb0 2024-07-26T12:48:58.475 INFO:teuthology.orchestra.run.smithi107.stdout:/ 1m 6m3h 2024-07-26T12:48:58.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 -- 172.21.15.107:0/794657459 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbb400b3cb0 msgr2=0x7fbb400b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbb400b3cb0 0x7fbb400b61c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c0095f0 tx=0x7fbb4c005f90 comp rx=0 tx=0).stop 2024-07-26T12:48:58.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 -- 172.21.15.107:0/794657459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb5c075800 msgr2=0x7fbb5c1a0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb5c075800 0x7fbb5c1a0200 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800ea60 tx=0x7fbb4800ef20 comp rx=0 tx=0).stop 2024-07-26T12:48:58.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 -- 172.21.15.107:0/794657459 shutdown_connections 2024-07-26T12:48:58.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fbb400b3cb0 0x7fbb400b61c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb5c113c00 0x7fbb5c1a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fbb5c076dc0 0x7fbb5c1a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 --2- 172.21.15.107:0/794657459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb5c075800 0x7fbb5c1a0200 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 -- 172.21.15.107:0/794657459 >> 172.21.15.107:0/794657459 conn(0x7fbb5c0fed60 msgr2=0x7fbb5c10fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:58.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.477+0000 7fbb5be00640 1 -- 172.21.15.107:0/794657459 shutdown_connections 2024-07-26T12:48:58.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.478+0000 7fbb5be00640 1 -- 172.21.15.107:0/794657459 wait complete. 2024-07-26T12:48:58.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-07-26T12:48:58.580 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-07-26T12:48:58.591 INFO:teuthology.run_tasks:Running task background_exec... 2024-07-26T12:48:58.601 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-07-26T12:48:58.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-07-26T12:48:58.607 INFO:teuthology.run_tasks:Running task workunit... 2024-07-26T12:48:58.619 INFO:tasks.workunit:Pulling workunits from ref e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:48:58.620 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-07-26T12:48:58.620 INFO:tasks.workunit:timeout=3h 2024-07-26T12:48:58.620 INFO:tasks.workunit:cleanup=True 2024-07-26T12:48:58.620 DEBUG:teuthology.orchestra.run.smithi107:> stat -- /home/ubuntu/cephtest/mnt.0 2024-07-26T12:48:58.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.649+0000 7ff1ef200640 1 -- 172.21.15.107:0/1146133985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e8108ed0 msgr2=0x7ff1e81092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.649+0000 7ff1ef200640 1 --2- 172.21.15.107:0/1146133985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e8108ed0 0x7ff1e81092e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009900 tx=0x7ff1d402a400 comp rx=0 tx=0).stop 2024-07-26T12:48:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.649+0000 7ff1ef200640 1 -- 172.21.15.107:0/1146133985 shutdown_connections 2024-07-26T12:48:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.649+0000 7ff1ef200640 1 --2- 172.21.15.107:0/1146133985 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1e8109820 0x7ff1e8110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.649+0000 7ff1ef200640 1 --2- 172.21.15.107:0/1146133985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e8108ed0 0x7ff1e81092e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.649+0000 7ff1ef200640 1 --2- 172.21.15.107:0/1146133985 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e8103590 0x7ff1e81039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.649+0000 7ff1ef200640 1 -- 172.21.15.107:0/1146133985 >> 172.21.15.107:0/1146133985 conn(0x7ff1e807b280 msgr2=0x7ff1e807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.649+0000 7ff1ef200640 1 -- 172.21.15.107:0/1146133985 shutdown_connections 2024-07-26T12:48:58.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.649+0000 7ff1ef200640 1 -- 172.21.15.107:0/1146133985 wait complete. 2024-07-26T12:48:58.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.651+0000 7ff1ef200640 1 Processor -- start 2024-07-26T12:48:58.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.651+0000 7ff1ef200640 1 -- start start 2024-07-26T12:48:58.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.651+0000 7ff1ef200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e8103590 0x7ff1e819c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.652+0000 7ff1ef200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1e8108ed0 0x7ff1e819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.652+0000 7ff1ef200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e8109820 0x7ff1e81a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.652+0000 7ff1ef200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e8113550 con 0x7ff1e8109820 2024-07-26T12:48:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.652+0000 7ff1ef200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e81133d0 con 0x7ff1e8108ed0 2024-07-26T12:48:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.652+0000 7ff1ef200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e81136d0 con 0x7ff1e8103590 2024-07-26T12:48:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.652+0000 7ff1e7400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1e8108ed0 0x7ff1e819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.652+0000 7ff1ed200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e8109820 0x7ff1e81a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.652+0000 7ff1e7400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1e8108ed0 0x7ff1e819c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:36252/0 (socket says 172.21.15.107:36252) 2024-07-26T12:48:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.652+0000 7ff1e7400640 1 -- 172.21.15.107:0/1081025526 learned_addr learned my addr 172.21.15.107:0/1081025526 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.652+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e8103590 0x7ff1e819c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.653+0000 7ff1e7e00640 1 -- 172.21.15.107:0/1081025526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1e8108ed0 msgr2=0x7ff1e819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.653+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1e8108ed0 0x7ff1e819c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.653+0000 7ff1e7e00640 1 -- 172.21.15.107:0/1081025526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e8109820 msgr2=0x7ff1e81a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.653+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e8109820 0x7ff1e81a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.653+0000 7ff1e7e00640 1 -- 172.21.15.107:0/1081025526 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e81a2130 con 0x7ff1e8103590 2024-07-26T12:48:58.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.653+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e8103590 0x7ff1e819c1c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800b3f0 tx=0x7ff1d800b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:58.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.653+0000 7ff1e4c00640 1 -- 172.21.15.107:0/1081025526 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8015020 con 0x7ff1e8103590 2024-07-26T12:48:58.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.653+0000 7ff1e4c00640 1 -- 172.21.15.107:0/1081025526 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d8011ce0 con 0x7ff1e8103590 2024-07-26T12:48:58.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.653+0000 7ff1ef200640 1 -- 172.21.15.107:0/1081025526 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e81a2420 con 0x7ff1e8103590 2024-07-26T12:48:58.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.654+0000 7ff1e4c00640 1 -- 172.21.15.107:0/1081025526 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8010de0 con 0x7ff1e8103590 2024-07-26T12:48:58.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.654+0000 7ff1ef200640 1 -- 172.21.15.107:0/1081025526 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e81a8b80 con 0x7ff1e8103590 2024-07-26T12:48:58.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.658+0000 7ff1f5eb2740 1 Processor -- start 2024-07-26T12:48:58.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.659+0000 7ff1f5eb2740 1 -- start start 2024-07-26T12:48:58.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.659+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eac2350 0x558b2ee66d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.659+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 0x558b2ee672d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.659+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee67810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.659+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558b2edddbf0 con 0x558b2eac2350 2024-07-26T12:48:58.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.659+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558b2edcd0f0 con 0x558b2ea99490 2024-07-26T12:48:58.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.659+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558b2edd1130 con 0x558b2eaa1670 2024-07-26T12:48:58.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.660+0000 7ff1e4c00640 1 -- 172.21.15.107:0/1081025526 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7ff1d8018020 con 0x7ff1e8103590 2024-07-26T12:48:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.660+0000 7ff1e7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee67810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.660+0000 7ff1e7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee67810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:58056/0 (socket says 172.21.15.107:58056) 2024-07-26T12:48:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.660+0000 7ff1e7400640 1 -- 172.21.15.107:0/3472828631 learned_addr learned my addr 172.21.15.107:0/3472828631 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.660+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/3472828631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 0x558b2ee672d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.660+0000 7ff1ed200640 1 --2- 172.21.15.107:0/3472828631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eac2350 0x558b2ee66d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.660+0000 7ff1e7400640 1 -- 172.21.15.107:0/3472828631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 msgr2=0x558b2ee672d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.660+0000 7ff1e7400640 1 --2- 172.21.15.107:0/3472828631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 0x558b2ee672d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.660+0000 7ff1e7400640 1 -- 172.21.15.107:0/3472828631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eac2350 msgr2=0x558b2ee66d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.660+0000 7ff1e7400640 1 --2- 172.21.15.107:0/3472828631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eac2350 0x558b2ee66d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.661+0000 7ff1e7400640 1 -- 172.21.15.107:0/3472828631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b2eb2d930 con 0x558b2eaa1670 2024-07-26T12:48:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.661+0000 7ff1e7400640 1 --2- 172.21.15.107:0/3472828631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee67810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff1d402a920 tx=0x7ff1d4004200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b44ffb8b0cd99d53 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.662+0000 7ff1dd600640 1 -- 172.21.15.107:0/3472828631 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4040070 con 0x558b2eaa1670 2024-07-26T12:48:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.662+0000 7ff1dd600640 1 -- 172.21.15.107:0/3472828631 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d40044b0 con 0x558b2eaa1670 2024-07-26T12:48:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.662+0000 7ff1dd600640 1 -- 172.21.15.107:0/3472828631 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4035040 con 0x558b2eaa1670 2024-07-26T12:48:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.662+0000 7ff1e4c00640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1cc0b3d30 0x7ff1cc0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.662+0000 7ff1e4c00640 1 -- 172.21.15.107:0/1081025526 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff1d80e3910 con 0x7ff1e8103590 2024-07-26T12:48:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.662+0000 7ff1ed200640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1cc0b3d30 0x7ff1cc0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.663+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3472828631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 msgr2=0x558b2ee67810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.663+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/3472828631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee67810 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff1d402a920 tx=0x7ff1d4004200 comp rx=0 tx=0).stop 2024-07-26T12:48:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.663+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3472828631 shutdown_connections 2024-07-26T12:48:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.664+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/3472828631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee67810 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.664+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/3472828631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 0x558b2ee672d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.664+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/3472828631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eac2350 0x558b2ee66d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.664+0000 7ff1ed200640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1cc0b3d30 0x7ff1cc0b6240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0005fd0 tx=0x7ff1e0009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.664+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3472828631 >> 172.21.15.107:0/3472828631 conn(0x558b2eab6740 msgr2=0x558b2ee64c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.664+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3472828631 shutdown_connections 2024-07-26T12:48:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.664+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3472828631 wait complete. 2024-07-26T12:48:58.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.665+0000 7ff1f5eb2740 1 -- start start 2024-07-26T12:48:58.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.665+0000 7ff1f5eb2740 1 Processor -- start 2024-07-26T12:48:58.667 INFO:teuthology.orchestra.run.smithi107.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-07-26T12:48:58.667 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-07-26T12:48:58.667 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-07-26T12:48:58.667 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T12:48:58.668 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-07-26T12:48:58.668 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-07-26 12:45:01.118308215 +0000 2024-07-26T12:48:58.668 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-07-26 12:45:01.118308215 +0000 2024-07-26T12:48:58.668 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-07-26 12:48:37.454376256 +0000 2024-07-26T12:48:58.668 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-07-26 12:45:01.118308215 +0000 2024-07-26T12:48:58.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 0x558b2ee724e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.669 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-07-26T12:48:58.669 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-07-26T12:48:58.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee72a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eac2350 0x558b2ee7e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558b2ec30890 con 0x558b2eac2350 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558b2ed69330 con 0x558b2ea99490 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558b2ee75510 con 0x558b2eaa1670 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1f5eb2740 10 client.0 _ll_register_callbacks cb 0x558b2ed95010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1e7e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 0x558b2ee724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1e7e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 0x558b2ee724e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:36272/0 (socket says 172.21.15.107:36272) 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1e7e00640 1 -- 172.21.15.107:0/164103508 learned_addr learned my addr 172.21.15.107:0/164103508 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1e7400640 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee72a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1ed200640 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eac2350 0x558b2ee7e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1e7e00640 1 -- 172.21.15.107:0/164103508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 msgr2=0x558b2ee72a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.666+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee72a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1e7e00640 1 -- 172.21.15.107:0/164103508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eac2350 msgr2=0x558b2ee7e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:58.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eac2350 0x558b2ee7e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1e7e00640 1 -- 172.21.15.107:0/164103508 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b2ee732a0 con 0x558b2ea99490 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 0x558b2ee724e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80a6390 tx=0x7ff1d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7285ca8f917fc81 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1d2000640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1d2000640 1 -- 172.21.15.107:0/164103508 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80aaa10 con 0x558b2ea99490 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/164103508 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d40044b0 con 0x558b2ea99490 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1d2000640 1 -- 172.21.15.107:0/164103508 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff1d8011830 con 0x558b2ea99490 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1d2000640 1 -- 172.21.15.107:0/164103508 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff1d80aa310 con 0x558b2ea99490 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1f5eb2740 10 client.25517 fetch_fsmap learned FSMap version 29 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.667+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/164103508 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558b2ee81480 con 0x558b2ea99490 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.668+0000 7ff1d2000640 1 -- 172.21.15.107:0/164103508 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d80c4e50 con 0x558b2ea99490 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.668+0000 7ff1d2000640 1 -- 172.21.15.107:0/164103508 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff1d80a5c80 con 0x558b2ea99490 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.668+0000 7ff1f5eb2740 10 client.25517 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:48:58.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.668+0000 7ff1f5eb2740 10 client.25517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.668+0000 7ff1f5eb2740 20 client.25517 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.668+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558b2ea80cd0 0x558b2ee848e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.668+0000 7ff1f5eb2740 4 client.25517 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.668+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/164103508 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b2eddfb40 con 0x558b2ea80cd0 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.668+0000 7ff1e7400640 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558b2ea80cd0 0x558b2ee848e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.669+0000 7ff1e7400640 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558b2ea80cd0 0x558b2ee848e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=428701105092c976 server_cookie=47a9ddbfa39467f6 in_seq=0 out_seq=0 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.669+0000 7ff1d2000640 10 client.25517 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1d2000640 1 -- 172.21.15.107:0/164103508 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558b2eddfb40 con 0x558b2ea80cd0 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1d2000640 10 client.25517 handle_command_reply: tid=0 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1f5eb2740 1 client.25517 shutdown 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/164103508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 msgr2=0x558b2ee724e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 0x558b2ee724e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff1d80a6390 tx=0x7ff1d80046c0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/164103508 shutdown_connections 2024-07-26T12:48:58.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558b2ea80cd0 0x558b2ee848e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eac2350 0x558b2ee7e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee72a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/164103508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2ea99490 0x558b2ee724e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/164103508 >> 172.21.15.107:0/164103508 conn(0x558b2eab6740 msgr2=0x558b2ee69f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:58.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1d2000640 0 client.25517 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:48:58.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.670+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/164103508 shutdown_connections 2024-07-26T12:48:58.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.671+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/164103508 wait complete. 2024-07-26T12:48:58.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.671+0000 7ff1f5eb2740 20 client.25517 trim_cache size 0 max 16384 2024-07-26T12:48:58.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.710+0000 7ff1f5eb2740 1 Processor -- start 2024-07-26T12:48:58.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.710+0000 7ff1f5eb2740 1 -- start start 2024-07-26T12:48:58.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.710+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2ea99490 0x558b2ee68890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.710+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee6dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.710+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 0x558b2ee8a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.710+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e81b2be0 con 0x558b2ea99490 2024-07-26T12:48:58.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.710+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e81b4720 con 0x558b2eac2350 2024-07-26T12:48:58.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.710+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e81b32b0 con 0x558b2eaa1670 2024-07-26T12:48:58.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.711+0000 7ff1e7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee6dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.711+0000 7ff1e7e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2ea99490 0x558b2ee68890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.711+0000 7ff1e7400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee6dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:58082/0 (socket says 172.21.15.107:58082) 2024-07-26T12:48:58.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.711+0000 7ff1e7400640 1 -- 172.21.15.107:0/2652630466 learned_addr learned my addr 172.21.15.107:0/2652630466 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:58.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.711+0000 7ff1ed200640 1 --2- 172.21.15.107:0/2652630466 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 0x558b2ee8a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.711+0000 7ff1e7400640 1 -- 172.21.15.107:0/2652630466 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 msgr2=0x558b2ee8a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:58.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.711+0000 7ff1e7400640 1 --2- 172.21.15.107:0/2652630466 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 0x558b2ee8a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.711+0000 7ff1e7400640 1 -- 172.21.15.107:0/2652630466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2ea99490 msgr2=0x558b2ee68890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:58.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.711+0000 7ff1e7400640 1 --2- 172.21.15.107:0/2652630466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2ea99490 0x558b2ee68890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.711+0000 7ff1e7400640 1 -- 172.21.15.107:0/2652630466 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b2ee69080 con 0x558b2eaa1670 2024-07-26T12:48:58.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.712+0000 7ff1e7400640 1 --2- 172.21.15.107:0/2652630466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee6dc10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4005f70 tx=0x7ff1d4004de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd5f424e7c7b2c39 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:58.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.712+0000 7ff1dcc00640 1 -- 172.21.15.107:0/2652630466 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4035040 con 0x558b2eaa1670 2024-07-26T12:48:58.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.712+0000 7ff1dcc00640 1 -- 172.21.15.107:0/2652630466 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d40053c0 con 0x558b2eaa1670 2024-07-26T12:48:58.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.712+0000 7ff1dcc00640 1 -- 172.21.15.107:0/2652630466 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4040040 con 0x558b2eaa1670 2024-07-26T12:48:58.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.713+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/2652630466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 msgr2=0x558b2ee6dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.713+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/2652630466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee6dc10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4005f70 tx=0x7ff1d4004de0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.713+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/2652630466 shutdown_connections 2024-07-26T12:48:58.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.713+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/2652630466 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 0x558b2ee8a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.713+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/2652630466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2eaa1670 0x558b2ee6dc10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.713+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/2652630466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2ea99490 0x558b2ee68890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.713+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/2652630466 >> 172.21.15.107:0/2652630466 conn(0x558b2eab6740 msgr2=0x558b2ee72390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:58.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.713+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/2652630466 shutdown_connections 2024-07-26T12:48:58.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.714+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/2652630466 wait complete. 2024-07-26T12:48:58.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.714+0000 7ff1f5eb2740 1 -- start start 2024-07-26T12:48:58.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.714+0000 7ff1f5eb2740 1 Processor -- start 2024-07-26T12:48:58.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.715+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2ea99490 0x558b2ee7ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.715+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eaa1670 0x558b2ee76290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.715+0000 7ff1f5eb2740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 0x558b2ee767d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.715+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e81b4050 con 0x558b2eaa1670 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.715+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e81b3980 con 0x558b2eac2350 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.715+0000 7ff1f5eb2740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e81b0bc0 con 0x558b2ea99490 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.715+0000 7ff1f5eb2740 10 client.0 _ll_register_callbacks cb 0x7ff1d8010f80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.715+0000 7ff1e7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2ea99490 0x558b2ee7ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.715+0000 7ff1e7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eaa1670 0x558b2ee76290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.716+0000 7ff1e7400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eaa1670 0x558b2ee76290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46374/0 (socket says 172.21.15.107:46374) 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.716+0000 7ff1ed200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 0x558b2ee767d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.716+0000 7ff1e7400640 1 -- 172.21.15.107:0/3467397091 learned_addr learned my addr 172.21.15.107:0/3467397091 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:48:58.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.715+0000 7ff1e7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2ea99490 0x558b2ee7ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:58090/0 (socket says 172.21.15.107:58090) 2024-07-26T12:48:58.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.716+0000 7ff1ed200640 1 -- 172.21.15.107:0/3467397091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2ea99490 msgr2=0x558b2ee7ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:58.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.716+0000 7ff1ed200640 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2ea99490 0x558b2ee7ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.716+0000 7ff1ed200640 1 -- 172.21.15.107:0/3467397091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eaa1670 msgr2=0x558b2ee76290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.716+0000 7ff1ed200640 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eaa1670 0x558b2ee76290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.716+0000 7ff1ed200640 1 -- 172.21.15.107:0/3467397091 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b2ee76f10 con 0x558b2eac2350 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.717+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2ea99490 0x558b2ee7ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.717+0000 7ff1ed200640 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 0x558b2ee767d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0004100 tx=0x7ff1e0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8c55ff8c92da83b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.717+0000 7ff1d2000640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.717+0000 7ff1d2000640 1 -- 172.21.15.107:0/3467397091 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0037060 con 0x558b2eac2350 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.717+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3467397091 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d40053c0 con 0x558b2eac2350 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.717+0000 7ff1d2000640 1 -- 172.21.15.107:0/3467397091 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff1e000fc10 con 0x558b2eac2350 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.717+0000 7ff1d2000640 1 -- 172.21.15.107:0/3467397091 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff1e003e050 con 0x558b2eac2350 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.717+0000 7ff1f5eb2740 10 client.25523 fetch_fsmap learned FSMap version 29 2024-07-26T12:48:58.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.717+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3467397091 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558b2eeed0a0 con 0x558b2eac2350 2024-07-26T12:48:58.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.718+0000 7ff1d2000640 1 -- 172.21.15.107:0/3467397091 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d40053c0 con 0x558b2eac2350 2024-07-26T12:48:58.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.718+0000 7ff1d2000640 1 -- 172.21.15.107:0/3467397091 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff1e0032440 con 0x558b2eac2350 2024-07-26T12:48:58.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.718+0000 7ff1f5eb2740 10 client.25523 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:48:58.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.718+0000 7ff1f5eb2740 10 client.25523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:48:58.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.718+0000 7ff1f5eb2740 20 client.25523 populate_metadata read hostname 'smithi107' 2024-07-26T12:48:58.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.718+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558b2ea56080 0x558b2eeef870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:48:58.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.718+0000 7ff1f5eb2740 4 client.25523 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T12:48:58.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.718+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3467397091 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558b2eddb5e0 con 0x558b2ea56080 2024-07-26T12:48:58.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.719+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558b2ea56080 0x558b2eeef870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:48:58.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.719+0000 7ff1d2000640 10 client.25523 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:48:58.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.719+0000 7ff1e7e00640 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558b2ea56080 0x558b2eeef870 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=331b8106d8fa98bf server_cookie=517b0342803b52e6 in_seq=0 out_seq=0 2024-07-26T12:48:58.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.720+0000 7ff1d2000640 1 -- 172.21.15.107:0/3467397091 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558b2eddb5e0 con 0x558b2ea56080 2024-07-26T12:48:58.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.720+0000 7ff1d2000640 10 client.25523 handle_command_reply: tid=0 2024-07-26T12:48:58.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.720+0000 7ff1f5eb2740 1 client.25523 shutdown 2024-07-26T12:48:58.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.721+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3467397091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 msgr2=0x558b2ee767d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.721+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 0x558b2ee767d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0004100 tx=0x7ff1e0017000 comp rx=0 tx=0).stop 2024-07-26T12:48:58.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.721+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3467397091 shutdown_connections 2024-07-26T12:48:58.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.721+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558b2ea56080 0x558b2eeef870 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.721+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558b2eac2350 0x558b2ee767d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.721+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558b2eaa1670 0x558b2ee76290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.721+0000 7ff1d2000640 0 client.25523 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:48:58.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.721+0000 7ff1f5eb2740 1 --2- 172.21.15.107:0/3467397091 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558b2ea99490 0x558b2ee7ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.721+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3467397091 >> 172.21.15.107:0/3467397091 conn(0x558b2eab6740 msgr2=0x558b2ee6caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:58.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.722+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3467397091 shutdown_connections 2024-07-26T12:48:58.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.722+0000 7ff1f5eb2740 1 -- 172.21.15.107:0/3467397091 wait complete. 2024-07-26T12:48:58.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.722+0000 7ff1f5eb2740 20 client.25523 trim_cache size 0 max 16384 2024-07-26T12:48:58.724 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:48:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-07-26T12:48:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "cd95c6b3-b9b8-4132-8db1-5f4d884380a4", 2024-07-26T12:48:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-07-26T12:48:58.724 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:48:58.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.723+0000 7ff1ef200640 1 -- 172.21.15.107:0/1081025526 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1cc0b3d30 msgr2=0x7ff1cc0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.723+0000 7ff1ef200640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1cc0b3d30 0x7ff1cc0b6240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0005fd0 tx=0x7ff1e0009500 comp rx=0 tx=0).stop 2024-07-26T12:48:58.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.723+0000 7ff1ef200640 1 -- 172.21.15.107:0/1081025526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e8103590 msgr2=0x7ff1e819c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:48:58.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.723+0000 7ff1ef200640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e8103590 0x7ff1e819c1c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800b3f0 tx=0x7ff1d800b8b0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.723+0000 7ff1ef200640 1 -- 172.21.15.107:0/1081025526 shutdown_connections 2024-07-26T12:48:58.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.723+0000 7ff1ef200640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff1cc0b3d30 0x7ff1cc0b6240 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.723+0000 7ff1ef200640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1e8109820 0x7ff1e81a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.723+0000 7ff1ef200640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1e8108ed0 0x7ff1e819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.723+0000 7ff1ef200640 1 --2- 172.21.15.107:0/1081025526 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e8103590 0x7ff1e819c1c0 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800b3f0 tx=0x7ff1d800b8b0 comp rx=0 tx=0).stop 2024-07-26T12:48:58.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.723+0000 7ff1ef200640 1 -- 172.21.15.107:0/1081025526 >> 172.21.15.107:0/1081025526 conn(0x7ff1e807b280 msgr2=0x7ff1e8106ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:48:58.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.724+0000 7ff1ef200640 1 -- 172.21.15.107:0/1081025526 shutdown_connections 2024-07-26T12:48:58.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:48:58.724+0000 7ff1ef200640 1 -- 172.21.15.107:0/1081025526 wait complete. 2024-07-26T12:48:58.749 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T12:48:58.833 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-07-26T12:48:59.265 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:58 smithi107 ceph-mon[32659]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:59.265 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:58 smithi107 ceph-mon[32659]: mgrmap e24: x(active, since 13m), standbys: z, y 2024-07-26T12:48:59.265 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:58 smithi107 ceph-mon[32659]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:59.265 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:58 smithi107 ceph-mon[32659]: pgmap v480: 129 pgs: 129 active+clean; 652 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T12:48:59.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:58 smithi132 ceph-mon[37580]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:59.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:58 smithi132 ceph-mon[37580]: mgrmap e24: x(active, since 13m), standbys: z, y 2024-07-26T12:48:59.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:58 smithi132 ceph-mon[37580]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:59.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:58 smithi132 ceph-mon[37580]: pgmap v480: 129 pgs: 129 active+clean; 652 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T12:48:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:58 smithi112 ceph-mon[37501]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:58 smithi112 ceph-mon[37501]: mgrmap e24: x(active, since 13m), standbys: z, y 2024-07-26T12:48:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:58 smithi112 ceph-mon[37501]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T12:48:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:58 smithi112 ceph-mon[37501]: pgmap v480: 129 pgs: 129 active+clean; 652 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-07-26T12:48:59.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T12:49:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:59 smithi107 ceph-mon[32659]: scrub queued for path: / 2024-07-26T12:49:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:59 smithi107 ceph-mon[32659]: scrub summary: idle+waiting paths [/] 2024-07-26T12:49:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:59 smithi107 ceph-mon[32659]: scrub summary: active paths [/] 2024-07-26T12:49:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:48:59 smithi107 ceph-mon[32659]: scrub summary: idle+waiting paths [/] 2024-07-26T12:49:00.306 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:49:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:59 smithi132 ceph-mon[37580]: scrub queued for path: / 2024-07-26T12:49:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:59 smithi132 ceph-mon[37580]: scrub summary: idle+waiting paths [/] 2024-07-26T12:49:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:59 smithi132 ceph-mon[37580]: scrub summary: active paths [/] 2024-07-26T12:49:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:48:59 smithi132 ceph-mon[37580]: scrub summary: idle+waiting paths [/] 2024-07-26T12:49:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:59 smithi112 ceph-mon[37501]: scrub queued for path: / 2024-07-26T12:49:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:59 smithi112 ceph-mon[37501]: scrub summary: idle+waiting paths [/] 2024-07-26T12:49:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:59 smithi112 ceph-mon[37501]: scrub summary: active paths [/] 2024-07-26T12:49:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:48:59 smithi112 ceph-mon[37501]: scrub summary: idle+waiting paths [/] 2024-07-26T12:49:00.686 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:49:00 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:49:00.364+0000 7f1f57e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T12:49:00.687 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:49:00 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:49:00.364+0000 7f1f57e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T12:49:00.687 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:49:00 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:49:00.364+0000 7f1f57e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T12:49:00.687 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:49:00 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:49:00.364+0000 7f1f57e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T12:49:00.687 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Jul 26 12:49:00 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mgr-x[32897]: 2024-07-26T12:49:00.364+0000 7f1f57e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T12:49:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:00 smithi107 ceph-mon[32659]: pgmap v481: 129 pgs: 129 active+clean; 1.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 73 KiB/s wr, 17 op/s 2024-07-26T12:49:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:00 smithi132 ceph-mon[37580]: pgmap v481: 129 pgs: 129 active+clean; 1.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 73 KiB/s wr, 17 op/s 2024-07-26T12:49:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:00 smithi112 ceph-mon[37501]: pgmap v481: 129 pgs: 129 active+clean; 1.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 73 KiB/s wr, 17 op/s 2024-07-26T12:49:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.854+0000 7faa10e00640 1 -- 172.21.15.107:0/912113022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa0c10a0a0 msgr2=0x7faa0c10a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.854+0000 7faa10e00640 1 --2- 172.21.15.107:0/912113022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa0c10a0a0 0x7faa0c10a4b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4009900 tx=0x7fa9f402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.854+0000 7faa10e00640 1 -- 172.21.15.107:0/912113022 shutdown_connections 2024-07-26T12:49:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.854+0000 7faa10e00640 1 --2- 172.21.15.107:0/912113022 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa0c10ab50 0x7faa0c11a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.854+0000 7faa10e00640 1 --2- 172.21.15.107:0/912113022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa0c10a0a0 0x7faa0c10a4b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.854+0000 7faa10e00640 1 --2- 172.21.15.107:0/912113022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa0c10c920 0x7faa0c10cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.854+0000 7faa10e00640 1 -- 172.21.15.107:0/912113022 >> 172.21.15.107:0/912113022 conn(0x7faa0c074110 msgr2=0x7faa0c074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:02.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.854+0000 7faa10e00640 1 -- 172.21.15.107:0/912113022 shutdown_connections 2024-07-26T12:49:02.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.854+0000 7faa10e00640 1 -- 172.21.15.107:0/912113022 wait complete. 2024-07-26T12:49:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.855+0000 7faa10e00640 1 Processor -- start 2024-07-26T12:49:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.855+0000 7faa10e00640 1 -- start start 2024-07-26T12:49:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.855+0000 7faa10e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa0c10a0a0 0x7faa0c1a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.856+0000 7faa10e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa0c10ab50 0x7faa0c1a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.856+0000 7faa10e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa0c10c920 0x7faa0c1aa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.856+0000 7faa10e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c11df00 con 0x7faa0c10c920 2024-07-26T12:49:02.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.856+0000 7faa10e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c11dd80 con 0x7faa0c10a0a0 2024-07-26T12:49:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.856+0000 7faa10e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c11e080 con 0x7faa0c10ab50 2024-07-26T12:49:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.856+0000 7faa09e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa0c10a0a0 0x7faa0c1a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.856+0000 7faa09e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa0c10a0a0 0x7faa0c1a4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:36342/0 (socket says 172.21.15.107:36342) 2024-07-26T12:49:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.856+0000 7faa09e00640 1 -- 172.21.15.107:0/618734976 learned_addr learned my addr 172.21.15.107:0/618734976 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.856+0000 7faa0a800640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa0c10c920 0x7faa0c1aa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.856+0000 7faa03e00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa0c10ab50 0x7faa0c1a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa09e00640 1 -- 172.21.15.107:0/618734976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa0c10ab50 msgr2=0x7faa0c1a5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa09e00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa0c10ab50 0x7faa0c1a5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa09e00640 1 -- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa0c10c920 msgr2=0x7faa0c1aa2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa09e00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa0c10c920 0x7faa0c1aa2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa09e00640 1 -- 172.21.15.107:0/618734976 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0c1aaa20 con 0x7faa0c10a0a0 2024-07-26T12:49:02.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa0a800640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa0c10c920 0x7faa0c1aa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa09e00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa0c10a0a0 0x7faa0c1a4ab0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00e9f0 tx=0x7fa9fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa02a00640 1 -- 172.21.15.107:0/618734976 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc00cd80 con 0x7faa0c10a0a0 2024-07-26T12:49:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa10e00640 1 -- 172.21.15.107:0/618734976 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa0c1aad10 con 0x7faa0c10a0a0 2024-07-26T12:49:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa02a00640 1 -- 172.21.15.107:0/618734976 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9fc0108e0 con 0x7faa0c10a0a0 2024-07-26T12:49:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.857+0000 7faa02a00640 1 -- 172.21.15.107:0/618734976 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc018980 con 0x7faa0c10a0a0 2024-07-26T12:49:02.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.858+0000 7faa10e00640 1 -- 172.21.15.107:0/618734976 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa0c1b1470 con 0x7faa0c10a0a0 2024-07-26T12:49:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.862+0000 7faa17a45740 1 Processor -- start 2024-07-26T12:49:02.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.862+0000 7faa17a45740 1 -- start start 2024-07-26T12:49:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa17a45740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb3017df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa17a45740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3018330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa17a45740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb3018870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa17a45740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564eb2f8eb90 con 0x564eb2c4a1b0 2024-07-26T12:49:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa17a45740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564eb2f7e090 con 0x564eb2c73070 2024-07-26T12:49:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa17a45740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564eb2f820d0 con 0x564eb2c52390 2024-07-26T12:49:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa02a00640 1 -- 172.21.15.107:0/618734976 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa9fc010430 con 0x7faa0c10a0a0 2024-07-26T12:49:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa03e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb3018870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa0a800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb3017df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa03e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb3018870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:58134/0 (socket says 172.21.15.107:58134) 2024-07-26T12:49:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa03e00640 1 -- 172.21.15.107:0/1159276651 learned_addr learned my addr 172.21.15.107:0/1159276651 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa0a800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb3017df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:36358/0 (socket says 172.21.15.107:36358) 2024-07-26T12:49:02.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.863+0000 7faa09e00640 1 --2- 172.21.15.107:0/1159276651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3018330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.864+0000 7faa03e00640 1 -- 172.21.15.107:0/1159276651 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 msgr2=0x564eb3017df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:02.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.864+0000 7faa03e00640 1 --2- 172.21.15.107:0/1159276651 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb3017df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.864+0000 7faa03e00640 1 -- 172.21.15.107:0/1159276651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 msgr2=0x564eb3018330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:02.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.864+0000 7faa03e00640 1 --2- 172.21.15.107:0/1159276651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3018330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.864+0000 7faa03e00640 1 -- 172.21.15.107:0/1159276651 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eb2cde650 con 0x564eb2c52390 2024-07-26T12:49:02.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.864+0000 7faa03e00640 1 --2- 172.21.15.107:0/1159276651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb3018870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4009900 tx=0x7fa9f4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42e2e92572e69ee6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:02.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.865+0000 7fa9fb400640 1 -- 172.21.15.107:0/1159276651 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4040070 con 0x564eb2c52390 2024-07-26T12:49:02.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.865+0000 7fa9fb400640 1 -- 172.21.15.107:0/1159276651 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f4004030 con 0x564eb2c52390 2024-07-26T12:49:02.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.865+0000 7fa9fb400640 1 -- 172.21.15.107:0/1159276651 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4035040 con 0x564eb2c52390 2024-07-26T12:49:02.869 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.865+0000 7faa02a00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa9ec0b3dc0 0x7fa9ec0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.865+0000 7faa02a00640 1 -- 172.21.15.107:0/618734976 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa9fc0e4570 con 0x7faa0c10a0a0 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.866+0000 7faa0a800640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa9ec0b3dc0 0x7fa9ec0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.866+0000 7faa17a45740 1 -- 172.21.15.107:0/1159276651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 msgr2=0x564eb3018870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.866+0000 7faa17a45740 1 --2- 172.21.15.107:0/1159276651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb3018870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4009900 tx=0x7fa9f4000bc0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.866+0000 7faa17a45740 1 -- 172.21.15.107:0/1159276651 shutdown_connections 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.866+0000 7faa17a45740 1 --2- 172.21.15.107:0/1159276651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb3018870 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.866+0000 7faa17a45740 1 --2- 172.21.15.107:0/1159276651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3018330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.866+0000 7faa17a45740 1 --2- 172.21.15.107:0/1159276651 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb3017df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.866+0000 7faa17a45740 1 -- 172.21.15.107:0/1159276651 >> 172.21.15.107:0/1159276651 conn(0x564eb2c67460 msgr2=0x564eb3015ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.866+0000 7faa0a800640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa9ec0b3dc0 0x7fa9ec0b62d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa04009fd0 tx=0x7faa04009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:02.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.866+0000 7faa17a45740 1 -- 172.21.15.107:0/1159276651 shutdown_connections 2024-07-26T12:49:02.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.867+0000 7faa17a45740 1 -- 172.21.15.107:0/1159276651 wait complete. 2024-07-26T12:49:02.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.867+0000 7faa17a45740 1 -- start start 2024-07-26T12:49:02.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.867+0000 7faa17a45740 1 Processor -- start 2024-07-26T12:49:02.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa17a45740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c4a1b0 0x564eb3023360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa17a45740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c52390 0x564eb30238a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa17a45740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb3023de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa17a45740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564eb2de1700 con 0x564eb2c52390 2024-07-26T12:49:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa17a45740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x564eb2f1a660 con 0x564eb2c73070 2024-07-26T12:49:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa17a45740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x564eb3026570 con 0x564eb2c4a1b0 2024-07-26T12:49:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa17a45740 10 client.0 _ll_register_callbacks cb 0x564eb2f2a110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa09e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c4a1b0 0x564eb3023360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa03e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c52390 0x564eb30238a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa09e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c4a1b0 0x564eb3023360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:58142/0 (socket says 172.21.15.107:58142) 2024-07-26T12:49:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa03e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c52390 0x564eb30238a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46460/0 (socket says 172.21.15.107:46460) 2024-07-26T12:49:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa09e00640 1 -- 172.21.15.107:0/1795255242 learned_addr learned my addr 172.21.15.107:0/1795255242 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.868+0000 7faa0a800640 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb3023de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.869+0000 7faa09e00640 1 -- 172.21.15.107:0/1795255242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 msgr2=0x564eb3023de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.869+0000 7faa09e00640 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb3023de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.869+0000 7faa09e00640 1 -- 172.21.15.107:0/1795255242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c52390 msgr2=0x564eb30238a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.869+0000 7faa09e00640 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c52390 0x564eb30238a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.869+0000 7faa09e00640 1 -- 172.21.15.107:0/1795255242 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eb2f31800 con 0x564eb2c4a1b0 2024-07-26T12:49:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.869+0000 7faa09e00640 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c4a1b0 0x564eb3023360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00c6a0 tx=0x7fa9fc00cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed4d33f619ebaca8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.869+0000 7fa9f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:49:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.869+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1795255242 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc0aa9c0 con 0x564eb2c4a1b0 2024-07-26T12:49:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.869+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1795255242 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa9fc0a7140 con 0x564eb2c4a1b0 2024-07-26T12:49:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.869+0000 7faa17a45740 1 -- 172.21.15.107:0/1795255242 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9f4004030 con 0x564eb2c4a1b0 2024-07-26T12:49:02.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.870+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1795255242 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa9fc0aaeb0 con 0x564eb2c4a1b0 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.870+0000 7faa17a45740 10 client.25546 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.870+0000 7faa17a45740 1 -- 172.21.15.107:0/1795255242 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564eb3032140 con 0x564eb2c4a1b0 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.870+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1795255242 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f4004030 con 0x564eb2c4a1b0 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.870+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1795255242 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa9fc0a6820 con 0x564eb2c4a1b0 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.871+0000 7faa17a45740 10 client.25546 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.871+0000 7faa17a45740 10 client.25546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.871+0000 7faa17a45740 20 client.25546 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.871+0000 7faa17a45740 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x564eb2c319f0 0x564eb30356c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.871+0000 7faa17a45740 4 client.25546 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.871+0000 7faa17a45740 1 -- 172.21.15.107:0/1795255242 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564eb2f90ae0 con 0x564eb2c319f0 2024-07-26T12:49:02.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.871+0000 7faa03e00640 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x564eb2c319f0 0x564eb30356c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.872+0000 7faa03e00640 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x564eb2c319f0 0x564eb30356c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ea0d784c8c80e93 server_cookie=d5b5365001300ed3 in_seq=0 out_seq=0 2024-07-26T12:49:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.872+0000 7fa9f3e00640 10 client.25546 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1795255242 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564eb2f90ae0 con 0x564eb2c319f0 2024-07-26T12:49:02.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7fa9f3e00640 10 client.25546 handle_command_reply: tid=0 2024-07-26T12:49:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 client.25546 shutdown 2024-07-26T12:49:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 -- 172.21.15.107:0/1795255242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c4a1b0 msgr2=0x564eb3023360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c4a1b0 0x564eb3023360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00c6a0 tx=0x7fa9fc00cf20 comp rx=0 tx=0).stop 2024-07-26T12:49:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 -- 172.21.15.107:0/1795255242 shutdown_connections 2024-07-26T12:49:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x564eb2c319f0 0x564eb30356c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb3023de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c52390 0x564eb30238a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7fa9f3e00640 0 client.25546 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 --2- 172.21.15.107:0/1795255242 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c4a1b0 0x564eb3023360 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 -- 172.21.15.107:0/1795255242 >> 172.21.15.107:0/1795255242 conn(0x564eb2c67460 msgr2=0x564eb301afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:02.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 -- 172.21.15.107:0/1795255242 shutdown_connections 2024-07-26T12:49:02.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 1 -- 172.21.15.107:0/1795255242 wait complete. 2024-07-26T12:49:02.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.873+0000 7faa17a45740 20 client.25546 trim_cache size 0 max 16384 2024-07-26T12:49:02.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.892+0000 7faa17a45740 1 Processor -- start 2024-07-26T12:49:02.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.892+0000 7faa17a45740 1 -- start start 2024-07-26T12:49:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.892+0000 7faa17a45740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3024f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.892+0000 7faa17a45740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb303ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.892+0000 7faa17a45740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb303b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.892+0000 7faa17a45740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c1bb300 con 0x564eb2c4a1b0 2024-07-26T12:49:02.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa17a45740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c1bce40 con 0x564eb2c73070 2024-07-26T12:49:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa17a45740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c1bb9d0 con 0x564eb2c52390 2024-07-26T12:49:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa09e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3024f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa09e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3024f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46468/0 (socket says 172.21.15.107:46468) 2024-07-26T12:49:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa09e00640 1 -- 172.21.15.107:0/2238798316 learned_addr learned my addr 172.21.15.107:0/2238798316 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa03e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb303ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa0a800640 1 --2- 172.21.15.107:0/2238798316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb303b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa03e00640 1 -- 172.21.15.107:0/2238798316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 msgr2=0x564eb303b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa03e00640 1 --2- 172.21.15.107:0/2238798316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb303b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa03e00640 1 -- 172.21.15.107:0/2238798316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 msgr2=0x564eb3024f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:02.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa03e00640 1 --2- 172.21.15.107:0/2238798316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3024f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa03e00640 1 -- 172.21.15.107:0/2238798316 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eb3021860 con 0x564eb2c52390 2024-07-26T12:49:02.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.893+0000 7faa09e00640 1 --2- 172.21.15.107:0/2238798316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3024f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.894+0000 7faa03e00640 1 --2- 172.21.15.107:0/2238798316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb303ae80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa9f402a910 tx=0x7fa9f4005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=648a2b0f43faceb5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.894+0000 7fa9faa00640 1 -- 172.21.15.107:0/2238798316 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4040040 con 0x564eb2c52390 2024-07-26T12:49:02.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.894+0000 7fa9faa00640 1 -- 172.21.15.107:0/2238798316 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f4004460 con 0x564eb2c52390 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.894+0000 7fa9faa00640 1 -- 172.21.15.107:0/2238798316 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4035040 con 0x564eb2c52390 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.895+0000 7faa17a45740 1 -- 172.21.15.107:0/2238798316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 msgr2=0x564eb303ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.895+0000 7faa17a45740 1 --2- 172.21.15.107:0/2238798316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb303ae80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa9f402a910 tx=0x7fa9f4005fd0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.895+0000 7faa17a45740 1 -- 172.21.15.107:0/2238798316 shutdown_connections 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.895+0000 7faa17a45740 1 --2- 172.21.15.107:0/2238798316 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c73070 0x564eb303b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.895+0000 7faa17a45740 1 --2- 172.21.15.107:0/2238798316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c52390 0x564eb303ae80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.895+0000 7faa17a45740 1 --2- 172.21.15.107:0/2238798316 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3024f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.895+0000 7faa17a45740 1 -- 172.21.15.107:0/2238798316 >> 172.21.15.107:0/2238798316 conn(0x564eb2c67460 msgr2=0x564eb3024880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.895+0000 7faa17a45740 1 -- 172.21.15.107:0/2238798316 shutdown_connections 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.895+0000 7faa17a45740 1 -- 172.21.15.107:0/2238798316 wait complete. 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.896+0000 7faa17a45740 1 -- start start 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.896+0000 7faa17a45740 1 Processor -- start 2024-07-26T12:49:02.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.896+0000 7faa17a45740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3026f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.896+0000 7faa17a45740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c52390 0x564eb3027530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.896+0000 7faa17a45740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c73070 0x564eb3095e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.896+0000 7faa17a45740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c1bc770 con 0x564eb2c4a1b0 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.896+0000 7faa17a45740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c1bc0a0 con 0x564eb2c52390 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.896+0000 7faa17a45740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7faa0c1b92e0 con 0x564eb2c73070 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.896+0000 7faa17a45740 10 client.0 _ll_register_callbacks cb 0x7fa9fc0a6e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa09e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3026f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa0a800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c73070 0x564eb3095e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa03e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c52390 0x564eb3027530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa09e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3026f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:46476/0 (socket says 172.21.15.107:46476) 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa09e00640 1 -- 172.21.15.107:0/1292013269 learned_addr learned my addr 172.21.15.107:0/1292013269 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:02.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa0a800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c73070 0x564eb3095e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:58172/0 (socket says 172.21.15.107:58172) 2024-07-26T12:49:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa03e00640 1 -- 172.21.15.107:0/1292013269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c73070 msgr2=0x564eb3095e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa03e00640 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c73070 0x564eb3095e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa03e00640 1 -- 172.21.15.107:0/1292013269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 msgr2=0x564eb3026f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa03e00640 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3026f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa03e00640 1 -- 172.21.15.107:0/1292013269 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eb3096450 con 0x564eb2c52390 2024-07-26T12:49:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7faa03e00640 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c52390 0x564eb3027530 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa9f402a910 tx=0x7fa9f402aaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f70b28631ee82528 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.897+0000 7fa9f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:49:02.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.898+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1292013269 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4035030 con 0x564eb2c52390 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.898+0000 7faa17a45740 1 -- 172.21.15.107:0/1292013269 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9f4004460 con 0x564eb2c52390 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.898+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1292013269 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa9f4005040 con 0x564eb2c52390 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.898+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1292013269 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa9f4067050 con 0x564eb2c52390 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.898+0000 7faa17a45740 10 client.25544 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.898+0000 7faa17a45740 1 -- 172.21.15.107:0/1292013269 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564eb309d790 con 0x564eb2c52390 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.899+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1292013269 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f4004460 con 0x564eb2c52390 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.899+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1292013269 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fa9f4004b30 con 0x564eb2c52390 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.899+0000 7faa17a45740 10 client.25544 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.899+0000 7faa17a45740 10 client.25544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.899+0000 7faa17a45740 20 client.25544 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.899+0000 7faa17a45740 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x564eb2c06da0 0x564eb309f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:02.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.899+0000 7faa17a45740 4 client.25544 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.899+0000 7faa17a45740 1 -- 172.21.15.107:0/1292013269 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564eb2f8c580 con 0x564eb2c06da0 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.899+0000 7faa09e00640 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x564eb2c06da0 0x564eb309f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.900+0000 7fa9f3e00640 10 client.25544 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.900+0000 7faa09e00640 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x564eb2c06da0 0x564eb309f8b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dacfe6e5a34da01 server_cookie=7f59bc73bb61a068 in_seq=0 out_seq=0 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.900+0000 7fa9f3e00640 1 -- 172.21.15.107:0/1292013269 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564eb2f8c580 con 0x564eb2c06da0 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.900+0000 7fa9f3e00640 10 client.25544 handle_command_reply: tid=0 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.900+0000 7faa17a45740 1 client.25544 shutdown 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.900+0000 7faa17a45740 1 -- 172.21.15.107:0/1292013269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c52390 msgr2=0x564eb3027530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.900+0000 7faa17a45740 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c52390 0x564eb3027530 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa9f402a910 tx=0x7fa9f402aaf0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.901+0000 7faa17a45740 1 -- 172.21.15.107:0/1292013269 shutdown_connections 2024-07-26T12:49:02.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.901+0000 7faa17a45740 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x564eb2c06da0 0x564eb309f8b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.901+0000 7faa17a45740 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x564eb2c73070 0x564eb3095e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.901+0000 7faa17a45740 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x564eb2c52390 0x564eb3027530 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.901+0000 7faa17a45740 1 --2- 172.21.15.107:0/1292013269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eb2c4a1b0 0x564eb3026f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.901+0000 7fa9f3e00640 0 client.25544 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.901+0000 7faa17a45740 1 -- 172.21.15.107:0/1292013269 >> 172.21.15.107:0/1292013269 conn(0x564eb2c67460 msgr2=0x564eb301e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.901+0000 7faa17a45740 1 -- 172.21.15.107:0/1292013269 shutdown_connections 2024-07-26T12:49:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.901+0000 7faa17a45740 1 -- 172.21.15.107:0/1292013269 wait complete. 2024-07-26T12:49:02.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.901+0000 7faa17a45740 20 client.25544 trim_cache size 0 max 16384 2024-07-26T12:49:02.907 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:49:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T12:49:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-07-26T12:49:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "cd95c6b3-b9b8-4132-8db1-5f4d884380a4": { 2024-07-26T12:49:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-07-26T12:49:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "cd95c6b3-b9b8-4132-8db1-5f4d884380a4", 2024-07-26T12:49:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-07-26T12:49:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-07-26T12:49:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-07-26T12:49:02.907 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:49:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 -- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa9ec0b3dc0 msgr2=0x7fa9ec0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa9ec0b3dc0 0x7fa9ec0b62d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7faa04009fd0 tx=0x7faa04009500 comp rx=0 tx=0).stop 2024-07-26T12:49:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 -- 172.21.15.107:0/618734976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa0c10a0a0 msgr2=0x7faa0c1a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa0c10a0a0 0x7faa0c1a4ab0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00e9f0 tx=0x7fa9fc00eeb0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 -- 172.21.15.107:0/618734976 shutdown_connections 2024-07-26T12:49:02.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa9ec0b3dc0 0x7fa9ec0b62d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faa0c10c920 0x7faa0c1aa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7faa0c10ab50 0x7faa0c1a5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 --2- 172.21.15.107:0/618734976 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7faa0c10a0a0 0x7faa0c1a4ab0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 -- 172.21.15.107:0/618734976 >> 172.21.15.107:0/618734976 conn(0x7faa0c074110 msgr2=0x7faa0c06eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 -- 172.21.15.107:0/618734976 shutdown_connections 2024-07-26T12:49:02.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:02.902+0000 7faa10e00640 1 -- 172.21.15.107:0/618734976 wait complete. 2024-07-26T12:49:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:02 smithi107 ceph-mon[32659]: mgrmap e25: x(active, since 13m), standbys: z, y 2024-07-26T12:49:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:02 smithi107 ceph-mon[32659]: pgmap v482: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 186 KiB/s wr, 71 op/s 2024-07-26T12:49:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:02 smithi132 ceph-mon[37580]: mgrmap e25: x(active, since 13m), standbys: z, y 2024-07-26T12:49:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:02 smithi132 ceph-mon[37580]: pgmap v482: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 186 KiB/s wr, 71 op/s 2024-07-26T12:49:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:02 smithi112 ceph-mon[37501]: mgrmap e25: x(active, since 13m), standbys: z, y 2024-07-26T12:49:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:02 smithi112 ceph-mon[37501]: pgmap v482: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 186 KiB/s wr, 71 op/s 2024-07-26T12:49:03.789 INFO:tasks.cephfs.filesystem:scrub status for tag:cd95c6b3-b9b8-4132-8db1-5f4d884380a4 - {'path': '/', 'tag': 'cd95c6b3-b9b8-4132-8db1-5f4d884380a4', 'options': 'recursive,force'} 2024-07-26T12:49:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:04 smithi112 ceph-mon[37501]: pgmap v483: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 198 KiB/s wr, 72 op/s 2024-07-26T12:49:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:04 smithi107 ceph-mon[32659]: pgmap v483: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 198 KiB/s wr, 72 op/s 2024-07-26T12:49:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:04 smithi132 ceph-mon[37580]: pgmap v483: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 198 KiB/s wr, 72 op/s 2024-07-26T12:49:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:06 smithi112 ceph-mon[37501]: pgmap v484: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 198 KiB/s wr, 71 op/s 2024-07-26T12:49:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:06 smithi107 ceph-mon[32659]: pgmap v484: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 198 KiB/s wr, 71 op/s 2024-07-26T12:49:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:06 smithi132 ceph-mon[37580]: pgmap v484: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 198 KiB/s wr, 71 op/s 2024-07-26T12:49:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:08 smithi112 ceph-mon[37501]: pgmap v485: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 198 KiB/s wr, 73 op/s 2024-07-26T12:49:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:08 smithi107 ceph-mon[32659]: pgmap v485: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 198 KiB/s wr, 73 op/s 2024-07-26T12:49:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:08 smithi132 ceph-mon[37580]: pgmap v485: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 198 KiB/s wr, 73 op/s 2024-07-26T12:49:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:10 smithi112 ceph-mon[37501]: pgmap v486: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 198 KiB/s wr, 71 op/s 2024-07-26T12:49:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:10 smithi107 ceph-mon[32659]: pgmap v486: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 198 KiB/s wr, 71 op/s 2024-07-26T12:49:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:10 smithi132 ceph-mon[37580]: pgmap v486: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 198 KiB/s wr, 71 op/s 2024-07-26T12:49:11.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.919+0000 7f9549e00640 1 -- 172.21.15.107:0/2605870146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954410b7d0 msgr2=0x7f9544112000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.919+0000 7f9549e00640 1 --2- 172.21.15.107:0/2605870146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954410b7d0 0x7f9544112000 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f953c009ca0 tx=0x7f953c02a8b0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.919+0000 7f9549e00640 1 -- 172.21.15.107:0/2605870146 shutdown_connections 2024-07-26T12:49:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.919+0000 7f9549e00640 1 --2- 172.21.15.107:0/2605870146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954410b7d0 0x7f9544112000 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.919+0000 7f9549e00640 1 --2- 172.21.15.107:0/2605870146 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f954410ae80 0x7f954410b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.919+0000 7f9549e00640 1 --2- 172.21.15.107:0/2605870146 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9544105550 0x7f9544105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.919+0000 7f9549e00640 1 -- 172.21.15.107:0/2605870146 >> 172.21.15.107:0/2605870146 conn(0x7f9544100f90 msgr2=0x7f95441033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.919+0000 7f9549e00640 1 -- 172.21.15.107:0/2605870146 shutdown_connections 2024-07-26T12:49:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.920+0000 7f9549e00640 1 -- 172.21.15.107:0/2605870146 wait complete. 2024-07-26T12:49:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.921+0000 7f9549e00640 1 Processor -- start 2024-07-26T12:49:11.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.921+0000 7f9549e00640 1 -- start start 2024-07-26T12:49:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.921+0000 7f9549e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9544105550 0x7f954419be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.921+0000 7f9549e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f954410ae80 0x7f954419c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.921+0000 7f9549e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954410b7d0 0x7f95441a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.921+0000 7f9549e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f95441154d0 con 0x7f954410b7d0 2024-07-26T12:49:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.921+0000 7f9549e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f9544115350 con 0x7f9544105550 2024-07-26T12:49:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.921+0000 7f9549e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f9544115650 con 0x7f954410ae80 2024-07-26T12:49:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9543e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f954410ae80 0x7f954419c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9549400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954410b7d0 0x7f95441a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9548a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9544105550 0x7f954419be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9543e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f954410ae80 0x7f954419c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:48566/0 (socket says 172.21.15.107:48566) 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9543e00640 1 -- 172.21.15.107:0/1024624944 learned_addr learned my addr 172.21.15.107:0/1024624944 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9543e00640 1 -- 172.21.15.107:0/1024624944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9544105550 msgr2=0x7f954419be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9543e00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9544105550 0x7f954419be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9543e00640 1 -- 172.21.15.107:0/1024624944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954410b7d0 msgr2=0x7f95441a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9543e00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954410b7d0 0x7f95441a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9543e00640 1 -- 172.21.15.107:0/1024624944 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95441a1d80 con 0x7f954410ae80 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9548a00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9544105550 0x7f954419be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.922+0000 7f9543e00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f954410ae80 0x7f954419c3f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f953400e970 tx=0x7f953400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.923+0000 7f9541600640 1 -- 172.21.15.107:0/1024624944 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953400cd80 con 0x7f954410ae80 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.923+0000 7f9541600640 1 -- 172.21.15.107:0/1024624944 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95340035f0 con 0x7f954410ae80 2024-07-26T12:49:11.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.923+0000 7f9549e00640 1 -- 172.21.15.107:0/1024624944 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95441a2070 con 0x7f954410ae80 2024-07-26T12:49:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.923+0000 7f9541600640 1 -- 172.21.15.107:0/1024624944 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9534010690 con 0x7f954410ae80 2024-07-26T12:49:11.927 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.923+0000 7f9549e00640 1 -- 172.21.15.107:0/1024624944 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95441a8800 con 0x7f954410ae80 2024-07-26T12:49:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.926+0000 7f9552412740 1 Processor -- start 2024-07-26T12:49:11.928 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.926+0000 7f9552412740 1 -- start start 2024-07-26T12:49:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9552412740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb71941b0 0x558fb7537c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9552412740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb716b2f0 0x558fb7538160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9552412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb75386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9552412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558fb74ac270 con 0x558fb71734d0 2024-07-26T12:49:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9552412740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558fb749b770 con 0x558fb716b2f0 2024-07-26T12:49:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9552412740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558fb749f7b0 con 0x558fb71941b0 2024-07-26T12:49:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9541600640 1 -- 172.21.15.107:0/1024624944 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f9534003aa0 con 0x7f954410ae80 2024-07-26T12:49:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9549400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb71941b0 0x558fb7537c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9549400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb71941b0 0x558fb7537c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:48574/0 (socket says 172.21.15.107:48574) 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9549400640 1 -- 172.21.15.107:0/114382856 learned_addr learned my addr 172.21.15.107:0/114382856 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9548a00640 1 --2- 172.21.15.107:0/114382856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb716b2f0 0x558fb7538160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.927+0000 7f9543e00640 1 --2- 172.21.15.107:0/114382856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb75386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.928+0000 7f9549400640 1 -- 172.21.15.107:0/114382856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb716b2f0 msgr2=0x558fb7538160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.928+0000 7f9549400640 1 --2- 172.21.15.107:0/114382856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb716b2f0 0x558fb7538160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.928+0000 7f9549400640 1 -- 172.21.15.107:0/114382856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 msgr2=0x558fb75386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.928+0000 7f9549400640 1 --2- 172.21.15.107:0/114382856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb75386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.928+0000 7f9549400640 1 -- 172.21.15.107:0/114382856 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fb71ff790 con 0x558fb71941b0 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.928+0000 7f9543e00640 1 --2- 172.21.15.107:0/114382856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb75386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.928+0000 7f9549400640 1 --2- 172.21.15.107:0/114382856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb71941b0 0x558fb7537c20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f953c035000 tx=0x7f953c030fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=398b9c291344dd2d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.928+0000 7f9531c00640 1 -- 172.21.15.107:0/114382856 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953c030800 con 0x558fb71941b0 2024-07-26T12:49:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.928+0000 7f9531c00640 1 -- 172.21.15.107:0/114382856 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f953c036070 con 0x558fb71941b0 2024-07-26T12:49:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.928+0000 7f9531c00640 1 -- 172.21.15.107:0/114382856 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953c0043a0 con 0x558fb71941b0 2024-07-26T12:49:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.929+0000 7f9541600640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f952c0b3dc0 0x7f952c0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.929+0000 7f9541600640 1 -- 172.21.15.107:0/1024624944 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f95340e45b0 con 0x7f954410ae80 2024-07-26T12:49:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.929+0000 7f9548a00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f952c0b3dc0 0x7f952c0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.929+0000 7f9552412740 1 -- 172.21.15.107:0/114382856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb71941b0 msgr2=0x558fb7537c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.929+0000 7f9552412740 1 --2- 172.21.15.107:0/114382856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb71941b0 0x558fb7537c20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f953c035000 tx=0x7f953c030fd0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.929+0000 7f9552412740 1 -- 172.21.15.107:0/114382856 shutdown_connections 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.929+0000 7f9552412740 1 --2- 172.21.15.107:0/114382856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb75386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.929+0000 7f9552412740 1 --2- 172.21.15.107:0/114382856 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb716b2f0 0x558fb7538160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.929+0000 7f9552412740 1 --2- 172.21.15.107:0/114382856 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb71941b0 0x558fb7537c20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.929+0000 7f9552412740 1 -- 172.21.15.107:0/114382856 >> 172.21.15.107:0/114382856 conn(0x558fb7188300 msgr2=0x558fb7535b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.930+0000 7f9552412740 1 -- 172.21.15.107:0/114382856 shutdown_connections 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.930+0000 7f9552412740 1 -- 172.21.15.107:0/114382856 wait complete. 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.930+0000 7f9548a00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f952c0b3dc0 0x7f952c0b62d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x558fb75368a0 tx=0x7f9538005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.930+0000 7f9552412740 1 -- start start 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.930+0000 7f9552412740 1 Processor -- start 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9552412740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7543810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9552412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb7543d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9552412740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 0x558fb754f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9552412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558fb73d5ab0 con 0x558fb71734d0 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9552412740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558fb7438d50 con 0x558fb71941b0 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9552412740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558fb7546340 con 0x558fb716b2f0 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9552412740 10 client.0 _ll_register_callbacks cb 0x558fb71aba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9548a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7543810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9543e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb7543d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9548a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7543810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:48586/0 (socket says 172.21.15.107:48586) 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9548a00640 1 -- 172.21.15.107:0/489274421 learned_addr learned my addr 172.21.15.107:0/489274421 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9543e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb7543d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59324/0 (socket says 172.21.15.107:59324) 2024-07-26T12:49:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.931+0000 7f9549400640 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 0x558fb754f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.932+0000 7f9548a00640 1 -- 172.21.15.107:0/489274421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 msgr2=0x558fb754f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.932+0000 7f9548a00640 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 0x558fb754f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.932+0000 7f9548a00640 1 -- 172.21.15.107:0/489274421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 msgr2=0x558fb7543d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.932+0000 7f9548a00640 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb7543d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.932+0000 7f9548a00640 1 -- 172.21.15.107:0/489274421 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fb7278620 con 0x558fb716b2f0 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.932+0000 7f9543e00640 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb7543d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.932+0000 7f9548a00640 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7543810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f95380035f0 tx=0x7f9538004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be86478a496f5e28 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.932+0000 7f9526000640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.932+0000 7f9526000640 1 -- 172.21.15.107:0/489274421 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953800f830 con 0x558fb716b2f0 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9552412740 1 -- 172.21.15.107:0/489274421 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f953c036070 con 0x558fb716b2f0 2024-07-26T12:49:11.939 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9526000640 1 -- 172.21.15.107:0/489274421 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f953800f9d0 con 0x558fb716b2f0 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9526000640 1 -- 172.21.15.107:0/489274421 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9538032050 con 0x558fb716b2f0 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9552412740 10 client.25564 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9552412740 1 -- 172.21.15.107:0/489274421 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558fb7552480 con 0x558fb716b2f0 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9526000640 1 -- 172.21.15.107:0/489274421 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f953c036070 con 0x558fb716b2f0 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9526000640 1 -- 172.21.15.107:0/489274421 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9538027020 con 0x558fb716b2f0 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9552412740 10 client.25564 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9552412740 10 client.25564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9552412740 20 client.25564 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9552412740 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558fb7152d80 0x558fb7555a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9552412740 4 client.25564 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.933+0000 7f9552412740 1 -- 172.21.15.107:0/489274421 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fb74ae1c0 con 0x558fb7152d80 2024-07-26T12:49:11.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.934+0000 7f9543e00640 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558fb7152d80 0x558fb7555a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.935+0000 7f9543e00640 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558fb7152d80 0x558fb7555a70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6df7a0e000e8d9cb server_cookie=e1d20cb3433afe83 in_seq=0 out_seq=0 2024-07-26T12:49:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.935+0000 7f9526000640 10 client.25564 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9526000640 1 -- 172.21.15.107:0/489274421 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558fb74ae1c0 con 0x558fb7152d80 2024-07-26T12:49:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9526000640 10 client.25564 handle_command_reply: tid=0 2024-07-26T12:49:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 client.25564 shutdown 2024-07-26T12:49:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 -- 172.21.15.107:0/489274421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 msgr2=0x558fb7543810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7543810 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f95380035f0 tx=0x7f9538004730 comp rx=0 tx=0).stop 2024-07-26T12:49:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 -- 172.21.15.107:0/489274421 shutdown_connections 2024-07-26T12:49:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558fb7152d80 0x558fb7555a70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 0x558fb754f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb7543d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9526000640 0 client.25564 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 --2- 172.21.15.107:0/489274421 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7543810 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 -- 172.21.15.107:0/489274421 >> 172.21.15.107:0/489274421 conn(0x558fb7188300 msgr2=0x558fb7538c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 -- 172.21.15.107:0/489274421 shutdown_connections 2024-07-26T12:49:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 1 -- 172.21.15.107:0/489274421 wait complete. 2024-07-26T12:49:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.936+0000 7f9552412740 20 client.25564 trim_cache size 0 max 16384 2024-07-26T12:49:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.953+0000 7f9552412740 1 Processor -- start 2024-07-26T12:49:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.953+0000 7f9552412740 1 -- start start 2024-07-26T12:49:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.953+0000 7f9552412740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7539720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.953+0000 7f9552412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb753dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.953+0000 7f9552412740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 0x558fb75454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.953+0000 7f9552412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f95441b2880 con 0x558fb71734d0 2024-07-26T12:49:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.953+0000 7f9552412740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f95441b43c0 con 0x558fb71941b0 2024-07-26T12:49:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.953+0000 7f9552412740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f95441b2f50 con 0x558fb716b2f0 2024-07-26T12:49:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.953+0000 7f9543e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb753dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9543e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb753dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59326/0 (socket says 172.21.15.107:59326) 2024-07-26T12:49:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9543e00640 1 -- 172.21.15.107:0/1644731982 learned_addr learned my addr 172.21.15.107:0/1644731982 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9549400640 1 --2- 172.21.15.107:0/1644731982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 0x558fb75454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9548a00640 1 --2- 172.21.15.107:0/1644731982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7539720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9543e00640 1 -- 172.21.15.107:0/1644731982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 msgr2=0x558fb7539720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9543e00640 1 --2- 172.21.15.107:0/1644731982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7539720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9543e00640 1 -- 172.21.15.107:0/1644731982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 msgr2=0x558fb75454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9543e00640 1 --2- 172.21.15.107:0/1644731982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 0x558fb75454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9543e00640 1 -- 172.21.15.107:0/1644731982 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fb7539f20 con 0x558fb71734d0 2024-07-26T12:49:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9549400640 1 --2- 172.21.15.107:0/1644731982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 0x558fb75454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.954+0000 7f9543e00640 1 --2- 172.21.15.107:0/1644731982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb753dd60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f95340a7380 tx=0x7f9534000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd9a0b58a26053e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.955+0000 7f9531200640 1 -- 172.21.15.107:0/1644731982 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95340ab4e0 con 0x558fb71734d0 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.955+0000 7f9531200640 1 -- 172.21.15.107:0/1644731982 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95340aade0 con 0x558fb71734d0 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.955+0000 7f9531200640 1 -- 172.21.15.107:0/1644731982 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95340e1da0 con 0x558fb71734d0 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.955+0000 7f9552412740 1 -- 172.21.15.107:0/1644731982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 msgr2=0x558fb753dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.955+0000 7f9552412740 1 --2- 172.21.15.107:0/1644731982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb753dd60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f95340a7380 tx=0x7f9534000bc0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.955+0000 7f9552412740 1 -- 172.21.15.107:0/1644731982 shutdown_connections 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.955+0000 7f9552412740 1 --2- 172.21.15.107:0/1644731982 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71941b0 0x558fb75454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.955+0000 7f9552412740 1 --2- 172.21.15.107:0/1644731982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71734d0 0x558fb753dd60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.955+0000 7f9552412740 1 --2- 172.21.15.107:0/1644731982 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7539720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.955+0000 7f9552412740 1 -- 172.21.15.107:0/1644731982 >> 172.21.15.107:0/1644731982 conn(0x558fb7188300 msgr2=0x558fb75340b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.956+0000 7f9552412740 1 -- 172.21.15.107:0/1644731982 shutdown_connections 2024-07-26T12:49:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.956+0000 7f9552412740 1 -- 172.21.15.107:0/1644731982 wait complete. 2024-07-26T12:49:11.960 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.956+0000 7f9552412740 1 -- start start 2024-07-26T12:49:11.960 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.956+0000 7f9552412740 1 Processor -- start 2024-07-26T12:49:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9552412740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7547d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9552412740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71734d0 0x558fb7548250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9552412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71941b0 0x558fb75bc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9552412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f95441b3cf0 con 0x558fb71941b0 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9552412740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f95441b3620 con 0x558fb71734d0 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9552412740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f95441b0860 con 0x558fb716b2f0 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9552412740 10 client.0 _ll_register_callbacks cb 0x7f9538009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9549400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71941b0 0x558fb75bc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9549400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71941b0 0x558fb75bc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59338/0 (socket says 172.21.15.107:59338) 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9549400640 1 -- 172.21.15.107:0/4068082081 learned_addr learned my addr 172.21.15.107:0/4068082081 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9543e00640 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71734d0 0x558fb7548250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9548a00640 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7547d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9549400640 1 -- 172.21.15.107:0/4068082081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 msgr2=0x558fb7547d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9549400640 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7547d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.957+0000 7f9549400640 1 -- 172.21.15.107:0/4068082081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71734d0 msgr2=0x558fb7548250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9549400640 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71734d0 0x558fb7548250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9549400640 1 -- 172.21.15.107:0/4068082081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fb7548af0 con 0x558fb71941b0 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9543e00640 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71734d0 0x558fb7548250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9549400640 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71941b0 0x558fb75bc1f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f953c004af0 tx=0x7f953c02a8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478d1898e260b738 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9526000640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9526000640 1 -- 172.21.15.107:0/4068082081 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953c030a10 con 0x558fb71941b0 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9526000640 1 -- 172.21.15.107:0/4068082081 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f953c030bb0 con 0x558fb71941b0 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9552412740 1 -- 172.21.15.107:0/4068082081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95340aade0 con 0x558fb71941b0 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9526000640 1 -- 172.21.15.107:0/4068082081 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f953c003f10 con 0x558fb71941b0 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9552412740 10 client.15429 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.958+0000 7f9552412740 1 -- 172.21.15.107:0/4068082081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558fb75bd790 con 0x558fb71941b0 2024-07-26T12:49:11.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.959+0000 7f9526000640 1 -- 172.21.15.107:0/4068082081 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95340aade0 con 0x558fb71941b0 2024-07-26T12:49:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.959+0000 7f9526000640 1 -- 172.21.15.107:0/4068082081 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f953c043020 con 0x558fb71941b0 2024-07-26T12:49:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.959+0000 7f9552412740 10 client.15429 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.959+0000 7f9552412740 10 client.15429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.959+0000 7f9552412740 20 client.15429 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.959+0000 7f9552412740 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558fb71281c0 0x558fb75bfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.959+0000 7f9552412740 4 client.15429 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T12:49:11.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.959+0000 7f9552412740 1 -- 172.21.15.107:0/4068082081 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558fb74a9c60 con 0x558fb71281c0 2024-07-26T12:49:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.960+0000 7f9548a00640 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558fb71281c0 0x558fb75bfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.960+0000 7f9548a00640 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558fb71281c0 0x558fb75bfad0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90f7952098e31d23 server_cookie=7d622618f7df9715 in_seq=0 out_seq=0 2024-07-26T12:49:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.960+0000 7f9526000640 10 client.15429 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9526000640 1 -- 172.21.15.107:0/4068082081 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558fb74a9c60 con 0x558fb71281c0 2024-07-26T12:49:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9526000640 10 client.15429 handle_command_reply: tid=0 2024-07-26T12:49:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9552412740 1 client.15429 shutdown 2024-07-26T12:49:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9552412740 1 -- 172.21.15.107:0/4068082081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71941b0 msgr2=0x558fb75bc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9552412740 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71941b0 0x558fb75bc1f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f953c004af0 tx=0x7f953c02a8b0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9552412740 1 -- 172.21.15.107:0/4068082081 shutdown_connections 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9552412740 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x558fb71281c0 0x558fb75bfad0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9552412740 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558fb71941b0 0x558fb75bc1f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9552412740 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558fb71734d0 0x558fb7548250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9552412740 1 --2- 172.21.15.107:0/4068082081 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558fb716b2f0 0x558fb7547d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9526000640 0 client.15429 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9552412740 1 -- 172.21.15.107:0/4068082081 >> 172.21.15.107:0/4068082081 conn(0x558fb7188300 msgr2=0x558fb75451d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.974+0000 7f9552412740 1 -- 172.21.15.107:0/4068082081 shutdown_connections 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9552412740 1 -- 172.21.15.107:0/4068082081 wait complete. 2024-07-26T12:49:11.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9552412740 20 client.15429 trim_cache size 0 max 16384 2024-07-26T12:49:11.978 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-07-26T12:49:11.978 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-07-26T12:49:11.978 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-07-26T12:49:11.978 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-07-26T12:49:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9549e00640 1 -- 172.21.15.107:0/1024624944 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f952c0b3dc0 msgr2=0x7f952c0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9549e00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f952c0b3dc0 0x7f952c0b62d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x558fb75368a0 tx=0x7f9538005f90 comp rx=0 tx=0).stop 2024-07-26T12:49:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9549e00640 1 -- 172.21.15.107:0/1024624944 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f954410ae80 msgr2=0x7f954419c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9549e00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f954410ae80 0x7f954419c3f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f953400e970 tx=0x7f953400ee30 comp rx=0 tx=0).stop 2024-07-26T12:49:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9549e00640 1 -- 172.21.15.107:0/1024624944 shutdown_connections 2024-07-26T12:49:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9549e00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f952c0b3dc0 0x7f952c0b62d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9549e00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f954410b7d0 0x7f95441a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9549e00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f954410ae80 0x7f954419c3f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.975+0000 7f9549e00640 1 --2- 172.21.15.107:0/1024624944 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f9544105550 0x7f954419be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.976+0000 7f9549e00640 1 -- 172.21.15.107:0/1024624944 >> 172.21.15.107:0/1024624944 conn(0x7f9544100f90 msgr2=0x7f95441033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.976+0000 7f9549e00640 1 -- 172.21.15.107:0/1024624944 shutdown_connections 2024-07-26T12:49:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:11.976+0000 7f9549e00640 1 -- 172.21.15.107:0/1024624944 wait complete. 2024-07-26T12:49:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:12 smithi112 ceph-mon[37501]: scrub summary: idle 2024-07-26T12:49:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:12 smithi107 ceph-mon[32659]: scrub summary: idle 2024-07-26T12:49:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:12 smithi132 ceph-mon[37580]: scrub summary: idle 2024-07-26T12:49:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:13 smithi112 ceph-mon[37501]: pgmap v487: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 126 KiB/s wr, 66 op/s 2024-07-26T12:49:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:13 smithi107 ceph-mon[32659]: pgmap v487: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 126 KiB/s wr, 66 op/s 2024-07-26T12:49:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:13 smithi132 ceph-mon[37580]: pgmap v487: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 126 KiB/s wr, 66 op/s 2024-07-26T12:49:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:14 smithi112 ceph-mon[37501]: pgmap v488: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-07-26T12:49:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:14 smithi107 ceph-mon[32659]: pgmap v488: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-07-26T12:49:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:14 smithi132 ceph-mon[37580]: pgmap v488: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-07-26T12:49:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:16 smithi112 ceph-mon[37501]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-07-26T12:49:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:16 smithi107 ceph-mon[32659]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-07-26T12:49:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:16 smithi132 ceph-mon[37580]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-07-26T12:49:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:18 smithi112 ceph-mon[37501]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-07-26T12:49:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:18 smithi107 ceph-mon[32659]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-07-26T12:49:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:18 smithi132 ceph-mon[37580]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-07-26T12:49:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:20 smithi112 ceph-mon[37501]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-07-26T12:49:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:20 smithi107 ceph-mon[32659]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-07-26T12:49:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:20 smithi132 ceph-mon[37580]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-07-26T12:49:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:23 smithi107 ceph-mon[32659]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 16 op/s 2024-07-26T12:49:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:49:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:23 smithi132 ceph-mon[37580]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 16 op/s 2024-07-26T12:49:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:49:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:23 smithi112 ceph-mon[37501]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 16 op/s 2024-07-26T12:49:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:49:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:24 smithi112 ceph-mon[37501]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 13 op/s 2024-07-26T12:49:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.232+0000 7f57c1200640 1 -- 172.21.15.107:0/344839210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57bc105540 msgr2=0x7f57bc105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.232+0000 7f57c1200640 1 --2- 172.21.15.107:0/344839210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57bc105540 0x7f57bc105950 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009900 tx=0x7f57ac02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.232+0000 7f57c1200640 1 -- 172.21.15.107:0/344839210 shutdown_connections 2024-07-26T12:49:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.232+0000 7f57c1200640 1 --2- 172.21.15.107:0/344839210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57bc10b7c0 0x7f57bc111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.232+0000 7f57c1200640 1 --2- 172.21.15.107:0/344839210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57bc10ae70 0x7f57bc10b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.232+0000 7f57c1200640 1 --2- 172.21.15.107:0/344839210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57bc105540 0x7f57bc105950 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.232+0000 7f57c1200640 1 -- 172.21.15.107:0/344839210 >> 172.21.15.107:0/344839210 conn(0x7f57bc100f20 msgr2=0x7f57bc103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.232+0000 7f57c1200640 1 -- 172.21.15.107:0/344839210 shutdown_connections 2024-07-26T12:49:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.232+0000 7f57c1200640 1 -- 172.21.15.107:0/344839210 wait complete. 2024-07-26T12:49:25.236 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.233+0000 7f57c1200640 1 Processor -- start 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57c1200640 1 -- start start 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57c1200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57bc105540 0x7f57bc19bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57c1200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57bc10ae70 0x7f57bc19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57c1200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57bc10b7c0 0x7f57bc1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57c1200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1154c0 con 0x7f57bc10ae70 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57c1200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc115340 con 0x7f57bc10b7c0 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57c1200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc115640 con 0x7f57bc105540 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57bb400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57bc105540 0x7f57bc19bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57baa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57bc10ae70 0x7f57bc19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57bb400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57bc105540 0x7f57bc19bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:35836/0 (socket says 172.21.15.107:35836) 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57bb400640 1 -- 172.21.15.107:0/1764109512 learned_addr learned my addr 172.21.15.107:0/1764109512 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.234+0000 7f57bbe00640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57bc10b7c0 0x7f57bc1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57bb400640 1 -- 172.21.15.107:0/1764109512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57bc10b7c0 msgr2=0x7f57bc1a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57bb400640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57bc10b7c0 0x7f57bc1a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57bb400640 1 -- 172.21.15.107:0/1764109512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57bc10ae70 msgr2=0x7f57bc19c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57bb400640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57bc10ae70 0x7f57bc19c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57bb400640 1 -- 172.21.15.107:0/1764109512 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57bc1a1e10 con 0x7f57bc105540 2024-07-26T12:49:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57baa00640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57bc10ae70 0x7f57bc19c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57bb400640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57bc105540 0x7f57bc19bee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f57ac005ab0 tx=0x7f57ac003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:25.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57b3e00640 1 -- 172.21.15.107:0/1764109512 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac03f070 con 0x7f57bc105540 2024-07-26T12:49:25.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57c1200640 1 -- 172.21.15.107:0/1764109512 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57bc1a20a0 con 0x7f57bc105540 2024-07-26T12:49:25.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57b3e00640 1 -- 172.21.15.107:0/1764109512 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57ac031b60 con 0x7f57bc105540 2024-07-26T12:49:25.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.235+0000 7f57b3e00640 1 -- 172.21.15.107:0/1764109512 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac035030 con 0x7f57bc105540 2024-07-26T12:49:25.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.236+0000 7f57c1200640 1 -- 172.21.15.107:0/1764109512 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57bc1a8890 con 0x7f57bc105540 2024-07-26T12:49:25.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.239+0000 7f57b3e00640 1 -- 172.21.15.107:0/1764109512 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f57ac042070 con 0x7f57bc105540 2024-07-26T12:49:25.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.240+0000 7f57c9799740 1 Processor -- start 2024-07-26T12:49:25.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.240+0000 7f57c9799740 1 -- start start 2024-07-26T12:49:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57c9799740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457211b0 0x559445ac4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57c9799740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594456f82f0 0x559445ac5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57c9799740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 0x559445ac56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57c9799740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559445a39270 con 0x5594456f82f0 2024-07-26T12:49:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57c9799740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559445a28770 con 0x5594457004d0 2024-07-26T12:49:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57c9799740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559445a2c7b0 con 0x5594457211b0 2024-07-26T12:49:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57b3e00640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f57a00b3d40 0x7f57a00b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57bb400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594456f82f0 0x559445ac5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57baa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 0x559445ac56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57bbe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457211b0 0x559445ac4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57baa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 0x559445ac56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:36516/0 (socket says 172.21.15.107:36516) 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57baa00640 1 -- 172.21.15.107:0/211521143 learned_addr learned my addr 172.21.15.107:0/211521143 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.241+0000 7f57b3e00640 1 -- 172.21.15.107:0/1764109512 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f57ac104ad0 con 0x7f57bc105540 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57bbe00640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f57a00b3d40 0x7f57a00b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57baa00640 1 -- 172.21.15.107:0/211521143 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457211b0 msgr2=0x559445ac4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57baa00640 1 --2- 172.21.15.107:0/211521143 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457211b0 0x559445ac4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57baa00640 1 -- 172.21.15.107:0/211521143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594456f82f0 msgr2=0x559445ac5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57baa00640 1 --2- 172.21.15.107:0/211521143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594456f82f0 0x559445ac5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57baa00640 1 -- 172.21.15.107:0/211521143 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55944578c790 con 0x5594457004d0 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57baa00640 1 --2- 172.21.15.107:0/211521143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 0x559445ac56c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f57a80114a0 tx=0x7f57a8011960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6eefc4a4c3227ef server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57b0c00640 1 -- 172.21.15.107:0/211521143 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a801e070 con 0x5594457004d0 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57bbe00640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f57a00b3d40 0x7f57a00b6250 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f57b4005c40 tx=0x7f57b4013040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57b0c00640 1 -- 172.21.15.107:0/211521143 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a800fad0 con 0x5594457004d0 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.242+0000 7f57b0c00640 1 -- 172.21.15.107:0/211521143 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a8019ac0 con 0x5594457004d0 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.244+0000 7f57c9799740 1 -- 172.21.15.107:0/211521143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 msgr2=0x559445ac56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.244+0000 7f57c9799740 1 --2- 172.21.15.107:0/211521143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 0x559445ac56c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f57a80114a0 tx=0x7f57a8011960 comp rx=0 tx=0).stop 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.244+0000 7f57c9799740 1 -- 172.21.15.107:0/211521143 shutdown_connections 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.244+0000 7f57c9799740 1 --2- 172.21.15.107:0/211521143 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 0x559445ac56c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.244+0000 7f57c9799740 1 --2- 172.21.15.107:0/211521143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594456f82f0 0x559445ac5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.244+0000 7f57c9799740 1 --2- 172.21.15.107:0/211521143 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457211b0 0x559445ac4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.244+0000 7f57c9799740 1 -- 172.21.15.107:0/211521143 >> 172.21.15.107:0/211521143 conn(0x559445715300 msgr2=0x559445ac2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.244+0000 7f57c9799740 1 -- 172.21.15.107:0/211521143 shutdown_connections 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.244+0000 7f57c9799740 1 -- 172.21.15.107:0/211521143 wait complete. 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.245+0000 7f57c9799740 1 -- start start 2024-07-26T12:49:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.245+0000 7f57c9799740 1 Processor -- start 2024-07-26T12:49:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57c9799740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594456f82f0 0x559445ad0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57c9799740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 0x559445ad0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57c9799740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457211b0 0x559445adc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57c9799740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559445962ab0 con 0x5594457211b0 2024-07-26T12:49:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57c9799740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5594459c5d50 con 0x5594457004d0 2024-07-26T12:49:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57c9799740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559445ad33c0 con 0x5594456f82f0 2024-07-26T12:49:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57c9799740 10 client.0 _ll_register_callbacks cb 0x559445738a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57bbe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457211b0 0x559445adc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57bb400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594456f82f0 0x559445ad0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57bb400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594456f82f0 0x559445ad0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:35854/0 (socket says 172.21.15.107:35854) 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57bb400640 1 -- 172.21.15.107:0/1303799170 learned_addr learned my addr 172.21.15.107:0/1303799170 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.246+0000 7f57baa00640 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 0x559445ad0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57bb400640 1 -- 172.21.15.107:0/1303799170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 msgr2=0x559445ad0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57bb400640 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 0x559445ad0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57bb400640 1 -- 172.21.15.107:0/1303799170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457211b0 msgr2=0x559445adc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57bb400640 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457211b0 0x559445adc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57bb400640 1 -- 172.21.15.107:0/1303799170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559445805620 con 0x5594456f82f0 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57bb400640 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594456f82f0 0x559445ad0820 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0cb0b0 tx=0x7f57ac030fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4fe0f28d3362a4a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57a5600640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57a5600640 1 -- 172.21.15.107:0/1303799170 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac049030 con 0x5594456f82f0 2024-07-26T12:49:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57a5600640 1 -- 172.21.15.107:0/1303799170 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f57ac1020c0 con 0x5594456f82f0 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57c9799740 1 -- 172.21.15.107:0/1303799170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57a800fad0 con 0x5594456f82f0 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57a5600640 1 -- 172.21.15.107:0/1303799170 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f57ac1022c0 con 0x5594456f82f0 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57c9799740 10 client.25582 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.247+0000 7f57c9799740 1 -- 172.21.15.107:0/1303799170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559445adf400 con 0x5594456f82f0 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.248+0000 7f57a5600640 1 -- 172.21.15.107:0/1303799170 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57a800fad0 con 0x5594456f82f0 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.248+0000 7f57a5600640 1 -- 172.21.15.107:0/1303799170 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f57ac0cb610 con 0x5594456f82f0 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.248+0000 7f57c9799740 10 client.25582 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.248+0000 7f57c9799740 10 client.25582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.248+0000 7f57c9799740 20 client.25582 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.248+0000 7f57c9799740 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5594456dfd80 0x559445ae2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.248+0000 7f57c9799740 4 client.25582 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.248+0000 7f57c9799740 1 -- 172.21.15.107:0/1303799170 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559445a3b1c0 con 0x5594456dfd80 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.248+0000 7f57baa00640 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5594456dfd80 0x559445ae2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.249+0000 7f57a5600640 10 client.25582 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.249+0000 7f57baa00640 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5594456dfd80 0x559445ae2900 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66f952205919cf1e server_cookie=e632cf808ae27ae8 in_seq=0 out_seq=0 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57a5600640 1 -- 172.21.15.107:0/1303799170 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559445a3b1c0 con 0x5594456dfd80 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57a5600640 10 client.25582 handle_command_reply: tid=0 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 client.25582 shutdown 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 -- 172.21.15.107:0/1303799170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594456f82f0 msgr2=0x559445ad0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594456f82f0 0x559445ad0820 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0cb0b0 tx=0x7f57ac030fc0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 -- 172.21.15.107:0/1303799170 shutdown_connections 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5594456dfd80 0x559445ae2900 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457211b0 0x559445adc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57a5600640 0 client.25582 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594457004d0 0x559445ad0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 --2- 172.21.15.107:0/1303799170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594456f82f0 0x559445ad0820 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 -- 172.21.15.107:0/1303799170 >> 172.21.15.107:0/1303799170 conn(0x559445715300 msgr2=0x559445ac5c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 -- 172.21.15.107:0/1303799170 shutdown_connections 2024-07-26T12:49:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.250+0000 7f57c9799740 1 -- 172.21.15.107:0/1303799170 wait complete. 2024-07-26T12:49:25.257 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.251+0000 7f57c9799740 20 client.25582 trim_cache size 0 max 16384 2024-07-26T12:49:25.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.276+0000 7f57c9799740 1 Processor -- start 2024-07-26T12:49:25.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.276+0000 7f57c9799740 1 -- start start 2024-07-26T12:49:25.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.276+0000 7f57c9799740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 0x559445ad2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.276+0000 7f57c9799740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457004d0 0x559445acadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.276+0000 7f57c9799740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457211b0 0x559445adc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.276+0000 7f57c9799740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1b2860 con 0x5594457211b0 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.276+0000 7f57c9799740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1b43a0 con 0x5594456f82f0 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.276+0000 7f57c9799740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1b2f30 con 0x5594457004d0 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57bb400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 0x559445ad2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57bbe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457211b0 0x559445adc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57bb400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 0x559445ad2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:36542/0 (socket says 172.21.15.107:36542) 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57bb400640 1 -- 172.21.15.107:0/1598964269 learned_addr learned my addr 172.21.15.107:0/1598964269 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57baa00640 1 --2- 172.21.15.107:0/1598964269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457004d0 0x559445acadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57bb400640 1 -- 172.21.15.107:0/1598964269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457004d0 msgr2=0x559445acadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57bb400640 1 --2- 172.21.15.107:0/1598964269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457004d0 0x559445acadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57bb400640 1 -- 172.21.15.107:0/1598964269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457211b0 msgr2=0x559445adc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:25.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57bb400640 1 --2- 172.21.15.107:0/1598964269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457211b0 0x559445adc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57bb400640 1 -- 172.21.15.107:0/1598964269 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559445ac7170 con 0x5594456f82f0 2024-07-26T12:49:25.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.277+0000 7f57bb400640 1 --2- 172.21.15.107:0/1598964269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 0x559445ad2600 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0cb9b0 tx=0x7f57ac009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2625274d89daaf54 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:25.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.278+0000 7f57a7e00640 1 -- 172.21.15.107:0/1598964269 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0c3030 con 0x5594456f82f0 2024-07-26T12:49:25.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.278+0000 7f57a7e00640 1 -- 172.21.15.107:0/1598964269 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57ac0cf900 con 0x5594456f82f0 2024-07-26T12:49:25.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.278+0000 7f57a7e00640 1 -- 172.21.15.107:0/1598964269 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0ca7a0 con 0x5594456f82f0 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.279+0000 7f57c9799740 1 -- 172.21.15.107:0/1598964269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 msgr2=0x559445ad2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.279+0000 7f57c9799740 1 --2- 172.21.15.107:0/1598964269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 0x559445ad2600 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0cb9b0 tx=0x7f57ac009260 comp rx=0 tx=0).stop 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.279+0000 7f57c9799740 1 -- 172.21.15.107:0/1598964269 shutdown_connections 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.279+0000 7f57c9799740 1 --2- 172.21.15.107:0/1598964269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457211b0 0x559445adc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.279+0000 7f57c9799740 1 --2- 172.21.15.107:0/1598964269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457004d0 0x559445acadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.279+0000 7f57c9799740 1 --2- 172.21.15.107:0/1598964269 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 0x559445ad2600 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.279+0000 7f57c9799740 1 -- 172.21.15.107:0/1598964269 >> 172.21.15.107:0/1598964269 conn(0x559445715300 msgr2=0x559445aca6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.279+0000 7f57c9799740 1 -- 172.21.15.107:0/1598964269 shutdown_connections 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.279+0000 7f57c9799740 1 -- 172.21.15.107:0/1598964269 wait complete. 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.280+0000 7f57c9799740 1 -- start start 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.280+0000 7f57c9799740 1 Processor -- start 2024-07-26T12:49:25.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57c9799740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 0x559445ad8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57c9799740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457004d0 0x559445ad3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57c9799740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457211b0 0x559445ad4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57c9799740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1b3cd0 con 0x5594457004d0 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57c9799740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1b3600 con 0x5594456f82f0 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57c9799740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f57bc1b0840 con 0x5594457211b0 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57c9799740 10 client.0 _ll_register_callbacks cb 0x7f57ac0cb180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57baa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457004d0 0x559445ad3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57baa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457004d0 0x559445ad3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37918/0 (socket says 172.21.15.107:37918) 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57baa00640 1 -- 172.21.15.107:0/2838996892 learned_addr learned my addr 172.21.15.107:0/2838996892 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57bb400640 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 0x559445ad8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57bbe00640 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457211b0 0x559445ad4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57baa00640 1 -- 172.21.15.107:0/2838996892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457211b0 msgr2=0x559445ad4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57baa00640 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457211b0 0x559445ad4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57baa00640 1 -- 172.21.15.107:0/2838996892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 msgr2=0x559445ad8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57baa00640 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 0x559445ad8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.281+0000 7f57baa00640 1 -- 172.21.15.107:0/2838996892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559445ad49c0 con 0x5594457004d0 2024-07-26T12:49:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.282+0000 7f57baa00640 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457004d0 0x559445ad3f40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f57a80058c0 tx=0x7f57a8009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d077b0e18220b10 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.282+0000 7f57a5600640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:49:25.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.282+0000 7f57a5600640 1 -- 172.21.15.107:0/2838996892 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57a801e040 con 0x5594457004d0 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.282+0000 7f57a5600640 1 -- 172.21.15.107:0/2838996892 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f57a8011040 con 0x5594457004d0 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.282+0000 7f57c9799740 1 -- 172.21.15.107:0/2838996892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57ac0cf900 con 0x5594457004d0 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.282+0000 7f57a5600640 1 -- 172.21.15.107:0/2838996892 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f57a8011240 con 0x5594457004d0 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.282+0000 7f57c9799740 10 client.15447 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.282+0000 7f57c9799740 1 -- 172.21.15.107:0/2838996892 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559445b4ae30 con 0x5594457004d0 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.283+0000 7f57a5600640 1 -- 172.21.15.107:0/2838996892 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57ac0cf900 con 0x5594457004d0 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.283+0000 7f57a5600640 1 -- 172.21.15.107:0/2838996892 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f57a80196b0 con 0x5594457004d0 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.283+0000 7f57c9799740 10 client.15447 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.283+0000 7f57c9799740 10 client.15447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.283+0000 7f57c9799740 20 client.15447 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.283+0000 7f57c9799740 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5594456b51c0 0x559445b4d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:25.287 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.283+0000 7f57c9799740 4 client.15447 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T12:49:25.288 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.283+0000 7f57c9799740 1 -- 172.21.15.107:0/2838996892 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559445a36c60 con 0x5594456b51c0 2024-07-26T12:49:25.288 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.284+0000 7f57bb400640 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5594456b51c0 0x559445b4d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:25.288 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.284+0000 7f57bb400640 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5594456b51c0 0x559445b4d460 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bab0b9d866324ec5 server_cookie=cb3a5fb49ddc68a0 in_seq=0 out_seq=0 2024-07-26T12:49:25.288 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.284+0000 7f57a5600640 10 client.15447 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:25.289 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57a5600640 1 -- 172.21.15.107:0/2838996892 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559445a36c60 con 0x5594456b51c0 2024-07-26T12:49:25.290 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57a5600640 10 client.15447 handle_command_reply: tid=0 2024-07-26T12:49:25.290 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57c9799740 1 client.15447 shutdown 2024-07-26T12:49:25.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57c9799740 1 -- 172.21.15.107:0/2838996892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457004d0 msgr2=0x559445ad3f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:25.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57c9799740 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457004d0 0x559445ad3f40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f57a80058c0 tx=0x7f57a8009fa0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57c9799740 1 -- 172.21.15.107:0/2838996892 shutdown_connections 2024-07-26T12:49:25.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57c9799740 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5594456b51c0 0x559445b4d460 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57c9799740 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5594457211b0 0x559445ad4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57c9799740 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5594457004d0 0x559445ad3f40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57a5600640 0 client.15447 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:25.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57c9799740 1 --2- 172.21.15.107:0/2838996892 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5594456f82f0 0x559445ad8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.288+0000 7f57c9799740 1 -- 172.21.15.107:0/2838996892 >> 172.21.15.107:0/2838996892 conn(0x559445715300 msgr2=0x559445adb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:25.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.289+0000 7f57c9799740 1 -- 172.21.15.107:0/2838996892 shutdown_connections 2024-07-26T12:49:25.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.289+0000 7f57c9799740 1 -- 172.21.15.107:0/2838996892 wait complete. 2024-07-26T12:49:25.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.289+0000 7f57c9799740 20 client.15447 trim_cache size 0 max 16384 2024-07-26T12:49:25.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-07-26T12:49:25.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-07-26T12:49:25.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-07-26T12:49:25.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-07-26T12:49:25.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 -- 172.21.15.107:0/1764109512 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f57a00b3d40 msgr2=0x7f57a00b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:25.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f57a00b3d40 0x7f57a00b6250 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f57b4005c40 tx=0x7f57b4013040 comp rx=0 tx=0).stop 2024-07-26T12:49:25.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 -- 172.21.15.107:0/1764109512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57bc105540 msgr2=0x7f57bc19bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:25.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57bc105540 0x7f57bc19bee0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f57ac005ab0 tx=0x7f57ac003c10 comp rx=0 tx=0).stop 2024-07-26T12:49:25.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 -- 172.21.15.107:0/1764109512 shutdown_connections 2024-07-26T12:49:25.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f57a00b3d40 0x7f57a00b6250 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f57bc10b7c0 0x7f57bc1a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57bc10ae70 0x7f57bc19c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 --2- 172.21.15.107:0/1764109512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57bc105540 0x7f57bc19bee0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:25.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 -- 172.21.15.107:0/1764109512 >> 172.21.15.107:0/1764109512 conn(0x7f57bc100f20 msgr2=0x7f57bc103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:25.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:24 smithi107 ceph-mon[32659]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 13 op/s 2024-07-26T12:49:25.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 -- 172.21.15.107:0/1764109512 shutdown_connections 2024-07-26T12:49:25.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:25.290+0000 7f57c1200640 1 -- 172.21.15.107:0/1764109512 wait complete. 2024-07-26T12:49:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:24 smithi132 ceph-mon[37580]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 938 B/s wr, 13 op/s 2024-07-26T12:49:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:26 smithi112 ceph-mon[37501]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T12:49:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:26 smithi107 ceph-mon[32659]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T12:49:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:26 smithi132 ceph-mon[37580]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 11 op/s 2024-07-26T12:49:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:28 smithi112 ceph-mon[37501]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T12:49:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:28 smithi107 ceph-mon[32659]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T12:49:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:28 smithi132 ceph-mon[37580]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T12:49:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:30 smithi112 ceph-mon[37501]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:49:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:30 smithi107 ceph-mon[32659]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:49:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:30 smithi132 ceph-mon[37580]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:49:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:32 smithi112 ceph-mon[37501]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-07-26T12:49:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:32 smithi107 ceph-mon[32659]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-07-26T12:49:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:32 smithi132 ceph-mon[37580]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-07-26T12:49:33.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T12:49:34.277 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:49:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:34 smithi112 ceph-mon[37501]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-07-26T12:49:35.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:34 smithi107 ceph-mon[32659]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-07-26T12:49:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:34 smithi132 ceph-mon[37580]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-07-26T12:49:37.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.139+0000 7ff363200640 1 -- 172.21.15.107:0/3896200237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff364076db0 msgr2=0x7ff3640751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:37.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.139+0000 7ff363200640 1 --2- 172.21.15.107:0/3896200237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff364076db0 0x7ff3640751e0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff354009900 tx=0x7ff35402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.139+0000 7ff363200640 1 -- 172.21.15.107:0/3896200237 shutdown_connections 2024-07-26T12:49:37.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.139+0000 7ff363200640 1 --2- 172.21.15.107:0/3896200237 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff364113cb0 0x7ff364116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.139+0000 7ff363200640 1 --2- 172.21.15.107:0/3896200237 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff364075800 0x7ff364075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.139+0000 7ff363200640 1 --2- 172.21.15.107:0/3896200237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff364076db0 0x7ff3640751e0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.139+0000 7ff363200640 1 -- 172.21.15.107:0/3896200237 >> 172.21.15.107:0/3896200237 conn(0x7ff3640fee70 msgr2=0x7ff3641012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:37.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.139+0000 7ff363200640 1 -- 172.21.15.107:0/3896200237 shutdown_connections 2024-07-26T12:49:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.139+0000 7ff363200640 1 -- 172.21.15.107:0/3896200237 wait complete. 2024-07-26T12:49:37.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.141+0000 7ff363200640 1 Processor -- start 2024-07-26T12:49:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.141+0000 7ff363200640 1 -- start start 2024-07-26T12:49:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.141+0000 7ff363200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff364075800 0x7ff3641a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.141+0000 7ff363200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff364076db0 0x7ff3641a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.141+0000 7ff363200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff364113cb0 0x7ff3641a5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.141+0000 7ff363200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff3641198f0 con 0x7ff364113cb0 2024-07-26T12:49:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.141+0000 7ff363200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff364119770 con 0x7ff364075800 2024-07-26T12:49:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.141+0000 7ff363200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff364119a70 con 0x7ff364076db0 2024-07-26T12:49:37.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.142+0000 7ff361e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff364075800 0x7ff3641a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.142+0000 7ff361e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff364075800 0x7ff3641a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:41420/0 (socket says 172.21.15.107:41420) 2024-07-26T12:49:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.142+0000 7ff361e00640 1 -- 172.21.15.107:0/269536189 learned_addr learned my addr 172.21.15.107:0/269536189 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.142+0000 7ff361400640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff364076db0 0x7ff3641a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.142+0000 7ff362800640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff364113cb0 0x7ff3641a5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.142+0000 7ff361e00640 1 -- 172.21.15.107:0/269536189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff364076db0 msgr2=0x7ff3641a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.142+0000 7ff361e00640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff364076db0 0x7ff3641a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.142+0000 7ff361e00640 1 -- 172.21.15.107:0/269536189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff364113cb0 msgr2=0x7ff3641a5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.142+0000 7ff361e00640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff364113cb0 0x7ff3641a5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.142+0000 7ff361e00640 1 -- 172.21.15.107:0/269536189 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3641a6290 con 0x7ff364075800 2024-07-26T12:49:37.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.143+0000 7ff361e00640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff364075800 0x7ff3641a0320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff354005940 tx=0x7ff3540040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:37.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.143+0000 7ff35aa00640 1 -- 172.21.15.107:0/269536189 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35403f030 con 0x7ff364075800 2024-07-26T12:49:37.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.143+0000 7ff35aa00640 1 -- 172.21.15.107:0/269536189 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3540044c0 con 0x7ff364075800 2024-07-26T12:49:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.145+0000 7ff363200640 1 -- 172.21.15.107:0/269536189 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3641a6520 con 0x7ff364075800 2024-07-26T12:49:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.145+0000 7ff363200640 1 -- 172.21.15.107:0/269536189 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3641acce0 con 0x7ff364075800 2024-07-26T12:49:37.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.145+0000 7ff35aa00640 1 -- 172.21.15.107:0/269536189 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff354035070 con 0x7ff364075800 2024-07-26T12:49:37.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.148+0000 7ff36fe49740 1 Processor -- start 2024-07-26T12:49:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.148+0000 7ff36fe49740 1 -- start start 2024-07-26T12:49:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff35aa00640 1 -- 172.21.15.107:0/269536189 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7ff354042030 con 0x7ff364075800 2024-07-26T12:49:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 0x55ce412f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f271b0 0x55ce412f5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f2f390 0x55ce412f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff36fe49740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ce4126bb90 con 0x55ce40f2f390 2024-07-26T12:49:37.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff36fe49740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ce4125b090 con 0x55ce40f271b0 2024-07-26T12:49:37.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff36fe49740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ce4125f0d0 con 0x55ce40f50070 2024-07-26T12:49:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff362800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 0x55ce412f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff361e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f271b0 0x55ce412f5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff362800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 0x55ce412f4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:59394/0 (socket says 172.21.15.107:59394) 2024-07-26T12:49:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff362800640 1 -- 172.21.15.107:0/2547645109 learned_addr learned my addr 172.21.15.107:0/2547645109 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.149+0000 7ff361400640 1 --2- 172.21.15.107:0/2547645109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f2f390 0x55ce412f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.150+0000 7ff362800640 1 -- 172.21.15.107:0/2547645109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f271b0 msgr2=0x55ce412f5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.150+0000 7ff362800640 1 --2- 172.21.15.107:0/2547645109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f271b0 0x55ce412f5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.150+0000 7ff362800640 1 -- 172.21.15.107:0/2547645109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f2f390 msgr2=0x55ce412f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:37.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.150+0000 7ff362800640 1 --2- 172.21.15.107:0/2547645109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f2f390 0x55ce412f5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.150+0000 7ff362800640 1 -- 172.21.15.107:0/2547645109 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce40fbb650 con 0x55ce40f50070 2024-07-26T12:49:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.151+0000 7ff361400640 1 --2- 172.21.15.107:0/2547645109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f2f390 0x55ce412f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.151+0000 7ff362800640 1 --2- 172.21.15.107:0/2547645109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 0x55ce412f4fd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009fc0 tx=0x7ff35c010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35a8a1bbcd82aeaf server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.151+0000 7ff34f400640 1 -- 172.21.15.107:0/2547645109 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c003b70 con 0x55ce40f50070 2024-07-26T12:49:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.151+0000 7ff34f400640 1 -- 172.21.15.107:0/2547645109 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff35c003d10 con 0x55ce40f50070 2024-07-26T12:49:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.151+0000 7ff34f400640 1 -- 172.21.15.107:0/2547645109 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c011a40 con 0x55ce40f50070 2024-07-26T12:49:37.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.152+0000 7ff35aa00640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff3480b3d40 0x7ff3480b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.152+0000 7ff35aa00640 1 -- 172.21.15.107:0/269536189 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff354104320 con 0x7ff364075800 2024-07-26T12:49:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.152+0000 7ff361e00640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff3480b3d40 0x7ff3480b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.152+0000 7ff36fe49740 1 -- 172.21.15.107:0/2547645109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 msgr2=0x55ce412f4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.152+0000 7ff36fe49740 1 --2- 172.21.15.107:0/2547645109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 0x55ce412f4fd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009fc0 tx=0x7ff35c010860 comp rx=0 tx=0).stop 2024-07-26T12:49:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.153+0000 7ff36fe49740 1 -- 172.21.15.107:0/2547645109 shutdown_connections 2024-07-26T12:49:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.153+0000 7ff36fe49740 1 --2- 172.21.15.107:0/2547645109 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f2f390 0x55ce412f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.153+0000 7ff36fe49740 1 --2- 172.21.15.107:0/2547645109 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f271b0 0x55ce412f5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.153+0000 7ff36fe49740 1 --2- 172.21.15.107:0/2547645109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 0x55ce412f4fd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.153+0000 7ff36fe49740 1 -- 172.21.15.107:0/2547645109 >> 172.21.15.107:0/2547645109 conn(0x55ce40f44460 msgr2=0x55ce412f2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.153+0000 7ff36fe49740 1 -- 172.21.15.107:0/2547645109 shutdown_connections 2024-07-26T12:49:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.153+0000 7ff36fe49740 1 -- 172.21.15.107:0/2547645109 wait complete. 2024-07-26T12:49:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.153+0000 7ff361e00640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff3480b3d40 0x7ff3480b6250 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55ce412f3c50 tx=0x7ff354107dd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:37.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.154+0000 7ff36fe49740 1 -- start start 2024-07-26T12:49:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.154+0000 7ff36fe49740 1 Processor -- start 2024-07-26T12:49:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41300560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 0x55ce41300aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f50070 0x55ce41300fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff36fe49740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ce410be700 con 0x55ce40f271b0 2024-07-26T12:49:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff36fe49740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55ce411f7660 con 0x55ce40f50070 2024-07-26T12:49:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff36fe49740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55ce413037b0 con 0x55ce40f2f390 2024-07-26T12:49:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff36fe49740 10 client.0 _ll_register_callbacks cb 0x55ce41207110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff362800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41300560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff362800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41300560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41118/0 (socket says 172.21.15.107:41118) 2024-07-26T12:49:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff362800640 1 -- 172.21.15.107:0/2371420966 learned_addr learned my addr 172.21.15.107:0/2371420966 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff361400640 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 0x55ce41300aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.155+0000 7ff362800640 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f50070 0x55ce41300fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.156+0000 7ff362800640 1 -- 172.21.15.107:0/2371420966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 msgr2=0x55ce41300aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:37.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.156+0000 7ff362800640 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 0x55ce41300aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.156+0000 7ff362800640 1 -- 172.21.15.107:0/2371420966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f50070 msgr2=0x55ce41300fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.156+0000 7ff362800640 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f50070 0x55ce41300fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.156+0000 7ff362800640 1 -- 172.21.15.107:0/2371420966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce4120e800 con 0x55ce40f271b0 2024-07-26T12:49:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.156+0000 7ff362800640 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41300560 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff35c004980 tx=0x7ff35c013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fa470ef8b8f352 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.156+0000 7ff343e00640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:49:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.157+0000 7ff343e00640 1 -- 172.21.15.107:0/2371420966 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c011d70 con 0x55ce40f271b0 2024-07-26T12:49:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.157+0000 7ff343e00640 1 -- 172.21.15.107:0/2371420966 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff35c004a40 con 0x55ce40f271b0 2024-07-26T12:49:37.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.157+0000 7ff36fe49740 1 -- 172.21.15.107:0/2371420966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff35c003d10 con 0x55ce40f271b0 2024-07-26T12:49:37.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.157+0000 7ff343e00640 1 -- 172.21.15.107:0/2371420966 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff35c004c40 con 0x55ce40f271b0 2024-07-26T12:49:37.162 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.157+0000 7ff36fe49740 10 client.15459 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.157+0000 7ff36fe49740 1 -- 172.21.15.107:0/2371420966 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ce4130f3c0 con 0x55ce40f271b0 2024-07-26T12:49:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.158+0000 7ff343e00640 1 -- 172.21.15.107:0/2371420966 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff35c003d10 con 0x55ce40f271b0 2024-07-26T12:49:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.158+0000 7ff343e00640 1 -- 172.21.15.107:0/2371420966 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff35c029040 con 0x55ce40f271b0 2024-07-26T12:49:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.158+0000 7ff36fe49740 10 client.15459 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.158+0000 7ff36fe49740 10 client.15459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.158+0000 7ff36fe49740 20 client.15459 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.158+0000 7ff36fe49740 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55ce40f0e9f0 0x55ce41312780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.158+0000 7ff36fe49740 4 client.15459 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:49:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.158+0000 7ff36fe49740 1 -- 172.21.15.107:0/2371420966 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce4126dae0 con 0x55ce40f0e9f0 2024-07-26T12:49:37.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.159+0000 7ff361400640 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55ce40f0e9f0 0x55ce41312780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.159+0000 7ff361400640 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55ce40f0e9f0 0x55ce41312780 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d96e63f8d714c07f server_cookie=13f5adecfd04e3a2 in_seq=0 out_seq=0 2024-07-26T12:49:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.159+0000 7ff343e00640 10 client.15459 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.160+0000 7ff343e00640 1 -- 172.21.15.107:0/2371420966 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ce4126dae0 con 0x55ce40f0e9f0 2024-07-26T12:49:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.160+0000 7ff343e00640 10 client.15459 handle_command_reply: tid=0 2024-07-26T12:49:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.160+0000 7ff36fe49740 1 client.15459 shutdown 2024-07-26T12:49:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.160+0000 7ff36fe49740 1 -- 172.21.15.107:0/2371420966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 msgr2=0x55ce41300560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.160+0000 7ff36fe49740 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41300560 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff35c004980 tx=0x7ff35c013040 comp rx=0 tx=0).stop 2024-07-26T12:49:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.160+0000 7ff36fe49740 1 -- 172.21.15.107:0/2371420966 shutdown_connections 2024-07-26T12:49:37.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.161+0000 7ff36fe49740 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55ce40f0e9f0 0x55ce41312780 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.161+0000 7ff36fe49740 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f50070 0x55ce41300fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.161+0000 7ff36fe49740 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 0x55ce41300aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.161+0000 7ff36fe49740 1 --2- 172.21.15.107:0/2371420966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41300560 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.161+0000 7ff36fe49740 1 -- 172.21.15.107:0/2371420966 >> 172.21.15.107:0/2371420966 conn(0x55ce40f44460 msgr2=0x55ce412f8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.161+0000 7ff343e00640 0 client.15459 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.161+0000 7ff36fe49740 1 -- 172.21.15.107:0/2371420966 shutdown_connections 2024-07-26T12:49:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.161+0000 7ff36fe49740 1 -- 172.21.15.107:0/2371420966 wait complete. 2024-07-26T12:49:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.161+0000 7ff36fe49740 20 client.15459 trim_cache size 0 max 16384 2024-07-26T12:49:37.180 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.179+0000 7ff36fe49740 1 Processor -- start 2024-07-26T12:49:37.181 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:36 smithi107 ceph-mon[32659]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-07-26T12:49:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.179+0000 7ff36fe49740 1 -- start start 2024-07-26T12:49:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f271b0 0x55ce41302440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 0x55ce412f1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f50070 0x55ce412f6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff36fe49740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff34400ff40 con 0x55ce40f50070 2024-07-26T12:49:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff36fe49740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff344011a80 con 0x55ce40f271b0 2024-07-26T12:49:37.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff36fe49740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff344010610 con 0x55ce40f2f390 2024-07-26T12:49:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff361400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 0x55ce412f1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff361400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 0x55ce412f1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:59422/0 (socket says 172.21.15.107:59422) 2024-07-26T12:49:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff361400640 1 -- 172.21.15.107:0/1122675918 learned_addr learned my addr 172.21.15.107:0/1122675918 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff362800640 1 --2- 172.21.15.107:0/1122675918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f50070 0x55ce412f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.180+0000 7ff362800640 1 --2- 172.21.15.107:0/1122675918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f271b0 0x55ce41302440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.181+0000 7ff361400640 1 -- 172.21.15.107:0/1122675918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f271b0 msgr2=0x55ce41302440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:37.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.181+0000 7ff361400640 1 --2- 172.21.15.107:0/1122675918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f271b0 0x55ce41302440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.181+0000 7ff361400640 1 -- 172.21.15.107:0/1122675918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f50070 msgr2=0x55ce412f6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.181+0000 7ff361400640 1 --2- 172.21.15.107:0/1122675918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f50070 0x55ce412f6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.181+0000 7ff361400640 1 -- 172.21.15.107:0/1122675918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce413032e0 con 0x55ce40f2f390 2024-07-26T12:49:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.181+0000 7ff362800640 1 --2- 172.21.15.107:0/1122675918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f50070 0x55ce412f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:49:37.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.181+0000 7ff361400640 1 --2- 172.21.15.107:0/1122675918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 0x55ce412f1500 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff350004700 tx=0x7ff350005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aef4351b09c6eb74 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:37.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.182+0000 7ff34ea00640 1 -- 172.21.15.107:0/1122675918 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff350022070 con 0x55ce40f2f390 2024-07-26T12:49:37.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.182+0000 7ff34ea00640 1 -- 172.21.15.107:0/1122675918 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff350003de0 con 0x55ce40f2f390 2024-07-26T12:49:37.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.182+0000 7ff34ea00640 1 -- 172.21.15.107:0/1122675918 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3500108f0 con 0x55ce40f2f390 2024-07-26T12:49:37.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.183+0000 7ff36fe49740 1 -- 172.21.15.107:0/1122675918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 msgr2=0x55ce412f1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.183+0000 7ff36fe49740 1 --2- 172.21.15.107:0/1122675918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 0x55ce412f1500 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff350004700 tx=0x7ff350005f90 comp rx=0 tx=0).stop 2024-07-26T12:49:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.183+0000 7ff36fe49740 1 -- 172.21.15.107:0/1122675918 shutdown_connections 2024-07-26T12:49:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.183+0000 7ff36fe49740 1 --2- 172.21.15.107:0/1122675918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f50070 0x55ce412f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.183+0000 7ff36fe49740 1 --2- 172.21.15.107:0/1122675918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f2f390 0x55ce412f1500 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.183+0000 7ff36fe49740 1 --2- 172.21.15.107:0/1122675918 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f271b0 0x55ce41302440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.183+0000 7ff36fe49740 1 -- 172.21.15.107:0/1122675918 >> 172.21.15.107:0/1122675918 conn(0x55ce40f44460 msgr2=0x55ce412f6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:37.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.184+0000 7ff36fe49740 1 -- 172.21.15.107:0/1122675918 shutdown_connections 2024-07-26T12:49:37.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.184+0000 7ff36fe49740 1 -- 172.21.15.107:0/1122675918 wait complete. 2024-07-26T12:49:37.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.185+0000 7ff36fe49740 1 -- start start 2024-07-26T12:49:37.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.185+0000 7ff36fe49740 1 Processor -- start 2024-07-26T12:49:37.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.185+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41306260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.185+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f2f390 0x55ce41306800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.185+0000 7ff36fe49740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 0x55ce41383d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.185+0000 7ff36fe49740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff3440113b0 con 0x55ce40f271b0 2024-07-26T12:49:37.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.185+0000 7ff36fe49740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff344010ce0 con 0x55ce40f2f390 2024-07-26T12:49:37.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.185+0000 7ff36fe49740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff34400df20 con 0x55ce40f50070 2024-07-26T12:49:37.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.186+0000 7ff36fe49740 10 client.0 _ll_register_callbacks cb 0x7ff35c011a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:37.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.186+0000 7ff362800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41306260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.186+0000 7ff362800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41306260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41142/0 (socket says 172.21.15.107:41142) 2024-07-26T12:49:37.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.186+0000 7ff362800640 1 -- 172.21.15.107:0/397158813 learned_addr learned my addr 172.21.15.107:0/397158813 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:37.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.186+0000 7ff362800640 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 0x55ce41383d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.186+0000 7ff361400640 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f2f390 0x55ce41306800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.187+0000 7ff362800640 1 -- 172.21.15.107:0/397158813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 msgr2=0x55ce41383d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.187+0000 7ff362800640 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 0x55ce41383d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.187+0000 7ff362800640 1 -- 172.21.15.107:0/397158813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f2f390 msgr2=0x55ce41306800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.187+0000 7ff362800640 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f2f390 0x55ce41306800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.187+0000 7ff362800640 1 -- 172.21.15.107:0/397158813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce412fc0f0 con 0x55ce40f271b0 2024-07-26T12:49:37.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.188+0000 7ff362800640 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41306260 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009f90 tx=0x7ff35c010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967f6e29b637bc0f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:37.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.188+0000 7ff343e00640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:49:37.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.188+0000 7ff343e00640 1 -- 172.21.15.107:0/397158813 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0049b0 con 0x55ce40f271b0 2024-07-26T12:49:37.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.188+0000 7ff343e00640 1 -- 172.21.15.107:0/397158813 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff35c002030 con 0x55ce40f271b0 2024-07-26T12:49:37.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.188+0000 7ff36fe49740 1 -- 172.21.15.107:0/397158813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff350003de0 con 0x55ce40f271b0 2024-07-26T12:49:37.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.188+0000 7ff343e00640 1 -- 172.21.15.107:0/397158813 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff35c004de0 con 0x55ce40f271b0 2024-07-26T12:49:37.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.188+0000 7ff36fe49740 10 client.15462 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:37.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.188+0000 7ff36fe49740 1 -- 172.21.15.107:0/397158813 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ce41370f00 con 0x55ce40f271b0 2024-07-26T12:49:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.189+0000 7ff343e00640 1 -- 172.21.15.107:0/397158813 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff350003de0 con 0x55ce40f271b0 2024-07-26T12:49:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.189+0000 7ff343e00640 1 -- 172.21.15.107:0/397158813 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff35c035930 con 0x55ce40f271b0 2024-07-26T12:49:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.189+0000 7ff36fe49740 10 client.15462 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.189+0000 7ff36fe49740 10 client.15462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.189+0000 7ff36fe49740 20 client.15462 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.189+0000 7ff36fe49740 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55ce40ee3da0 0x55ce413736c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.189+0000 7ff36fe49740 4 client.15462 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T12:49:37.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.189+0000 7ff36fe49740 1 -- 172.21.15.107:0/397158813 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ce41269580 con 0x55ce40ee3da0 2024-07-26T12:49:37.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.190+0000 7ff361400640 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55ce40ee3da0 0x55ce413736c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:37.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.191+0000 7ff361400640 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55ce40ee3da0 0x55ce413736c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e20d345219a138f server_cookie=eff766eef733d65e in_seq=0 out_seq=0 2024-07-26T12:49:37.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.191+0000 7ff343e00640 10 client.15462 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:37.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.191+0000 7ff343e00640 1 -- 172.21.15.107:0/397158813 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ce41269580 con 0x55ce40ee3da0 2024-07-26T12:49:37.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.191+0000 7ff343e00640 10 client.15462 handle_command_reply: tid=0 2024-07-26T12:49:37.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.191+0000 7ff36fe49740 1 client.15462 shutdown 2024-07-26T12:49:37.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.192+0000 7ff36fe49740 1 -- 172.21.15.107:0/397158813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 msgr2=0x55ce41306260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.192+0000 7ff36fe49740 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41306260 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009f90 tx=0x7ff35c010fd0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.192+0000 7ff36fe49740 1 -- 172.21.15.107:0/397158813 shutdown_connections 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.192+0000 7ff36fe49740 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55ce40ee3da0 0x55ce413736c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.192+0000 7ff343e00640 0 client.15462 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.192+0000 7ff36fe49740 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55ce40f50070 0x55ce41383d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.192+0000 7ff36fe49740 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55ce40f2f390 0x55ce41306800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.192+0000 7ff36fe49740 1 --2- 172.21.15.107:0/397158813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ce40f271b0 0x55ce41306260 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.192+0000 7ff36fe49740 1 -- 172.21.15.107:0/397158813 >> 172.21.15.107:0/397158813 conn(0x55ce40f44460 msgr2=0x55ce41300e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.193+0000 7ff36fe49740 1 -- 172.21.15.107:0/397158813 shutdown_connections 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.193+0000 7ff36fe49740 1 -- 172.21.15.107:0/397158813 wait complete. 2024-07-26T12:49:37.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.193+0000 7ff36fe49740 20 client.15462 trim_cache size 0 max 16384 2024-07-26T12:49:37.196 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:49:37.196 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-07-26T12:49:37.196 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-07-26T12:49:37.197 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:49:37.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.194+0000 7ff363200640 1 -- 172.21.15.107:0/269536189 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff3480b3d40 msgr2=0x7ff3480b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:37.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.194+0000 7ff363200640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff3480b3d40 0x7ff3480b6250 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55ce412f3c50 tx=0x7ff354107dd0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.194+0000 7ff363200640 1 -- 172.21.15.107:0/269536189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff364075800 msgr2=0x7ff3641a0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.194+0000 7ff363200640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff364075800 0x7ff3641a0320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff354005940 tx=0x7ff3540040c0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.194+0000 7ff363200640 1 -- 172.21.15.107:0/269536189 shutdown_connections 2024-07-26T12:49:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.194+0000 7ff363200640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff3480b3d40 0x7ff3480b6250 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.194+0000 7ff363200640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff364113cb0 0x7ff3641a5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.194+0000 7ff363200640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff364076db0 0x7ff3641a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.194+0000 7ff363200640 1 --2- 172.21.15.107:0/269536189 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff364075800 0x7ff3641a0320 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff354005940 tx=0x7ff3540040c0 comp rx=0 tx=0).stop 2024-07-26T12:49:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.194+0000 7ff363200640 1 -- 172.21.15.107:0/269536189 >> 172.21.15.107:0/269536189 conn(0x7ff3640fee70 msgr2=0x7ff36410fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.195+0000 7ff363200640 1 -- 172.21.15.107:0/269536189 shutdown_connections 2024-07-26T12:49:37.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:37.195+0000 7ff363200640 1 -- 172.21.15.107:0/269536189 wait complete. 2024-07-26T12:49:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:36 smithi112 ceph-mon[37501]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-07-26T12:49:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:36 smithi132 ceph-mon[37580]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-07-26T12:49:38.086 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T12:49:38.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:49:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.528+0000 7f2451a00640 1 -- 172.21.15.107:0/1306433574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f244c113840 msgr2=0x7f244c113c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.528+0000 7f2451a00640 1 --2- 172.21.15.107:0/1306433574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f244c113840 0x7f244c113c50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2438009860 tx=0x7f243802a360 comp rx=0 tx=0).stop 2024-07-26T12:49:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.529+0000 7f2451a00640 1 -- 172.21.15.107:0/1306433574 shutdown_connections 2024-07-26T12:49:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.529+0000 7f2451a00640 1 --2- 172.21.15.107:0/1306433574 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f244c114190 0x7f244c11a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.531 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.529+0000 7f2451a00640 1 --2- 172.21.15.107:0/1306433574 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f244c113840 0x7f244c113c50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.531 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.529+0000 7f2451a00640 1 --2- 172.21.15.107:0/1306433574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f244c10df10 0x7f244c10e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.531 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.529+0000 7f2451a00640 1 -- 172.21.15.107:0/1306433574 >> 172.21.15.107:0/1306433574 conn(0x7f244c109df0 msgr2=0x7f244c10c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:38.531 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.529+0000 7f2451a00640 1 -- 172.21.15.107:0/1306433574 shutdown_connections 2024-07-26T12:49:38.531 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.529+0000 7f2451a00640 1 -- 172.21.15.107:0/1306433574 wait complete. 2024-07-26T12:49:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.530+0000 7f2451a00640 1 Processor -- start 2024-07-26T12:49:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.530+0000 7f2451a00640 1 -- start start 2024-07-26T12:49:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.530+0000 7f2451a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f244c10df10 0x7f244c1199b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.530+0000 7f2451a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f244c113840 0x7f244c119f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.530+0000 7f2451a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f244c114190 0x7f244c1ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.530+0000 7f2451a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f244c06f870 con 0x7f244c113840 2024-07-26T12:49:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.530+0000 7f2451a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f244c06f6f0 con 0x7f244c10df10 2024-07-26T12:49:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.530+0000 7f2451a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f244c06f9f0 con 0x7f244c114190 2024-07-26T12:49:38.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.531+0000 7f244aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f244c10df10 0x7f244c1199b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.531+0000 7f244aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f244c10df10 0x7f244c1199b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:41470/0 (socket says 172.21.15.107:41470) 2024-07-26T12:49:38.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.531+0000 7f244aa00640 1 -- 172.21.15.107:0/1783811959 learned_addr learned my addr 172.21.15.107:0/1783811959 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:38.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.531+0000 7f244a000640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f244c113840 0x7f244c119f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.531+0000 7f244b400640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f244c114190 0x7f244c1ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.532+0000 7f244aa00640 1 -- 172.21.15.107:0/1783811959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f244c114190 msgr2=0x7f244c1ae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:38.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.532+0000 7f244aa00640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f244c114190 0x7f244c1ae4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.532+0000 7f244aa00640 1 -- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f244c113840 msgr2=0x7f244c119f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:38.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.532+0000 7f244aa00640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f244c113840 0x7f244c119f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.532+0000 7f244aa00640 1 -- 172.21.15.107:0/1783811959 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c1aebb0 con 0x7f244c10df10 2024-07-26T12:49:38.535 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.532+0000 7f244a000640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f244c113840 0x7f244c119f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:38.535 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.532+0000 7f244aa00640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f244c10df10 0x7f244c1199b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f243c00e9c0 tx=0x7f243c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:38.535 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.533+0000 7f2443400640 1 -- 172.21.15.107:0/1783811959 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c00ccc0 con 0x7f244c10df10 2024-07-26T12:49:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.533+0000 7f2451a00640 1 -- 172.21.15.107:0/1783811959 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244c1aeea0 con 0x7f244c10df10 2024-07-26T12:49:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.533+0000 7f2443400640 1 -- 172.21.15.107:0/1783811959 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f243c0054d0 con 0x7f244c10df10 2024-07-26T12:49:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.533+0000 7f2443400640 1 -- 172.21.15.107:0/1783811959 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c010430 con 0x7f244c10df10 2024-07-26T12:49:38.536 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.533+0000 7f2451a00640 1 -- 172.21.15.107:0/1783811959 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244c1b5630 con 0x7f244c10df10 2024-07-26T12:49:38.536 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:49:38.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.538+0000 7f2458a29740 1 Processor -- start 2024-07-26T12:49:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.538+0000 7f2458a29740 1 -- start start 2024-07-26T12:49:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.538+0000 7f2458a29740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fdf1b0 0x5643e3382a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.539+0000 7f2443400640 1 -- 172.21.15.107:0/1783811959 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f243c004f80 con 0x7f244c10df10 2024-07-26T12:49:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.539+0000 7f2458a29740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fb62f0 0x5643e3382f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.539+0000 7f2458a29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33834c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.539+0000 7f2458a29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5643e32f7270 con 0x5643e2fbe4d0 2024-07-26T12:49:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.539+0000 7f2458a29740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5643e32e6770 con 0x5643e2fdf1b0 2024-07-26T12:49:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.539+0000 7f2458a29740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5643e32ea7b0 con 0x5643e2fb62f0 2024-07-26T12:49:38.540 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.539+0000 7f244a000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33834c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.540+0000 7f244a000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33834c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:41158/0 (socket says 172.21.15.107:41158) 2024-07-26T12:49:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.540+0000 7f244a000640 1 -- 172.21.15.107:0/3020870244 learned_addr learned my addr 172.21.15.107:0/3020870244 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:38.541 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.540+0000 7f244a000640 1 -- 172.21.15.107:0/3020870244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fb62f0 msgr2=0x5643e3382f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T12:49:38.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.541+0000 7f244a000640 1 --2- 172.21.15.107:0/3020870244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fb62f0 0x5643e3382f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.541+0000 7f244a000640 1 -- 172.21.15.107:0/3020870244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fdf1b0 msgr2=0x5643e3382a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:38.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.541+0000 7f244a000640 1 --2- 172.21.15.107:0/3020870244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fdf1b0 0x5643e3382a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.541+0000 7f244a000640 1 -- 172.21.15.107:0/3020870244 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643e304a790 con 0x5643e2fbe4d0 2024-07-26T12:49:38.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.541+0000 7f2443400640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24300b3dc0 0x7f24300b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.541+0000 7f2443400640 1 -- 172.21.15.107:0/1783811959 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f243c0e47f0 con 0x7f244c10df10 2024-07-26T12:49:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.542+0000 7f244a000640 1 --2- 172.21.15.107:0/3020870244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33834c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f243802a950 tx=0x7f24380303c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b10bb9c62f56d6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.542+0000 7f2437e00640 1 -- 172.21.15.107:0/3020870244 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438042020 con 0x5643e2fbe4d0 2024-07-26T12:49:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.543+0000 7f244b400640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24300b3dc0 0x7f24300b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.543+0000 7f2437e00640 1 -- 172.21.15.107:0/3020870244 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2438038040 con 0x5643e2fbe4d0 2024-07-26T12:49:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.543+0000 7f2437e00640 1 -- 172.21.15.107:0/3020870244 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438037040 con 0x5643e2fbe4d0 2024-07-26T12:49:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.544+0000 7f2458a29740 1 -- 172.21.15.107:0/3020870244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 msgr2=0x5643e33834c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.544+0000 7f244b400640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24300b3dc0 0x7f24300b62d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2444005fd0 tx=0x7f2444009b10 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.544+0000 7f2458a29740 1 --2- 172.21.15.107:0/3020870244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33834c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f243802a950 tx=0x7f24380303c0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.544+0000 7f2458a29740 1 -- 172.21.15.107:0/3020870244 shutdown_connections 2024-07-26T12:49:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.544+0000 7f2458a29740 1 --2- 172.21.15.107:0/3020870244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33834c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.544+0000 7f2458a29740 1 --2- 172.21.15.107:0/3020870244 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fb62f0 0x5643e3382f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.544+0000 7f2458a29740 1 --2- 172.21.15.107:0/3020870244 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fdf1b0 0x5643e3382a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.544+0000 7f2458a29740 1 -- 172.21.15.107:0/3020870244 >> 172.21.15.107:0/3020870244 conn(0x5643e2fd3300 msgr2=0x5643e3380930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:38.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.544+0000 7f2458a29740 1 -- 172.21.15.107:0/3020870244 shutdown_connections 2024-07-26T12:49:38.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.544+0000 7f2458a29740 1 -- 172.21.15.107:0/3020870244 wait complete. 2024-07-26T12:49:38.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.545+0000 7f2458a29740 1 -- start start 2024-07-26T12:49:38.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.546+0000 7f2458a29740 1 Processor -- start 2024-07-26T12:49:38.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.546+0000 7f2458a29740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e338e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.546+0000 7f2458a29740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fbe4d0 0x5643e338ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.546+0000 7f2458a29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fdf1b0 0x5643e339a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.546+0000 7f2458a29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5643e3220ab0 con 0x5643e2fdf1b0 2024-07-26T12:49:38.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.546+0000 7f2458a29740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5643e3283d50 con 0x5643e2fb62f0 2024-07-26T12:49:38.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.546+0000 7f2458a29740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5643e33911b0 con 0x5643e2fbe4d0 2024-07-26T12:49:38.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.546+0000 7f2458a29740 10 client.0 _ll_register_callbacks cb 0x5643e2ff6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:38.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.547+0000 7f244aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e338e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.547+0000 7f244aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e338e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:41496/0 (socket says 172.21.15.107:41496) 2024-07-26T12:49:38.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.547+0000 7f244aa00640 1 -- 172.21.15.107:0/1536202644 learned_addr learned my addr 172.21.15.107:0/1536202644 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.547+0000 7f244a000640 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fbe4d0 0x5643e338ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.547+0000 7f244aa00640 1 -- 172.21.15.107:0/1536202644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fbe4d0 msgr2=0x5643e338ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.547+0000 7f244aa00640 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fbe4d0 0x5643e338ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.547+0000 7f244aa00640 1 -- 172.21.15.107:0/1536202644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fdf1b0 msgr2=0x5643e339a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.547+0000 7f244aa00640 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fdf1b0 0x5643e339a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.547+0000 7f244aa00640 1 -- 172.21.15.107:0/1536202644 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643e30c3620 con 0x5643e2fb62f0 2024-07-26T12:49:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.547+0000 7f244aa00640 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e338e6d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f243c0aad20 tx=0x7f243c0aa790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=930e0fcb342815b6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.548+0000 7f2434c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:49:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.548+0000 7f2434c00640 1 -- 172.21.15.107:0/1536202644 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c0af3e0 con 0x5643e2fb62f0 2024-07-26T12:49:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.548+0000 7f2434c00640 1 -- 172.21.15.107:0/1536202644 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f243c0a72b0 con 0x5643e2fb62f0 2024-07-26T12:49:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.548+0000 7f2458a29740 1 -- 172.21.15.107:0/1536202644 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2438038040 con 0x5643e2fb62f0 2024-07-26T12:49:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.548+0000 7f2434c00640 1 -- 172.21.15.107:0/1536202644 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f243c0ab020 con 0x5643e2fb62f0 2024-07-26T12:49:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.548+0000 7f2458a29740 10 client.25604 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.548+0000 7f2458a29740 1 -- 172.21.15.107:0/1536202644 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5643e339d280 con 0x5643e2fb62f0 2024-07-26T12:49:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.548+0000 7f2434c00640 1 -- 172.21.15.107:0/1536202644 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2438038040 con 0x5643e2fb62f0 2024-07-26T12:49:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.550+0000 7f2434c00640 1 -- 172.21.15.107:0/1536202644 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f243c0e43f0 con 0x5643e2fb62f0 2024-07-26T12:49:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.550+0000 7f2458a29740 10 client.25604 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.550+0000 7f2458a29740 10 client.25604 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.550+0000 7f2458a29740 20 client.25604 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.550+0000 7f2458a29740 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5643e2f9dd80 0x5643e33a0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.550+0000 7f2458a29740 4 client.25604 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:49:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.550+0000 7f2458a29740 1 -- 172.21.15.107:0/1536202644 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643e32f91c0 con 0x5643e2f9dd80 2024-07-26T12:49:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.551+0000 7f244a000640 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5643e2f9dd80 0x5643e33a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.551+0000 7f244a000640 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5643e2f9dd80 0x5643e33a0870 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4aa881074ebd3433 server_cookie=dffbf032aac0fdfc in_seq=0 out_seq=0 2024-07-26T12:49:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.551+0000 7f2434c00640 10 client.25604 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.552+0000 7f2434c00640 1 -- 172.21.15.107:0/1536202644 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5643e32f91c0 con 0x5643e2f9dd80 2024-07-26T12:49:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.552+0000 7f2434c00640 10 client.25604 handle_command_reply: tid=0 2024-07-26T12:49:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.552+0000 7f2458a29740 1 client.25604 shutdown 2024-07-26T12:49:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.552+0000 7f2458a29740 1 -- 172.21.15.107:0/1536202644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 msgr2=0x5643e338e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:38.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.552+0000 7f2458a29740 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e338e6d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f243c0aad20 tx=0x7f243c0aa790 comp rx=0 tx=0).stop 2024-07-26T12:49:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.553+0000 7f2458a29740 1 -- 172.21.15.107:0/1536202644 shutdown_connections 2024-07-26T12:49:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.553+0000 7f2458a29740 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5643e2f9dd80 0x5643e33a0870 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.553+0000 7f2458a29740 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fdf1b0 0x5643e339a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.553+0000 7f2458a29740 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fbe4d0 0x5643e338ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.555 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.553+0000 7f2458a29740 1 --2- 172.21.15.107:0/1536202644 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e338e6d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.553+0000 7f2458a29740 1 -- 172.21.15.107:0/1536202644 >> 172.21.15.107:0/1536202644 conn(0x5643e2fd3300 msgr2=0x5643e337f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.553+0000 7f2434c00640 0 client.25604 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.553+0000 7f2458a29740 1 -- 172.21.15.107:0/1536202644 shutdown_connections 2024-07-26T12:49:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.553+0000 7f2458a29740 1 -- 172.21.15.107:0/1536202644 wait complete. 2024-07-26T12:49:38.556 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.553+0000 7f2458a29740 20 client.25604 trim_cache size 0 max 16384 2024-07-26T12:49:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.584+0000 7f2458a29740 1 Processor -- start 2024-07-26T12:49:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.584+0000 7f2458a29740 1 -- start start 2024-07-26T12:49:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.585+0000 7f2458a29740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e339a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.585+0000 7f2458a29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33806b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.585+0000 7f2458a29740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fdf1b0 0x5643e33a6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.585+0000 7f2458a29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1bf600 con 0x5643e2fbe4d0 2024-07-26T12:49:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.585+0000 7f2458a29740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1c1140 con 0x5643e2fb62f0 2024-07-26T12:49:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.585+0000 7f2458a29740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1bfcd0 con 0x5643e2fdf1b0 2024-07-26T12:49:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.585+0000 7f244aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e339a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.585+0000 7f244a000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33806b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.585+0000 7f244aa00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e339a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:41506/0 (socket says 172.21.15.107:41506) 2024-07-26T12:49:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.585+0000 7f244aa00640 1 -- 172.21.15.107:0/2620850698 learned_addr learned my addr 172.21.15.107:0/2620850698 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.586+0000 7f244b400640 1 --2- 172.21.15.107:0/2620850698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fdf1b0 0x5643e33a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.586+0000 7f244aa00640 1 -- 172.21.15.107:0/2620850698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fdf1b0 msgr2=0x5643e33a6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.586+0000 7f244aa00640 1 --2- 172.21.15.107:0/2620850698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fdf1b0 0x5643e33a6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.586+0000 7f244aa00640 1 -- 172.21.15.107:0/2620850698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 msgr2=0x5643e33806b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.586+0000 7f244aa00640 1 --2- 172.21.15.107:0/2620850698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33806b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.586+0000 7f244aa00640 1 -- 172.21.15.107:0/2620850698 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643e33816a0 con 0x5643e2fb62f0 2024-07-26T12:49:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.587+0000 7f244a000640 1 --2- 172.21.15.107:0/2620850698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33806b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.587+0000 7f244aa00640 1 --2- 172.21.15.107:0/2620850698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e339a290 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f243c0a70a0 tx=0x7f243c004a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f7f932e23321b60 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.587+0000 7f2437400640 1 -- 172.21.15.107:0/2620850698 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c0c9070 con 0x5643e2fb62f0 2024-07-26T12:49:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.587+0000 7f2437400640 1 -- 172.21.15.107:0/2620850698 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f243c0aab30 con 0x5643e2fb62f0 2024-07-26T12:49:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.587+0000 7f2437400640 1 -- 172.21.15.107:0/2620850698 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c0c5b60 con 0x5643e2fb62f0 2024-07-26T12:49:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.588+0000 7f2458a29740 1 -- 172.21.15.107:0/2620850698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 msgr2=0x5643e339a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.588+0000 7f2458a29740 1 --2- 172.21.15.107:0/2620850698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e339a290 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f243c0a70a0 tx=0x7f243c004a20 comp rx=0 tx=0).stop 2024-07-26T12:49:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.589+0000 7f2458a29740 1 -- 172.21.15.107:0/2620850698 shutdown_connections 2024-07-26T12:49:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.589+0000 7f2458a29740 1 --2- 172.21.15.107:0/2620850698 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fdf1b0 0x5643e33a6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.589+0000 7f2458a29740 1 --2- 172.21.15.107:0/2620850698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fbe4d0 0x5643e33806b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.589+0000 7f2458a29740 1 --2- 172.21.15.107:0/2620850698 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fb62f0 0x5643e339a290 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.589+0000 7f2458a29740 1 -- 172.21.15.107:0/2620850698 >> 172.21.15.107:0/2620850698 conn(0x5643e2fd3300 msgr2=0x5643e337ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:38.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.589+0000 7f2458a29740 1 -- 172.21.15.107:0/2620850698 shutdown_connections 2024-07-26T12:49:38.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.589+0000 7f2458a29740 1 -- 172.21.15.107:0/2620850698 wait complete. 2024-07-26T12:49:38.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.589+0000 7f2458a29740 1 -- start start 2024-07-26T12:49:38.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.590+0000 7f2458a29740 1 Processor -- start 2024-07-26T12:49:38.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.590+0000 7f2458a29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fb62f0 0x5643e33968c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.590+0000 7f2458a29740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fbe4d0 0x5643e3391d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.590+0000 7f2458a29740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fdf1b0 0x5643e33922a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.590+0000 7f2458a29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1c0a70 con 0x5643e2fb62f0 2024-07-26T12:49:38.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.590+0000 7f2458a29740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1c03a0 con 0x5643e2fbe4d0 2024-07-26T12:49:38.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.590+0000 7f2458a29740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f244c1bd5e0 con 0x5643e2fdf1b0 2024-07-26T12:49:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.590+0000 7f2458a29740 10 client.0 _ll_register_callbacks cb 0x7f243c0aadf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.591+0000 7f244a000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fbe4d0 0x5643e3391d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.591+0000 7f244b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fdf1b0 0x5643e33922a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.591+0000 7f244a000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fbe4d0 0x5643e3391d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:41512/0 (socket says 172.21.15.107:41512) 2024-07-26T12:49:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.591+0000 7f244a000640 1 -- 172.21.15.107:0/3705241032 learned_addr learned my addr 172.21.15.107:0/3705241032 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.591+0000 7f244aa00640 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fb62f0 0x5643e33968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.591+0000 7f244a000640 1 -- 172.21.15.107:0/3705241032 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fdf1b0 msgr2=0x5643e33922a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.591+0000 7f244a000640 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fdf1b0 0x5643e33922a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.591+0000 7f244a000640 1 -- 172.21.15.107:0/3705241032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fb62f0 msgr2=0x5643e33968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.591+0000 7f244a000640 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fb62f0 0x5643e33968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.591+0000 7f244a000640 1 -- 172.21.15.107:0/3705241032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643e3392810 con 0x5643e2fbe4d0 2024-07-26T12:49:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.592+0000 7f244a000640 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fbe4d0 0x5643e3391d60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2438005eb0 tx=0x7f243802a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae0c7a7e324ca224 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:38.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.592+0000 7f2434c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:49:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.592+0000 7f2434c00640 1 -- 172.21.15.107:0/3705241032 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438037040 con 0x5643e2fbe4d0 2024-07-26T12:49:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.592+0000 7f2434c00640 1 -- 172.21.15.107:0/3705241032 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f24380304f0 con 0x5643e2fbe4d0 2024-07-26T12:49:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.592+0000 7f2458a29740 1 -- 172.21.15.107:0/3705241032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f243c0aab30 con 0x5643e2fbe4d0 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2434c00640 1 -- 172.21.15.107:0/3705241032 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2438063050 con 0x5643e2fbe4d0 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2458a29740 10 client.25610 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2458a29740 1 -- 172.21.15.107:0/3705241032 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5643e3408d30 con 0x5643e2fbe4d0 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2434c00640 1 -- 172.21.15.107:0/3705241032 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f243c0aab30 con 0x5643e2fbe4d0 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2434c00640 1 -- 172.21.15.107:0/3705241032 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f2438051330 con 0x5643e2fbe4d0 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2458a29740 10 client.25610 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2458a29740 10 client.25610 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2458a29740 20 client.25610 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2458a29740 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5643e2f731c0 0x5643e340b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2458a29740 4 client.25610 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T12:49:38.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.593+0000 7f2458a29740 1 -- 172.21.15.107:0/3705241032 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5643e32f4c60 con 0x5643e2f731c0 2024-07-26T12:49:38.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.594+0000 7f244aa00640 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5643e2f731c0 0x5643e340b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:38.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.594+0000 7f244aa00640 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5643e2f731c0 0x5643e340b080 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46598ab527de0614 server_cookie=833a9c46eceb9207 in_seq=0 out_seq=0 2024-07-26T12:49:38.596 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.594+0000 7f2434c00640 10 client.25610 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.605+0000 7f2434c00640 1 -- 172.21.15.107:0/3705241032 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5643e32f4c60 con 0x5643e2f731c0 2024-07-26T12:49:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.605+0000 7f2434c00640 10 client.25610 handle_command_reply: tid=0 2024-07-26T12:49:38.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.605+0000 7f2458a29740 1 client.25610 shutdown 2024-07-26T12:49:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.606+0000 7f2458a29740 1 -- 172.21.15.107:0/3705241032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fbe4d0 msgr2=0x5643e3391d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.606+0000 7f2458a29740 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fbe4d0 0x5643e3391d60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2438005eb0 tx=0x7f243802a360 comp rx=0 tx=0).stop 2024-07-26T12:49:38.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.607+0000 7f2458a29740 1 -- 172.21.15.107:0/3705241032 shutdown_connections 2024-07-26T12:49:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.607+0000 7f2458a29740 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5643e2f731c0 0x5643e340b080 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.607+0000 7f2458a29740 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5643e2fdf1b0 0x5643e33922a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.607+0000 7f2458a29740 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5643e2fbe4d0 0x5643e3391d60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.607+0000 7f2458a29740 1 --2- 172.21.15.107:0/3705241032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5643e2fb62f0 0x5643e33968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.607+0000 7f2458a29740 1 -- 172.21.15.107:0/3705241032 >> 172.21.15.107:0/3705241032 conn(0x5643e2fd3300 msgr2=0x5643e3383a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.607+0000 7f2434c00640 0 client.25610 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.607+0000 7f2458a29740 1 -- 172.21.15.107:0/3705241032 shutdown_connections 2024-07-26T12:49:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.607+0000 7f2458a29740 1 -- 172.21.15.107:0/3705241032 wait complete. 2024-07-26T12:49:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.607+0000 7f2458a29740 20 client.25610 trim_cache size 0 max 16384 2024-07-26T12:49:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-07-26T12:49:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-07-26T12:49:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-07-26T12:49:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-07-26T12:49:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.608+0000 7f2451a00640 1 -- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24300b3dc0 msgr2=0x7f24300b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.608+0000 7f2451a00640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24300b3dc0 0x7f24300b62d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2444005fd0 tx=0x7f2444009b10 comp rx=0 tx=0).stop 2024-07-26T12:49:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.608+0000 7f2451a00640 1 -- 172.21.15.107:0/1783811959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f244c10df10 msgr2=0x7f244c1199b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:38.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.608+0000 7f2451a00640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f244c10df10 0x7f244c1199b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f243c00e9c0 tx=0x7f243c00ee80 comp rx=0 tx=0).stop 2024-07-26T12:49:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.608+0000 7f2451a00640 1 -- 172.21.15.107:0/1783811959 shutdown_connections 2024-07-26T12:49:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.608+0000 7f2451a00640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f24300b3dc0 0x7f24300b62d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.608+0000 7f2451a00640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f244c114190 0x7f244c1ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.608+0000 7f2451a00640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f244c113840 0x7f244c119f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.608+0000 7f2451a00640 1 --2- 172.21.15.107:0/1783811959 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f244c10df10 0x7f244c1199b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.608+0000 7f2451a00640 1 -- 172.21.15.107:0/1783811959 >> 172.21.15.107:0/1783811959 conn(0x7f244c109df0 msgr2=0x7f244c10d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:38.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.610+0000 7f2451a00640 1 -- 172.21.15.107:0/1783811959 shutdown_connections 2024-07-26T12:49:38.611 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:38.610+0000 7f2451a00640 1 -- 172.21.15.107:0/1783811959 wait complete. 2024-07-26T12:49:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:38 smithi107 ceph-mon[32659]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-07-26T12:49:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:38 smithi112 ceph-mon[37501]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-07-26T12:49:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:38 smithi132 ceph-mon[37580]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-07-26T12:49:40.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.982+0000 7f7e67200640 1 -- 172.21.15.107:0/1014557318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e68108d20 msgr2=0x7f7e68109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:40.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.982+0000 7f7e67200640 1 --2- 172.21.15.107:0/1014557318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e68108d20 0x7f7e68109130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7e54009860 tx=0x7f7e5402a340 comp rx=0 tx=0).stop 2024-07-26T12:49:40.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.982+0000 7f7e67200640 1 -- 172.21.15.107:0/1014557318 shutdown_connections 2024-07-26T12:49:40.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.982+0000 7f7e67200640 1 --2- 172.21.15.107:0/1014557318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e68109670 0x7f7e6810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:40.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.982+0000 7f7e67200640 1 --2- 172.21.15.107:0/1014557318 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e68108d20 0x7f7e68109130 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:40.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.982+0000 7f7e67200640 1 --2- 172.21.15.107:0/1014557318 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e681033e0 0x7f7e681037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:40.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.982+0000 7f7e67200640 1 -- 172.21.15.107:0/1014557318 >> 172.21.15.107:0/1014557318 conn(0x7f7e680fee70 msgr2=0x7f7e681012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:40.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.983+0000 7f7e67200640 1 -- 172.21.15.107:0/1014557318 shutdown_connections 2024-07-26T12:49:40.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.983+0000 7f7e67200640 1 -- 172.21.15.107:0/1014557318 wait complete. 2024-07-26T12:49:40.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.984+0000 7f7e67200640 1 Processor -- start 2024-07-26T12:49:40.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.985+0000 7f7e67200640 1 -- start start 2024-07-26T12:49:40.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.985+0000 7f7e67200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e681033e0 0x7f7e6819bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:40.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.985+0000 7f7e67200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e68108d20 0x7f7e6819c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:40.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.985+0000 7f7e67200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e68109670 0x7f7e681a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:40.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.985+0000 7f7e67200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7e681132e0 con 0x7f7e68109670 2024-07-26T12:49:40.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.985+0000 7f7e67200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7e68113160 con 0x7f7e681033e0 2024-07-26T12:49:40.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.985+0000 7f7e67200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7e68113460 con 0x7f7e68108d20 2024-07-26T12:49:40.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.986+0000 7f7e65400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e68108d20 0x7f7e6819c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:40.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.986+0000 7f7e65400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e68108d20 0x7f7e6819c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:59470/0 (socket says 172.21.15.107:59470) 2024-07-26T12:49:40.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.986+0000 7f7e65400640 1 -- 172.21.15.107:0/3428813192 learned_addr learned my addr 172.21.15.107:0/3428813192 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:40.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.986+0000 7f7e66800640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e68109670 0x7f7e681a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:40.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.986+0000 7f7e65e00640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e681033e0 0x7f7e6819bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:40.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.986+0000 7f7e65400640 1 -- 172.21.15.107:0/3428813192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e681033e0 msgr2=0x7f7e6819bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:40.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.986+0000 7f7e65400640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e681033e0 0x7f7e6819bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:40.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.986+0000 7f7e65400640 1 -- 172.21.15.107:0/3428813192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e68109670 msgr2=0x7f7e681a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:40.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.986+0000 7f7e65400640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e68109670 0x7f7e681a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:40.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.987+0000 7f7e65400640 1 -- 172.21.15.107:0/3428813192 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e681a1f00 con 0x7f7e68108d20 2024-07-26T12:49:40.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.987+0000 7f7e65e00640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e681033e0 0x7f7e6819bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:49:40.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.987+0000 7f7e65400640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e68108d20 0x7f7e6819c530 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7e54035040 tx=0x7f7e54032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:40.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.987+0000 7f7e5ea00640 1 -- 172.21.15.107:0/3428813192 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e54037040 con 0x7f7e68108d20 2024-07-26T12:49:40.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.987+0000 7f7e5ea00640 1 -- 172.21.15.107:0/3428813192 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e54004050 con 0x7f7e68108d20 2024-07-26T12:49:40.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.987+0000 7f7e5ea00640 1 -- 172.21.15.107:0/3428813192 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e540035f0 con 0x7f7e68108d20 2024-07-26T12:49:40.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.988+0000 7f7e67200640 1 -- 172.21.15.107:0/3428813192 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e681a2190 con 0x7f7e68108d20 2024-07-26T12:49:40.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.988+0000 7f7e67200640 1 -- 172.21.15.107:0/3428813192 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e681a25c0 con 0x7f7e68108d20 2024-07-26T12:49:40.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.989+0000 7f7e67200640 1 -- 172.21.15.107:0/3428813192 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e681037f0 con 0x7f7e68108d20 2024-07-26T12:49:40.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.992+0000 7f7e5ea00640 1 -- 172.21.15.107:0/3428813192 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f7e54043020 con 0x7f7e68108d20 2024-07-26T12:49:40.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.993+0000 7f7e5ea00640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7e4c0b3e40 0x7f7e4c0b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:40.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.993+0000 7f7e5ea00640 1 -- 172.21.15.107:0/3428813192 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7e54103550 con 0x7f7e68108d20 2024-07-26T12:49:40.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.994+0000 7f7e65e00640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7e4c0b3e40 0x7f7e4c0b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:40.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.994+0000 7f7e65e00640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7e4c0b3e40 0x7f7e4c0b6350 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7e58005fd0 tx=0x7f7e58009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:40.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:40.997+0000 7f7e5ea00640 1 -- 172.21.15.107:0/3428813192 <== mon.1 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7e5404a060 con 0x7f7e68108d20 2024-07-26T12:49:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:40 smithi112 ceph-mon[37501]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-07-26T12:49:41.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.229+0000 7f7e67200640 1 -- 172.21.15.107:0/3428813192 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e6819d0c0 con 0x7f7e68108d20 2024-07-26T12:49:41.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:40 smithi107 ceph-mon[32659]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-07-26T12:49:41.231 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:49:41.231 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":29,"btime":"2024-07-26T12:48:38:656386+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:48:38.606034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24716,"mds_1":24715,"mds_2":24703,"mds_3":24694,"mds_4":24682},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24632":{"gid":24632,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24634":{"gid":24634,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24652":{"gid":24652,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24662":{"gid":24662,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24680":{"gid":24680,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/3739854241","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":3739854241},{"type":"v1","addr":"172.21.15.112:6837","nonce":3739854241}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":32,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24694":{"gid":24694,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":31,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":29,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":28,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24716,"qdb_cluster":[24715,24716,24703,24694,24682]},"id":1}]} 2024-07-26T12:49:41.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.230+0000 7f7e5ea00640 1 -- 172.21.15.107:0/3428813192 <== mon.1 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11993 (secure 0 0 0) 0x7f7e540ca590 con 0x7f7e68108d20 2024-07-26T12:49:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 29 2024-07-26T12:49:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.233+0000 7f7e67200640 1 -- 172.21.15.107:0/3428813192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7e4c0b3e40 msgr2=0x7f7e4c0b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.233+0000 7f7e67200640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7e4c0b3e40 0x7f7e4c0b6350 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7e58005fd0 tx=0x7f7e58009260 comp rx=0 tx=0).stop 2024-07-26T12:49:41.234 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.233+0000 7f7e67200640 1 -- 172.21.15.107:0/3428813192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e68108d20 msgr2=0x7f7e6819c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:41.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.233+0000 7f7e67200640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e68108d20 0x7f7e6819c530 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7e54035040 tx=0x7f7e54032d50 comp rx=0 tx=0).stop 2024-07-26T12:49:41.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.233+0000 7f7e67200640 1 -- 172.21.15.107:0/3428813192 shutdown_connections 2024-07-26T12:49:41.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.233+0000 7f7e67200640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7e4c0b3e40 0x7f7e4c0b6350 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:41.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.233+0000 7f7e67200640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e68109670 0x7f7e681a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:41.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.233+0000 7f7e67200640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e68108d20 0x7f7e6819c530 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:41.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.233+0000 7f7e67200640 1 --2- 172.21.15.107:0/3428813192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7e681033e0 0x7f7e6819bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:41.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.233+0000 7f7e67200640 1 -- 172.21.15.107:0/3428813192 >> 172.21.15.107:0/3428813192 conn(0x7f7e680fee70 msgr2=0x7f7e68107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:41.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.234+0000 7f7e67200640 1 -- 172.21.15.107:0/3428813192 shutdown_connections 2024-07-26T12:49:41.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:41.234+0000 7f7e67200640 1 -- 172.21.15.107:0/3428813192 wait complete. 2024-07-26T12:49:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:40 smithi132 ceph-mon[37580]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-07-26T12:49:42.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T12:49:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:41 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/3428813192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:49:42.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:41 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/3428813192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:49:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:41 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/3428813192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:49:42.812 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:49:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:42 smithi112 ceph-mon[37501]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-07-26T12:49:43.209 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:42 smithi107 ceph-mon[32659]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-07-26T12:49:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:42 smithi132 ceph-mon[37580]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-07-26T12:49:45.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.082+0000 7fc5a6e00640 1 -- 172.21.15.107:0/129595593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5a0108d20 msgr2=0x7fc5a0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:45.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.082+0000 7fc5a6e00640 1 --2- 172.21.15.107:0/129595593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5a0108d20 0x7fc5a0109130 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009900 tx=0x7fc58c02a410 comp rx=0 tx=0).stop 2024-07-26T12:49:45.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.082+0000 7fc5a6e00640 1 -- 172.21.15.107:0/129595593 shutdown_connections 2024-07-26T12:49:45.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.082+0000 7fc5a6e00640 1 --2- 172.21.15.107:0/129595593 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5a0109670 0x7fc5a010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.082+0000 7fc5a6e00640 1 --2- 172.21.15.107:0/129595593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5a0108d20 0x7fc5a0109130 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.082+0000 7fc5a6e00640 1 --2- 172.21.15.107:0/129595593 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5a01033e0 0x7fc5a01037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.082+0000 7fc5a6e00640 1 -- 172.21.15.107:0/129595593 >> 172.21.15.107:0/129595593 conn(0x7fc5a00fee70 msgr2=0x7fc5a01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.082+0000 7fc5a6e00640 1 -- 172.21.15.107:0/129595593 shutdown_connections 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.082+0000 7fc5a6e00640 1 -- 172.21.15.107:0/129595593 wait complete. 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.083+0000 7fc5a6e00640 1 Processor -- start 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.083+0000 7fc5a6e00640 1 -- start start 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a6e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5a01033e0 0x7fc5a0199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a6e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5a0108d20 0x7fc5a019a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a6e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5a0109670 0x7fc5a019f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a6e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a0113350 con 0x7fc5a0108d20 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a6e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a01131d0 con 0x7fc5a0109670 2024-07-26T12:49:45.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a6e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a01134d0 con 0x7fc5a01033e0 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a5a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5a01033e0 0x7fc5a0199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a5a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5a01033e0 0x7fc5a0199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33072/0 (socket says 172.21.15.107:33072) 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a5a00640 1 -- 172.21.15.107:0/2727711099 learned_addr learned my addr 172.21.15.107:0/2727711099 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a6400640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5a0109670 0x7fc5a019f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a5a00640 1 -- 172.21.15.107:0/2727711099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5a0109670 msgr2=0x7fc5a019f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a5a00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5a0109670 0x7fc5a019f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a5a00640 1 -- 172.21.15.107:0/2727711099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5a0108d20 msgr2=0x7fc5a019a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a5a00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5a0108d20 0x7fc5a019a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.084+0000 7fc5a5a00640 1 -- 172.21.15.107:0/2727711099 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a019fd10 con 0x7fc5a01033e0 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.085+0000 7fc5a5a00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5a01033e0 0x7fc5a0199da0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc59400e9f0 tx=0x7fc59400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.085+0000 7fc59e000640 1 -- 172.21.15.107:0/2727711099 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59400cd80 con 0x7fc5a01033e0 2024-07-26T12:49:45.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.085+0000 7fc5a6e00640 1 -- 172.21.15.107:0/2727711099 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a01a0000 con 0x7fc5a01033e0 2024-07-26T12:49:45.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.085+0000 7fc59e000640 1 -- 172.21.15.107:0/2727711099 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5940054d0 con 0x7fc5a01033e0 2024-07-26T12:49:45.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.085+0000 7fc59e000640 1 -- 172.21.15.107:0/2727711099 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594010690 con 0x7fc5a01033e0 2024-07-26T12:49:45.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.085+0000 7fc5a6e00640 1 -- 172.21.15.107:0/2727711099 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a0075020 con 0x7fc5a01033e0 2024-07-26T12:49:45.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.089+0000 7fc5af088740 1 Processor -- start 2024-07-26T12:49:45.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.089+0000 7fc5af088740 1 -- start start 2024-07-26T12:49:45.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.090+0000 7fc5af088740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 0x55d40f68e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.090+0000 7fc5af088740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c01b0 0x55d40f68e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.090+0000 7fc5af088740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c8390 0x55d40f68eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.090+0000 7fc5af088740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d40f604bf0 con 0x55d40f2c8390 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.090+0000 7fc5af088740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d40f5f40f0 con 0x55d40f2c01b0 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.090+0000 7fc5af088740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d40f5f8130 con 0x55d40f2e9070 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.090+0000 7fc5a6400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 0x55d40f68e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.090+0000 7fc5a6400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 0x55d40f68e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33084/0 (socket says 172.21.15.107:33084) 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.090+0000 7fc5a6400640 1 -- 172.21.15.107:0/1676404175 learned_addr learned my addr 172.21.15.107:0/1676404175 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.091+0000 7fc59e000640 1 -- 172.21.15.107:0/2727711099 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fc594004e90 con 0x7fc5a01033e0 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.091+0000 7fc5a6400640 1 -- 172.21.15.107:0/1676404175 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c01b0 msgr2=0x55d40f68e570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.091+0000 7fc5a6400640 1 --2- 172.21.15.107:0/1676404175 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c01b0 0x55d40f68e570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.091+0000 7fc5a6400640 1 -- 172.21.15.107:0/1676404175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c8390 msgr2=0x55d40f68eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.091+0000 7fc5a6400640 1 --2- 172.21.15.107:0/1676404175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c8390 0x55d40f68eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.091+0000 7fc5a6400640 1 -- 172.21.15.107:0/1676404175 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d40f354650 con 0x55d40f2e9070 2024-07-26T12:49:45.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.091+0000 7fc5a6400640 1 --2- 172.21.15.107:0/1676404175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 0x55d40f68e030 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc598009fd0 tx=0x7fc5980118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0de37ed16ddd83e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:45.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.092+0000 7fc592a00640 1 -- 172.21.15.107:0/1676404175 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598004d20 con 0x55d40f2e9070 2024-07-26T12:49:45.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.092+0000 7fc592a00640 1 -- 172.21.15.107:0/1676404175 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc598004050 con 0x55d40f2e9070 2024-07-26T12:49:45.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.092+0000 7fc592a00640 1 -- 172.21.15.107:0/1676404175 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598005280 con 0x55d40f2e9070 2024-07-26T12:49:45.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.092+0000 7fc59e000640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5880b3d40 0x7fc5880b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.093+0000 7fc59e000640 1 -- 172.21.15.107:0/2727711099 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc5940e4880 con 0x7fc5a01033e0 2024-07-26T12:49:45.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.093+0000 7fc5a5a00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5880b3d40 0x7fc5880b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.093+0000 7fc5af088740 1 -- 172.21.15.107:0/1676404175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 msgr2=0x55d40f68e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:45.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.093+0000 7fc5af088740 1 --2- 172.21.15.107:0/1676404175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 0x55d40f68e030 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc598009fd0 tx=0x7fc5980118d0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.093+0000 7fc5af088740 1 -- 172.21.15.107:0/1676404175 shutdown_connections 2024-07-26T12:49:45.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.093+0000 7fc5af088740 1 --2- 172.21.15.107:0/1676404175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c8390 0x55d40f68eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.093+0000 7fc5af088740 1 --2- 172.21.15.107:0/1676404175 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c01b0 0x55d40f68e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.093+0000 7fc5af088740 1 --2- 172.21.15.107:0/1676404175 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 0x55d40f68e030 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.093+0000 7fc5af088740 1 -- 172.21.15.107:0/1676404175 >> 172.21.15.107:0/1676404175 conn(0x55d40f2dd460 msgr2=0x55d40f68bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:45.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.094+0000 7fc5af088740 1 -- 172.21.15.107:0/1676404175 shutdown_connections 2024-07-26T12:49:45.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.094+0000 7fc5af088740 1 -- 172.21.15.107:0/1676404175 wait complete. 2024-07-26T12:49:45.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.094+0000 7fc5a5a00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5880b3d40 0x7fc5880b6250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55d40f68ccb0 tx=0x7fc59400cf20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:45.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.095+0000 7fc5af088740 1 -- start start 2024-07-26T12:49:45.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.095+0000 7fc5af088740 1 Processor -- start 2024-07-26T12:49:45.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.095+0000 7fc5af088740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6994a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5af088740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c8390 0x55d40f6999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5af088740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2e9070 0x55d40f699f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5af088740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d40f457700 con 0x55d40f2c8390 2024-07-26T12:49:45.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5af088740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55d40f5906c0 con 0x55d40f2e9070 2024-07-26T12:49:45.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5af088740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55d40f69c810 con 0x55d40f2c01b0 2024-07-26T12:49:45.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5af088740 10 client.0 _ll_register_callbacks cb 0x55d40f5a0170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:45.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5a6400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5a6400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33096/0 (socket says 172.21.15.107:33096) 2024-07-26T12:49:45.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5a6400640 1 -- 172.21.15.107:0/2676498936 learned_addr learned my addr 172.21.15.107:0/2676498936 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:45.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5a5000640 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c8390 0x55d40f6999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.096+0000 7fc5a6400640 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2e9070 0x55d40f699f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.097+0000 7fc5a6400640 1 -- 172.21.15.107:0/2676498936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2e9070 msgr2=0x55d40f699f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:45.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.097+0000 7fc5a6400640 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2e9070 0x55d40f699f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.097+0000 7fc5a6400640 1 -- 172.21.15.107:0/2676498936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c8390 msgr2=0x55d40f6999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:45.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.097+0000 7fc5a6400640 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c8390 0x55d40f6999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.097+0000 7fc5a6400640 1 -- 172.21.15.107:0/2676498936 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d40f5a7860 con 0x55d40f2c01b0 2024-07-26T12:49:45.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.097+0000 7fc5a5000640 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c8390 0x55d40f6999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:49:45.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.098+0000 7fc5a6400640 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6994a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc598004910 tx=0x7fc598005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a60ed84798f9d2b1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:45.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.098+0000 7fc587400640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:49:45.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.098+0000 7fc587400640 1 -- 172.21.15.107:0/2676498936 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59800f8f0 con 0x55d40f2c01b0 2024-07-26T12:49:45.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.098+0000 7fc587400640 1 -- 172.21.15.107:0/2676498936 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc59800fa90 con 0x55d40f2c01b0 2024-07-26T12:49:45.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.098+0000 7fc5af088740 1 -- 172.21.15.107:0/2676498936 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc598004050 con 0x55d40f2c01b0 2024-07-26T12:49:45.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.099+0000 7fc587400640 1 -- 172.21.15.107:0/2676498936 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc59800fc90 con 0x55d40f2c01b0 2024-07-26T12:49:45.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.099+0000 7fc5af088740 10 client.25636 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:45.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.099+0000 7fc5af088740 1 -- 172.21.15.107:0/2676498936 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d40f6b0b50 con 0x55d40f2c01b0 2024-07-26T12:49:45.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.099+0000 7fc587400640 1 -- 172.21.15.107:0/2676498936 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc598004050 con 0x55d40f2c01b0 2024-07-26T12:49:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.099+0000 7fc587400640 1 -- 172.21.15.107:0/2676498936 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc598025040 con 0x55d40f2c01b0 2024-07-26T12:49:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.100+0000 7fc5af088740 10 client.25636 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.100+0000 7fc5af088740 10 client.25636 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-07-26T12:49:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.100+0000 7fc5af088740 20 client.25636 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.100+0000 7fc5af088740 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55d40f2a79f0 0x55d40f6b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.100+0000 7fc5af088740 4 client.25636 mds_command: new command op to 24682 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:49:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.100+0000 7fc5af088740 1 -- 172.21.15.107:0/2676498936 --> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d40f606b40 con 0x55d40f2a79f0 2024-07-26T12:49:45.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.100+0000 7fc5a5000640 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55d40f2a79f0 0x55d40f6b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.101+0000 7fc5a5000640 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55d40f2a79f0 0x55d40f6b3f60 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=799e50adab0e214b server_cookie=87b10c8e529ca1c6 in_seq=0 out_seq=0 2024-07-26T12:49:45.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.101+0000 7fc587400640 10 client.25636 ms_handle_connect on v2:172.21.15.132:6834/1242618859 2024-07-26T12:49:45.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc587400640 1 -- 172.21.15.107:0/2676498936 <== mds.4 v2:172.21.15.132:6834/1242618859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d40f606b40 con 0x55d40f2a79f0 2024-07-26T12:49:45.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc587400640 10 client.25636 handle_command_reply: tid=0 2024-07-26T12:49:45.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc5af088740 1 client.25636 shutdown 2024-07-26T12:49:45.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc5af088740 1 -- 172.21.15.107:0/2676498936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 msgr2=0x55d40f6994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:45.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc5af088740 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6994a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc598004910 tx=0x7fc598005bd0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc5af088740 1 -- 172.21.15.107:0/2676498936 shutdown_connections 2024-07-26T12:49:45.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc5af088740 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55d40f2a79f0 0x55d40f6b3f60 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc5af088740 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2e9070 0x55d40f699f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc5af088740 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c8390 0x55d40f6999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc5af088740 1 --2- 172.21.15.107:0/2676498936 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6994a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc5af088740 1 -- 172.21.15.107:0/2676498936 >> 172.21.15.107:0/2676498936 conn(0x55d40f2dd460 msgr2=0x55d40f6911c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:45.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.102+0000 7fc587400640 0 client.25636 ms_handle_reset on v2:172.21.15.132:6834/1242618859 2024-07-26T12:49:45.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.103+0000 7fc5af088740 1 -- 172.21.15.107:0/2676498936 shutdown_connections 2024-07-26T12:49:45.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.103+0000 7fc5af088740 1 -- 172.21.15.107:0/2676498936 wait complete. 2024-07-26T12:49:45.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.103+0000 7fc5af088740 20 client.25636 trim_cache size 0 max 16384 2024-07-26T12:49:45.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.118+0000 7fc5af088740 1 Processor -- start 2024-07-26T12:49:45.119 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:44 smithi107 ceph-mon[32659]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T12:49:45.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.118+0000 7fc5af088740 1 -- start start 2024-07-26T12:49:45.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.118+0000 7fc5af088740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c01b0 0x55d40f6ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.118+0000 7fc5af088740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 0x55d40f699960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.119+0000 7fc5af088740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 0x55d40f699ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.119+0000 7fc5af088740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a01ae630 con 0x55d40f2c01b0 2024-07-26T12:49:45.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.119+0000 7fc5af088740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a01b0170 con 0x55d40f2c8390 2024-07-26T12:49:45.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.119+0000 7fc5af088740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a01aed00 con 0x55d40f2e9070 2024-07-26T12:49:45.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.119+0000 7fc5a6400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c01b0 0x55d40f6ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.119+0000 7fc5a5000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 0x55d40f699960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.119+0000 7fc5a5000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 0x55d40f699960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57506/0 (socket says 172.21.15.107:57506) 2024-07-26T12:49:45.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.119+0000 7fc5a6400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c01b0 0x55d40f6ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:49864/0 (socket says 172.21.15.107:49864) 2024-07-26T12:49:45.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.119+0000 7fc5a5000640 1 -- 172.21.15.107:0/3868173106 learned_addr learned my addr 172.21.15.107:0/3868173106 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:45.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.119+0000 7fc5a6400640 1 --2- 172.21.15.107:0/3868173106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 0x55d40f699ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.120+0000 7fc5a6400640 1 -- 172.21.15.107:0/3868173106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 msgr2=0x55d40f699ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.120+0000 7fc5a6400640 1 --2- 172.21.15.107:0/3868173106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 0x55d40f699ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.120+0000 7fc5a6400640 1 -- 172.21.15.107:0/3868173106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 msgr2=0x55d40f699960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.120+0000 7fc5a6400640 1 --2- 172.21.15.107:0/3868173106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 0x55d40f699960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.120+0000 7fc5a6400640 1 -- 172.21.15.107:0/3868173106 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d40f690320 con 0x55d40f2c01b0 2024-07-26T12:49:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.120+0000 7fc5a6400640 1 --2- 172.21.15.107:0/3868173106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c01b0 0x55d40f6ae1e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc598005d50 tx=0x7fc598009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2e9401e133fadf9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.121+0000 7fc592000640 1 -- 172.21.15.107:0/3868173106 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59803a040 con 0x55d40f2c01b0 2024-07-26T12:49:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.121+0000 7fc592000640 1 -- 172.21.15.107:0/3868173106 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc59802d070 con 0x55d40f2c01b0 2024-07-26T12:49:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.121+0000 7fc592000640 1 -- 172.21.15.107:0/3868173106 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59800f950 con 0x55d40f2c01b0 2024-07-26T12:49:45.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.122+0000 7fc5af088740 1 -- 172.21.15.107:0/3868173106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c01b0 msgr2=0x55d40f6ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.122+0000 7fc5af088740 1 --2- 172.21.15.107:0/3868173106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c01b0 0x55d40f6ae1e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc598005d50 tx=0x7fc598009230 comp rx=0 tx=0).stop 2024-07-26T12:49:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.122+0000 7fc5af088740 1 -- 172.21.15.107:0/3868173106 shutdown_connections 2024-07-26T12:49:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.122+0000 7fc5af088740 1 --2- 172.21.15.107:0/3868173106 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2e9070 0x55d40f699ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.122+0000 7fc5af088740 1 --2- 172.21.15.107:0/3868173106 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 0x55d40f699960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.122+0000 7fc5af088740 1 --2- 172.21.15.107:0/3868173106 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2c01b0 0x55d40f6ae1e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.122+0000 7fc5af088740 1 -- 172.21.15.107:0/3868173106 >> 172.21.15.107:0/3868173106 conn(0x55d40f2dd460 msgr2=0x55d40f699250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:45.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.122+0000 7fc5af088740 1 -- 172.21.15.107:0/3868173106 shutdown_connections 2024-07-26T12:49:45.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.123+0000 7fc5af088740 1 -- 172.21.15.107:0/3868173106 wait complete. 2024-07-26T12:49:45.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.123+0000 7fc5af088740 1 -- start start 2024-07-26T12:49:45.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.123+0000 7fc5af088740 1 Processor -- start 2024-07-26T12:49:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.124+0000 7fc5af088740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6a7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.124+0000 7fc5af088740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 0x55d40f6a8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.124+0000 7fc5af088740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2e9070 0x55d40f713520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.124+0000 7fc5af088740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a01afaa0 con 0x55d40f2e9070 2024-07-26T12:49:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.124+0000 7fc5af088740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a01af3d0 con 0x55d40f2c8390 2024-07-26T12:49:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.124+0000 7fc5af088740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a01ac610 con 0x55d40f2c01b0 2024-07-26T12:49:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.124+0000 7fc5af088740 10 client.0 _ll_register_callbacks cb 0x7fc598004650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:45.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.125+0000 7fc5a6400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6a7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.125+0000 7fc5a6400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6a7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33116/0 (socket says 172.21.15.107:33116) 2024-07-26T12:49:45.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.125+0000 7fc5a6400640 1 -- 172.21.15.107:0/1570267072 learned_addr learned my addr 172.21.15.107:0/1570267072 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:45.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.125+0000 7fc5a6400640 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2e9070 0x55d40f713520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.125+0000 7fc5a5000640 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 0x55d40f6a8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.126+0000 7fc5a6400640 1 -- 172.21.15.107:0/1570267072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 msgr2=0x55d40f6a7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.126+0000 7fc5a6400640 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6a7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.126+0000 7fc5a6400640 1 -- 172.21.15.107:0/1570267072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 msgr2=0x55d40f6a8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.126+0000 7fc5a6400640 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 0x55d40f6a8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.126+0000 7fc5a6400640 1 -- 172.21.15.107:0/1570267072 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d40f713b90 con 0x55d40f2e9070 2024-07-26T12:49:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.127+0000 7fc5a6400640 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2e9070 0x55d40f713520 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc59800a010 tx=0x7fc598000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0dc33a30e4b0e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.127+0000 7fc587400640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:49:45.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.127+0000 7fc587400640 1 -- 172.21.15.107:0/1570267072 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59803a070 con 0x55d40f2e9070 2024-07-26T12:49:45.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.128+0000 7fc5af088740 1 -- 172.21.15.107:0/1570267072 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc59802d070 con 0x55d40f2e9070 2024-07-26T12:49:45.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.128+0000 7fc587400640 1 -- 172.21.15.107:0/1570267072 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc59800f8f0 con 0x55d40f2e9070 2024-07-26T12:49:45.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.128+0000 7fc587400640 1 -- 172.21.15.107:0/1570267072 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc598050050 con 0x55d40f2e9070 2024-07-26T12:49:45.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.128+0000 7fc5af088740 10 client.15495 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:45.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.128+0000 7fc5af088740 1 -- 172.21.15.107:0/1570267072 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d40f71c050 con 0x55d40f2e9070 2024-07-26T12:49:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.129+0000 7fc587400640 1 -- 172.21.15.107:0/1570267072 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc59802d070 con 0x55d40f2e9070 2024-07-26T12:49:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.129+0000 7fc587400640 1 -- 172.21.15.107:0/1570267072 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc598028810 con 0x55d40f2e9070 2024-07-26T12:49:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.129+0000 7fc5af088740 10 client.15495 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:45.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.129+0000 7fc5af088740 10 client.15495 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-07-26T12:49:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.129+0000 7fc5af088740 20 client.15495 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.129+0000 7fc5af088740 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55d40f27cda0 0x55d40f71e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.129+0000 7fc5af088740 4 client.15495 mds_command: new command op to 24682 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T12:49:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.129+0000 7fc5af088740 1 -- 172.21.15.107:0/1570267072 --> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d40f6025e0 con 0x55d40f27cda0 2024-07-26T12:49:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.130+0000 7fc5a5000640 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55d40f27cda0 0x55d40f71e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.131+0000 7fc587400640 10 client.15495 ms_handle_connect on v2:172.21.15.132:6834/1242618859 2024-07-26T12:49:45.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.131+0000 7fc5a5000640 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55d40f27cda0 0x55d40f71e670 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=21297bd5505c6fe2 server_cookie=2c0f691e1f3f6d2a in_seq=0 out_seq=0 2024-07-26T12:49:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.131+0000 7fc587400640 1 -- 172.21.15.107:0/1570267072 <== mds.4 v2:172.21.15.132:6834/1242618859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d40f6025e0 con 0x55d40f27cda0 2024-07-26T12:49:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.131+0000 7fc587400640 10 client.15495 handle_command_reply: tid=0 2024-07-26T12:49:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.131+0000 7fc5af088740 1 client.15495 shutdown 2024-07-26T12:49:45.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.132+0000 7fc5af088740 1 -- 172.21.15.107:0/1570267072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2e9070 msgr2=0x55d40f713520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.132+0000 7fc5af088740 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2e9070 0x55d40f713520 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc59800a010 tx=0x7fc598000f40 comp rx=0 tx=0).stop 2024-07-26T12:49:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.132+0000 7fc5af088740 1 -- 172.21.15.107:0/1570267072 shutdown_connections 2024-07-26T12:49:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.132+0000 7fc5af088740 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x55d40f27cda0 0x55d40f71e670 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.132+0000 7fc5af088740 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d40f2e9070 0x55d40f713520 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.132+0000 7fc5af088740 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55d40f2c8390 0x55d40f6a8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.132+0000 7fc587400640 0 client.15495 ms_handle_reset on v2:172.21.15.132:6834/1242618859 2024-07-26T12:49:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.132+0000 7fc5af088740 1 --2- 172.21.15.107:0/1570267072 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55d40f2c01b0 0x55d40f6a7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.132+0000 7fc5af088740 1 -- 172.21.15.107:0/1570267072 >> 172.21.15.107:0/1570267072 conn(0x55d40f2dd460 msgr2=0x55d40f699c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.133+0000 7fc5af088740 1 -- 172.21.15.107:0/1570267072 shutdown_connections 2024-07-26T12:49:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.133+0000 7fc5af088740 1 -- 172.21.15.107:0/1570267072 wait complete. 2024-07-26T12:49:45.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.133+0000 7fc5af088740 20 client.15495 trim_cache size 0 max 16384 2024-07-26T12:49:45.134 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-07-26T12:49:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.134+0000 7fc5a6e00640 1 -- 172.21.15.107:0/2727711099 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5880b3d40 msgr2=0x7fc5880b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.134+0000 7fc5a6e00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5880b3d40 0x7fc5880b6250 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55d40f68ccb0 tx=0x7fc59400cf20 comp rx=0 tx=0).stop 2024-07-26T12:49:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.134+0000 7fc5a6e00640 1 -- 172.21.15.107:0/2727711099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5a01033e0 msgr2=0x7fc5a0199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:45.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.134+0000 7fc5a6e00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5a01033e0 0x7fc5a0199da0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc59400e9f0 tx=0x7fc59400edf0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.134+0000 7fc5a6e00640 1 -- 172.21.15.107:0/2727711099 shutdown_connections 2024-07-26T12:49:45.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.134+0000 7fc5a6e00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fc5880b3d40 0x7fc5880b6250 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.134+0000 7fc5a6e00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fc5a0109670 0x7fc5a019f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.134+0000 7fc5a6e00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5a0108d20 0x7fc5a019a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.134+0000 7fc5a6e00640 1 --2- 172.21.15.107:0/2727711099 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc5a01033e0 0x7fc5a0199da0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:45.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.134+0000 7fc5a6e00640 1 -- 172.21.15.107:0/2727711099 >> 172.21.15.107:0/2727711099 conn(0x7fc5a00fee70 msgr2=0x7fc5a0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:45.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.135+0000 7fc5a6e00640 1 -- 172.21.15.107:0/2727711099 shutdown_connections 2024-07-26T12:49:45.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:45.135+0000 7fc5a6e00640 1 -- 172.21.15.107:0/2727711099 wait complete. 2024-07-26T12:49:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:44 smithi112 ceph-mon[37501]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T12:49:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:44 smithi132 ceph-mon[37580]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T12:49:46.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T12:49:46.472 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:49:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:46 smithi112 ceph-mon[37501]: pgmap v504: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T12:49:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:46 smithi107 ceph-mon[32659]: pgmap v504: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T12:49:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:46 smithi132 ceph-mon[37580]: pgmap v504: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T12:49:48.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:48 smithi107 ceph-mon[32659]: pgmap v505: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T12:49:48.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.967+0000 7f3588a00640 1 -- 172.21.15.107:0/932023051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3584075ab0 msgr2=0x7f358410f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:48.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.967+0000 7f3588a00640 1 --2- 172.21.15.107:0/932023051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3584075ab0 0x7f358410f480 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3570009860 tx=0x7f357002a340 comp rx=0 tx=0).stop 2024-07-26T12:49:48.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.967+0000 7f3588a00640 1 -- 172.21.15.107:0/932023051 shutdown_connections 2024-07-26T12:49:48.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.967+0000 7f3588a00640 1 --2- 172.21.15.107:0/932023051 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f358410f9c0 0x7f3584111de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.967+0000 7f3588a00640 1 --2- 172.21.15.107:0/932023051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3584075ab0 0x7f358410f480 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.967+0000 7f3588a00640 1 --2- 172.21.15.107:0/932023051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3584075160 0x7f3584075570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.967+0000 7f3588a00640 1 -- 172.21.15.107:0/932023051 >> 172.21.15.107:0/932023051 conn(0x7f35840fee30 msgr2=0x7f3584101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:48.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.967+0000 7f3588a00640 1 -- 172.21.15.107:0/932023051 shutdown_connections 2024-07-26T12:49:48.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.967+0000 7f3588a00640 1 -- 172.21.15.107:0/932023051 wait complete. 2024-07-26T12:49:48.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.968+0000 7f3588a00640 1 Processor -- start 2024-07-26T12:49:48.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.969+0000 7f3588a00640 1 -- start start 2024-07-26T12:49:48.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.969+0000 7f3588a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3584075160 0x7f358419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.969+0000 7f3588a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3584075ab0 0x7f358419c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.969+0000 7f3588a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f358410f9c0 0x7f35841a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.969+0000 7f3588a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f35841154d0 con 0x7f3584075ab0 2024-07-26T12:49:48.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.969+0000 7f3588a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3584115350 con 0x7f3584075160 2024-07-26T12:49:48.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.969+0000 7f3588a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3584115650 con 0x7f358410f9c0 2024-07-26T12:49:48.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.970+0000 7f3582800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3584075160 0x7f358419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:48.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.970+0000 7f3582800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3584075160 0x7f358419c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57544/0 (socket says 172.21.15.107:57544) 2024-07-26T12:49:48.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.970+0000 7f3582800640 1 -- 172.21.15.107:0/1024858481 learned_addr learned my addr 172.21.15.107:0/1024858481 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:48.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.970+0000 7f3581e00640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3584075ab0 0x7f358419c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:48.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.970+0000 7f3582800640 1 -- 172.21.15.107:0/1024858481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f358410f9c0 msgr2=0x7f35841a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:48.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.970+0000 7f3582800640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f358410f9c0 0x7f35841a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.970+0000 7f3582800640 1 -- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3584075ab0 msgr2=0x7f358419c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:48.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.970+0000 7f3582800640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3584075ab0 0x7f358419c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.970+0000 7f3582800640 1 -- 172.21.15.107:0/1024858481 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35841a1f70 con 0x7f3584075160 2024-07-26T12:49:48.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.970+0000 7f3581e00640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3584075ab0 0x7f358419c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:49:48.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.971+0000 7f3582800640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3584075160 0x7f358419c000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f357400e9f0 tx=0x7f357400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:48.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.971+0000 7f357b400640 1 -- 172.21.15.107:0/1024858481 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357400cd80 con 0x7f3584075160 2024-07-26T12:49:48.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.971+0000 7f357b400640 1 -- 172.21.15.107:0/1024858481 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35740108e0 con 0x7f3584075160 2024-07-26T12:49:48.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.971+0000 7f357b400640 1 -- 172.21.15.107:0/1024858481 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3574018980 con 0x7f3584075160 2024-07-26T12:49:48.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.971+0000 7f3588a00640 1 -- 172.21.15.107:0/1024858481 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35841a2260 con 0x7f3584075160 2024-07-26T12:49:48.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.971+0000 7f3588a00640 1 -- 172.21.15.107:0/1024858481 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35841a89c0 con 0x7f3584075160 2024-07-26T12:49:48.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.975+0000 7f357b400640 1 -- 172.21.15.107:0/1024858481 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f3574010430 con 0x7f3584075160 2024-07-26T12:49:48.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.976+0000 7f35900ca740 1 Processor -- start 2024-07-26T12:49:48.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.976+0000 7f35900ca740 1 -- start start 2024-07-26T12:49:48.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.976+0000 7f35900ca740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 0x5602f2371030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f35900ca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 0x5602f2371570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f35900ca740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f2371ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f35900ca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5602f22e7bf0 con 0x5602f1fa31b0 2024-07-26T12:49:48.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f35900ca740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5602f22d70f0 con 0x5602f1fcc070 2024-07-26T12:49:48.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f35900ca740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5602f22db130 con 0x5602f1fab390 2024-07-26T12:49:48.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f3581e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f2371ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:48.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f3581e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f2371ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33150/0 (socket says 172.21.15.107:33150) 2024-07-26T12:49:48.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f3581e00640 1 -- 172.21.15.107:0/469841243 learned_addr learned my addr 172.21.15.107:0/469841243 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:48.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f357b400640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35680b3d40 0x7f35680b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f3583200640 1 --2- 172.21.15.107:0/469841243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 0x5602f2371030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:48.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.977+0000 7f357b400640 1 -- 172.21.15.107:0/1024858481 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f35740e4810 con 0x7f3584075160 2024-07-26T12:49:48.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.978+0000 7f3582800640 1 --2- 172.21.15.107:0/469841243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 0x5602f2371570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:48.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.978+0000 7f3581e00640 1 -- 172.21.15.107:0/469841243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 msgr2=0x5602f2371030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:48.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.978+0000 7f3581e00640 1 --2- 172.21.15.107:0/469841243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 0x5602f2371030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.978+0000 7f3581e00640 1 -- 172.21.15.107:0/469841243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 msgr2=0x5602f2371570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:48.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.978+0000 7f3581e00640 1 --2- 172.21.15.107:0/469841243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 0x5602f2371570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.978+0000 7f3581e00640 1 -- 172.21.15.107:0/469841243 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f2037650 con 0x5602f1fab390 2024-07-26T12:49:48.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.978+0000 7f3583200640 1 --2- 172.21.15.107:0/469841243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 0x5602f2371030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:49:48.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.980+0000 7f3583200640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35680b3d40 0x7f35680b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:48.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.980+0000 7f3581e00640 1 --2- 172.21.15.107:0/469841243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f2371ab0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f357002a860 tx=0x7f3570030b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb06c783a8115741 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:48.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.980+0000 7f3583200640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35680b3d40 0x7f35680b6250 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f357c005fd0 tx=0x7f357c004a40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:48.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.981+0000 7f356f400640 1 -- 172.21.15.107:0/469841243 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570040020 con 0x5602f1fab390 2024-07-26T12:49:48.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.981+0000 7f356f400640 1 -- 172.21.15.107:0/469841243 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3570038040 con 0x5602f1fab390 2024-07-26T12:49:48.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.981+0000 7f356f400640 1 -- 172.21.15.107:0/469841243 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570037070 con 0x5602f1fab390 2024-07-26T12:49:48.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.982+0000 7f35900ca740 1 -- 172.21.15.107:0/469841243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 msgr2=0x5602f2371ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:48.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.982+0000 7f35900ca740 1 --2- 172.21.15.107:0/469841243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f2371ab0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f357002a860 tx=0x7f3570030b50 comp rx=0 tx=0).stop 2024-07-26T12:49:48.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.982+0000 7f35900ca740 1 -- 172.21.15.107:0/469841243 shutdown_connections 2024-07-26T12:49:48.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.982+0000 7f35900ca740 1 --2- 172.21.15.107:0/469841243 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f2371ab0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.982+0000 7f35900ca740 1 --2- 172.21.15.107:0/469841243 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 0x5602f2371570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.982+0000 7f35900ca740 1 --2- 172.21.15.107:0/469841243 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 0x5602f2371030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.982+0000 7f35900ca740 1 -- 172.21.15.107:0/469841243 >> 172.21.15.107:0/469841243 conn(0x5602f1fc0460 msgr2=0x5602f236ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:48.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.983+0000 7f35900ca740 1 -- 172.21.15.107:0/469841243 shutdown_connections 2024-07-26T12:49:48.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.983+0000 7f35900ca740 1 -- 172.21.15.107:0/469841243 wait complete. 2024-07-26T12:49:48.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.984+0000 7f35900ca740 1 -- start start 2024-07-26T12:49:48.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.984+0000 7f35900ca740 1 Processor -- start 2024-07-26T12:49:48.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.985+0000 7f35900ca740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 0x5602f237c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.985+0000 7f35900ca740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f237cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.985+0000 7f35900ca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 0x5602f237d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.985+0000 7f35900ca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5602f213a700 con 0x5602f1fcc070 2024-07-26T12:49:48.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.985+0000 7f35900ca740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5602f22736c0 con 0x5602f1fa31b0 2024-07-26T12:49:48.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.985+0000 7f35900ca740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5602f237f7b0 con 0x5602f1fab390 2024-07-26T12:49:48.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.985+0000 7f35900ca740 10 client.0 _ll_register_callbacks cb 0x5602f2283170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:48.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.985+0000 7f3581e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f237cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:48.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.986+0000 7f3582800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 0x5602f237c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:48.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.986+0000 7f3583200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 0x5602f237d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:48.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.986+0000 7f3582800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 0x5602f237c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57558/0 (socket says 172.21.15.107:57558) 2024-07-26T12:49:48.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.986+0000 7f3582800640 1 -- 172.21.15.107:0/470417867 learned_addr learned my addr 172.21.15.107:0/470417867 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:48.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.986+0000 7f3583200640 1 -- 172.21.15.107:0/470417867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 msgr2=0x5602f237cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:48.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.986+0000 7f3583200640 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f237cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.986+0000 7f3583200640 1 -- 172.21.15.107:0/470417867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 msgr2=0x5602f237c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:48.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.986+0000 7f3583200640 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 0x5602f237c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.986+0000 7f3583200640 1 -- 172.21.15.107:0/470417867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f228a860 con 0x5602f1fcc070 2024-07-26T12:49:48.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.987+0000 7f3583200640 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 0x5602f237d070 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f357c01be20 tx=0x7f357c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1b363b609ee2b4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:48.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.987+0000 7f3563e00640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:49:48.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.987+0000 7f3563e00640 1 -- 172.21.15.107:0/470417867 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357c028060 con 0x5602f1fcc070 2024-07-26T12:49:48.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.988+0000 7f35900ca740 1 -- 172.21.15.107:0/470417867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3570038040 con 0x5602f1fcc070 2024-07-26T12:49:48.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.988+0000 7f3563e00640 1 -- 172.21.15.107:0/470417867 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f357c023630 con 0x5602f1fcc070 2024-07-26T12:49:48.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.988+0000 7f3563e00640 1 -- 172.21.15.107:0/470417867 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f357c02f050 con 0x5602f1fcc070 2024-07-26T12:49:48.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.988+0000 7f35900ca740 10 client.15504 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:48.989 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.988+0000 7f35900ca740 1 -- 172.21.15.107:0/470417867 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5602f238b380 con 0x5602f1fcc070 2024-07-26T12:49:48.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.988+0000 7f3563e00640 1 -- 172.21.15.107:0/470417867 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3570038040 con 0x5602f1fcc070 2024-07-26T12:49:48.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.989+0000 7f3563e00640 1 -- 172.21.15.107:0/470417867 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f357c00b040 con 0x5602f1fcc070 2024-07-26T12:49:48.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.989+0000 7f35900ca740 10 client.15504 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:48.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.989+0000 7f35900ca740 10 client.15504 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T12:49:48.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.989+0000 7f35900ca740 20 client.15504 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:48.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.989+0000 7f35900ca740 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x5602f1f8a9f0 0x5602f238e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:48.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.989+0000 7f35900ca740 4 client.15504 mds_command: new command op to 24694 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:49:48.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.989+0000 7f35900ca740 1 -- 172.21.15.107:0/470417867 --> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602f22e9b40 con 0x5602f1f8a9f0 2024-07-26T12:49:48.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.990+0000 7f3582800640 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x5602f1f8a9f0 0x5602f238e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:48.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.990+0000 7f3582800640 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x5602f1f8a9f0 0x5602f238e8a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=75557d66a9e373ba server_cookie=40861ea848761d58 in_seq=0 out_seq=0 2024-07-26T12:49:48.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.990+0000 7f3563e00640 10 client.15504 ms_handle_connect on v2:172.21.15.132:6836/735918310 2024-07-26T12:49:48.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.991+0000 7f3563e00640 1 -- 172.21.15.107:0/470417867 <== mds.3 v2:172.21.15.132:6836/735918310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5602f22e9b40 con 0x5602f1f8a9f0 2024-07-26T12:49:48.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.991+0000 7f3563e00640 10 client.15504 handle_command_reply: tid=0 2024-07-26T12:49:48.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.991+0000 7f35900ca740 1 client.15504 shutdown 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.992+0000 7f35900ca740 1 -- 172.21.15.107:0/470417867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 msgr2=0x5602f237d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.992+0000 7f35900ca740 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 0x5602f237d070 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f357c01be20 tx=0x7f357c0095c0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.992+0000 7f35900ca740 1 -- 172.21.15.107:0/470417867 shutdown_connections 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.992+0000 7f35900ca740 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x5602f1f8a9f0 0x5602f238e8a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.992+0000 7f35900ca740 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 0x5602f237d070 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.992+0000 7f35900ca740 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f237cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.992+0000 7f35900ca740 1 --2- 172.21.15.107:0/470417867 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 0x5602f237c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.992+0000 7f35900ca740 1 -- 172.21.15.107:0/470417867 >> 172.21.15.107:0/470417867 conn(0x5602f1fc0460 msgr2=0x5602f23741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.992+0000 7f3563e00640 0 client.15504 ms_handle_reset on v2:172.21.15.132:6836/735918310 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.993+0000 7f35900ca740 1 -- 172.21.15.107:0/470417867 shutdown_connections 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.993+0000 7f35900ca740 1 -- 172.21.15.107:0/470417867 wait complete. 2024-07-26T12:49:48.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:48.993+0000 7f35900ca740 20 client.15504 trim_cache size 0 max 16384 2024-07-26T12:49:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.019+0000 7f35900ca740 1 Processor -- start 2024-07-26T12:49:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.019+0000 7f35900ca740 1 -- start start 2024-07-26T12:49:49.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f35900ca740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 0x5602f237cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f35900ca740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f236d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f35900ca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 0x5602f2372770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f35900ca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f35841b2a10 con 0x5602f1fcc070 2024-07-26T12:49:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f35900ca740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35841b4550 con 0x5602f1fa31b0 2024-07-26T12:49:49.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f35900ca740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f35841b30e0 con 0x5602f1fab390 2024-07-26T12:49:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f3582800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 0x5602f237cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f3581e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f236d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f3582800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 0x5602f237cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57564/0 (socket says 172.21.15.107:57564) 2024-07-26T12:49:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f3581e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f236d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33168/0 (socket says 172.21.15.107:33168) 2024-07-26T12:49:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.020+0000 7f3582800640 1 -- 172.21.15.107:0/1928758163 learned_addr learned my addr 172.21.15.107:0/1928758163 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.021+0000 7f3583200640 1 --2- 172.21.15.107:0/1928758163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 0x5602f2372770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.021+0000 7f3581e00640 1 -- 172.21.15.107:0/1928758163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 msgr2=0x5602f237cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.021+0000 7f3581e00640 1 --2- 172.21.15.107:0/1928758163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 0x5602f237cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.021+0000 7f3581e00640 1 -- 172.21.15.107:0/1928758163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 msgr2=0x5602f2372770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:49.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.021+0000 7f3581e00640 1 --2- 172.21.15.107:0/1928758163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 0x5602f2372770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.021+0000 7f3581e00640 1 -- 172.21.15.107:0/1928758163 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f2371e40 con 0x5602f1fab390 2024-07-26T12:49:49.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.021+0000 7f3583200640 1 --2- 172.21.15.107:0/1928758163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 0x5602f2372770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:49:49.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.021+0000 7f3581e00640 1 --2- 172.21.15.107:0/1928758163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f236d560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f35700303f0 tx=0x7f3570005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6add7a9d39c57e3e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:49.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.022+0000 7f356ea00640 1 -- 172.21.15.107:0/1928758163 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570040070 con 0x5602f1fab390 2024-07-26T12:49:49.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.022+0000 7f356ea00640 1 -- 172.21.15.107:0/1928758163 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f357004a070 con 0x5602f1fab390 2024-07-26T12:49:49.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.022+0000 7f356ea00640 1 -- 172.21.15.107:0/1928758163 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570037070 con 0x5602f1fab390 2024-07-26T12:49:49.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.023+0000 7f35900ca740 1 -- 172.21.15.107:0/1928758163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 msgr2=0x5602f236d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:49.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.023+0000 7f35900ca740 1 --2- 172.21.15.107:0/1928758163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f236d560 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f35700303f0 tx=0x7f3570005f10 comp rx=0 tx=0).stop 2024-07-26T12:49:49.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.023+0000 7f35900ca740 1 -- 172.21.15.107:0/1928758163 shutdown_connections 2024-07-26T12:49:49.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.023+0000 7f35900ca740 1 --2- 172.21.15.107:0/1928758163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fcc070 0x5602f2372770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.023+0000 7f35900ca740 1 --2- 172.21.15.107:0/1928758163 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f236d560 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.023+0000 7f35900ca740 1 --2- 172.21.15.107:0/1928758163 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fa31b0 0x5602f237cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.023+0000 7f35900ca740 1 -- 172.21.15.107:0/1928758163 >> 172.21.15.107:0/1928758163 conn(0x5602f1fc0460 msgr2=0x5602f2372060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:49.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.023+0000 7f35900ca740 1 -- 172.21.15.107:0/1928758163 shutdown_connections 2024-07-26T12:49:49.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.023+0000 7f35900ca740 1 -- 172.21.15.107:0/1928758163 wait complete. 2024-07-26T12:49:49.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.024+0000 7f35900ca740 1 -- start start 2024-07-26T12:49:49.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.024+0000 7f35900ca740 1 Processor -- start 2024-07-26T12:49:49.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.025+0000 7f35900ca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 0x5602f2382240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:49.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.025+0000 7f35900ca740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f23827e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:49.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.025+0000 7f35900ca740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 0x5602f23f7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:49.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.025+0000 7f35900ca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f35841b3e80 con 0x5602f1fa31b0 2024-07-26T12:49:49.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.025+0000 7f35900ca740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f35841b37b0 con 0x5602f1fcc070 2024-07-26T12:49:49.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.025+0000 7f35900ca740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f35841b09f0 con 0x5602f1fab390 2024-07-26T12:49:49.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.025+0000 7f35900ca740 10 client.0 _ll_register_callbacks cb 0x7f357c01bad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:49.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.025+0000 7f3581e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f23827e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:49.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.025+0000 7f3582800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 0x5602f2382240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:49.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.026+0000 7f3581e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f23827e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33184/0 (socket says 172.21.15.107:33184) 2024-07-26T12:49:49.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.026+0000 7f3581e00640 1 -- 172.21.15.107:0/884524473 learned_addr learned my addr 172.21.15.107:0/884524473 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:49.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.026+0000 7f3583200640 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 0x5602f23f7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:49.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.026+0000 7f3582800640 1 -- 172.21.15.107:0/884524473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 msgr2=0x5602f23827e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:49.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.026+0000 7f3582800640 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f23827e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.026+0000 7f3582800640 1 -- 172.21.15.107:0/884524473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 msgr2=0x5602f23f7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:49.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.026+0000 7f3582800640 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 0x5602f23f7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.026+0000 7f3582800640 1 -- 172.21.15.107:0/884524473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f23f7ef0 con 0x5602f1fa31b0 2024-07-26T12:49:49.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.026+0000 7f3581e00640 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f23827e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:49.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.027+0000 7f3582800640 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 0x5602f2382240 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f357400e9c0 tx=0x7f35740aae10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c57556ca228be8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:49.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.027+0000 7f3563e00640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:49:49.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.027+0000 7f3563e00640 1 -- 172.21.15.107:0/884524473 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35740e2000 con 0x5602f1fa31b0 2024-07-26T12:49:49.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.027+0000 7f3563e00640 1 -- 172.21.15.107:0/884524473 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f35740b3c60 con 0x5602f1fa31b0 2024-07-26T12:49:49.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.027+0000 7f35900ca740 1 -- 172.21.15.107:0/884524473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f357004a070 con 0x5602f1fa31b0 2024-07-26T12:49:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.028+0000 7f3563e00640 1 -- 172.21.15.107:0/884524473 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f35740b3e80 con 0x5602f1fa31b0 2024-07-26T12:49:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.028+0000 7f35900ca740 10 client.15507 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.028+0000 7f35900ca740 1 -- 172.21.15.107:0/884524473 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5602f23ff180 con 0x5602f1fa31b0 2024-07-26T12:49:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.028+0000 7f3563e00640 1 -- 172.21.15.107:0/884524473 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f357004a070 con 0x5602f1fa31b0 2024-07-26T12:49:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.029+0000 7f3563e00640 1 -- 172.21.15.107:0/884524473 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f35740a69d0 con 0x5602f1fa31b0 2024-07-26T12:49:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.029+0000 7f35900ca740 10 client.15507 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.029+0000 7f35900ca740 10 client.15507 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T12:49:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.029+0000 7f35900ca740 20 client.15507 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:49.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.029+0000 7f35900ca740 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x5602f1f5fda0 0x5602f2401940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.029+0000 7f35900ca740 4 client.15507 mds_command: new command op to 24694 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T12:49:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.029+0000 7f35900ca740 1 -- 172.21.15.107:0/884524473 --> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5602f22e55e0 con 0x5602f1f5fda0 2024-07-26T12:49:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.029+0000 7f3581e00640 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x5602f1f5fda0 0x5602f2401940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:49.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.030+0000 7f3581e00640 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x5602f1f5fda0 0x5602f2401940 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c096cb72d5b5bca2 server_cookie=ea9163f434308346 in_seq=0 out_seq=0 2024-07-26T12:49:49.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.030+0000 7f3563e00640 10 client.15507 ms_handle_connect on v2:172.21.15.132:6836/735918310 2024-07-26T12:49:49.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.031+0000 7f3563e00640 1 -- 172.21.15.107:0/884524473 <== mds.3 v2:172.21.15.132:6836/735918310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5602f22e55e0 con 0x5602f1f5fda0 2024-07-26T12:49:49.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.031+0000 7f3563e00640 10 client.15507 handle_command_reply: tid=0 2024-07-26T12:49:49.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.031+0000 7f35900ca740 1 client.15507 shutdown 2024-07-26T12:49:49.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.031+0000 7f35900ca740 1 -- 172.21.15.107:0/884524473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 msgr2=0x5602f2382240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:49.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.031+0000 7f35900ca740 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 0x5602f2382240 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f357400e9c0 tx=0x7f35740aae10 comp rx=0 tx=0).stop 2024-07-26T12:49:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.031+0000 7f35900ca740 1 -- 172.21.15.107:0/884524473 shutdown_connections 2024-07-26T12:49:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.032+0000 7f35900ca740 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x5602f1f5fda0 0x5602f2401940 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.032+0000 7f35900ca740 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5602f1fcc070 0x5602f23f7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.032+0000 7f35900ca740 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5602f1fab390 0x5602f23827e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.032+0000 7f35900ca740 1 --2- 172.21.15.107:0/884524473 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5602f1fa31b0 0x5602f2382240 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.032+0000 7f35900ca740 1 -- 172.21.15.107:0/884524473 >> 172.21.15.107:0/884524473 conn(0x5602f1fc0460 msgr2=0x5602f2372a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.032+0000 7f3563e00640 0 client.15507 ms_handle_reset on v2:172.21.15.132:6836/735918310 2024-07-26T12:49:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.032+0000 7f35900ca740 1 -- 172.21.15.107:0/884524473 shutdown_connections 2024-07-26T12:49:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.032+0000 7f35900ca740 1 -- 172.21.15.107:0/884524473 wait complete. 2024-07-26T12:49:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.032+0000 7f35900ca740 20 client.15507 trim_cache size 0 max 16384 2024-07-26T12:49:49.034 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-07-26T12:49:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.033+0000 7f3588a00640 1 -- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35680b3d40 msgr2=0x7f35680b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.033+0000 7f3588a00640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35680b3d40 0x7f35680b6250 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f357c005fd0 tx=0x7f357c004a40 comp rx=0 tx=0).stop 2024-07-26T12:49:49.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.033+0000 7f3588a00640 1 -- 172.21.15.107:0/1024858481 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3584075160 msgr2=0x7f358419c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:49.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.033+0000 7f3588a00640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3584075160 0x7f358419c000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f357400e9f0 tx=0x7f357400eeb0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.034+0000 7f3588a00640 1 -- 172.21.15.107:0/1024858481 shutdown_connections 2024-07-26T12:49:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.034+0000 7f3588a00640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f35680b3d40 0x7f35680b6250 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.034+0000 7f3588a00640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f358410f9c0 0x7f35841a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.034+0000 7f3588a00640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3584075ab0 0x7f358419c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.034+0000 7f3588a00640 1 --2- 172.21.15.107:0/1024858481 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3584075160 0x7f358419c000 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.034+0000 7f3588a00640 1 -- 172.21.15.107:0/1024858481 >> 172.21.15.107:0/1024858481 conn(0x7f35840fee30 msgr2=0x7f35841095a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.034+0000 7f3588a00640 1 -- 172.21.15.107:0/1024858481 shutdown_connections 2024-07-26T12:49:49.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:49.034+0000 7f3588a00640 1 -- 172.21.15.107:0/1024858481 wait complete. 2024-07-26T12:49:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:48 smithi112 ceph-mon[37501]: pgmap v505: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T12:49:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:48 smithi132 ceph-mon[37580]: pgmap v505: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T12:49:49.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T12:49:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:49:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:50 smithi112 ceph-mon[37501]: pgmap v506: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T12:49:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:50 smithi107 ceph-mon[32659]: pgmap v506: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T12:49:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:50 smithi132 ceph-mon[37580]: pgmap v506: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T12:49:51.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.821+0000 7f821a400640 1 -- 172.21.15.107:0/506322129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8214103340 msgr2=0x7f8214103750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:51.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.821+0000 7f821a400640 1 --2- 172.21.15.107:0/506322129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8214103340 0x7f8214103750 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8208009900 tx=0x7f820802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.821+0000 7f821a400640 1 -- 172.21.15.107:0/506322129 shutdown_connections 2024-07-26T12:49:51.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.821+0000 7f821a400640 1 --2- 172.21.15.107:0/506322129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82141095c0 0x7f821410fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.821+0000 7f821a400640 1 --2- 172.21.15.107:0/506322129 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8214108c70 0x7f8214109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.821+0000 7f821a400640 1 --2- 172.21.15.107:0/506322129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8214103340 0x7f8214103750 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.821+0000 7f821a400640 1 -- 172.21.15.107:0/506322129 >> 172.21.15.107:0/506322129 conn(0x7f82140fed60 msgr2=0x7f82141011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:51.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.821+0000 7f821a400640 1 -- 172.21.15.107:0/506322129 shutdown_connections 2024-07-26T12:49:51.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.821+0000 7f821a400640 1 -- 172.21.15.107:0/506322129 wait complete. 2024-07-26T12:49:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.823+0000 7f821a400640 1 Processor -- start 2024-07-26T12:49:51.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.823+0000 7f821a400640 1 -- start start 2024-07-26T12:49:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f821a400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8214103340 0x7f821419bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f821a400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8214108c70 0x7f821419c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f821a400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82141095c0 0x7f82141a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f821a400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8214113290 con 0x7f82141095c0 2024-07-26T12:49:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f821a400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f8214113110 con 0x7f8214108c70 2024-07-26T12:49:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f821a400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f8214113410 con 0x7f8214103340 2024-07-26T12:49:51.825 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f8219000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8214103340 0x7f821419bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f8219a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82141095c0 0x7f82141a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f8219000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8214103340 0x7f821419bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33200/0 (socket says 172.21.15.107:33200) 2024-07-26T12:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f8219000640 1 -- 172.21.15.107:0/1898135556 learned_addr learned my addr 172.21.15.107:0/1898135556 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:51.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.824+0000 7f8213e00640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8214108c70 0x7f821419c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.825+0000 7f8213e00640 1 -- 172.21.15.107:0/1898135556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8214103340 msgr2=0x7f821419bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.825+0000 7f8213e00640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8214103340 0x7f821419bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.825+0000 7f8213e00640 1 -- 172.21.15.107:0/1898135556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82141095c0 msgr2=0x7f82141a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.825+0000 7f8213e00640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82141095c0 0x7f82141a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.825+0000 7f8213e00640 1 -- 172.21.15.107:0/1898135556 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82141a1e00 con 0x7f8214108c70 2024-07-26T12:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.825+0000 7f8219000640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8214103340 0x7f821419bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.825+0000 7f8213e00640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8214108c70 0x7f821419c470 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f821406c0c0 tx=0x7f820000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:51.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.825+0000 7f8211600640 1 -- 172.21.15.107:0/1898135556 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820000cc40 con 0x7f8214108c70 2024-07-26T12:49:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.826+0000 7f8211600640 1 -- 172.21.15.107:0/1898135556 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f820000cde0 con 0x7f8214108c70 2024-07-26T12:49:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.826+0000 7f8211600640 1 -- 172.21.15.107:0/1898135556 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200004a00 con 0x7f8214108c70 2024-07-26T12:49:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.826+0000 7f821a400640 1 -- 172.21.15.107:0/1898135556 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82141a20f0 con 0x7f8214108c70 2024-07-26T12:49:51.828 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.826+0000 7f821a400640 1 -- 172.21.15.107:0/1898135556 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82141a8880 con 0x7f8214108c70 2024-07-26T12:49:51.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.831+0000 7f82229d0740 1 Processor -- start 2024-07-26T12:49:51.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.831+0000 7f8211600640 1 -- 172.21.15.107:0/1898135556 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f8200004ba0 con 0x7f8214108c70 2024-07-26T12:49:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.832+0000 7f82229d0740 1 -- start start 2024-07-26T12:49:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.832+0000 7f82229d0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836c91b0 0x556383a6cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.832+0000 7f82229d0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836a02f0 0x556383a6d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.832+0000 7f82229d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383a6d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.832+0000 7f82229d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5563839e1270 con 0x5563836a84d0 2024-07-26T12:49:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.832+0000 7f82229d0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5563839d0770 con 0x5563836c91b0 2024-07-26T12:49:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.832+0000 7f82229d0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5563839d47b0 con 0x5563836a02f0 2024-07-26T12:49:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.833+0000 7f8213e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383a6d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.833+0000 7f8219a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836c91b0 0x556383a6cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.833+0000 7f8219a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836c91b0 0x556383a6cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57598/0 (socket says 172.21.15.107:57598) 2024-07-26T12:49:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.833+0000 7f8219a00640 1 -- 172.21.15.107:0/3573167101 learned_addr learned my addr 172.21.15.107:0/3573167101 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:51.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.833+0000 7f8219a00640 1 -- 172.21.15.107:0/3573167101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836a02f0 msgr2=0x556383a6d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:51.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.833+0000 7f8219a00640 1 --2- 172.21.15.107:0/3573167101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836a02f0 0x556383a6d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.833+0000 7f8219a00640 1 -- 172.21.15.107:0/3573167101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 msgr2=0x556383a6d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.833+0000 7f8219a00640 1 --2- 172.21.15.107:0/3573167101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383a6d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.833+0000 7f8219a00640 1 -- 172.21.15.107:0/3573167101 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556383734790 con 0x5563836c91b0 2024-07-26T12:49:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.833+0000 7f8211600640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f81fc0b3d40 0x7f81fc0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.834+0000 7f8211600640 1 -- 172.21.15.107:0/1898135556 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f82000e4320 con 0x7f8214108c70 2024-07-26T12:49:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.834+0000 7f8219a00640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f81fc0b3d40 0x7f81fc0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.834+0000 7f8219a00640 1 --2- 172.21.15.107:0/3573167101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836c91b0 0x556383a6cc40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f820c009fd0 tx=0x7f820c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9df6ecb104b36e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.834+0000 7f8206000640 1 -- 172.21.15.107:0/3573167101 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c004250 con 0x5563836c91b0 2024-07-26T12:49:51.836 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.835+0000 7f8206000640 1 -- 172.21.15.107:0/3573167101 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f820c003db0 con 0x5563836c91b0 2024-07-26T12:49:51.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.835+0000 7f8206000640 1 -- 172.21.15.107:0/3573167101 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c01bb80 con 0x5563836c91b0 2024-07-26T12:49:51.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.836+0000 7f82229d0740 1 -- 172.21.15.107:0/3573167101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836c91b0 msgr2=0x556383a6cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:51.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.836+0000 7f8219a00640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f81fc0b3d40 0x7f81fc0b6250 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f820c020000 tx=0x7f820c021000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.836+0000 7f82229d0740 1 --2- 172.21.15.107:0/3573167101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836c91b0 0x556383a6cc40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f820c009fd0 tx=0x7f820c0118d0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.836+0000 7f82229d0740 1 -- 172.21.15.107:0/3573167101 shutdown_connections 2024-07-26T12:49:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.836+0000 7f82229d0740 1 --2- 172.21.15.107:0/3573167101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383a6d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.836+0000 7f82229d0740 1 --2- 172.21.15.107:0/3573167101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836a02f0 0x556383a6d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.837+0000 7f82229d0740 1 --2- 172.21.15.107:0/3573167101 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836c91b0 0x556383a6cc40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.837+0000 7f82229d0740 1 -- 172.21.15.107:0/3573167101 >> 172.21.15.107:0/3573167101 conn(0x5563836bd300 msgr2=0x556383a6ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.837+0000 7f82229d0740 1 -- 172.21.15.107:0/3573167101 shutdown_connections 2024-07-26T12:49:51.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.837+0000 7f82229d0740 1 -- 172.21.15.107:0/3573167101 wait complete. 2024-07-26T12:49:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.838+0000 7f82229d0740 1 -- start start 2024-07-26T12:49:51.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.838+0000 7f82229d0740 1 Processor -- start 2024-07-26T12:49:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.839+0000 7f82229d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a02f0 0x556383a78eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.839+0000 7f82229d0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a84d0 0x556383a79470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.839+0000 7f82229d0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836c91b0 0x556383a7e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.839+0000 7f82229d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55638390aab0 con 0x5563836a02f0 2024-07-26T12:49:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.839+0000 7f82229d0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55638396dd50 con 0x5563836a84d0 2024-07-26T12:49:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.839+0000 7f82229d0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556383a73320 con 0x5563836c91b0 2024-07-26T12:49:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f82229d0740 10 client.0 _ll_register_callbacks cb 0x5563836e0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f8219000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a02f0 0x556383a78eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f8219a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836c91b0 0x556383a7e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f8213e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a84d0 0x556383a79470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f8213e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a84d0 0x556383a79470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57610/0 (socket says 172.21.15.107:57610) 2024-07-26T12:49:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f8213e00640 1 -- 172.21.15.107:0/2039425575 learned_addr learned my addr 172.21.15.107:0/2039425575 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f8213e00640 1 -- 172.21.15.107:0/2039425575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836c91b0 msgr2=0x556383a7e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f8213e00640 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836c91b0 0x556383a7e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f8213e00640 1 -- 172.21.15.107:0/2039425575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a02f0 msgr2=0x556383a78eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f8213e00640 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a02f0 0x556383a78eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.840+0000 7f8213e00640 1 -- 172.21.15.107:0/2039425575 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563837ad620 con 0x5563836a84d0 2024-07-26T12:49:51.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.841+0000 7f8213e00640 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a84d0 0x556383a79470 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f82000a6bd0 tx=0x7f82000057b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4af5a5095e76e4bc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.841+0000 7f81faa00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:49:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.841+0000 7f81faa00640 1 -- 172.21.15.107:0/2039425575 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820001c070 con 0x5563836a84d0 2024-07-26T12:49:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.841+0000 7f81faa00640 1 -- 172.21.15.107:0/2039425575 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f82000aadb0 con 0x5563836a84d0 2024-07-26T12:49:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.841+0000 7f82229d0740 1 -- 172.21.15.107:0/2039425575 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f820c003db0 con 0x5563836a84d0 2024-07-26T12:49:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.841+0000 7f81faa00640 1 -- 172.21.15.107:0/2039425575 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82000aa4a0 con 0x5563836a84d0 2024-07-26T12:49:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.842+0000 7f82229d0740 10 client.25655 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.842+0000 7f82229d0740 1 -- 172.21.15.107:0/2039425575 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556383a98470 con 0x5563836a84d0 2024-07-26T12:49:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.842+0000 7f81faa00640 1 -- 172.21.15.107:0/2039425575 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f820c003db0 con 0x5563836a84d0 2024-07-26T12:49:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.842+0000 7f81faa00640 1 -- 172.21.15.107:0/2039425575 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f82000b3770 con 0x5563836a84d0 2024-07-26T12:49:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.842+0000 7f82229d0740 10 client.25655 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.842+0000 7f82229d0740 10 client.25655 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.842+0000 7f82229d0740 20 client.25655 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.842+0000 7f82229d0740 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556383687d80 0x556383a9b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.842+0000 7f82229d0740 4 client.25655 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:49:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.842+0000 7f82229d0740 1 -- 172.21.15.107:0/2039425575 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563839e31c0 con 0x556383687d80 2024-07-26T12:49:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.843+0000 7f8219000640 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556383687d80 0x556383a9b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.844+0000 7f8219000640 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556383687d80 0x556383a9b990 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f375025340073280 server_cookie=f9be76eb27ecb4e3 in_seq=0 out_seq=0 2024-07-26T12:49:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.844+0000 7f81faa00640 10 client.25655 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.844+0000 7f81faa00640 1 -- 172.21.15.107:0/2039425575 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5563839e31c0 con 0x556383687d80 2024-07-26T12:49:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.844+0000 7f81faa00640 10 client.25655 handle_command_reply: tid=0 2024-07-26T12:49:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.845+0000 7f82229d0740 1 client.25655 shutdown 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.845+0000 7f82229d0740 1 -- 172.21.15.107:0/2039425575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a84d0 msgr2=0x556383a79470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.845+0000 7f82229d0740 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a84d0 0x556383a79470 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f82000a6bd0 tx=0x7f82000057b0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.845+0000 7f82229d0740 1 -- 172.21.15.107:0/2039425575 shutdown_connections 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.845+0000 7f82229d0740 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556383687d80 0x556383a9b990 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.845+0000 7f81faa00640 0 client.25655 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.846+0000 7f82229d0740 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836c91b0 0x556383a7e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.846+0000 7f82229d0740 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a84d0 0x556383a79470 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.846+0000 7f82229d0740 1 --2- 172.21.15.107:0/2039425575 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a02f0 0x556383a78eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.846+0000 7f82229d0740 1 -- 172.21.15.107:0/2039425575 >> 172.21.15.107:0/2039425575 conn(0x5563836bd300 msgr2=0x556383a6fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.846+0000 7f82229d0740 1 -- 172.21.15.107:0/2039425575 shutdown_connections 2024-07-26T12:49:51.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.846+0000 7f82229d0740 1 -- 172.21.15.107:0/2039425575 wait complete. 2024-07-26T12:49:51.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.846+0000 7f82229d0740 20 client.25655 trim_cache size 0 max 16384 2024-07-26T12:49:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.874+0000 7f82229d0740 1 Processor -- start 2024-07-26T12:49:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.874+0000 7f82229d0740 1 -- start start 2024-07-26T12:49:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f82229d0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556383687d80 0x556383a73e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f82229d0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a02f0 0x556383a74350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f82229d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383a77500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f82229d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f82141ae570 con 0x5563836a84d0 2024-07-26T12:49:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f82229d0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f82141b00b0 con 0x5563836a02f0 2024-07-26T12:49:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f82229d0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f82141aec40 con 0x556383687d80 2024-07-26T12:49:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f8219000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556383687d80 0x556383a73e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f8213e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a02f0 0x556383a74350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f8219000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556383687d80 0x556383a73e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33232/0 (socket says 172.21.15.107:33232) 2024-07-26T12:49:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f8219000640 1 -- 172.21.15.107:0/4107361606 learned_addr learned my addr 172.21.15.107:0/4107361606 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.875+0000 7f8219a00640 1 --2- 172.21.15.107:0/4107361606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383a77500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.876+0000 7f8213e00640 1 -- 172.21.15.107:0/4107361606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556383687d80 msgr2=0x556383a73e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.876+0000 7f8213e00640 1 --2- 172.21.15.107:0/4107361606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556383687d80 0x556383a73e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.876+0000 7f8213e00640 1 -- 172.21.15.107:0/4107361606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 msgr2=0x556383a77500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.876+0000 7f8213e00640 1 --2- 172.21.15.107:0/4107361606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383a77500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.876+0000 7f8213e00640 1 -- 172.21.15.107:0/4107361606 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556383a78a20 con 0x5563836a02f0 2024-07-26T12:49:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.876+0000 7f8213e00640 1 --2- 172.21.15.107:0/4107361606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a02f0 0x556383a74350 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8200009230 tx=0x7f820000cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35b3d51f0c0b3258 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.876+0000 7f8205600640 1 -- 172.21.15.107:0/4107361606 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820001c070 con 0x5563836a02f0 2024-07-26T12:49:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.877+0000 7f8205600640 1 -- 172.21.15.107:0/4107361606 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82000aa660 con 0x5563836a02f0 2024-07-26T12:49:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.877+0000 7f8205600640 1 -- 172.21.15.107:0/4107361606 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82000e1b20 con 0x5563836a02f0 2024-07-26T12:49:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.878+0000 7f82229d0740 1 -- 172.21.15.107:0/4107361606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a02f0 msgr2=0x556383a74350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.878+0000 7f82229d0740 1 --2- 172.21.15.107:0/4107361606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a02f0 0x556383a74350 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8200009230 tx=0x7f820000cf90 comp rx=0 tx=0).stop 2024-07-26T12:49:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.878+0000 7f82229d0740 1 -- 172.21.15.107:0/4107361606 shutdown_connections 2024-07-26T12:49:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.878+0000 7f82229d0740 1 --2- 172.21.15.107:0/4107361606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383a77500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.878+0000 7f82229d0740 1 --2- 172.21.15.107:0/4107361606 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563836a02f0 0x556383a74350 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.878+0000 7f82229d0740 1 --2- 172.21.15.107:0/4107361606 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556383687d80 0x556383a73e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.878+0000 7f82229d0740 1 -- 172.21.15.107:0/4107361606 >> 172.21.15.107:0/4107361606 conn(0x5563836bd300 msgr2=0x556383a7e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.878+0000 7f82229d0740 1 -- 172.21.15.107:0/4107361606 shutdown_connections 2024-07-26T12:49:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.878+0000 7f82229d0740 1 -- 172.21.15.107:0/4107361606 wait complete. 2024-07-26T12:49:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.879+0000 7f82229d0740 1 -- start start 2024-07-26T12:49:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.879+0000 7f82229d0740 1 Processor -- start 2024-07-26T12:49:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f82229d0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556383687d80 0x556383aa02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f82229d0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836a02f0 0x556383aa07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f82229d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383aabe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f82229d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f82141af9e0 con 0x5563836a84d0 2024-07-26T12:49:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f82229d0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f82141af310 con 0x556383687d80 2024-07-26T12:49:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f82229d0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f82141ac570 con 0x5563836a02f0 2024-07-26T12:49:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f82229d0740 10 client.0 _ll_register_callbacks cb 0x7f82000e7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f8219000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556383687d80 0x556383aa02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f8213e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836a02f0 0x556383aa07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f8219000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556383687d80 0x556383aa02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57628/0 (socket says 172.21.15.107:57628) 2024-07-26T12:49:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f8219000640 1 -- 172.21.15.107:0/824422980 learned_addr learned my addr 172.21.15.107:0/824422980 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.880+0000 7f8219a00640 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383aabe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f8213e00640 1 -- 172.21.15.107:0/824422980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556383687d80 msgr2=0x556383aa02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f8213e00640 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556383687d80 0x556383aa02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f8213e00640 1 -- 172.21.15.107:0/824422980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 msgr2=0x556383aabe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f8213e00640 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383aabe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f8213e00640 1 -- 172.21.15.107:0/824422980 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556383a70d30 con 0x5563836a02f0 2024-07-26T12:49:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f8219000640 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556383687d80 0x556383aa02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:49:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f8213e00640 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836a02f0 0x556383aa07e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8200009230 tx=0x7f82000057b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=111d9afa98dd4f1e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f81faa00640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:49:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f81faa00640 1 -- 172.21.15.107:0/824422980 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820001c070 con 0x5563836a02f0 2024-07-26T12:49:51.884 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f81faa00640 1 -- 172.21.15.107:0/824422980 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f82000aafc0 con 0x5563836a02f0 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.881+0000 7f82229d0740 1 -- 172.21.15.107:0/824422980 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82000aa660 con 0x5563836a02f0 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f81faa00640 1 -- 172.21.15.107:0/824422980 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f82000a62e0 con 0x5563836a02f0 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f82229d0740 10 client.25672 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f82229d0740 1 -- 172.21.15.107:0/824422980 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556383a6eb80 con 0x5563836a02f0 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f81faa00640 1 -- 172.21.15.107:0/824422980 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82000aa660 con 0x5563836a02f0 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f81faa00640 1 -- 172.21.15.107:0/824422980 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f82000a64c0 con 0x5563836a02f0 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f82229d0740 10 client.25672 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f82229d0740 10 client.25672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f82229d0740 20 client.25672 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f82229d0740 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563836c91b0 0x556383aad400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f82229d0740 4 client.25672 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T12:49:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.882+0000 7f82229d0740 1 -- 172.21.15.107:0/824422980 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563839dec60 con 0x5563836c91b0 2024-07-26T12:49:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.883+0000 7f8219000640 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563836c91b0 0x556383aad400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.884+0000 7f8219000640 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563836c91b0 0x556383aad400 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96ce41a32337c859 server_cookie=8d58f707e222dcd8 in_seq=0 out_seq=0 2024-07-26T12:49:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.884+0000 7f81faa00640 10 client.25672 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:51.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.888+0000 7f81faa00640 1 -- 172.21.15.107:0/824422980 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563839dec60 con 0x5563836c91b0 2024-07-26T12:49:51.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.888+0000 7f81faa00640 10 client.25672 handle_command_reply: tid=0 2024-07-26T12:49:51.888 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.888+0000 7f82229d0740 1 client.25672 shutdown 2024-07-26T12:49:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.889+0000 7f82229d0740 1 -- 172.21.15.107:0/824422980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836a02f0 msgr2=0x556383aa07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.889+0000 7f82229d0740 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836a02f0 0x556383aa07e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8200009230 tx=0x7f82000057b0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.889+0000 7f82229d0740 1 -- 172.21.15.107:0/824422980 shutdown_connections 2024-07-26T12:49:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.889+0000 7f82229d0740 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563836c91b0 0x556383aad400 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.889+0000 7f82229d0740 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563836a84d0 0x556383aabe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.889+0000 7f82229d0740 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563836a02f0 0x556383aa07e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.889+0000 7f82229d0740 1 --2- 172.21.15.107:0/824422980 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556383687d80 0x556383aa02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.889+0000 7f82229d0740 1 -- 172.21.15.107:0/824422980 >> 172.21.15.107:0/824422980 conn(0x5563836bd300 msgr2=0x556383a6fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.889+0000 7f81faa00640 0 client.25672 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:49:51.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.890+0000 7f82229d0740 1 -- 172.21.15.107:0/824422980 shutdown_connections 2024-07-26T12:49:51.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.890+0000 7f82229d0740 1 -- 172.21.15.107:0/824422980 wait complete. 2024-07-26T12:49:51.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.890+0000 7f82229d0740 20 client.25672 trim_cache size 0 max 16384 2024-07-26T12:49:51.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-07-26T12:49:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-07-26T12:49:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-07-26T12:49:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-07-26T12:49:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.891+0000 7f821a400640 1 -- 172.21.15.107:0/1898135556 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f81fc0b3d40 msgr2=0x7f81fc0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.891+0000 7f821a400640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f81fc0b3d40 0x7f81fc0b6250 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f820c020000 tx=0x7f820c021000 comp rx=0 tx=0).stop 2024-07-26T12:49:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.891+0000 7f821a400640 1 -- 172.21.15.107:0/1898135556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8214108c70 msgr2=0x7f821419c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:51.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.891+0000 7f821a400640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8214108c70 0x7f821419c470 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f821406c0c0 tx=0x7f820000ee50 comp rx=0 tx=0).stop 2024-07-26T12:49:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.892+0000 7f821a400640 1 -- 172.21.15.107:0/1898135556 shutdown_connections 2024-07-26T12:49:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.892+0000 7f821a400640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f81fc0b3d40 0x7f81fc0b6250 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.892+0000 7f821a400640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82141095c0 0x7f82141a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.892+0000 7f821a400640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f8214108c70 0x7f821419c470 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.892+0000 7f821a400640 1 --2- 172.21.15.107:0/1898135556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8214103340 0x7f821419bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.892+0000 7f821a400640 1 -- 172.21.15.107:0/1898135556 >> 172.21.15.107:0/1898135556 conn(0x7f82140fed60 msgr2=0x7f821410dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.892+0000 7f821a400640 1 -- 172.21.15.107:0/1898135556 shutdown_connections 2024-07-26T12:49:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:49:51.892+0000 7f821a400640 1 -- 172.21.15.107:0/1898135556 wait complete. 2024-07-26T12:49:52.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.819+0000 7f3c09c00640 1 -- 172.21.15.107:0/790593426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c04075b30 msgr2=0x7f3c0410f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.819+0000 7f3c09c00640 1 --2- 172.21.15.107:0/790593426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c04075b30 0x7f3c0410f400 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0009900 tx=0x7f3bf002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.820+0000 7f3c09c00640 1 -- 172.21.15.107:0/790593426 shutdown_connections 2024-07-26T12:49:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.820+0000 7f3c09c00640 1 --2- 172.21.15.107:0/790593426 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c0410f940 0x7f3c04111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.820+0000 7f3c09c00640 1 --2- 172.21.15.107:0/790593426 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c04075b30 0x7f3c0410f400 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.820+0000 7f3c09c00640 1 --2- 172.21.15.107:0/790593426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c040751e0 0x7f3c040755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.820+0000 7f3c09c00640 1 -- 172.21.15.107:0/790593426 >> 172.21.15.107:0/790593426 conn(0x7f3c040fee70 msgr2=0x7f3c041012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.820+0000 7f3c09c00640 1 -- 172.21.15.107:0/790593426 shutdown_connections 2024-07-26T12:49:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.820+0000 7f3c09c00640 1 -- 172.21.15.107:0/790593426 wait complete. 2024-07-26T12:49:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.821+0000 7f3c09c00640 1 Processor -- start 2024-07-26T12:49:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.822+0000 7f3c09c00640 1 -- start start 2024-07-26T12:49:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.822+0000 7f3c09c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c040751e0 0x7f3c0419bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.822+0000 7f3c09c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04075b30 0x7f3c0419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.822+0000 7f3c09c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c0410f940 0x7f3c041a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.822+0000 7f3c09c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3c041154c0 con 0x7f3c04075b30 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.822+0000 7f3c09c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3c04115340 con 0x7f3c040751e0 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.822+0000 7f3c09c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3c04115640 con 0x7f3c0410f940 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.823+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c040751e0 0x7f3c0419bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.823+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c040751e0 0x7f3c0419bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57648/0 (socket says 172.21.15.107:57648) 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.823+0000 7f3c03e00640 1 -- 172.21.15.107:0/3960761354 learned_addr learned my addr 172.21.15.107:0/3960761354 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.823+0000 7f3c03400640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04075b30 0x7f3c0419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.823+0000 7f3c09200640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c0410f940 0x7f3c041a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.823+0000 7f3c03e00640 1 -- 172.21.15.107:0/3960761354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c0410f940 msgr2=0x7f3c041a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:52.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.823+0000 7f3c03e00640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c0410f940 0x7f3c041a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.823+0000 7f3c03e00640 1 -- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04075b30 msgr2=0x7f3c0419c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:52.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.824+0000 7f3c03e00640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04075b30 0x7f3c0419c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.824+0000 7f3c03e00640 1 -- 172.21.15.107:0/3960761354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c041a1ec0 con 0x7f3c040751e0 2024-07-26T12:49:52.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.824+0000 7f3c09200640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c0410f940 0x7f3c041a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:49:52.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.824+0000 7f3c03400640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04075b30 0x7f3c0419c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:49:52.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.824+0000 7f3c03e00640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c040751e0 0x7f3c0419bf50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400e9f0 tx=0x7f3bf400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.824+0000 7f3c00c00640 1 -- 172.21.15.107:0/3960761354 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf400cd80 con 0x7f3c040751e0 2024-07-26T12:49:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.825+0000 7f3c00c00640 1 -- 172.21.15.107:0/3960761354 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf40108e0 con 0x7f3c040751e0 2024-07-26T12:49:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.825+0000 7f3c09c00640 1 -- 172.21.15.107:0/3960761354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c041a21b0 con 0x7f3c040751e0 2024-07-26T12:49:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.825+0000 7f3c00c00640 1 -- 172.21.15.107:0/3960761354 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4018860 con 0x7f3c040751e0 2024-07-26T12:49:52.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.825+0000 7f3c09c00640 1 -- 172.21.15.107:0/3960761354 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c041a8910 con 0x7f3c040751e0 2024-07-26T12:49:52.830 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.830+0000 7f3c11ddc740 1 Processor -- start 2024-07-26T12:49:52.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.830+0000 7f3c11ddc740 1 -- start start 2024-07-26T12:49:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.830+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67818010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.830+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b6744a1b0 0x557b67818550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.830+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 0x557b67818a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.830+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557b6778ebd0 con 0x557b67452390 2024-07-26T12:49:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.830+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557b6777e0d0 con 0x557b67473070 2024-07-26T12:49:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.830+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557b67782110 con 0x557b6744a1b0 2024-07-26T12:49:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.831+0000 7f3c09200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67818010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.831+0000 7f3c09200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67818010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57660/0 (socket says 172.21.15.107:57660) 2024-07-26T12:49:52.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.831+0000 7f3c09200640 1 -- 172.21.15.107:0/3530367542 learned_addr learned my addr 172.21.15.107:0/3530367542 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.831+0000 7f3c03400640 1 --2- 172.21.15.107:0/3530367542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 0x557b67818a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.832+0000 7f3c09200640 1 -- 172.21.15.107:0/3530367542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b6744a1b0 msgr2=0x557b67818550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T12:49:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.832+0000 7f3c09200640 1 --2- 172.21.15.107:0/3530367542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b6744a1b0 0x557b67818550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.832+0000 7f3c09200640 1 -- 172.21.15.107:0/3530367542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 msgr2=0x557b67818a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.832+0000 7f3c09200640 1 --2- 172.21.15.107:0/3530367542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 0x557b67818a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.832+0000 7f3c09200640 1 -- 172.21.15.107:0/3530367542 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b674de650 con 0x557b67473070 2024-07-26T12:49:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.832+0000 7f3c03400640 1 --2- 172.21.15.107:0/3530367542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 0x557b67818a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:49:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.832+0000 7f3c00c00640 1 -- 172.21.15.107:0/3960761354 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f3bf4010430 con 0x7f3c040751e0 2024-07-26T12:49:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.833+0000 7f3c09200640 1 --2- 172.21.15.107:0/3530367542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67818010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009fd0 tx=0x7f3bfc0108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcbc9ba7922fcbf2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.833+0000 7f3bf9600640 1 -- 172.21.15.107:0/3530367542 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc011760 con 0x557b67473070 2024-07-26T12:49:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.833+0000 7f3bf9600640 1 -- 172.21.15.107:0/3530367542 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bfc011db0 con 0x557b67473070 2024-07-26T12:49:52.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.833+0000 7f3bf9600640 1 -- 172.21.15.107:0/3530367542 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc01a380 con 0x557b67473070 2024-07-26T12:49:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.834+0000 7f3c11ddc740 1 -- 172.21.15.107:0/3530367542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 msgr2=0x557b67818010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:52.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.834+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/3530367542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67818010 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009fd0 tx=0x7f3bfc0108e0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.835+0000 7f3c11ddc740 1 -- 172.21.15.107:0/3530367542 shutdown_connections 2024-07-26T12:49:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.835+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/3530367542 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 0x557b67818a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.835+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/3530367542 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b6744a1b0 0x557b67818550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.835+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/3530367542 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67818010 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.835+0000 7f3c11ddc740 1 -- 172.21.15.107:0/3530367542 >> 172.21.15.107:0/3530367542 conn(0x557b67467460 msgr2=0x557b67815f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.835+0000 7f3c11ddc740 1 -- 172.21.15.107:0/3530367542 shutdown_connections 2024-07-26T12:49:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.835+0000 7f3c00c00640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3be80b3cd0 0x7f3be80b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.835+0000 7f3c11ddc740 1 -- 172.21.15.107:0/3530367542 wait complete. 2024-07-26T12:49:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.835+0000 7f3c00c00640 1 -- 172.21.15.107:0/3960761354 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3bf40e47b0 con 0x7f3c040751e0 2024-07-26T12:49:52.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.836+0000 7f3c11ddc740 1 -- start start 2024-07-26T12:49:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.836+0000 7f3c11ddc740 1 Processor -- start 2024-07-26T12:49:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.836+0000 7f3c03400640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3be80b3cd0 0x7f3be80b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.837+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b6744a1b0 0x557b67823ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.837+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67452390 0x557b67824470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.838 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.837+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b678294f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.837+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557b675e1700 con 0x557b6744a1b0 2024-07-26T12:49:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.837+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x557b6771a6a0 con 0x557b67473070 2024-07-26T12:49:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.837+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x557b6781e6c0 con 0x557b67452390 2024-07-26T12:49:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.837+0000 7f3c11ddc740 10 client.0 _ll_register_callbacks cb 0x557b6772a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.838+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b6744a1b0 0x557b67823ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.838+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b6744a1b0 0x557b67823ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:50036/0 (socket says 172.21.15.107:50036) 2024-07-26T12:49:52.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.838+0000 7f3c03e00640 1 -- 172.21.15.107:0/1038832929 learned_addr learned my addr 172.21.15.107:0/1038832929 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.838+0000 7f3c09200640 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b678294f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.839+0000 7f3c03e00640 1 -- 172.21.15.107:0/1038832929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67452390 msgr2=0x557b67824470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T12:49:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.839+0000 7f3c03e00640 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67452390 0x557b67824470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.839+0000 7f3c03e00640 1 -- 172.21.15.107:0/1038832929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 msgr2=0x557b678294f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.839+0000 7f3c03e00640 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b678294f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.839+0000 7f3c03e00640 1 -- 172.21.15.107:0/1038832929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b67731840 con 0x557b6744a1b0 2024-07-26T12:49:52.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.839+0000 7f3c03400640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3be80b3cd0 0x7f3be80b61e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0000c00 tx=0x7f3bf00058b0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.840+0000 7f3c03e00640 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b6744a1b0 0x557b67823ed0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4000c00 tx=0x7f3bf40aaaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da80fb0929c1ddde server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.840+0000 7f3bee000640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:49:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.840+0000 7f3bee000640 1 -- 172.21.15.107:0/1038832929 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf40af5e0 con 0x557b6744a1b0 2024-07-26T12:49:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.840+0000 7f3bee000640 1 -- 172.21.15.107:0/1038832929 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3bf40e43b0 con 0x557b6744a1b0 2024-07-26T12:49:52.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.840+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1038832929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bfc011db0 con 0x557b6744a1b0 2024-07-26T12:49:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.841+0000 7f3bee000640 1 -- 172.21.15.107:0/1038832929 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3bf40e45b0 con 0x557b6744a1b0 2024-07-26T12:49:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.841+0000 7f3c11ddc740 10 client.15537 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:52.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.841+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1038832929 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557b67843400 con 0x557b6744a1b0 2024-07-26T12:49:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.841+0000 7f3bee000640 1 -- 172.21.15.107:0/1038832929 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bfc011db0 con 0x557b6744a1b0 2024-07-26T12:49:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.841+0000 7f3bee000640 1 -- 172.21.15.107:0/1038832929 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3bf40a6950 con 0x557b6744a1b0 2024-07-26T12:49:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.842+0000 7f3c11ddc740 10 client.15537 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.842+0000 7f3c11ddc740 10 client.15537 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T12:49:52.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.842+0000 7f3c11ddc740 20 client.15537 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:52.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.842+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x557b674319f0 0x557b67846940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.842+0000 7f3c11ddc740 4 client.15537 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:49:52.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.842+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1038832929 --> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b67790b20 con 0x557b674319f0 2024-07-26T12:49:52.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.843+0000 7f3c03400640 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x557b674319f0 0x557b67846940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.844+0000 7f3c03400640 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x557b674319f0 0x557b67846940 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e4d48df6ea19e99c server_cookie=a7ef0551dad2f781 in_seq=0 out_seq=0 2024-07-26T12:49:52.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.844+0000 7f3bee000640 10 client.15537 ms_handle_connect on v2:172.21.15.132:6838/3489644052 2024-07-26T12:49:52.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.845+0000 7f3bee000640 1 -- 172.21.15.107:0/1038832929 <== mds.2 v2:172.21.15.132:6838/3489644052 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557b67790b20 con 0x557b674319f0 2024-07-26T12:49:52.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.845+0000 7f3bee000640 10 client.15537 handle_command_reply: tid=0 2024-07-26T12:49:52.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.845+0000 7f3c11ddc740 1 client.15537 shutdown 2024-07-26T12:49:52.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.845+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1038832929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b6744a1b0 msgr2=0x557b67823ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:52.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.845+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b6744a1b0 0x557b67823ed0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4000c00 tx=0x7f3bf40aaaf0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.846+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1038832929 shutdown_connections 2024-07-26T12:49:52.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.846+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x557b674319f0 0x557b67846940 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.846+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b678294f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.846+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67452390 0x557b67824470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.846+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1038832929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b6744a1b0 0x557b67823ed0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.846+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1038832929 >> 172.21.15.107:0/1038832929 conn(0x557b67467460 msgr2=0x557b67814540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:52.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.846+0000 7f3bee000640 0 client.15537 ms_handle_reset on v2:172.21.15.132:6838/3489644052 2024-07-26T12:49:52.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.846+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1038832929 shutdown_connections 2024-07-26T12:49:52.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.846+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1038832929 wait complete. 2024-07-26T12:49:52.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.847+0000 7f3c11ddc740 20 client.15537 trim_cache size 0 max 16384 2024-07-26T12:49:52.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.874+0000 7f3c11ddc740 1 Processor -- start 2024-07-26T12:49:52.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.875+0000 7f3c11ddc740 1 -- start start 2024-07-26T12:49:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.875+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 0x557b67829460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.875+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67452390 0x557b6784c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.875+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67473070 0x557b6784c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.875+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3c041ae590 con 0x557b67473070 2024-07-26T12:49:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.875+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3c041b00d0 con 0x557b67452390 2024-07-26T12:49:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.875+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3c041aec60 con 0x557b67467460 2024-07-26T12:49:52.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.876+0000 7f3c03400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67452390 0x557b6784c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.876+0000 7f3c03400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67452390 0x557b6784c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57670/0 (socket says 172.21.15.107:57670) 2024-07-26T12:49:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.876+0000 7f3c03400640 1 -- 172.21.15.107:0/112283224 learned_addr learned my addr 172.21.15.107:0/112283224 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.876+0000 7f3c03e00640 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 0x557b67829460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.876+0000 7f3c03400640 1 -- 172.21.15.107:0/112283224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 msgr2=0x557b67829460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.876+0000 7f3c03400640 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 0x557b67829460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.876+0000 7f3c03400640 1 -- 172.21.15.107:0/112283224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67473070 msgr2=0x557b6784c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.876+0000 7f3c09200640 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67473070 0x557b6784c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.877+0000 7f3c03400640 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67473070 0x557b6784c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.877+0000 7f3c03400640 1 -- 172.21.15.107:0/112283224 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b6781ae20 con 0x557b67452390 2024-07-26T12:49:52.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.877+0000 7f3c03e00640 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 0x557b67829460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:49:52.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.877+0000 7f3c09200640 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67473070 0x557b6784c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:49:52.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.878+0000 7f3c03400640 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67452390 0x557b6784c100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0005940 tx=0x7f3bf003e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9a7018b4013f283 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:52.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.878+0000 7f3bf8c00640 1 -- 172.21.15.107:0/112283224 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0035040 con 0x557b67452390 2024-07-26T12:49:52.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.878+0000 7f3bf8c00640 1 -- 172.21.15.107:0/112283224 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf0055070 con 0x557b67452390 2024-07-26T12:49:52.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.878+0000 7f3bf8c00640 1 -- 172.21.15.107:0/112283224 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0003be0 con 0x557b67452390 2024-07-26T12:49:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.879+0000 7f3c11ddc740 1 -- 172.21.15.107:0/112283224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67452390 msgr2=0x557b6784c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.879+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67452390 0x557b6784c100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0005940 tx=0x7f3bf003e000 comp rx=0 tx=0).stop 2024-07-26T12:49:52.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.880+0000 7f3c11ddc740 1 -- 172.21.15.107:0/112283224 shutdown_connections 2024-07-26T12:49:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.880+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67473070 0x557b6784c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.880+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67452390 0x557b6784c100 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.880+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/112283224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 0x557b67829460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.880+0000 7f3c11ddc740 1 -- 172.21.15.107:0/112283224 >> 172.21.15.107:0/112283224 conn(0x557b67846c90 msgr2=0x557b67822570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.880+0000 7f3c11ddc740 1 -- 172.21.15.107:0/112283224 shutdown_connections 2024-07-26T12:49:52.881 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.880+0000 7f3c11ddc740 1 -- 172.21.15.107:0/112283224 wait complete. 2024-07-26T12:49:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.880+0000 7f3c11ddc740 1 -- start start 2024-07-26T12:49:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.881+0000 7f3c11ddc740 1 Processor -- start 2024-07-26T12:49:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.881+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 0x557b67858290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.881+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 0x557b67853790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.881+0000 7f3c11ddc740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67853cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.882+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3c041afa00 con 0x557b67452390 2024-07-26T12:49:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.882+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3c041af330 con 0x557b67473070 2024-07-26T12:49:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.882+0000 7f3c11ddc740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3c041ac590 con 0x557b67467460 2024-07-26T12:49:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.882+0000 7f3c11ddc740 10 client.0 _ll_register_callbacks cb 0x7f3bf40aa7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.882+0000 7f3c03400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 0x557b67853790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.882+0000 7f3c09200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67853cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.882+0000 7f3c09200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67853cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57686/0 (socket says 172.21.15.107:57686) 2024-07-26T12:49:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.882+0000 7f3c09200640 1 -- 172.21.15.107:0/1771132628 learned_addr learned my addr 172.21.15.107:0/1771132628 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.882+0000 7f3c03e00640 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 0x557b67858290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.882+0000 7f3c03400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 0x557b67853790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33284/0 (socket says 172.21.15.107:33284) 2024-07-26T12:49:52.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.883+0000 7f3c09200640 1 -- 172.21.15.107:0/1771132628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 msgr2=0x557b67853790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:52.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.883+0000 7f3c09200640 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 0x557b67853790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.883+0000 7f3c09200640 1 -- 172.21.15.107:0/1771132628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 msgr2=0x557b67858290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:52.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.883+0000 7f3c09200640 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 0x557b67858290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.883+0000 7f3c09200640 1 -- 172.21.15.107:0/1771132628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b67854210 con 0x557b67473070 2024-07-26T12:49:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.883+0000 7f3c09200640 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67853cd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc006010 tx=0x7f3bfc004f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8050fcdf08d184a0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.884+0000 7f3bee000640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:49:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.884+0000 7f3bee000640 1 -- 172.21.15.107:0/1771132628 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc02c020 con 0x557b67473070 2024-07-26T12:49:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.884+0000 7f3bee000640 1 -- 172.21.15.107:0/1771132628 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3bfc011560 con 0x557b67473070 2024-07-26T12:49:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.884+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1771132628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bf0055070 con 0x557b67473070 2024-07-26T12:49:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.884+0000 7f3bee000640 1 -- 172.21.15.107:0/1771132628 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3bfc034050 con 0x557b67473070 2024-07-26T12:49:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.884+0000 7f3c11ddc740 10 client.25679 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.884+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1771132628 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557b678aeb80 con 0x557b67473070 2024-07-26T12:49:52.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.885+0000 7f3bee000640 1 -- 172.21.15.107:0/1771132628 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bf0055070 con 0x557b67473070 2024-07-26T12:49:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.885+0000 7f3bee000640 1 -- 172.21.15.107:0/1771132628 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3bfc028720 con 0x557b67473070 2024-07-26T12:49:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.886+0000 7f3c11ddc740 10 client.25679 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.886+0000 7f3c11ddc740 10 client.25679 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T12:49:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.886+0000 7f3c11ddc740 20 client.25679 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.886+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x557b67406da0 0x557b678b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.886+0000 7f3c11ddc740 4 client.25679 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T12:49:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.886+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1771132628 --> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b6778c5c0 con 0x557b67406da0 2024-07-26T12:49:52.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.886+0000 7f3c03e00640 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x557b67406da0 0x557b678b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.887+0000 7f3c03e00640 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x557b67406da0 0x557b678b10c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=32bd66c3165634b6 server_cookie=9033b5f5a51198ec in_seq=0 out_seq=0 2024-07-26T12:49:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.887+0000 7f3bee000640 10 client.25679 ms_handle_connect on v2:172.21.15.132:6838/3489644052 2024-07-26T12:49:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.887+0000 7f3bee000640 1 -- 172.21.15.107:0/1771132628 <== mds.2 v2:172.21.15.132:6838/3489644052 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b6778c5c0 con 0x557b67406da0 2024-07-26T12:49:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.887+0000 7f3bee000640 10 client.25679 handle_command_reply: tid=0 2024-07-26T12:49:52.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.887+0000 7f3c11ddc740 1 client.25679 shutdown 2024-07-26T12:49:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.888+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1771132628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 msgr2=0x557b67853cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.888+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67853cd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc006010 tx=0x7f3bfc004f90 comp rx=0 tx=0).stop 2024-07-26T12:49:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.888+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1771132628 shutdown_connections 2024-07-26T12:49:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.888+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x557b67406da0 0x557b678b10c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.888+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x557b67473070 0x557b67853cd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.888+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x557b67467460 0x557b67853790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.888+0000 7f3c11ddc740 1 --2- 172.21.15.107:0/1771132628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b67452390 0x557b67858290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.888+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1771132628 >> 172.21.15.107:0/1771132628 conn(0x557b67846c90 msgr2=0x557b6781e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.888+0000 7f3bee000640 0 client.25679 ms_handle_reset on v2:172.21.15.132:6838/3489644052 2024-07-26T12:49:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.888+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1771132628 shutdown_connections 2024-07-26T12:49:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.889+0000 7f3c11ddc740 1 -- 172.21.15.107:0/1771132628 wait complete. 2024-07-26T12:49:52.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.889+0000 7f3c11ddc740 20 client.25679 trim_cache size 0 max 16384 2024-07-26T12:49:52.891 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-07-26T12:49:52.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.890+0000 7f3c09c00640 1 -- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3be80b3cd0 msgr2=0x7f3be80b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:52.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.890+0000 7f3c09c00640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3be80b3cd0 0x7f3be80b61e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0000c00 tx=0x7f3bf00058b0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.890+0000 7f3c09c00640 1 -- 172.21.15.107:0/3960761354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c040751e0 msgr2=0x7f3c0419bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:52.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.890+0000 7f3c09c00640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c040751e0 0x7f3c0419bf50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400e9f0 tx=0x7f3bf400eeb0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.890+0000 7f3c09c00640 1 -- 172.21.15.107:0/3960761354 shutdown_connections 2024-07-26T12:49:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.890+0000 7f3c09c00640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f3be80b3cd0 0x7f3be80b61e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.890+0000 7f3c09c00640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3c0410f940 0x7f3c041a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.890+0000 7f3c09c00640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3c04075b30 0x7f3c0419c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.890+0000 7f3c09c00640 1 --2- 172.21.15.107:0/3960761354 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f3c040751e0 0x7f3c0419bf50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.890+0000 7f3c09c00640 1 -- 172.21.15.107:0/3960761354 >> 172.21.15.107:0/3960761354 conn(0x7f3c040fee70 msgr2=0x7f3c04109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.891+0000 7f3c09c00640 1 -- 172.21.15.107:0/3960761354 shutdown_connections 2024-07-26T12:49:52.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:52.891+0000 7f3c09c00640 1 -- 172.21.15.107:0/3960761354 wait complete. 2024-07-26T12:49:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:52 smithi112 ceph-mon[37501]: pgmap v507: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-07-26T12:49:53.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:52 smithi107 ceph-mon[32659]: pgmap v507: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-07-26T12:49:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:52 smithi132 ceph-mon[37580]: pgmap v507: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-07-26T12:49:53.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T12:49:54.293 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:49:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:54 smithi112 ceph-mon[37501]: pgmap v508: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:49:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:54 smithi107 ceph-mon[32659]: pgmap v508: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:49:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:54 smithi132 ceph-mon[37580]: pgmap v508: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:49:56.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.657+0000 7ff119c00640 1 -- 172.21.15.107:0/3643527128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff114109670 msgr2=0x7ff11410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:56.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.657+0000 7ff119c00640 1 --2- 172.21.15.107:0/3643527128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff114109670 0x7ff11410fea0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff10c009ca0 tx=0x7ff10c02a8c0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.658+0000 7ff119c00640 1 -- 172.21.15.107:0/3643527128 shutdown_connections 2024-07-26T12:49:56.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.658+0000 7ff119c00640 1 --2- 172.21.15.107:0/3643527128 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff114109670 0x7ff11410fea0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.658+0000 7ff119c00640 1 --2- 172.21.15.107:0/3643527128 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff114108d20 0x7ff114109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.658+0000 7ff119c00640 1 --2- 172.21.15.107:0/3643527128 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1141033e0 0x7ff1141037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.658+0000 7ff119c00640 1 -- 172.21.15.107:0/3643527128 >> 172.21.15.107:0/3643527128 conn(0x7ff1140fee70 msgr2=0x7ff1141012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:56.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.658+0000 7ff119c00640 1 -- 172.21.15.107:0/3643527128 shutdown_connections 2024-07-26T12:49:56.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.658+0000 7ff119c00640 1 -- 172.21.15.107:0/3643527128 wait complete. 2024-07-26T12:49:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.659+0000 7ff119c00640 1 Processor -- start 2024-07-26T12:49:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.659+0000 7ff119c00640 1 -- start start 2024-07-26T12:49:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.660+0000 7ff119c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1141033e0 0x7ff114074870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.660+0000 7ff119c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff114108d20 0x7ff11406db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.660+0000 7ff119c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff114109670 0x7ff11406e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.660+0000 7ff119c00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff114113340 con 0x7ff114108d20 2024-07-26T12:49:56.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.660+0000 7ff119c00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1141131c0 con 0x7ff1141033e0 2024-07-26T12:49:56.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.660+0000 7ff119c00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1141134c0 con 0x7ff114109670 2024-07-26T12:49:56.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.661+0000 7ff113400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff114108d20 0x7ff11406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.661+0000 7ff113400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff114108d20 0x7ff11406db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37842/0 (socket says 172.21.15.107:37842) 2024-07-26T12:49:56.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.661+0000 7ff113400640 1 -- 172.21.15.107:0/883732911 learned_addr learned my addr 172.21.15.107:0/883732911 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:56.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.661+0000 7ff113e00640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1141033e0 0x7ff114074870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.661+0000 7ff119200640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff114109670 0x7ff11406e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.661+0000 7ff113400640 1 -- 172.21.15.107:0/883732911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff114109670 msgr2=0x7ff11406e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:56.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.661+0000 7ff113400640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff114109670 0x7ff11406e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.661+0000 7ff113400640 1 -- 172.21.15.107:0/883732911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1141033e0 msgr2=0x7ff114074870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:56.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.661+0000 7ff113400640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1141033e0 0x7ff114074870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.661+0000 7ff113400640 1 -- 172.21.15.107:0/883732911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff11406e8c0 con 0x7ff114108d20 2024-07-26T12:49:56.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.662+0000 7ff113400640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff114108d20 0x7ff11406db40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff10000e9c0 tx=0x7ff10000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:56.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.662+0000 7ff110c00640 1 -- 172.21.15.107:0/883732911 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10000ccc0 con 0x7ff114108d20 2024-07-26T12:49:56.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.662+0000 7ff110c00640 1 -- 172.21.15.107:0/883732911 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1000054d0 con 0x7ff114108d20 2024-07-26T12:49:56.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.662+0000 7ff119c00640 1 -- 172.21.15.107:0/883732911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff114073540 con 0x7ff114108d20 2024-07-26T12:49:56.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.662+0000 7ff119c00640 1 -- 172.21.15.107:0/883732911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff114073a80 con 0x7ff114108d20 2024-07-26T12:49:56.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.663+0000 7ff110c00640 1 -- 172.21.15.107:0/883732911 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff100010430 con 0x7ff114108d20 2024-07-26T12:49:56.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.667+0000 7ff110c00640 1 -- 172.21.15.107:0/883732911 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7ff1000105d0 con 0x7ff114108d20 2024-07-26T12:49:56.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.667+0000 7ff121ddd740 1 Processor -- start 2024-07-26T12:49:56.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.667+0000 7ff121ddd740 1 -- start start 2024-07-26T12:49:56.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.668+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 0x55c59eae4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.668+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eae4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.668+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e71e390 0x55c59eae4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.668+0000 7ff121ddd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c59ea5abd0 con 0x55c59e7161b0 2024-07-26T12:49:56.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.668+0000 7ff121ddd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c59ea4a0d0 con 0x55c59e71e390 2024-07-26T12:49:56.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.668+0000 7ff121ddd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c59ea4e110 con 0x55c59e73f070 2024-07-26T12:49:56.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.669+0000 7ff119200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 0x55c59eae4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.669+0000 7ff113e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eae4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.669+0000 7ff119200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 0x55c59eae4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:35342/0 (socket says 172.21.15.107:35342) 2024-07-26T12:49:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.669+0000 7ff119200640 1 -- 172.21.15.107:0/546876439 learned_addr learned my addr 172.21.15.107:0/546876439 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.669+0000 7ff113400640 1 --2- 172.21.15.107:0/546876439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e71e390 0x55c59eae4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.669+0000 7ff110c00640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0f80b3dc0 0x7ff0f80b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.670+0000 7ff110c00640 1 -- 172.21.15.107:0/883732911 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff1000e43b0 con 0x7ff114108d20 2024-07-26T12:49:56.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.670+0000 7ff113e00640 1 -- 172.21.15.107:0/546876439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 msgr2=0x55c59eae4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:56.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.670+0000 7ff113e00640 1 --2- 172.21.15.107:0/546876439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 0x55c59eae4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.670+0000 7ff113e00640 1 -- 172.21.15.107:0/546876439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e71e390 msgr2=0x55c59eae4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:56.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.670+0000 7ff113e00640 1 --2- 172.21.15.107:0/546876439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e71e390 0x55c59eae4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.670+0000 7ff113e00640 1 -- 172.21.15.107:0/546876439 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c59e7aa650 con 0x55c59e7161b0 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.670+0000 7ff113400640 1 --2- 172.21.15.107:0/546876439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e71e390 0x55c59eae4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.670+0000 7ff113e00640 1 --2- 172.21.15.107:0/546876439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eae4550 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff104005fd0 tx=0x7ff1040107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfd69d1a4692632 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.670+0000 7ff119200640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0f80b3dc0 0x7ff0f80b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.671+0000 7ff109600640 1 -- 172.21.15.107:0/546876439 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104004220 con 0x55c59e7161b0 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.671+0000 7ff109600640 1 -- 172.21.15.107:0/546876439 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff10400be20 con 0x55c59e7161b0 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.671+0000 7ff109600640 1 -- 172.21.15.107:0/546876439 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104004220 con 0x55c59e7161b0 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.672+0000 7ff121ddd740 1 -- 172.21.15.107:0/546876439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 msgr2=0x55c59eae4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.672+0000 7ff121ddd740 1 --2- 172.21.15.107:0/546876439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eae4550 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff104005fd0 tx=0x7ff1040107a0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.672+0000 7ff121ddd740 1 -- 172.21.15.107:0/546876439 shutdown_connections 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.672+0000 7ff121ddd740 1 --2- 172.21.15.107:0/546876439 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e71e390 0x55c59eae4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.672+0000 7ff121ddd740 1 --2- 172.21.15.107:0/546876439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eae4550 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.672+0000 7ff121ddd740 1 --2- 172.21.15.107:0/546876439 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 0x55c59eae4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.672+0000 7ff121ddd740 1 -- 172.21.15.107:0/546876439 >> 172.21.15.107:0/546876439 conn(0x55c59e733460 msgr2=0x55c59eae1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:56.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.673+0000 7ff119200640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0f80b3dc0 0x7ff0f80b62d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff10c02f500 tx=0x7ff10c02a8c0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:56.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.673+0000 7ff121ddd740 1 -- 172.21.15.107:0/546876439 shutdown_connections 2024-07-26T12:49:56.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.673+0000 7ff121ddd740 1 -- 172.21.15.107:0/546876439 wait complete. 2024-07-26T12:49:56.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.673+0000 7ff121ddd740 1 -- start start 2024-07-26T12:49:56.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.674+0000 7ff121ddd740 1 Processor -- start 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.674+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eaf7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.674+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eaf8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.674+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e73f070 0x55c59eaf8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.675+0000 7ff121ddd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c59e8ad700 con 0x55c59e7161b0 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.675+0000 7ff121ddd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55c59e9e66a0 con 0x55c59e73f070 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.675+0000 7ff121ddd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55c59eafaf20 con 0x55c59e71e390 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.675+0000 7ff121ddd740 10 client.0 _ll_register_callbacks cb 0x55c59e9f6150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.675+0000 7ff119200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e73f070 0x55c59eaf8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.675+0000 7ff119200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e73f070 0x55c59eaf8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:50618/0 (socket says 172.21.15.107:50618) 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.675+0000 7ff119200640 1 -- 172.21.15.107:0/3392613760 learned_addr learned my addr 172.21.15.107:0/3392613760 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.675+0000 7ff113400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eaf8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.675+0000 7ff113e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eaf7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.676+0000 7ff113400640 1 -- 172.21.15.107:0/3392613760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e73f070 msgr2=0x55c59eaf8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:56.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.676+0000 7ff113400640 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e73f070 0x55c59eaf8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.676+0000 7ff113400640 1 -- 172.21.15.107:0/3392613760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 msgr2=0x55c59eaf7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:56.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.676+0000 7ff113400640 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eaf7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.676+0000 7ff113400640 1 -- 172.21.15.107:0/3392613760 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c59e9fd840 con 0x55c59e71e390 2024-07-26T12:49:56.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.676+0000 7ff113400640 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eaf8220 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff1000e1720 tx=0x7ff10000cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbb6e7f2db9332fb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.676+0000 7ff0fe000640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.676+0000 7ff0fe000640 1 -- 172.21.15.107:0/3392613760 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff100005680 con 0x55c59e71e390 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.677+0000 7ff0fe000640 1 -- 172.21.15.107:0/3392613760 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff1000922c0 con 0x55c59e71e390 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.677+0000 7ff121ddd740 1 -- 172.21.15.107:0/3392613760 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff10400be20 con 0x55c59e71e390 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.677+0000 7ff0fe000640 1 -- 172.21.15.107:0/3392613760 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff1000ab2e0 con 0x55c59e71e390 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.677+0000 7ff121ddd740 10 client.25696 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.677+0000 7ff121ddd740 1 -- 172.21.15.107:0/3392613760 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c59eb06ab0 con 0x55c59e71e390 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.677+0000 7ff0fe000640 1 -- 172.21.15.107:0/3392613760 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff10400be20 con 0x55c59e71e390 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.678+0000 7ff0fe000640 1 -- 172.21.15.107:0/3392613760 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff1000b3800 con 0x55c59e71e390 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.678+0000 7ff121ddd740 10 client.25696 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.678+0000 7ff121ddd740 10 client.25696 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.678+0000 7ff121ddd740 20 client.25696 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.678+0000 7ff121ddd740 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x55c59e6fd9f0 0x55c59eb0a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.678+0000 7ff121ddd740 4 client.25696 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:49:56.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.678+0000 7ff121ddd740 1 -- 172.21.15.107:0/3392613760 --> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c59ea5cb20 con 0x55c59e6fd9f0 2024-07-26T12:49:56.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.678+0000 7ff113e00640 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x55c59e6fd9f0 0x55c59eb0a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.679+0000 7ff113e00640 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x55c59e6fd9f0 0x55c59eb0a070 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8fbefeb1498cd2cd server_cookie=104c8df2e2c4b39d in_seq=0 out_seq=0 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.679+0000 7ff0fe000640 10 client.25696 ms_handle_connect on v2:172.21.15.107:6838/137526780 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.680+0000 7ff0fe000640 1 -- 172.21.15.107:0/3392613760 <== mds.1 v2:172.21.15.107:6838/137526780 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c59ea5cb20 con 0x55c59e6fd9f0 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.680+0000 7ff0fe000640 10 client.25696 handle_command_reply: tid=0 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.680+0000 7ff121ddd740 1 client.25696 shutdown 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.680+0000 7ff121ddd740 1 -- 172.21.15.107:0/3392613760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 msgr2=0x55c59eaf8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.680+0000 7ff121ddd740 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eaf8220 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff1000e1720 tx=0x7ff10000cf20 comp rx=0 tx=0).stop 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.681+0000 7ff121ddd740 1 -- 172.21.15.107:0/3392613760 shutdown_connections 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.681+0000 7ff121ddd740 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x55c59e6fd9f0 0x55c59eb0a070 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.681+0000 7ff121ddd740 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e73f070 0x55c59eaf8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.681+0000 7ff121ddd740 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eaf8220 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.681+0000 7ff121ddd740 1 --2- 172.21.15.107:0/3392613760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eaf7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.681+0000 7ff121ddd740 1 -- 172.21.15.107:0/3392613760 >> 172.21.15.107:0/3392613760 conn(0x55c59e733460 msgr2=0x55c59eae0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:56.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.681+0000 7ff0fe000640 0 client.25696 ms_handle_reset on v2:172.21.15.107:6838/137526780 2024-07-26T12:49:56.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.681+0000 7ff121ddd740 1 -- 172.21.15.107:0/3392613760 shutdown_connections 2024-07-26T12:49:56.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.681+0000 7ff121ddd740 1 -- 172.21.15.107:0/3392613760 wait complete. 2024-07-26T12:49:56.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.681+0000 7ff121ddd740 20 client.25696 trim_cache size 0 max 16384 2024-07-26T12:49:56.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.695+0000 7ff121ddd740 1 Processor -- start 2024-07-26T12:49:56.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.696+0000 7ff121ddd740 1 -- start start 2024-07-26T12:49:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.696+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eaf9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.696+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eae5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.696+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e73f070 0x55c59eb0f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.696+0000 7ff121ddd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f400ff00 con 0x55c59e7161b0 2024-07-26T12:49:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.696+0000 7ff121ddd740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f4011a40 con 0x55c59e73f070 2024-07-26T12:49:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.696+0000 7ff121ddd740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f40105d0 con 0x55c59e71e390 2024-07-26T12:49:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.697+0000 7ff113e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eaf9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.697+0000 7ff113400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eae5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.697+0000 7ff113400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eae5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:35348/0 (socket says 172.21.15.107:35348) 2024-07-26T12:49:56.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.697+0000 7ff113e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eaf9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37880/0 (socket says 172.21.15.107:37880) 2024-07-26T12:49:56.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.697+0000 7ff113400640 1 -- 172.21.15.107:0/1496565227 learned_addr learned my addr 172.21.15.107:0/1496565227 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:56.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.697+0000 7ff113400640 1 -- 172.21.15.107:0/1496565227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e73f070 msgr2=0x55c59eb0f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:56.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.697+0000 7ff113400640 1 --2- 172.21.15.107:0/1496565227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e73f070 0x55c59eb0f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.697+0000 7ff113400640 1 -- 172.21.15.107:0/1496565227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 msgr2=0x55c59eaf9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:56.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.697+0000 7ff113400640 1 --2- 172.21.15.107:0/1496565227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eaf9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.697+0000 7ff113400640 1 -- 172.21.15.107:0/1496565227 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c59eaf8d80 con 0x55c59e71e390 2024-07-26T12:49:56.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.698+0000 7ff113400640 1 --2- 172.21.15.107:0/1496565227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eae5750 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff1000aa8e0 tx=0x7ff1000e4630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7039dc820ccb2e9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:56.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.698+0000 7ff108c00640 1 -- 172.21.15.107:0/1496565227 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1000cb030 con 0x55c59e71e390 2024-07-26T12:49:56.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.698+0000 7ff108c00640 1 -- 172.21.15.107:0/1496565227 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff100005680 con 0x55c59e71e390 2024-07-26T12:49:56.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.698+0000 7ff108c00640 1 -- 172.21.15.107:0/1496565227 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1000b3800 con 0x55c59e71e390 2024-07-26T12:49:56.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.699+0000 7ff121ddd740 1 -- 172.21.15.107:0/1496565227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 msgr2=0x55c59eae5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:56.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.699+0000 7ff121ddd740 1 --2- 172.21.15.107:0/1496565227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eae5750 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff1000aa8e0 tx=0x7ff1000e4630 comp rx=0 tx=0).stop 2024-07-26T12:49:56.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.699+0000 7ff121ddd740 1 -- 172.21.15.107:0/1496565227 shutdown_connections 2024-07-26T12:49:56.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.699+0000 7ff121ddd740 1 --2- 172.21.15.107:0/1496565227 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e73f070 0x55c59eb0f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.699+0000 7ff121ddd740 1 --2- 172.21.15.107:0/1496565227 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e71e390 0x55c59eae5750 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.699+0000 7ff121ddd740 1 --2- 172.21.15.107:0/1496565227 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e7161b0 0x55c59eaf9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.699+0000 7ff121ddd740 1 -- 172.21.15.107:0/1496565227 >> 172.21.15.107:0/1496565227 conn(0x55c59e733460 msgr2=0x55c59eae5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:56.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.699+0000 7ff121ddd740 1 -- 172.21.15.107:0/1496565227 shutdown_connections 2024-07-26T12:49:56.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.699+0000 7ff121ddd740 1 -- 172.21.15.107:0/1496565227 wait complete. 2024-07-26T12:49:56.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.700+0000 7ff121ddd740 1 -- start start 2024-07-26T12:49:56.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.700+0000 7ff121ddd740 1 Processor -- start 2024-07-26T12:49:56.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.701+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e7161b0 0x55c59eafc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.701+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e71e390 0x55c59eafce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.701+0000 7ff121ddd740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 0x55c59eb71800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.701+0000 7ff113e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e7161b0 0x55c59eafc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.701+0000 7ff113e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e7161b0 0x55c59eafc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:50644/0 (socket says 172.21.15.107:50644) 2024-07-26T12:49:56.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.701+0000 7ff113e00640 1 -- 172.21.15.107:0/2098445477 learned_addr learned my addr 172.21.15.107:0/2098445477 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:49:56.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.701+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f4011370 con 0x55c59e71e390 2024-07-26T12:49:56.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.701+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f4010ca0 con 0x55c59e7161b0 2024-07-26T12:49:56.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.701+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f400dee0 con 0x55c59e73f070 2024-07-26T12:49:56.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.701+0000 7ff121ddd740 10 client.0 _ll_register_callbacks cb 0x7ff1000aa370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:49:56.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.702+0000 7ff119200640 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 0x55c59eb71800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.702+0000 7ff113400640 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e71e390 0x55c59eafce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.702+0000 7ff113e00640 1 -- 172.21.15.107:0/2098445477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 msgr2=0x55c59eb71800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:56.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.702+0000 7ff113e00640 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 0x55c59eb71800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.702+0000 7ff113e00640 1 -- 172.21.15.107:0/2098445477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e71e390 msgr2=0x55c59eafce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:49:56.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.702+0000 7ff113e00640 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e71e390 0x55c59eafce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.702+0000 7ff113e00640 1 -- 172.21.15.107:0/2098445477 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c59eb71e10 con 0x55c59e7161b0 2024-07-26T12:49:56.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.702+0000 7ff113400640 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e71e390 0x55c59eafce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:49:56.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.703+0000 7ff113e00640 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e7161b0 0x55c59eafc900 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff104003f10 tx=0x7ff10400bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a534512706a920ec server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:49:56.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.703+0000 7ff0fe000640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:49:56.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.703+0000 7ff0fe000640 1 -- 172.21.15.107:0/2098445477 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104034060 con 0x55c59e7161b0 2024-07-26T12:49:56.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.704+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff100005680 con 0x55c59e7161b0 2024-07-26T12:49:56.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.704+0000 7ff0fe000640 1 -- 172.21.15.107:0/2098445477 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff1040035f0 con 0x55c59e7161b0 2024-07-26T12:49:56.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.704+0000 7ff0fe000640 1 -- 172.21.15.107:0/2098445477 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff10403b050 con 0x55c59e7161b0 2024-07-26T12:49:56.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.704+0000 7ff121ddd740 10 client.25691 fetch_fsmap learned FSMap version 29 2024-07-26T12:49:56.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.704+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c59eb73090 con 0x55c59e7161b0 2024-07-26T12:49:56.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.704+0000 7ff0fe000640 1 -- 172.21.15.107:0/2098445477 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff100005680 con 0x55c59e7161b0 2024-07-26T12:49:56.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.705+0000 7ff0fe000640 1 -- 172.21.15.107:0/2098445477 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7ff1040119e0 con 0x55c59e7161b0 2024-07-26T12:49:56.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.705+0000 7ff121ddd740 10 client.25691 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:49:56.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.705+0000 7ff121ddd740 10 client.25691 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-07-26T12:49:56.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.705+0000 7ff121ddd740 20 client.25691 populate_metadata read hostname 'smithi107' 2024-07-26T12:49:56.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.705+0000 7ff121ddd740 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x55c59e6d2da0 0x55c59eb756c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:49:56.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.706+0000 7ff121ddd740 4 client.25691 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T12:49:56.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.706+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 --> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c59ea585c0 con 0x55c59e6d2da0 2024-07-26T12:49:56.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.706+0000 7ff113400640 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x55c59e6d2da0 0x55c59eb756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:49:56.708 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.707+0000 7ff113400640 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x55c59e6d2da0 0x55c59eb756c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc8b8aa6884ca9ee server_cookie=55a53b50cf1ef6dd in_seq=0 out_seq=0 2024-07-26T12:49:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.707+0000 7ff0fe000640 10 client.25691 ms_handle_connect on v2:172.21.15.107:6838/137526780 2024-07-26T12:49:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.707+0000 7ff0fe000640 1 -- 172.21.15.107:0/2098445477 <== mds.1 v2:172.21.15.107:6838/137526780 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c59ea585c0 con 0x55c59e6d2da0 2024-07-26T12:49:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.707+0000 7ff0fe000640 10 client.25691 handle_command_reply: tid=0 2024-07-26T12:49:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.707+0000 7ff121ddd740 1 client.25691 shutdown 2024-07-26T12:49:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.708+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e7161b0 msgr2=0x55c59eafc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.708+0000 7ff121ddd740 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e7161b0 0x55c59eafc900 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff104003f10 tx=0x7ff10400bef0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.708+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 shutdown_connections 2024-07-26T12:49:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.708+0000 7ff121ddd740 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x55c59e6d2da0 0x55c59eb756c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.708+0000 7ff121ddd740 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55c59e73f070 0x55c59eb71800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.708+0000 7ff121ddd740 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c59e71e390 0x55c59eafce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.708+0000 7ff121ddd740 1 --2- 172.21.15.107:0/2098445477 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55c59e7161b0 0x55c59eafc900 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.708+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 >> 172.21.15.107:0/2098445477 conn(0x55c59e733460 msgr2=0x55c59eaf9920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.708+0000 7ff0fe000640 0 client.25691 ms_handle_reset on v2:172.21.15.107:6838/137526780 2024-07-26T12:49:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.709+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 shutdown_connections 2024-07-26T12:49:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.709+0000 7ff121ddd740 1 -- 172.21.15.107:0/2098445477 wait complete. 2024-07-26T12:49:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.709+0000 7ff121ddd740 20 client.25691 trim_cache size 0 max 16384 2024-07-26T12:49:56.711 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-07-26T12:49:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 -- 172.21.15.107:0/883732911 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0f80b3dc0 msgr2=0x7ff0f80b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:56.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0f80b3dc0 0x7ff0f80b62d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff10c02f500 tx=0x7ff10c02a8c0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 -- 172.21.15.107:0/883732911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff114108d20 msgr2=0x7ff11406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:49:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff114108d20 0x7ff11406db40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff10000e9c0 tx=0x7ff10000ee80 comp rx=0 tx=0).stop 2024-07-26T12:49:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 -- 172.21.15.107:0/883732911 shutdown_connections 2024-07-26T12:49:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0f80b3dc0 0x7ff0f80b62d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff114109670 0x7ff11406e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff114108d20 0x7ff11406db40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 --2- 172.21.15.107:0/883732911 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff1141033e0 0x7ff114074870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:49:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 -- 172.21.15.107:0/883732911 >> 172.21.15.107:0/883732911 conn(0x7ff1140fee70 msgr2=0x7ff114101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:49:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 -- 172.21.15.107:0/883732911 shutdown_connections 2024-07-26T12:49:56.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:49:56.710+0000 7ff119c00640 1 -- 172.21.15.107:0/883732911 wait complete. 2024-07-26T12:49:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:56 smithi107 ceph-mon[32659]: pgmap v509: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:49:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:56 smithi112 ceph-mon[37501]: pgmap v509: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:49:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:56 smithi132 ceph-mon[37580]: pgmap v509: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:49:57.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T12:49:58.109 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:49:59.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:49:58 smithi107 ceph-mon[32659]: pgmap v510: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T12:49:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:49:58 smithi112 ceph-mon[37501]: pgmap v510: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T12:49:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:49:58 smithi132 ceph-mon[37580]: pgmap v510: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T12:50:00.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:00 smithi107 ceph-mon[32659]: overall HEALTH_OK 2024-07-26T12:50:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:00 smithi132 ceph-mon[37580]: overall HEALTH_OK 2024-07-26T12:50:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.419+0000 7fabc3a00640 1 -- 172.21.15.107:0/4293236833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabbc076db0 msgr2=0x7fabbc0751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.419+0000 7fabc3a00640 1 --2- 172.21.15.107:0/4293236833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabbc076db0 0x7fabbc0751e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fabb0009900 tx=0x7fabb002a410 comp rx=0 tx=0).stop 2024-07-26T12:50:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.419+0000 7fabc3a00640 1 -- 172.21.15.107:0/4293236833 shutdown_connections 2024-07-26T12:50:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.419+0000 7fabc3a00640 1 --2- 172.21.15.107:0/4293236833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fabbc113e70 0x7fabbc116340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.419+0000 7fabc3a00640 1 --2- 172.21.15.107:0/4293236833 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabbc075800 0x7fabbc075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.419+0000 7fabc3a00640 1 --2- 172.21.15.107:0/4293236833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabbc076db0 0x7fabbc0751e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.419+0000 7fabc3a00640 1 -- 172.21.15.107:0/4293236833 >> 172.21.15.107:0/4293236833 conn(0x7fabbc0ff010 msgr2=0x7fabbc101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.419+0000 7fabc3a00640 1 -- 172.21.15.107:0/4293236833 shutdown_connections 2024-07-26T12:50:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.419+0000 7fabc3a00640 1 -- 172.21.15.107:0/4293236833 wait complete. 2024-07-26T12:50:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.421+0000 7fabc3a00640 1 Processor -- start 2024-07-26T12:50:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.421+0000 7fabc3a00640 1 -- start start 2024-07-26T12:50:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.421+0000 7fabc3a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabbc075800 0x7fabbc1a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.421+0000 7fabc3a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabbc076db0 0x7fabbc1a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.421+0000 7fabc3a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fabbc113e70 0x7fabbc1a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.421+0000 7fabc3a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fabbc119ab0 con 0x7fabbc076db0 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.421+0000 7fabc3a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fabbc119930 con 0x7fabbc075800 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.421+0000 7fabc3a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fabbc119c30 con 0x7fabbc113e70 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabc1000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabbc075800 0x7fabbc1a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabc1000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabbc075800 0x7fabbc1a04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:50670/0 (socket says 172.21.15.107:50670) 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabbbe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabbc076db0 0x7fabbc1a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabc1000640 1 -- 172.21.15.107:0/2987762211 learned_addr learned my addr 172.21.15.107:0/2987762211 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabc1a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fabbc113e70 0x7fabbc1a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabc1a00640 1 -- 172.21.15.107:0/2987762211 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabbc075800 msgr2=0x7fabbc1a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabc1a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabbc075800 0x7fabbc1a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabc1a00640 1 -- 172.21.15.107:0/2987762211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabbc076db0 msgr2=0x7fabbc1a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:00.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabc1a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabbc076db0 0x7fabbc1a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabc1a00640 1 -- 172.21.15.107:0/2987762211 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabbc1a6450 con 0x7fabbc113e70 2024-07-26T12:50:00.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.422+0000 7fabbbe00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabbc076db0 0x7fabbc1a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:00.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.423+0000 7fabc1a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fabbc113e70 0x7fabbc1a5d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009fd0 tx=0x7fabb400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:00.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.423+0000 7fabb9600640 1 -- 172.21.15.107:0/2987762211 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb400ce20 con 0x7fabbc113e70 2024-07-26T12:50:00.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.423+0000 7fabb9600640 1 -- 172.21.15.107:0/2987762211 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb4005020 con 0x7fabbc113e70 2024-07-26T12:50:00.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.423+0000 7fabc3a00640 1 -- 172.21.15.107:0/2987762211 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabbc1a6740 con 0x7fabbc113e70 2024-07-26T12:50:00.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.423+0000 7fabc3a00640 1 -- 172.21.15.107:0/2987762211 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabbc1acea0 con 0x7fabbc113e70 2024-07-26T12:50:00.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.424+0000 7fabb9600640 1 -- 172.21.15.107:0/2987762211 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb4010690 con 0x7fabbc113e70 2024-07-26T12:50:00.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.427+0000 7fabb9600640 1 -- 172.21.15.107:0/2987762211 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fabb4004a40 con 0x7fabbc113e70 2024-07-26T12:50:00.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.428+0000 7fabca826740 1 Processor -- start 2024-07-26T12:50:00.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.428+0000 7fabca826740 1 -- start start 2024-07-26T12:50:00.429 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabca826740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbe6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabca826740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbe7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabca826740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 0x56130dbe78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabca826740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56130db5dbf0 con 0x56130d8191b0 2024-07-26T12:50:00.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabca826740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56130db4d0f0 con 0x56130d842070 2024-07-26T12:50:00.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabca826740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56130db51130 con 0x56130d821390 2024-07-26T12:50:00.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabbbe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbe6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabbbe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbe6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:50684/0 (socket says 172.21.15.107:50684) 2024-07-26T12:50:00.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabbbe00640 1 -- 172.21.15.107:0/297301984 learned_addr learned my addr 172.21.15.107:0/297301984 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:00.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabbbe00640 1 --2- 172.21.15.107:0/297301984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 0x56130dbe78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabc1000640 1 --2- 172.21.15.107:0/297301984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbe7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabbbe00640 1 -- 172.21.15.107:0/297301984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 msgr2=0x56130dbe78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:00.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabbbe00640 1 --2- 172.21.15.107:0/297301984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 0x56130dbe78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabbbe00640 1 -- 172.21.15.107:0/297301984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 msgr2=0x56130dbe7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:00.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabbbe00640 1 --2- 172.21.15.107:0/297301984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbe7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabb9600640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faba40b3dc0 0x7faba40b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.429+0000 7fabbbe00640 1 -- 172.21.15.107:0/297301984 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56130d8ad650 con 0x56130d842070 2024-07-26T12:50:00.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.430+0000 7fabc1000640 1 --2- 172.21.15.107:0/297301984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbe7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:50:00.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.430+0000 7fabb9600640 1 -- 172.21.15.107:0/2987762211 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fabb40e48d0 con 0x7fabbc113e70 2024-07-26T12:50:00.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.430+0000 7fabc1a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faba40b3dc0 0x7faba40b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.430+0000 7fabbbe00640 1 --2- 172.21.15.107:0/297301984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbe6e50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faba8013950 tx=0x7faba8005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b407972bec8ffca server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:00.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.430+0000 7fabae000640 1 -- 172.21.15.107:0/297301984 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8004030 con 0x56130d842070 2024-07-26T12:50:00.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.431+0000 7fabae000640 1 -- 172.21.15.107:0/297301984 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faba800f940 con 0x56130d842070 2024-07-26T12:50:00.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.431+0000 7fabae000640 1 -- 172.21.15.107:0/297301984 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8003ac0 con 0x56130d842070 2024-07-26T12:50:00.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.432+0000 7fabca826740 1 -- 172.21.15.107:0/297301984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 msgr2=0x56130dbe6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.432+0000 7fabc1a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faba40b3dc0 0x7faba40b62d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fabb40a8370 tx=0x7fabb40a6fc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.432+0000 7fabca826740 1 --2- 172.21.15.107:0/297301984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbe6e50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faba8013950 tx=0x7faba8005850 comp rx=0 tx=0).stop 2024-07-26T12:50:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.432+0000 7fabca826740 1 -- 172.21.15.107:0/297301984 shutdown_connections 2024-07-26T12:50:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.432+0000 7fabca826740 1 --2- 172.21.15.107:0/297301984 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 0x56130dbe78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.432+0000 7fabca826740 1 --2- 172.21.15.107:0/297301984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbe7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.432+0000 7fabca826740 1 --2- 172.21.15.107:0/297301984 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbe6e50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.432+0000 7fabca826740 1 -- 172.21.15.107:0/297301984 >> 172.21.15.107:0/297301984 conn(0x56130d836460 msgr2=0x56130dbe4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.432+0000 7fabca826740 1 -- 172.21.15.107:0/297301984 shutdown_connections 2024-07-26T12:50:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.432+0000 7fabca826740 1 -- 172.21.15.107:0/297301984 wait complete. 2024-07-26T12:50:00.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.433+0000 7fabca826740 1 -- start start 2024-07-26T12:50:00.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.433+0000 7fabca826740 1 Processor -- start 2024-07-26T12:50:00.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabca826740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbf24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabca826740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 0x56130dbf2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabca826740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbf2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabca826740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56130d9b0700 con 0x56130d8191b0 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabca826740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56130dae96c0 con 0x56130d842070 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabca826740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56130dbf5630 con 0x56130d821390 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabca826740 10 client.0 _ll_register_callbacks cb 0x56130daf9170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabc1000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 0x56130dbf2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabc1000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 0x56130dbf2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:35392/0 (socket says 172.21.15.107:35392) 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabc1000640 1 -- 172.21.15.107:0/3213137002 learned_addr learned my addr 172.21.15.107:0/3213137002 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabbbe00640 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbf24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabbbe00640 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbf2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.434+0000 7fabc1000640 1 -- 172.21.15.107:0/3213137002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 msgr2=0x56130dbf2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.435+0000 7fabc1000640 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbf2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.435+0000 7fabc1000640 1 -- 172.21.15.107:0/3213137002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 msgr2=0x56130dbf24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.435+0000 7fabc1000640 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbf24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.435+0000 7fabc1000640 1 -- 172.21.15.107:0/3213137002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56130db00860 con 0x56130d821390 2024-07-26T12:50:00.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.435+0000 7fabbbe00640 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbf2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:50:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.435+0000 7fabc1000640 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 0x56130dbf2a10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fabb0006010 tx=0x7fabb0031d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88312b9d878a8b45 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.435+0000 7faba2a00640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:50:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.435+0000 7faba2a00640 1 -- 172.21.15.107:0/3213137002 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0040070 con 0x56130d821390 2024-07-26T12:50:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.435+0000 7fabca826740 1 -- 172.21.15.107:0/3213137002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faba800f940 con 0x56130d821390 2024-07-26T12:50:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.435+0000 7faba2a00640 1 -- 172.21.15.107:0/3213137002 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fabb00318b0 con 0x56130d821390 2024-07-26T12:50:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7faba2a00640 1 -- 172.21.15.107:0/3213137002 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fabb0035040 con 0x56130d821390 2024-07-26T12:50:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7fabca826740 10 client.25714 fetch_fsmap learned FSMap version 29 2024-07-26T12:50:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7fabca826740 1 -- 172.21.15.107:0/3213137002 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56130dc01280 con 0x56130d821390 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7faba2a00640 1 -- 172.21.15.107:0/3213137002 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faba800f940 con 0x56130d821390 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7faba2a00640 1 -- 172.21.15.107:0/3213137002 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fabb00047a0 con 0x56130d821390 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7fabca826740 10 client.25714 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7fabca826740 10 client.25714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7fabca826740 20 client.25714 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7fabca826740 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56130d8009f0 0x56130dc04750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7fabca826740 4 client.25714 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.436+0000 7fabca826740 1 -- 172.21.15.107:0/3213137002 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56130db5fb40 con 0x56130d8009f0 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.437+0000 7fabbbe00640 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56130d8009f0 0x56130dc04750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.437+0000 7fabbbe00640 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56130d8009f0 0x56130dc04750 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0f384a5234778fb server_cookie=2a6b30181c98ddbf in_seq=0 out_seq=0 2024-07-26T12:50:00.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.437+0000 7faba2a00640 10 client.25714 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.438+0000 7faba2a00640 1 -- 172.21.15.107:0/3213137002 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56130db5fb40 con 0x56130d8009f0 2024-07-26T12:50:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.438+0000 7faba2a00640 10 client.25714 handle_command_reply: tid=0 2024-07-26T12:50:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.438+0000 7fabca826740 1 client.25714 shutdown 2024-07-26T12:50:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.438+0000 7fabca826740 1 -- 172.21.15.107:0/3213137002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 msgr2=0x56130dbf2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:00.440 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7fabca826740 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 0x56130dbf2a10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fabb0006010 tx=0x7fabb0031d40 comp rx=0 tx=0).stop 2024-07-26T12:50:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7fabca826740 1 -- 172.21.15.107:0/3213137002 shutdown_connections 2024-07-26T12:50:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7fabca826740 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56130d8009f0 0x56130dc04750 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7fabca826740 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d842070 0x56130dbf2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7fabca826740 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d821390 0x56130dbf2a10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7fabca826740 1 --2- 172.21.15.107:0/3213137002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbf24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7faba2a00640 0 client.25714 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7fabca826740 1 -- 172.21.15.107:0/3213137002 >> 172.21.15.107:0/3213137002 conn(0x56130d836460 msgr2=0x56130dbe9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7fabca826740 1 -- 172.21.15.107:0/3213137002 shutdown_connections 2024-07-26T12:50:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7fabca826740 1 -- 172.21.15.107:0/3213137002 wait complete. 2024-07-26T12:50:00.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.439+0000 7fabca826740 20 client.25714 trim_cache size 0 max 16384 2024-07-26T12:50:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:00 smithi112 ceph-mon[37501]: overall HEALTH_OK 2024-07-26T12:50:00.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.453+0000 7fabca826740 1 Processor -- start 2024-07-26T12:50:00.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.453+0000 7fabca826740 1 -- start start 2024-07-26T12:50:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.453+0000 7fabca826740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbedc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.453+0000 7fabca826740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.453+0000 7fabca826740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 0x56130dbebee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.453+0000 7fabca826740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c00ff20 con 0x56130d8191b0 2024-07-26T12:50:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.453+0000 7fabca826740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c011a60 con 0x56130d821390 2024-07-26T12:50:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.453+0000 7fabca826740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c0105f0 con 0x56130d842070 2024-07-26T12:50:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabc1000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabbbe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbedc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabc1000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:50704/0 (socket says 172.21.15.107:50704) 2024-07-26T12:50:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabc1000640 1 -- 172.21.15.107:0/2403809853 learned_addr learned my addr 172.21.15.107:0/2403809853 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:00.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabbbe00640 1 --2- 172.21.15.107:0/2403809853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 0x56130dbebee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabc1000640 1 -- 172.21.15.107:0/2403809853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 msgr2=0x56130dbebee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabc1000640 1 --2- 172.21.15.107:0/2403809853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 0x56130dbebee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabc1000640 1 -- 172.21.15.107:0/2403809853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 msgr2=0x56130dbedc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabc1000640 1 --2- 172.21.15.107:0/2403809853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbedc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabc1000640 1 -- 172.21.15.107:0/2403809853 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56130dbf3790 con 0x56130d821390 2024-07-26T12:50:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.454+0000 7fabbbe00640 1 --2- 172.21.15.107:0/2403809853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 0x56130dbebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:50:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.455+0000 7fabc1000640 1 --2- 172.21.15.107:0/2403809853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf44c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fabb0000c00 tx=0x7fabb002ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e8dd9094b143f5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:00.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.455+0000 7fabad600640 1 -- 172.21.15.107:0/2403809853 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0040030 con 0x56130d821390 2024-07-26T12:50:00.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.455+0000 7fabad600640 1 -- 172.21.15.107:0/2403809853 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb0004770 con 0x56130d821390 2024-07-26T12:50:00.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.455+0000 7fabad600640 1 -- 172.21.15.107:0/2403809853 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb00054f0 con 0x56130d821390 2024-07-26T12:50:00.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.456+0000 7fabca826740 1 -- 172.21.15.107:0/2403809853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 msgr2=0x56130dbf44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:00.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.456+0000 7fabca826740 1 --2- 172.21.15.107:0/2403809853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf44c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fabb0000c00 tx=0x7fabb002ab10 comp rx=0 tx=0).stop 2024-07-26T12:50:00.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.456+0000 7fabca826740 1 -- 172.21.15.107:0/2403809853 shutdown_connections 2024-07-26T12:50:00.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.456+0000 7fabca826740 1 --2- 172.21.15.107:0/2403809853 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 0x56130dbebee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.456+0000 7fabca826740 1 --2- 172.21.15.107:0/2403809853 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf44c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.456+0000 7fabca826740 1 --2- 172.21.15.107:0/2403809853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbedc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.456+0000 7fabca826740 1 -- 172.21.15.107:0/2403809853 >> 172.21.15.107:0/2403809853 conn(0x56130d836460 msgr2=0x56130dbf2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:00.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.456+0000 7fabca826740 1 -- 172.21.15.107:0/2403809853 shutdown_connections 2024-07-26T12:50:00.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.456+0000 7fabca826740 1 -- 172.21.15.107:0/2403809853 wait complete. 2024-07-26T12:50:00.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.457+0000 7fabca826740 1 -- start start 2024-07-26T12:50:00.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.457+0000 7fabca826740 1 Processor -- start 2024-07-26T12:50:00.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.457+0000 7fabca826740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbf6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.457+0000 7fabca826740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.457+0000 7fabca826740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 0x56130dc6bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.457+0000 7fabca826740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c011390 con 0x56130d8191b0 2024-07-26T12:50:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.457+0000 7fabca826740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c010cc0 con 0x56130d821390 2024-07-26T12:50:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.457+0000 7fabca826740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c00df00 con 0x56130d842070 2024-07-26T12:50:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.457+0000 7fabca826740 10 client.0 _ll_register_callbacks cb 0x7fabb0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.458+0000 7fabc1000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.458+0000 7fabbbe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbf6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.458+0000 7fabc1000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:50714/0 (socket says 172.21.15.107:50714) 2024-07-26T12:50:00.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.458+0000 7fabc1000640 1 -- 172.21.15.107:0/2587383859 learned_addr learned my addr 172.21.15.107:0/2587383859 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:00.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.458+0000 7fabbbe00640 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 0x56130dc6bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.458+0000 7fabc1000640 1 -- 172.21.15.107:0/2587383859 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 msgr2=0x56130dc6bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:00.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.458+0000 7fabc1000640 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 0x56130dc6bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.458+0000 7fabc1000640 1 -- 172.21.15.107:0/2587383859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 msgr2=0x56130dbf6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:00.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.458+0000 7fabc1000640 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbf6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.458+0000 7fabc1000640 1 -- 172.21.15.107:0/2587383859 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56130dbf7840 con 0x56130d821390 2024-07-26T12:50:00.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.459+0000 7fabc1000640 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf6fc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fabb00092a0 tx=0x7fabb00055d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b113c708ce93e66e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:00.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.459+0000 7faba2a00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:50:00.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.459+0000 7faba2a00640 1 -- 172.21.15.107:0/2587383859 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0060060 con 0x56130d821390 2024-07-26T12:50:00.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.459+0000 7fabca826740 1 -- 172.21.15.107:0/2587383859 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabb0004770 con 0x56130d821390 2024-07-26T12:50:00.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.459+0000 7faba2a00640 1 -- 172.21.15.107:0/2587383859 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fabb00316b0 con 0x56130d821390 2024-07-26T12:50:00.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.459+0000 7faba2a00640 1 -- 172.21.15.107:0/2587383859 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fabb0040070 con 0x56130d821390 2024-07-26T12:50:00.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.459+0000 7fabca826740 10 client.25706 fetch_fsmap learned FSMap version 29 2024-07-26T12:50:00.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.459+0000 7fabca826740 1 -- 172.21.15.107:0/2587383859 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56130dc6d3e0 con 0x56130d821390 2024-07-26T12:50:00.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.460+0000 7faba2a00640 1 -- 172.21.15.107:0/2587383859 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb0004770 con 0x56130d821390 2024-07-26T12:50:00.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.460+0000 7faba2a00640 1 -- 172.21.15.107:0/2587383859 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fabb005bc00 con 0x56130d821390 2024-07-26T12:50:00.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.460+0000 7fabca826740 10 client.25706 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:50:00.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.460+0000 7fabca826740 10 client.25706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:00.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.460+0000 7fabca826740 20 client.25706 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:00.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.460+0000 7fabca826740 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56130d7d5da0 0x56130dc6f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:00.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.460+0000 7fabca826740 4 client.25706 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T12:50:00.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.460+0000 7fabca826740 1 -- 172.21.15.107:0/2587383859 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56130db5b5e0 con 0x56130d7d5da0 2024-07-26T12:50:00.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.460+0000 7fabbbe00640 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56130d7d5da0 0x56130dc6f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:00.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7fabbbe00640 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56130d7d5da0 0x56130dc6f730 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc73e97089f97ea3 server_cookie=1b18d3ca098df3d6 in_seq=0 out_seq=0 2024-07-26T12:50:00.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7faba2a00640 10 client.25706 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:00.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7faba2a00640 1 -- 172.21.15.107:0/2587383859 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56130db5b5e0 con 0x56130d7d5da0 2024-07-26T12:50:00.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7faba2a00640 10 client.25706 handle_command_reply: tid=0 2024-07-26T12:50:00.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7fabca826740 1 client.25706 shutdown 2024-07-26T12:50:00.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7fabca826740 1 -- 172.21.15.107:0/2587383859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 msgr2=0x56130dbf6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:00.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7fabca826740 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf6fc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fabb00092a0 tx=0x7fabb00055d0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7fabca826740 1 -- 172.21.15.107:0/2587383859 shutdown_connections 2024-07-26T12:50:00.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7fabca826740 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56130d7d5da0 0x56130dc6f730 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7fabca826740 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56130d842070 0x56130dc6bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7fabca826740 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56130d821390 0x56130dbf6fc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7fabca826740 1 --2- 172.21.15.107:0/2587383859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56130d8191b0 0x56130dbf6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.461+0000 7fabca826740 1 -- 172.21.15.107:0/2587383859 >> 172.21.15.107:0/2587383859 conn(0x56130d836460 msgr2=0x56130dbf2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.462+0000 7faba2a00640 0 client.25706 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.462+0000 7fabca826740 1 -- 172.21.15.107:0/2587383859 shutdown_connections 2024-07-26T12:50:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.462+0000 7fabca826740 1 -- 172.21.15.107:0/2587383859 wait complete. 2024-07-26T12:50:00.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.462+0000 7fabca826740 20 client.25706 trim_cache size 0 max 16384 2024-07-26T12:50:00.473 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-07-26T12:50:00.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 -- 172.21.15.107:0/2987762211 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faba40b3dc0 msgr2=0x7faba40b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:00.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faba40b3dc0 0x7faba40b62d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fabb40a8370 tx=0x7fabb40a6fc0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 -- 172.21.15.107:0/2987762211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fabbc113e70 msgr2=0x7fabbc1a5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:00.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fabbc113e70 0x7fabbc1a5d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009fd0 tx=0x7fabb400ef10 comp rx=0 tx=0).stop 2024-07-26T12:50:00.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 -- 172.21.15.107:0/2987762211 shutdown_connections 2024-07-26T12:50:00.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7faba40b3dc0 0x7faba40b62d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fabbc113e70 0x7fabbc1a5d20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fabbc076db0 0x7fabbc1a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 --2- 172.21.15.107:0/2987762211 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fabbc075800 0x7fabbc1a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:00.474 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 -- 172.21.15.107:0/2987762211 >> 172.21.15.107:0/2987762211 conn(0x7fabbc0ff010 msgr2=0x7fabbc10fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:00.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 -- 172.21.15.107:0/2987762211 shutdown_connections 2024-07-26T12:50:00.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:00.463+0000 7fabc3a00640 1 -- 172.21.15.107:0/2987762211 wait complete. 2024-07-26T12:50:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:01 smithi107 ceph-mon[32659]: pgmap v511: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:50:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:01 smithi112 ceph-mon[37501]: pgmap v511: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:50:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:01 smithi132 ceph-mon[37580]: pgmap v511: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-07-26T12:50:02.738 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T12:50:02.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T12:50:03.182 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:50:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:02 smithi112 ceph-mon[37501]: pgmap v512: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 24 op/s 2024-07-26T12:50:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:02 smithi107 ceph-mon[32659]: pgmap v512: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 24 op/s 2024-07-26T12:50:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:02 smithi132 ceph-mon[37580]: pgmap v512: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 24 op/s 2024-07-26T12:50:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:04 smithi107 ceph-mon[32659]: pgmap v513: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-07-26T12:50:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.142+0000 7f7d25000640 1 -- 172.21.15.107:0/2598183838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d20108c80 msgr2=0x7f7d20109090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.142+0000 7f7d25000640 1 --2- 172.21.15.107:0/2598183838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d20108c80 0x7f7d20109090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c009900 tx=0x7f7d0c02a410 comp rx=0 tx=0).stop 2024-07-26T12:50:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.142+0000 7f7d25000640 1 -- 172.21.15.107:0/2598183838 shutdown_connections 2024-07-26T12:50:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.142+0000 7f7d25000640 1 --2- 172.21.15.107:0/2598183838 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d201095d0 0x7f7d2010fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.142+0000 7f7d25000640 1 --2- 172.21.15.107:0/2598183838 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d20108c80 0x7f7d20109090 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.142+0000 7f7d25000640 1 --2- 172.21.15.107:0/2598183838 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d20103350 0x7f7d20103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.144 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.142+0000 7f7d25000640 1 -- 172.21.15.107:0/2598183838 >> 172.21.15.107:0/2598183838 conn(0x7f7d200fed50 msgr2=0x7f7d201011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.142+0000 7f7d25000640 1 -- 172.21.15.107:0/2598183838 shutdown_connections 2024-07-26T12:50:05.145 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.142+0000 7f7d25000640 1 -- 172.21.15.107:0/2598183838 wait complete. 2024-07-26T12:50:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.143+0000 7f7d25000640 1 Processor -- start 2024-07-26T12:50:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.143+0000 7f7d25000640 1 -- start start 2024-07-26T12:50:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d25000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d20103350 0x7f7d2019bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d25000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d20108c80 0x7f7d2019c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d25000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d201095d0 0x7f7d201a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d25000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201132b0 con 0x7f7d201095d0 2024-07-26T12:50:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d25000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d20113130 con 0x7f7d20108c80 2024-07-26T12:50:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d25000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7d20113430 con 0x7f7d20103350 2024-07-26T12:50:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d201095d0 0x7f7d201a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.146 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d201095d0 0x7f7d201a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36060/0 (socket says 172.21.15.107:36060) 2024-07-26T12:50:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d20108c80 0x7f7d2019c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d1f400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d20103350 0x7f7d2019bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d1fe00640 1 -- 172.21.15.107:0/3746673416 learned_addr learned my addr 172.21.15.107:0/3746673416 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d1fe00640 1 -- 172.21.15.107:0/3746673416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d20103350 msgr2=0x7f7d2019bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d20103350 0x7f7d2019bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d1fe00640 1 -- 172.21.15.107:0/3746673416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d20108c80 msgr2=0x7f7d2019c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.144+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d20108c80 0x7f7d2019c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.145+0000 7f7d1fe00640 1 -- 172.21.15.107:0/3746673416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d201a1e20 con 0x7f7d201095d0 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.145+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d20108c80 0x7f7d2019c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.145+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d201095d0 0x7f7d201a1720 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800eb10 tx=0x7f7d1800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.145+0000 7f7d17e00640 1 -- 172.21.15.107:0/3746673416 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18004b20 con 0x7f7d201095d0 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.145+0000 7f7d17e00640 1 -- 172.21.15.107:0/3746673416 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d18004cc0 con 0x7f7d201095d0 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.145+0000 7f7d25000640 1 -- 172.21.15.107:0/3746673416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d201a2110 con 0x7f7d201095d0 2024-07-26T12:50:05.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.145+0000 7f7d17e00640 1 -- 172.21.15.107:0/3746673416 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18010670 con 0x7f7d201095d0 2024-07-26T12:50:05.149 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.145+0000 7f7d25000640 1 -- 172.21.15.107:0/3746673416 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d201a88a0 con 0x7f7d201095d0 2024-07-26T12:50:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.149+0000 7f7d17e00640 1 -- 172.21.15.107:0/3746673416 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f7d18005020 con 0x7f7d201095d0 2024-07-26T12:50:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.150+0000 7f7d2d611740 1 Processor -- start 2024-07-26T12:50:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.151+0000 7f7d2d611740 1 -- start start 2024-07-26T12:50:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.151+0000 7f7d17e00640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3dc0 0x7f7d040b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.151+0000 7f7d17e00640 1 -- 172.21.15.107:0/3746673416 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7d180e4900 con 0x7f7d201095d0 2024-07-26T12:50:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.151+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 0x55b00afe5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.151+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac192f0 0x55b00afe6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.151+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac214d0 0x55b00afe66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.151+0000 7f7d2d611740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b00af5a270 con 0x55b00ac214d0 2024-07-26T12:50:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.151+0000 7f7d2d611740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b00af49770 con 0x55b00ac421b0 2024-07-26T12:50:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.151+0000 7f7d2d611740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b00af4d7b0 con 0x55b00ac192f0 2024-07-26T12:50:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.151+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3dc0 0x7f7d040b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac214d0 0x55b00afe66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac214d0 0x55b00afe66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36074/0 (socket says 172.21.15.107:36074) 2024-07-26T12:50:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1fe00640 1 -- 172.21.15.107:0/668920895 learned_addr learned my addr 172.21.15.107:0/668920895 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1f400640 1 --2- 172.21.15.107:0/668920895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 0x55b00afe5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1fe00640 1 -- 172.21.15.107:0/668920895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac192f0 msgr2=0x55b00afe6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/668920895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac192f0 0x55b00afe6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1fe00640 1 -- 172.21.15.107:0/668920895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 msgr2=0x55b00afe5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/668920895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 0x55b00afe5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1fe00640 1 -- 172.21.15.107:0/668920895 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b00acad790 con 0x55b00ac214d0 2024-07-26T12:50:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1f400640 1 --2- 172.21.15.107:0/668920895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 0x55b00afe5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3dc0 0x7f7d040b62d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c02aa00 tx=0x7f7d0c005fd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.152+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/668920895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac214d0 0x55b00afe66c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7d180ab0c0 tx=0x7f7d180aab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ccdcf7584be3a0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.153+0000 7f7d14c00640 1 -- 172.21.15.107:0/668920895 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d180c5070 con 0x55b00ac214d0 2024-07-26T12:50:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.153+0000 7f7d14c00640 1 -- 172.21.15.107:0/668920895 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d180ab130 con 0x55b00ac214d0 2024-07-26T12:50:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.153+0000 7f7d14c00640 1 -- 172.21.15.107:0/668920895 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d180ab390 con 0x55b00ac214d0 2024-07-26T12:50:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.154+0000 7f7d2d611740 1 -- 172.21.15.107:0/668920895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac214d0 msgr2=0x55b00afe66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.154+0000 7f7d2d611740 1 --2- 172.21.15.107:0/668920895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac214d0 0x55b00afe66c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7d180ab0c0 tx=0x7f7d180aab40 comp rx=0 tx=0).stop 2024-07-26T12:50:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.154+0000 7f7d2d611740 1 -- 172.21.15.107:0/668920895 shutdown_connections 2024-07-26T12:50:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.154+0000 7f7d2d611740 1 --2- 172.21.15.107:0/668920895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac214d0 0x55b00afe66c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.154+0000 7f7d2d611740 1 --2- 172.21.15.107:0/668920895 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac192f0 0x55b00afe6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.154+0000 7f7d2d611740 1 --2- 172.21.15.107:0/668920895 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 0x55b00afe5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.154+0000 7f7d2d611740 1 -- 172.21.15.107:0/668920895 >> 172.21.15.107:0/668920895 conn(0x55b00ac36300 msgr2=0x55b00afe3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.154+0000 7f7d2d611740 1 -- 172.21.15.107:0/668920895 shutdown_connections 2024-07-26T12:50:05.157 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.154+0000 7f7d2d611740 1 -- 172.21.15.107:0/668920895 wait complete. 2024-07-26T12:50:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.155+0000 7f7d2d611740 1 -- start start 2024-07-26T12:50:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.155+0000 7f7d2d611740 1 Processor -- start 2024-07-26T12:50:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.155+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac192f0 0x55b00aff18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.155+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00aff1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.155+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac421b0 0x55b00affd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.155+0000 7f7d2d611740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b00ae83ab0 con 0x55b00ac421b0 2024-07-26T12:50:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.155+0000 7f7d2d611740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55b00aee6d50 con 0x55b00ac214d0 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.155+0000 7f7d2d611740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55b00aff43c0 con 0x55b00ac192f0 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.155+0000 7f7d2d611740 10 client.0 _ll_register_callbacks cb 0x55b00ac59a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac421b0 0x55b00affd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac421b0 0x55b00affd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36084/0 (socket says 172.21.15.107:36084) 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1fe00640 1 -- 172.21.15.107:0/167088085 learned_addr learned my addr 172.21.15.107:0/167088085 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00aff1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1f400640 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac192f0 0x55b00aff18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1fe00640 1 -- 172.21.15.107:0/167088085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac192f0 msgr2=0x55b00aff18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac192f0 0x55b00aff18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1fe00640 1 -- 172.21.15.107:0/167088085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 msgr2=0x55b00aff1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00aff1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1fe00640 1 -- 172.21.15.107:0/167088085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b00ad26620 con 0x55b00ac421b0 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00aff1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac421b0 0x55b00affd6b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7d180049d0 tx=0x7f7d180e1c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea374ccc0056dcf server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.156+0000 7f7d09600640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.157+0000 7f7d09600640 1 -- 172.21.15.107:0/167088085 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d180ca060 con 0x55b00ac421b0 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.157+0000 7f7d2d611740 1 -- 172.21.15.107:0/167088085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d180ab130 con 0x55b00ac421b0 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.157+0000 7f7d09600640 1 -- 172.21.15.107:0/167088085 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7d180a7720 con 0x55b00ac421b0 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.157+0000 7f7d09600640 1 -- 172.21.15.107:0/167088085 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7d180c5070 con 0x55b00ac421b0 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.157+0000 7f7d2d611740 10 client.15585 fetch_fsmap learned FSMap version 29 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.157+0000 7f7d2d611740 1 -- 172.21.15.107:0/167088085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b00b000480 con 0x55b00ac421b0 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.157+0000 7f7d09600640 1 -- 172.21.15.107:0/167088085 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d180ab130 con 0x55b00ac421b0 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.158+0000 7f7d09600640 1 -- 172.21.15.107:0/167088085 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7d180a6aa0 con 0x55b00ac421b0 2024-07-26T12:50:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.158+0000 7f7d2d611740 10 client.15585 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:50:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.158+0000 7f7d2d611740 10 client.15585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.158+0000 7f7d2d611740 20 client.15585 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.158+0000 7f7d2d611740 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55b00ac00d80 0x55b00b0037e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.158+0000 7f7d2d611740 4 client.15585 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:50:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.158+0000 7f7d2d611740 1 -- 172.21.15.107:0/167088085 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b00af5c1c0 con 0x55b00ac00d80 2024-07-26T12:50:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.158+0000 7f7d1f400640 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55b00ac00d80 0x55b00b0037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.159+0000 7f7d1f400640 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55b00ac00d80 0x55b00b0037e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1233311574eb8bb server_cookie=50929591ccea4b73 in_seq=0 out_seq=0 2024-07-26T12:50:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.159+0000 7f7d09600640 10 client.15585 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.159+0000 7f7d09600640 1 -- 172.21.15.107:0/167088085 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b00af5c1c0 con 0x55b00ac00d80 2024-07-26T12:50:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.159+0000 7f7d09600640 10 client.15585 handle_command_reply: tid=0 2024-07-26T12:50:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.159+0000 7f7d2d611740 1 client.15585 shutdown 2024-07-26T12:50:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 1 -- 172.21.15.107:0/167088085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac421b0 msgr2=0x55b00affd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac421b0 0x55b00affd6b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7d180049d0 tx=0x7f7d180e1c50 comp rx=0 tx=0).stop 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 1 -- 172.21.15.107:0/167088085 shutdown_connections 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55b00ac00d80 0x55b00b0037e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac421b0 0x55b00affd6b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00aff1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d09600640 0 client.15585 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 1 --2- 172.21.15.107:0/167088085 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac192f0 0x55b00aff18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 1 -- 172.21.15.107:0/167088085 >> 172.21.15.107:0/167088085 conn(0x55b00ac36300 msgr2=0x55b00afe6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 1 -- 172.21.15.107:0/167088085 shutdown_connections 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 1 -- 172.21.15.107:0/167088085 wait complete. 2024-07-26T12:50:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.160+0000 7f7d2d611740 20 client.15585 trim_cache size 0 max 16384 2024-07-26T12:50:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.177+0000 7f7d2d611740 1 Processor -- start 2024-07-26T12:50:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.177+0000 7f7d2d611740 1 -- start start 2024-07-26T12:50:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.177+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 0x55b00afeb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.177+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00affd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.177+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac421b0 0x55b00affdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.177+0000 7f7d2d611740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b2930 con 0x55b00ac192f0 2024-07-26T12:50:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.177+0000 7f7d2d611740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b4470 con 0x55b00ac214d0 2024-07-26T12:50:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.177+0000 7f7d2d611740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b3000 con 0x55b00ac421b0 2024-07-26T12:50:05.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.177+0000 7f7d1f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 0x55b00afeb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00affd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac421b0 0x55b00affdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.180 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00affd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:33524/0 (socket says 172.21.15.107:33524) 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1ea00640 1 -- 172.21.15.107:0/1125207612 learned_addr learned my addr 172.21.15.107:0/1125207612 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1ea00640 1 -- 172.21.15.107:0/1125207612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac421b0 msgr2=0x55b00affdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/1125207612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac421b0 0x55b00affdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1ea00640 1 -- 172.21.15.107:0/1125207612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 msgr2=0x55b00afeb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/1125207612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 0x55b00afeb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1ea00640 1 -- 172.21.15.107:0/1125207612 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b00aff3dd0 con 0x55b00ac214d0 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/1125207612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac421b0 0x55b00affdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/1125207612 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00affd620 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c031440 tx=0x7f7d0c03f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=522691697b905fdc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.178+0000 7f7d0be00640 1 -- 172.21.15.107:0/1125207612 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c035040 con 0x55b00ac214d0 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.179+0000 7f7d0be00640 1 -- 172.21.15.107:0/1125207612 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d0c041040 con 0x55b00ac214d0 2024-07-26T12:50:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.179+0000 7f7d0be00640 1 -- 172.21.15.107:0/1125207612 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c003e30 con 0x55b00ac214d0 2024-07-26T12:50:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.179+0000 7f7d2d611740 1 -- 172.21.15.107:0/1125207612 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 msgr2=0x55b00affd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.179+0000 7f7d2d611740 1 --2- 172.21.15.107:0/1125207612 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00affd620 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c031440 tx=0x7f7d0c03f000 comp rx=0 tx=0).stop 2024-07-26T12:50:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.180+0000 7f7d2d611740 1 -- 172.21.15.107:0/1125207612 shutdown_connections 2024-07-26T12:50:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.180+0000 7f7d2d611740 1 --2- 172.21.15.107:0/1125207612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac421b0 0x55b00affdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.180+0000 7f7d2d611740 1 --2- 172.21.15.107:0/1125207612 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac214d0 0x55b00affd620 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.180+0000 7f7d2d611740 1 --2- 172.21.15.107:0/1125207612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 0x55b00afeb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.180+0000 7f7d2d611740 1 -- 172.21.15.107:0/1125207612 >> 172.21.15.107:0/1125207612 conn(0x55b00ac36300 msgr2=0x55b00afeacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.180+0000 7f7d2d611740 1 -- 172.21.15.107:0/1125207612 shutdown_connections 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.180+0000 7f7d2d611740 1 -- 172.21.15.107:0/1125207612 wait complete. 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.180+0000 7f7d2d611740 1 -- start start 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.180+0000 7f7d2d611740 1 Processor -- start 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 0x55b00b003950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac214d0 0x55b00b003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d2d611740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 0x55b00b069d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d2d611740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b3da0 con 0x55b00ac192f0 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d2d611740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b36d0 con 0x55b00ac421b0 2024-07-26T12:50:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d2d611740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7d201b0910 con 0x55b00ac214d0 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d2d611740 10 client.0 _ll_register_callbacks cb 0x7f7d180c3070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d1f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 0x55b00b003950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d1f400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 0x55b00b003950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36100/0 (socket says 172.21.15.107:36100) 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d1f400640 1 -- 172.21.15.107:0/2521294381 learned_addr learned my addr 172.21.15.107:0/2521294381 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac214d0 0x55b00b003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.181+0000 7f7d1fe00640 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 0x55b00b069d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.182+0000 7f7d1f400640 1 -- 172.21.15.107:0/2521294381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac214d0 msgr2=0x55b00b003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.182+0000 7f7d1f400640 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac214d0 0x55b00b003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.182+0000 7f7d1f400640 1 -- 172.21.15.107:0/2521294381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 msgr2=0x55b00b069d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.182+0000 7f7d1f400640 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 0x55b00b069d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.182+0000 7f7d1f400640 1 -- 172.21.15.107:0/2521294381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b00b0046a0 con 0x55b00ac192f0 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.182+0000 7f7d1f400640 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 0x55b00b003950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x55b00af5a380 tx=0x7f7d10010a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf8e39c066e1345 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.182+0000 7f7d09600640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.182+0000 7f7d09600640 1 -- 172.21.15.107:0/2521294381 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10022070 con 0x55b00ac192f0 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.182+0000 7f7d09600640 1 -- 172.21.15.107:0/2521294381 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7d1001f520 con 0x55b00ac192f0 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.182+0000 7f7d2d611740 1 -- 172.21.15.107:0/2521294381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d0c041040 con 0x55b00ac192f0 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d09600640 1 -- 172.21.15.107:0/2521294381 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7d10030050 con 0x55b00ac192f0 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d2d611740 10 client.15591 fetch_fsmap learned FSMap version 29 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d2d611740 1 -- 172.21.15.107:0/2521294381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b00b06b2c0 con 0x55b00ac192f0 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d09600640 1 -- 172.21.15.107:0/2521294381 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d0c041040 con 0x55b00ac192f0 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d09600640 1 -- 172.21.15.107:0/2521294381 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f7d10003ae0 con 0x55b00ac192f0 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d2d611740 10 client.15591 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:50:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d2d611740 10 client.15591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d2d611740 20 client.15591 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d2d611740 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55b00abd61c0 0x55b00b06da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d2d611740 4 client.15591 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T12:50:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.183+0000 7f7d2d611740 1 -- 172.21.15.107:0/2521294381 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b00af57c60 con 0x55b00abd61c0 2024-07-26T12:50:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.184+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55b00abd61c0 0x55b00b06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.184+0000 7f7d1ea00640 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55b00abd61c0 0x55b00b06da90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4278fd022072735c server_cookie=b26dc8fd419895fe in_seq=0 out_seq=0 2024-07-26T12:50:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.184+0000 7f7d09600640 10 client.15591 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d09600640 1 -- 172.21.15.107:0/2521294381 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b00af57c60 con 0x55b00abd61c0 2024-07-26T12:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d09600640 10 client.15591 handle_command_reply: tid=0 2024-07-26T12:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 client.15591 shutdown 2024-07-26T12:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 -- 172.21.15.107:0/2521294381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 msgr2=0x55b00b003950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 0x55b00b003950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x55b00af5a380 tx=0x7f7d10010a50 comp rx=0 tx=0).stop 2024-07-26T12:50:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 -- 172.21.15.107:0/2521294381 shutdown_connections 2024-07-26T12:50:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55b00abd61c0 0x55b00b06da90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55b00ac421b0 0x55b00b069d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d09600640 0 client.15591 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55b00ac214d0 0x55b00b003e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 --2- 172.21.15.107:0/2521294381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b00ac192f0 0x55b00b003950 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 -- 172.21.15.107:0/2521294381 >> 172.21.15.107:0/2521294381 conn(0x55b00ac36300 msgr2=0x55b00afe6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 -- 172.21.15.107:0/2521294381 shutdown_connections 2024-07-26T12:50:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.191+0000 7f7d2d611740 1 -- 172.21.15.107:0/2521294381 wait complete. 2024-07-26T12:50:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d2d611740 20 client.15591 trim_cache size 0 max 16384 2024-07-26T12:50:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-07-26T12:50:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-07-26T12:50:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-07-26T12:50:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-07-26T12:50:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d25000640 1 -- 172.21.15.107:0/3746673416 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3dc0 msgr2=0x7f7d040b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d25000640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3dc0 0x7f7d040b62d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c02aa00 tx=0x7f7d0c005fd0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d25000640 1 -- 172.21.15.107:0/3746673416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d201095d0 msgr2=0x7f7d201a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d25000640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d201095d0 0x7f7d201a1720 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800eb10 tx=0x7f7d1800c2d0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d25000640 1 -- 172.21.15.107:0/3746673416 shutdown_connections 2024-07-26T12:50:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d25000640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7d040b3dc0 0x7f7d040b62d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d25000640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d201095d0 0x7f7d201a1720 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d25000640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7d20108c80 0x7f7d2019c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d25000640 1 --2- 172.21.15.107:0/3746673416 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7d20103350 0x7f7d2019bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.192+0000 7f7d25000640 1 -- 172.21.15.107:0/3746673416 >> 172.21.15.107:0/3746673416 conn(0x7f7d200fed50 msgr2=0x7f7d20106200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.193+0000 7f7d25000640 1 -- 172.21.15.107:0/3746673416 shutdown_connections 2024-07-26T12:50:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:05.193+0000 7f7d25000640 1 -- 172.21.15.107:0/3746673416 wait complete. 2024-07-26T12:50:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:04 smithi112 ceph-mon[37501]: pgmap v513: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-07-26T12:50:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:04 smithi132 ceph-mon[37580]: pgmap v513: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-07-26T12:50:07.075 INFO:tasks.workunit.client.0.smithi107.stderr:Updating files: 59% (7510/12697) Updating files: 60% (7619/12697) Updating files: 61% (7746/12697) Updating files: 62% (7873/12697) Updating files: 63% (8000/12697) Updating files: 64% (8127/12697) Updating files: 65% (8254/12697) Updating files: 66% (8381/12697) Updating files: 67% (8507/12697) Updating files: 68% (8634/12697) Updating files: 69% (8761/12697) Updating files: 70% (8888/12697) Updating files: 71% (9015/12697) Updating files: 72% (9142/12697) Updating files: 73% (9269/12697) Updating files: 74% (9396/12697) Updating files: 75% (9523/12697) Updating files: 76% (9650/12697) Updating files: 77% (9777/12697) Updating files: 78% (9904/12697) Updating files: 79% (10031/12697) Updating files: 80% (10158/12697) Updating files: 81% (10285/12697) Updating files: 82% (10412/12697) Updating files: 83% (10539/12697) Updating files: 84% (10666/12697) Updating files: 85% (10793/12697) Updating files: 86% (10920/12697) Updating files: 87% (11047/12697) Updating files: 88% (11174/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 97% (12348/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-07-26T12:50:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:06 smithi112 ceph-mon[37501]: pgmap v514: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-07-26T12:50:07.210 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:06 smithi107 ceph-mon[32659]: pgmap v514: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-07-26T12:50:07.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.300+0000 7fb84b600640 1 -- 172.21.15.107:0/2466920869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb844109820 msgr2=0x7fb844110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:07.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.300+0000 7fb84b600640 1 --2- 172.21.15.107:0/2466920869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb844109820 0x7fb844110050 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb83c009860 tx=0x7fb83c02a340 comp rx=0 tx=0).stop 2024-07-26T12:50:07.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.300+0000 7fb84b600640 1 -- 172.21.15.107:0/2466920869 shutdown_connections 2024-07-26T12:50:07.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.300+0000 7fb84b600640 1 --2- 172.21.15.107:0/2466920869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb844109820 0x7fb844110050 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.300+0000 7fb84b600640 1 --2- 172.21.15.107:0/2466920869 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb844108ed0 0x7fb8441092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.300+0000 7fb84b600640 1 --2- 172.21.15.107:0/2466920869 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb844103590 0x7fb8441039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.300+0000 7fb84b600640 1 -- 172.21.15.107:0/2466920869 >> 172.21.15.107:0/2466920869 conn(0x7fb84407b280 msgr2=0x7fb84407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:07.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.301+0000 7fb84b600640 1 -- 172.21.15.107:0/2466920869 shutdown_connections 2024-07-26T12:50:07.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.301+0000 7fb84b600640 1 -- 172.21.15.107:0/2466920869 wait complete. 2024-07-26T12:50:07.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.302+0000 7fb84b600640 1 Processor -- start 2024-07-26T12:50:07.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.302+0000 7fb84b600640 1 -- start start 2024-07-26T12:50:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb84b600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb844103590 0x7fb844199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb84b600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb844108ed0 0x7fb84419a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb84b600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb844109820 0x7fb84419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb84b600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb8441134d0 con 0x7fb844108ed0 2024-07-26T12:50:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb84b600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb844113350 con 0x7fb844103590 2024-07-26T12:50:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb84b600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb844113650 con 0x7fb844109820 2024-07-26T12:50:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb849600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb844109820 0x7fb84419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb843e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb844108ed0 0x7fb84419a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb849600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb844109820 0x7fb84419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33800/0 (socket says 172.21.15.107:33800) 2024-07-26T12:50:07.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb848c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb844103590 0x7fb844199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.303+0000 7fb849600640 1 -- 172.21.15.107:0/1634576728 learned_addr learned my addr 172.21.15.107:0/1634576728 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:07.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.304+0000 7fb849600640 1 -- 172.21.15.107:0/1634576728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb844103590 msgr2=0x7fb844199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.304+0000 7fb849600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb844103590 0x7fb844199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.304+0000 7fb849600640 1 -- 172.21.15.107:0/1634576728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb844108ed0 msgr2=0x7fb84419a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.304+0000 7fb849600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb844108ed0 0x7fb84419a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.304+0000 7fb849600640 1 -- 172.21.15.107:0/1634576728 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb84419fe10 con 0x7fb844109820 2024-07-26T12:50:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.304+0000 7fb849600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb844109820 0x7fb84419f6e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb83c02a930 tx=0x7fb83c032d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.304+0000 7fb841600640 1 -- 172.21.15.107:0/1634576728 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c042070 con 0x7fb844109820 2024-07-26T12:50:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.305+0000 7fb84b600640 1 -- 172.21.15.107:0/1634576728 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8441a00a0 con 0x7fb844109820 2024-07-26T12:50:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.305+0000 7fb841600640 1 -- 172.21.15.107:0/1634576728 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb83c030d50 con 0x7fb844109820 2024-07-26T12:50:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.305+0000 7fb841600640 1 -- 172.21.15.107:0/1634576728 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb83c037040 con 0x7fb844109820 2024-07-26T12:50:07.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.305+0000 7fb84b600640 1 -- 172.21.15.107:0/1634576728 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8441a6860 con 0x7fb844109820 2024-07-26T12:50:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:06 smithi132 ceph-mon[37580]: pgmap v514: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-07-26T12:50:07.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.309+0000 7fb841600640 1 -- 172.21.15.107:0/1634576728 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fb83c045070 con 0x7fb844109820 2024-07-26T12:50:07.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.309+0000 7fb852242740 1 Processor -- start 2024-07-26T12:50:07.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.310+0000 7fb852242740 1 -- start start 2024-07-26T12:50:07.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.310+0000 7fb852242740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5772f350 0x55df57ad3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.310+0000 7fb852242740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57ad42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.310+0000 7fb852242740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 0x55df57ad4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.312 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.310+0000 7fb852242740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55df57a4abf0 con 0x55df5772f350 2024-07-26T12:50:07.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.310+0000 7fb852242740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55df57a3a0f0 con 0x55df57706490 2024-07-26T12:50:07.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.310+0000 7fb852242740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55df57a3e130 con 0x55df5770e670 2024-07-26T12:50:07.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.310+0000 7fb841600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb8240b3d40 0x7fb8240b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb843e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5772f350 0x55df57ad3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb843e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5772f350 0x55df57ad3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36126/0 (socket says 172.21.15.107:36126) 2024-07-26T12:50:07.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb843e00640 1 -- 172.21.15.107:0/200825890 learned_addr learned my addr 172.21.15.107:0/200825890 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:07.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb848c00640 1 --2- 172.21.15.107:0/200825890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57ad42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb841600640 1 -- 172.21.15.107:0/1634576728 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb83c103960 con 0x7fb844109820 2024-07-26T12:50:07.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb849600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb8240b3d40 0x7fb8240b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb843e00640 1 --2- 172.21.15.107:0/200825890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 0x55df57ad4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb848c00640 1 -- 172.21.15.107:0/200825890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 msgr2=0x55df57ad4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:07.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb848c00640 1 --2- 172.21.15.107:0/200825890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 0x55df57ad4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb848c00640 1 -- 172.21.15.107:0/200825890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5772f350 msgr2=0x55df57ad3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb848c00640 1 --2- 172.21.15.107:0/200825890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5772f350 0x55df57ad3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb848c00640 1 -- 172.21.15.107:0/200825890 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df5779a930 con 0x55df57706490 2024-07-26T12:50:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.311+0000 7fb843e00640 1 --2- 172.21.15.107:0/200825890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5772f350 0x55df57ad3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.312+0000 7fb848c00640 1 --2- 172.21.15.107:0/200825890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57ad42d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb834009c20 tx=0x7fb834011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1780170ed6d3b118 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.312+0000 7fb839c00640 1 -- 172.21.15.107:0/200825890 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834004050 con 0x55df57706490 2024-07-26T12:50:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.312+0000 7fb849600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb8240b3d40 0x7fb8240b6250 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb83c005820 tx=0x7fb83c030750 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.312+0000 7fb839c00640 1 -- 172.21.15.107:0/200825890 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8340054d0 con 0x55df57706490 2024-07-26T12:50:07.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.312+0000 7fb839c00640 1 -- 172.21.15.107:0/200825890 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834004be0 con 0x55df57706490 2024-07-26T12:50:07.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.313+0000 7fb852242740 1 -- 172.21.15.107:0/200825890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 msgr2=0x55df57ad42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:07.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.313+0000 7fb852242740 1 --2- 172.21.15.107:0/200825890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57ad42d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb834009c20 tx=0x7fb834011810 comp rx=0 tx=0).stop 2024-07-26T12:50:07.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.313+0000 7fb852242740 1 -- 172.21.15.107:0/200825890 shutdown_connections 2024-07-26T12:50:07.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.313+0000 7fb852242740 1 --2- 172.21.15.107:0/200825890 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 0x55df57ad4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.313+0000 7fb852242740 1 --2- 172.21.15.107:0/200825890 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57ad42d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.313+0000 7fb852242740 1 --2- 172.21.15.107:0/200825890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5772f350 0x55df57ad3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.313+0000 7fb852242740 1 -- 172.21.15.107:0/200825890 >> 172.21.15.107:0/200825890 conn(0x55df57723740 msgr2=0x55df57ad1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.313+0000 7fb852242740 1 -- 172.21.15.107:0/200825890 shutdown_connections 2024-07-26T12:50:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.314+0000 7fb852242740 1 -- 172.21.15.107:0/200825890 wait complete. 2024-07-26T12:50:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.314+0000 7fb852242740 1 -- start start 2024-07-26T12:50:07.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.315+0000 7fb852242740 1 Processor -- start 2024-07-26T12:50:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.315+0000 7fb852242740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57adf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.315+0000 7fb852242740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 0x55df57adf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.315+0000 7fb852242740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5772f350 0x55df57adfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.315+0000 7fb852242740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55df5789d890 con 0x55df5770e670 2024-07-26T12:50:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.315+0000 7fb852242740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55df579d6330 con 0x55df57706490 2024-07-26T12:50:07.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.315+0000 7fb852242740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55df57ae2570 con 0x55df5772f350 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb852242740 10 client.0 _ll_register_callbacks cb 0x55df57a02010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb843e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57adf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb848c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 0x55df57adf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb843e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57adf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:33570/0 (socket says 172.21.15.107:33570) 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb843e00640 1 -- 172.21.15.107:0/1112625026 learned_addr learned my addr 172.21.15.107:0/1112625026 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb843e00640 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5772f350 0x55df57adfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb843e00640 1 -- 172.21.15.107:0/1112625026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5772f350 msgr2=0x55df57adfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb843e00640 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5772f350 0x55df57adfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb843e00640 1 -- 172.21.15.107:0/1112625026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 msgr2=0x55df57adf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb843e00640 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 0x55df57adf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.316+0000 7fb843e00640 1 -- 172.21.15.107:0/1112625026 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df57aeba90 con 0x55df57706490 2024-07-26T12:50:07.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.317+0000 7fb843e00640 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57adf440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55df57ae21c0 tx=0x7fb82c01e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a74f947695fc4690 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:07.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.317+0000 7fb832000640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:50:07.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.317+0000 7fb832000640 1 -- 172.21.15.107:0/1112625026 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c01c0d0 con 0x55df57706490 2024-07-26T12:50:07.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.317+0000 7fb852242740 1 -- 172.21.15.107:0/1112625026 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8340054d0 con 0x55df57706490 2024-07-26T12:50:07.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.317+0000 7fb832000640 1 -- 172.21.15.107:0/1112625026 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb82c01c270 con 0x55df57706490 2024-07-26T12:50:07.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.317+0000 7fb832000640 1 -- 172.21.15.107:0/1112625026 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb82c02e050 con 0x55df57706490 2024-07-26T12:50:07.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.317+0000 7fb852242740 10 client.25736 fetch_fsmap learned FSMap version 29 2024-07-26T12:50:07.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.317+0000 7fb852242740 1 -- 172.21.15.107:0/1112625026 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55df57aee3a0 con 0x55df57706490 2024-07-26T12:50:07.321 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.318+0000 7fb832000640 1 -- 172.21.15.107:0/1112625026 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8340054d0 con 0x55df57706490 2024-07-26T12:50:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.318+0000 7fb832000640 1 -- 172.21.15.107:0/1112625026 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb82c029030 con 0x55df57706490 2024-07-26T12:50:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.318+0000 7fb852242740 10 client.25736 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:50:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.318+0000 7fb852242740 10 client.25736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.318+0000 7fb852242740 20 client.25736 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.318+0000 7fb852242740 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55df576edcd0 0x55df57af18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.318+0000 7fb852242740 4 client.25736 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:50:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.318+0000 7fb852242740 1 -- 172.21.15.107:0/1112625026 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df57a4cb40 con 0x55df576edcd0 2024-07-26T12:50:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.319+0000 7fb848c00640 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55df576edcd0 0x55df57af18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.319+0000 7fb848c00640 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55df576edcd0 0x55df57af18b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71377e501faf2f80 server_cookie=490144ecc0a44358 in_seq=0 out_seq=0 2024-07-26T12:50:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.319+0000 7fb832000640 10 client.25736 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:07.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb832000640 1 -- 172.21.15.107:0/1112625026 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55df57a4cb40 con 0x55df576edcd0 2024-07-26T12:50:07.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb832000640 10 client.25736 handle_command_reply: tid=0 2024-07-26T12:50:07.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb852242740 1 client.25736 shutdown 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb852242740 1 -- 172.21.15.107:0/1112625026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 msgr2=0x55df57adf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb852242740 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57adf440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x55df57ae21c0 tx=0x7fb82c01e040 comp rx=0 tx=0).stop 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb852242740 1 -- 172.21.15.107:0/1112625026 shutdown_connections 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb852242740 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55df576edcd0 0x55df57af18b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb852242740 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5772f350 0x55df57adfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb852242740 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 0x55df57adf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb832000640 0 client.25736 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb852242740 1 --2- 172.21.15.107:0/1112625026 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57adf440 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.320+0000 7fb852242740 1 -- 172.21.15.107:0/1112625026 >> 172.21.15.107:0/1112625026 conn(0x55df57723740 msgr2=0x55df57ad6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.321+0000 7fb852242740 1 -- 172.21.15.107:0/1112625026 shutdown_connections 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.321+0000 7fb852242740 1 -- 172.21.15.107:0/1112625026 wait complete. 2024-07-26T12:50:07.324 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.321+0000 7fb852242740 20 client.25736 trim_cache size 0 max 16384 2024-07-26T12:50:07.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.340+0000 7fb852242740 1 Processor -- start 2024-07-26T12:50:07.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.340+0000 7fb852242740 1 -- start start 2024-07-26T12:50:07.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb852242740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57ad7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb852242740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 0x55df57ae1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb852242740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5772f350 0x55df57adfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb852242740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb8441ae610 con 0x55df5770e670 2024-07-26T12:50:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb852242740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb8441b0150 con 0x55df57706490 2024-07-26T12:50:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb852242740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb8441aece0 con 0x55df5772f350 2024-07-26T12:50:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb848c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 0x55df57ae1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb848c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 0x55df57ae1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36136/0 (socket says 172.21.15.107:36136) 2024-07-26T12:50:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb843e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57ad7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb848c00640 1 -- 172.21.15.107:0/1068119508 learned_addr learned my addr 172.21.15.107:0/1068119508 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:07.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.341+0000 7fb843e00640 1 --2- 172.21.15.107:0/1068119508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5772f350 0x55df57adfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.342+0000 7fb843e00640 1 -- 172.21.15.107:0/1068119508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5772f350 msgr2=0x55df57adfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.342+0000 7fb843e00640 1 --2- 172.21.15.107:0/1068119508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5772f350 0x55df57adfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.342+0000 7fb843e00640 1 -- 172.21.15.107:0/1068119508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 msgr2=0x55df57ae1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.342+0000 7fb843e00640 1 --2- 172.21.15.107:0/1068119508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 0x55df57ae1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.342+0000 7fb843e00640 1 -- 172.21.15.107:0/1068119508 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df57ad6080 con 0x55df57706490 2024-07-26T12:50:07.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.342+0000 7fb848c00640 1 --2- 172.21.15.107:0/1068119508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 0x55df57ae1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.342+0000 7fb843e00640 1 --2- 172.21.15.107:0/1068119508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57ad7960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb82c029440 tx=0x7fb82c01c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7ddf040581d19f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.342+0000 7fb839200640 1 -- 172.21.15.107:0/1068119508 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c004a40 con 0x55df57706490 2024-07-26T12:50:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.342+0000 7fb839200640 1 -- 172.21.15.107:0/1068119508 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb82c01a5e0 con 0x55df57706490 2024-07-26T12:50:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.342+0000 7fb839200640 1 -- 172.21.15.107:0/1068119508 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82c029720 con 0x55df57706490 2024-07-26T12:50:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.343+0000 7fb852242740 1 -- 172.21.15.107:0/1068119508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 msgr2=0x55df57ad7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.343+0000 7fb852242740 1 --2- 172.21.15.107:0/1068119508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57ad7960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb82c029440 tx=0x7fb82c01c960 comp rx=0 tx=0).stop 2024-07-26T12:50:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.343+0000 7fb852242740 1 -- 172.21.15.107:0/1068119508 shutdown_connections 2024-07-26T12:50:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.343+0000 7fb852242740 1 --2- 172.21.15.107:0/1068119508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5772f350 0x55df57adfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.343+0000 7fb852242740 1 --2- 172.21.15.107:0/1068119508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df5770e670 0x55df57ae1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.343+0000 7fb852242740 1 --2- 172.21.15.107:0/1068119508 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df57706490 0x55df57ad7960 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.343+0000 7fb852242740 1 -- 172.21.15.107:0/1068119508 >> 172.21.15.107:0/1068119508 conn(0x55df57723740 msgr2=0x55df57adf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.344+0000 7fb852242740 1 -- 172.21.15.107:0/1068119508 shutdown_connections 2024-07-26T12:50:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.344+0000 7fb852242740 1 -- 172.21.15.107:0/1068119508 wait complete. 2024-07-26T12:50:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.344+0000 7fb852242740 1 -- start start 2024-07-26T12:50:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.344+0000 7fb852242740 1 Processor -- start 2024-07-26T12:50:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb852242740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df57706490 0x55df57ae8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb852242740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 0x55df57ae3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb852242740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df5772f350 0x55df57ae3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb852242740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb8441afa80 con 0x55df57706490 2024-07-26T12:50:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb852242740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fb8441af3b0 con 0x55df5772f350 2024-07-26T12:50:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb852242740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fb8441ac5f0 con 0x55df5770e670 2024-07-26T12:50:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb852242740 10 client.0 _ll_register_callbacks cb 0x7fb82c00fae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:07.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb843e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df57706490 0x55df57ae8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb848c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 0x55df57ae3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb843e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df57706490 0x55df57ae8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36138/0 (socket says 172.21.15.107:36138) 2024-07-26T12:50:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb843e00640 1 -- 172.21.15.107:0/3387078429 learned_addr learned my addr 172.21.15.107:0/3387078429 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.345+0000 7fb843e00640 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df5772f350 0x55df57ae3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.346+0000 7fb848c00640 1 -- 172.21.15.107:0/3387078429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df5772f350 msgr2=0x55df57ae3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.346+0000 7fb848c00640 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df5772f350 0x55df57ae3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.346+0000 7fb848c00640 1 -- 172.21.15.107:0/3387078429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df57706490 msgr2=0x55df57ae8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.346+0000 7fb848c00640 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df57706490 0x55df57ae8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.346+0000 7fb848c00640 1 -- 172.21.15.107:0/3387078429 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df57b59040 con 0x55df5770e670 2024-07-26T12:50:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.346+0000 7fb848c00640 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 0x55df57ae3520 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb834014040 tx=0x7fb834005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9bf37d5aadbc3a5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.346+0000 7fb832000640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:50:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.347+0000 7fb832000640 1 -- 172.21.15.107:0/3387078429 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb834032070 con 0x55df5770e670 2024-07-26T12:50:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.347+0000 7fb852242740 1 -- 172.21.15.107:0/3387078429 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb82c01a5e0 con 0x55df5770e670 2024-07-26T12:50:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.347+0000 7fb832000640 1 -- 172.21.15.107:0/3387078429 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb83402e750 con 0x55df5770e670 2024-07-26T12:50:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.347+0000 7fb832000640 1 -- 172.21.15.107:0/3387078429 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb83403d050 con 0x55df5770e670 2024-07-26T12:50:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.347+0000 7fb852242740 10 client.25750 fetch_fsmap learned FSMap version 29 2024-07-26T12:50:07.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.347+0000 7fb852242740 1 -- 172.21.15.107:0/3387078429 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55df57b5a170 con 0x55df5770e670 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.347+0000 7fb832000640 1 -- 172.21.15.107:0/3387078429 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb82c01a5e0 con 0x55df5770e670 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.348+0000 7fb832000640 1 -- 172.21.15.107:0/3387078429 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fb8340199e0 con 0x55df5770e670 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.348+0000 7fb852242740 10 client.25750 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.348+0000 7fb852242740 10 client.25750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.348+0000 7fb852242740 20 client.25750 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.348+0000 7fb852242740 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55df576c3080 0x55df57b5c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.348+0000 7fb852242740 4 client.25750 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.348+0000 7fb852242740 1 -- 172.21.15.107:0/3387078429 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55df57a485e0 con 0x55df576c3080 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.348+0000 7fb843e00640 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55df576c3080 0x55df57b5c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.349+0000 7fb832000640 10 client.25750 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.349+0000 7fb843e00640 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55df576c3080 0x55df57b5c740 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a88196450582e9b server_cookie=6a0ae240b8ee1d29 in_seq=0 out_seq=0 2024-07-26T12:50:07.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.350+0000 7fb832000640 1 -- 172.21.15.107:0/3387078429 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55df57a485e0 con 0x55df576c3080 2024-07-26T12:50:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.350+0000 7fb832000640 10 client.25750 handle_command_reply: tid=0 2024-07-26T12:50:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.350+0000 7fb852242740 1 client.25750 shutdown 2024-07-26T12:50:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 1 -- 172.21.15.107:0/3387078429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 msgr2=0x55df57ae3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 0x55df57ae3520 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb834014040 tx=0x7fb834005d30 comp rx=0 tx=0).stop 2024-07-26T12:50:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 1 -- 172.21.15.107:0/3387078429 shutdown_connections 2024-07-26T12:50:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55df576c3080 0x55df57b5c740 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55df5772f350 0x55df57ae3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55df5770e670 0x55df57ae3520 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 1 --2- 172.21.15.107:0/3387078429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df57706490 0x55df57ae8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 1 -- 172.21.15.107:0/3387078429 >> 172.21.15.107:0/3387078429 conn(0x55df57723740 msgr2=0x55df57ae1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb832000640 0 client.25750 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 1 -- 172.21.15.107:0/3387078429 shutdown_connections 2024-07-26T12:50:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 1 -- 172.21.15.107:0/3387078429 wait complete. 2024-07-26T12:50:07.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.351+0000 7fb852242740 20 client.25750 trim_cache size 0 max 16384 2024-07-26T12:50:07.358 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:50:07.358 INFO:teuthology.orchestra.run.smithi107.stdout: "return_code": 0, 2024-07-26T12:50:07.358 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_tag": "340f6e5a-91eb-4f67-a1ae-ead087775e9c", 2024-07-26T12:50:07.358 INFO:teuthology.orchestra.run.smithi107.stdout: "mode": "asynchronous" 2024-07-26T12:50:07.358 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:50:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.352+0000 7fb84b600640 1 -- 172.21.15.107:0/1634576728 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb8240b3d40 msgr2=0x7fb8240b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.352+0000 7fb84b600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb8240b3d40 0x7fb8240b6250 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb83c005820 tx=0x7fb83c030750 comp rx=0 tx=0).stop 2024-07-26T12:50:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.352+0000 7fb84b600640 1 -- 172.21.15.107:0/1634576728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb844109820 msgr2=0x7fb84419f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.352+0000 7fb84b600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb844109820 0x7fb84419f6e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb83c02a930 tx=0x7fb83c032d40 comp rx=0 tx=0).stop 2024-07-26T12:50:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.352+0000 7fb84b600640 1 -- 172.21.15.107:0/1634576728 shutdown_connections 2024-07-26T12:50:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.352+0000 7fb84b600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fb8240b3d40 0x7fb8240b6250 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.352+0000 7fb84b600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb844109820 0x7fb84419f6e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.352+0000 7fb84b600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb844108ed0 0x7fb84419a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.352+0000 7fb84b600640 1 --2- 172.21.15.107:0/1634576728 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fb844103590 0x7fb844199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:07.359 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.352+0000 7fb84b600640 1 -- 172.21.15.107:0/1634576728 >> 172.21.15.107:0/1634576728 conn(0x7fb84407b280 msgr2=0x7fb844106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.353+0000 7fb84b600640 1 -- 172.21.15.107:0/1634576728 shutdown_connections 2024-07-26T12:50:07.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:07.353+0000 7fb84b600640 1 -- 172.21.15.107:0/1634576728 wait complete. 2024-07-26T12:50:07.436 INFO:tasks.workunit.client.0.smithi107.stderr:Note: switching to 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'. 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr:state without impacting any branches by switching back to a branch. 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr: git switch -c 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr:Or undo this operation with: 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:07.437 INFO:tasks.workunit.client.0.smithi107.stderr: git switch - 2024-07-26T12:50:07.438 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:07.438 INFO:tasks.workunit.client.0.smithi107.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-26T12:50:07.438 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:07.438 INFO:tasks.workunit.client.0.smithi107.stderr:HEAD is now at e8f88bb1fe0 Merge PR #54435 into wip-vshankar-testing-20240725.121331-debug 2024-07-26T12:50:07.445 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-07-26T12:50:07.574 INFO:tasks.workunit.client.0.smithi107.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-07-26T12:50:07.582 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T12:50:07.582 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-07-26T12:50:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:08 smithi107 ceph-mon[32659]: scrub queued for path: / 2024-07-26T12:50:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:08 smithi107 ceph-mon[32659]: scrub summary: idle+waiting paths [/] 2024-07-26T12:50:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:08 smithi107 ceph-mon[32659]: scrub summary: active paths [/] 2024-07-26T12:50:08.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:08 smithi107 ceph-mon[32659]: scrub summary: idle+waiting paths [/] 2024-07-26T12:50:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:08 smithi112 ceph-mon[37501]: scrub queued for path: / 2024-07-26T12:50:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:08 smithi112 ceph-mon[37501]: scrub summary: idle+waiting paths [/] 2024-07-26T12:50:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:08 smithi112 ceph-mon[37501]: scrub summary: active paths [/] 2024-07-26T12:50:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:08 smithi112 ceph-mon[37501]: scrub summary: idle+waiting paths [/] 2024-07-26T12:50:08.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T12:50:08.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:08 smithi132 ceph-mon[37580]: scrub queued for path: / 2024-07-26T12:50:08.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:08 smithi132 ceph-mon[37580]: scrub summary: idle+waiting paths [/] 2024-07-26T12:50:08.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:08 smithi132 ceph-mon[37580]: scrub summary: active paths [/] 2024-07-26T12:50:08.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:08 smithi132 ceph-mon[37580]: scrub summary: idle+waiting paths [/] 2024-07-26T12:50:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:50:09.027 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-07-26T12:50:09.222 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-07-26T12:50:09.301 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T12:50:09.308 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T12:50:09.308 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-07-26T12:50:09.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:09 smithi107 ceph-mon[32659]: pgmap v515: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-07-26T12:50:09.397 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T12:50:09.405 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-07-26T12:50:09.405 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-07-26T12:50:09.446 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-07-26T12:50:09.448 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-07-26T12:50:09.448 DEBUG:teuthology.orchestra.run.smithi107:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-07-26T12:50:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:09 smithi112 ceph-mon[37501]: pgmap v515: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-07-26T12:50:09.555 INFO:tasks.workunit.client.0.smithi107.stderr:+ set -e 2024-07-26T12:50:09.555 INFO:tasks.workunit.client.0.smithi107.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-07-26T12:50:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:09 smithi132 ceph-mon[37580]: pgmap v515: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 26 op/s 2024-07-26T12:50:09.563 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into 'xfstests-dev'... 2024-07-26T12:50:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:10 smithi112 ceph-mon[37501]: pgmap v516: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-07-26T12:50:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:10 smithi107 ceph-mon[32659]: pgmap v516: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-07-26T12:50:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:10 smithi132 ceph-mon[37580]: pgmap v516: 129 pgs: 129 active+clean; 1.4 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-07-26T12:50:11.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.529+0000 7fcc83e00640 1 -- 172.21.15.107:0/1324351783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc7c1057a0 msgr2=0x7fcc7c105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:11.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.529+0000 7fcc83e00640 1 --2- 172.21.15.107:0/1324351783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc7c1057a0 0x7fcc7c105bb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcc70009900 tx=0x7fcc7002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.529+0000 7fcc83e00640 1 -- 172.21.15.107:0/1324351783 shutdown_connections 2024-07-26T12:50:11.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.529+0000 7fcc83e00640 1 --2- 172.21.15.107:0/1324351783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc7c10ba30 0x7fcc7c112260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.529+0000 7fcc83e00640 1 --2- 172.21.15.107:0/1324351783 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc7c10b0e0 0x7fcc7c10b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.529+0000 7fcc83e00640 1 --2- 172.21.15.107:0/1324351783 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc7c1057a0 0x7fcc7c105bb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.529+0000 7fcc83e00640 1 -- 172.21.15.107:0/1324351783 >> 172.21.15.107:0/1324351783 conn(0x7fcc7c1011b0 msgr2=0x7fcc7c103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:11.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.530+0000 7fcc83e00640 1 -- 172.21.15.107:0/1324351783 shutdown_connections 2024-07-26T12:50:11.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.530+0000 7fcc83e00640 1 -- 172.21.15.107:0/1324351783 wait complete. 2024-07-26T12:50:11.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.531+0000 7fcc83e00640 1 Processor -- start 2024-07-26T12:50:11.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.532+0000 7fcc83e00640 1 -- start start 2024-07-26T12:50:11.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.532+0000 7fcc83e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc7c1057a0 0x7fcc7c19c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.532+0000 7fcc83e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc7c10b0e0 0x7fcc7c19c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.532+0000 7fcc83e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc7c10ba30 0x7fcc7c1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.532+0000 7fcc83e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c115730 con 0x7fcc7c10ba30 2024-07-26T12:50:11.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.532+0000 7fcc83e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c1155b0 con 0x7fcc7c10b0e0 2024-07-26T12:50:11.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.532+0000 7fcc83e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c1158b0 con 0x7fcc7c1057a0 2024-07-26T12:50:11.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.533+0000 7fcc80a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc7c10b0e0 0x7fcc7c19c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.533+0000 7fcc81400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc7c1057a0 0x7fcc7c19c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.533+0000 7fcc81e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc7c10ba30 0x7fcc7c1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.533+0000 7fcc80a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc7c10b0e0 0x7fcc7c19c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:33604/0 (socket says 172.21.15.107:33604) 2024-07-26T12:50:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.533+0000 7fcc80a00640 1 -- 172.21.15.107:0/342845339 learned_addr learned my addr 172.21.15.107:0/342845339 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.533+0000 7fcc81400640 1 -- 172.21.15.107:0/342845339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc7c10b0e0 msgr2=0x7fcc7c19c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.533+0000 7fcc81400640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc7c10b0e0 0x7fcc7c19c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.533+0000 7fcc81400640 1 -- 172.21.15.107:0/342845339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc7c10ba30 msgr2=0x7fcc7c1a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.533+0000 7fcc81400640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc7c10ba30 0x7fcc7c1a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.533+0000 7fcc81400640 1 -- 172.21.15.107:0/342845339 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7c1a2070 con 0x7fcc7c1057a0 2024-07-26T12:50:11.536 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.534+0000 7fcc81400640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc7c1057a0 0x7fcc7c19c100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcc7002a9e0 tx=0x7fcc70003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.534+0000 7fcc7a000640 1 -- 172.21.15.107:0/342845339 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc7003f070 con 0x7fcc7c1057a0 2024-07-26T12:50:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.534+0000 7fcc83e00640 1 -- 172.21.15.107:0/342845339 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc7c1a2300 con 0x7fcc7c1057a0 2024-07-26T12:50:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.534+0000 7fcc7a000640 1 -- 172.21.15.107:0/342845339 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc70031b50 con 0x7fcc7c1057a0 2024-07-26T12:50:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.534+0000 7fcc7a000640 1 -- 172.21.15.107:0/342845339 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc70035030 con 0x7fcc7c1057a0 2024-07-26T12:50:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.535+0000 7fcc83e00640 1 -- 172.21.15.107:0/342845339 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc7c1a8ac0 con 0x7fcc7c1057a0 2024-07-26T12:50:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.538+0000 7fcc8ac21740 1 Processor -- start 2024-07-26T12:50:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.538+0000 7fcc8ac21740 1 -- start start 2024-07-26T12:50:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabc87e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8ba1b0 0x55fcabc88360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8c2390 0x55fcabc888a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fcabbfebc0 con 0x55fcab8ba1b0 2024-07-26T12:50:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fcabbee0c0 con 0x55fcab8c2390 2024-07-26T12:50:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fcabbf2100 con 0x55fcab8e3070 2024-07-26T12:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc81e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabc87e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc81e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabc87e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:33868/0 (socket says 172.21.15.107:33868) 2024-07-26T12:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc81e00640 1 -- 172.21.15.107:0/2541315434 learned_addr learned my addr 172.21.15.107:0/2541315434 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc81400640 1 --2- 172.21.15.107:0/2541315434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8ba1b0 0x55fcabc88360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.539+0000 7fcc80a00640 1 --2- 172.21.15.107:0/2541315434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8c2390 0x55fcabc888a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.540+0000 7fcc81e00640 1 -- 172.21.15.107:0/2541315434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8c2390 msgr2=0x55fcabc888a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.540+0000 7fcc81e00640 1 --2- 172.21.15.107:0/2541315434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8c2390 0x55fcabc888a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.540+0000 7fcc81e00640 1 -- 172.21.15.107:0/2541315434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8ba1b0 msgr2=0x55fcabc88360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.540+0000 7fcc81e00640 1 --2- 172.21.15.107:0/2541315434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8ba1b0 0x55fcabc88360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.540+0000 7fcc81e00640 1 -- 172.21.15.107:0/2541315434 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcab94e650 con 0x55fcab8e3070 2024-07-26T12:50:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.540+0000 7fcc81e00640 1 --2- 172.21.15.107:0/2541315434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabc87e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009fd0 tx=0x7fcc74011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2e954273d85a0b1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.540+0000 7fcc7a000640 1 -- 172.21.15.107:0/342845339 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fcc70042070 con 0x7fcc7c1057a0 2024-07-26T12:50:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.541+0000 7fcc6e600640 1 -- 172.21.15.107:0/2541315434 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74004d20 con 0x55fcab8e3070 2024-07-26T12:50:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.541+0000 7fcc6e600640 1 -- 172.21.15.107:0/2541315434 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc74004030 con 0x55fcab8e3070 2024-07-26T12:50:11.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.541+0000 7fcc6e600640 1 -- 172.21.15.107:0/2541315434 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74005280 con 0x55fcab8e3070 2024-07-26T12:50:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.542+0000 7fcc8ac21740 1 -- 172.21.15.107:0/2541315434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 msgr2=0x55fcabc87e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.542+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/2541315434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabc87e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009fd0 tx=0x7fcc74011810 comp rx=0 tx=0).stop 2024-07-26T12:50:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.542+0000 7fcc8ac21740 1 -- 172.21.15.107:0/2541315434 shutdown_connections 2024-07-26T12:50:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.542+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/2541315434 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8c2390 0x55fcabc888a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.542+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/2541315434 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8ba1b0 0x55fcabc88360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.542+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/2541315434 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabc87e20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.542+0000 7fcc8ac21740 1 -- 172.21.15.107:0/2541315434 >> 172.21.15.107:0/2541315434 conn(0x55fcab8d7460 msgr2=0x55fcabc85d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.542+0000 7fcc8ac21740 1 -- 172.21.15.107:0/2541315434 shutdown_connections 2024-07-26T12:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.542+0000 7fcc8ac21740 1 -- 172.21.15.107:0/2541315434 wait complete. 2024-07-26T12:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.543+0000 7fcc8ac21740 1 -- start start 2024-07-26T12:50:11.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.543+0000 7fcc7a000640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcc640b3d60 0x7fcc640b6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.544+0000 7fcc8ac21740 1 Processor -- start 2024-07-26T12:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.544+0000 7fcc7a000640 1 -- 172.21.15.107:0/342845339 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcc70104210 con 0x7fcc7c1057a0 2024-07-26T12:50:11.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.544+0000 7fcc81e00640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcc640b3d60 0x7fcc640b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 0x55fcabc93400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc93940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 0x55fcabc93e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fcaba51700 con 0x55fcab8c2390 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55fcabb8a690 con 0x55fcab8e3070 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55fcabc96600 con 0x55fcab8ba1b0 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc8ac21740 10 client.0 _ll_register_callbacks cb 0x55fcabb9a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc80a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc93940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc80a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc93940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36184/0 (socket says 172.21.15.107:36184) 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc80a00640 1 -- 172.21.15.107:0/1564025642 learned_addr learned my addr 172.21.15.107:0/1564025642 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:11.547 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.545+0000 7fcc81400640 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 0x55fcabc93400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.546+0000 7fcc80a00640 1 -- 172.21.15.107:0/1564025642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 msgr2=0x55fcabc93400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.546+0000 7fcc80a00640 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 0x55fcabc93400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.546+0000 7fcc80a00640 1 -- 172.21.15.107:0/1564025642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 msgr2=0x55fcabc93e80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.546+0000 7fcc80a00640 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 0x55fcabc93e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.546+0000 7fcc80a00640 1 -- 172.21.15.107:0/1564025642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcabba1830 con 0x55fcab8c2390 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.546+0000 7fcc81400640 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 0x55fcabc93400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.546+0000 7fcc80a00640 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc93940 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcc68012750 tx=0x7fcc68012c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62a31013f49bd93 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.546+0000 7fcc81e00640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcc640b3d60 0x7fcc640b6270 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009270 tx=0x7fcc74000f40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.546+0000 7fcc62a00640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.546+0000 7fcc62a00640 1 -- 172.21.15.107:0/1564025642 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc68012e60 con 0x55fcab8c2390 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.547+0000 7fcc8ac21740 1 -- 172.21.15.107:0/1564025642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc74004030 con 0x55fcab8c2390 2024-07-26T12:50:11.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.547+0000 7fcc62a00640 1 -- 172.21.15.107:0/1564025642 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcc68004050 con 0x55fcab8c2390 2024-07-26T12:50:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.547+0000 7fcc62a00640 1 -- 172.21.15.107:0/1564025642 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc68025050 con 0x55fcab8c2390 2024-07-26T12:50:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.547+0000 7fcc8ac21740 10 client.15621 fetch_fsmap learned FSMap version 29 2024-07-26T12:50:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.547+0000 7fcc8ac21740 1 -- 172.21.15.107:0/1564025642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcabcaab20 con 0x55fcab8c2390 2024-07-26T12:50:11.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.547+0000 7fcc62a00640 1 -- 172.21.15.107:0/1564025642 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc74004030 con 0x55fcab8c2390 2024-07-26T12:50:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.548+0000 7fcc62a00640 1 -- 172.21.15.107:0/1564025642 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcc68003890 con 0x55fcab8c2390 2024-07-26T12:50:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.548+0000 7fcc8ac21740 10 client.15621 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:50:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.548+0000 7fcc8ac21740 10 client.15621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.548+0000 7fcc8ac21740 20 client.15621 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.548+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55fcab8a19f0 0x55fcabcae120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.548+0000 7fcc8ac21740 4 client.15621 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:50:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.548+0000 7fcc8ac21740 1 -- 172.21.15.107:0/1564025642 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcabc00b10 con 0x55fcab8a19f0 2024-07-26T12:50:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.548+0000 7fcc81400640 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55fcab8a19f0 0x55fcabcae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.549+0000 7fcc62a00640 10 client.15621 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:11.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.549+0000 7fcc81400640 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55fcab8a19f0 0x55fcabcae120 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=222c71f41f2972e2 server_cookie=e14423387bbbd109 in_seq=0 out_seq=0 2024-07-26T12:50:11.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.550+0000 7fcc62a00640 1 -- 172.21.15.107:0/1564025642 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fcabc00b10 con 0x55fcab8a19f0 2024-07-26T12:50:11.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.550+0000 7fcc62a00640 10 client.15621 handle_command_reply: tid=0 2024-07-26T12:50:11.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.550+0000 7fcc8ac21740 1 client.15621 shutdown 2024-07-26T12:50:11.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.550+0000 7fcc8ac21740 1 -- 172.21.15.107:0/1564025642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 msgr2=0x55fcabc93940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:11.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.550+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc93940 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcc68012750 tx=0x7fcc68012c10 comp rx=0 tx=0).stop 2024-07-26T12:50:11.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.551+0000 7fcc8ac21740 1 -- 172.21.15.107:0/1564025642 shutdown_connections 2024-07-26T12:50:11.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.551+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55fcab8a19f0 0x55fcabcae120 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.551+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 0x55fcabc93e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.551+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc93940 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.551+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/1564025642 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 0x55fcabc93400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.551+0000 7fcc8ac21740 1 -- 172.21.15.107:0/1564025642 >> 172.21.15.107:0/1564025642 conn(0x55fcab8d7460 msgr2=0x55fcabc8afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:11.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.551+0000 7fcc62a00640 0 client.15621 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:11.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.552+0000 7fcc8ac21740 1 -- 172.21.15.107:0/1564025642 shutdown_connections 2024-07-26T12:50:11.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.552+0000 7fcc8ac21740 1 -- 172.21.15.107:0/1564025642 wait complete. 2024-07-26T12:50:11.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.552+0000 7fcc8ac21740 20 client.15621 trim_cache size 0 max 16384 2024-07-26T12:50:11.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.585+0000 7fcc8ac21740 1 Processor -- start 2024-07-26T12:50:11.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.585+0000 7fcc8ac21740 1 -- start start 2024-07-26T12:50:11.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.586+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8ba1b0 0x55fcabc952e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.586+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc89560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.586+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabcb3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.586+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c1ae620 con 0x55fcab8c2390 2024-07-26T12:50:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.586+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c1b0160 con 0x55fcab8ba1b0 2024-07-26T12:50:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.586+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c1aecf0 con 0x55fcab8e3070 2024-07-26T12:50:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.586+0000 7fcc81400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8ba1b0 0x55fcabc952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.586+0000 7fcc81e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabcb3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.586+0000 7fcc81400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8ba1b0 0x55fcabc952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:33628/0 (socket says 172.21.15.107:33628) 2024-07-26T12:50:11.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.586+0000 7fcc81400640 1 -- 172.21.15.107:0/3363451557 learned_addr learned my addr 172.21.15.107:0/3363451557 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.587+0000 7fcc81e00640 1 -- 172.21.15.107:0/3363451557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8ba1b0 msgr2=0x55fcabc952e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.587+0000 7fcc81e00640 1 --2- 172.21.15.107:0/3363451557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8ba1b0 0x55fcabc952e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.587+0000 7fcc81e00640 1 -- 172.21.15.107:0/3363451557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 msgr2=0x55fcabc89560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.587+0000 7fcc80a00640 1 --2- 172.21.15.107:0/3363451557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc89560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.587+0000 7fcc81e00640 1 --2- 172.21.15.107:0/3363451557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc89560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.587+0000 7fcc81e00640 1 -- 172.21.15.107:0/3363451557 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcabc8ac80 con 0x55fcab8e3070 2024-07-26T12:50:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.587+0000 7fcc80a00640 1 --2- 172.21.15.107:0/3363451557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc89560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:50:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.588+0000 7fcc81e00640 1 --2- 172.21.15.107:0/3363451557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabcb3920 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009fa0 tx=0x7fcc74009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f2498dadacdfa8e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:11.590 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.588+0000 7fcc6dc00640 1 -- 172.21.15.107:0/3363451557 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74004300 con 0x55fcab8e3070 2024-07-26T12:50:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.588+0000 7fcc6dc00640 1 -- 172.21.15.107:0/3363451557 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc74009c50 con 0x55fcab8e3070 2024-07-26T12:50:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.588+0000 7fcc6dc00640 1 -- 172.21.15.107:0/3363451557 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc740236c0 con 0x55fcab8e3070 2024-07-26T12:50:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.590+0000 7fcc8ac21740 1 -- 172.21.15.107:0/3363451557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 msgr2=0x55fcabcb3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:11.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.590+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/3363451557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabcb3920 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009fa0 tx=0x7fcc74009500 comp rx=0 tx=0).stop 2024-07-26T12:50:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.590+0000 7fcc8ac21740 1 -- 172.21.15.107:0/3363451557 shutdown_connections 2024-07-26T12:50:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.590+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/3363451557 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8e3070 0x55fcabcb3920 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.590+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/3363451557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc89560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.590+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/3363451557 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8ba1b0 0x55fcabc952e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.590+0000 7fcc8ac21740 1 -- 172.21.15.107:0/3363451557 >> 172.21.15.107:0/3363451557 conn(0x55fcab8d7460 msgr2=0x55fcabc88e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.591+0000 7fcc8ac21740 1 -- 172.21.15.107:0/3363451557 shutdown_connections 2024-07-26T12:50:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.591+0000 7fcc8ac21740 1 -- 172.21.15.107:0/3363451557 wait complete. 2024-07-26T12:50:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.591+0000 7fcc8ac21740 1 -- start start 2024-07-26T12:50:11.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.592+0000 7fcc8ac21740 1 Processor -- start 2024-07-26T12:50:11.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.592+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 0x55fcabc9f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.592+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc9fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.592+0000 7fcc8ac21740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 0x55fcabd148e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.592+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c1afa90 con 0x55fcab8c2390 2024-07-26T12:50:11.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.592+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c1af3c0 con 0x55fcab8e3070 2024-07-26T12:50:11.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.592+0000 7fcc8ac21740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7c1ac600 con 0x55fcab8ba1b0 2024-07-26T12:50:11.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.592+0000 7fcc8ac21740 10 client.0 _ll_register_callbacks cb 0x7fcc68009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:11.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.593+0000 7fcc81e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 0x55fcabd148e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.593+0000 7fcc81e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 0x55fcabd148e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:33634/0 (socket says 172.21.15.107:33634) 2024-07-26T12:50:11.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.593+0000 7fcc81e00640 1 -- 172.21.15.107:0/4087006100 learned_addr learned my addr 172.21.15.107:0/4087006100 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:11.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.593+0000 7fcc81400640 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 0x55fcabc9f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.593+0000 7fcc80a00640 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc9fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.593+0000 7fcc81e00640 1 -- 172.21.15.107:0/4087006100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 msgr2=0x55fcabc9f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.593+0000 7fcc81e00640 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 0x55fcabc9f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.593+0000 7fcc81e00640 1 -- 172.21.15.107:0/4087006100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 msgr2=0x55fcabc9fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.593+0000 7fcc81e00640 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc9fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.593+0000 7fcc81e00640 1 -- 172.21.15.107:0/4087006100 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcabd14e20 con 0x55fcab8e3070 2024-07-26T12:50:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.594+0000 7fcc81e00640 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 0x55fcabd148e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcc74005570 tx=0x7fcc740055a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e64b01247347c069 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.594+0000 7fcc62a00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:50:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.594+0000 7fcc62a00640 1 -- 172.21.15.107:0/4087006100 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc74004300 con 0x55fcab8e3070 2024-07-26T12:50:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.594+0000 7fcc8ac21740 1 -- 172.21.15.107:0/4087006100 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fcabbfc5b0 con 0x55fcab8e3070 2024-07-26T12:50:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.595+0000 7fcc62a00640 1 -- 172.21.15.107:0/4087006100 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcc7402c040 con 0x55fcab8e3070 2024-07-26T12:50:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.595+0000 7fcc62a00640 1 -- 172.21.15.107:0/4087006100 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc74003f00 con 0x55fcab8e3070 2024-07-26T12:50:11.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.595+0000 7fcc8ac21740 10 client.25757 fetch_fsmap learned FSMap version 29 2024-07-26T12:50:11.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.595+0000 7fcc8ac21740 1 -- 172.21.15.107:0/4087006100 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fcabd15f30 con 0x55fcab8e3070 2024-07-26T12:50:11.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.595+0000 7fcc62a00640 1 -- 172.21.15.107:0/4087006100 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55fcabbfc5b0 con 0x55fcab8e3070 2024-07-26T12:50:11.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.596+0000 7fcc62a00640 1 -- 172.21.15.107:0/4087006100 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fcc74018070 con 0x55fcab8e3070 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.596+0000 7fcc8ac21740 10 client.25757 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.596+0000 7fcc8ac21740 10 client.25757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.596+0000 7fcc8ac21740 20 client.25757 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.596+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55fcab876da0 0x55fcabd18500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.596+0000 7fcc8ac21740 4 client.25757 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.596+0000 7fcc8ac21740 1 -- 172.21.15.107:0/4087006100 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fcabbf9cf0 con 0x55fcab876da0 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.596+0000 7fcc81400640 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55fcab876da0 0x55fcabd18500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.597+0000 7fcc81400640 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55fcab876da0 0x55fcabd18500 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b5397ab4cc80ec3 server_cookie=8ace93efd56f3c3 in_seq=0 out_seq=0 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.597+0000 7fcc62a00640 10 client.25757 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.597+0000 7fcc62a00640 1 -- 172.21.15.107:0/4087006100 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fcabbf9cf0 con 0x55fcab876da0 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.597+0000 7fcc62a00640 10 client.25757 handle_command_reply: tid=0 2024-07-26T12:50:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.597+0000 7fcc8ac21740 1 client.25757 shutdown 2024-07-26T12:50:11.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.597+0000 7fcc8ac21740 1 -- 172.21.15.107:0/4087006100 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 msgr2=0x55fcabd148e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:11.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.597+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 0x55fcabd148e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcc74005570 tx=0x7fcc740055a0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.598+0000 7fcc8ac21740 1 -- 172.21.15.107:0/4087006100 shutdown_connections 2024-07-26T12:50:11.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.598+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55fcab876da0 0x55fcabd18500 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.598+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55fcab8e3070 0x55fcabd148e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.598+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fcab8c2390 0x55fcabc9fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.598+0000 7fcc8ac21740 1 --2- 172.21.15.107:0/4087006100 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55fcab8ba1b0 0x55fcabc9f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.598+0000 7fcc62a00640 0 client.25757 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.598+0000 7fcc8ac21740 1 -- 172.21.15.107:0/4087006100 >> 172.21.15.107:0/4087006100 conn(0x55fcab8d7460 msgr2=0x55fcabc895e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.598+0000 7fcc8ac21740 1 -- 172.21.15.107:0/4087006100 shutdown_connections 2024-07-26T12:50:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.598+0000 7fcc8ac21740 1 -- 172.21.15.107:0/4087006100 wait complete. 2024-07-26T12:50:11.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.598+0000 7fcc8ac21740 20 client.25757 trim_cache size 0 max 16384 2024-07-26T12:50:11.601 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:50:11.602 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T12:50:11.602 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": { 2024-07-26T12:50:11.602 INFO:teuthology.orchestra.run.smithi107.stdout: "340f6e5a-91eb-4f67-a1ae-ead087775e9c": { 2024-07-26T12:50:11.602 INFO:teuthology.orchestra.run.smithi107.stdout: "path": "/", 2024-07-26T12:50:11.602 INFO:teuthology.orchestra.run.smithi107.stdout: "tag": "340f6e5a-91eb-4f67-a1ae-ead087775e9c", 2024-07-26T12:50:11.602 INFO:teuthology.orchestra.run.smithi107.stdout: "options": "recursive,force" 2024-07-26T12:50:11.602 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-07-26T12:50:11.602 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-07-26T12:50:11.602 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:50:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.599+0000 7fcc83e00640 1 -- 172.21.15.107:0/342845339 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcc640b3d60 msgr2=0x7fcc640b6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.599+0000 7fcc83e00640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcc640b3d60 0x7fcc640b6270 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcc74009270 tx=0x7fcc74000f40 comp rx=0 tx=0).stop 2024-07-26T12:50:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.599+0000 7fcc83e00640 1 -- 172.21.15.107:0/342845339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc7c1057a0 msgr2=0x7fcc7c19c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.599+0000 7fcc83e00640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc7c1057a0 0x7fcc7c19c100 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcc7002a9e0 tx=0x7fcc70003b90 comp rx=0 tx=0).stop 2024-07-26T12:50:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.600+0000 7fcc83e00640 1 -- 172.21.15.107:0/342845339 shutdown_connections 2024-07-26T12:50:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.600+0000 7fcc83e00640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fcc640b3d60 0x7fcc640b6270 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.600+0000 7fcc83e00640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc7c10ba30 0x7fcc7c1a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.600+0000 7fcc83e00640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fcc7c10b0e0 0x7fcc7c19c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.600+0000 7fcc83e00640 1 --2- 172.21.15.107:0/342845339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc7c1057a0 0x7fcc7c19c100 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcc7002a9e0 tx=0x7fcc70003b90 comp rx=0 tx=0).stop 2024-07-26T12:50:11.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.600+0000 7fcc83e00640 1 -- 172.21.15.107:0/342845339 >> 172.21.15.107:0/342845339 conn(0x7fcc7c1011b0 msgr2=0x7fcc7c1035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.601+0000 7fcc83e00640 1 -- 172.21.15.107:0/342845339 shutdown_connections 2024-07-26T12:50:11.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:11.601+0000 7fcc83e00640 1 -- 172.21.15.107:0/342845339 wait complete. 2024-07-26T12:50:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:12 smithi107 ceph-mon[32659]: scrub summary: idle 2024-07-26T12:50:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:12 smithi112 ceph-mon[37501]: scrub summary: idle 2024-07-26T12:50:12.538 INFO:tasks.cephfs.filesystem:scrub status for tag:340f6e5a-91eb-4f67-a1ae-ead087775e9c - {'path': '/', 'tag': '340f6e5a-91eb-4f67-a1ae-ead087775e9c', 'options': 'recursive,force'} 2024-07-26T12:50:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:12 smithi132 ceph-mon[37580]: scrub summary: idle 2024-07-26T12:50:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:13 smithi112 ceph-mon[37501]: pgmap v517: 129 pgs: 129 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 57 KiB/s wr, 29 op/s 2024-07-26T12:50:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:13 smithi107 ceph-mon[32659]: pgmap v517: 129 pgs: 129 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 57 KiB/s wr, 29 op/s 2024-07-26T12:50:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:13 smithi132 ceph-mon[37580]: pgmap v517: 129 pgs: 129 active+clean; 1.8 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 57 KiB/s wr, 29 op/s 2024-07-26T12:50:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:14 smithi112 ceph-mon[37501]: pgmap v518: 129 pgs: 129 active+clean; 11 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 838 KiB/s wr, 15 op/s 2024-07-26T12:50:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:14 smithi107 ceph-mon[32659]: pgmap v518: 129 pgs: 129 active+clean; 11 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 838 KiB/s wr, 15 op/s 2024-07-26T12:50:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:14 smithi132 ceph-mon[37580]: pgmap v518: 129 pgs: 129 active+clean; 11 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 838 KiB/s wr, 15 op/s 2024-07-26T12:50:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:16 smithi112 ceph-mon[37501]: pgmap v519: 129 pgs: 129 active+clean; 15 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-07-26T12:50:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:16 smithi107 ceph-mon[32659]: pgmap v519: 129 pgs: 129 active+clean; 15 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-07-26T12:50:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:16 smithi132 ceph-mon[37580]: pgmap v519: 129 pgs: 129 active+clean; 15 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-07-26T12:50:18.413 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.411+0000 7f31a7000640 1 -- 172.21.15.107:0/3551864979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31a0075b30 msgr2=0x7f31a010f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:18.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.411+0000 7f31a7000640 1 --2- 172.21.15.107:0/3551864979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31a0075b30 0x7f31a010f370 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f318c009900 tx=0x7f318c02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.411+0000 7f31a7000640 1 -- 172.21.15.107:0/3551864979 shutdown_connections 2024-07-26T12:50:18.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.411+0000 7f31a7000640 1 --2- 172.21.15.107:0/3551864979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31a010f8b0 0x7f31a0111d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.411+0000 7f31a7000640 1 --2- 172.21.15.107:0/3551864979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31a0075b30 0x7f31a010f370 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.411+0000 7f31a7000640 1 --2- 172.21.15.107:0/3551864979 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31a00751e0 0x7f31a00755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.411+0000 7f31a7000640 1 -- 172.21.15.107:0/3551864979 >> 172.21.15.107:0/3551864979 conn(0x7f31a00fedc0 msgr2=0x7f31a0101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:18.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.412+0000 7f31a7000640 1 -- 172.21.15.107:0/3551864979 shutdown_connections 2024-07-26T12:50:18.414 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.412+0000 7f31a7000640 1 -- 172.21.15.107:0/3551864979 wait complete. 2024-07-26T12:50:18.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.413+0000 7f31a7000640 1 Processor -- start 2024-07-26T12:50:18.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.413+0000 7f31a7000640 1 -- start start 2024-07-26T12:50:18.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.413+0000 7f31a7000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31a00751e0 0x7f31a019bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.413+0000 7f31a7000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31a0075b30 0x7f31a019c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.413+0000 7f31a7000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31a010f8b0 0x7f31a01a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.413+0000 7f31a7000640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0115480 con 0x7f31a0075b30 2024-07-26T12:50:18.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.413+0000 7f31a7000640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0115300 con 0x7f31a00751e0 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.413+0000 7f31a7000640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0115600 con 0x7f31a010f8b0 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a5200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31a0075b30 0x7f31a019c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a5c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31a00751e0 0x7f31a019bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a5200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31a0075b30 0x7f31a019c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59256/0 (socket says 172.21.15.107:59256) 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a5200640 1 -- 172.21.15.107:0/3256594736 learned_addr learned my addr 172.21.15.107:0/3256594736 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a6600640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31a010f8b0 0x7f31a01a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a5c00640 1 -- 172.21.15.107:0/3256594736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31a010f8b0 msgr2=0x7f31a01a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a5c00640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31a010f8b0 0x7f31a01a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a5c00640 1 -- 172.21.15.107:0/3256594736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31a0075b30 msgr2=0x7f31a019c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a5c00640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31a0075b30 0x7f31a019c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a5c00640 1 -- 172.21.15.107:0/3256594736 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a01a1e30 con 0x7f31a00751e0 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a6600640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31a010f8b0 0x7f31a01a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:18.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.414+0000 7f31a5200640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31a0075b30 0x7f31a019c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.415+0000 7f31a5c00640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31a00751e0 0x7f31a019bf00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f319000ec80 tx=0x7f319000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.415+0000 7f319e000640 1 -- 172.21.15.107:0/3256594736 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190003b70 con 0x7f31a00751e0 2024-07-26T12:50:18.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.415+0000 7f319e000640 1 -- 172.21.15.107:0/3256594736 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3190003d10 con 0x7f31a00751e0 2024-07-26T12:50:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.415+0000 7f31a7000640 1 -- 172.21.15.107:0/3256594736 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a01a2120 con 0x7f31a00751e0 2024-07-26T12:50:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.415+0000 7f319e000640 1 -- 172.21.15.107:0/3256594736 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190010910 con 0x7f31a00751e0 2024-07-26T12:50:18.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.415+0000 7f31a7000640 1 -- 172.21.15.107:0/3256594736 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a01a88b0 con 0x7f31a00751e0 2024-07-26T12:50:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.418+0000 7f31af5f0740 1 Processor -- start 2024-07-26T12:50:18.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.418+0000 7f31af5f0740 1 -- start start 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 0x56040ac62c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8962f0 0x56040ac63180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac636c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31af5f0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56040abd7270 con 0x56040a89e4d0 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31af5f0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56040abc6770 con 0x56040a8bf1b0 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31af5f0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56040abca7b0 con 0x56040a8962f0 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f319e000640 1 -- 172.21.15.107:0/3256594736 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f3190004560 con 0x7f31a00751e0 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31a5200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac636c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31a5200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac636c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59260/0 (socket says 172.21.15.107:59260) 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31a5200640 1 -- 172.21.15.107:0/776250887 learned_addr learned my addr 172.21.15.107:0/776250887 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:18.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31a6600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 0x56040ac62c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.422 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.419+0000 7f31a5c00640 1 --2- 172.21.15.107:0/776250887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8962f0 0x56040ac63180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.420+0000 7f31a5200640 1 -- 172.21.15.107:0/776250887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8962f0 msgr2=0x56040ac63180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.420+0000 7f31a5200640 1 --2- 172.21.15.107:0/776250887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8962f0 0x56040ac63180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.420+0000 7f31a5200640 1 -- 172.21.15.107:0/776250887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 msgr2=0x56040ac62c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.420+0000 7f31a5200640 1 --2- 172.21.15.107:0/776250887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 0x56040ac62c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.420+0000 7f31a5200640 1 -- 172.21.15.107:0/776250887 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56040a92a790 con 0x56040a89e4d0 2024-07-26T12:50:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.420+0000 7f31a6600640 1 --2- 172.21.15.107:0/776250887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 0x56040ac62c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.420+0000 7f31a5200640 1 --2- 172.21.15.107:0/776250887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac636c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f318c006010 tx=0x7f318c005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bd9fe691a5f3bd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.420+0000 7f3196a00640 1 -- 172.21.15.107:0/776250887 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c035040 con 0x56040a89e4d0 2024-07-26T12:50:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.420+0000 7f3196a00640 1 -- 172.21.15.107:0/776250887 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318c031720 con 0x56040a89e4d0 2024-07-26T12:50:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.420+0000 7f3196a00640 1 -- 172.21.15.107:0/776250887 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c0049c0 con 0x56040a89e4d0 2024-07-26T12:50:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.421+0000 7f31af5f0740 1 -- 172.21.15.107:0/776250887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 msgr2=0x56040ac636c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.421+0000 7f31af5f0740 1 --2- 172.21.15.107:0/776250887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac636c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f318c006010 tx=0x7f318c005e20 comp rx=0 tx=0).stop 2024-07-26T12:50:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f31af5f0740 1 -- 172.21.15.107:0/776250887 shutdown_connections 2024-07-26T12:50:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f31af5f0740 1 --2- 172.21.15.107:0/776250887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac636c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f31af5f0740 1 --2- 172.21.15.107:0/776250887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8962f0 0x56040ac63180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f31af5f0740 1 --2- 172.21.15.107:0/776250887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 0x56040ac62c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f31af5f0740 1 -- 172.21.15.107:0/776250887 >> 172.21.15.107:0/776250887 conn(0x56040a8b3300 msgr2=0x56040ac60b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f31af5f0740 1 -- 172.21.15.107:0/776250887 shutdown_connections 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f31af5f0740 1 -- 172.21.15.107:0/776250887 wait complete. 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f319e000640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31840b3cd0 0x7f31840b61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f319e000640 1 -- 172.21.15.107:0/3256594736 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f31900e4a80 con 0x7f31a00751e0 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f31a5200640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31840b3cd0 0x7f31840b61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.422+0000 7f31af5f0740 1 -- start start 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31af5f0740 1 Processor -- start 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31a5200640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31840b3cd0 0x7f31840b61e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f318c0053c0 tx=0x7f318c031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 0x56040ac6e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac6ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8bf1b0 0x56040ac7a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31af5f0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56040ab00ab0 con 0x56040a89e4d0 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31af5f0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x56040ab63d50 con 0x56040a8962f0 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31af5f0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x56040ac71360 con 0x56040a8bf1b0 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31af5f0740 10 client.0 _ll_register_callbacks cb 0x56040a8d6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31a5200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac6ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.423+0000 7f31a5c00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 0x56040ac6e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f31a5200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac6ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59266/0 (socket says 172.21.15.107:59266) 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f31a5200640 1 -- 172.21.15.107:0/15964893 learned_addr learned my addr 172.21.15.107:0/15964893 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f31a6600640 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8bf1b0 0x56040ac7a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f31a5200640 1 -- 172.21.15.107:0/15964893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8bf1b0 msgr2=0x56040ac7a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f31a5200640 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8bf1b0 0x56040ac7a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f31a5200640 1 -- 172.21.15.107:0/15964893 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 msgr2=0x56040ac6e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f31a5200640 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 0x56040ac6e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f31a5200640 1 -- 172.21.15.107:0/15964893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56040a9a3620 con 0x56040a89e4d0 2024-07-26T12:50:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f31a6600640 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8bf1b0 0x56040ac7a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f31a5200640 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac6ecb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f318c0042c0 tx=0x7f318c005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe21ef3e533ffa61 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.424+0000 7f318b400640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:50:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.425+0000 7f318b400640 1 -- 172.21.15.107:0/15964893 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c035040 con 0x56040a89e4d0 2024-07-26T12:50:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.425+0000 7f318b400640 1 -- 172.21.15.107:0/15964893 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f318c003800 con 0x56040a89e4d0 2024-07-26T12:50:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.425+0000 7f31af5f0740 1 -- 172.21.15.107:0/15964893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f318c031720 con 0x56040a89e4d0 2024-07-26T12:50:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.425+0000 7f318b400640 1 -- 172.21.15.107:0/15964893 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f318c064050 con 0x56040a89e4d0 2024-07-26T12:50:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.425+0000 7f31af5f0740 10 client.15636 fetch_fsmap learned FSMap version 29 2024-07-26T12:50:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.425+0000 7f31af5f0740 1 -- 172.21.15.107:0/15964893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56040ac7d370 con 0x56040a89e4d0 2024-07-26T12:50:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.425+0000 7f318b400640 1 -- 172.21.15.107:0/15964893 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318c031720 con 0x56040a89e4d0 2024-07-26T12:50:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.425+0000 7f318b400640 1 -- 172.21.15.107:0/15964893 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f318c004bd0 con 0x56040a89e4d0 2024-07-26T12:50:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.426+0000 7f31af5f0740 10 client.15636 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:50:18.431 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.426+0000 7f31af5f0740 10 client.15636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.426+0000 7f31af5f0740 20 client.15636 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.426+0000 7f31af5f0740 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56040a87dd80 0x56040ac806d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.426+0000 7f31af5f0740 4 client.15636 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.426+0000 7f31af5f0740 1 -- 172.21.15.107:0/15964893 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56040abd91c0 con 0x56040a87dd80 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.426+0000 7f31a5c00640 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56040a87dd80 0x56040ac806d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.426+0000 7f31a5c00640 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56040a87dd80 0x56040ac806d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=241bf6afbf51c591 server_cookie=1a3b02aec76eb025 in_seq=0 out_seq=0 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.426+0000 7f318b400640 10 client.15636 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.427+0000 7f318b400640 1 -- 172.21.15.107:0/15964893 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56040abd91c0 con 0x56040a87dd80 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.427+0000 7f318b400640 10 client.15636 handle_command_reply: tid=0 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.427+0000 7f31af5f0740 1 client.15636 shutdown 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 1 -- 172.21.15.107:0/15964893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 msgr2=0x56040ac6ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac6ecb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f318c0042c0 tx=0x7f318c005420 comp rx=0 tx=0).stop 2024-07-26T12:50:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 1 -- 172.21.15.107:0/15964893 shutdown_connections 2024-07-26T12:50:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56040a87dd80 0x56040ac806d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8bf1b0 0x56040ac7a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac6ecb0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f318b400640 0 client.15636 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 1 --2- 172.21.15.107:0/15964893 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 0x56040ac6e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 1 -- 172.21.15.107:0/15964893 >> 172.21.15.107:0/15964893 conn(0x56040a8b3300 msgr2=0x56040ac63c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 1 -- 172.21.15.107:0/15964893 shutdown_connections 2024-07-26T12:50:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 1 -- 172.21.15.107:0/15964893 wait complete. 2024-07-26T12:50:18.433 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.428+0000 7f31af5f0740 20 client.15636 trim_cache size 0 max 16384 2024-07-26T12:50:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.444+0000 7f31af5f0740 1 Processor -- start 2024-07-26T12:50:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.444+0000 7f31af5f0740 1 -- start start 2024-07-26T12:50:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8962f0 0x56040ac68d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac64380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 0x56040ac7a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31af5f0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f31a01b27e0 con 0x56040a89e4d0 2024-07-26T12:50:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31af5f0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f31a01b4320 con 0x56040a8bf1b0 2024-07-26T12:50:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31af5f0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f31a01b2eb0 con 0x56040a8962f0 2024-07-26T12:50:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31a5200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac64380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31a5200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac64380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59282/0 (socket says 172.21.15.107:59282) 2024-07-26T12:50:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31a5200640 1 -- 172.21.15.107:0/3404711414 learned_addr learned my addr 172.21.15.107:0/3404711414 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31a5c00640 1 --2- 172.21.15.107:0/3404711414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8962f0 0x56040ac68d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31a6600640 1 --2- 172.21.15.107:0/3404711414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 0x56040ac7a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31a5200640 1 -- 172.21.15.107:0/3404711414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8962f0 msgr2=0x56040ac68d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.445+0000 7f31a5200640 1 --2- 172.21.15.107:0/3404711414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8962f0 0x56040ac68d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.446+0000 7f31a5200640 1 -- 172.21.15.107:0/3404711414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 msgr2=0x56040ac7a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.446+0000 7f31a5200640 1 --2- 172.21.15.107:0/3404711414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 0x56040ac7a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.446+0000 7f31a5200640 1 -- 172.21.15.107:0/3404711414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56040ac61430 con 0x56040a89e4d0 2024-07-26T12:50:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.446+0000 7f31a5200640 1 --2- 172.21.15.107:0/3404711414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac64380 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f318c004fa0 tx=0x7f318c02a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37fdd65d144fd64 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.446+0000 7f3196000640 1 -- 172.21.15.107:0/3404711414 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c035040 con 0x56040a89e4d0 2024-07-26T12:50:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.446+0000 7f3196000640 1 -- 172.21.15.107:0/3404711414 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318c056070 con 0x56040a89e4d0 2024-07-26T12:50:18.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.446+0000 7f3196000640 1 -- 172.21.15.107:0/3404711414 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c003a60 con 0x56040a89e4d0 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.447+0000 7f31af5f0740 1 -- 172.21.15.107:0/3404711414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 msgr2=0x56040ac64380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.447+0000 7f31af5f0740 1 --2- 172.21.15.107:0/3404711414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac64380 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f318c004fa0 tx=0x7f318c02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.447+0000 7f31af5f0740 1 -- 172.21.15.107:0/3404711414 shutdown_connections 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.447+0000 7f31af5f0740 1 --2- 172.21.15.107:0/3404711414 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8bf1b0 0x56040ac7a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.447+0000 7f31af5f0740 1 --2- 172.21.15.107:0/3404711414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a89e4d0 0x56040ac64380 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.447+0000 7f31af5f0740 1 --2- 172.21.15.107:0/3404711414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a8962f0 0x56040ac68d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.447+0000 7f31af5f0740 1 -- 172.21.15.107:0/3404711414 >> 172.21.15.107:0/3404711414 conn(0x56040a8b3300 msgr2=0x56040ac63c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.447+0000 7f31af5f0740 1 -- 172.21.15.107:0/3404711414 shutdown_connections 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.447+0000 7f31af5f0740 1 -- 172.21.15.107:0/3404711414 wait complete. 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.448+0000 7f31af5f0740 1 -- start start 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.448+0000 7f31af5f0740 1 Processor -- start 2024-07-26T12:50:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.448+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 0x56040ac71600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.448+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a89e4d0 0x56040ac71b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.448+0000 7f31af5f0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a8bf1b0 0x56040ac72080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.448+0000 7f31af5f0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f31a01b3c50 con 0x56040a8bf1b0 2024-07-26T12:50:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.448+0000 7f31af5f0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f31a01b3580 con 0x56040a8962f0 2024-07-26T12:50:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.448+0000 7f31af5f0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f31a01b07c0 con 0x56040a89e4d0 2024-07-26T12:50:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.448+0000 7f31af5f0740 10 client.0 _ll_register_callbacks cb 0x7f318c040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a6600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a8bf1b0 0x56040ac72080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a6600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a8bf1b0 0x56040ac72080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59284/0 (socket says 172.21.15.107:59284) 2024-07-26T12:50:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a6600640 1 -- 172.21.15.107:0/3580585192 learned_addr learned my addr 172.21.15.107:0/3580585192 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a5c00640 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 0x56040ac71600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a5200640 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a89e4d0 0x56040ac71b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a6600640 1 -- 172.21.15.107:0/3580585192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a89e4d0 msgr2=0x56040ac71b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a6600640 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a89e4d0 0x56040ac71b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a6600640 1 -- 172.21.15.107:0/3580585192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 msgr2=0x56040ac71600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a6600640 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 0x56040ac71600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a6600640 1 -- 172.21.15.107:0/3580585192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56040ace6cb0 con 0x56040a8bf1b0 2024-07-26T12:50:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a5c00640 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 0x56040ac71600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f31a6600640 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a8bf1b0 0x56040ac72080 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f31980058c0 tx=0x7f3198012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471c1c9ff0c775ca server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.449+0000 7f318b400640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:50:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.450+0000 7f318b400640 1 -- 172.21.15.107:0/3580585192 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319801d670 con 0x56040a8bf1b0 2024-07-26T12:50:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.450+0000 7f318b400640 1 -- 172.21.15.107:0/3580585192 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f319801d810 con 0x56040a8bf1b0 2024-07-26T12:50:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.450+0000 7f31af5f0740 1 -- 172.21.15.107:0/3580585192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f318c056070 con 0x56040a8bf1b0 2024-07-26T12:50:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.450+0000 7f318b400640 1 -- 172.21.15.107:0/3580585192 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f319802b050 con 0x56040a8bf1b0 2024-07-26T12:50:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.450+0000 7f31af5f0740 10 client.15642 fetch_fsmap learned FSMap version 29 2024-07-26T12:50:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.450+0000 7f31af5f0740 1 -- 172.21.15.107:0/3580585192 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56040ace7f40 con 0x56040a8bf1b0 2024-07-26T12:50:18.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.450+0000 7f318b400640 1 -- 172.21.15.107:0/3580585192 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318c056070 con 0x56040a8bf1b0 2024-07-26T12:50:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.450+0000 7f318b400640 1 -- 172.21.15.107:0/3580585192 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f3198010a00 con 0x56040a8bf1b0 2024-07-26T12:50:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.451+0000 7f31af5f0740 10 client.15642 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T12:50:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.451+0000 7f31af5f0740 10 client.15642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.451+0000 7f31af5f0740 20 client.15642 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.451+0000 7f31af5f0740 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56040a8531c0 0x56040acea780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:18.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.451+0000 7f31af5f0740 4 client.15642 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T12:50:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.451+0000 7f31af5f0740 1 -- 172.21.15.107:0/3580585192 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56040abd4c60 con 0x56040a8531c0 2024-07-26T12:50:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.451+0000 7f31a5c00640 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56040a8531c0 0x56040acea780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.451+0000 7f318b400640 10 client.15642 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.451+0000 7f31a5c00640 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56040a8531c0 0x56040acea780 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2acdbd67a79038f server_cookie=3119968f4971a9a5 in_seq=0 out_seq=0 2024-07-26T12:50:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f318b400640 1 -- 172.21.15.107:0/3580585192 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56040abd4c60 con 0x56040a8531c0 2024-07-26T12:50:18.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f318b400640 10 client.15642 handle_command_reply: tid=0 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 client.15642 shutdown 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 -- 172.21.15.107:0/3580585192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a8bf1b0 msgr2=0x56040ac72080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a8bf1b0 0x56040ac72080 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f31980058c0 tx=0x7f3198012f90 comp rx=0 tx=0).stop 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 -- 172.21.15.107:0/3580585192 shutdown_connections 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x56040a8531c0 0x56040acea780 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56040a8bf1b0 0x56040ac72080 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56040a89e4d0 0x56040ac71b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 --2- 172.21.15.107:0/3580585192 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x56040a8962f0 0x56040ac71600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f318b400640 0 client.15642 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 -- 172.21.15.107:0/3580585192 >> 172.21.15.107:0/3580585192 conn(0x56040a8b3300 msgr2=0x56040ac60ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 -- 172.21.15.107:0/3580585192 shutdown_connections 2024-07-26T12:50:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.778+0000 7f31af5f0740 1 -- 172.21.15.107:0/3580585192 wait complete. 2024-07-26T12:50:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31af5f0740 20 client.15642 trim_cache size 0 max 16384 2024-07-26T12:50:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-07-26T12:50:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-07-26T12:50:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-07-26T12:50:18.782 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-07-26T12:50:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31a7000640 1 -- 172.21.15.107:0/3256594736 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31840b3cd0 msgr2=0x7f31840b61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31a7000640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31840b3cd0 0x7f31840b61e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f318c0053c0 tx=0x7f318c031400 comp rx=0 tx=0).stop 2024-07-26T12:50:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31a7000640 1 -- 172.21.15.107:0/3256594736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31a00751e0 msgr2=0x7f31a019bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31a7000640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31a00751e0 0x7f31a019bf00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f319000ec80 tx=0x7f319000c6a0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31a7000640 1 -- 172.21.15.107:0/3256594736 shutdown_connections 2024-07-26T12:50:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31a7000640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f31840b3cd0 0x7f31840b61e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31a7000640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f31a010f8b0 0x7f31a01a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31a7000640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31a0075b30 0x7f31a019c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31a7000640 1 --2- 172.21.15.107:0/3256594736 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f31a00751e0 0x7f31a019bf00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.779+0000 7f31a7000640 1 -- 172.21.15.107:0/3256594736 >> 172.21.15.107:0/3256594736 conn(0x7f31a00fedc0 msgr2=0x7f31a0109450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.780+0000 7f31a7000640 1 -- 172.21.15.107:0/3256594736 shutdown_connections 2024-07-26T12:50:18.784 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:18.780+0000 7f31a7000640 1 -- 172.21.15.107:0/3256594736 wait complete. 2024-07-26T12:50:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:18 smithi112 ceph-mon[37501]: pgmap v520: 129 pgs: 129 active+clean; 16 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-26T12:50:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:18 smithi107 ceph-mon[32659]: pgmap v520: 129 pgs: 129 active+clean; 16 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-26T12:50:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:18 smithi132 ceph-mon[37580]: pgmap v520: 129 pgs: 129 active+clean; 16 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-07-26T12:50:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:20 smithi112 ceph-mon[37501]: pgmap v521: 129 pgs: 129 active+clean; 16 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.2 MiB/s wr, 44 op/s 2024-07-26T12:50:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:20 smithi107 ceph-mon[32659]: pgmap v521: 129 pgs: 129 active+clean; 16 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.2 MiB/s wr, 44 op/s 2024-07-26T12:50:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:20 smithi132 ceph-mon[37580]: pgmap v521: 129 pgs: 129 active+clean; 16 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 1.2 MiB/s wr, 44 op/s 2024-07-26T12:50:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:22 smithi112 ceph-mon[37501]: pgmap v522: 129 pgs: 129 active+clean; 18 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-07-26T12:50:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:22 smithi107 ceph-mon[32659]: pgmap v522: 129 pgs: 129 active+clean; 18 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-07-26T12:50:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:22 smithi132 ceph-mon[37580]: pgmap v522: 129 pgs: 129 active+clean; 18 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-07-26T12:50:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:50:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:24 smithi112 ceph-mon[37501]: pgmap v523: 129 pgs: 129 active+clean; 19 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-07-26T12:50:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:50:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:24 smithi107 ceph-mon[32659]: pgmap v523: 129 pgs: 129 active+clean; 19 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-07-26T12:50:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:50:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:24 smithi132 ceph-mon[37580]: pgmap v523: 129 pgs: 129 active+clean; 19 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 1.5 MiB/s wr, 169 op/s 2024-07-26T12:50:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:26 smithi112 ceph-mon[37501]: pgmap v524: 129 pgs: 129 active+clean; 19 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 770 KiB/s wr, 174 op/s 2024-07-26T12:50:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:26 smithi107 ceph-mon[32659]: pgmap v524: 129 pgs: 129 active+clean; 19 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 770 KiB/s wr, 174 op/s 2024-07-26T12:50:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:26 smithi132 ceph-mon[37580]: pgmap v524: 129 pgs: 129 active+clean; 19 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 770 KiB/s wr, 174 op/s 2024-07-26T12:50:28.200 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:27 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[68819]: *** Caught signal (Segmentation fault) ** 2024-07-26T12:50:28.200 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:27 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[68819]: in thread 7fec88b74640 thread_name:ms_dispatch 2024-07-26T12:50:28.200 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:28 smithi112 podman[71710]: 2024-07-26 12:50:28.015747459 +0000 UTC m=+0.019482227 container died 93b16afe7021e1f4efdf1baade876c8322e78fa93de44d75ba5f7f3bf4cb3a8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-07-26T12:50:28.996 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:28 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:28.996 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:28 smithi112 ceph-mon[37501]: pgmap v525: 129 pgs: 129 active+clean; 20 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 545 KiB/s wr, 183 op/s 2024-07-26T12:50:28.997 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:28 smithi112 podman[71710]: 2024-07-26 12:50:28.742889546 +0000 UTC m=+0.746624313 container remove 93b16afe7021e1f4efdf1baade876c8322e78fa93de44d75ba5f7f3bf4cb3a8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:50:28.997 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:28 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Main process exited, code=exited, status=139/n/a 2024-07-26T12:50:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:28 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:28 smithi107 ceph-mon[32659]: pgmap v525: 129 pgs: 129 active+clean; 20 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 545 KiB/s wr, 183 op/s 2024-07-26T12:50:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:28 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:28 smithi132 ceph-mon[37580]: pgmap v525: 129 pgs: 129 active+clean; 20 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 545 KiB/s wr, 183 op/s 2024-07-26T12:50:29.450 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:28 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T12:50:29.450 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:28 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Consumed 1.749s CPU time. 2024-07-26T12:50:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:30 smithi112 ceph-mon[37501]: pgmap v526: 129 pgs: 129 active+clean; 23 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 739 KiB/s wr, 185 op/s 2024-07-26T12:50:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:30 smithi107 ceph-mon[32659]: pgmap v526: 129 pgs: 129 active+clean; 23 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 739 KiB/s wr, 185 op/s 2024-07-26T12:50:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:30 smithi132 ceph-mon[37580]: pgmap v526: 129 pgs: 129 active+clean; 23 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 739 KiB/s wr, 185 op/s 2024-07-26T12:50:31.588 INFO:tasks.workunit.client.0.smithi107.stderr:Updating files: 7% (322/4329) Updating files: 8% (347/4329) Updating files: 9% (390/4329) Updating files: 10% (433/4329) Updating files: 11% (477/4329) Updating files: 11% (491/4329) Updating files: 12% (520/4329) Updating files: 13% (563/4329) Updating files: 14% (607/4329) Updating files: 15% (650/4329) Updating files: 16% (693/4329) Updating files: 17% (736/4329) Updating files: 18% (780/4329) Updating files: 19% (823/4329) Updating files: 20% (866/4329) Updating files: 21% (910/4329) Updating files: 22% (953/4329) Updating files: 22% (979/4329) Updating files: 23% (996/4329) Updating files: 24% (1039/4329) Updating files: 25% (1083/4329) Updating files: 26% (1126/4329) Updating files: 27% (1169/4329) Updating files: 28% (1213/4329) Updating files: 29% (1256/4329) Updating files: 29% (1289/4329) Updating files: 30% (1299/4329) Updating files: 31% (1342/4329) Updating files: 32% (1386/4329) Updating files: 33% (1429/4329) Updating files: 33% (1436/4329) Updating files: 34% (1472/4329) Updating files: 35% (1516/4329) Updating files: 36% (1559/4329) Updating files: 37% (1602/4329) Updating files: 38% (1646/4329) Updating files: 39% (1689/4329) Updating files: 40% (1732/4329) Updating files: 41% (1775/4329) Updating files: 42% (1819/4329) Updating files: 43% (1862/4329) Updating files: 44% (1905/4329) Updating files: 45% (1949/4329) Updating files: 45% (1973/4329) Updating files: 46% (1992/4329) Updating files: 47% (2035/4329) Updating files: 48% (2078/4329) Updating files: 49% (2122/4329) Updating files: 50% (2165/4329) Updating files: 51% (2208/4329) Updating files: 52% (2252/4329) Updating files: 53% (2295/4329) Updating files: 54% (2338/4329) Updating files: 55% (2381/4329) Updating files: 55% (2388/4329) Updating files: 56% (2425/4329) Updating files: 57% (2468/4329) Updating files: 58% (2511/4329) Updating files: 59% (2555/4329) Updating files: 60% (2598/4329) Updating files: 61% (2641/4329) Updating files: 62% (2684/4329) Updating files: 63% (2728/4329) Updating files: 64% (2771/4329) Updating files: 64% (2791/4329) Updating files: 65% (2814/4329) Updating files: 66% (2858/4329) Updating files: 67% (2901/4329) Updating files: 68% (2944/4329) Updating files: 69% (2988/4329) Updating files: 69% (3022/4329) Updating files: 70% (3031/4329) Updating files: 71% (3074/4329) Updating files: 72% (3117/4329) Updating files: 73% (3161/4329) Updating files: 74% (3204/4329) Updating files: 75% (3247/4329) Updating files: 76% (3291/4329) Updating files: 76% (3333/4329) Updating files: 77% (3334/4329) Updating files: 78% (3377/4329) Updating files: 79% (3420/4329) Updating files: 80% (3464/4329) Updating files: 81% (3507/4329) Updating files: 82% (3550/4329) Updating files: 83% (3594/4329) Updating files: 84% (3637/4329) Updating files: 85% (3680/4329) Updating files: 86% (3723/4329) Updating files: 87% (3767/4329) Updating files: 88% (3810/4329) Updating files: 89% (3853/4329) Updating files: 89% (3870/4329) Updating files: 90% (3897/4329) Updating files: 91% (3940/4329) Updating files: 92% (3983/4329) Updating files: 92% (4020/4329) Updating files: 93% (4026/4329) Updating files: 94% (4070/4329) Updating files: 95% (4113/4329) Updating files: 96% (4156/4329) Updating files: 97% (4200/4329) Updating files: 98% (4243/4329) Updating files: 99% (4286/4329) Updating files: 99% (4322/4329) Updating files: 100% (4329/4329) Updating files: 100% (4329/4329), done. 2024-07-26T12:50:31.619 INFO:tasks.workunit.client.0.smithi107.stderr:+ cd xfstests-dev 2024-07-26T12:50:31.619 INFO:tasks.workunit.client.0.smithi107.stderr:+ git checkout 0e5c12dfd008efc2848c98108c9237487e91ef35 2024-07-26T12:50:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:31.999+0000 7f84f5a00640 1 -- 172.21.15.107:0/2648273027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84f01056f0 msgr2=0x7f84f0105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.000+0000 7f84f5a00640 1 --2- 172.21.15.107:0/2648273027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84f01056f0 0x7f84f0105b00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009ca0 tx=0x7f84e002a8c0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.000+0000 7f84f5a00640 1 -- 172.21.15.107:0/2648273027 shutdown_connections 2024-07-26T12:50:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.000+0000 7f84f5a00640 1 --2- 172.21.15.107:0/2648273027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84f010b970 0x7f84f01121a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.000+0000 7f84f5a00640 1 --2- 172.21.15.107:0/2648273027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84f010b020 0x7f84f010b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.000+0000 7f84f5a00640 1 --2- 172.21.15.107:0/2648273027 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84f01056f0 0x7f84f0105b00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.000+0000 7f84f5a00640 1 -- 172.21.15.107:0/2648273027 >> 172.21.15.107:0/2648273027 conn(0x7f84f0101110 msgr2=0x7f84f0103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.000+0000 7f84f5a00640 1 -- 172.21.15.107:0/2648273027 shutdown_connections 2024-07-26T12:50:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.000+0000 7f84f5a00640 1 -- 172.21.15.107:0/2648273027 wait complete. 2024-07-26T12:50:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.001+0000 7f84f5a00640 1 Processor -- start 2024-07-26T12:50:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.001+0000 7f84f5a00640 1 -- start start 2024-07-26T12:50:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.001+0000 7f84f5a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84f01056f0 0x7f84f019c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.001+0000 7f84f5a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84f010b020 0x7f84f019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.001+0000 7f84f5a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84f010b970 0x7f84f01a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.001+0000 7f84f5a00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f84f0115690 con 0x7f84f010b020 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.001+0000 7f84f5a00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f84f0115510 con 0x7f84f01056f0 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.001+0000 7f84f5a00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f84f0115810 con 0x7f84f010b970 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84eea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84f01056f0 0x7f84f019c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84eea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84f01056f0 0x7f84f019c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57734/0 (socket says 172.21.15.107:57734) 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84eea00640 1 -- 172.21.15.107:0/1727284760 learned_addr learned my addr 172.21.15.107:0/1727284760 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84ee000640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84f010b020 0x7f84f019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84ef400640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84f010b970 0x7f84f01a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84ef400640 1 -- 172.21.15.107:0/1727284760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84f01056f0 msgr2=0x7f84f019c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84ef400640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84f01056f0 0x7f84f019c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84ef400640 1 -- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84f010b020 msgr2=0x7f84f019c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84ef400640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84f010b020 0x7f84f019c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84ef400640 1 -- 172.21.15.107:0/1727284760 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f01a1fe0 con 0x7f84f010b970 2024-07-26T12:50:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.002+0000 7f84ee000640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84f010b020 0x7f84f019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.003+0000 7f84ef400640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84f010b970 0x7f84f01a18e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f84e800d6b0 tx=0x7f84e800db70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.003+0000 7f84e7400640 1 -- 172.21.15.107:0/1727284760 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8017070 con 0x7f84f010b970 2024-07-26T12:50:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.003+0000 7f84f5a00640 1 -- 172.21.15.107:0/1727284760 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f01a22d0 con 0x7f84f010b970 2024-07-26T12:50:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.003+0000 7f84e7400640 1 -- 172.21.15.107:0/1727284760 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84e8009cd0 con 0x7f84f010b970 2024-07-26T12:50:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.003+0000 7f84e7400640 1 -- 172.21.15.107:0/1727284760 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8003cc0 con 0x7f84f010b970 2024-07-26T12:50:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.003+0000 7f84f5a00640 1 -- 172.21.15.107:0/1727284760 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f01a8a60 con 0x7f84f010b970 2024-07-26T12:50:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.006+0000 7f84fcbd7740 1 Processor -- start 2024-07-26T12:50:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.007+0000 7f84e7400640 1 -- 172.21.15.107:0/1727284760 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f84e8009820 con 0x7f84f010b970 2024-07-26T12:50:32.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.007+0000 7f84fcbd7740 1 -- start start 2024-07-26T12:50:32.009 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c46181b0 0x5574c49bba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c45ef2f0 0x5574c49bbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45f74d0 0x5574c49bc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5574c4930270 con 0x5574c45f74d0 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5574c491f770 con 0x5574c45ef2f0 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5574c49237b0 con 0x5574c46181b0 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84eea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c45ef2f0 0x5574c49bbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84eea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c45ef2f0 0x5574c49bbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57738/0 (socket says 172.21.15.107:57738) 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84eea00640 1 -- 172.21.15.107:0/1411722352 learned_addr learned my addr 172.21.15.107:0/1411722352 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84ee000640 1 --2- 172.21.15.107:0/1411722352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45f74d0 0x5574c49bc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84ee000640 1 --2- 172.21.15.107:0/1411722352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c46181b0 0x5574c49bba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84eea00640 1 -- 172.21.15.107:0/1411722352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c46181b0 msgr2=0x5574c49bba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:32.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84eea00640 1 --2- 172.21.15.107:0/1411722352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c46181b0 0x5574c49bba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84eea00640 1 -- 172.21.15.107:0/1411722352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45f74d0 msgr2=0x5574c49bc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:32.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84eea00640 1 --2- 172.21.15.107:0/1411722352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45f74d0 0x5574c49bc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.008+0000 7f84eea00640 1 -- 172.21.15.107:0/1411722352 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c4683790 con 0x5574c45ef2f0 2024-07-26T12:50:32.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.009+0000 7f84ee000640 1 --2- 172.21.15.107:0/1411722352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45f74d0 0x5574c49bc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:50:32.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.009+0000 7f84eea00640 1 --2- 172.21.15.107:0/1411722352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c45ef2f0 0x5574c49bbf80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f84e002f390 tx=0x7f84e002f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=189f1cfb09d32c76 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:32.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.009+0000 7f84e7400640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84d40b3d40 0x7f84d40b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.009+0000 7f84dbe00640 1 -- 172.21.15.107:0/1411722352 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e0031d70 con 0x5574c45ef2f0 2024-07-26T12:50:32.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.009+0000 7f84dbe00640 1 -- 172.21.15.107:0/1411722352 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84e0036070 con 0x5574c45ef2f0 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.009+0000 7f84e7400640 1 -- 172.21.15.107:0/1727284760 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f84e80e3c00 con 0x7f84f010b970 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.009+0000 7f84dbe00640 1 -- 172.21.15.107:0/1411722352 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e00042a0 con 0x5574c45ef2f0 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.010+0000 7f84ee000640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84d40b3d40 0x7f84d40b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.010+0000 7f84fcbd7740 1 -- 172.21.15.107:0/1411722352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c45ef2f0 msgr2=0x5574c49bbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.010+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/1411722352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c45ef2f0 0x5574c49bbf80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f84e002f390 tx=0x7f84e002f3c0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.011+0000 7f84fcbd7740 1 -- 172.21.15.107:0/1411722352 shutdown_connections 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.011+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/1411722352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45f74d0 0x5574c49bc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.011+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/1411722352 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c45ef2f0 0x5574c49bbf80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.011+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/1411722352 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c46181b0 0x5574c49bba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.011+0000 7f84fcbd7740 1 -- 172.21.15.107:0/1411722352 >> 172.21.15.107:0/1411722352 conn(0x5574c460c300 msgr2=0x5574c49b9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.011+0000 7f84fcbd7740 1 -- 172.21.15.107:0/1411722352 shutdown_connections 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.011+0000 7f84ee000640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84d40b3d40 0x7f84d40b6250 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0058c0 tx=0x7f84dc005850 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:32.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.011+0000 7f84fcbd7740 1 -- 172.21.15.107:0/1411722352 wait complete. 2024-07-26T12:50:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.012+0000 7f84fcbd7740 1 -- start start 2024-07-26T12:50:32.014 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.012+0000 7f84fcbd7740 1 Processor -- start 2024-07-26T12:50:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5574c4859ab0 con 0x5574c45ef2f0 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5574c48bcd50 con 0x5574c46181b0 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5574c49c9fe0 con 0x5574c45f74d0 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84fcbd7740 10 client.0 _ll_register_callbacks cb 0x5574c462fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84eea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84ee000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84ef400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84eea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49c76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57502/0 (socket says 172.21.15.107:57502) 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84eea00640 1 -- 172.21.15.107:0/2312489623 learned_addr learned my addr 172.21.15.107:0/2312489623 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84ee000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49c7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:45726/0 (socket says 172.21.15.107:45726) 2024-07-26T12:50:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84eea00640 1 -- 172.21.15.107:0/2312489623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 msgr2=0x5574c49c7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84eea00640 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49c7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84eea00640 1 -- 172.21.15.107:0/2312489623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 msgr2=0x5574c49d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84eea00640 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84eea00640 1 -- 172.21.15.107:0/2312489623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c46fc620 con 0x5574c45ef2f0 2024-07-26T12:50:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84ef400640 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.013+0000 7f84ee000640 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.014+0000 7f84eea00640 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49c76d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f84e0000f80 tx=0x7f84e0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b045ef58bdb43b15 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.014+0000 7f84d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:50:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.014+0000 7f84d8c00640 1 -- 172.21.15.107:0/2312489623 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e0031860 con 0x5574c45ef2f0 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.014+0000 7f84d8c00640 1 -- 172.21.15.107:0/2312489623 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f84e0003fb0 con 0x5574c45ef2f0 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.014+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2312489623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84e0036070 con 0x5574c45ef2f0 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.014+0000 7f84d8c00640 1 -- 172.21.15.107:0/2312489623 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f84e005b050 con 0x5574c45ef2f0 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.015+0000 7f84fcbd7740 10 client.15654 fetch_fsmap learned FSMap version 30 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.015+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2312489623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5574c49d60b0 con 0x5574c45ef2f0 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.015+0000 7f84d8c00640 1 -- 172.21.15.107:0/2312489623 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84e0036070 con 0x5574c45ef2f0 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.015+0000 7f84d8c00640 1 -- 172.21.15.107:0/2312489623 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f84e0043020 con 0x5574c45ef2f0 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.015+0000 7f84fcbd7740 10 client.15654 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.015+0000 7f84fcbd7740 10 client.15654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.015+0000 7f84fcbd7740 20 client.15654 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.015+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5574c45d6d80 0x5574c49d9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.015+0000 7f84fcbd7740 4 client.15654 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:50:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.015+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2312489623 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574c49321c0 con 0x5574c45d6d80 2024-07-26T12:50:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.016+0000 7f84ee000640 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5574c45d6d80 0x5574c49d9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.016+0000 7f84d8c00640 10 client.15654 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.016+0000 7f84ee000640 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5574c45d6d80 0x5574c49d9410 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5056b65198987a9 server_cookie=c63edc4578e6986e in_seq=0 out_seq=0 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.017+0000 7f84d8c00640 1 -- 172.21.15.107:0/2312489623 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5574c49321c0 con 0x5574c45d6d80 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.017+0000 7f84d8c00640 10 client.15654 handle_command_reply: tid=0 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.017+0000 7f84fcbd7740 1 client.15654 shutdown 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2312489623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 msgr2=0x5574c49c76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49c76d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f84e0000f80 tx=0x7f84e0005bd0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2312489623 shutdown_connections 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5574c45d6d80 0x5574c49d9410 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49c7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2312489623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49c76d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84d8c00640 0 client.15654 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2312489623 >> 172.21.15.107:0/2312489623 conn(0x5574c460c300 msgr2=0x5574c49b80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2312489623 shutdown_connections 2024-07-26T12:50:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2312489623 wait complete. 2024-07-26T12:50:32.022 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.018+0000 7f84fcbd7740 20 client.15654 trim_cache size 0 max 16384 2024-07-26T12:50:32.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.035+0000 7f84fcbd7740 1 Processor -- start 2024-07-26T12:50:32.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.035+0000 7f84fcbd7740 1 -- start start 2024-07-26T12:50:32.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49b8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49c1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01b2960 con 0x5574c45ef2f0 2024-07-26T12:50:32.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01b44d0 con 0x5574c46181b0 2024-07-26T12:50:32.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01b3030 con 0x5574c45f74d0 2024-07-26T12:50:32.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84ef400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84eea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84ef400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57760/0 (socket says 172.21.15.107:57760) 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84ef400640 1 -- 172.21.15.107:0/2846502654 learned_addr learned my addr 172.21.15.107:0/2846502654 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84ee000640 1 --2- 172.21.15.107:0/2846502654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49c1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84ef400640 1 -- 172.21.15.107:0/2846502654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 msgr2=0x5574c49c1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84ef400640 1 --2- 172.21.15.107:0/2846502654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49c1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84ef400640 1 -- 172.21.15.107:0/2846502654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 msgr2=0x5574c49b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.036+0000 7f84ef400640 1 --2- 172.21.15.107:0/2846502654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.037+0000 7f84ef400640 1 -- 172.21.15.107:0/2846502654 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574c49b5350 con 0x5574c46181b0 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.037+0000 7f84ef400640 1 --2- 172.21.15.107:0/2846502654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d30c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f84e80a6560 tx=0x7f84e80a9c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=726b30886cd39e91 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.037+0000 7f84db400640 1 -- 172.21.15.107:0/2846502654 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8020040 con 0x5574c46181b0 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.037+0000 7f84db400640 1 -- 172.21.15.107:0/2846502654 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84e8003e60 con 0x5574c46181b0 2024-07-26T12:50:32.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.037+0000 7f84db400640 1 -- 172.21.15.107:0/2846502654 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e80b34e0 con 0x5574c46181b0 2024-07-26T12:50:32.043 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.038+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2846502654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 msgr2=0x5574c49d30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.038+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2846502654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d30c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f84e80a6560 tx=0x7f84e80a9c50 comp rx=0 tx=0).stop 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.038+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2846502654 shutdown_connections 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.038+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2846502654 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49d30c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.038+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2846502654 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49c1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.038+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2846502654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.038+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2846502654 >> 172.21.15.107:0/2846502654 conn(0x5574c460c300 msgr2=0x5574c49c0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.038+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2846502654 shutdown_connections 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.038+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2846502654 wait complete. 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.039+0000 7f84fcbd7740 1 -- start start 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.039+0000 7f84fcbd7740 1 Processor -- start 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.044 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49ca730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84fcbd7740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49cac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01b3dd0 con 0x5574c45ef2f0 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01b3700 con 0x5574c46181b0 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84fcbd7740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f84f01b0940 con 0x5574c45f74d0 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84fcbd7740 10 client.0 _ll_register_callbacks cb 0x7f84e00095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84eea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84ef400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84ee000640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84ef400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49cac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:57772/0 (socket says 172.21.15.107:57772) 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84eea00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49ca1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57518/0 (socket says 172.21.15.107:57518) 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84ef400640 1 -- 172.21.15.107:0/2061250604 learned_addr learned my addr 172.21.15.107:0/2061250604 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84ef400640 1 -- 172.21.15.107:0/2061250604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 msgr2=0x5574c49ca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:32.045 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84ef400640 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49ca730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84ef400640 1 -- 172.21.15.107:0/2061250604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 msgr2=0x5574c49ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84ef400640 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.040+0000 7f84ef400640 1 -- 172.21.15.107:0/2061250604 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574c4a3fa70 con 0x5574c46181b0 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.041+0000 7f84ee000640 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.041+0000 7f84ef400640 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49cac70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f84e8005fd0 tx=0x7f84e80a9d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db63ddc7855ea3b2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.041+0000 7f84d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.041+0000 7f84d8c00640 1 -- 172.21.15.107:0/2061250604 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8014030 con 0x5574c46181b0 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.041+0000 7f84d8c00640 1 -- 172.21.15.107:0/2061250604 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f84e80a6060 con 0x5574c46181b0 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.041+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2061250604 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5574c492dc60 con 0x5574c46181b0 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.041+0000 7f84d8c00640 1 -- 172.21.15.107:0/2061250604 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f84e8020050 con 0x5574c46181b0 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.041+0000 7f84fcbd7740 10 client.25793 fetch_fsmap learned FSMap version 30 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.041+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2061250604 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5574c4a40d00 con 0x5574c46181b0 2024-07-26T12:50:32.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.041+0000 7f84d8c00640 1 -- 172.21.15.107:0/2061250604 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5574c492dc60 con 0x5574c46181b0 2024-07-26T12:50:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.042+0000 7f84d8c00640 1 -- 172.21.15.107:0/2061250604 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 30) ==== 8374+0+0 (secure 0 0 0) 0x7f84e80a91b0 con 0x5574c46181b0 2024-07-26T12:50:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.042+0000 7f84fcbd7740 10 client.25793 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T12:50:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.042+0000 7f84fcbd7740 10 client.25793 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.042+0000 7f84fcbd7740 20 client.25793 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.042+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5574c45ac1c0 0x5574c4a43410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.042+0000 7f84fcbd7740 4 client.25793 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T12:50:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.042+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2061250604 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5574c492b3a0 con 0x5574c45ac1c0 2024-07-26T12:50:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.042+0000 7f84eea00640 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5574c45ac1c0 0x5574c4a43410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.043+0000 7f84eea00640 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5574c45ac1c0 0x5574c4a43410 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38cb03614d6a9c3d server_cookie=657e7f2d11a6bc1 in_seq=0 out_seq=0 2024-07-26T12:50:32.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:32.043+0000 7f84d8c00640 10 client.25793 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:33.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84d8c00640 1 -- 172.21.15.107:0/2061250604 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5574c492b3a0 con 0x5574c45ac1c0 2024-07-26T12:50:33.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84d8c00640 10 client.25793 handle_command_reply: tid=0 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 client.25793 shutdown 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2061250604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 msgr2=0x5574c49cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49cac70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f84e8005fd0 tx=0x7f84e80a9d80 comp rx=0 tx=0).stop 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2061250604 shutdown_connections 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5574c45ac1c0 0x5574c4a43410 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5574c46181b0 0x5574c49cac70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5574c45f74d0 0x5574c49ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 --2- 172.21.15.107:0/2061250604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5574c45ef2f0 0x5574c49ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84d8c00640 0 client.25793 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2061250604 >> 172.21.15.107:0/2061250604 conn(0x5574c460c300 msgr2=0x5574c49b9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2061250604 shutdown_connections 2024-07-26T12:50:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 1 -- 172.21.15.107:0/2061250604 wait complete. 2024-07-26T12:50:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.416+0000 7f84fcbd7740 20 client.25793 trim_cache size 0 max 16384 2024-07-26T12:50:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-07-26T12:50:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-07-26T12:50:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-07-26T12:50:33.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-07-26T12:50:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 -- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84d40b3d40 msgr2=0x7f84d40b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84d40b3d40 0x7f84d40b6250 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f84dc0058c0 tx=0x7f84dc005850 comp rx=0 tx=0).stop 2024-07-26T12:50:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 -- 172.21.15.107:0/1727284760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84f010b970 msgr2=0x7f84f01a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84f010b970 0x7f84f01a18e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f84e800d6b0 tx=0x7f84e800db70 comp rx=0 tx=0).stop 2024-07-26T12:50:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 -- 172.21.15.107:0/1727284760 shutdown_connections 2024-07-26T12:50:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f84d40b3d40 0x7f84d40b6250 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84f010b970 0x7f84f01a18e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:33.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84f010b020 0x7f84f019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 --2- 172.21.15.107:0/1727284760 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f84f01056f0 0x7f84f019c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 -- 172.21.15.107:0/1727284760 >> 172.21.15.107:0/1727284760 conn(0x7f84f0101110 msgr2=0x7f84f0103530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 -- 172.21.15.107:0/1727284760 shutdown_connections 2024-07-26T12:50:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:33.417+0000 7f84f5a00640 1 -- 172.21.15.107:0/1727284760 wait complete. 2024-07-26T12:50:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:33 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:33 smithi112 ceph-mon[37501]: pgmap v527: 129 pgs: 129 active+clean; 29 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 1.3 MiB/s wr, 357 op/s 2024-07-26T12:50:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:33 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:33 smithi107 ceph-mon[32659]: pgmap v527: 129 pgs: 129 active+clean; 29 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 1.3 MiB/s wr, 357 op/s 2024-07-26T12:50:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:33 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:33 smithi132 ceph-mon[37580]: pgmap v527: 129 pgs: 129 active+clean; 29 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 1.3 MiB/s wr, 357 op/s 2024-07-26T12:50:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:34 smithi112 ceph-mon[37501]: pgmap v528: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.3 MiB/s wr, 579 op/s 2024-07-26T12:50:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:34 smithi107 ceph-mon[32659]: pgmap v528: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.3 MiB/s wr, 579 op/s 2024-07-26T12:50:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:34 smithi132 ceph-mon[37580]: pgmap v528: 129 pgs: 129 active+clean; 27 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 1.3 MiB/s wr, 579 op/s 2024-07-26T12:50:37.112 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:36 smithi107 ceph-mon[32659]: pgmap v529: 129 pgs: 129 active+clean; 29 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 1.9 MiB/s wr, 1.14k op/s 2024-07-26T12:50:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:36 smithi132 ceph-mon[37580]: pgmap v529: 129 pgs: 129 active+clean; 29 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 1.9 MiB/s wr, 1.14k op/s 2024-07-26T12:50:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:36 smithi112 ceph-mon[37501]: pgmap v529: 129 pgs: 129 active+clean; 29 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 1.9 MiB/s wr, 1.14k op/s 2024-07-26T12:50:37.921 INFO:tasks.workunit.client.0.smithi107.stderr:Note: switching to '0e5c12dfd008efc2848c98108c9237487e91ef35'. 2024-07-26T12:50:37.921 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr:state without impacting any branches by switching back to a branch. 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr: git switch -c 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr:Or undo this operation with: 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr: git switch - 2024-07-26T12:50:37.922 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:37.923 INFO:tasks.workunit.client.0.smithi107.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-26T12:50:37.923 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-07-26T12:50:37.923 INFO:tasks.workunit.client.0.smithi107.stderr:HEAD is now at 0e5c12df xfs/522: use reflink instead of crc as test feature 2024-07-26T12:50:37.936 INFO:tasks.workunit.client.0.smithi107.stderr:+ make -j4 2024-07-26T12:50:37.956 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize -cfi 2024-07-26T12:50:39.097 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: putting auxiliary files in '.'. 2024-07-26T12:50:39.097 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: copying file './config.guess' 2024-07-26T12:50:39.110 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: copying file './config.sub' 2024-07-26T12:50:39.125 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: copying file './install-sh' 2024-07-26T12:50:39.145 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: copying file './ltmain.sh' 2024-07-26T12:50:39.227 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-07-26T12:50:39.228 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-07-26T12:50:39.246 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-07-26T12:50:39.265 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-07-26T12:50:39.288 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-07-26T12:50:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:38 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:38 smithi107 ceph-mon[32659]: pgmap v530: 129 pgs: 129 active+clean; 30 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 2.4 MiB/s wr, 1.36k op/s 2024-07-26T12:50:39.304 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-07-26T12:50:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:38 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:38 smithi132 ceph-mon[37580]: pgmap v530: 129 pgs: 129 active+clean; 30 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 2.4 MiB/s wr, 1.36k op/s 2024-07-26T12:50:39.322 INFO:tasks.workunit.client.0.smithi107.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-07-26T12:50:39.325 INFO:tasks.workunit.client.0.smithi107.stdout:cp include/install-sh . 2024-07-26T12:50:39.337 INFO:tasks.workunit.client.0.smithi107.stdout:aclocal -I m4 2024-07-26T12:50:39.436 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:38 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:39.437 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:38 smithi112 ceph-mon[37501]: pgmap v530: 129 pgs: 129 active+clean; 30 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 2.4 MiB/s wr, 1.36k op/s 2024-07-26T12:50:39.437 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:39 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-07-26T12:50:39.437 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:39 smithi112 systemd[1]: Stopped Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:50:39.437 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:39 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Consumed 1.749s CPU time. 2024-07-26T12:50:39.437 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:39 smithi112 systemd[1]: Starting Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:50:39.700 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:39 smithi112 podman[71808]: 2024-07-26 12:50:39.339486011 +0000 UTC m=+0.018527708 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:50:40.174 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:39 smithi112 podman[71808]: 2024-07-26 12:50:39.920355395 +0000 UTC m=+0.599397084 container create 822dce7256d42518a49e8b68bed60df239d35cccf42005bb09287bf302bb8891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T12:50:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:40 smithi112 ceph-mon[37501]: daemon mds.h restarted 2024-07-26T12:50:40.450 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:40 smithi112 podman[71808]: 2024-07-26 12:50:40.173700918 +0000 UTC m=+0.852742606 container init 822dce7256d42518a49e8b68bed60df239d35cccf42005bb09287bf302bb8891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-07-26T12:50:40.451 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:40 smithi112 podman[71808]: 2024-07-26 12:50:40.176876667 +0000 UTC m=+0.855918364 container start 822dce7256d42518a49e8b68bed60df239d35cccf42005bb09287bf302bb8891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-07-26T12:50:40.451 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:40 smithi112 ceph-mds[71828]: -- 172.21.15.112:0/3195072673 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557658a9f860 con 0x55765cc41600 2024-07-26T12:50:40.451 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:40 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: starting mds.h at 2024-07-26T12:50:40.451 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:40 smithi112 bash[71808]: 822dce7256d42518a49e8b68bed60df239d35cccf42005bb09287bf302bb8891 2024-07-26T12:50:40.451 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:50:40 smithi112 systemd[1]: Started Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:50:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:40 smithi107 ceph-mon[32659]: daemon mds.h restarted 2024-07-26T12:50:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:40 smithi132 ceph-mon[37580]: daemon mds.h restarted 2024-07-26T12:50:41.064 INFO:tasks.workunit.client.0.smithi107.stdout:autoheader 2024-07-26T12:50:41.506 INFO:tasks.workunit.client.0.smithi107.stdout:autoconf 2024-07-26T12:50:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:50:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:41 smithi132 ceph-mon[37580]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T12:50:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:41 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:boot 2024-07-26T12:50:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:41 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:50:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:41 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:41 smithi132 ceph-mon[37580]: pgmap v532: 129 pgs: 129 active+clean; 32 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 2.7 MiB/s wr, 1.70k op/s 2024-07-26T12:50:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:50:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:41 smithi112 ceph-mon[37501]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T12:50:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:41 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:boot 2024-07-26T12:50:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:41 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:50:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:41 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:41 smithi112 ceph-mon[37501]: pgmap v532: 129 pgs: 129 active+clean; 32 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 2.7 MiB/s wr, 1.70k op/s 2024-07-26T12:50:41.781 INFO:tasks.workunit.client.0.smithi107.stdout:./configure \ 2024-07-26T12:50:41.781 INFO:tasks.workunit.client.0.smithi107.stdout: --libexecdir=/usr/lib \ 2024-07-26T12:50:41.782 INFO:tasks.workunit.client.0.smithi107.stdout: --exec_prefix=/var/lib 2024-07-26T12:50:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:50:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:41 smithi107 ceph-mon[32659]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T12:50:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:41 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:boot 2024-07-26T12:50:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:41 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:50:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:41 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:41 smithi107 ceph-mon[32659]: pgmap v532: 129 pgs: 129 active+clean; 32 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 2.7 MiB/s wr, 1.70k op/s 2024-07-26T12:50:42.204 INFO:tasks.workunit.client.0.smithi107.stdout:checking build system type... x86_64-pc-linux-gnu 2024-07-26T12:50:42.205 INFO:tasks.workunit.client.0.smithi107.stdout:checking host system type... x86_64-pc-linux-gnu 2024-07-26T12:50:42.209 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to print strings... printf 2024-07-26T12:50:42.209 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... gcc 2024-07-26T12:50:42.409 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the C compiler works... yes 2024-07-26T12:50:42.409 INFO:tasks.workunit.client.0.smithi107.stdout:checking for C compiler default output file name... a.out 2024-07-26T12:50:42.520 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of executables... 2024-07-26T12:50:42.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T12:50:42.590 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are cross compiling... no 2024-07-26T12:50:42.650 INFO:tasks.workunit.client.0.smithi107.stdout:checking for suffix of object files... o 2024-07-26T12:50:42.703 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are using the GNU C compiler... yes 2024-07-26T12:50:42.753 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... yes 2024-07-26T12:50:42.873 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to accept ISO C89... none needed 2024-07-26T12:50:42.883 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-07-26T12:50:42.891 INFO:tasks.workunit.client.0.smithi107.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-07-26T12:50:42.895 INFO:tasks.workunit.client.0.smithi107.stdout:checking for egrep... /usr/bin/grep -E 2024-07-26T12:50:42.899 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fgrep... /usr/bin/grep -F 2024-07-26T12:50:42.909 INFO:tasks.workunit.client.0.smithi107.stdout:checking for ld used by gcc... /usr/bin/ld 2024-07-26T12:50:42.914 INFO:tasks.workunit.client.0.smithi107.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-07-26T12:50:42.930 INFO:tasks.workunit.client.0.smithi107.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-07-26T12:50:42.980 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:50:43.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:43 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:43.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:43 smithi107 ceph-mon[32659]: pgmap v533: 129 pgs: 129 active+clean; 34 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 1.53k op/s 2024-07-26T12:50:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:43 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:43 smithi112 ceph-mon[37501]: pgmap v533: 129 pgs: 129 active+clean; 34 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 1.53k op/s 2024-07-26T12:50:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:43 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:50:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:43 smithi132 ceph-mon[37580]: pgmap v533: 129 pgs: 129 active+clean; 34 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 1.53k op/s 2024-07-26T12:50:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:44 smithi107 ceph-mon[32659]: pgmap v534: 129 pgs: 129 active+clean; 34 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 1.13k op/s 2024-07-26T12:50:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:44 smithi132 ceph-mon[37580]: pgmap v534: 129 pgs: 129 active+clean; 34 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 1.13k op/s 2024-07-26T12:50:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:44 smithi112 ceph-mon[37501]: pgmap v534: 129 pgs: 129 active+clean; 34 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 1.13k op/s 2024-07-26T12:50:45.742 INFO:tasks.workunit.client.0.smithi107.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-07-26T12:50:45.742 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether ln -s works... yes 2024-07-26T12:50:45.772 INFO:tasks.workunit.client.0.smithi107.stdout:checking the maximum length of command line arguments... 1572864 2024-07-26T12:50:45.784 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-07-26T12:50:45.784 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-07-26T12:50:45.784 INFO:tasks.workunit.client.0.smithi107.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-07-26T12:50:45.784 INFO:tasks.workunit.client.0.smithi107.stdout:checking for objdump... objdump 2024-07-26T12:50:45.784 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to recognize dependent libraries... pass_all 2024-07-26T12:50:45.785 INFO:tasks.workunit.client.0.smithi107.stdout:checking for dlltool... no 2024-07-26T12:50:45.785 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-07-26T12:50:45.786 INFO:tasks.workunit.client.0.smithi107.stdout:checking for ar... ar 2024-07-26T12:50:45.973 INFO:tasks.workunit.client.0.smithi107.stdout:checking for archiver @FILE support... @ 2024-07-26T12:50:45.973 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strip... strip 2024-07-26T12:50:45.974 INFO:tasks.workunit.client.0.smithi107.stdout:checking for ranlib... ranlib 2024-07-26T12:50:45.974 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gawk... gawk 2024-07-26T12:50:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.026+0000 7ff19d400640 1 -- 172.21.15.107:0/2866033551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff198114170 msgr2=0x7ff198116590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.026+0000 7ff19d400640 1 --2- 172.21.15.107:0/2866033551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff198114170 0x7ff198116590 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff190009900 tx=0x7ff19002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.027+0000 7ff19d400640 1 -- 172.21.15.107:0/2866033551 shutdown_connections 2024-07-26T12:50:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.027+0000 7ff19d400640 1 --2- 172.21.15.107:0/2866033551 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff198114170 0x7ff198116590 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.027+0000 7ff19d400640 1 --2- 172.21.15.107:0/2866033551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff198077cc0 0x7ff198113c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.027+0000 7ff19d400640 1 --2- 172.21.15.107:0/2866033551 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff198077370 0x7ff198077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.027+0000 7ff19d400640 1 -- 172.21.15.107:0/2866033551 >> 172.21.15.107:0/2866033551 conn(0x7ff198101240 msgr2=0x7ff198103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.027+0000 7ff19d400640 1 -- 172.21.15.107:0/2866033551 shutdown_connections 2024-07-26T12:50:46.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.027+0000 7ff19d400640 1 -- 172.21.15.107:0/2866033551 wait complete. 2024-07-26T12:50:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.028+0000 7ff19d400640 1 Processor -- start 2024-07-26T12:50:46.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff19d400640 1 -- start start 2024-07-26T12:50:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff19d400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff198077370 0x7ff1981a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff19d400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff198077cc0 0x7ff1981a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff19d400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff198114170 0x7ff1981a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff19d400640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff198119b20 con 0x7ff198077cc0 2024-07-26T12:50:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff19d400640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1981199a0 con 0x7ff198114170 2024-07-26T12:50:46.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff19d400640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff198119ca0 con 0x7ff198077370 2024-07-26T12:50:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff197200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff198114170 0x7ff1981a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff196800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff198077370 0x7ff1981a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff197200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff198114170 0x7ff1981a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:38474/0 (socket says 172.21.15.107:38474) 2024-07-26T12:50:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff197200640 1 -- 172.21.15.107:0/3083556616 learned_addr learned my addr 172.21.15.107:0/3083556616 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.029+0000 7ff195e00640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff198077cc0 0x7ff1981a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.030+0000 7ff197200640 1 -- 172.21.15.107:0/3083556616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff198077370 msgr2=0x7ff1981a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.030+0000 7ff197200640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff198077370 0x7ff1981a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.030+0000 7ff197200640 1 -- 172.21.15.107:0/3083556616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff198077cc0 msgr2=0x7ff1981a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.030+0000 7ff197200640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff198077cc0 0x7ff1981a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.030+0000 7ff197200640 1 -- 172.21.15.107:0/3083556616 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1981a6440 con 0x7ff198114170 2024-07-26T12:50:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.030+0000 7ff196800640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff198077370 0x7ff1981a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.030+0000 7ff197200640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff198114170 0x7ff1981a5d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff19002a910 tx=0x7ff190003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.031+0000 7ff18f400640 1 -- 172.21.15.107:0/3083556616 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19003f070 con 0x7ff198114170 2024-07-26T12:50:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.031+0000 7ff18f400640 1 -- 172.21.15.107:0/3083556616 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff190031b50 con 0x7ff198114170 2024-07-26T12:50:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.031+0000 7ff19d400640 1 -- 172.21.15.107:0/3083556616 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1981a66d0 con 0x7ff198114170 2024-07-26T12:50:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.031+0000 7ff18f400640 1 -- 172.21.15.107:0/3083556616 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190035030 con 0x7ff198114170 2024-07-26T12:50:46.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.031+0000 7ff19d400640 1 -- 172.21.15.107:0/3083556616 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1981ace90 con 0x7ff198114170 2024-07-26T12:50:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.035+0000 7ff1a422c740 1 Processor -- start 2024-07-26T12:50:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.035+0000 7ff18f400640 1 -- 172.21.15.107:0/3083556616 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7ff190042070 con 0x7ff198114170 2024-07-26T12:50:46.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.035+0000 7ff1a422c740 1 -- start start 2024-07-26T12:50:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.036+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb38e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.036+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb39360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.036+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 0x560fdbb398a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.036+0000 7ff1a422c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560fdbaafbc0 con 0x560fdb76b1b0 2024-07-26T12:50:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.036+0000 7ff1a422c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560fdba9f0c0 con 0x560fdb773390 2024-07-26T12:50:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.036+0000 7ff1a422c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560fdbaa3100 con 0x560fdb794070 2024-07-26T12:50:46.037 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.036+0000 7ff195e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb38e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.036+0000 7ff195e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb38e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:57388/0 (socket says 172.21.15.107:57388) 2024-07-26T12:50:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.036+0000 7ff195e00640 1 -- 172.21.15.107:0/1333358743 learned_addr learned my addr 172.21.15.107:0/1333358743 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:46.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.036+0000 7ff195e00640 1 --2- 172.21.15.107:0/1333358743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 0x560fdbb398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.037+0000 7ff196800640 1 --2- 172.21.15.107:0/1333358743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb39360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.037+0000 7ff195e00640 1 -- 172.21.15.107:0/1333358743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 msgr2=0x560fdbb398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.037+0000 7ff195e00640 1 --2- 172.21.15.107:0/1333358743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 0x560fdbb398a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.037+0000 7ff195e00640 1 -- 172.21.15.107:0/1333358743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 msgr2=0x560fdbb39360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.037+0000 7ff195e00640 1 --2- 172.21.15.107:0/1333358743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb39360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.037+0000 7ff195e00640 1 -- 172.21.15.107:0/1333358743 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fdb7ff650 con 0x560fdb794070 2024-07-26T12:50:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.037+0000 7ff196800640 1 --2- 172.21.15.107:0/1333358743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb39360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:50:46.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.037+0000 7ff18f400640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff17c0b3e90 0x7ff17c0b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.038+0000 7ff18f400640 1 -- 172.21.15.107:0/3083556616 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7ff1901047b0 con 0x7ff198114170 2024-07-26T12:50:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.038+0000 7ff195e00640 1 --2- 172.21.15.107:0/1333358743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb38e20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff188013aa0 tx=0x7ff188009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=313ce59865181d63 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.038+0000 7ff187400640 1 -- 172.21.15.107:0/1333358743 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188013170 con 0x560fdb794070 2024-07-26T12:50:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.038+0000 7ff187400640 1 -- 172.21.15.107:0/1333358743 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff18800f880 con 0x560fdb794070 2024-07-26T12:50:46.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.038+0000 7ff187400640 1 -- 172.21.15.107:0/1333358743 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188005280 con 0x560fdb794070 2024-07-26T12:50:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.039+0000 7ff196800640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff17c0b3e90 0x7ff17c0b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.039+0000 7ff1a422c740 1 -- 172.21.15.107:0/1333358743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 msgr2=0x560fdbb38e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.039+0000 7ff1a422c740 1 --2- 172.21.15.107:0/1333358743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb38e20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff188013aa0 tx=0x7ff188009500 comp rx=0 tx=0).stop 2024-07-26T12:50:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.039+0000 7ff196800640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff17c0b3e90 0x7ff17c0b63a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x560fdbb37aa0 tx=0x7ff180005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.040+0000 7ff1a422c740 1 -- 172.21.15.107:0/1333358743 shutdown_connections 2024-07-26T12:50:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.040+0000 7ff1a422c740 1 --2- 172.21.15.107:0/1333358743 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 0x560fdbb398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.040+0000 7ff1a422c740 1 --2- 172.21.15.107:0/1333358743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb39360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.040+0000 7ff1a422c740 1 --2- 172.21.15.107:0/1333358743 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb38e20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.040+0000 7ff1a422c740 1 -- 172.21.15.107:0/1333358743 >> 172.21.15.107:0/1333358743 conn(0x560fdb788460 msgr2=0x560fdbb36d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.040+0000 7ff1a422c740 1 -- 172.21.15.107:0/1333358743 shutdown_connections 2024-07-26T12:50:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.040+0000 7ff1a422c740 1 -- 172.21.15.107:0/1333358743 wait complete. 2024-07-26T12:50:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.041+0000 7ff1a422c740 1 -- start start 2024-07-26T12:50:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.041+0000 7ff1a422c740 1 Processor -- start 2024-07-26T12:50:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb44420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.042 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 0x560fdbb44960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb44ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff1a422c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560fdb902700 con 0x560fdb76b1b0 2024-07-26T12:50:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff1a422c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x560fdba3b690 con 0x560fdb773390 2024-07-26T12:50:46.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff1a422c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x560fdbb47600 con 0x560fdb794070 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff1a422c740 10 client.0 _ll_register_callbacks cb 0x560fdba4b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff195e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 0x560fdbb44960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff195e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 0x560fdbb44960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:38490/0 (socket says 172.21.15.107:38490) 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff195e00640 1 -- 172.21.15.107:0/2206764931 learned_addr learned my addr 172.21.15.107:0/2206764931 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff197200640 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb44ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.042+0000 7ff196800640 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb44420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.043+0000 7ff195e00640 1 -- 172.21.15.107:0/2206764931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 msgr2=0x560fdbb44ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.043+0000 7ff195e00640 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb44ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.043+0000 7ff195e00640 1 -- 172.21.15.107:0/2206764931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 msgr2=0x560fdbb44420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.043+0000 7ff195e00640 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb44420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.043+0000 7ff195e00640 1 -- 172.21.15.107:0/2206764931 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fdba52830 con 0x560fdb773390 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.043+0000 7ff195e00640 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 0x560fdbb44960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff188013fa0 tx=0x7ff188004f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89790c24ae36889b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.043+0000 7ff17be00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.043+0000 7ff17be00640 1 -- 172.21.15.107:0/2206764931 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18800fd00 con 0x560fdb773390 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.043+0000 7ff17be00640 1 -- 172.21.15.107:0/2206764931 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7ff188005000 con 0x560fdb773390 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.043+0000 7ff1a422c740 1 -- 172.21.15.107:0/2206764931 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff18800f880 con 0x560fdb773390 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.044+0000 7ff17be00640 1 -- 172.21.15.107:0/2206764931 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff18802f050 con 0x560fdb773390 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.044+0000 7ff1a422c740 10 client.25811 fetch_fsmap learned FSMap version 35 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.044+0000 7ff1a422c740 1 -- 172.21.15.107:0/2206764931 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560fdbb53230 con 0x560fdb773390 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.044+0000 7ff17be00640 1 -- 172.21.15.107:0/2206764931 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff18800f880 con 0x560fdb773390 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.044+0000 7ff17be00640 1 -- 172.21.15.107:0/2206764931 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7ff188010550 con 0x560fdb773390 2024-07-26T12:50:46.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.045+0000 7ff1a422c740 10 client.25811 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:50:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.045+0000 7ff1a422c740 10 client.25811 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.045+0000 7ff1a422c740 20 client.25811 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.045+0000 7ff1a422c740 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x560fdb7529f0 0x560fdbb56590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.045+0000 7ff1a422c740 4 client.25811 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:50:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.045+0000 7ff1a422c740 1 -- 172.21.15.107:0/2206764931 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fdbab1b10 con 0x560fdb7529f0 2024-07-26T12:50:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.045+0000 7ff196800640 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x560fdb7529f0 0x560fdbb56590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.046+0000 7ff196800640 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x560fdb7529f0 0x560fdbb56590 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bf27c42e9affbc0 server_cookie=56dfe467ae9a9dc6 in_seq=0 out_seq=0 2024-07-26T12:50:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.046+0000 7ff17be00640 10 client.25811 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.047+0000 7ff17be00640 1 -- 172.21.15.107:0/2206764931 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560fdbab1b10 con 0x560fdb7529f0 2024-07-26T12:50:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.047+0000 7ff17be00640 10 client.25811 handle_command_reply: tid=0 2024-07-26T12:50:46.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.047+0000 7ff1a422c740 1 client.25811 shutdown 2024-07-26T12:50:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 1 -- 172.21.15.107:0/2206764931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 msgr2=0x560fdbb44960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 0x560fdbb44960 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff188013fa0 tx=0x7ff188004f60 comp rx=0 tx=0).stop 2024-07-26T12:50:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 1 -- 172.21.15.107:0/2206764931 shutdown_connections 2024-07-26T12:50:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x560fdb7529f0 0x560fdbb56590 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb44ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb773390 0x560fdbb44960 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 1 --2- 172.21.15.107:0/2206764931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb44420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff17be00640 0 client.25811 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:46.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 1 -- 172.21.15.107:0/2206764931 >> 172.21.15.107:0/2206764931 conn(0x560fdb788460 msgr2=0x560fdbb3bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 1 -- 172.21.15.107:0/2206764931 shutdown_connections 2024-07-26T12:50:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 1 -- 172.21.15.107:0/2206764931 wait complete. 2024-07-26T12:50:46.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.048+0000 7ff1a422c740 20 client.25811 trim_cache size 0 max 16384 2024-07-26T12:50:46.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.067+0000 7ff1a422c740 1 Processor -- start 2024-07-26T12:50:46.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.067+0000 7ff1a422c740 1 -- start start 2024-07-26T12:50:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb76b1b0 0x560fdbb44e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb773390 0x560fdbb46300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb3a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff1a422c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1981b6e70 con 0x560fdb773390 2024-07-26T12:50:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff1a422c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1981b8a70 con 0x560fdb76b1b0 2024-07-26T12:50:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff1a422c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1981b7540 con 0x560fdb794070 2024-07-26T12:50:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff195e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb773390 0x560fdbb46300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff196800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb76b1b0 0x560fdbb44e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff196800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb76b1b0 0x560fdbb44e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:38500/0 (socket says 172.21.15.107:38500) 2024-07-26T12:50:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff195e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb773390 0x560fdbb46300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59952/0 (socket says 172.21.15.107:59952) 2024-07-26T12:50:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.068+0000 7ff196800640 1 -- 172.21.15.107:0/99829263 learned_addr learned my addr 172.21.15.107:0/99829263 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.069+0000 7ff197200640 1 --2- 172.21.15.107:0/99829263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb3a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.069+0000 7ff196800640 1 -- 172.21.15.107:0/99829263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 msgr2=0x560fdbb3a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.069+0000 7ff196800640 1 --2- 172.21.15.107:0/99829263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb3a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.069+0000 7ff196800640 1 -- 172.21.15.107:0/99829263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb773390 msgr2=0x560fdbb46300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.069+0000 7ff196800640 1 --2- 172.21.15.107:0/99829263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb773390 0x560fdbb46300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.069+0000 7ff196800640 1 -- 172.21.15.107:0/99829263 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fdbb3cea0 con 0x560fdb76b1b0 2024-07-26T12:50:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.069+0000 7ff196800640 1 --2- 172.21.15.107:0/99829263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb76b1b0 0x560fdbb44e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff180005e00 tx=0x7ff180019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3208f3b28b4b7fb8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.069+0000 7ff186a00640 1 -- 172.21.15.107:0/99829263 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18000fd10 con 0x560fdb76b1b0 2024-07-26T12:50:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.070+0000 7ff186a00640 1 -- 172.21.15.107:0/99829263 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff18001dd10 con 0x560fdb76b1b0 2024-07-26T12:50:46.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.070+0000 7ff186a00640 1 -- 172.21.15.107:0/99829263 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18002ac50 con 0x560fdb76b1b0 2024-07-26T12:50:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.071+0000 7ff1a422c740 1 -- 172.21.15.107:0/99829263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb76b1b0 msgr2=0x560fdbb44e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.071+0000 7ff1a422c740 1 --2- 172.21.15.107:0/99829263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb76b1b0 0x560fdbb44e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff180005e00 tx=0x7ff180019000 comp rx=0 tx=0).stop 2024-07-26T12:50:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.071+0000 7ff1a422c740 1 -- 172.21.15.107:0/99829263 shutdown_connections 2024-07-26T12:50:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.071+0000 7ff1a422c740 1 --2- 172.21.15.107:0/99829263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb794070 0x560fdbb3a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.071+0000 7ff1a422c740 1 --2- 172.21.15.107:0/99829263 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb773390 0x560fdbb46300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.071+0000 7ff1a422c740 1 --2- 172.21.15.107:0/99829263 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb76b1b0 0x560fdbb44e60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.071+0000 7ff1a422c740 1 -- 172.21.15.107:0/99829263 >> 172.21.15.107:0/99829263 conn(0x560fdb788460 msgr2=0x560fdbb39e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.071+0000 7ff1a422c740 1 -- 172.21.15.107:0/99829263 shutdown_connections 2024-07-26T12:50:46.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.071+0000 7ff1a422c740 1 -- 172.21.15.107:0/99829263 wait complete. 2024-07-26T12:50:46.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.072+0000 7ff1a422c740 1 -- start start 2024-07-26T12:50:46.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.072+0000 7ff1a422c740 1 Processor -- start 2024-07-26T12:50:46.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb48050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb773390 0x560fdbb48590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff1a422c740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb794070 0x560fdbbbcc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff1a422c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1981b83a0 con 0x560fdb76b1b0 2024-07-26T12:50:46.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff1a422c740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff1981b7c10 con 0x560fdb794070 2024-07-26T12:50:46.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff1a422c740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff1981b4e50 con 0x560fdb773390 2024-07-26T12:50:46.074 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff1a422c740 10 client.0 _ll_register_callbacks cb 0x7ff188004e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff195e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb773390 0x560fdbb48590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff195e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb773390 0x560fdbb48590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:57416/0 (socket says 172.21.15.107:57416) 2024-07-26T12:50:46.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff195e00640 1 -- 172.21.15.107:0/1311550997 learned_addr learned my addr 172.21.15.107:0/1311550997 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff197200640 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb794070 0x560fdbbbcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.073+0000 7ff196800640 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb48050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.074+0000 7ff197200640 1 -- 172.21.15.107:0/1311550997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb773390 msgr2=0x560fdbb48590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.074+0000 7ff197200640 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb773390 0x560fdbb48590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.074+0000 7ff197200640 1 -- 172.21.15.107:0/1311550997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 msgr2=0x560fdbb48050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.074+0000 7ff197200640 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb48050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.074+0000 7ff197200640 1 -- 172.21.15.107:0/1311550997 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fdbbbd1e0 con 0x560fdb794070 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.074+0000 7ff197200640 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb794070 0x560fdbbbcc70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff190101b20 tx=0x7ff1900caaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a4856cb9d53d795 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.074+0000 7ff17be00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.074+0000 7ff17be00640 1 -- 172.21.15.107:0/1311550997 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1900cb090 con 0x560fdb794070 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.074+0000 7ff17be00640 1 -- 172.21.15.107:0/1311550997 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7ff1900d3c00 con 0x560fdb794070 2024-07-26T12:50:46.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.074+0000 7ff1a422c740 1 -- 172.21.15.107:0/1311550997 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff18001dd10 con 0x560fdb794070 2024-07-26T12:50:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff17be00640 1 -- 172.21.15.107:0/1311550997 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff1900cf5e0 con 0x560fdb794070 2024-07-26T12:50:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff1a422c740 10 client.25817 fetch_fsmap learned FSMap version 35 2024-07-26T12:50:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff1a422c740 1 -- 172.21.15.107:0/1311550997 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560fdbbbe470 con 0x560fdb794070 2024-07-26T12:50:46.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff17be00640 1 -- 172.21.15.107:0/1311550997 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff18001dd10 con 0x560fdb794070 2024-07-26T12:50:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff17be00640 1 -- 172.21.15.107:0/1311550997 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7ff1901042f0 con 0x560fdb794070 2024-07-26T12:50:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff1a422c740 10 client.25817 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:50:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff1a422c740 10 client.25817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff1a422c740 20 client.25817 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff1a422c740 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x560fdb727da0 0x560fdbbc0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff1a422c740 4 client.25817 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T12:50:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.075+0000 7ff1a422c740 1 -- 172.21.15.107:0/1311550997 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560fdbaad5b0 con 0x560fdb727da0 2024-07-26T12:50:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.076+0000 7ff196800640 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x560fdb727da0 0x560fdbbc0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.077+0000 7ff196800640 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x560fdb727da0 0x560fdbbc0d40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3fc273aeefa0266 server_cookie=9aaa6a8de9cbe13d in_seq=0 out_seq=0 2024-07-26T12:50:46.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.077+0000 7ff17be00640 10 client.25817 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.077+0000 7ff17be00640 1 -- 172.21.15.107:0/1311550997 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560fdbaad5b0 con 0x560fdb727da0 2024-07-26T12:50:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.077+0000 7ff17be00640 10 client.25817 handle_command_reply: tid=0 2024-07-26T12:50:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.077+0000 7ff1a422c740 1 client.25817 shutdown 2024-07-26T12:50:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.078+0000 7ff1a422c740 1 -- 172.21.15.107:0/1311550997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb794070 msgr2=0x560fdbbbcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.078+0000 7ff1a422c740 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb794070 0x560fdbbbcc70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff190101b20 tx=0x7ff1900caaa0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.079+0000 7ff1a422c740 1 -- 172.21.15.107:0/1311550997 shutdown_connections 2024-07-26T12:50:46.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.079+0000 7ff1a422c740 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x560fdb727da0 0x560fdbbc0d40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.079+0000 7ff1a422c740 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x560fdb794070 0x560fdbbbcc70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.079+0000 7ff1a422c740 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x560fdb773390 0x560fdbb48590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.079+0000 7ff1a422c740 1 --2- 172.21.15.107:0/1311550997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560fdb76b1b0 0x560fdbb48050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.079+0000 7ff17be00640 0 client.25817 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:46.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.079+0000 7ff1a422c740 1 -- 172.21.15.107:0/1311550997 >> 172.21.15.107:0/1311550997 conn(0x560fdb788460 msgr2=0x560fdbb3a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.080+0000 7ff1a422c740 1 -- 172.21.15.107:0/1311550997 shutdown_connections 2024-07-26T12:50:46.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.080+0000 7ff1a422c740 1 -- 172.21.15.107:0/1311550997 wait complete. 2024-07-26T12:50:46.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.080+0000 7ff1a422c740 20 client.25817 trim_cache size 0 max 16384 2024-07-26T12:50:46.081 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-07-26T12:50:46.081 INFO:teuthology.orchestra.run.smithi107.stdout: "status": "no active scrubs running", 2024-07-26T12:50:46.081 INFO:teuthology.orchestra.run.smithi107.stdout: "scrubs": {} 2024-07-26T12:50:46.081 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-07-26T12:50:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 -- 172.21.15.107:0/3083556616 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff17c0b3e90 msgr2=0x7ff17c0b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff17c0b3e90 0x7ff17c0b63a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x560fdbb37aa0 tx=0x7ff180005f90 comp rx=0 tx=0).stop 2024-07-26T12:50:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 -- 172.21.15.107:0/3083556616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff198114170 msgr2=0x7ff1981a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff198114170 0x7ff1981a5d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff19002a910 tx=0x7ff190003b90 comp rx=0 tx=0).stop 2024-07-26T12:50:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 -- 172.21.15.107:0/3083556616 shutdown_connections 2024-07-26T12:50:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff17c0b3e90 0x7ff17c0b63a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff198114170 0x7ff1981a5d10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff198077cc0 0x7ff1981a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 --2- 172.21.15.107:0/3083556616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff198077370 0x7ff1981a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 -- 172.21.15.107:0/3083556616 >> 172.21.15.107:0/3083556616 conn(0x7ff198101240 msgr2=0x7ff198116150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 -- 172.21.15.107:0/3083556616 shutdown_connections 2024-07-26T12:50:46.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:46.081+0000 7ff19d400640 1 -- 172.21.15.107:0/3083556616 wait complete. 2024-07-26T12:50:46.168 INFO:tasks.workunit.client.0.smithi107.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-07-26T12:50:46.171 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sysroot... no 2024-07-26T12:50:46.188 INFO:tasks.workunit.client.0.smithi107.stdout:checking for a working dd... /usr/bin/dd 2024-07-26T12:50:46.204 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-07-26T12:50:46.266 INFO:tasks.workunit.client.0.smithi107.stdout:checking for mt... no 2024-07-26T12:50:46.290 INFO:tasks.workunit.client.0.smithi107.stdout:checking if : is a manifest tool... no 2024-07-26T12:50:46.544 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to run the C preprocessor... gcc -E 2024-07-26T12:50:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.640+0000 7f83d1e00640 1 -- 172.21.15.107:0/570469775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83cc107d10 msgr2=0x7f83cc108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.640+0000 7f83d1e00640 1 --2- 172.21.15.107:0/570469775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83cc107d10 0x7f83cc108120 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009900 tx=0x7f83b802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.640+0000 7f83d1e00640 1 -- 172.21.15.107:0/570469775 shutdown_connections 2024-07-26T12:50:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.640+0000 7f83d1e00640 1 --2- 172.21.15.107:0/570469775 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83cc108660 0x7f83cc10ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.640+0000 7f83d1e00640 1 --2- 172.21.15.107:0/570469775 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83cc107d10 0x7f83cc108120 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.640+0000 7f83d1e00640 1 --2- 172.21.15.107:0/570469775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83cc1023e0 0x7f83cc1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.640+0000 7f83d1e00640 1 -- 172.21.15.107:0/570469775 >> 172.21.15.107:0/570469775 conn(0x7f83cc07a170 msgr2=0x7f83cc07a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.640+0000 7f83d1e00640 1 -- 172.21.15.107:0/570469775 shutdown_connections 2024-07-26T12:50:46.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.640+0000 7f83d1e00640 1 -- 172.21.15.107:0/570469775 wait complete. 2024-07-26T12:50:46.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.641+0000 7f83d1e00640 1 Processor -- start 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83d1e00640 1 -- start start 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83d1e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83cc1023e0 0x7f83cc199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83d1e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83cc107d10 0x7f83cc19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83d1e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83cc108660 0x7f83cc19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83d1e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc113470 con 0x7f83cc1023e0 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83d1e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc1132f0 con 0x7f83cc108660 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83d1e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc1135f0 con 0x7f83cc107d10 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83caa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83cc1023e0 0x7f83cc199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83caa00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83cc1023e0 0x7f83cc199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59982/0 (socket says 172.21.15.107:59982) 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83caa00640 1 -- 172.21.15.107:0/732195755 learned_addr learned my addr 172.21.15.107:0/732195755 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83cb400640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83cc108660 0x7f83cc19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.642+0000 7f83ca000640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83cc107d10 0x7f83cc19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83cb400640 1 -- 172.21.15.107:0/732195755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83cc107d10 msgr2=0x7f83cc19a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83cb400640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83cc107d10 0x7f83cc19a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83cb400640 1 -- 172.21.15.107:0/732195755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83cc1023e0 msgr2=0x7f83cc199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83cb400640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83cc1023e0 0x7f83cc199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83cb400640 1 -- 172.21.15.107:0/732195755 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83cc19fde0 con 0x7f83cc108660 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83caa00640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83cc1023e0 0x7f83cc199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83cb400640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83cc108660 0x7f83cc19f6e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f83c4009fd0 tx=0x7f83c400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83d0a00640 1 -- 172.21.15.107:0/732195755 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c400ce20 con 0x7f83cc108660 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83d0a00640 1 -- 172.21.15.107:0/732195755 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83c4005020 con 0x7f83cc108660 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83d0a00640 1 -- 172.21.15.107:0/732195755 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4010690 con 0x7f83cc108660 2024-07-26T12:50:46.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.643+0000 7f83d1e00640 1 -- 172.21.15.107:0/732195755 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83cc1a00d0 con 0x7f83cc108660 2024-07-26T12:50:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.644+0000 7f83d1e00640 1 -- 172.21.15.107:0/732195755 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83cc1a6860 con 0x7f83cc108660 2024-07-26T12:50:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.647+0000 7f83d0a00640 1 -- 172.21.15.107:0/732195755 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f83c40054d0 con 0x7f83cc108660 2024-07-26T12:50:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.647+0000 7f83d8df0740 1 Processor -- start 2024-07-26T12:50:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83d8df0740 1 -- start start 2024-07-26T12:50:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 0x556048ee9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b1d2f0 0x556048ee9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b254d0 0x556048eea4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83d8df0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556048e5e270 con 0x556048b1d2f0 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83d8df0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556048e4d770 con 0x556048b461b0 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83d8df0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556048e517b0 con 0x556048b254d0 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83ca000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 0x556048ee9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83ca000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 0x556048ee9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:38530/0 (socket says 172.21.15.107:38530) 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83ca000640 1 -- 172.21.15.107:0/3016219204 learned_addr learned my addr 172.21.15.107:0/3016219204 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83caa00640 1 --2- 172.21.15.107:0/3016219204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b1d2f0 0x556048ee9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.648+0000 7f83ca000640 1 --2- 172.21.15.107:0/3016219204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b254d0 0x556048eea4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.649+0000 7f83caa00640 1 -- 172.21.15.107:0/3016219204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b254d0 msgr2=0x556048eea4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.649+0000 7f83caa00640 1 --2- 172.21.15.107:0/3016219204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b254d0 0x556048eea4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.649+0000 7f83caa00640 1 -- 172.21.15.107:0/3016219204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 msgr2=0x556048ee9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.649+0000 7f83caa00640 1 --2- 172.21.15.107:0/3016219204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 0x556048ee9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.649+0000 7f83caa00640 1 -- 172.21.15.107:0/3016219204 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556048bb1790 con 0x556048b1d2f0 2024-07-26T12:50:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.649+0000 7f83d0a00640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f83b00b3d40 0x7f83b00b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.649+0000 7f83caa00640 1 --2- 172.21.15.107:0/3016219204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b1d2f0 0x556048ee9f80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f83bc010580 tx=0x7f83bc00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88a94c34232dc17 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.649+0000 7f83d0a00640 1 -- 172.21.15.107:0/732195755 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f83c40e4a10 con 0x7f83cc108660 2024-07-26T12:50:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.649+0000 7f83ca000640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f83b00b3d40 0x7f83b00b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.650+0000 7f83b7e00640 1 -- 172.21.15.107:0/3016219204 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc011760 con 0x556048b1d2f0 2024-07-26T12:50:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.650+0000 7f83b7e00640 1 -- 172.21.15.107:0/3016219204 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83bc011db0 con 0x556048b1d2f0 2024-07-26T12:50:46.652 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.650+0000 7f83b7e00640 1 -- 172.21.15.107:0/3016219204 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc00ee50 con 0x556048b1d2f0 2024-07-26T12:50:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.650+0000 7f83d8df0740 1 -- 172.21.15.107:0/3016219204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b1d2f0 msgr2=0x556048ee9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.650+0000 7f83d8df0740 1 --2- 172.21.15.107:0/3016219204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b1d2f0 0x556048ee9f80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f83bc010580 tx=0x7f83bc00bf90 comp rx=0 tx=0).stop 2024-07-26T12:50:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.650+0000 7f83ca000640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f83b00b3d40 0x7f83b00b6250 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f83b80096c0 tx=0x7f83b8041040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.651+0000 7f83d8df0740 1 -- 172.21.15.107:0/3016219204 shutdown_connections 2024-07-26T12:50:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.651+0000 7f83d8df0740 1 --2- 172.21.15.107:0/3016219204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b254d0 0x556048eea4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.651+0000 7f83d8df0740 1 --2- 172.21.15.107:0/3016219204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b1d2f0 0x556048ee9f80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.651+0000 7f83d8df0740 1 --2- 172.21.15.107:0/3016219204 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 0x556048ee9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.651+0000 7f83d8df0740 1 -- 172.21.15.107:0/3016219204 >> 172.21.15.107:0/3016219204 conn(0x556048b3a300 msgr2=0x556048ee7930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.651+0000 7f83d8df0740 1 -- 172.21.15.107:0/3016219204 shutdown_connections 2024-07-26T12:50:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.651+0000 7f83d8df0740 1 -- 172.21.15.107:0/3016219204 wait complete. 2024-07-26T12:50:46.654 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.651+0000 7f83d8df0740 1 -- start start 2024-07-26T12:50:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.652+0000 7f83d8df0740 1 Processor -- start 2024-07-26T12:50:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.652+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048ef98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.652+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048ef9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.652+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b461b0 0x556048f058d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.652+0000 7f83d8df0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556048d87ab0 con 0x556048b461b0 2024-07-26T12:50:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.652+0000 7f83d8df0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x556048dead50 con 0x556048b254d0 2024-07-26T12:50:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.652+0000 7f83d8df0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x556048efc5a0 con 0x556048b1d2f0 2024-07-26T12:50:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83d8df0740 10 client.0 _ll_register_callbacks cb 0x556048b5da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83ca000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048ef9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83ca000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048ef9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:38532/0 (socket says 172.21.15.107:38532) 2024-07-26T12:50:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83caa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048ef98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83ca000640 1 -- 172.21.15.107:0/4017495540 learned_addr learned my addr 172.21.15.107:0/4017495540 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83cb400640 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b461b0 0x556048f058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83caa00640 1 -- 172.21.15.107:0/4017495540 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 msgr2=0x556048ef9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.658 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83caa00640 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048ef9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83caa00640 1 -- 172.21.15.107:0/4017495540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b461b0 msgr2=0x556048f058d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83caa00640 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b461b0 0x556048f058d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83caa00640 1 -- 172.21.15.107:0/4017495540 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556048c2a620 con 0x556048b1d2f0 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83caa00640 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048ef98f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00e9d0 tx=0x7f83bc00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de1906dd01c28730 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.653+0000 7f83b4c00640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.654+0000 7f83b4c00640 1 -- 172.21.15.107:0/4017495540 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc01f040 con 0x556048b1d2f0 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.654+0000 7f83b4c00640 1 -- 172.21.15.107:0/4017495540 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f83bc00e630 con 0x556048b1d2f0 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.654+0000 7f83d8df0740 1 -- 172.21.15.107:0/4017495540 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83bc011db0 con 0x556048b1d2f0 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.654+0000 7f83b4c00640 1 -- 172.21.15.107:0/4017495540 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f83bc00ee70 con 0x556048b1d2f0 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.654+0000 7f83d8df0740 10 client.25840 fetch_fsmap learned FSMap version 35 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.654+0000 7f83d8df0740 1 -- 172.21.15.107:0/4017495540 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556048f084d0 con 0x556048b1d2f0 2024-07-26T12:50:46.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.654+0000 7f83b4c00640 1 -- 172.21.15.107:0/4017495540 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83bc011db0 con 0x556048b1d2f0 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.655+0000 7f83b4c00640 1 -- 172.21.15.107:0/4017495540 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7f83bc022070 con 0x556048b1d2f0 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.655+0000 7f83d8df0740 10 client.25840 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.655+0000 7f83d8df0740 10 client.25840 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.655+0000 7f83d8df0740 20 client.25840 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.655+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556048b04d80 0x556048f0b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.655+0000 7f83d8df0740 4 client.25840 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.655+0000 7f83d8df0740 1 -- 172.21.15.107:0/4017495540 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556048e601c0 con 0x556048b04d80 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.655+0000 7f83ca000640 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556048b04d80 0x556048f0b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.656+0000 7f83ca000640 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556048b04d80 0x556048f0b940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cfbf4968efeba9 server_cookie=765772cd75081989 in_seq=0 out_seq=0 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.656+0000 7f83b4c00640 10 client.25840 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.656+0000 7f83b4c00640 1 -- 172.21.15.107:0/4017495540 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556048e601c0 con 0x556048b04d80 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.656+0000 7f83b4c00640 10 client.25840 handle_command_reply: tid=0 2024-07-26T12:50:46.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 client.25840 shutdown 2024-07-26T12:50:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 -- 172.21.15.107:0/4017495540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 msgr2=0x556048ef98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048ef98f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00e9d0 tx=0x7f83bc00ec80 comp rx=0 tx=0).stop 2024-07-26T12:50:46.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 -- 172.21.15.107:0/4017495540 shutdown_connections 2024-07-26T12:50:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556048b04d80 0x556048f0b940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b461b0 0x556048f058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048ef9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83b4c00640 0 client.25840 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4017495540 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048ef98f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 -- 172.21.15.107:0/4017495540 >> 172.21.15.107:0/4017495540 conn(0x556048b3a300 msgr2=0x556048ee60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 -- 172.21.15.107:0/4017495540 shutdown_connections 2024-07-26T12:50:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 1 -- 172.21.15.107:0/4017495540 wait complete. 2024-07-26T12:50:46.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.657+0000 7f83d8df0740 20 client.25840 trim_cache size 0 max 16384 2024-07-26T12:50:46.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.674+0000 7f83d8df0740 1 Processor -- start 2024-07-26T12:50:46.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.674+0000 7f83d8df0740 1 -- start start 2024-07-26T12:50:46.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048efb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048f056c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b461b0 0x556048f05c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.676 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83d8df0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc1ae6a0 con 0x556048b461b0 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83d8df0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc1b02a0 con 0x556048b254d0 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83d8df0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc1aed70 con 0x556048b1d2f0 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83ca000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048f056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83cb400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b461b0 0x556048f05c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83ca000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048f056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:38544/0 (socket says 172.21.15.107:38544) 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83ca000640 1 -- 172.21.15.107:0/3499478690 learned_addr learned my addr 172.21.15.107:0/3499478690 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.675+0000 7f83caa00640 1 --2- 172.21.15.107:0/3499478690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048efb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.676+0000 7f83ca000640 1 -- 172.21.15.107:0/3499478690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 msgr2=0x556048efb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.676+0000 7f83ca000640 1 --2- 172.21.15.107:0/3499478690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048efb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.676+0000 7f83ca000640 1 -- 172.21.15.107:0/3499478690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b461b0 msgr2=0x556048f05c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.676+0000 7f83ca000640 1 --2- 172.21.15.107:0/3499478690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b461b0 0x556048f05c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.676+0000 7f83ca000640 1 -- 172.21.15.107:0/3499478690 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556048ee1b60 con 0x556048b254d0 2024-07-26T12:50:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.676+0000 7f83ca000640 1 --2- 172.21.15.107:0/3499478690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048f056c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f83b8004650 tx=0x7f83b802a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69c0ae81ab006aa7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.676+0000 7f83b7400640 1 -- 172.21.15.107:0/3499478690 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b805a070 con 0x556048b254d0 2024-07-26T12:50:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.676+0000 7f83b7400640 1 -- 172.21.15.107:0/3499478690 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83b8004da0 con 0x556048b254d0 2024-07-26T12:50:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.676+0000 7f83b7400640 1 -- 172.21.15.107:0/3499478690 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8035040 con 0x556048b254d0 2024-07-26T12:50:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.677+0000 7f83d8df0740 1 -- 172.21.15.107:0/3499478690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 msgr2=0x556048f056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.677+0000 7f83d8df0740 1 --2- 172.21.15.107:0/3499478690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048f056c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f83b8004650 tx=0x7f83b802a3f0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.677+0000 7f83d8df0740 1 -- 172.21.15.107:0/3499478690 shutdown_connections 2024-07-26T12:50:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.677+0000 7f83d8df0740 1 --2- 172.21.15.107:0/3499478690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b461b0 0x556048f05c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.677+0000 7f83d8df0740 1 --2- 172.21.15.107:0/3499478690 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b254d0 0x556048f056c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.679 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.677+0000 7f83d8df0740 1 --2- 172.21.15.107:0/3499478690 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048efb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.677+0000 7f83d8df0740 1 -- 172.21.15.107:0/3499478690 >> 172.21.15.107:0/3499478690 conn(0x556048b3a300 msgr2=0x556048ee6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.677+0000 7f83d8df0740 1 -- 172.21.15.107:0/3499478690 shutdown_connections 2024-07-26T12:50:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.678+0000 7f83d8df0740 1 -- 172.21.15.107:0/3499478690 wait complete. 2024-07-26T12:50:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.678+0000 7f83d8df0740 1 -- start start 2024-07-26T12:50:46.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.678+0000 7f83d8df0740 1 Processor -- start 2024-07-26T12:50:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048f02000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b254d0 0x556048efe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83d8df0740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 0x556048efeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83d8df0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc1afbd0 con 0x556048b254d0 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83d8df0740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc1af440 con 0x556048b461b0 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83d8df0740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f83cc1ac680 con 0x556048b1d2f0 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83d8df0740 10 client.0 _ll_register_callbacks cb 0x7f83bc00be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83ca000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b254d0 0x556048efe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83caa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048f02000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83ca000640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b254d0 0x556048efe4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60020/0 (socket says 172.21.15.107:60020) 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83ca000640 1 -- 172.21.15.107:0/4257209486 learned_addr learned my addr 172.21.15.107:0/4257209486 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83caa00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048f02000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:57478/0 (socket says 172.21.15.107:57478) 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.679+0000 7f83cb400640 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 0x556048efeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83ca000640 1 -- 172.21.15.107:0/4257209486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 msgr2=0x556048f02000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83ca000640 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048f02000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83ca000640 1 -- 172.21.15.107:0/4257209486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 msgr2=0x556048efeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83ca000640 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 0x556048efeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83ca000640 1 -- 172.21.15.107:0/4257209486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556048eff270 con 0x556048b254d0 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83caa00640 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048f02000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83cb400640 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 0x556048efeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83ca000640 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b254d0 0x556048efe4b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009230 tx=0x7f83b80056c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91648f11fc49f1e2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83b4c00640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83b4c00640 1 -- 172.21.15.107:0/4257209486 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8063060 con 0x556048b254d0 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83b4c00640 1 -- 172.21.15.107:0/4257209486 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f83b80035f0 con 0x556048b254d0 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83d8df0740 1 -- 172.21.15.107:0/4257209486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83b8004da0 con 0x556048b254d0 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83b4c00640 1 -- 172.21.15.107:0/4257209486 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f83b805a030 con 0x556048b254d0 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83d8df0740 10 client.15702 fetch_fsmap learned FSMap version 35 2024-07-26T12:50:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.680+0000 7f83d8df0740 1 -- 172.21.15.107:0/4257209486 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556048f73120 con 0x556048b254d0 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.681+0000 7f83b4c00640 1 -- 172.21.15.107:0/4257209486 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83b8004da0 con 0x556048b254d0 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.681+0000 7f83b4c00640 1 -- 172.21.15.107:0/4257209486 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7f83b8032af0 con 0x556048b254d0 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.681+0000 7f83d8df0740 10 client.15702 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.681+0000 7f83d8df0740 10 client.15702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.681+0000 7f83d8df0740 20 client.15702 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.681+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556048ada1c0 0x556048f758f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.681+0000 7f83d8df0740 4 client.15702 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.681+0000 7f83d8df0740 1 -- 172.21.15.107:0/4257209486 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556048e5bc60 con 0x556048ada1c0 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.681+0000 7f83caa00640 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556048ada1c0 0x556048f758f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.682+0000 7f83caa00640 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556048ada1c0 0x556048f758f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4d3dfa3d0b2aa25 server_cookie=3d5f793f18a54d1c in_seq=0 out_seq=0 2024-07-26T12:50:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.682+0000 7f83b4c00640 10 client.15702 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.837+0000 7f83b4c00640 1 -- 172.21.15.107:0/4257209486 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556048e5bc60 con 0x556048ada1c0 2024-07-26T12:50:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.837+0000 7f83b4c00640 10 client.15702 handle_command_reply: tid=0 2024-07-26T12:50:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.837+0000 7f83d8df0740 1 client.15702 shutdown 2024-07-26T12:50:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.837+0000 7f83d8df0740 1 -- 172.21.15.107:0/4257209486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b254d0 msgr2=0x556048efe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.837+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b254d0 0x556048efe4b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f83b8009230 tx=0x7f83b80056c0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.837+0000 7f83d8df0740 1 -- 172.21.15.107:0/4257209486 shutdown_connections 2024-07-26T12:50:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.837+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x556048ada1c0 0x556048f758f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.837+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x556048b461b0 0x556048efeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.837+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556048b254d0 0x556048efe4b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.837+0000 7f83d8df0740 1 --2- 172.21.15.107:0/4257209486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x556048b1d2f0 0x556048f02000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.838+0000 7f83d8df0740 1 -- 172.21.15.107:0/4257209486 >> 172.21.15.107:0/4257209486 conn(0x556048b3a300 msgr2=0x556048ee6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.838+0000 7f83b4c00640 0 client.15702 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:50:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.838+0000 7f83d8df0740 1 -- 172.21.15.107:0/4257209486 shutdown_connections 2024-07-26T12:50:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.838+0000 7f83d8df0740 1 -- 172.21.15.107:0/4257209486 wait complete. 2024-07-26T12:50:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.838+0000 7f83d8df0740 20 client.15702 trim_cache size 0 max 16384 2024-07-26T12:50:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-07-26T12:50:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-07-26T12:50:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-07-26T12:50:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-07-26T12:50:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.839+0000 7f83d1e00640 1 -- 172.21.15.107:0/732195755 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f83b00b3d40 msgr2=0x7f83b00b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.839+0000 7f83d1e00640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f83b00b3d40 0x7f83b00b6250 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f83b80096c0 tx=0x7f83b8041040 comp rx=0 tx=0).stop 2024-07-26T12:50:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.839+0000 7f83d1e00640 1 -- 172.21.15.107:0/732195755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83cc108660 msgr2=0x7f83cc19f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.839+0000 7f83d1e00640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83cc108660 0x7f83cc19f6e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f83c4009fd0 tx=0x7f83c400ef10 comp rx=0 tx=0).stop 2024-07-26T12:50:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.839+0000 7f83d1e00640 1 -- 172.21.15.107:0/732195755 shutdown_connections 2024-07-26T12:50:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.839+0000 7f83d1e00640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f83b00b3d40 0x7f83b00b6250 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.839+0000 7f83d1e00640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f83cc108660 0x7f83cc19f6e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.839+0000 7f83d1e00640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83cc107d10 0x7f83cc19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.839+0000 7f83d1e00640 1 --2- 172.21.15.107:0/732195755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83cc1023e0 0x7f83cc199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.839+0000 7f83d1e00640 1 -- 172.21.15.107:0/732195755 >> 172.21.15.107:0/732195755 conn(0x7f83cc07a170 msgr2=0x7f83cc105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.840+0000 7f83d1e00640 1 -- 172.21.15.107:0/732195755 shutdown_connections 2024-07-26T12:50:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:50:46.840+0000 7f83d1e00640 1 -- 172.21.15.107:0/732195755 wait complete. 2024-07-26T12:50:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:46 smithi107 ceph-mon[32659]: pgmap v535: 129 pgs: 129 active+clean; 35 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 463 op/s 2024-07-26T12:50:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:46 smithi132 ceph-mon[37580]: pgmap v535: 129 pgs: 129 active+clean; 35 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 463 op/s 2024-07-26T12:50:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:46 smithi112 ceph-mon[37501]: pgmap v535: 129 pgs: 129 active+clean; 35 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 463 op/s 2024-07-26T12:50:47.606 INFO:tasks.workunit.client.0.smithi107.stdout:checking for ANSI C header files... yes 2024-07-26T12:50:47.681 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/types.h... yes 2024-07-26T12:50:47.739 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T12:50:47.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T12:50:47.766 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/stat.h... yes 2024-07-26T12:50:47.846 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdlib.h... yes 2024-07-26T12:50:47.928 INFO:tasks.workunit.client.0.smithi107.stdout:checking for string.h... yes 2024-07-26T12:50:48.011 INFO:tasks.workunit.client.0.smithi107.stdout:checking for memory.h... yes 2024-07-26T12:50:48.087 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strings.h... yes 2024-07-26T12:50:48.162 INFO:tasks.workunit.client.0.smithi107.stdout:checking for inttypes.h... yes 2024-07-26T12:50:48.176 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:50:48.242 INFO:tasks.workunit.client.0.smithi107.stdout:checking for stdint.h... yes 2024-07-26T12:50:48.328 INFO:tasks.workunit.client.0.smithi107.stdout:checking for unistd.h... yes 2024-07-26T12:50:48.445 INFO:tasks.workunit.client.0.smithi107.stdout:checking for dlfcn.h... yes 2024-07-26T12:50:48.463 INFO:tasks.workunit.client.0.smithi107.stdout:checking for objdir... .libs 2024-07-26T12:50:48.702 INFO:tasks.workunit.client.0.smithi107.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-07-26T12:50:48.702 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-07-26T12:50:48.771 INFO:tasks.workunit.client.0.smithi107.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-07-26T12:50:48.873 INFO:tasks.workunit.client.0.smithi107.stdout:checking if gcc static flag -static works... no 2024-07-26T12:50:49.008 INFO:tasks.workunit.client.0.smithi107.stdout:checking if gcc supports -c -o file.o... yes 2024-07-26T12:50:49.008 INFO:tasks.workunit.client.0.smithi107.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-07-26T12:50:49.032 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-07-26T12:50:49.122 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether -lc should be explicitly linked in... no 2024-07-26T12:50:49.252 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:48 smithi112 ceph-mon[37501]: pgmap v536: 129 pgs: 129 active+clean; 31 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 682 KiB/s wr, 227 op/s 2024-07-26T12:50:49.285 INFO:tasks.workunit.client.0.smithi107.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-07-26T12:50:49.285 INFO:tasks.workunit.client.0.smithi107.stdout:checking how to hardcode library paths into programs... immediate 2024-07-26T12:50:49.290 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether stripping libraries is possible... yes 2024-07-26T12:50:49.290 INFO:tasks.workunit.client.0.smithi107.stdout:checking if libtool supports shared libraries... yes 2024-07-26T12:50:49.290 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether to build shared libraries... yes 2024-07-26T12:50:49.291 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether to build static libraries... yes 2024-07-26T12:50:49.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:48 smithi107 ceph-mon[32659]: pgmap v536: 129 pgs: 129 active+clean; 31 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 682 KiB/s wr, 227 op/s 2024-07-26T12:50:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:48 smithi132 ceph-mon[37580]: pgmap v536: 129 pgs: 129 active+clean; 31 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 682 KiB/s wr, 227 op/s 2024-07-26T12:50:49.310 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc... (cached) gcc 2024-07-26T12:50:49.390 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-07-26T12:50:49.390 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc accepts -g... (cached) yes 2024-07-26T12:50:49.391 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-07-26T12:50:49.391 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gmake... /usr/bin/gmake 2024-07-26T12:50:49.391 INFO:tasks.workunit.client.0.smithi107.stderr:./configure: line 12082: test: $(SHELL): binary operator expected 2024-07-26T12:50:49.392 INFO:tasks.workunit.client.0.smithi107.stdout:checking for tar... /bin/tar 2024-07-26T12:50:49.392 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gzip... /bin/gzip 2024-07-26T12:50:49.459 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether gcc -MM is supported... yes 2024-07-26T12:50:49.460 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sort... /bin/sort 2024-07-26T12:50:49.460 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether ln -s works... yes 2024-07-26T12:50:49.461 INFO:tasks.workunit.client.0.smithi107.stdout:checking for rpm... /bin/rpm 2024-07-26T12:50:49.472 INFO:tasks.workunit.client.0.smithi107.stdout:checking for rpmbuild... no 2024-07-26T12:50:49.632 INFO:tasks.workunit.client.0.smithi107.stdout:checking assert.h usability... yes 2024-07-26T12:50:49.670 INFO:tasks.workunit.client.0.smithi107.stdout:checking assert.h presence... yes 2024-07-26T12:50:49.670 INFO:tasks.workunit.client.0.smithi107.stdout:checking for assert.h... yes 2024-07-26T12:50:49.772 INFO:tasks.workunit.client.0.smithi107.stdout:checking bstring.h usability... no 2024-07-26T12:50:49.829 INFO:tasks.workunit.client.0.smithi107.stdout:checking bstring.h presence... no 2024-07-26T12:50:49.830 INFO:tasks.workunit.client.0.smithi107.stdout:checking for bstring.h... no 2024-07-26T12:50:49.913 INFO:tasks.workunit.client.0.smithi107.stdout:checking libgen.h usability... yes 2024-07-26T12:50:49.944 INFO:tasks.workunit.client.0.smithi107.stdout:checking libgen.h presence... yes 2024-07-26T12:50:49.945 INFO:tasks.workunit.client.0.smithi107.stdout:checking for libgen.h... yes 2024-07-26T12:50:50.500 INFO:tasks.workunit.client.0.smithi107.stdout:checking dirent.h usability... yes 2024-07-26T12:50:50.531 INFO:tasks.workunit.client.0.smithi107.stdout:checking dirent.h presence... yes 2024-07-26T12:50:50.531 INFO:tasks.workunit.client.0.smithi107.stdout:checking for dirent.h... yes 2024-07-26T12:50:50.618 INFO:tasks.workunit.client.0.smithi107.stdout:checking errno.h usability... yes 2024-07-26T12:50:50.652 INFO:tasks.workunit.client.0.smithi107.stdout:checking errno.h presence... yes 2024-07-26T12:50:50.653 INFO:tasks.workunit.client.0.smithi107.stdout:checking for errno.h... yes 2024-07-26T12:50:50.741 INFO:tasks.workunit.client.0.smithi107.stdout:checking malloc.h usability... yes 2024-07-26T12:50:50.775 INFO:tasks.workunit.client.0.smithi107.stdout:checking malloc.h presence... yes 2024-07-26T12:50:50.775 INFO:tasks.workunit.client.0.smithi107.stdout:checking for malloc.h... yes 2024-07-26T12:50:50.893 INFO:tasks.workunit.client.0.smithi107.stdout:checking uuid.h usability... no 2024-07-26T12:50:50.946 INFO:tasks.workunit.client.0.smithi107.stdout:checking uuid.h presence... no 2024-07-26T12:50:50.947 INFO:tasks.workunit.client.0.smithi107.stdout:checking for uuid.h... no 2024-07-26T12:50:51.041 INFO:tasks.workunit.client.0.smithi107.stdout:checking uuid/uuid.h usability... yes 2024-07-26T12:50:51.082 INFO:tasks.workunit.client.0.smithi107.stdout:checking uuid/uuid.h presence... yes 2024-07-26T12:50:51.082 INFO:tasks.workunit.client.0.smithi107.stdout:checking for uuid/uuid.h... yes 2024-07-26T12:50:51.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.084+0000 7f51da200640 1 -- 172.21.15.107:0/3749780778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51d4103590 msgr2=0x7f51d41039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:51.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.084+0000 7f51da200640 1 --2- 172.21.15.107:0/3749780778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51d4103590 0x7f51d41039a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009900 tx=0x7f51c402a3f0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.084+0000 7f51da200640 1 -- 172.21.15.107:0/3749780778 shutdown_connections 2024-07-26T12:50:51.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.084+0000 7f51da200640 1 --2- 172.21.15.107:0/3749780778 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51d4109820 0x7f51d4110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.084+0000 7f51da200640 1 --2- 172.21.15.107:0/3749780778 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51d4108ed0 0x7f51d41092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.084+0000 7f51da200640 1 --2- 172.21.15.107:0/3749780778 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51d4103590 0x7f51d41039a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.084+0000 7f51da200640 1 -- 172.21.15.107:0/3749780778 >> 172.21.15.107:0/3749780778 conn(0x7f51d407b280 msgr2=0x7f51d407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.085+0000 7f51da200640 1 -- 172.21.15.107:0/3749780778 shutdown_connections 2024-07-26T12:50:51.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.085+0000 7f51da200640 1 -- 172.21.15.107:0/3749780778 wait complete. 2024-07-26T12:50:51.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.086+0000 7f51da200640 1 Processor -- start 2024-07-26T12:50:51.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.086+0000 7f51da200640 1 -- start start 2024-07-26T12:50:51.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.087+0000 7f51da200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51d4103590 0x7f51d419c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:51.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.087+0000 7f51da200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51d4108ed0 0x7f51d419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:51.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.087+0000 7f51da200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51d4109820 0x7f51d41a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:51.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.087+0000 7f51da200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f51d41134d0 con 0x7f51d4109820 2024-07-26T12:50:51.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.087+0000 7f51da200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f51d4113350 con 0x7f51d4108ed0 2024-07-26T12:50:51.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.087+0000 7f51da200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f51d4113650 con 0x7f51d4103590 2024-07-26T12:50:51.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.087+0000 7f51d2a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51d4108ed0 0x7f51d419c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.087+0000 7f51d2a00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51d4108ed0 0x7f51d419c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:38576/0 (socket says 172.21.15.107:38576) 2024-07-26T12:50:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.087+0000 7f51d2a00640 1 -- 172.21.15.107:0/736496391 learned_addr learned my addr 172.21.15.107:0/736496391 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.087+0000 7f51d3e00640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51d4109820 0x7f51d41a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.088+0000 7f51d2a00640 1 -- 172.21.15.107:0/736496391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51d4103590 msgr2=0x7f51d419c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.088+0000 7f51d3400640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51d4103590 0x7f51d419c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.088+0000 7f51d2a00640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51d4103590 0x7f51d419c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.088+0000 7f51d2a00640 1 -- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51d4109820 msgr2=0x7f51d41a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.088+0000 7f51d2a00640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51d4109820 0x7f51d41a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.088+0000 7f51d2a00640 1 -- 172.21.15.107:0/736496391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d41a2090 con 0x7f51d4108ed0 2024-07-26T12:50:51.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.088+0000 7f51d3e00640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51d4109820 0x7f51d41a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:50:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.088+0000 7f51d2a00640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51d4108ed0 0x7f51d419c6c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f51c000e9f0 tx=0x7f51c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.088+0000 7f51d3400640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51d4103590 0x7f51d419c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:50:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.089+0000 7f51cbe00640 1 -- 172.21.15.107:0/736496391 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c000ccc0 con 0x7f51d4108ed0 2024-07-26T12:50:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.089+0000 7f51cbe00640 1 -- 172.21.15.107:0/736496391 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51c00108e0 con 0x7f51d4108ed0 2024-07-26T12:50:51.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.089+0000 7f51cbe00640 1 -- 172.21.15.107:0/736496391 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c00188e0 con 0x7f51d4108ed0 2024-07-26T12:50:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.089+0000 7f51da200640 1 -- 172.21.15.107:0/736496391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d41a2380 con 0x7f51d4108ed0 2024-07-26T12:50:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.089+0000 7f51da200640 1 -- 172.21.15.107:0/736496391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d41a8ae0 con 0x7f51d4108ed0 2024-07-26T12:50:51.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.090+0000 7f51da200640 1 -- 172.21.15.107:0/736496391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d41039a0 con 0x7f51d4108ed0 2024-07-26T12:50:51.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.093+0000 7f51cbe00640 1 -- 172.21.15.107:0/736496391 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f51c0010430 con 0x7f51d4108ed0 2024-07-26T12:50:51.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.095+0000 7f51cbe00640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f51b80b3dc0 0x7f51b80b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:51.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.095+0000 7f51cbe00640 1 -- 172.21.15.107:0/736496391 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f51c00e48b0 con 0x7f51d4108ed0 2024-07-26T12:50:51.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.095+0000 7f51d3400640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f51b80b3dc0 0x7f51b80b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:51.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.096+0000 7f51d3400640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f51b80b3dc0 0x7f51b80b62d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f51c40096c0 tx=0x7f51c4031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.099+0000 7f51cbe00640 1 -- 172.21.15.107:0/736496391 <== mon.2 v2:172.21.15.112:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f51c0055600 con 0x7f51d4108ed0 2024-07-26T12:50:51.176 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/uuid.h usability... no 2024-07-26T12:50:51.236 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/uuid.h presence... no 2024-07-26T12:50:51.236 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/uuid.h... no 2024-07-26T12:50:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:50 smithi107 ceph-mon[32659]: pgmap v537: 129 pgs: 129 active+clean; 31 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 431 KiB/s wr, 148 op/s 2024-07-26T12:50:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:50 smithi132 ceph-mon[37580]: pgmap v537: 129 pgs: 129 active+clean; 31 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 431 KiB/s wr, 148 op/s 2024-07-26T12:50:51.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.342+0000 7f51da200640 1 -- 172.21.15.107:0/736496391 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51d41a8d10 con 0x7f51d4108ed0 2024-07-26T12:50:51.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.343+0000 7f51cbe00640 1 -- 172.21.15.107:0/736496391 <== mon.2 v2:172.21.15.112:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11997 (secure 0 0 0) 0x7f51c00ab7e0 con 0x7f51d4108ed0 2024-07-26T12:50:51.344 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-07-26T12:50:51.344 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":35,"btime":"2024-07-26T12:50:42:293847+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/4197833362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":4197833362},{"type":"v1","addr":"172.21.15.107:6837","nonce":4197833362}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14580,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6841/300465136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6840","nonce":300465136},{"type":"v1","addr":"172.21.15.107:6841","nonce":300465136}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T12:43:15.198289+0000","modified":"2024-07-26T12:50:42.011337+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24716,"mds_1":24715,"mds_2":24703,"mds_3":24694,"mds_4":24682},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24632":{"gid":24632,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6833/2931369116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6832","nonce":2931369116},{"type":"v1","addr":"172.21.15.112:6833","nonce":2931369116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24634":{"gid":24634,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.132:6833/277912155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6832","nonce":277912155},{"type":"v1","addr":"172.21.15.132:6833","nonce":277912155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24652":{"gid":24652,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6835/3403405722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3403405722},{"type":"v1","addr":"172.21.15.107:6835","nonce":3403405722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24662":{"gid":24662,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6835/1038904649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6834","nonce":1038904649},{"type":"v1","addr":"172.21.15.112:6835","nonce":1038904649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":32,"addr":"172.21.15.132:6835/1242618859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6834","nonce":1242618859},{"type":"v1","addr":"172.21.15.132:6835","nonce":1242618859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24694":{"gid":24694,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":31,"addr":"172.21.15.132:6837/735918310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6836","nonce":735918310},{"type":"v1","addr":"172.21.15.132:6837","nonce":735918310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":29,"addr":"172.21.15.132:6839/3489644052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:6838","nonce":3489644052},{"type":"v1","addr":"172.21.15.132:6839","nonce":3489644052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24715":{"gid":24715,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":28,"addr":"172.21.15.107:6839/137526780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":137526780},{"type":"v1","addr":"172.21.15.107:6839","nonce":137526780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24716":{"gid":24716,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.112:6839/3779803434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6838","nonce":3779803434},{"type":"v1","addr":"172.21.15.112:6839","nonce":3779803434}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25799":{"gid":25799,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.112:6837/540940561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6836","nonce":540940561},{"type":"v1","addr":"172.21.15.112:6837","nonce":540940561}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24716,"qdb_cluster":[24715,24716,24703,24694,24682]},"id":1}]} 2024-07-26T12:50:51.345 INFO:teuthology.orchestra.run.smithi107.stderr:dumped fsmap epoch 35 2024-07-26T12:50:51.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 -- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f51b80b3dc0 msgr2=0x7f51b80b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f51b80b3dc0 0x7f51b80b62d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f51c40096c0 tx=0x7f51c4031400 comp rx=0 tx=0).stop 2024-07-26T12:50:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 -- 172.21.15.107:0/736496391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51d4108ed0 msgr2=0x7f51d419c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51d4108ed0 0x7f51d419c6c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f51c000e9f0 tx=0x7f51c000eeb0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 -- 172.21.15.107:0/736496391 shutdown_connections 2024-07-26T12:50:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f51b80b3dc0 0x7f51b80b62d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51d4109820 0x7f51d41a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f51d4108ed0 0x7f51d419c6c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 --2- 172.21.15.107:0/736496391 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51d4103590 0x7f51d419c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:51.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 -- 172.21.15.107:0/736496391 >> 172.21.15.107:0/736496391 conn(0x7f51d407b280 msgr2=0x7f51d4106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:51.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 -- 172.21.15.107:0/736496391 shutdown_connections 2024-07-26T12:50:51.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:51.346+0000 7f51da200640 1 -- 172.21.15.107:0/736496391 wait complete. 2024-07-26T12:50:51.352 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/file.h usability... yes 2024-07-26T12:50:51.390 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/file.h presence... yes 2024-07-26T12:50:51.390 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/file.h... yes 2024-07-26T12:50:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:50 smithi112 ceph-mon[37501]: pgmap v537: 129 pgs: 129 active+clean; 31 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 431 KiB/s wr, 148 op/s 2024-07-26T12:50:51.494 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/fcntl.h usability... yes 2024-07-26T12:50:51.528 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/fcntl.h presence... yes 2024-07-26T12:50:51.528 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/fcntl.h... yes 2024-07-26T12:50:51.634 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/syssgi.h usability... no 2024-07-26T12:50:51.683 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/syssgi.h presence... no 2024-07-26T12:50:51.683 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/syssgi.h... no 2024-07-26T12:50:52.102 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/param.h usability... yes 2024-07-26T12:50:52.193 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/param.h presence... yes 2024-07-26T12:50:52.193 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/param.h... yes 2024-07-26T12:50:52.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:51 smithi107 ceph-mon[32659]: from='client.? 172.21.15.107:0/736496391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:50:52.286 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/stat.h... (cached) yes 2024-07-26T12:50:52.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T12:50:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:51 smithi132 ceph-mon[37580]: from='client.? 172.21.15.107:0/736496391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:50:52.416 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/statvfs.h usability... yes 2024-07-26T12:50:52.444 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/statvfs.h presence... yes 2024-07-26T12:50:52.445 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/statvfs.h... yes 2024-07-26T12:50:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:51 smithi112 ceph-mon[37501]: from='client.? 172.21.15.107:0/736496391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T12:50:52.530 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/time.h usability... yes 2024-07-26T12:50:52.563 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/time.h presence... yes 2024-07-26T12:50:52.563 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/time.h... yes 2024-07-26T12:50:52.664 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/ioctl.h usability... yes 2024-07-26T12:50:52.694 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/ioctl.h presence... yes 2024-07-26T12:50:52.695 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/ioctl.h... yes 2024-07-26T12:50:52.785 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:50:52.791 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/wait.h usability... yes 2024-07-26T12:50:52.826 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/wait.h presence... yes 2024-07-26T12:50:52.826 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/wait.h... yes 2024-07-26T12:50:52.852 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/types.h... (cached) yes 2024-07-26T12:50:52.865 INFO:tasks.workunit.client.0.smithi107.stdout:checking for strings.h... (cached) yes 2024-07-26T12:50:52.942 INFO:tasks.workunit.client.0.smithi107.stdout:checking err.h usability... yes 2024-07-26T12:50:52.980 INFO:tasks.workunit.client.0.smithi107.stdout:checking err.h presence... yes 2024-07-26T12:50:52.980 INFO:tasks.workunit.client.0.smithi107.stdout:checking for err.h... yes 2024-07-26T12:50:53.151 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/falloc.h usability... yes 2024-07-26T12:50:53.187 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/falloc.h presence... yes 2024-07-26T12:50:53.187 INFO:tasks.workunit.client.0.smithi107.stdout:checking for linux/falloc.h... yes 2024-07-26T12:50:53.240 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:52 smithi107 ceph-mon[32659]: pgmap v538: 129 pgs: 129 active+clean; 29 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 605 KiB/s wr, 167 op/s 2024-07-26T12:50:53.285 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-07-26T12:50:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:52 smithi132 ceph-mon[37580]: pgmap v538: 129 pgs: 129 active+clean; 29 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 605 KiB/s wr, 167 op/s 2024-07-26T12:50:53.339 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-07-26T12:50:53.339 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/fs/xfs_fsops.h... no 2024-07-26T12:50:53.429 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/fs/xfs_itable.h usability... no 2024-07-26T12:50:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:52 smithi112 ceph-mon[37501]: pgmap v538: 129 pgs: 129 active+clean; 29 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 605 KiB/s wr, 167 op/s 2024-07-26T12:50:53.477 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/fs/xfs_itable.h presence... no 2024-07-26T12:50:53.478 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/fs/xfs_itable.h... no 2024-07-26T12:50:53.564 INFO:tasks.workunit.client.0.smithi107.stdout:checking xfs/platform_defs.h usability... no 2024-07-26T12:50:53.622 INFO:tasks.workunit.client.0.smithi107.stdout:checking xfs/platform_defs.h presence... no 2024-07-26T12:50:53.622 INFO:tasks.workunit.client.0.smithi107.stdout:checking for xfs/platform_defs.h... no 2024-07-26T12:50:53.721 INFO:tasks.workunit.client.0.smithi107.stdout:checking btrfs/ioctl.h usability... no 2024-07-26T12:50:53.784 INFO:tasks.workunit.client.0.smithi107.stdout:checking btrfs/ioctl.h presence... no 2024-07-26T12:50:53.784 INFO:tasks.workunit.client.0.smithi107.stdout:checking for btrfs/ioctl.h... no 2024-07-26T12:50:53.876 INFO:tasks.workunit.client.0.smithi107.stdout:checking cifs/ioctl.h usability... no 2024-07-26T12:50:53.939 INFO:tasks.workunit.client.0.smithi107.stdout:checking cifs/ioctl.h presence... no 2024-07-26T12:50:53.939 INFO:tasks.workunit.client.0.smithi107.stdout:checking for cifs/ioctl.h... no 2024-07-26T12:50:54.068 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/mman.h usability... yes 2024-07-26T12:50:54.106 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/mman.h presence... yes 2024-07-26T12:50:54.106 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/mman.h... yes 2024-07-26T12:50:54.222 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/ext4.h usability... no 2024-07-26T12:50:54.284 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/ext4.h presence... no 2024-07-26T12:50:54.285 INFO:tasks.workunit.client.0.smithi107.stdout:checking for linux/ext4.h... no 2024-07-26T12:50:54.360 INFO:tasks.workunit.client.0.smithi107.stdout:checking for xfs/xfs_log_format.h... no 2024-07-26T12:50:54.477 INFO:tasks.workunit.client.0.smithi107.stdout:checking for uuid_compare... no 2024-07-26T12:50:54.568 INFO:tasks.workunit.client.0.smithi107.stdout:checking for uuid_compare in -luuid... yes 2024-07-26T12:50:54.738 INFO:tasks.workunit.client.0.smithi107.stdout:checking for xfs/xfs.h... yes 2024-07-26T12:50:54.834 INFO:tasks.workunit.client.0.smithi107.stdout:checking for xfs/libxfs.h... no 2024-07-26T12:50:54.910 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether xlog_assign_lsn is declared... no 2024-07-26T12:50:54.997 INFO:tasks.workunit.client.0.smithi107.stdout:checking for xfs/xqm.h... yes 2024-07-26T12:50:54.999 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:54 smithi107 ceph-mon[32659]: pgmap v539: 129 pgs: 129 active+clean; 30 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 478 KiB/s wr, 113 op/s 2024-07-26T12:50:55.101 INFO:tasks.workunit.client.0.smithi107.stdout:checking xfsctl from xfs/xfs.h... ok 2024-07-26T12:50:55.174 INFO:tasks.workunit.client.0.smithi107.stdout:checking xfs/handle.h usability... yes 2024-07-26T12:50:55.210 INFO:tasks.workunit.client.0.smithi107.stdout:checking xfs/handle.h presence... yes 2024-07-26T12:50:55.210 INFO:tasks.workunit.client.0.smithi107.stdout:checking for xfs/handle.h... yes 2024-07-26T12:50:55.295 INFO:tasks.workunit.client.0.smithi107.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-07-26T12:50:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:54 smithi132 ceph-mon[37580]: pgmap v539: 129 pgs: 129 active+clean; 30 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 478 KiB/s wr, 113 op/s 2024-07-26T12:50:55.367 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/acl.h usability... yes 2024-07-26T12:50:55.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.387+0000 7f7b4cc00640 1 -- 172.21.15.107:0/2906045887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b481033e0 msgr2=0x7f7b481037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:55.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.387+0000 7f7b4cc00640 1 --2- 172.21.15.107:0/2906045887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b481033e0 0x7f7b481037f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009900 tx=0x7f7b3802a410 comp rx=0 tx=0).stop 2024-07-26T12:50:55.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.387+0000 7f7b4cc00640 1 -- 172.21.15.107:0/2906045887 shutdown_connections 2024-07-26T12:50:55.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.387+0000 7f7b4cc00640 1 --2- 172.21.15.107:0/2906045887 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b48109670 0x7f7b4810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.387+0000 7f7b4cc00640 1 --2- 172.21.15.107:0/2906045887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b48108d20 0x7f7b48109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.387+0000 7f7b4cc00640 1 --2- 172.21.15.107:0/2906045887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b481033e0 0x7f7b481037f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.387+0000 7f7b4cc00640 1 -- 172.21.15.107:0/2906045887 >> 172.21.15.107:0/2906045887 conn(0x7f7b480fee70 msgr2=0x7f7b481012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.387+0000 7f7b4cc00640 1 -- 172.21.15.107:0/2906045887 shutdown_connections 2024-07-26T12:50:55.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.387+0000 7f7b4cc00640 1 -- 172.21.15.107:0/2906045887 wait complete. 2024-07-26T12:50:55.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.389+0000 7f7b4cc00640 1 Processor -- start 2024-07-26T12:50:55.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.389+0000 7f7b4cc00640 1 -- start start 2024-07-26T12:50:55.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b4cc00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b481033e0 0x7f7b48075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b4cc00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b48108d20 0x7f7b48075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b4cc00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b48109670 0x7f7b48075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b4cc00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48113320 con 0x7f7b48109670 2024-07-26T12:50:55.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b4cc00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481131a0 con 0x7f7b481033e0 2024-07-26T12:50:55.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b4cc00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481134a0 con 0x7f7b48108d20 2024-07-26T12:50:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b47e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b48109670 0x7f7b48075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b3ea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b48108d20 0x7f7b48075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b47400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b481033e0 0x7f7b48075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b3ea00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b48108d20 0x7f7b48075680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:44330/0 (socket says 172.21.15.107:44330) 2024-07-26T12:50:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.390+0000 7f7b3ea00640 1 -- 172.21.15.107:0/486521417 learned_addr learned my addr 172.21.15.107:0/486521417 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.391+0000 7f7b3ea00640 1 -- 172.21.15.107:0/486521417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b481033e0 msgr2=0x7f7b48075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.391+0000 7f7b3ea00640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b481033e0 0x7f7b48075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.391+0000 7f7b3ea00640 1 -- 172.21.15.107:0/486521417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b48109670 msgr2=0x7f7b48075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.391+0000 7f7b3ea00640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b48109670 0x7f7b48075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.391+0000 7f7b3ea00640 1 -- 172.21.15.107:0/486521417 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b481a8070 con 0x7f7b48108d20 2024-07-26T12:50:55.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.391+0000 7f7b3ea00640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b48108d20 0x7f7b48075680 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000e9f0 tx=0x7f7b3000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:55.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.391+0000 7f7b44c00640 1 -- 172.21.15.107:0/486521417 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3000ccc0 con 0x7f7b48108d20 2024-07-26T12:50:55.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.391+0000 7f7b44c00640 1 -- 172.21.15.107:0/486521417 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b300108e0 con 0x7f7b48108d20 2024-07-26T12:50:55.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.391+0000 7f7b44c00640 1 -- 172.21.15.107:0/486521417 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b30018880 con 0x7f7b48108d20 2024-07-26T12:50:55.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.391+0000 7f7b4cc00640 1 -- 172.21.15.107:0/486521417 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b481a8360 con 0x7f7b48108d20 2024-07-26T12:50:55.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.392+0000 7f7b4cc00640 1 -- 172.21.15.107:0/486521417 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b481a8850 con 0x7f7b48108d20 2024-07-26T12:50:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.395+0000 7f7b54e9e740 1 Processor -- start 2024-07-26T12:50:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.395+0000 7f7b54e9e740 1 -- start start 2024-07-26T12:50:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b44c00640 1 -- 172.21.15.107:0/486521417 <== mon.1 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f7b30010430 con 0x7f7b48108d20 2024-07-26T12:50:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5df1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5df1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5df1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5645e5d67bf0 con 0x5645e5a231b0 2024-07-26T12:50:55.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5645e5d570f0 con 0x5645e5a2b390 2024-07-26T12:50:55.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5645e5d5b130 con 0x5645e5a4c070 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b47e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5df1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b47e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5df1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:44334/0 (socket says 172.21.15.107:44334) 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b47e00640 1 -- 172.21.15.107:0/171254238 learned_addr learned my addr 172.21.15.107:0/171254238 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b47400640 1 --2- 172.21.15.107:0/171254238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5df1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.396+0000 7f7b3ea00640 1 --2- 172.21.15.107:0/171254238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5df1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.397+0000 7f7b47e00640 1 -- 172.21.15.107:0/171254238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 msgr2=0x5645e5df1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.397+0000 7f7b47e00640 1 --2- 172.21.15.107:0/171254238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5df1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.397+0000 7f7b47e00640 1 -- 172.21.15.107:0/171254238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 msgr2=0x5645e5df1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.397+0000 7f7b47e00640 1 --2- 172.21.15.107:0/171254238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5df1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.397+0000 7f7b47e00640 1 -- 172.21.15.107:0/171254238 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645e5ab7650 con 0x5645e5a4c070 2024-07-26T12:50:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.397+0000 7f7b3ea00640 1 --2- 172.21.15.107:0/171254238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5df1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:55.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.397+0000 7f7b47e00640 1 --2- 172.21.15.107:0/171254238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5df1220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7b40011440 tx=0x7f7b40011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bad6fb1104a112c1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:55.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.397+0000 7f7b37e00640 1 -- 172.21.15.107:0/171254238 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40003c30 con 0x5645e5a4c070 2024-07-26T12:50:55.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.397+0000 7f7b37e00640 1 -- 172.21.15.107:0/171254238 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b40003dd0 con 0x5645e5a4c070 2024-07-26T12:50:55.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.397+0000 7f7b37e00640 1 -- 172.21.15.107:0/171254238 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4000fae0 con 0x5645e5a4c070 2024-07-26T12:50:55.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.398+0000 7f7b44c00640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b2c0b3dc0 0x7f7b2c0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.398+0000 7f7b44c00640 1 -- 172.21.15.107:0/486521417 <== mon.1 v2:172.21.15.132:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f7b300e46f0 con 0x7f7b48108d20 2024-07-26T12:50:55.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.398+0000 7f7b47400640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b2c0b3dc0 0x7f7b2c0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.399+0000 7f7b54e9e740 1 -- 172.21.15.107:0/171254238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 msgr2=0x5645e5df1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:55.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.399+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/171254238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5df1220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7b40011440 tx=0x7f7b40011900 comp rx=0 tx=0).stop 2024-07-26T12:50:55.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.399+0000 7f7b54e9e740 1 -- 172.21.15.107:0/171254238 shutdown_connections 2024-07-26T12:50:55.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.399+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/171254238 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5df1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.399+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/171254238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5df1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.399+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/171254238 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5df1220 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.399+0000 7f7b54e9e740 1 -- 172.21.15.107:0/171254238 >> 172.21.15.107:0/171254238 conn(0x5645e5a40460 msgr2=0x5645e5def110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:55.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.399+0000 7f7b54e9e740 1 -- 172.21.15.107:0/171254238 shutdown_connections 2024-07-26T12:50:55.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.399+0000 7f7b47400640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b2c0b3dc0 0x7f7b2c0b62d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7b38000c00 tx=0x7f7b38031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:55.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.399+0000 7f7b54e9e740 1 -- 172.21.15.107:0/171254238 wait complete. 2024-07-26T12:50:55.401 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.400+0000 7f7b54e9e740 1 -- start start 2024-07-26T12:50:55.402 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/acl.h presence... yes 2024-07-26T12:50:55.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.400+0000 7f7b54e9e740 1 Processor -- start 2024-07-26T12:50:55.402 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/acl.h... yes 2024-07-26T12:50:55.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5dfc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a2b390 0x5645e5dfcca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a4c070 0x5645e5dfd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5645e5bba700 con 0x5645e5a231b0 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5645e5cf36c0 con 0x5645e5a4c070 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5645e5dffa00 con 0x5645e5a2b390 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b54e9e740 10 client.0 _ll_register_callbacks cb 0x5645e5d03170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b47400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5dfc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b47400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5dfc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59334/0 (socket says 172.21.15.107:59334) 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b47400640 1 -- 172.21.15.107:0/3177161254 learned_addr learned my addr 172.21.15.107:0/3177161254 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.401+0000 7f7b47e00640 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a4c070 0x5645e5dfd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b47e00640 1 -- 172.21.15.107:0/3177161254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a2b390 msgr2=0x5645e5dfcca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b47e00640 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a2b390 0x5645e5dfcca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b47e00640 1 -- 172.21.15.107:0/3177161254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 msgr2=0x5645e5dfc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b47e00640 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5dfc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b47e00640 1 -- 172.21.15.107:0/3177161254 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645e5d0a860 con 0x5645e5a4c070 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b47400640 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5dfc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b47e00640 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a4c070 0x5645e5dfd1e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7b4001b040 tx=0x7f7b40003d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=612b12bfa10d15c5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b34c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b34c00640 1 -- 172.21.15.107:0/3177161254 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40028070 con 0x5645e5a4c070 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b54e9e740 1 -- 172.21.15.107:0/3177161254 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b40003dd0 con 0x5645e5a4c070 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.402+0000 7f7b34c00640 1 -- 172.21.15.107:0/3177161254 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f7b4000fa30 con 0x5645e5a4c070 2024-07-26T12:50:55.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.403+0000 7f7b34c00640 1 -- 172.21.15.107:0/3177161254 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7b40034050 con 0x5645e5a4c070 2024-07-26T12:50:55.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.403+0000 7f7b54e9e740 10 client.25853 fetch_fsmap learned FSMap version 35 2024-07-26T12:50:55.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.403+0000 7f7b54e9e740 1 -- 172.21.15.107:0/3177161254 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5645e5e0b590 con 0x5645e5a4c070 2024-07-26T12:50:55.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.403+0000 7f7b34c00640 1 -- 172.21.15.107:0/3177161254 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b40003dd0 con 0x5645e5a4c070 2024-07-26T12:50:55.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.403+0000 7f7b34c00640 1 -- 172.21.15.107:0/3177161254 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7f7b40004550 con 0x5645e5a4c070 2024-07-26T12:50:55.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.403+0000 7f7b54e9e740 10 client.25853 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:50:55.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.403+0000 7f7b54e9e740 10 client.25853 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-07-26T12:50:55.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.403+0000 7f7b54e9e740 20 client.25853 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:55.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.404+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x5645e5a0a9f0 0x5645e5e0e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.404+0000 7f7b54e9e740 4 client.25853 mds_command: new command op to 24682 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:50:55.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.404+0000 7f7b54e9e740 1 -- 172.21.15.107:0/3177161254 --> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645e5d69b40 con 0x5645e5a0a9f0 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.404+0000 7f7b47400640 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x5645e5a0a9f0 0x5645e5e0e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.405+0000 7f7b47400640 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x5645e5a0a9f0 0x5645e5e0e9b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2fb5418c8af08c59 server_cookie=60faa5fe4b66a124 in_seq=0 out_seq=0 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.405+0000 7f7b34c00640 10 client.25853 ms_handle_connect on v2:172.21.15.132:6834/1242618859 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.405+0000 7f7b34c00640 1 -- 172.21.15.107:0/3177161254 <== mds.4 v2:172.21.15.132:6834/1242618859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5645e5d69b40 con 0x5645e5a0a9f0 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b34c00640 10 client.25853 handle_command_reply: tid=0 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 client.25853 shutdown 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 -- 172.21.15.107:0/3177161254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a4c070 msgr2=0x5645e5dfd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a4c070 0x5645e5dfd1e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7b4001b040 tx=0x7f7b40003d60 comp rx=0 tx=0).stop 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 -- 172.21.15.107:0/3177161254 shutdown_connections 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x5645e5a0a9f0 0x5645e5e0e9b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a4c070 0x5645e5dfd1e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a2b390 0x5645e5dfcca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b34c00640 0 client.25853 ms_handle_reset on v2:172.21.15.132:6834/1242618859 2024-07-26T12:50:55.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/3177161254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5dfc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 -- 172.21.15.107:0/3177161254 >> 172.21.15.107:0/3177161254 conn(0x5645e5a40460 msgr2=0x5645e5df43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:55.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 -- 172.21.15.107:0/3177161254 shutdown_connections 2024-07-26T12:50:55.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 1 -- 172.21.15.107:0/3177161254 wait complete. 2024-07-26T12:50:55.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.406+0000 7f7b54e9e740 20 client.25853 trim_cache size 0 max 16384 2024-07-26T12:50:55.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.422+0000 7f7b54e9e740 1 Processor -- start 2024-07-26T12:50:55.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.422+0000 7f7b54e9e740 1 -- start start 2024-07-26T12:50:55.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a231b0 0x5645e5dfe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a2b390 0x5645e5df2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5dfcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481b2a30 con 0x5645e5a2b390 2024-07-26T12:50:55.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481b4630 con 0x5645e5a231b0 2024-07-26T12:50:55.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481b3100 con 0x5645e5a4c070 2024-07-26T12:50:55.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b47400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a231b0 0x5645e5dfe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.424 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b47e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5dfcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b47400640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a231b0 0x5645e5dfe750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44684/0 (socket says 172.21.15.107:44684) 2024-07-26T12:50:55.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b47e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5dfcde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:44346/0 (socket says 172.21.15.107:44346) 2024-07-26T12:50:55.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b47400640 1 -- 172.21.15.107:0/1465844981 learned_addr learned my addr 172.21.15.107:0/1465844981 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:55.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.423+0000 7f7b3ea00640 1 --2- 172.21.15.107:0/1465844981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a2b390 0x5645e5df2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.424+0000 7f7b47e00640 1 -- 172.21.15.107:0/1465844981 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a231b0 msgr2=0x5645e5dfe750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.424+0000 7f7b47e00640 1 --2- 172.21.15.107:0/1465844981 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a231b0 0x5645e5dfe750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.424+0000 7f7b47e00640 1 -- 172.21.15.107:0/1465844981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a2b390 msgr2=0x5645e5df2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.424+0000 7f7b47e00640 1 --2- 172.21.15.107:0/1465844981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a2b390 0x5645e5df2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.424+0000 7f7b47e00640 1 -- 172.21.15.107:0/1465844981 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645e5dfda20 con 0x5645e5a4c070 2024-07-26T12:50:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.424+0000 7f7b47400640 1 --2- 172.21.15.107:0/1465844981 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a231b0 0x5645e5dfe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.424+0000 7f7b47e00640 1 --2- 172.21.15.107:0/1465844981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5dfcde0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b40011440 tx=0x7f7b40009d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61e63db91e252f24 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.424+0000 7f7b37400640 1 -- 172.21.15.107:0/1465844981 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40028040 con 0x5645e5a4c070 2024-07-26T12:50:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.424+0000 7f7b37400640 1 -- 172.21.15.107:0/1465844981 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b4000f5b0 con 0x5645e5a4c070 2024-07-26T12:50:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.424+0000 7f7b37400640 1 -- 172.21.15.107:0/1465844981 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40005860 con 0x5645e5a4c070 2024-07-26T12:50:55.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.425+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1465844981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 msgr2=0x5645e5dfcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:55.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.425+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/1465844981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5dfcde0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b40011440 tx=0x7f7b40009d80 comp rx=0 tx=0).stop 2024-07-26T12:50:55.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.425+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1465844981 shutdown_connections 2024-07-26T12:50:55.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.425+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/1465844981 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5dfcde0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.425+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/1465844981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a2b390 0x5645e5df2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.425+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/1465844981 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a231b0 0x5645e5dfe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.425+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1465844981 >> 172.21.15.107:0/1465844981 conn(0x5645e5a40460 msgr2=0x5645e5dfc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:55.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.426+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1465844981 shutdown_connections 2024-07-26T12:50:55.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.426+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1465844981 wait complete. 2024-07-26T12:50:55.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.426+0000 7f7b54e9e740 1 -- start start 2024-07-26T12:50:55.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b54e9e740 1 Processor -- start 2024-07-26T12:50:55.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5e01fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5e025c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b54e9e740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5e74fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481b3f60 con 0x5645e5a231b0 2024-07-26T12:50:55.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481b37d0 con 0x5645e5a2b390 2024-07-26T12:50:55.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b54e9e740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481b0a10 con 0x5645e5a4c070 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b54e9e740 10 client.0 _ll_register_callbacks cb 0x7f7b40009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b3ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5e025c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b3ea00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5e025c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44692/0 (socket says 172.21.15.107:44692) 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b3ea00640 1 -- 172.21.15.107:0/1447814819 learned_addr learned my addr 172.21.15.107:0/1447814819 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.427+0000 7f7b47400640 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5e01fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.428+0000 7f7b47e00640 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5e74fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.428+0000 7f7b3ea00640 1 -- 172.21.15.107:0/1447814819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 msgr2=0x5645e5e74fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.428+0000 7f7b3ea00640 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5e74fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.428+0000 7f7b3ea00640 1 -- 172.21.15.107:0/1447814819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 msgr2=0x5645e5e01fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.428+0000 7f7b3ea00640 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5e01fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.428+0000 7f7b3ea00640 1 -- 172.21.15.107:0/1447814819 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645e5e75520 con 0x5645e5a2b390 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.428+0000 7f7b47e00640 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5e74fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.428+0000 7f7b3ea00640 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5e025c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7b300a6fa0 tx=0x7f7b30005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=630a42c16121bc9f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.428+0000 7f7b34c00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.428+0000 7f7b34c00640 1 -- 172.21.15.107:0/1447814819 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b300aa690 con 0x5645e5a2b390 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b34c00640 1 -- 172.21.15.107:0/1447814819 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f7b300e42b0 con 0x5645e5a2b390 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1447814819 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b4000f5b0 con 0x5645e5a2b390 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b34c00640 1 -- 172.21.15.107:0/1447814819 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7b300e1ef0 con 0x5645e5a2b390 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b54e9e740 10 client.25859 fetch_fsmap learned FSMap version 35 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1447814819 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5645e5e766d0 con 0x5645e5a2b390 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b34c00640 1 -- 172.21.15.107:0/1447814819 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b4000f5b0 con 0x5645e5a2b390 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b34c00640 1 -- 172.21.15.107:0/1447814819 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7f7b300e4ae0 con 0x5645e5a2b390 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b54e9e740 10 client.25859 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:50:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b54e9e740 10 client.25859 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-07-26T12:50:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b54e9e740 20 client.25859 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x5645e59dfda0 0x5645e5e78e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b54e9e740 4 client.25859 mds_command: new command op to 24682 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T12:50:55.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.429+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1447814819 --> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5645e5d655e0 con 0x5645e59dfda0 2024-07-26T12:50:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.430+0000 7f7b47400640 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x5645e59dfda0 0x5645e5e78e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.430+0000 7f7b34c00640 10 client.25859 ms_handle_connect on v2:172.21.15.132:6834/1242618859 2024-07-26T12:50:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.430+0000 7f7b47400640 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x5645e59dfda0 0x5645e5e78e60 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=26320e4aa7c8eb83 server_cookie=ffa1b23422f23907 in_seq=0 out_seq=0 2024-07-26T12:50:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b34c00640 1 -- 172.21.15.107:0/1447814819 <== mds.4 v2:172.21.15.132:6834/1242618859 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5645e5d655e0 con 0x5645e59dfda0 2024-07-26T12:50:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b34c00640 10 client.25859 handle_command_reply: tid=0 2024-07-26T12:50:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 client.25859 shutdown 2024-07-26T12:50:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1447814819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 msgr2=0x5645e5e025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:55.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5e025c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7b300a6fa0 tx=0x7f7b30005f10 comp rx=0 tx=0).stop 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1447814819 shutdown_connections 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.132:6834/1242618859,v1:172.21.15.132:6835/1242618859] conn(0x5645e59dfda0 0x5645e5e78e60 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5645e5a4c070 0x5645e5e74fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5645e5a2b390 0x5645e5e025c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b34c00640 0 client.25859 ms_handle_reset on v2:172.21.15.132:6834/1242618859 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 --2- 172.21.15.107:0/1447814819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5645e5a231b0 0x5645e5e01fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1447814819 >> 172.21.15.107:0/1447814819 conn(0x5645e5a40460 msgr2=0x5645e5e07ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1447814819 shutdown_connections 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 1 -- 172.21.15.107:0/1447814819 wait complete. 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.431+0000 7f7b54e9e740 20 client.25859 trim_cache size 0 max 16384 2024-07-26T12:50:55.435 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-07-26T12:50:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 -- 172.21.15.107:0/486521417 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b2c0b3dc0 msgr2=0x7f7b2c0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b2c0b3dc0 0x7f7b2c0b62d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7b38000c00 tx=0x7f7b38031400 comp rx=0 tx=0).stop 2024-07-26T12:50:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 -- 172.21.15.107:0/486521417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b48108d20 msgr2=0x7f7b48075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b48108d20 0x7f7b48075680 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000e9f0 tx=0x7f7b3000edf0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 -- 172.21.15.107:0/486521417 shutdown_connections 2024-07-26T12:50:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f7b2c0b3dc0 0x7f7b2c0b62d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b48109670 0x7f7b48075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b48108d20 0x7f7b48075680 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 --2- 172.21.15.107:0/486521417 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7b481033e0 0x7f7b48075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:55.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 -- 172.21.15.107:0/486521417 >> 172.21.15.107:0/486521417 conn(0x7f7b480fee70 msgr2=0x7f7b48101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:55.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 -- 172.21.15.107:0/486521417 shutdown_connections 2024-07-26T12:50:55.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:55.432+0000 7f7b4cc00640 1 -- 172.21.15.107:0/486521417 wait complete. 2024-07-26T12:50:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:54 smithi112 ceph-mon[37501]: pgmap v539: 129 pgs: 129 active+clean; 30 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 478 KiB/s wr, 113 op/s 2024-07-26T12:50:55.492 INFO:tasks.workunit.client.0.smithi107.stdout:checking acl/libacl.h usability... yes 2024-07-26T12:50:55.524 INFO:tasks.workunit.client.0.smithi107.stdout:checking acl/libacl.h presence... yes 2024-07-26T12:50:55.524 INFO:tasks.workunit.client.0.smithi107.stdout:checking for acl/libacl.h... yes 2024-07-26T12:50:55.603 INFO:tasks.workunit.client.0.smithi107.stdout:checking for acl_init in -lacl... yes 2024-07-26T12:50:55.722 INFO:tasks.workunit.client.0.smithi107.stdout:checking gdbm-ndbm.h usability... no 2024-07-26T12:50:55.782 INFO:tasks.workunit.client.0.smithi107.stdout:checking gdbm-ndbm.h presence... no 2024-07-26T12:50:55.782 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gdbm-ndbm.h... no 2024-07-26T12:50:55.881 INFO:tasks.workunit.client.0.smithi107.stdout:checking gdbm/ndbm.h usability... no 2024-07-26T12:50:55.939 INFO:tasks.workunit.client.0.smithi107.stdout:checking gdbm/ndbm.h presence... no 2024-07-26T12:50:55.939 INFO:tasks.workunit.client.0.smithi107.stdout:checking for gdbm/ndbm.h... no 2024-07-26T12:50:56.019 INFO:tasks.workunit.client.0.smithi107.stdout:checking ndbm.h usability... no 2024-07-26T12:50:56.080 INFO:tasks.workunit.client.0.smithi107.stdout:checking ndbm.h presence... no 2024-07-26T12:50:56.080 INFO:tasks.workunit.client.0.smithi107.stdout:checking for ndbm.h... no 2024-07-26T12:50:56.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T12:50:56.951 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:50:57.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:56 smithi107 ceph-mon[32659]: pgmap v540: 129 pgs: 129 active+clean; 36 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.0 MiB/s wr, 160 op/s 2024-07-26T12:50:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:56 smithi132 ceph-mon[37580]: pgmap v540: 129 pgs: 129 active+clean; 36 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.0 MiB/s wr, 160 op/s 2024-07-26T12:50:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:56 smithi112 ceph-mon[37501]: pgmap v540: 129 pgs: 129 active+clean; 36 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 1.0 MiB/s wr, 160 op/s 2024-07-26T12:50:58.836 INFO:tasks.workunit.client.0.smithi107.stdout:checking libaio.h usability... yes 2024-07-26T12:50:58.868 INFO:tasks.workunit.client.0.smithi107.stdout:checking libaio.h presence... yes 2024-07-26T12:50:58.868 INFO:tasks.workunit.client.0.smithi107.stdout:checking for libaio.h... yes 2024-07-26T12:50:58.973 INFO:tasks.workunit.client.0.smithi107.stdout:checking liburing.h usability... no 2024-07-26T12:50:59.030 INFO:tasks.workunit.client.0.smithi107.stdout:checking liburing.h presence... no 2024-07-26T12:50:59.031 INFO:tasks.workunit.client.0.smithi107.stdout:checking for liburing.h... no 2024-07-26T12:50:59.122 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/capability.h usability... no 2024-07-26T12:50:59.176 INFO:tasks.workunit.client.0.smithi107.stdout:checking sys/capability.h presence... no 2024-07-26T12:50:59.176 INFO:tasks.workunit.client.0.smithi107.stdout:checking for sys/capability.h... no 2024-07-26T12:50:59.252 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/fiemap.h usability... yes 2024-07-26T12:50:59.283 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/fiemap.h presence... yes 2024-07-26T12:50:59.284 INFO:tasks.workunit.client.0.smithi107.stdout:checking for linux/fiemap.h... yes 2024-07-26T12:50:59.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:50:58 smithi107 ceph-mon[32659]: pgmap v541: 129 pgs: 129 active+clean; 39 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 1.2 MiB/s wr, 175 op/s 2024-07-26T12:50:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:50:58 smithi132 ceph-mon[37580]: pgmap v541: 129 pgs: 129 active+clean; 39 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 1.2 MiB/s wr, 175 op/s 2024-07-26T12:50:59.369 INFO:tasks.workunit.client.0.smithi107.stdout:checking for fallocate... yes 2024-07-26T12:50:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:50:58 smithi112 ceph-mon[37501]: pgmap v541: 129 pgs: 129 active+clean; 39 MiB data, 581 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 1.2 MiB/s wr, 175 op/s 2024-07-26T12:50:59.470 INFO:tasks.workunit.client.0.smithi107.stdout:checking for open_by_handle_at... yes 2024-07-26T12:50:59.560 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/fs.h usability... yes 2024-07-26T12:50:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.559+0000 7fa381600640 1 -- 172.21.15.107:0/3859543904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa37c076db0 msgr2=0x7fa37c0751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.559+0000 7fa381600640 1 --2- 172.21.15.107:0/3859543904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa37c076db0 0x7fa37c0751e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa36c009900 tx=0x7fa36c02a3f0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.559+0000 7fa381600640 1 -- 172.21.15.107:0/3859543904 shutdown_connections 2024-07-26T12:50:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.559+0000 7fa381600640 1 --2- 172.21.15.107:0/3859543904 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa37c113cb0 0x7fa37c116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.559+0000 7fa381600640 1 --2- 172.21.15.107:0/3859543904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa37c075800 0x7fa37c075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.559+0000 7fa381600640 1 --2- 172.21.15.107:0/3859543904 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa37c076db0 0x7fa37c0751e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.559+0000 7fa381600640 1 -- 172.21.15.107:0/3859543904 >> 172.21.15.107:0/3859543904 conn(0x7fa37c0fee30 msgr2=0x7fa37c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:59.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.560+0000 7fa381600640 1 -- 172.21.15.107:0/3859543904 shutdown_connections 2024-07-26T12:50:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.560+0000 7fa381600640 1 -- 172.21.15.107:0/3859543904 wait complete. 2024-07-26T12:50:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa381600640 1 Processor -- start 2024-07-26T12:50:59.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa381600640 1 -- start start 2024-07-26T12:50:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa381600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa37c075800 0x7fa37c0748b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa381600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa37c076db0 0x7fa37c06db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa381600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa37c113cb0 0x7fa37c06e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa381600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1198f0 con 0x7fa37c076db0 2024-07-26T12:50:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa381600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c119770 con 0x7fa37c075800 2024-07-26T12:50:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa381600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c119a70 con 0x7fa37c113cb0 2024-07-26T12:50:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa37be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa37c075800 0x7fa37c0748b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa37b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa37c076db0 0x7fa37c06db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa37be00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa37c075800 0x7fa37c0748b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44702/0 (socket says 172.21.15.107:44702) 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.561+0000 7fa37be00640 1 -- 172.21.15.107:0/3809027306 learned_addr learned my addr 172.21.15.107:0/3809027306 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.562+0000 7fa380c00640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa37c113cb0 0x7fa37c06e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.562+0000 7fa37b400640 1 -- 172.21.15.107:0/3809027306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa37c113cb0 msgr2=0x7fa37c06e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.562+0000 7fa37b400640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa37c113cb0 0x7fa37c06e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.562+0000 7fa37b400640 1 -- 172.21.15.107:0/3809027306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa37c075800 msgr2=0x7fa37c0748b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.562+0000 7fa37b400640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa37c075800 0x7fa37c0748b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.562+0000 7fa37b400640 1 -- 172.21.15.107:0/3809027306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa37c06e940 con 0x7fa37c076db0 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.562+0000 7fa37b400640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa37c076db0 0x7fa37c06db80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa36800e9c0 tx=0x7fa36800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.563+0000 7fa378c00640 1 -- 172.21.15.107:0/3809027306 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36800ccc0 con 0x7fa37c076db0 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.563+0000 7fa381600640 1 -- 172.21.15.107:0/3809027306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa37c0735e0 con 0x7fa37c076db0 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.563+0000 7fa378c00640 1 -- 172.21.15.107:0/3809027306 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3680108e0 con 0x7fa37c076db0 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.563+0000 7fa378c00640 1 -- 172.21.15.107:0/3809027306 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368018760 con 0x7fa37c076db0 2024-07-26T12:50:59.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.563+0000 7fa381600640 1 -- 172.21.15.107:0/3809027306 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa37c073b70 con 0x7fa37c076db0 2024-07-26T12:50:59.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.566+0000 7fa3899c4740 1 Processor -- start 2024-07-26T12:50:59.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.566+0000 7fa3899c4740 1 -- start start 2024-07-26T12:50:59.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.566+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 0x55cfc350c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc350c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3146390 0x55cfc350ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.568 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa3899c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cfc3482bd0 con 0x55cfc3167070 2024-07-26T12:50:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa3899c4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cfc34720d0 con 0x55cfc3146390 2024-07-26T12:50:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa3899c4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cfc3476110 con 0x55cfc313e1b0 2024-07-26T12:50:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa37be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc350c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa380c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 0x55cfc350c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa378c00640 1 -- 172.21.15.107:0/3809027306 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa368010430 con 0x7fa37c076db0 2024-07-26T12:50:59.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa380c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 0x55cfc350c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59388/0 (socket says 172.21.15.107:59388) 2024-07-26T12:50:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa380c00640 1 -- 172.21.15.107:0/3350713688 learned_addr learned my addr 172.21.15.107:0/3350713688 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.567+0000 7fa37b400640 1 --2- 172.21.15.107:0/3350713688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3146390 0x55cfc350ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.568+0000 7fa37b400640 1 -- 172.21.15.107:0/3350713688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 msgr2=0x55cfc350c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.568+0000 7fa37b400640 1 --2- 172.21.15.107:0/3350713688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc350c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.568+0000 7fa37b400640 1 -- 172.21.15.107:0/3350713688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 msgr2=0x55cfc350c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.568+0000 7fa37b400640 1 --2- 172.21.15.107:0/3350713688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 0x55cfc350c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.568+0000 7fa37b400640 1 -- 172.21.15.107:0/3350713688 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfc31d2650 con 0x55cfc3146390 2024-07-26T12:50:59.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.568+0000 7fa37b400640 1 --2- 172.21.15.107:0/3350713688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3146390 0x55cfc350ca90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa3680e1920 tx=0x7fa3680ec990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f56b17e93e3533df server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:59.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.569+0000 7fa371200640 1 -- 172.21.15.107:0/3350713688 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3680f8070 con 0x55cfc3146390 2024-07-26T12:50:59.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.569+0000 7fa371200640 1 -- 172.21.15.107:0/3350713688 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3680e64e0 con 0x55cfc3146390 2024-07-26T12:50:59.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.569+0000 7fa371200640 1 -- 172.21.15.107:0/3350713688 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3680f0980 con 0x55cfc3146390 2024-07-26T12:50:59.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.569+0000 7fa378c00640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa3600b3fc0 0x7fa3600b64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.569+0000 7fa378c00640 1 -- 172.21.15.107:0/3809027306 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7fa368018500 con 0x7fa37c076db0 2024-07-26T12:50:59.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.570+0000 7fa3899c4740 1 -- 172.21.15.107:0/3350713688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3146390 msgr2=0x55cfc350ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:59.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.570+0000 7fa3899c4740 1 --2- 172.21.15.107:0/3350713688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3146390 0x55cfc350ca90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa3680e1920 tx=0x7fa3680ec990 comp rx=0 tx=0).stop 2024-07-26T12:50:59.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.571+0000 7fa380c00640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa3600b3fc0 0x7fa3600b64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.571+0000 7fa3899c4740 1 -- 172.21.15.107:0/3350713688 shutdown_connections 2024-07-26T12:50:59.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.571+0000 7fa3899c4740 1 --2- 172.21.15.107:0/3350713688 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3146390 0x55cfc350ca90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.571+0000 7fa3899c4740 1 --2- 172.21.15.107:0/3350713688 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc350c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.571+0000 7fa3899c4740 1 --2- 172.21.15.107:0/3350713688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 0x55cfc350c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.571+0000 7fa3899c4740 1 -- 172.21.15.107:0/3350713688 >> 172.21.15.107:0/3350713688 conn(0x55cfc315b460 msgr2=0x55cfc3509f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:59.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.571+0000 7fa3899c4740 1 -- 172.21.15.107:0/3350713688 shutdown_connections 2024-07-26T12:50:59.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.571+0000 7fa3899c4740 1 -- 172.21.15.107:0/3350713688 wait complete. 2024-07-26T12:50:59.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.572+0000 7fa3899c4740 1 -- start start 2024-07-26T12:50:59.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.572+0000 7fa3899c4740 1 Processor -- start 2024-07-26T12:50:59.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.572+0000 7fa380c00640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa3600b3fc0 0x7fa3600b64d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa374005f40 tx=0x7fa3740096a0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:59.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.573+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc313e1b0 0x55cfc3517660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.573+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc3146390 0x55cfc3517ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.573+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 0x55cfc3523b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa380c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 0x55cfc3523b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa37b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc3146390 0x55cfc3517ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa37b400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc3146390 0x55cfc3517ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:44386/0 (socket says 172.21.15.107:44386) 2024-07-26T12:50:59.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa37b400640 1 -- 172.21.15.107:0/629937274 learned_addr learned my addr 172.21.15.107:0/629937274 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:59.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cfc32d5700 con 0x55cfc3167070 2024-07-26T12:50:59.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55cfc340e6a0 con 0x55cfc313e1b0 2024-07-26T12:50:59.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55cfc351a7f0 con 0x55cfc3146390 2024-07-26T12:50:59.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa37b400640 1 -- 172.21.15.107:0/629937274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc313e1b0 msgr2=0x55cfc3517660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T12:50:59.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa37b400640 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc313e1b0 0x55cfc3517660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa37b400640 1 -- 172.21.15.107:0/629937274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 msgr2=0x55cfc3523b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:59.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.574+0000 7fa3899c4740 10 client.0 _ll_register_callbacks cb 0x55cfc341e150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:59.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.575+0000 7fa37b400640 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 0x55cfc3523b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.575+0000 7fa37b400640 1 -- 172.21.15.107:0/629937274 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfc3425840 con 0x55cfc3146390 2024-07-26T12:50:59.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.575+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3680e64e0 con 0x55cfc3146390 2024-07-26T12:50:59.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.575+0000 7fa380c00640 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 0x55cfc3523b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:50:59.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.575+0000 7fa37b400640 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc3146390 0x55cfc3517ba0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa3680aad70 tx=0x7fa368009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1762ca72d027b7e2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:59.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.575+0000 7fa365600640 10 client.25876 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:50:59.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.575+0000 7fa365600640 1 -- 172.21.15.107:0/629937274 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3680ae070 con 0x55cfc3146390 2024-07-26T12:50:59.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.575+0000 7fa365600640 1 -- 172.21.15.107:0/629937274 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7fa3680e62e0 con 0x55cfc3146390 2024-07-26T12:50:59.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.575+0000 7fa365600640 1 -- 172.21.15.107:0/629937274 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa3680a6ca0 con 0x55cfc3146390 2024-07-26T12:50:59.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.575+0000 7fa3899c4740 10 client.25876 fetch_fsmap learned FSMap version 35 2024-07-26T12:50:59.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.576+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55cfc3526290 con 0x55cfc3146390 2024-07-26T12:50:59.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.576+0000 7fa365600640 1 -- 172.21.15.107:0/629937274 <== mon.1 v2:172.21.15.132:3300/0 4 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7fa3680a7070 con 0x55cfc3146390 2024-07-26T12:50:59.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.576+0000 7fa3899c4740 10 client.25876 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:50:59.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.576+0000 7fa3899c4740 10 client.25876 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T12:50:59.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.576+0000 7fa3899c4740 20 client.25876 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:59.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.576+0000 7fa3899c4740 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x55cfc31259f0 0x55cfc35295f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.576+0000 7fa3899c4740 4 client.25876 mds_command: new command op to 24694 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:50:59.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.576+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 --> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cfc3484b20 con 0x55cfc31259f0 2024-07-26T12:50:59.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.576+0000 7fa37be00640 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x55cfc31259f0 0x55cfc35295f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.577+0000 7fa37be00640 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x55cfc31259f0 0x55cfc35295f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=227746877f926052 server_cookie=e5edef82b85ac135 in_seq=0 out_seq=0 2024-07-26T12:50:59.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.577+0000 7fa365600640 10 client.25876 ms_handle_connect on v2:172.21.15.132:6836/735918310 2024-07-26T12:50:59.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.578+0000 7fa365600640 1 -- 172.21.15.107:0/629937274 <== mds.3 v2:172.21.15.132:6836/735918310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cfc3484b20 con 0x55cfc31259f0 2024-07-26T12:50:59.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.578+0000 7fa365600640 10 client.25876 handle_command_reply: tid=0 2024-07-26T12:50:59.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.578+0000 7fa3899c4740 1 client.25876 shutdown 2024-07-26T12:50:59.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc3146390 msgr2=0x55cfc3517ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc3146390 0x55cfc3517ba0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa3680aad70 tx=0x7fa368009700 comp rx=0 tx=0).stop 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 shutdown_connections 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x55cfc31259f0 0x55cfc35295f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3167070 0x55cfc3523b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc3146390 0x55cfc3517ba0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 1 --2- 172.21.15.107:0/629937274 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc313e1b0 0x55cfc3517660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 >> 172.21.15.107:0/629937274 conn(0x55cfc315b460 msgr2=0x55cfc350f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa365600640 0 client.25876 ms_handle_reset on v2:172.21.15.132:6836/735918310 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 shutdown_connections 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 1 -- 172.21.15.107:0/629937274 wait complete. 2024-07-26T12:50:59.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.579+0000 7fa3899c4740 20 client.25876 trim_cache size 0 max 16384 2024-07-26T12:50:59.589 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/fs.h presence... yes 2024-07-26T12:50:59.589 INFO:tasks.workunit.client.0.smithi107.stdout:checking for linux/fs.h... yes 2024-07-26T12:50:59.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.594+0000 7fa3899c4740 1 Processor -- start 2024-07-26T12:50:59.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.594+0000 7fa3899c4740 1 -- start start 2024-07-26T12:50:59.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.594+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc3509010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.594+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 0x55cfc3519540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.594+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3167070 0x55cfc350d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.594+0000 7fa3899c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1bf550 con 0x55cfc3146390 2024-07-26T12:50:59.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.594+0000 7fa3899c4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1c1150 con 0x55cfc3167070 2024-07-26T12:50:59.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.594+0000 7fa3899c4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1bfc20 con 0x55cfc313e1b0 2024-07-26T12:50:59.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa37be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc3509010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa37b400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 0x55cfc3519540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa37be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc3509010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:44402/0 (socket says 172.21.15.107:44402) 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa37be00640 1 -- 172.21.15.107:0/2802372567 learned_addr learned my addr 172.21.15.107:0/2802372567 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa380c00640 1 --2- 172.21.15.107:0/2802372567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3167070 0x55cfc350d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa37be00640 1 -- 172.21.15.107:0/2802372567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3167070 msgr2=0x55cfc350d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa37be00640 1 --2- 172.21.15.107:0/2802372567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3167070 0x55cfc350d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa37be00640 1 -- 172.21.15.107:0/2802372567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 msgr2=0x55cfc3519540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa37be00640 1 --2- 172.21.15.107:0/2802372567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 0x55cfc3519540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa37be00640 1 -- 172.21.15.107:0/2802372567 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfc3125c20 con 0x55cfc313e1b0 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.595+0000 7fa37be00640 1 --2- 172.21.15.107:0/2802372567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc3509010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa36c000c00 tx=0x7fa36c0058b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e53176468fee9f6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.596+0000 7fa367e00640 1 -- 172.21.15.107:0/2802372567 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c049070 con 0x55cfc313e1b0 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.596+0000 7fa367e00640 1 -- 172.21.15.107:0/2802372567 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa36c005030 con 0x55cfc313e1b0 2024-07-26T12:50:59.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.596+0000 7fa367e00640 1 -- 172.21.15.107:0/2802372567 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c035040 con 0x55cfc313e1b0 2024-07-26T12:50:59.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.597+0000 7fa3899c4740 1 -- 172.21.15.107:0/2802372567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 msgr2=0x55cfc3509010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:59.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.597+0000 7fa3899c4740 1 --2- 172.21.15.107:0/2802372567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc3509010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa36c000c00 tx=0x7fa36c0058b0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.597+0000 7fa3899c4740 1 -- 172.21.15.107:0/2802372567 shutdown_connections 2024-07-26T12:50:59.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.597+0000 7fa3899c4740 1 --2- 172.21.15.107:0/2802372567 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3167070 0x55cfc350d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.597+0000 7fa3899c4740 1 --2- 172.21.15.107:0/2802372567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 0x55cfc3519540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.597+0000 7fa3899c4740 1 --2- 172.21.15.107:0/2802372567 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc3509010 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.597+0000 7fa3899c4740 1 -- 172.21.15.107:0/2802372567 >> 172.21.15.107:0/2802372567 conn(0x55cfc315b460 msgr2=0x55cfc350d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:59.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.597+0000 7fa3899c4740 1 -- 172.21.15.107:0/2802372567 shutdown_connections 2024-07-26T12:50:59.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.597+0000 7fa3899c4740 1 -- 172.21.15.107:0/2802372567 wait complete. 2024-07-26T12:50:59.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.597+0000 7fa3899c4740 1 -- start start 2024-07-26T12:50:59.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.598+0000 7fa3899c4740 1 Processor -- start 2024-07-26T12:50:59.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.598+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc352b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.599+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 0x55cfc352ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.599+0000 7fa3899c4740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3167070 0x55cfc352bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.599+0000 7fa3899c4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1c0a80 con 0x55cfc3146390 2024-07-26T12:50:59.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.599+0000 7fa37be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc352b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.599+0000 7fa3899c4740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1c02f0 con 0x55cfc3167070 2024-07-26T12:50:59.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.599+0000 7fa3899c4740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa37c1bd530 con 0x55cfc313e1b0 2024-07-26T12:50:59.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.599+0000 7fa37be00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc352b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:44416/0 (socket says 172.21.15.107:44416) 2024-07-26T12:50:59.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.599+0000 7fa37be00640 1 -- 172.21.15.107:0/1740385348 learned_addr learned my addr 172.21.15.107:0/1740385348 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:50:59.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.599+0000 7fa3899c4740 10 client.0 _ll_register_callbacks cb 0x7fa368005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:50:59.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.599+0000 7fa380c00640 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3167070 0x55cfc352bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.600+0000 7fa37b400640 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 0x55cfc352ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.600+0000 7fa37be00640 1 -- 172.21.15.107:0/1740385348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3167070 msgr2=0x55cfc352bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:59.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.600+0000 7fa37be00640 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3167070 0x55cfc352bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.600+0000 7fa37be00640 1 -- 172.21.15.107:0/1740385348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 msgr2=0x55cfc352ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:50:59.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.600+0000 7fa37be00640 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 0x55cfc352ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.600+0000 7fa37be00640 1 -- 172.21.15.107:0/1740385348 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfc358fb40 con 0x55cfc313e1b0 2024-07-26T12:50:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.600+0000 7fa37b400640 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 0x55cfc352ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:50:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.601+0000 7fa37be00640 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc352b510 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa36c032de0 tx=0x7fa36c032e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=764c142d20529d98 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:50:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.601+0000 7fa365600640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:50:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.601+0000 7fa365600640 1 -- 172.21.15.107:0/1740385348 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36c035070 con 0x55cfc313e1b0 2024-07-26T12:50:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.601+0000 7fa3899c4740 1 -- 172.21.15.107:0/1740385348 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa36c005030 con 0x55cfc313e1b0 2024-07-26T12:50:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.601+0000 7fa365600640 1 -- 172.21.15.107:0/1740385348 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7fa36c0052e0 con 0x55cfc313e1b0 2024-07-26T12:50:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.601+0000 7fa365600640 1 -- 172.21.15.107:0/1740385348 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa36c049040 con 0x55cfc313e1b0 2024-07-26T12:50:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.601+0000 7fa3899c4740 10 client.25882 fetch_fsmap learned FSMap version 35 2024-07-26T12:50:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.601+0000 7fa3899c4740 1 -- 172.21.15.107:0/1740385348 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55cfc3590dc0 con 0x55cfc313e1b0 2024-07-26T12:50:59.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.602+0000 7fa365600640 1 -- 172.21.15.107:0/1740385348 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa36c005030 con 0x55cfc313e1b0 2024-07-26T12:50:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.602+0000 7fa365600640 1 -- 172.21.15.107:0/1740385348 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7fa36c055030 con 0x55cfc313e1b0 2024-07-26T12:50:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.602+0000 7fa3899c4740 10 client.25882 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:50:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.602+0000 7fa3899c4740 10 client.25882 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T12:50:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.602+0000 7fa3899c4740 20 client.25882 populate_metadata read hostname 'smithi107' 2024-07-26T12:50:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.602+0000 7fa3899c4740 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x55cfc30fada0 0x55cfc3593530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:50:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.602+0000 7fa3899c4740 4 client.25882 mds_command: new command op to 24694 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T12:50:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.602+0000 7fa3899c4740 1 -- 172.21.15.107:0/1740385348 --> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cfc34805c0 con 0x55cfc30fada0 2024-07-26T12:50:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.603+0000 7fa37b400640 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x55cfc30fada0 0x55cfc3593530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:50:59.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.603+0000 7fa37b400640 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x55cfc30fada0 0x55cfc3593530 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e354a6a907c4632c server_cookie=e418add6901abf9c in_seq=0 out_seq=0 2024-07-26T12:50:59.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.603+0000 7fa365600640 10 client.25882 ms_handle_connect on v2:172.21.15.132:6836/735918310 2024-07-26T12:50:59.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.603+0000 7fa365600640 1 -- 172.21.15.107:0/1740385348 <== mds.3 v2:172.21.15.132:6836/735918310 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cfc34805c0 con 0x55cfc30fada0 2024-07-26T12:50:59.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.603+0000 7fa365600640 10 client.25882 handle_command_reply: tid=0 2024-07-26T12:50:59.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.603+0000 7fa3899c4740 1 client.25882 shutdown 2024-07-26T12:50:59.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.604+0000 7fa3899c4740 1 -- 172.21.15.107:0/1740385348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 msgr2=0x55cfc352b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.604+0000 7fa3899c4740 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc352b510 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa36c032de0 tx=0x7fa36c032e10 comp rx=0 tx=0).stop 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.604+0000 7fa3899c4740 1 -- 172.21.15.107:0/1740385348 shutdown_connections 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.604+0000 7fa3899c4740 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.132:6836/735918310,v1:172.21.15.132:6837/735918310] conn(0x55cfc30fada0 0x55cfc3593530 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.604+0000 7fa3899c4740 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55cfc3167070 0x55cfc352bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.604+0000 7fa3899c4740 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cfc3146390 0x55cfc352ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.604+0000 7fa3899c4740 1 --2- 172.21.15.107:0/1740385348 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55cfc313e1b0 0x55cfc352b510 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.604+0000 7fa3899c4740 1 -- 172.21.15.107:0/1740385348 >> 172.21.15.107:0/1740385348 conn(0x55cfc315b460 msgr2=0x55cfc35192e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.604+0000 7fa365600640 0 client.25882 ms_handle_reset on v2:172.21.15.132:6836/735918310 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.606+0000 7fa3899c4740 1 -- 172.21.15.107:0/1740385348 shutdown_connections 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.606+0000 7fa3899c4740 1 -- 172.21.15.107:0/1740385348 wait complete. 2024-07-26T12:50:59.609 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.606+0000 7fa3899c4740 20 client.25882 trim_cache size 0 max 16384 2024-07-26T12:50:59.610 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-07-26T12:50:59.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 -- 172.21.15.107:0/3809027306 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa3600b3fc0 msgr2=0x7fa3600b64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:59.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa3600b3fc0 0x7fa3600b64d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa374005f40 tx=0x7fa3740096a0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 -- 172.21.15.107:0/3809027306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa37c076db0 msgr2=0x7fa37c06db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:50:59.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa37c076db0 0x7fa37c06db80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa36800e9c0 tx=0x7fa36800ee80 comp rx=0 tx=0).stop 2024-07-26T12:50:59.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 -- 172.21.15.107:0/3809027306 shutdown_connections 2024-07-26T12:50:59.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa3600b3fc0 0x7fa3600b64d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa37c113cb0 0x7fa37c06e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa37c076db0 0x7fa37c06db80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 --2- 172.21.15.107:0/3809027306 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa37c075800 0x7fa37c0748b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:50:59.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 -- 172.21.15.107:0/3809027306 >> 172.21.15.107:0/3809027306 conn(0x7fa37c0fee30 msgr2=0x7fa37c10fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:50:59.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 -- 172.21.15.107:0/3809027306 shutdown_connections 2024-07-26T12:50:59.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:50:59.607+0000 7fa381600640 1 -- 172.21.15.107:0/3809027306 wait complete. 2024-07-26T12:50:59.673 INFO:tasks.workunit.client.0.smithi107.stdout:checking btrfsutil.h usability... no 2024-07-26T12:50:59.719 INFO:tasks.workunit.client.0.smithi107.stdout:checking btrfsutil.h presence... no 2024-07-26T12:50:59.719 INFO:tasks.workunit.client.0.smithi107.stdout:checking for btrfsutil.h... no 2024-07-26T12:50:59.871 INFO:tasks.workunit.client.0.smithi107.stdout:checking for copy_file_range... yes 2024-07-26T12:50:59.961 INFO:tasks.workunit.client.0.smithi107.stdout:checking for SEEK_DATA... yes 2024-07-26T12:51:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.049+0000 7f349b200640 1 -- 172.21.15.107:0/777365479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3494108e50 msgr2=0x7f3494109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.049+0000 7f349b200640 1 --2- 172.21.15.107:0/777365479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3494108e50 0x7f3494109260 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3480009900 tx=0x7f348002a410 comp rx=0 tx=0).stop 2024-07-26T12:51:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.049+0000 7f349b200640 1 -- 172.21.15.107:0/777365479 shutdown_connections 2024-07-26T12:51:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.049+0000 7f349b200640 1 --2- 172.21.15.107:0/777365479 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34941097a0 0x7f349410ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.049+0000 7f349b200640 1 --2- 172.21.15.107:0/777365479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3494108e50 0x7f3494109260 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.049+0000 7f349b200640 1 --2- 172.21.15.107:0/777365479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3494103520 0x7f3494103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.049+0000 7f349b200640 1 -- 172.21.15.107:0/777365479 >> 172.21.15.107:0/777365479 conn(0x7f34940fefa0 msgr2=0x7f34941013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:00.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.049+0000 7f349b200640 1 -- 172.21.15.107:0/777365479 shutdown_connections 2024-07-26T12:51:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.049+0000 7f349b200640 1 -- 172.21.15.107:0/777365479 wait complete. 2024-07-26T12:51:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.050+0000 7f349b200640 1 Processor -- start 2024-07-26T12:51:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.050+0000 7f349b200640 1 -- start start 2024-07-26T12:51:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.051+0000 7f349b200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3494103520 0x7f349419c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.051+0000 7f349b200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3494108e50 0x7f349419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.051+0000 7f349b200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34941097a0 0x7f34941a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.051+0000 7f349b200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3494075cd0 con 0x7f3494103520 2024-07-26T12:51:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.051+0000 7f349b200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f3494075b50 con 0x7f34941097a0 2024-07-26T12:51:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.051+0000 7f349b200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f3494075e50 con 0x7f3494108e50 2024-07-26T12:51:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.051+0000 7f3493e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3494103520 0x7f349419c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.051+0000 7f3493e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3494103520 0x7f349419c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59442/0 (socket says 172.21.15.107:59442) 2024-07-26T12:51:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.051+0000 7f3493e00640 1 -- 172.21.15.107:0/372911425 learned_addr learned my addr 172.21.15.107:0/372911425 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.051+0000 7f3499200640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34941097a0 0x7f34941a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f3493e00640 1 -- 172.21.15.107:0/372911425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3494108e50 msgr2=0x7f349419c670 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T12:51:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f3493e00640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3494108e50 0x7f349419c670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f3493e00640 1 -- 172.21.15.107:0/372911425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34941097a0 msgr2=0x7f34941a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f3493e00640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34941097a0 0x7f34941a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f3493e00640 1 -- 172.21.15.107:0/372911425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34941a2000 con 0x7f3494103520 2024-07-26T12:51:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f3493e00640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3494103520 0x7f349419c0d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f348400e9f0 tx=0x7f348400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f3490c00640 1 -- 172.21.15.107:0/372911425 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348400cd80 con 0x7f3494103520 2024-07-26T12:51:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f3490c00640 1 -- 172.21.15.107:0/372911425 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34840108e0 con 0x7f3494103520 2024-07-26T12:51:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f3490c00640 1 -- 172.21.15.107:0/372911425 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484018980 con 0x7f3494103520 2024-07-26T12:51:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f349b200640 1 -- 172.21.15.107:0/372911425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34941a22f0 con 0x7f3494103520 2024-07-26T12:51:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.052+0000 7f349b200640 1 -- 172.21.15.107:0/372911425 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34941a8a80 con 0x7f3494103520 2024-07-26T12:51:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.056+0000 7f3490c00640 1 -- 172.21.15.107:0/372911425 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f3484010430 con 0x7f3494103520 2024-07-26T12:51:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.056+0000 7f34a2238740 1 Processor -- start 2024-07-26T12:51:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f34a2238740 1 -- start start 2024-07-26T12:51:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f34a2238740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e641b0 0x5563a4207a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f34a2238740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 0x5563a4207f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f34a2238740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e434d0 0x5563a42084c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f34a2238740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5563a417c270 con 0x5563a3e641b0 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f34a2238740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5563a416b770 con 0x5563a3e3b2f0 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f34a2238740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5563a416f7b0 con 0x5563a3e434d0 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f3499200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e641b0 0x5563a4207a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f3493400640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e434d0 0x5563a42084c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f3499200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e641b0 0x5563a4207a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59456/0 (socket says 172.21.15.107:59456) 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f3499200640 1 -- 172.21.15.107:0/2359017931 learned_addr learned my addr 172.21.15.107:0/2359017931 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.057+0000 7f3493e00640 1 --2- 172.21.15.107:0/2359017931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 0x5563a4207f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.058+0000 7f3493400640 1 -- 172.21.15.107:0/2359017931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 msgr2=0x5563a4207f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.058+0000 7f3493400640 1 --2- 172.21.15.107:0/2359017931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 0x5563a4207f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.058+0000 7f3493400640 1 -- 172.21.15.107:0/2359017931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e641b0 msgr2=0x5563a4207a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.058+0000 7f3493400640 1 --2- 172.21.15.107:0/2359017931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e641b0 0x5563a4207a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.058+0000 7f3490c00640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34780b3d40 0x7f34780b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.058+0000 7f3493400640 1 -- 172.21.15.107:0/2359017931 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563a3ecf790 con 0x5563a3e434d0 2024-07-26T12:51:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.058+0000 7f3490c00640 1 -- 172.21.15.107:0/372911425 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f34840e47a0 con 0x7f3494103520 2024-07-26T12:51:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.058+0000 7f3499200640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34780b3d40 0x7f34780b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.061 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.058+0000 7f3493400640 1 --2- 172.21.15.107:0/2359017931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e434d0 0x5563a42084c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f349406c050 tx=0x7f3480005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834f2059f26f789a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.059+0000 7f3489600640 1 -- 172.21.15.107:0/2359017931 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348003e030 con 0x5563a3e434d0 2024-07-26T12:51:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.059+0000 7f3489600640 1 -- 172.21.15.107:0/2359017931 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3480032ce0 con 0x5563a3e434d0 2024-07-26T12:51:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.059+0000 7f3489600640 1 -- 172.21.15.107:0/2359017931 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480035040 con 0x5563a3e434d0 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.060+0000 7f34a2238740 1 -- 172.21.15.107:0/2359017931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e434d0 msgr2=0x5563a42084c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.060+0000 7f3499200640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34780b3d40 0x7f34780b6250 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f348c011690 tx=0x7f348c009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.060+0000 7f34a2238740 1 --2- 172.21.15.107:0/2359017931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e434d0 0x5563a42084c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f349406c050 tx=0x7f3480005e50 comp rx=0 tx=0).stop 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.060+0000 7f34a2238740 1 -- 172.21.15.107:0/2359017931 shutdown_connections 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.060+0000 7f34a2238740 1 --2- 172.21.15.107:0/2359017931 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e434d0 0x5563a42084c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.060+0000 7f34a2238740 1 --2- 172.21.15.107:0/2359017931 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 0x5563a4207f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.060+0000 7f34a2238740 1 --2- 172.21.15.107:0/2359017931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e641b0 0x5563a4207a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.060+0000 7f34a2238740 1 -- 172.21.15.107:0/2359017931 >> 172.21.15.107:0/2359017931 conn(0x5563a3e58300 msgr2=0x5563a4205930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.060+0000 7f34a2238740 1 -- 172.21.15.107:0/2359017931 shutdown_connections 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.060+0000 7f34a2238740 1 -- 172.21.15.107:0/2359017931 wait complete. 2024-07-26T12:51:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.061+0000 7f34a2238740 1 -- start start 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.061+0000 7f34a2238740 1 Processor -- start 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.061+0000 7f34a2238740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a42136a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.061+0000 7f34a2238740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4213c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.061+0000 7f34a2238740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 0x5563a421f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.061+0000 7f34a2238740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5563a40a5ab0 con 0x5563a3e434d0 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.061+0000 7f34a2238740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x5563a4108d50 con 0x5563a3e641b0 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.061+0000 7f34a2238740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x5563a4216160 con 0x5563a3e3b2f0 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.061+0000 7f34a2238740 10 client.0 _ll_register_callbacks cb 0x5563a3e7ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4213c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a42136a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4213c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:59472/0 (socket says 172.21.15.107:59472) 2024-07-26T12:51:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493400640 1 -- 172.21.15.107:0/1133610252 learned_addr learned my addr 172.21.15.107:0/1133610252 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3499200640 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 0x5563a421f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493e00640 1 -- 172.21.15.107:0/1133610252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 msgr2=0x5563a421f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493e00640 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 0x5563a421f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493e00640 1 -- 172.21.15.107:0/1133610252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 msgr2=0x5563a4213c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493e00640 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4213c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493e00640 1 -- 172.21.15.107:0/1133610252 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563a3f48620 con 0x5563a3e3b2f0 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493400640 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4213c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.062+0000 7f3493e00640 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a42136a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f34840e1b10 tx=0x7f34840aae60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=646043dd468f6416 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.063+0000 7f347e000640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.063+0000 7f347e000640 1 -- 172.21.15.107:0/1133610252 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34840e1fa0 con 0x5563a3e3b2f0 2024-07-26T12:51:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.063+0000 7f347e000640 1 -- 172.21.15.107:0/1133610252 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f34840e4360 con 0x5563a3e3b2f0 2024-07-26T12:51:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.063+0000 7f34a2238740 1 -- 172.21.15.107:0/1133610252 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3480032ce0 con 0x5563a3e3b2f0 2024-07-26T12:51:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.063+0000 7f347e000640 1 -- 172.21.15.107:0/1133610252 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f34840aaae0 con 0x5563a3e3b2f0 2024-07-26T12:51:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.063+0000 7f34a2238740 10 client.25891 fetch_fsmap learned FSMap version 35 2024-07-26T12:51:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.063+0000 7f34a2238740 1 -- 172.21.15.107:0/1133610252 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5563a4222270 con 0x5563a3e3b2f0 2024-07-26T12:51:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.063+0000 7f347e000640 1 -- 172.21.15.107:0/1133610252 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3480032ce0 con 0x5563a3e3b2f0 2024-07-26T12:51:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.064+0000 7f347e000640 1 -- 172.21.15.107:0/1133610252 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7f34840e4b90 con 0x5563a3e3b2f0 2024-07-26T12:51:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.064+0000 7f34a2238740 10 client.25891 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:51:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.064+0000 7f34a2238740 10 client.25891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:51:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.064+0000 7f34a2238740 20 client.25891 populate_metadata read hostname 'smithi107' 2024-07-26T12:51:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.064+0000 7f34a2238740 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563a3e22d80 0x5563a4225810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.064+0000 7f34a2238740 4 client.25891 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:51:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.064+0000 7f34a2238740 1 -- 172.21.15.107:0/1133610252 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563a417e1c0 con 0x5563a3e22d80 2024-07-26T12:51:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.064+0000 7f3493400640 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563a3e22d80 0x5563a4225810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.065+0000 7f3493400640 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563a3e22d80 0x5563a4225810 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1afa80d47c565840 server_cookie=76e049f9a2b1d0cd in_seq=0 out_seq=0 2024-07-26T12:51:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.065+0000 7f347e000640 10 client.25891 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.065+0000 7f347e000640 1 -- 172.21.15.107:0/1133610252 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5563a417e1c0 con 0x5563a3e22d80 2024-07-26T12:51:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.065+0000 7f347e000640 10 client.25891 handle_command_reply: tid=0 2024-07-26T12:51:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.065+0000 7f34a2238740 1 client.25891 shutdown 2024-07-26T12:51:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 1 -- 172.21.15.107:0/1133610252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 msgr2=0x5563a42136a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a42136a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f34840e1b10 tx=0x7f34840aae60 comp rx=0 tx=0).stop 2024-07-26T12:51:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 1 -- 172.21.15.107:0/1133610252 shutdown_connections 2024-07-26T12:51:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563a3e22d80 0x5563a4225810 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 0x5563a421f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4213c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f347e000640 0 client.25891 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 1 --2- 172.21.15.107:0/1133610252 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a42136a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 1 -- 172.21.15.107:0/1133610252 >> 172.21.15.107:0/1133610252 conn(0x5563a3e58300 msgr2=0x5563a42040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 1 -- 172.21.15.107:0/1133610252 shutdown_connections 2024-07-26T12:51:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 1 -- 172.21.15.107:0/1133610252 wait complete. 2024-07-26T12:51:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.066+0000 7f34a2238740 20 client.25891 trim_cache size 0 max 16384 2024-07-26T12:51:00.075 INFO:tasks.workunit.client.0.smithi107.stdout:checking for BMV_OF_SHARED... yes 2024-07-26T12:51:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f34a2238740 1 Processor -- start 2024-07-26T12:51:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f34a2238740 1 -- start start 2024-07-26T12:51:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f34a2238740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 0x5563a420d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f34a2238740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a421f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f34a2238740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e641b0 0x5563a4205250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f34a2238740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b2a40 con 0x5563a3e434d0 2024-07-26T12:51:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f34a2238740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b4640 con 0x5563a3e3b2f0 2024-07-26T12:51:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f34a2238740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b3110 con 0x5563a3e641b0 2024-07-26T12:51:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f3499200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e641b0 0x5563a4205250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f3499200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e641b0 0x5563a4205250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:44456/0 (socket says 172.21.15.107:44456) 2024-07-26T12:51:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f3499200640 1 -- 172.21.15.107:0/2283955281 learned_addr learned my addr 172.21.15.107:0/2283955281 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f3493e00640 1 --2- 172.21.15.107:0/2283955281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 0x5563a420d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.083+0000 7f3493400640 1 --2- 172.21.15.107:0/2283955281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a421f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.084+0000 7f3493e00640 1 -- 172.21.15.107:0/2283955281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e641b0 msgr2=0x5563a4205250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.084+0000 7f3493e00640 1 --2- 172.21.15.107:0/2283955281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e641b0 0x5563a4205250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.084+0000 7f3493e00640 1 -- 172.21.15.107:0/2283955281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 msgr2=0x5563a421f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.084+0000 7f3493e00640 1 --2- 172.21.15.107:0/2283955281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a421f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.084+0000 7f3493e00640 1 -- 172.21.15.107:0/2283955281 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563a4209d40 con 0x5563a3e3b2f0 2024-07-26T12:51:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.084+0000 7f3499200640 1 --2- 172.21.15.107:0/2283955281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e641b0 0x5563a4205250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.084+0000 7f3493e00640 1 --2- 172.21.15.107:0/2283955281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 0x5563a420d210 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f34840aaf60 tx=0x7f34840aaed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8438c884c70816f8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.084+0000 7f3488c00640 1 -- 172.21.15.107:0/2283955281 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34840aa740 con 0x5563a3e3b2f0 2024-07-26T12:51:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.084+0000 7f3488c00640 1 -- 172.21.15.107:0/2283955281 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34840ea040 con 0x5563a3e3b2f0 2024-07-26T12:51:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.084+0000 7f3488c00640 1 -- 172.21.15.107:0/2283955281 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34840a6a80 con 0x5563a3e3b2f0 2024-07-26T12:51:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.085+0000 7f34a2238740 1 -- 172.21.15.107:0/2283955281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 msgr2=0x5563a420d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.085+0000 7f34a2238740 1 --2- 172.21.15.107:0/2283955281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 0x5563a420d210 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f34840aaf60 tx=0x7f34840aaed0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.085+0000 7f34a2238740 1 -- 172.21.15.107:0/2283955281 shutdown_connections 2024-07-26T12:51:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.085+0000 7f34a2238740 1 --2- 172.21.15.107:0/2283955281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e641b0 0x5563a4205250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.085+0000 7f34a2238740 1 --2- 172.21.15.107:0/2283955281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a421f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.085+0000 7f34a2238740 1 --2- 172.21.15.107:0/2283955281 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e3b2f0 0x5563a420d210 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.085+0000 7f34a2238740 1 -- 172.21.15.107:0/2283955281 >> 172.21.15.107:0/2283955281 conn(0x5563a3e58300 msgr2=0x5563a420cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.085+0000 7f34a2238740 1 -- 172.21.15.107:0/2283955281 shutdown_connections 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.085+0000 7f34a2238740 1 -- 172.21.15.107:0/2283955281 wait complete. 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.086+0000 7f34a2238740 1 -- start start 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.086+0000 7f34a2238740 1 Processor -- start 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.086+0000 7f34a2238740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a421ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.086+0000 7f34a2238740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4216370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.086+0000 7f34a2238740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 0x5563a42168b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.086+0000 7f34a2238740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b3f70 con 0x5563a3e434d0 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.086+0000 7f34a2238740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b37e0 con 0x5563a3e641b0 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.086+0000 7f34a2238740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f34941b0a20 con 0x5563a3e3b2f0 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f34a2238740 10 client.0 _ll_register_callbacks cb 0x7f34840ea220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3493e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a421ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3499200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 0x5563a42168b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3493400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4216370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3499200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 0x5563a42168b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44808/0 (socket says 172.21.15.107:44808) 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3493e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a421ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:44460/0 (socket says 172.21.15.107:44460) 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3499200640 1 -- 172.21.15.107:0/1481367296 learned_addr learned my addr 172.21.15.107:0/1481367296 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3493400640 1 -- 172.21.15.107:0/1481367296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 msgr2=0x5563a421ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3493400640 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a421ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3493400640 1 -- 172.21.15.107:0/1481367296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 msgr2=0x5563a42168b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3493400640 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 0x5563a42168b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3493400640 1 -- 172.21.15.107:0/1481367296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563a4216ff0 con 0x5563a3e434d0 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3499200640 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 0x5563a42168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3493e00640 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a421ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f3493400640 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4216370 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x5563a4225f70 tx=0x7f348002aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b962cd79298824 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.087+0000 7f347e000640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:51:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.088+0000 7f347e000640 1 -- 172.21.15.107:0/1481367296 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480035040 con 0x5563a3e434d0 2024-07-26T12:51:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.088+0000 7f347e000640 1 -- 172.21.15.107:0/1481367296 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f34800318c0 con 0x5563a3e434d0 2024-07-26T12:51:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.088+0000 7f34a2238740 1 -- 172.21.15.107:0/1481367296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34840ea040 con 0x5563a3e434d0 2024-07-26T12:51:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.088+0000 7f347e000640 1 -- 172.21.15.107:0/1481367296 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f348005c050 con 0x5563a3e434d0 2024-07-26T12:51:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.088+0000 7f34a2238740 10 client.15759 fetch_fsmap learned FSMap version 35 2024-07-26T12:51:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.088+0000 7f34a2238740 1 -- 172.21.15.107:0/1481367296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5563a428ce90 con 0x5563a3e434d0 2024-07-26T12:51:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.088+0000 7f347e000640 1 -- 172.21.15.107:0/1481367296 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f34840ea040 con 0x5563a3e434d0 2024-07-26T12:51:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.089+0000 7f347e000640 1 -- 172.21.15.107:0/1481367296 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7f3480004aa0 con 0x5563a3e434d0 2024-07-26T12:51:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.089+0000 7f34a2238740 10 client.15759 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:51:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.089+0000 7f34a2238740 10 client.15759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:51:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.089+0000 7f34a2238740 20 client.15759 populate_metadata read hostname 'smithi107' 2024-07-26T12:51:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.089+0000 7f34a2238740 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563a3df81c0 0x5563a428f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.089+0000 7f34a2238740 4 client.15759 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T12:51:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.089+0000 7f34a2238740 1 -- 172.21.15.107:0/1481367296 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563a4179c60 con 0x5563a3df81c0 2024-07-26T12:51:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.089+0000 7f3493e00640 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563a3df81c0 0x5563a428f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:00.095 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.089+0000 7f3493e00640 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563a3df81c0 0x5563a428f2a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1743fac602fa28c0 server_cookie=b2cad344f8c02588 in_seq=0 out_seq=0 2024-07-26T12:51:00.095 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.089+0000 7f347e000640 10 client.15759 ms_handle_connect on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:00.177 INFO:tasks.workunit.client.0.smithi107.stdout:checking for nftw... yes 2024-07-26T12:51:00.341 INFO:tasks.workunit.client.0.smithi107.stdout:checking for RLIMIT_NOFILE... yes 2024-07-26T12:51:00.450 INFO:tasks.workunit.client.0.smithi107.stdout:checking for XFS_IOC_EXCHANGE_RANGE... no 2024-07-26T12:51:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f347e000640 1 -- 172.21.15.107:0/1481367296 <== mds.0 v2:172.21.15.112:6838/3779803434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563a4179c60 con 0x5563a3df81c0 2024-07-26T12:51:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f347e000640 10 client.15759 handle_command_reply: tid=0 2024-07-26T12:51:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f34a2238740 1 client.15759 shutdown 2024-07-26T12:51:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f34a2238740 1 -- 172.21.15.107:0/1481367296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 msgr2=0x5563a4216370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:00.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f34a2238740 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4216370 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x5563a4225f70 tx=0x7f348002aa90 comp rx=0 tx=0).stop 2024-07-26T12:51:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f34a2238740 1 -- 172.21.15.107:0/1481367296 shutdown_connections 2024-07-26T12:51:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f34a2238740 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5563a3df81c0 0x5563a428f2a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f34a2238740 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5563a3e641b0 0x5563a42168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f34a2238740 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5563a3e434d0 0x5563a4216370 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f34a2238740 1 --2- 172.21.15.107:0/1481367296 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5563a3e3b2f0 0x5563a421ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f34a2238740 1 -- 172.21.15.107:0/1481367296 >> 172.21.15.107:0/1481367296 conn(0x5563a3e58300 msgr2=0x5563a42051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.514+0000 7f347e000640 0 client.15759 ms_handle_reset on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:00.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.515+0000 7f34a2238740 1 -- 172.21.15.107:0/1481367296 shutdown_connections 2024-07-26T12:51:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.515+0000 7f34a2238740 1 -- 172.21.15.107:0/1481367296 wait complete. 2024-07-26T12:51:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.515+0000 7f34a2238740 20 client.15759 trim_cache size 0 max 16384 2024-07-26T12:51:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:{ 2024-07-26T12:51:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "message": "", 2024-07-26T12:51:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout: "return_code": 0 2024-07-26T12:51:00.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stdout:} 2024-07-26T12:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.516+0000 7f349b200640 1 -- 172.21.15.107:0/372911425 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34780b3d40 msgr2=0x7f34780b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.516+0000 7f349b200640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34780b3d40 0x7f34780b6250 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f348c011690 tx=0x7f348c009500 comp rx=0 tx=0).stop 2024-07-26T12:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.516+0000 7f349b200640 1 -- 172.21.15.107:0/372911425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3494103520 msgr2=0x7f349419c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:00.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.516+0000 7f349b200640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3494103520 0x7f349419c0d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f348400e9f0 tx=0x7f348400eeb0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.516+0000 7f349b200640 1 -- 172.21.15.107:0/372911425 shutdown_connections 2024-07-26T12:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.516+0000 7f349b200640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f34780b3d40 0x7f34780b6250 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.516+0000 7f349b200640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f34941097a0 0x7f34941a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.516+0000 7f349b200640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3494108e50 0x7f349419c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.516+0000 7f349b200640 1 --2- 172.21.15.107:0/372911425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3494103520 0x7f349419c0d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.516+0000 7f349b200640 1 -- 172.21.15.107:0/372911425 >> 172.21.15.107:0/372911425 conn(0x7f34940fefa0 msgr2=0x7f3494107950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.517+0000 7f349b200640 1 -- 172.21.15.107:0/372911425 shutdown_connections 2024-07-26T12:51:00.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:00.517+0000 7f349b200640 1 -- 172.21.15.107:0/372911425 wait complete. 2024-07-26T12:51:00.576 INFO:tasks.workunit.client.0.smithi107.stdout:checking for FICLONE... yes 2024-07-26T12:51:00.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T12:51:00.684 INFO:tasks.workunit.client.0.smithi107.stdout:checking for renameat2... yes 2024-07-26T12:51:00.765 INFO:tasks.workunit.client.0.smithi107.stdout:checking for reallocarray... yes 2024-07-26T12:51:00.875 INFO:tasks.workunit.client.0.smithi107.stdout:checking for struct mount_attr... yes 2024-07-26T12:51:00.988 INFO:tasks.workunit.client.0.smithi107.stdout:checking for struct btrfs_qgroup_limit... yes 2024-07-26T12:51:01.032 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:51:01.096 INFO:tasks.workunit.client.0.smithi107.stdout:checking for struct btrfs_qgroup_inherit... yes 2024-07-26T12:51:01.211 INFO:tasks.workunit.client.0.smithi107.stdout:checking for struct btrfs_ioctl_vol_args... yes 2024-07-26T12:51:01.221 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:00 smithi107 ceph-mon[32659]: pgmap v542: 129 pgs: 129 active+clean; 40 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 1.3 MiB/s wr, 185 op/s 2024-07-26T12:51:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:00 smithi132 ceph-mon[37580]: pgmap v542: 129 pgs: 129 active+clean; 40 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 1.3 MiB/s wr, 185 op/s 2024-07-26T12:51:01.364 INFO:tasks.workunit.client.0.smithi107.stdout:checking for struct btrfs_ioctl_vol_args_v2... yes 2024-07-26T12:51:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:00 smithi112 ceph-mon[37501]: pgmap v542: 129 pgs: 129 active+clean; 40 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 1.3 MiB/s wr, 185 op/s 2024-07-26T12:51:01.489 INFO:tasks.workunit.client.0.smithi107.stdout:checking for struct btrfs_ioctl_ino_lookup_args... yes 2024-07-26T12:51:01.614 INFO:tasks.workunit.client.0.smithi107.stdout:checking for struct btrfs_ioctl_ino_lookup_user_args... yes 2024-07-26T12:51:01.726 INFO:tasks.workunit.client.0.smithi107.stdout:checking for struct btrfs_ioctl_get_subvol_rootref_args... yes 2024-07-26T12:51:01.805 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/btrfs.h usability... yes 2024-07-26T12:51:01.846 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/btrfs.h presence... yes 2024-07-26T12:51:01.846 INFO:tasks.workunit.client.0.smithi107.stdout:checking for linux/btrfs.h... yes 2024-07-26T12:51:01.961 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/btrfs_tree.h usability... yes 2024-07-26T12:51:02.002 INFO:tasks.workunit.client.0.smithi107.stdout:checking linux/btrfs_tree.h presence... yes 2024-07-26T12:51:02.002 INFO:tasks.workunit.client.0.smithi107.stdout:checking for linux/btrfs_tree.h... yes 2024-07-26T12:51:02.087 INFO:tasks.workunit.client.0.smithi107.stdout:checking for struct btrfs_ioctl_vol_args_v2.subvolid... yes 2024-07-26T12:51:02.152 INFO:tasks.workunit.client.0.smithi107.stdout:checking whether BTRFS_IOC_SNAP_DESTROY_V2 is declared... yes 2024-07-26T12:51:02.197 INFO:tasks.workunit.client.0.smithi107.stdout:configure: creating ./config.status 2024-07-26T12:51:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:02 smithi107 ceph-mon[32659]: pgmap v543: 129 pgs: 129 active+clean; 37 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 1.4 MiB/s wr, 232 op/s 2024-07-26T12:51:03.298 INFO:tasks.workunit.client.0.smithi107.stdout:config.status: creating include/builddefs 2024-07-26T12:51:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:02 smithi132 ceph-mon[37580]: pgmap v543: 129 pgs: 129 active+clean; 37 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 1.4 MiB/s wr, 232 op/s 2024-07-26T12:51:03.347 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:02 smithi112 ceph-mon[37501]: pgmap v543: 129 pgs: 129 active+clean; 37 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 1.4 MiB/s wr, 232 op/s 2024-07-26T12:51:03.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:51:03.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: *** Caught signal (Aborted) ** 2024-07-26T12:51:03.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: in thread 7f357b35e640 thread_name:ms_dispatch 2024-07-26T12:51:03.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:03.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f35807356f0] 2024-07-26T12:51:03.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f358078294c] 2024-07-26T12:51:03.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 3: raise() 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 4: abort() 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 5: /lib64/libc.so.6(+0x2871b) [0x7f358071f71b] 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 6: /lib64/libc.so.6(+0x37386) [0x7f358072e386] 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55eaabf2a6b9] 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55eaac0e880c] 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55eaac036cde] 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55eaabff8763] 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55eaabff9a1b] 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55eaabfdd752] 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55eaabf2ba56] 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55eaabf2f637] 2024-07-26T12:51:03.702 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55eaabf2ff5e] 2024-07-26T12:51:03.703 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55eaabf1303a] 2024-07-26T12:51:03.703 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3581002407] 2024-07-26T12:51:03.703 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 18: (DispatchQueue::entry()+0x60f) [0x7f3581002cf9] 2024-07-26T12:51:03.703 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f35810df5a1] 2024-07-26T12:51:03.704 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 20: (Thread::entry_wrapper()+0x43) [0x7f3580e8ccdd] 2024-07-26T12:51:03.704 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3580e8ccf9] 2024-07-26T12:51:03.704 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3580780c02] 2024-07-26T12:51:03.704 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3580805c40] 2024-07-26T12:51:03.704 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2024-07-26T12:51:03.347+0000 7f357b35e640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:03.704 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: in thread 7f357b35e640 thread_name:ms_dispatch 2024-07-26T12:51:03.704 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2024-07-26T12:51:03.704 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:03.704 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f35807356f0] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f358078294c] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 3: raise() 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 4: abort() 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 5: /lib64/libc.so.6(+0x2871b) [0x7f358071f71b] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 6: /lib64/libc.so.6(+0x37386) [0x7f358072e386] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55eaabf2a6b9] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55eaac0e880c] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55eaac036cde] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55eaabff8763] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55eaabff9a1b] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55eaabfdd752] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55eaabf2ba56] 2024-07-26T12:51:03.705 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55eaabf2f637] 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55eaabf2ff5e] 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55eaabf1303a] 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3581002407] 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 18: (DispatchQueue::entry()+0x60f) [0x7f3581002cf9] 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f35810df5a1] 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 20: (Thread::entry_wrapper()+0x43) [0x7f3580e8ccdd] 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3580e8ccf9] 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3580780c02] 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3580805c40] 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 0> 2024-07-26T12:51:03.347+0000 7f357b35e640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:03.706 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: in thread 7f357b35e640 thread_name:ms_dispatch 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f35807356f0] 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f358078294c] 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 3: raise() 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 4: abort() 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 5: /lib64/libc.so.6(+0x2871b) [0x7f358071f71b] 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 6: /lib64/libc.so.6(+0x37386) [0x7f358072e386] 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55eaabf2a6b9] 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55eaac0e880c] 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55eaac036cde] 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55eaabff8763] 2024-07-26T12:51:03.707 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55eaabff9a1b] 2024-07-26T12:51:03.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55eaabfdd752] 2024-07-26T12:51:03.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55eaabf2ba56] 2024-07-26T12:51:03.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55eaabf2f637] 2024-07-26T12:51:03.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55eaabf2ff5e] 2024-07-26T12:51:03.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55eaabf1303a] 2024-07-26T12:51:03.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3581002407] 2024-07-26T12:51:03.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 18: (DispatchQueue::entry()+0x60f) [0x7f3581002cf9] 2024-07-26T12:51:03.708 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f35810df5a1] 2024-07-26T12:51:03.709 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 20: (Thread::entry_wrapper()+0x43) [0x7f3580e8ccdd] 2024-07-26T12:51:03.709 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3580e8ccf9] 2024-07-26T12:51:03.709 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3580780c02] 2024-07-26T12:51:03.709 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3580805c40] 2024-07-26T12:51:03.709 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:03.709 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2024-07-26T12:51:03.709 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 0> 2024-07-26T12:51:03.347+0000 7f357b35e640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:03.709 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: in thread 7f357b35e640 thread_name:ms_dispatch 2024-07-26T12:51:03.709 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2024-07-26T12:51:03.709 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f35807356f0] 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f358078294c] 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 3: raise() 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 4: abort() 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 5: /lib64/libc.so.6(+0x2871b) [0x7f358071f71b] 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 6: /lib64/libc.so.6(+0x37386) [0x7f358072e386] 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55eaabf2a6b9] 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55eaac0e880c] 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55eaac036cde] 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55eaabff8763] 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55eaabff9a1b] 2024-07-26T12:51:03.710 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55eaabfdd752] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55eaabf2ba56] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55eaabf2f637] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55eaabf2ff5e] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55eaabf1303a] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3581002407] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 18: (DispatchQueue::entry()+0x60f) [0x7f3581002cf9] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f35810df5a1] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 20: (Thread::entry_wrapper()+0x43) [0x7f3580e8ccdd] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3580e8ccf9] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3580780c02] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3580805c40] 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:03.711 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[69133]: 2024-07-26T12:51:03.712 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:03 smithi112 podman[71903]: 2024-07-26 12:51:03.412206605 +0000 UTC m=+0.020570609 container died 79e81ecd91c8b7344c1b48f68400eb126e9209f3fc49cb85561891912b22dad9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-07-26T12:51:03.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.921+0000 7fa045600640 1 -- 172.21.15.107:0/1137469626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0401055d0 msgr2=0x7fa0401059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.921+0000 7fa045600640 1 --2- 172.21.15.107:0/1137469626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0401055d0 0x7fa0401059e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa030009900 tx=0x7fa03002a3f0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.921+0000 7fa045600640 1 -- 172.21.15.107:0/1137469626 shutdown_connections 2024-07-26T12:51:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.921+0000 7fa045600640 1 --2- 172.21.15.107:0/1137469626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa04010b860 0x7fa040112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.921+0000 7fa045600640 1 --2- 172.21.15.107:0/1137469626 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa04010af10 0x7fa04010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.921+0000 7fa045600640 1 --2- 172.21.15.107:0/1137469626 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0401055d0 0x7fa0401059e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.921+0000 7fa045600640 1 -- 172.21.15.107:0/1137469626 >> 172.21.15.107:0/1137469626 conn(0x7fa040101040 msgr2=0x7fa040103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.921+0000 7fa045600640 1 -- 172.21.15.107:0/1137469626 shutdown_connections 2024-07-26T12:51:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.921+0000 7fa045600640 1 -- 172.21.15.107:0/1137469626 wait complete. 2024-07-26T12:51:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.923+0000 7fa045600640 1 Processor -- start 2024-07-26T12:51:03.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.923+0000 7fa045600640 1 -- start start 2024-07-26T12:51:03.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa045600640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0401055d0 0x7fa04019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa045600640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa04010af10 0x7fa04019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa045600640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04010b860 0x7fa0401a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa045600640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa040115560 con 0x7fa04010b860 2024-07-26T12:51:03.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa045600640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401153e0 con 0x7fa0401055d0 2024-07-26T12:51:03.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa045600640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401156e0 con 0x7fa04010af10 2024-07-26T12:51:03.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa03fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0401055d0 0x7fa04019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa03fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0401055d0 0x7fa04019bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:43986/0 (socket says 172.21.15.107:43986) 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa03fe00640 1 -- 172.21.15.107:0/4142786732 learned_addr learned my addr 172.21.15.107:0/4142786732 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa044c00640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04010b860 0x7fa0401a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.924+0000 7fa037e00640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa04010af10 0x7fa04019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.925+0000 7fa03fe00640 1 -- 172.21.15.107:0/4142786732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa04010af10 msgr2=0x7fa04019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.925+0000 7fa03fe00640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa04010af10 0x7fa04019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.925+0000 7fa03fe00640 1 -- 172.21.15.107:0/4142786732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04010b860 msgr2=0x7fa0401a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.925+0000 7fa03fe00640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04010b860 0x7fa0401a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.925+0000 7fa03fe00640 1 -- 172.21.15.107:0/4142786732 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0401a1ec0 con 0x7fa0401055d0 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.925+0000 7fa03fe00640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0401055d0 0x7fa04019bf50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa030005ab0 tx=0x7fa030003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.926+0000 7fa03d600640 1 -- 172.21.15.107:0/4142786732 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03003f070 con 0x7fa0401055d0 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.926+0000 7fa03d600640 1 -- 172.21.15.107:0/4142786732 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa030031b60 con 0x7fa0401055d0 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.926+0000 7fa045600640 1 -- 172.21.15.107:0/4142786732 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0401a2150 con 0x7fa0401055d0 2024-07-26T12:51:03.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.926+0000 7fa03d600640 1 -- 172.21.15.107:0/4142786732 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030035030 con 0x7fa0401055d0 2024-07-26T12:51:03.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.926+0000 7fa045600640 1 -- 172.21.15.107:0/4142786732 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0401a8910 con 0x7fa0401055d0 2024-07-26T12:51:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.929+0000 7fa04d7c6740 1 Processor -- start 2024-07-26T12:51:03.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.929+0000 7fa04d7c6740 1 -- start start 2024-07-26T12:51:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559954fa4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fa4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fa4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559954f1abd0 con 0x559954bff070 2024-07-26T12:51:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa03d600640 1 -- 172.21.15.107:0/4142786732 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa030042070 con 0x7fa0401055d0 2024-07-26T12:51:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559954f0a0d0 con 0x559954bd61b0 2024-07-26T12:51:03.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559954f0e110 con 0x559954bde390 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa037e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fa4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa044c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559954fa4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa037e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fa4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:47614/0 (socket says 172.21.15.107:47614) 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa037e00640 1 -- 172.21.15.107:0/3282281483 learned_addr learned my addr 172.21.15.107:0/3282281483 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.930+0000 7fa03fe00640 1 --2- 172.21.15.107:0/3282281483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fa4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.931+0000 7fa037e00640 1 -- 172.21.15.107:0/3282281483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 msgr2=0x559954fa4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.931+0000 7fa037e00640 1 --2- 172.21.15.107:0/3282281483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fa4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.931+0000 7fa037e00640 1 -- 172.21.15.107:0/3282281483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 msgr2=0x559954fa4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.931+0000 7fa037e00640 1 --2- 172.21.15.107:0/3282281483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559954fa4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.931+0000 7fa037e00640 1 -- 172.21.15.107:0/3282281483 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559954c6a650 con 0x559954bde390 2024-07-26T12:51:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.931+0000 7fa037e00640 1 --2- 172.21.15.107:0/3282281483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fa4a90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa028009c20 tx=0x7fa028011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74008ba2cf9e855a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.931+0000 7fa035200640 1 -- 172.21.15.107:0/3282281483 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028004230 con 0x559954bde390 2024-07-26T12:51:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.932+0000 7fa035200640 1 -- 172.21.15.107:0/3282281483 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0280043d0 con 0x559954bde390 2024-07-26T12:51:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.932+0000 7fa035200640 1 -- 172.21.15.107:0/3282281483 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028004ca0 con 0x559954bde390 2024-07-26T12:51:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.932+0000 7fa03d600640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa0240b3dc0 0x7fa0240b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.932+0000 7fa03d600640 1 -- 172.21.15.107:0/4142786732 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7fa0301049c0 con 0x7fa0401055d0 2024-07-26T12:51:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.932+0000 7fa044c00640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa0240b3dc0 0x7fa0240b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.933+0000 7fa04d7c6740 1 -- 172.21.15.107:0/3282281483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 msgr2=0x559954fa4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:03.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.933+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/3282281483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fa4a90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa028009c20 tx=0x7fa028011810 comp rx=0 tx=0).stop 2024-07-26T12:51:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.933+0000 7fa04d7c6740 1 -- 172.21.15.107:0/3282281483 shutdown_connections 2024-07-26T12:51:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.933+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/3282281483 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fa4a90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.933+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/3282281483 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fa4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.933+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/3282281483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559954fa4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.933+0000 7fa04d7c6740 1 -- 172.21.15.107:0/3282281483 >> 172.21.15.107:0/3282281483 conn(0x559954bf3460 msgr2=0x559954fa1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.933+0000 7fa04d7c6740 1 -- 172.21.15.107:0/3282281483 shutdown_connections 2024-07-26T12:51:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.933+0000 7fa04d7c6740 1 -- 172.21.15.107:0/3282281483 wait complete. 2024-07-26T12:51:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.933+0000 7fa044c00640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa0240b3dc0 0x7fa0240b62d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa038005fd0 tx=0x7fa038009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.934+0000 7fa04d7c6740 1 -- start start 2024-07-26T12:51:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.934+0000 7fa04d7c6740 1 Processor -- start 2024-07-26T12:51:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.934+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954faf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.934+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fafac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.934+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559954fb0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559954d6d700 con 0x559954bff070 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x559954ea66a0 con 0x559954bd61b0 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x559954fb27f0 con 0x559954bde390 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa04d7c6740 10 client.0 _ll_register_callbacks cb 0x559954eb6150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa037e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fafac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa044c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559954fb0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa037e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fafac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:47622/0 (socket says 172.21.15.107:47622) 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa037e00640 1 -- 172.21.15.107:0/349882311 learned_addr learned my addr 172.21.15.107:0/349882311 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa03fe00640 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954faf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa037e00640 1 -- 172.21.15.107:0/349882311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 msgr2=0x559954faf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa037e00640 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954faf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa037e00640 1 -- 172.21.15.107:0/349882311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 msgr2=0x559954fb0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa037e00640 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559954fb0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa037e00640 1 -- 172.21.15.107:0/349882311 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559954ebd840 con 0x559954bde390 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.935+0000 7fa03fe00640 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954faf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.936+0000 7fa037e00640 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fafac0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa028000c00 tx=0x7fa028019ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f74bbd2d3db6c774 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.936+0000 7fa02d600640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.936+0000 7fa02d600640 1 -- 172.21.15.107:0/349882311 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028028020 con 0x559954bde390 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.936+0000 7fa04d7c6740 1 -- 172.21.15.107:0/349882311 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0280043d0 con 0x559954bde390 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.936+0000 7fa02d600640 1 -- 172.21.15.107:0/349882311 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7fa028024620 con 0x559954bde390 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.936+0000 7fa02d600640 1 -- 172.21.15.107:0/349882311 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa028033050 con 0x559954bde390 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.936+0000 7fa04d7c6740 10 client.25909 fetch_fsmap learned FSMap version 35 2024-07-26T12:51:03.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.936+0000 7fa04d7c6740 1 -- 172.21.15.107:0/349882311 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559954fbe380 con 0x559954bde390 2024-07-26T12:51:03.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.937+0000 7fa02d600640 1 -- 172.21.15.107:0/349882311 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0280043d0 con 0x559954bde390 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.937+0000 7fa02d600640 1 -- 172.21.15.107:0/349882311 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7fa028018070 con 0x559954bde390 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.937+0000 7fa04d7c6740 10 client.25909 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.937+0000 7fa04d7c6740 10 client.25909 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.937+0000 7fa04d7c6740 20 client.25909 populate_metadata read hostname 'smithi107' 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.937+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x559954bbd9f0 0x559954fc1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.937+0000 7fa04d7c6740 4 client.25909 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.937+0000 7fa04d7c6740 1 -- 172.21.15.107:0/349882311 --> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559954f1cb20 con 0x559954bbd9f0 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.937+0000 7fa03fe00640 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x559954bbd9f0 0x559954fc1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.938+0000 7fa03fe00640 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x559954bbd9f0 0x559954fc1740 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e0fb0e71810ae71 server_cookie=d57d7eda6e33e623 in_seq=0 out_seq=0 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.938+0000 7fa02d600640 10 client.25909 ms_handle_connect on v2:172.21.15.132:6838/3489644052 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa02d600640 1 -- 172.21.15.107:0/349882311 <== mds.2 v2:172.21.15.132:6838/3489644052 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559954f1cb20 con 0x559954bbd9f0 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa02d600640 10 client.25909 handle_command_reply: tid=0 2024-07-26T12:51:03.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 client.25909 shutdown 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 -- 172.21.15.107:0/349882311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 msgr2=0x559954fafac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fafac0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa028000c00 tx=0x7fa028019ae0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 -- 172.21.15.107:0/349882311 shutdown_connections 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559954fb0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fafac0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x559954bbd9f0 0x559954fc1740 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/349882311 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954faf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 -- 172.21.15.107:0/349882311 >> 172.21.15.107:0/349882311 conn(0x559954bf3460 msgr2=0x559954fa71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa02d600640 0 client.25909 ms_handle_reset on v2:172.21.15.132:6838/3489644052 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 -- 172.21.15.107:0/349882311 shutdown_connections 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.939+0000 7fa04d7c6740 1 -- 172.21.15.107:0/349882311 wait complete. 2024-07-26T12:51:03.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.940+0000 7fa04d7c6740 20 client.25909 trim_cache size 0 max 16384 2024-07-26T12:51:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.954+0000 7fa04d7c6740 1 Processor -- start 2024-07-26T12:51:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.954+0000 7fa04d7c6740 1 -- start start 2024-07-26T12:51:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.954+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.954+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bde390 0x559954fbb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.954+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bff070 0x559954fc6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.954+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401b2860 con 0x559954bde390 2024-07-26T12:51:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.954+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401b4460 con 0x559954bd61b0 2024-07-26T12:51:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.954+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401b2f30 con 0x559954bff070 2024-07-26T12:51:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa03fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa037e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bde390 0x559954fbb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa044c00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bff070 0x559954fc6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa03fe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44016/0 (socket says 172.21.15.107:44016) 2024-07-26T12:51:03.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa03fe00640 1 -- 172.21.15.107:0/3292199645 learned_addr learned my addr 172.21.15.107:0/3292199645 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:03.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa03fe00640 1 -- 172.21.15.107:0/3292199645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bff070 msgr2=0x559954fc6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:03.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa03fe00640 1 --2- 172.21.15.107:0/3292199645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bff070 0x559954fc6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa03fe00640 1 -- 172.21.15.107:0/3292199645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bde390 msgr2=0x559954fbb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:03.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa03fe00640 1 --2- 172.21.15.107:0/3292199645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bde390 0x559954fbb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa03fe00640 1 -- 172.21.15.107:0/3292199645 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559954fa6300 con 0x559954bd61b0 2024-07-26T12:51:03.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.955+0000 7fa03fe00640 1 --2- 172.21.15.107:0/3292199645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb1340 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa030031440 tx=0x7fa030009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b763ce3d15651ab server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:03.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.956+0000 7fa02fe00640 1 -- 172.21.15.107:0/3292199645 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030049040 con 0x559954bd61b0 2024-07-26T12:51:03.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.956+0000 7fa02fe00640 1 -- 172.21.15.107:0/3292199645 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0300cae10 con 0x559954bd61b0 2024-07-26T12:51:03.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.956+0000 7fa02fe00640 1 -- 172.21.15.107:0/3292199645 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0300d4300 con 0x559954bd61b0 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.956+0000 7fa04d7c6740 1 -- 172.21.15.107:0/3292199645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 msgr2=0x559954fb1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.956+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/3292199645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb1340 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa030031440 tx=0x7fa030009260 comp rx=0 tx=0).stop 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.956+0000 7fa04d7c6740 1 -- 172.21.15.107:0/3292199645 shutdown_connections 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.956+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/3292199645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bff070 0x559954fc6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.956+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/3292199645 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bde390 0x559954fbb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.956+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/3292199645 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb1340 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.956+0000 7fa04d7c6740 1 -- 172.21.15.107:0/3292199645 >> 172.21.15.107:0/3292199645 conn(0x559954bf3460 msgr2=0x559954fa5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.957+0000 7fa04d7c6740 1 -- 172.21.15.107:0/3292199645 shutdown_connections 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.957+0000 7fa04d7c6740 1 -- 172.21.15.107:0/3292199645 wait complete. 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.957+0000 7fa04d7c6740 1 -- start start 2024-07-26T12:51:03.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.957+0000 7fa04d7c6740 1 Processor -- start 2024-07-26T12:51:03.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fb5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa04d7c6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559955027da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401b3d90 con 0x559954bff070 2024-07-26T12:51:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401b3600 con 0x559954bd61b0 2024-07-26T12:51:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa04d7c6740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401b0840 con 0x559954bde390 2024-07-26T12:51:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa04d7c6740 10 client.0 _ll_register_callbacks cb 0x7fa028009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:51:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa037e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fb5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa044c00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559955027da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa037e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fb5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:47638/0 (socket says 172.21.15.107:47638) 2024-07-26T12:51:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.958+0000 7fa037e00640 1 -- 172.21.15.107:0/2483606029 learned_addr learned my addr 172.21.15.107:0/2483606029 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.959+0000 7fa037e00640 1 -- 172.21.15.107:0/2483606029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 msgr2=0x559954fb4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:03.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa03fe00640 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa037e00640 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa037e00640 1 -- 172.21.15.107:0/2483606029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 msgr2=0x559955027da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa037e00640 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559955027da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa037e00640 1 -- 172.21.15.107:0/2483606029 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599550282e0 con 0x559954bde390 2024-07-26T12:51:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa044c00640 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559955027da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa03fe00640 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T12:51:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa037e00640 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fb5390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa0280059f0 tx=0x7fa028005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1874485092ca0bc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa02d600640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:51:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa02d600640 1 -- 172.21.15.107:0/2483606029 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028019e60 con 0x559954bde390 2024-07-26T12:51:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa02d600640 1 -- 172.21.15.107:0/2483606029 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7fa028018a70 con 0x559954bde390 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.960+0000 7fa04d7c6740 1 -- 172.21.15.107:0/2483606029 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0300cae10 con 0x559954bde390 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa02d600640 1 -- 172.21.15.107:0/2483606029 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa028028040 con 0x559954bde390 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa04d7c6740 10 client.25915 fetch_fsmap learned FSMap version 35 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa04d7c6740 1 -- 172.21.15.107:0/2483606029 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559955029490 con 0x559954bde390 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa02d600640 1 -- 172.21.15.107:0/2483606029 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0300cae10 con 0x559954bde390 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa02d600640 1 -- 172.21.15.107:0/2483606029 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7fa028018040 con 0x559954bde390 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa04d7c6740 10 client.25915 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa04d7c6740 10 client.25915 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa04d7c6740 20 client.25915 populate_metadata read hostname 'smithi107' 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x559954b92da0 0x55995502bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa04d7c6740 4 client.25915 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T12:51:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.961+0000 7fa04d7c6740 1 -- 172.21.15.107:0/2483606029 --> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559954f185c0 con 0x559954b92da0 2024-07-26T12:51:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.962+0000 7fa03fe00640 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x559954b92da0 0x55995502bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.962+0000 7fa02d600640 10 client.25915 ms_handle_connect on v2:172.21.15.132:6838/3489644052 2024-07-26T12:51:03.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.962+0000 7fa03fe00640 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x559954b92da0 0x55995502bc70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5512bcc32e3b9ea server_cookie=6bb3ae4f1424868e in_seq=0 out_seq=0 2024-07-26T12:51:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa02d600640 1 -- 172.21.15.107:0/2483606029 <== mds.2 v2:172.21.15.132:6838/3489644052 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559954f185c0 con 0x559954b92da0 2024-07-26T12:51:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa02d600640 10 client.25915 handle_command_reply: tid=0 2024-07-26T12:51:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 client.25915 shutdown 2024-07-26T12:51:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 -- 172.21.15.107:0/2483606029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 msgr2=0x559954fb5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fb5390 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa0280059f0 tx=0x7fa028005f10 comp rx=0 tx=0).stop 2024-07-26T12:51:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 -- 172.21.15.107:0/2483606029 shutdown_connections 2024-07-26T12:51:03.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559954bff070 0x559955027da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x559954bde390 0x559954fb5390 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] conn(0x559954b92da0 0x55995502bc70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 --2- 172.21.15.107:0/2483606029 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x559954bd61b0 0x559954fb4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa02d600640 0 client.25915 ms_handle_reset on v2:172.21.15.132:6838/3489644052 2024-07-26T12:51:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 -- 172.21.15.107:0/2483606029 >> 172.21.15.107:0/2483606029 conn(0x559954bf3460 msgr2=0x559954fa5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 -- 172.21.15.107:0/2483606029 shutdown_connections 2024-07-26T12:51:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 1 -- 172.21.15.107:0/2483606029 wait complete. 2024-07-26T12:51:03.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.963+0000 7fa04d7c6740 20 client.25915 trim_cache size 0 max 16384 2024-07-26T12:51:03.971 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-07-26T12:51:03.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 -- 172.21.15.107:0/4142786732 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa0240b3dc0 msgr2=0x7fa0240b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa0240b3dc0 0x7fa0240b62d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa038005fd0 tx=0x7fa038009500 comp rx=0 tx=0).stop 2024-07-26T12:51:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 -- 172.21.15.107:0/4142786732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0401055d0 msgr2=0x7fa04019bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0401055d0 0x7fa04019bf50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa030005ab0 tx=0x7fa030003c10 comp rx=0 tx=0).stop 2024-07-26T12:51:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 -- 172.21.15.107:0/4142786732 shutdown_connections 2024-07-26T12:51:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7fa0240b3dc0 0x7fa0240b62d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa04010b860 0x7fa0401a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa04010af10 0x7fa04019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 --2- 172.21.15.107:0/4142786732 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7fa0401055d0 0x7fa04019bf50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 -- 172.21.15.107:0/4142786732 >> 172.21.15.107:0/4142786732 conn(0x7fa040101040 msgr2=0x7fa040103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:03.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 -- 172.21.15.107:0/4142786732 shutdown_connections 2024-07-26T12:51:03.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:03.964+0000 7fa045600640 1 -- 172.21.15.107:0/4142786732 wait complete. 2024-07-26T12:51:04.329 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:04 smithi112 podman[71903]: 2024-07-26 12:51:04.070415713 +0000 UTC m=+0.678779715 container remove 79e81ecd91c8b7344c1b48f68400eb126e9209f3fc49cb85561891912b22dad9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T12:51:04.330 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:04 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:51:04.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:04 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T12:51:04.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:04 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Consumed 28.453s CPU time. 2024-07-26T12:51:04.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T12:51:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:04 smithi107 ceph-mon[32659]: pgmap v544: 129 pgs: 129 active+clean; 37 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.2 MiB/s wr, 208 op/s 2024-07-26T12:51:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:04 smithi132 ceph-mon[37580]: pgmap v544: 129 pgs: 129 active+clean; 37 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.2 MiB/s wr, 208 op/s 2024-07-26T12:51:05.406 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:51:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:04 smithi112 ceph-mon[37501]: pgmap v544: 129 pgs: 129 active+clean; 37 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.2 MiB/s wr, 208 op/s 2024-07-26T12:51:07.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:06 smithi107 ceph-mon[32659]: pgmap v545: 129 pgs: 129 active+clean; 37 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 1.1 MiB/s wr, 203 op/s 2024-07-26T12:51:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:06 smithi132 ceph-mon[37580]: pgmap v545: 129 pgs: 129 active+clean; 37 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 1.1 MiB/s wr, 203 op/s 2024-07-26T12:51:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:06 smithi112 ceph-mon[37501]: pgmap v545: 129 pgs: 129 active+clean; 37 MiB data, 605 MiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 1.1 MiB/s wr, 203 op/s 2024-07-26T12:51:07.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.841+0000 7f7403e00640 1 -- 172.21.15.107:0/631501617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7404101810 msgr2=0x7f7404101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:07.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.841+0000 7f7403e00640 1 --2- 172.21.15.107:0/631501617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7404101810 0x7f7404101c20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009860 tx=0x7f73f402a340 comp rx=0 tx=0).stop 2024-07-26T12:51:07.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.841+0000 7f7403e00640 1 -- 172.21.15.107:0/631501617 shutdown_connections 2024-07-26T12:51:07.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.841+0000 7f7403e00640 1 --2- 172.21.15.107:0/631501617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7404106440 0x7f740410fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.841+0000 7f7403e00640 1 --2- 172.21.15.107:0/631501617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7404105af0 0x7f7404105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.841+0000 7f7403e00640 1 --2- 172.21.15.107:0/631501617 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7404101810 0x7f7404101c20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.841+0000 7f7403e00640 1 -- 172.21.15.107:0/631501617 >> 172.21.15.107:0/631501617 conn(0x7f74040774e0 msgr2=0x7f7404077920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:07.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.841+0000 7f7403e00640 1 -- 172.21.15.107:0/631501617 shutdown_connections 2024-07-26T12:51:07.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.841+0000 7f7403e00640 1 -- 172.21.15.107:0/631501617 wait complete. 2024-07-26T12:51:07.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.842+0000 7f7403e00640 1 Processor -- start 2024-07-26T12:51:07.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.842+0000 7f7403e00640 1 -- start start 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f7403e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7404101810 0x7f7404199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f7403e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7404105af0 0x7f740419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f7403e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7404106440 0x7f740419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f7403e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f74041132f0 con 0x7f7404101810 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f7403e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f7404113170 con 0x7f7404105af0 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f7403e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f7404113470 con 0x7f7404106440 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f73fbe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7404105af0 0x7f740419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f7402a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7404101810 0x7f7404199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f73fbe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7404105af0 0x7f740419a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44052/0 (socket says 172.21.15.107:44052) 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f73fbe00640 1 -- 172.21.15.107:0/4250990210 learned_addr learned my addr 172.21.15.107:0/4250990210 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:07.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.843+0000 7f7403400640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7404106440 0x7f740419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.844+0000 7f7402a00640 1 -- 172.21.15.107:0/4250990210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7404106440 msgr2=0x7f740419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:07.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.844+0000 7f7402a00640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7404106440 0x7f740419f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.844+0000 7f7402a00640 1 -- 172.21.15.107:0/4250990210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7404105af0 msgr2=0x7f740419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:07.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.844+0000 7f7402a00640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7404105af0 0x7f740419a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.844+0000 7f7402a00640 1 -- 172.21.15.107:0/4250990210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740419fd50 con 0x7f7404101810 2024-07-26T12:51:07.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.844+0000 7f73fbe00640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7404105af0 0x7f740419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:07.847 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.844+0000 7f7402a00640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7404101810 0x7f7404199de0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f73f4035040 tx=0x7f73f4032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:07.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.845+0000 7f73fb400640 1 -- 172.21.15.107:0/4250990210 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4040020 con 0x7f7404101810 2024-07-26T12:51:07.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.845+0000 7f73fb400640 1 -- 172.21.15.107:0/4250990210 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73f4030e10 con 0x7f7404101810 2024-07-26T12:51:07.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.845+0000 7f7403e00640 1 -- 172.21.15.107:0/4250990210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f740419ffe0 con 0x7f7404101810 2024-07-26T12:51:07.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.845+0000 7f73fb400640 1 -- 172.21.15.107:0/4250990210 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f4037040 con 0x7f7404101810 2024-07-26T12:51:07.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.845+0000 7f7403e00640 1 -- 172.21.15.107:0/4250990210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74041a67a0 con 0x7f7404101810 2024-07-26T12:51:07.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.849+0000 7f7410254740 1 Processor -- start 2024-07-26T12:51:07.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.850+0000 7f7410254740 1 -- start start 2024-07-26T12:51:07.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.850+0000 7f7410254740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e90b070 0x558a4ecb0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.850+0000 7f7410254740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4ecb0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.850+0000 7f7410254740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e8ea390 0x558a4ecb0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.850+0000 7f7410254740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558a4ec26bd0 con 0x558a4e90b070 2024-07-26T12:51:07.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.850+0000 7f7410254740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558a4ec160d0 con 0x558a4e8ea390 2024-07-26T12:51:07.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.850+0000 7f7410254740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558a4ec1a110 con 0x558a4e8e21b0 2024-07-26T12:51:07.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f73fbe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e8ea390 0x558a4ecb0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f7403400640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e90b070 0x558a4ecb0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f73fbe00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e8ea390 0x558a4ecb0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44060/0 (socket says 172.21.15.107:44060) 2024-07-26T12:51:07.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f73fbe00640 1 -- 172.21.15.107:0/2596849560 learned_addr learned my addr 172.21.15.107:0/2596849560 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:07.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f73fb400640 1 -- 172.21.15.107:0/4250990210 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f73f4043020 con 0x7f7404101810 2024-07-26T12:51:07.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f73fbe00640 1 -- 172.21.15.107:0/2596849560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 msgr2=0x558a4ecb0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:07.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f73fbe00640 1 --2- 172.21.15.107:0/2596849560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4ecb0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f73fbe00640 1 -- 172.21.15.107:0/2596849560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e90b070 msgr2=0x558a4ecb0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f73fbe00640 1 --2- 172.21.15.107:0/2596849560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e90b070 0x558a4ecb0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f73fbe00640 1 -- 172.21.15.107:0/2596849560 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a4e976650 con 0x558a4e8ea390 2024-07-26T12:51:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.851+0000 7f7403400640 1 --2- 172.21.15.107:0/2596849560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e90b070 0x558a4ecb0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.852+0000 7f73fbe00640 1 --2- 172.21.15.107:0/2596849560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e8ea390 0x558a4ecb0a90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f73ec011540 tx=0x7f73ec011940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df2b8b264ace6209 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.852+0000 7f73f3400640 1 -- 172.21.15.107:0/2596849560 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec01e070 con 0x558a4e8ea390 2024-07-26T12:51:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.852+0000 7f73f3400640 1 -- 172.21.15.107:0/2596849560 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73ec00fa70 con 0x558a4e8ea390 2024-07-26T12:51:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.852+0000 7f73f3400640 1 -- 172.21.15.107:0/2596849560 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec019b90 con 0x558a4e8ea390 2024-07-26T12:51:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f73fb400640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f73e80b3d40 0x7f73e80b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f7410254740 1 -- 172.21.15.107:0/2596849560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e8ea390 msgr2=0x558a4ecb0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f7410254740 1 --2- 172.21.15.107:0/2596849560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e8ea390 0x558a4ecb0a90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f73ec011540 tx=0x7f73ec011940 comp rx=0 tx=0).stop 2024-07-26T12:51:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f73fb400640 1 -- 172.21.15.107:0/4250990210 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f73f4103e40 con 0x7f7404101810 2024-07-26T12:51:07.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f7410254740 1 -- 172.21.15.107:0/2596849560 shutdown_connections 2024-07-26T12:51:07.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f7410254740 1 --2- 172.21.15.107:0/2596849560 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e8ea390 0x558a4ecb0a90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f7403400640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f73e80b3d40 0x7f73e80b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.864 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f7410254740 1 --2- 172.21.15.107:0/2596849560 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4ecb0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f7410254740 1 --2- 172.21.15.107:0/2596849560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e90b070 0x558a4ecb0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f7410254740 1 -- 172.21.15.107:0/2596849560 >> 172.21.15.107:0/2596849560 conn(0x558a4e8ff460 msgr2=0x558a4ecadf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:07.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f7410254740 1 -- 172.21.15.107:0/2596849560 shutdown_connections 2024-07-26T12:51:07.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.853+0000 7f7410254740 1 -- 172.21.15.107:0/2596849560 wait complete. 2024-07-26T12:51:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.854+0000 7f7403400640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f73e80b3d40 0x7f73e80b6250 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f73fc005fd0 tx=0x7f73fc010040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.854+0000 7f7410254740 1 -- start start 2024-07-26T12:51:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.854+0000 7f7410254740 1 Processor -- start 2024-07-26T12:51:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f7410254740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8e21b0 0x558a4ecc2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f7410254740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8ea390 0x558a4ecbb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f7410254740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecbbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f7410254740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558a4ea79700 con 0x558a4e8e21b0 2024-07-26T12:51:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f7410254740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x558a4ebb26a0 con 0x558a4e90b070 2024-07-26T12:51:07.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f7410254740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x558a4ecc2b20 con 0x558a4e8ea390 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f7410254740 10 client.0 _ll_register_callbacks cb 0x558a4ebc2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f7402a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8e21b0 0x558a4ecc2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f7402a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8e21b0 0x558a4ecc2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57476/0 (socket says 172.21.15.107:57476) 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f7402a00640 1 -- 172.21.15.107:0/400705458 learned_addr learned my addr 172.21.15.107:0/400705458 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.855+0000 7f73fbe00640 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8ea390 0x558a4ecbb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f7402a00640 1 -- 172.21.15.107:0/400705458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8ea390 msgr2=0x558a4ecbb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f7402a00640 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8ea390 0x558a4ecbb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f7402a00640 1 -- 172.21.15.107:0/400705458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 msgr2=0x558a4ecbbd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f7402a00640 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecbbd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f7402a00640 1 -- 172.21.15.107:0/400705458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a4ebc9840 con 0x558a4e8e21b0 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f73fbe00640 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8ea390 0x558a4ecbb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:07.877 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f7402a00640 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8e21b0 0x558a4ecc2160 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f73f40c6800 tx=0x7f73f4004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1ff0b8196b7457 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f73e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f73e7e00640 1 -- 172.21.15.107:0/400705458 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f40d3170 con 0x558a4e8e21b0 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f73e7e00640 1 -- 172.21.15.107:0/400705458 <== mon.0 v2:172.21.15.107:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f73f40d33a0 con 0x558a4e8e21b0 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.856+0000 7f7410254740 1 -- 172.21.15.107:0/400705458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73ec00fa70 con 0x558a4e8e21b0 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f73e7e00640 1 -- 172.21.15.107:0/400705458 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f73f40ca290 con 0x558a4e8e21b0 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f7410254740 10 client.15789 fetch_fsmap learned FSMap version 35 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f7410254740 1 -- 172.21.15.107:0/400705458 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558a4ecd2ce0 con 0x558a4e8e21b0 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f73e7e00640 1 -- 172.21.15.107:0/400705458 <== mon.0 v2:172.21.15.107:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73ec00fa70 con 0x558a4e8e21b0 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f73e7e00640 1 -- 172.21.15.107:0/400705458 <== mon.0 v2:172.21.15.107:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7f73f40c60f0 con 0x558a4e8e21b0 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f7410254740 10 client.15789 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f7410254740 10 client.15789 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-07-26T12:51:07.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f7410254740 20 client.15789 populate_metadata read hostname 'smithi107' 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f7410254740 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558a4e8c99f0 0x558a4ecd60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f7410254740 4 client.15789 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.857+0000 7f7410254740 1 -- 172.21.15.107:0/400705458 --> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a4ec28b20 con 0x558a4e8c99f0 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.858+0000 7f73fbe00640 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558a4e8c99f0 0x558a4ecd60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.858+0000 7f73e7e00640 10 client.15789 ms_handle_connect on v2:172.21.15.107:6838/137526780 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.858+0000 7f73fbe00640 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558a4e8c99f0 0x558a4ecd60b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a46a3ee664e89652 server_cookie=e84dc1ecf0ba15ee in_seq=0 out_seq=0 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.859+0000 7f73e7e00640 1 -- 172.21.15.107:0/400705458 <== mds.1 v2:172.21.15.107:6838/137526780 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558a4ec28b20 con 0x558a4e8c99f0 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.859+0000 7f73e7e00640 10 client.15789 handle_command_reply: tid=0 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.859+0000 7f7410254740 1 client.15789 shutdown 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.859+0000 7f7410254740 1 -- 172.21.15.107:0/400705458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8e21b0 msgr2=0x558a4ecc2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.859+0000 7f7410254740 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8e21b0 0x558a4ecc2160 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f73f40c6800 tx=0x7f73f4004610 comp rx=0 tx=0).stop 2024-07-26T12:51:07.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.860+0000 7f7410254740 1 -- 172.21.15.107:0/400705458 shutdown_connections 2024-07-26T12:51:07.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.860+0000 7f7410254740 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558a4e8c99f0 0x558a4ecd60b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.860+0000 7f7410254740 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecbbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.860+0000 7f7410254740 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8ea390 0x558a4ecbb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.860+0000 7f73e7e00640 0 client.15789 ms_handle_reset on v2:172.21.15.107:6838/137526780 2024-07-26T12:51:07.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.860+0000 7f7410254740 1 --2- 172.21.15.107:0/400705458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8e21b0 0x558a4ecc2160 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.860+0000 7f7410254740 1 -- 172.21.15.107:0/400705458 >> 172.21.15.107:0/400705458 conn(0x558a4e8ff460 msgr2=0x558a4ecb3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:07.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.860+0000 7f7410254740 1 -- 172.21.15.107:0/400705458 shutdown_connections 2024-07-26T12:51:07.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.860+0000 7f7410254740 1 -- 172.21.15.107:0/400705458 wait complete. 2024-07-26T12:51:07.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.860+0000 7f7410254740 20 client.15789 trim_cache size 0 max 16384 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.875+0000 7f7410254740 1 Processor -- start 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.875+0000 7f7410254740 1 -- start start 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.875+0000 7f7410254740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4ecb37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.875+0000 7f7410254740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 0x558a4ecdb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.875+0000 7f7410254740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecdbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.875+0000 7f7410254740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f74041ae4a0 con 0x558a4e8ea390 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.875+0000 7f7410254740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f74041b00a0 con 0x558a4e90b070 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.875+0000 7f7410254740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f74041aeb70 con 0x558a4e8e21b0 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f7402a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4ecb37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f7402a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4ecb37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:47682/0 (socket says 172.21.15.107:47682) 2024-07-26T12:51:07.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f7402a00640 1 -- 172.21.15.107:0/303536367 learned_addr learned my addr 172.21.15.107:0/303536367 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f73fbe00640 1 --2- 172.21.15.107:0/303536367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 0x558a4ecdb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f7403400640 1 --2- 172.21.15.107:0/303536367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecdbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f7402a00640 1 -- 172.21.15.107:0/303536367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 msgr2=0x558a4ecdbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f7402a00640 1 --2- 172.21.15.107:0/303536367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecdbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f7402a00640 1 -- 172.21.15.107:0/303536367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 msgr2=0x558a4ecdb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f7402a00640 1 --2- 172.21.15.107:0/303536367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 0x558a4ecdb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f7402a00640 1 -- 172.21.15.107:0/303536367 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a4ecb1f50 con 0x558a4e8e21b0 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f73fbe00640 1 --2- 172.21.15.107:0/303536367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 0x558a4ecdb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.876+0000 7f7402a00640 1 --2- 172.21.15.107:0/303536367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4ecb37c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f73f402a860 tx=0x7f73f4101770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e62d6dcfc85e5c3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.877+0000 7f73f2a00640 1 -- 172.21.15.107:0/303536367 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f40ca0d0 con 0x558a4e8e21b0 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.877+0000 7f73f2a00640 1 -- 172.21.15.107:0/303536367 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73f40d3170 con 0x558a4e8e21b0 2024-07-26T12:51:07.885 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.877+0000 7f73f2a00640 1 -- 172.21.15.107:0/303536367 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f40ca780 con 0x558a4e8e21b0 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.878+0000 7f7410254740 1 -- 172.21.15.107:0/303536367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 msgr2=0x558a4ecb37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.878+0000 7f7410254740 1 --2- 172.21.15.107:0/303536367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4ecb37c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f73f402a860 tx=0x7f73f4101770 comp rx=0 tx=0).stop 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.878+0000 7f7410254740 1 -- 172.21.15.107:0/303536367 shutdown_connections 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.878+0000 7f7410254740 1 --2- 172.21.15.107:0/303536367 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecdbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.878+0000 7f7410254740 1 --2- 172.21.15.107:0/303536367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 0x558a4ecdb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.878+0000 7f7410254740 1 --2- 172.21.15.107:0/303536367 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4ecb37c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.878+0000 7f7410254740 1 -- 172.21.15.107:0/303536367 >> 172.21.15.107:0/303536367 conn(0x558a4e8ff460 msgr2=0x558a4ecb30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.878+0000 7f7410254740 1 -- 172.21.15.107:0/303536367 shutdown_connections 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.878+0000 7f7410254740 1 -- 172.21.15.107:0/303536367 wait complete. 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.878+0000 7f7410254740 1 -- start start 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.879+0000 7f7410254740 1 Processor -- start 2024-07-26T12:51:07.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.879+0000 7f7410254740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4eccc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.879+0000 7f7410254740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 0x558a4eccc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.879+0000 7f7410254740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecd05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.879+0000 7f7410254740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f74041af9d0 con 0x558a4e8ea390 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.879+0000 7f7410254740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f74041af240 con 0x558a4e90b070 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.879+0000 7f7410254740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f74041ac480 con 0x558a4e8e21b0 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.879+0000 7f7410254740 10 client.0 _ll_register_callbacks cb 0x7f73f4030680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.879+0000 7f7402a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4eccc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.879+0000 7f73fbe00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 0x558a4eccc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f7402a00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4eccc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:47694/0 (socket says 172.21.15.107:47694) 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f7402a00640 1 -- 172.21.15.107:0/3975664224 learned_addr learned my addr 172.21.15.107:0/3975664224 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f7403400640 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecd05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f7402a00640 1 -- 172.21.15.107:0/3975664224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 msgr2=0x558a4ecd05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f7402a00640 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecd05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f7402a00640 1 -- 172.21.15.107:0/3975664224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 msgr2=0x558a4eccc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f7402a00640 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 0x558a4eccc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f7402a00640 1 -- 172.21.15.107:0/3975664224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a4ecd0c10 con 0x558a4e8e21b0 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f73fbe00640 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 0x558a4eccc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f7402a00640 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4eccc0f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f73f4000c00 tx=0x7f73f4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c38d04552ffe18a5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.880+0000 7f73e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f73e7e00640 1 -- 172.21.15.107:0/3975664224 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f40a8030 con 0x558a4e8e21b0 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f73e7e00640 1 -- 172.21.15.107:0/3975664224 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7f73f40c6d70 con 0x558a4e8e21b0 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f7410254740 1 -- 172.21.15.107:0/3975664224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558a4ec245c0 con 0x558a4e8e21b0 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f73e7e00640 1 -- 172.21.15.107:0/3975664224 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f73f40ced80 con 0x558a4e8e21b0 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f7410254740 10 client.25930 fetch_fsmap learned FSMap version 35 2024-07-26T12:51:07.888 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f7410254740 1 -- 172.21.15.107:0/3975664224 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558a4ecd0c10 con 0x558a4e8e21b0 2024-07-26T12:51:07.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f73e7e00640 1 -- 172.21.15.107:0/3975664224 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558a4ec245c0 con 0x558a4e8e21b0 2024-07-26T12:51:07.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f73e7e00640 1 -- 172.21.15.107:0/3975664224 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7f73f40c94e0 con 0x558a4e8e21b0 2024-07-26T12:51:07.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f7410254740 10 client.25930 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:51:07.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f7410254740 10 client.25930 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-07-26T12:51:07.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.881+0000 7f7410254740 20 client.25930 populate_metadata read hostname 'smithi107' 2024-07-26T12:51:07.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.882+0000 7f7410254740 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558a4e89eda0 0x558a4ed405c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:07.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.882+0000 7f7410254740 4 client.25930 mds_command: new command op to 24715 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T12:51:07.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.882+0000 7f7410254740 1 -- 172.21.15.107:0/3975664224 --> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a4ec21d00 con 0x558a4e89eda0 2024-07-26T12:51:07.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.882+0000 7f73fbe00640 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558a4e89eda0 0x558a4ed405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:07.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.882+0000 7f73fbe00640 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558a4e89eda0 0x558a4ed405c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f0903b54d8116e77 server_cookie=f25fabcd04acb2 in_seq=0 out_seq=0 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.882+0000 7f73e7e00640 10 client.25930 ms_handle_connect on v2:172.21.15.107:6838/137526780 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f73e7e00640 1 -- 172.21.15.107:0/3975664224 <== mds.1 v2:172.21.15.107:6838/137526780 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a4ec21d00 con 0x558a4e89eda0 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f73e7e00640 10 client.25930 handle_command_reply: tid=0 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 client.25930 shutdown 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 -- 172.21.15.107:0/3975664224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 msgr2=0x558a4eccc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4eccc0f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f73f4000c00 tx=0x7f73f4005f90 comp rx=0 tx=0).stop 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 -- 172.21.15.107:0/3975664224 shutdown_connections 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.107:6838/137526780,v1:172.21.15.107:6839/137526780] conn(0x558a4e89eda0 0x558a4ed405c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x558a4e90b070 0x558a4ecd05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a4e8ea390 0x558a4eccc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f73e7e00640 0 client.25930 ms_handle_reset on v2:172.21.15.107:6838/137526780 2024-07-26T12:51:07.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 --2- 172.21.15.107:0/3975664224 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x558a4e8e21b0 0x558a4eccc0f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 -- 172.21.15.107:0/3975664224 >> 172.21.15.107:0/3975664224 conn(0x558a4e8ff460 msgr2=0x558a4ecbcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:07.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 -- 172.21.15.107:0/3975664224 shutdown_connections 2024-07-26T12:51:07.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.883+0000 7f7410254740 1 -- 172.21.15.107:0/3975664224 wait complete. 2024-07-26T12:51:07.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7410254740 20 client.25930 trim_cache size 0 max 16384 2024-07-26T12:51:07.892 INFO:teuthology.orchestra.run.smithi107.stdout:[] 2024-07-26T12:51:07.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7403e00640 1 -- 172.21.15.107:0/4250990210 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f73e80b3d40 msgr2=0x7f73e80b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:07.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7403e00640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f73e80b3d40 0x7f73e80b6250 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f73fc005fd0 tx=0x7f73fc010040 comp rx=0 tx=0).stop 2024-07-26T12:51:07.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7403e00640 1 -- 172.21.15.107:0/4250990210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7404101810 msgr2=0x7f7404199de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:07.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7403e00640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7404101810 0x7f7404199de0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f73f4035040 tx=0x7f73f4032d50 comp rx=0 tx=0).stop 2024-07-26T12:51:07.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7403e00640 1 -- 172.21.15.107:0/4250990210 shutdown_connections 2024-07-26T12:51:07.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7403e00640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f73e80b3d40 0x7f73e80b6250 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7403e00640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7404106440 0x7f740419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7403e00640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f7404105af0 0x7f740419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7403e00640 1 --2- 172.21.15.107:0/4250990210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7404101810 0x7f7404199de0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:07.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.884+0000 7f7403e00640 1 -- 172.21.15.107:0/4250990210 >> 172.21.15.107:0/4250990210 conn(0x7f74040774e0 msgr2=0x7f7404106f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:07.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.885+0000 7f7403e00640 1 -- 172.21.15.107:0/4250990210 shutdown_connections 2024-07-26T12:51:07.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:07.885+0000 7f7403e00640 1 -- 172.21.15.107:0/4250990210 wait complete. 2024-07-26T12:51:08.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 14de5e40-4b4b-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T12:51:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:08 smithi107 ceph-mon[32659]: pgmap v546: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 614 KiB/s wr, 152 op/s 2024-07-26T12:51:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:08 smithi132 ceph-mon[37580]: pgmap v546: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 614 KiB/s wr, 152 op/s 2024-07-26T12:51:09.312 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/14de5e40-4b4b-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T12:51:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:08 smithi112 ceph-mon[37501]: pgmap v546: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 614 KiB/s wr, 152 op/s 2024-07-26T12:51:11.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:10 smithi107 ceph-mon[32659]: pgmap v547: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 340 KiB/s wr, 119 op/s 2024-07-26T12:51:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:10 smithi132 ceph-mon[37580]: pgmap v547: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 340 KiB/s wr, 119 op/s 2024-07-26T12:51:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:10 smithi112 ceph-mon[37501]: pgmap v547: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 340 KiB/s wr, 119 op/s 2024-07-26T12:51:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.624+0000 7ff10a200640 1 -- 172.21.15.107:0/1684473845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff104102930 msgr2=0x7ff104106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.624+0000 7ff10a200640 1 --2- 172.21.15.107:0/1684473845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff104102930 0x7ff104106e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009860 tx=0x7ff0f002a340 comp rx=0 tx=0).stop 2024-07-26T12:51:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.624+0000 7ff10a200640 1 -- 172.21.15.107:0/1684473845 shutdown_connections 2024-07-26T12:51:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.624+0000 7ff10a200640 1 --2- 172.21.15.107:0/1684473845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff104107420 0x7ff104107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.624+0000 7ff10a200640 1 --2- 172.21.15.107:0/1684473845 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff104102930 0x7ff104106e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.624+0000 7ff10a200640 1 --2- 172.21.15.107:0/1684473845 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff104101fe0 0x7ff1041023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.624+0000 7ff10a200640 1 -- 172.21.15.107:0/1684473845 >> 172.21.15.107:0/1684473845 conn(0x7ff1040fdcf0 msgr2=0x7ff104100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.624+0000 7ff10a200640 1 -- 172.21.15.107:0/1684473845 shutdown_connections 2024-07-26T12:51:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.624+0000 7ff10a200640 1 -- 172.21.15.107:0/1684473845 wait complete. 2024-07-26T12:51:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.625+0000 7ff10a200640 1 Processor -- start 2024-07-26T12:51:11.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.625+0000 7ff10a200640 1 -- start start 2024-07-26T12:51:11.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff10a200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff104101fe0 0x7ff1041a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff10a200640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff104102930 0x7ff1041a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff10a200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff104107420 0x7ff1041a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff10a200640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1041198c0 con 0x7ff104101fe0 2024-07-26T12:51:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff10a200640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7ff104119740 con 0x7ff104102930 2024-07-26T12:51:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff10a200640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7ff104119a40 con 0x7ff104107420 2024-07-26T12:51:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff108e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff104101fe0 0x7ff1041a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff103e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff104102930 0x7ff1041a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff109800640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff104107420 0x7ff1041a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff103e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff104102930 0x7ff1041a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44098/0 (socket says 172.21.15.107:44098) 2024-07-26T12:51:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.626+0000 7ff103e00640 1 -- 172.21.15.107:0/1770124043 learned_addr learned my addr 172.21.15.107:0/1770124043 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.627+0000 7ff103e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff104107420 msgr2=0x7ff1041a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.627+0000 7ff103e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff104107420 0x7ff1041a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.627+0000 7ff103e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff104101fe0 msgr2=0x7ff1041a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.627+0000 7ff103e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff104101fe0 0x7ff1041a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.627+0000 7ff103e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1041a6280 con 0x7ff104102930 2024-07-26T12:51:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.627+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff104101fe0 0x7ff1041a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.627+0000 7ff103e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7ff104102930 0x7ff1041a08b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0005940 tx=0x7ff0f0032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.627+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0040020 con 0x7ff104102930 2024-07-26T12:51:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.627+0000 7ff10a200640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1041a6510 con 0x7ff104102930 2024-07-26T12:51:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.627+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0f0030d40 con 0x7ff104102930 2024-07-26T12:51:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.628+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0037040 con 0x7ff104102930 2024-07-26T12:51:11.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.628+0000 7ff10a200640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1041accd0 con 0x7ff104102930 2024-07-26T12:51:11.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.631+0000 7ff11240d740 1 Processor -- start 2024-07-26T12:51:11.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.631+0000 7ff11240d740 1 -- start start 2024-07-26T12:51:11.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.631+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7ff0f0043020 con 0x7ff104102930 2024-07-26T12:51:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.631+0000 7ff11240d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654219b4070 0x565421d59010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.631+0000 7ff11240d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56542198b1b0 0x565421d59550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.631+0000 7ff11240d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565421993390 0x565421d59a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.631+0000 7ff11240d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x565421ccfbd0 con 0x565421993390 2024-07-26T12:51:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.631+0000 7ff11240d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x565421cbf0d0 con 0x5654219b4070 2024-07-26T12:51:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.631+0000 7ff11240d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x565421cc3110 con 0x56542198b1b0 2024-07-26T12:51:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.632+0000 7ff109800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654219b4070 0x565421d59010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.632+0000 7ff109800640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654219b4070 0x565421d59010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44110/0 (socket says 172.21.15.107:44110) 2024-07-26T12:51:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.632+0000 7ff109800640 1 -- 172.21.15.107:0/442797836 learned_addr learned my addr 172.21.15.107:0/442797836 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.632+0000 7ff108e00640 1 --2- 172.21.15.107:0/442797836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56542198b1b0 0x565421d59550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.632+0000 7ff103e00640 1 --2- 172.21.15.107:0/442797836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565421993390 0x565421d59a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.632+0000 7ff109800640 1 -- 172.21.15.107:0/442797836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56542198b1b0 msgr2=0x565421d59550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.632+0000 7ff109800640 1 --2- 172.21.15.107:0/442797836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56542198b1b0 0x565421d59550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.632+0000 7ff109800640 1 -- 172.21.15.107:0/442797836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565421993390 msgr2=0x565421d59a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.632+0000 7ff109800640 1 --2- 172.21.15.107:0/442797836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565421993390 0x565421d59a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.632+0000 7ff109800640 1 -- 172.21.15.107:0/442797836 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565421a1f650 con 0x5654219b4070 2024-07-26T12:51:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.633+0000 7ff108e00640 1 --2- 172.21.15.107:0/442797836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56542198b1b0 0x565421d59550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.633+0000 7ff109800640 1 --2- 172.21.15.107:0/442797836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654219b4070 0x565421d59010 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009fd0 tx=0x7ff0fc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fd9bb335fbc3c2e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.633+0000 7ff0f6000640 1 -- 172.21.15.107:0/442797836 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc004d20 con 0x5654219b4070 2024-07-26T12:51:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.633+0000 7ff0f6000640 1 -- 172.21.15.107:0/442797836 <== mon.2 v2:172.21.15.112:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0fc004050 con 0x5654219b4070 2024-07-26T12:51:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.633+0000 7ff0f6000640 1 -- 172.21.15.107:0/442797836 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc005260 con 0x5654219b4070 2024-07-26T12:51:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.634+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.634+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7ff0f0103f40 con 0x7ff104102930 2024-07-26T12:51:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.634+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.634+0000 7ff11240d740 1 -- 172.21.15.107:0/442797836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654219b4070 msgr2=0x565421d59010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.634+0000 7ff11240d740 1 --2- 172.21.15.107:0/442797836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654219b4070 0x565421d59010 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc009fd0 tx=0x7ff0fc0118d0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.634+0000 7ff11240d740 1 -- 172.21.15.107:0/442797836 shutdown_connections 2024-07-26T12:51:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.634+0000 7ff11240d740 1 --2- 172.21.15.107:0/442797836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565421993390 0x565421d59a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.634+0000 7ff11240d740 1 --2- 172.21.15.107:0/442797836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x56542198b1b0 0x565421d59550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.634+0000 7ff11240d740 1 --2- 172.21.15.107:0/442797836 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x5654219b4070 0x565421d59010 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.634+0000 7ff11240d740 1 -- 172.21.15.107:0/442797836 >> 172.21.15.107:0/442797836 conn(0x5654219a8460 msgr2=0x565421d56f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:11.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.635+0000 7ff11240d740 1 -- 172.21.15.107:0/442797836 shutdown_connections 2024-07-26T12:51:11.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.635+0000 7ff11240d740 1 -- 172.21.15.107:0/442797836 wait complete. 2024-07-26T12:51:11.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.635+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0b6250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x565421d57c90 tx=0x7ff0f8009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:11.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.635+0000 7ff11240d740 1 -- start start 2024-07-26T12:51:11.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.636+0000 7ff11240d740 1 Processor -- start 2024-07-26T12:51:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.636+0000 7ff11240d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56542198b1b0 0x565421d64520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.636+0000 7ff11240d740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x565421993390 0x565421d64a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.636+0000 7ff11240d740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654219b4070 0x565421d64fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.636+0000 7ff11240d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x565421b22700 con 0x56542198b1b0 2024-07-26T12:51:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.636+0000 7ff11240d740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x565421c5b6a0 con 0x565421993390 2024-07-26T12:51:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.636+0000 7ff11240d740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x565421d677f0 con 0x5654219b4070 2024-07-26T12:51:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.636+0000 7ff11240d740 10 client.0 _ll_register_callbacks cb 0x565421c6b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:51:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.636+0000 7ff108e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56542198b1b0 0x565421d64520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.636+0000 7ff103e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x565421993390 0x565421d64a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff103e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x565421993390 0x565421d64a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.112:3300/0 says I am v2:172.21.15.107:44122/0 (socket says 172.21.15.107:44122) 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff103e00640 1 -- 172.21.15.107:0/4208142334 learned_addr learned my addr 172.21.15.107:0/4208142334 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56542198b1b0 0x565421d64520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57532/0 (socket says 172.21.15.107:57532) 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff109800640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654219b4070 0x565421d64fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff103e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654219b4070 msgr2=0x565421d64fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff103e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x5654219b4070 0x565421d64fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff103e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56542198b1b0 msgr2=0x565421d64520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff103e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56542198b1b0 0x565421d64520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff103e00640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565421c72840 con 0x565421993390 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff103e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x565421993390 0x565421d64a60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00c7b50 tx=0x7ff0f0107680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6bccbf4cef4d0fd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:11.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff0eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.112:3300/0 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.637+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f00c67a0 con 0x565421993390 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7ff0f0106080 con 0x565421993390 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff11240d740 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0fc004050 con 0x565421993390 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff0f00ca050 con 0x565421993390 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff11240d740 10 client.25934 fetch_fsmap learned FSMap version 35 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff11240d740 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x565421d73380 con 0x565421993390 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0fc004050 con 0x565421993390 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7ff0f004d060 con 0x565421993390 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff11240d740 10 client.25934 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff11240d740 10 client.25934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff11240d740 20 client.25934 populate_metadata read hostname 'smithi107' 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.638+0000 7ff11240d740 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.639+0000 7ff11240d740 4 client.25934 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.639+0000 7ff11240d740 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565421cd1b20 con 0x5654219729f0 2024-07-26T12:51:11.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.639+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:11.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.639+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:11.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.639+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T12:51:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.841+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.841+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:11.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:11.841+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T12:51:12.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:12.243+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:12.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:12.243+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:12.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:12.243+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T12:51:13.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:13.045+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:13.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:13.045+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:13.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:13.045+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T12:51:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:12 smithi107 ceph-mon[32659]: pgmap v548: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 218 KiB/s wr, 87 op/s 2024-07-26T12:51:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:12 smithi132 ceph-mon[37580]: pgmap v548: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 218 KiB/s wr, 87 op/s 2024-07-26T12:51:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:12 smithi112 ceph-mon[37501]: pgmap v548: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 218 KiB/s wr, 87 op/s 2024-07-26T12:51:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.732+0000 7efc7ca00640 1 -- 172.21.15.107:0/2657649049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc78108c70 msgr2=0x7efc78109080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.732+0000 7efc7ca00640 1 --2- 172.21.15.107:0/2657649049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc78108c70 0x7efc78109080 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efc60009860 tx=0x7efc6002a340 comp rx=0 tx=0).stop 2024-07-26T12:51:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.732+0000 7efc7ca00640 1 -- 172.21.15.107:0/2657649049 shutdown_connections 2024-07-26T12:51:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.732+0000 7efc7ca00640 1 --2- 172.21.15.107:0/2657649049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc781095c0 0x7efc7810fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.732+0000 7efc7ca00640 1 --2- 172.21.15.107:0/2657649049 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc78108c70 0x7efc78109080 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.732+0000 7efc7ca00640 1 --2- 172.21.15.107:0/2657649049 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc78103340 0x7efc78103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.732+0000 7efc7ca00640 1 -- 172.21.15.107:0/2657649049 >> 172.21.15.107:0/2657649049 conn(0x7efc780fed60 msgr2=0x7efc781011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.732+0000 7efc7ca00640 1 -- 172.21.15.107:0/2657649049 shutdown_connections 2024-07-26T12:51:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.732+0000 7efc7ca00640 1 -- 172.21.15.107:0/2657649049 wait complete. 2024-07-26T12:51:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.734+0000 7efc7ca00640 1 Processor -- start 2024-07-26T12:51:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.734+0000 7efc7ca00640 1 -- start start 2024-07-26T12:51:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.734+0000 7efc7ca00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc78103340 0x7efc78199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.734+0000 7efc7ca00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc78108c70 0x7efc7819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.734+0000 7efc7ca00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc781095c0 0x7efc7819f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.734+0000 7efc7ca00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efc78113230 con 0x7efc78103340 2024-07-26T12:51:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.734+0000 7efc7ca00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7efc781130b0 con 0x7efc781095c0 2024-07-26T12:51:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.734+0000 7efc7ca00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7efc781133b0 con 0x7efc78108c70 2024-07-26T12:51:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.735+0000 7efc76800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc78103340 0x7efc78199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.735+0000 7efc76800640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc78103340 0x7efc78199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:57552/0 (socket says 172.21.15.107:57552) 2024-07-26T12:51:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.735+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 learned_addr learned my addr 172.21.15.107:0/3520391650 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.735+0000 7efc6fe00640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc78108c70 0x7efc7819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.735+0000 7efc77200640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc781095c0 0x7efc7819f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.735+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc78108c70 msgr2=0x7efc7819a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.735+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efc78108c70 0x7efc7819a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.735+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc781095c0 msgr2=0x7efc7819f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.735+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7efc781095c0 0x7efc7819f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.735+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7819fc60 con 0x7efc78103340 2024-07-26T12:51:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.736+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc78103340 0x7efc78199d30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efc6800e9f0 tx=0x7efc6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.736+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6800cd80 con 0x7efc78103340 2024-07-26T12:51:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.736+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc680108e0 con 0x7efc78103340 2024-07-26T12:51:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.736+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc68018980 con 0x7efc78103340 2024-07-26T12:51:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.736+0000 7efc7ca00640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc7819ff50 con 0x7efc78103340 2024-07-26T12:51:13.739 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.736+0000 7efc7ca00640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc781a66e0 con 0x7efc78103340 2024-07-26T12:51:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.740+0000 7efc84578740 1 Processor -- start 2024-07-26T12:51:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.740+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7efc68010430 con 0x7efc78103340 2024-07-26T12:51:13.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.740+0000 7efc84578740 1 -- start start 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc84578740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6d71b0 0x55aecda7f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc84578740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aecd6ae2f0 0x55aecda7f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc84578740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aecd6b64d0 0x55aecda7fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc84578740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55aecd9ef270 con 0x55aecd6ae2f0 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc84578740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55aecd9de770 con 0x55aecd6b64d0 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc84578740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55aecd9e27b0 con 0x55aecd6d71b0 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc77200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6d71b0 0x55aecda7f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc77200640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6d71b0 0x55aecda7f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:47754/0 (socket says 172.21.15.107:47754) 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc77200640 1 -- 172.21.15.107:0/2866935844 learned_addr learned my addr 172.21.15.107:0/2866935844 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc76800640 1 --2- 172.21.15.107:0/2866935844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aecd6ae2f0 0x55aecda7f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.741+0000 7efc6fe00640 1 --2- 172.21.15.107:0/2866935844 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aecd6b64d0 0x55aecda7fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:13.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.742+0000 7efc77200640 1 -- 172.21.15.107:0/2866935844 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aecd6b64d0 msgr2=0x55aecda7fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.742+0000 7efc77200640 1 --2- 172.21.15.107:0/2866935844 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aecd6b64d0 0x55aecda7fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.742+0000 7efc77200640 1 -- 172.21.15.107:0/2866935844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aecd6ae2f0 msgr2=0x55aecda7f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.742+0000 7efc77200640 1 --2- 172.21.15.107:0/2866935844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aecd6ae2f0 0x55aecda7f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.742+0000 7efc77200640 1 -- 172.21.15.107:0/2866935844 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aecd742790 con 0x55aecd6d71b0 2024-07-26T12:51:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.742+0000 7efc6fe00640 1 --2- 172.21.15.107:0/2866935844 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aecd6b64d0 0x55aecda7fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T12:51:13.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.742+0000 7efc77200640 1 --2- 172.21.15.107:0/2866935844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6d71b0 0x55aecda7f030 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efc70009a20 tx=0x7efc700095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=962320395fa722ab server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.742+0000 7efc67400640 1 -- 172.21.15.107:0/2866935844 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7001d070 con 0x55aecd6d71b0 2024-07-26T12:51:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.743+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.743+0000 7efc67400640 1 -- 172.21.15.107:0/2866935844 <== mon.1 v2:172.21.15.132:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc70003e40 con 0x55aecd6d71b0 2024-07-26T12:51:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.743+0000 7efc67400640 1 -- 172.21.15.107:0/2866935844 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc700189e0 con 0x55aecd6d71b0 2024-07-26T12:51:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.743+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7efc680e4690 con 0x7efc78103340 2024-07-26T12:51:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.743+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc84578740 1 -- 172.21.15.107:0/2866935844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6d71b0 msgr2=0x55aecda7f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T12:51:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc84578740 1 --2- 172.21.15.107:0/2866935844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6d71b0 0x55aecda7f030 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efc70009a20 tx=0x7efc700095c0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0b6250 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efc680e1b10 tx=0x7efc680ea550 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc84578740 1 -- 172.21.15.107:0/2866935844 shutdown_connections 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc84578740 1 --2- 172.21.15.107:0/2866935844 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aecd6b64d0 0x55aecda7fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc84578740 1 --2- 172.21.15.107:0/2866935844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aecd6ae2f0 0x55aecda7f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc84578740 1 --2- 172.21.15.107:0/2866935844 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6d71b0 0x55aecda7f030 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc84578740 1 -- 172.21.15.107:0/2866935844 >> 172.21.15.107:0/2866935844 conn(0x55aecd6cb300 msgr2=0x55aecda7cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc84578740 1 -- 172.21.15.107:0/2866935844 shutdown_connections 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc84578740 1 -- 172.21.15.107:0/2866935844 wait complete. 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.744+0000 7efc84578740 1 -- start start 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.745+0000 7efc84578740 1 Processor -- start 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.745+0000 7efc84578740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aecd6ae2f0 0x55aecda8acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.745+0000 7efc84578740 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6b64d0 0x55aecda8b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.745+0000 7efc84578740 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aecd6d71b0 0x55aecda96ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.745+0000 7efc84578740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55aecd918ab0 con 0x55aecd6ae2f0 2024-07-26T12:51:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.745+0000 7efc84578740 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x55aecd97bd50 con 0x55aecd6d71b0 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.745+0000 7efc84578740 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x55aecda8d7b0 con 0x55aecd6b64d0 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.745+0000 7efc84578740 10 client.0 _ll_register_callbacks cb 0x55aecd6eea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc77200640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aecd6ae2f0 0x55aecda8acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc6fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6b64d0 0x55aecda8b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc6fe00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6b64d0 0x55aecda8b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.107:47762/0 (socket says 172.21.15.107:47762) 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc6fe00640 1 -- 172.21.15.107:0/3975622631 learned_addr learned my addr 172.21.15.107:0/3975622631 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aecd6d71b0 0x55aecda96ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc6fe00640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aecd6d71b0 msgr2=0x55aecda96ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc6fe00640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x55aecd6d71b0 0x55aecda96ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc6fe00640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aecd6ae2f0 msgr2=0x55aecda8acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc6fe00640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aecd6ae2f0 0x55aecda8acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:51:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc6fe00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aecd7bb620 con 0x55aecd6b64d0 2024-07-26T12:51:13.751 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55aecd6ae2f0 0x55aecda8acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T12:51:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc6fe00640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x55aecd6b64d0 0x55aecda8b290 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efc6002a930 tx=0x7efc60004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b985ab86876c33de server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:51:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.746+0000 7efc5be00640 10 client.0 ms_handle_connect on v2:172.21.15.132:3300/0 2024-07-26T12:51:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.747+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc60041070 con 0x55aecd6b64d0 2024-07-26T12:51:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.747+0000 7efc84578740 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55aecd9f11c0 con 0x55aecd6b64d0 2024-07-26T12:51:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.747+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9423+0+0 (secure 0 0 0) 0x7efc60030450 con 0x55aecd6b64d0 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.747+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efc6004d050 con 0x55aecd6b64d0 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.747+0000 7efc84578740 10 client.25954 fetch_fsmap learned FSMap version 35 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.747+0000 7efc84578740 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aecda99900 con 0x55aecd6b64d0 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.747+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55aecd9f11c0 con 0x55aecd6b64d0 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.748+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 5 ==== fsmap(e 35) ==== 8386+0+0 (secure 0 0 0) 0x7efc6003d450 con 0x55aecd6b64d0 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.748+0000 7efc84578740 10 client.25954 fetch_fsmap finished waiting for FSMap version 35 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.748+0000 7efc84578740 10 client.25954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.748+0000 7efc84578740 20 client.25954 populate_metadata read hostname 'smithi107' 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.748+0000 7efc84578740 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.748+0000 7efc84578740 4 client.25954 mds_command: new command op to 24716 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.748+0000 7efc84578740 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aecd9ecc60 con 0x55aecd695d80 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.748+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.748+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.748+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T12:51:13.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.949+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:13.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.949+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:13.951 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:13.949+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T12:51:14.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:14.350+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:14.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:14.350+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:14.352 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:14.350+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T12:51:14.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:14.646+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:14.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:14.647+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:14.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:14.647+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-26T12:51:14.714 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:14 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-07-26T12:51:14.714 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:14 smithi112 systemd[1]: Stopped Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:51:14.714 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:14 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Consumed 28.453s CPU time. 2024-07-26T12:51:14.714 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:14 smithi112 systemd[1]: Starting Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:51:14.714 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:14 smithi112 podman[71997]: 2024-07-26 12:51:14.584496529 +0000 UTC m=+0.017970173 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:51:14.714 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:14 smithi112 podman[71997]: 2024-07-26 12:51:14.713277632 +0000 UTC m=+0.146751313 container create fe53c02f25a73b94b98ab6dfa4aa5c150f79b3a18ecd9707d3abe30c8cbab24e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:51:15.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:15.151+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:15.152+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T12:51:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:15.153+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:15.154 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:15.153+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:15.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:15.153+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T12:51:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:14 smithi107 ceph-mon[32659]: pgmap v549: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 23 op/s 2024-07-26T12:51:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:14 smithi132 ceph-mon[37580]: pgmap v549: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 23 op/s 2024-07-26T12:51:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:14 smithi112 ceph-mon[37501]: pgmap v549: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 23 op/s 2024-07-26T12:51:15.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:14 smithi112 podman[71997]: 2024-07-26 12:51:14.966254276 +0000 UTC m=+0.399727916 container init fe53c02f25a73b94b98ab6dfa4aa5c150f79b3a18ecd9707d3abe30c8cbab24e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-07-26T12:51:15.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:14 smithi112 podman[71997]: 2024-07-26 12:51:14.969042874 +0000 UTC m=+0.402516518 container start fe53c02f25a73b94b98ab6dfa4aa5c150f79b3a18ecd9707d3abe30c8cbab24e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=) 2024-07-26T12:51:15.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:15 smithi112 ceph-mds[72016]: -- 172.21.15.112:0/1578979029 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8a03d1860 con 0x55f8a4573600 2024-07-26T12:51:15.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:15 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: starting mds.b at 2024-07-26T12:51:15.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:15 smithi112 bash[71997]: fe53c02f25a73b94b98ab6dfa4aa5c150f79b3a18ecd9707d3abe30c8cbab24e 2024-07-26T12:51:15.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:15 smithi112 systemd[1]: Started Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:51:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:15 smithi107 ceph-mon[32659]: daemon mds.b restarted 2024-07-26T12:51:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:15 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:51:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:51:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:15 smithi107 ceph-mon[32659]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T12:51:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:15 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:boot 2024-07-26T12:51:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:15 smithi107 ceph-mon[32659]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:51:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:15 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:51:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:15 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:15 smithi132 ceph-mon[37580]: daemon mds.b restarted 2024-07-26T12:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:15 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:15 smithi132 ceph-mon[37580]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T12:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:15 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:boot 2024-07-26T12:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:15 smithi132 ceph-mon[37580]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:15 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:15 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:51:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:15 smithi112 ceph-mon[37501]: daemon mds.b restarted 2024-07-26T12:51:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:15 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:51:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:51:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:15 smithi112 ceph-mon[37501]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T12:51:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:15 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:boot 2024-07-26T12:51:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:15 smithi112 ceph-mon[37501]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:51:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:15 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:51:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:15 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:51:16.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:16.755+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:16.756+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T12:51:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:16.756+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:16.756+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:16.757+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-26T12:51:17.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:17 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:resolve 2024-07-26T12:51:17.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:17 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:51:17.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:17 smithi107 ceph-mon[32659]: pgmap v551: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-07-26T12:51:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:17 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:resolve 2024-07-26T12:51:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:17 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:51:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:17 smithi132 ceph-mon[37580]: pgmap v551: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-07-26T12:51:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:17 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:resolve 2024-07-26T12:51:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:17 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:51:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:17 smithi112 ceph-mon[37501]: pgmap v551: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 20 KiB/s wr, 12 op/s 2024-07-26T12:51:17.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:17.851+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:17.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:17.852+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T12:51:17.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:17.852+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:17.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:17.852+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:17.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:17.852+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-26T12:51:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:18 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:51:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:18 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:reconnect 2024-07-26T12:51:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:18 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:51:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:18 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0939975 2024-07-26T12:51:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:18 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:51:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:18 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:reconnect 2024-07-26T12:51:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:18 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:51:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:18 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0939975 2024-07-26T12:51:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:18 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:51:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:18 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:reconnect 2024-07-26T12:51:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:18 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:51:18.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:18 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0939975 2024-07-26T12:51:19.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:rejoin 2024-07-26T12:51:19.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:51:19.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-mon[37501]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:51:19.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-mon[37501]: pgmap v552: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-07-26T12:51:19.352 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:51:19.352 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: *** Caught signal (Aborted) ** 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: in thread 7f90c9386640 thread_name:ms_dispatch 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f90ce75d6f0] 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f90ce7aa94c] 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 3: raise() 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 4: abort() 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 5: /lib64/libc.so.6(+0x2871b) [0x7f90ce74771b] 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 6: /lib64/libc.so.6(+0x37386) [0x7f90ce756386] 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55765688c6b9] 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557656a4a80c] 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557656998cde] 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55765695a763] 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55765695ba1b] 2024-07-26T12:51:19.353 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55765693f752] 2024-07-26T12:51:19.354 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55765688da56] 2024-07-26T12:51:19.354 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557656891637] 2024-07-26T12:51:19.354 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557656891f5e] 2024-07-26T12:51:19.354 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55765687503a] 2024-07-26T12:51:19.354 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f90cf02a407] 2024-07-26T12:51:19.354 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 18: (DispatchQueue::entry()+0x60f) [0x7f90cf02acf9] 2024-07-26T12:51:19.354 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f90cf1075a1] 2024-07-26T12:51:19.355 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 20: (Thread::entry_wrapper()+0x43) [0x7f90ceeb4cdd] 2024-07-26T12:51:19.355 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 21: (Thread::_entry_func(void*)+0xd) [0x7f90ceeb4cf9] 2024-07-26T12:51:19.355 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 22: /lib64/libc.so.6(+0x89c02) [0x7f90ce7a8c02] 2024-07-26T12:51:19.355 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f90ce82dc40] 2024-07-26T12:51:19.355 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2024-07-26T12:51:19.082+0000 7f90c9386640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:19.355 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: in thread 7f90c9386640 thread_name:ms_dispatch 2024-07-26T12:51:19.355 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f90ce75d6f0] 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f90ce7aa94c] 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 3: raise() 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 4: abort() 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 5: /lib64/libc.so.6(+0x2871b) [0x7f90ce74771b] 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 6: /lib64/libc.so.6(+0x37386) [0x7f90ce756386] 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55765688c6b9] 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557656a4a80c] 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557656998cde] 2024-07-26T12:51:19.356 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55765695a763] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55765695ba1b] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55765693f752] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55765688da56] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557656891637] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557656891f5e] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55765687503a] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f90cf02a407] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 18: (DispatchQueue::entry()+0x60f) [0x7f90cf02acf9] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f90cf1075a1] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 20: (Thread::entry_wrapper()+0x43) [0x7f90ceeb4cdd] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 21: (Thread::_entry_func(void*)+0xd) [0x7f90ceeb4cf9] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 22: /lib64/libc.so.6(+0x89c02) [0x7f90ce7a8c02] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f90ce82dc40] 2024-07-26T12:51:19.357 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 0> 2024-07-26T12:51:19.082+0000 7f90c9386640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: in thread 7f90c9386640 thread_name:ms_dispatch 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f90ce75d6f0] 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f90ce7aa94c] 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 3: raise() 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 4: abort() 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 5: /lib64/libc.so.6(+0x2871b) [0x7f90ce74771b] 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 6: /lib64/libc.so.6(+0x37386) [0x7f90ce756386] 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55765688c6b9] 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557656a4a80c] 2024-07-26T12:51:19.358 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557656998cde] 2024-07-26T12:51:19.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55765695a763] 2024-07-26T12:51:19.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55765695ba1b] 2024-07-26T12:51:19.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55765693f752] 2024-07-26T12:51:19.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55765688da56] 2024-07-26T12:51:19.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557656891637] 2024-07-26T12:51:19.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557656891f5e] 2024-07-26T12:51:19.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55765687503a] 2024-07-26T12:51:19.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f90cf02a407] 2024-07-26T12:51:19.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 18: (DispatchQueue::entry()+0x60f) [0x7f90cf02acf9] 2024-07-26T12:51:19.359 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f90cf1075a1] 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 20: (Thread::entry_wrapper()+0x43) [0x7f90ceeb4cdd] 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 21: (Thread::_entry_func(void*)+0xd) [0x7f90ceeb4cf9] 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 22: /lib64/libc.so.6(+0x89c02) [0x7f90ce7a8c02] 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f90ce82dc40] 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 0> 2024-07-26T12:51:19.082+0000 7f90c9386640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: in thread 7f90c9386640 thread_name:ms_dispatch 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:19.360 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f90ce75d6f0] 2024-07-26T12:51:19.361 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f90ce7aa94c] 2024-07-26T12:51:19.361 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 3: raise() 2024-07-26T12:51:19.361 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 4: abort() 2024-07-26T12:51:19.361 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 5: /lib64/libc.so.6(+0x2871b) [0x7f90ce74771b] 2024-07-26T12:51:19.361 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 6: /lib64/libc.so.6(+0x37386) [0x7f90ce756386] 2024-07-26T12:51:19.361 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55765688c6b9] 2024-07-26T12:51:19.361 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557656a4a80c] 2024-07-26T12:51:19.361 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x557656998cde] 2024-07-26T12:51:19.361 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55765695a763] 2024-07-26T12:51:19.362 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55765695ba1b] 2024-07-26T12:51:19.362 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55765693f752] 2024-07-26T12:51:19.362 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55765688da56] 2024-07-26T12:51:19.362 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557656891637] 2024-07-26T12:51:19.362 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557656891f5e] 2024-07-26T12:51:19.362 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55765687503a] 2024-07-26T12:51:19.362 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f90cf02a407] 2024-07-26T12:51:19.362 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 18: (DispatchQueue::entry()+0x60f) [0x7f90cf02acf9] 2024-07-26T12:51:19.363 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f90cf1075a1] 2024-07-26T12:51:19.363 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 20: (Thread::entry_wrapper()+0x43) [0x7f90ceeb4cdd] 2024-07-26T12:51:19.363 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 21: (Thread::_entry_func(void*)+0xd) [0x7f90ceeb4cf9] 2024-07-26T12:51:19.363 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 22: /lib64/libc.so.6(+0x89c02) [0x7f90ce7a8c02] 2024-07-26T12:51:19.363 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f90ce82dc40] 2024-07-26T12:51:19.363 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:19.363 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[71823]: 2024-07-26T12:51:19.363 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 podman[72051]: 2024-07-26 12:51:19.143878727 +0000 UTC m=+0.021774731 container died 822dce7256d42518a49e8b68bed60df239d35cccf42005bb09287bf302bb8891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-07-26T12:51:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:19 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:rejoin 2024-07-26T12:51:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:19 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:51:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:19 smithi107 ceph-mon[32659]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:51:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:19 smithi107 ceph-mon[32659]: pgmap v552: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-07-26T12:51:19.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:19 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:rejoin 2024-07-26T12:51:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:19 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:51:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:19 smithi132 ceph-mon[37580]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:51:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:19 smithi132 ceph-mon[37580]: pgmap v552: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-07-26T12:51:19.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:19.961+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:19.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:19.962+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T12:51:19.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:19.962+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:19.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:19.962+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:19.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:19.962+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-26T12:51:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:20 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:51:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:20 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:51:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:20 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:active 2024-07-26T12:51:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:20 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:51:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:20 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:51:20.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 podman[72051]: 2024-07-26 12:51:19.888206427 +0000 UTC m=+0.766102432 container remove 822dce7256d42518a49e8b68bed60df239d35cccf42005bb09287bf302bb8891 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-07-26T12:51:20.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:19 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:51:20.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:20 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T12:51:20.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:20 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Consumed 1.933s CPU time. 2024-07-26T12:51:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:20 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:51:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:20 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:51:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:20 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:active 2024-07-26T12:51:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:20 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:51:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:20 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:51:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:20 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:51:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:20 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:51:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:20 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/540940561,v1:172.21.15.112:6837/540940561] up:active 2024-07-26T12:51:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:20 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:51:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:20 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:51:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:21 smithi112 ceph-mon[37501]: pgmap v553: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T12:51:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:21 smithi107 ceph-mon[32659]: pgmap v553: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T12:51:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:21 smithi132 ceph-mon[37580]: pgmap v553: 129 pgs: 129 active+clean; 37 MiB data, 606 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T12:51:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:22 smithi107 ceph-mon[32659]: pgmap v554: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-07-26T12:51:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:22 smithi132 ceph-mon[37580]: pgmap v554: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-07-26T12:51:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:22 smithi112 ceph-mon[37501]: pgmap v554: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 23 KiB/s wr, 18 op/s 2024-07-26T12:51:24.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:24.259+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:24.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:24.261+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T12:51:24.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:24.261+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:24.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:24.261+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:24.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:24.261+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-26T12:51:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:51:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:51:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:51:26.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:26.369+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:26.370+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T12:51:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:26.370+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:26.370+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:26.370+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-26T12:51:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:26 smithi112 ceph-mon[37501]: pgmap v555: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 39 KiB/s wr, 21 op/s 2024-07-26T12:51:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:26 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T12:51:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:26 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T12:51:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:26 smithi107 ceph-mon[32659]: pgmap v555: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 39 KiB/s wr, 21 op/s 2024-07-26T12:51:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:26 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T12:51:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:26 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T12:51:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:26 smithi132 ceph-mon[37580]: pgmap v555: 129 pgs: 129 active+clean; 37 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 39 KiB/s wr, 21 op/s 2024-07-26T12:51:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:26 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T12:51:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:26 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T12:51:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:28 smithi107 ceph-mon[32659]: pgmap v556: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-07-26T12:51:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:28 smithi132 ceph-mon[37580]: pgmap v556: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-07-26T12:51:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:28 smithi112 ceph-mon[37501]: pgmap v556: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-07-26T12:51:30.456 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-07-26T12:51:30.456 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 systemd[1]: Stopped Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:51:30.456 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Consumed 1.933s CPU time. 2024-07-26T12:51:30.456 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 systemd[1]: Starting Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:51:30.456 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 podman[72275]: 2024-07-26 12:51:30.33907592 +0000 UTC m=+0.017702274 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:51:30.456 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 podman[72275]: 2024-07-26 12:51:30.455896975 +0000 UTC m=+0.134523321 container create 1d2b226c9861768e089f904dcca81fe68c4193cb37500ef3f5bba65323a28e50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, ceph=True) 2024-07-26T12:51:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:30 smithi107 ceph-mon[32659]: pgmap v557: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 46 KiB/s wr, 23 op/s 2024-07-26T12:51:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:30 smithi132 ceph-mon[37580]: pgmap v557: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 46 KiB/s wr, 23 op/s 2024-07-26T12:51:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:30 smithi112 ceph-mon[37501]: pgmap v557: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 46 KiB/s wr, 23 op/s 2024-07-26T12:51:31.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 podman[72275]: 2024-07-26 12:51:30.708366222 +0000 UTC m=+0.386992581 container init 1d2b226c9861768e089f904dcca81fe68c4193cb37500ef3f5bba65323a28e50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:51:31.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 podman[72275]: 2024-07-26 12:51:30.711128235 +0000 UTC m=+0.389754584 container start 1d2b226c9861768e089f904dcca81fe68c4193cb37500ef3f5bba65323a28e50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-07-26T12:51:31.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 ceph-mds[72294]: -- 172.21.15.112:0/2239998314 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bf84d21860 con 0x55bf88ec3600 2024-07-26T12:51:31.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: starting mds.h at 2024-07-26T12:51:31.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 bash[72275]: 1d2b226c9861768e089f904dcca81fe68c4193cb37500ef3f5bba65323a28e50 2024-07-26T12:51:31.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:30 smithi112 systemd[1]: Started Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:51:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:31 smithi107 ceph-mon[32659]: daemon mds.h restarted 2024-07-26T12:51:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:31 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:51:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:51:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:31 smithi107 ceph-mon[32659]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T12:51:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:31 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:boot 2024-07-26T12:51:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:31 smithi107 ceph-mon[32659]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:51:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:31 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:51:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:31 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:51:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:31 smithi132 ceph-mon[37580]: daemon mds.h restarted 2024-07-26T12:51:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:31 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:51:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:51:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:31 smithi132 ceph-mon[37580]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T12:51:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:31 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:boot 2024-07-26T12:51:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:31 smithi132 ceph-mon[37580]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:51:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:31 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:51:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:31 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:51:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:31 smithi112 ceph-mon[37501]: daemon mds.h restarted 2024-07-26T12:51:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:31 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:51:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:51:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:31 smithi112 ceph-mon[37501]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T12:51:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:31 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:boot 2024-07-26T12:51:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:31 smithi112 ceph-mon[37501]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:51:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:31 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:51:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:31 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:51:33.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:32 smithi107 ceph-mon[32659]: pgmap v559: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 58 KiB/s wr, 24 op/s 2024-07-26T12:51:33.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:32 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:resolve 2024-07-26T12:51:33.037 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:32 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:51:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:32 smithi132 ceph-mon[37580]: pgmap v559: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 58 KiB/s wr, 24 op/s 2024-07-26T12:51:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:32 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:resolve 2024-07-26T12:51:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:32 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:51:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:32 smithi112 ceph-mon[37501]: pgmap v559: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 58 KiB/s wr, 24 op/s 2024-07-26T12:51:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:32 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:resolve 2024-07-26T12:51:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:32 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:51:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:33 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.000999974 2024-07-26T12:51:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:33 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:reconnect 2024-07-26T12:51:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:33 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:51:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:33 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00899976 2024-07-26T12:51:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:33 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.000999974 2024-07-26T12:51:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:33 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:reconnect 2024-07-26T12:51:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:33 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:51:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:33 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00899976 2024-07-26T12:51:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:33 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.000999974 2024-07-26T12:51:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:33 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:reconnect 2024-07-26T12:51:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:33 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:51:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:33 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00899976 2024-07-26T12:51:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-mon[37501]: pgmap v560: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 35 KiB/s wr, 16 op/s 2024-07-26T12:51:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:rejoin 2024-07-26T12:51:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:51:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-mon[37501]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:51:35.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:51:35.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: *** Caught signal (Aborted) ** 2024-07-26T12:51:35.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: in thread 7fa7f61e1640 thread_name:ms_dispatch 2024-07-26T12:51:35.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:35.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa7fb5b86f0] 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa7fb60594c] 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 3: raise() 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 4: abort() 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa7fb5a271b] 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 6: /lib64/libc.so.6(+0x37386) [0x7fa7fb5b1386] 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f89dbf96b9] 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f89ddb780c] 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55f89dd05cde] 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55f89dcc7763] 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f89dcc8a1b] 2024-07-26T12:51:35.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f89dcac752] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f89dbfaa56] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f89dbfe637] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f89dbfef5e] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f89dbe203a] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa7fbe85407] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 18: (DispatchQueue::entry()+0x60f) [0x7fa7fbe85cf9] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa7fbf625a1] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 20: (Thread::entry_wrapper()+0x43) [0x7fa7fbd0fcdd] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa7fbd0fcf9] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa7fb603c02] 2024-07-26T12:51:35.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa7fb688c40] 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2024-07-26T12:51:34.826+0000 7fa7f61e1640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: in thread 7fa7f61e1640 thread_name:ms_dispatch 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa7fb5b86f0] 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa7fb60594c] 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 3: raise() 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 4: abort() 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa7fb5a271b] 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 6: /lib64/libc.so.6(+0x37386) [0x7fa7fb5b1386] 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f89dbf96b9] 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f89ddb780c] 2024-07-26T12:51:35.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55f89dd05cde] 2024-07-26T12:51:35.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55f89dcc7763] 2024-07-26T12:51:35.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f89dcc8a1b] 2024-07-26T12:51:35.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f89dcac752] 2024-07-26T12:51:35.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f89dbfaa56] 2024-07-26T12:51:35.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f89dbfe637] 2024-07-26T12:51:35.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f89dbfef5e] 2024-07-26T12:51:35.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f89dbe203a] 2024-07-26T12:51:35.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa7fbe85407] 2024-07-26T12:51:35.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 18: (DispatchQueue::entry()+0x60f) [0x7fa7fbe85cf9] 2024-07-26T12:51:35.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa7fbf625a1] 2024-07-26T12:51:35.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 20: (Thread::entry_wrapper()+0x43) [0x7fa7fbd0fcdd] 2024-07-26T12:51:35.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa7fbd0fcf9] 2024-07-26T12:51:35.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa7fb603c02] 2024-07-26T12:51:35.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa7fb688c40] 2024-07-26T12:51:35.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:35.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2024-07-26T12:51:35.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 0> 2024-07-26T12:51:34.826+0000 7fa7f61e1640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: in thread 7fa7f61e1640 thread_name:ms_dispatch 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa7fb5b86f0] 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa7fb60594c] 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 3: raise() 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 4: abort() 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa7fb5a271b] 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 6: /lib64/libc.so.6(+0x37386) [0x7fa7fb5b1386] 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f89dbf96b9] 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f89ddb780c] 2024-07-26T12:51:35.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55f89dd05cde] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55f89dcc7763] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f89dcc8a1b] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f89dcac752] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f89dbfaa56] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f89dbfe637] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f89dbfef5e] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f89dbe203a] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa7fbe85407] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 18: (DispatchQueue::entry()+0x60f) [0x7fa7fbe85cf9] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa7fbf625a1] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 20: (Thread::entry_wrapper()+0x43) [0x7fa7fbd0fcdd] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa7fbd0fcf9] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa7fb603c02] 2024-07-26T12:51:35.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa7fb688c40] 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 0> 2024-07-26T12:51:34.826+0000 7fa7f61e1640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: in thread 7fa7f61e1640 thread_name:ms_dispatch 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa7fb5b86f0] 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa7fb60594c] 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 3: raise() 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 4: abort() 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa7fb5a271b] 2024-07-26T12:51:35.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 6: /lib64/libc.so.6(+0x37386) [0x7fa7fb5b1386] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f89dbf96b9] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f89ddb780c] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55f89dd05cde] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55f89dcc7763] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f89dcc8a1b] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f89dcac752] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f89dbfaa56] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f89dbfe637] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f89dbfef5e] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f89dbe203a] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa7fbe85407] 2024-07-26T12:51:35.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 18: (DispatchQueue::entry()+0x60f) [0x7fa7fbe85cf9] 2024-07-26T12:51:35.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa7fbf625a1] 2024-07-26T12:51:35.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 20: (Thread::entry_wrapper()+0x43) [0x7fa7fbd0fcdd] 2024-07-26T12:51:35.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa7fbd0fcf9] 2024-07-26T12:51:35.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa7fb603c02] 2024-07-26T12:51:35.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa7fb688c40] 2024-07-26T12:51:35.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:35.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72012]: 2024-07-26T12:51:35.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:34 smithi112 podman[72329]: 2024-07-26 12:51:34.894474219 +0000 UTC m=+0.022423643 container died fe53c02f25a73b94b98ab6dfa4aa5c150f79b3a18ecd9707d3abe30c8cbab24e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:51:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:34 smithi107 ceph-mon[32659]: pgmap v560: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 35 KiB/s wr, 16 op/s 2024-07-26T12:51:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:34 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:rejoin 2024-07-26T12:51:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:34 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:51:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:34 smithi107 ceph-mon[32659]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:51:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:34 smithi132 ceph-mon[37580]: pgmap v560: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 35 KiB/s wr, 16 op/s 2024-07-26T12:51:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:34 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:rejoin 2024-07-26T12:51:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:34 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:51:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:34 smithi132 ceph-mon[37580]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:51:35.884 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:35 smithi112 podman[72329]: 2024-07-26 12:51:35.630722691 +0000 UTC m=+0.758672117 container remove fe53c02f25a73b94b98ab6dfa4aa5c150f79b3a18ecd9707d3abe30c8cbab24e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD) 2024-07-26T12:51:35.884 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:35 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:51:35.884 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:35 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:51:35.884 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:35 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:51:35.884 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:35 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:active 2024-07-26T12:51:35.884 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:35 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:51:35.884 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:35 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:51:36.200 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:35 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T12:51:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:35 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:51:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:35 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:51:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:35 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:active 2024-07-26T12:51:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:35 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:51:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:35 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:35 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:35 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:35 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/790856392,v1:172.21.15.112:6839/790856392] up:active 2024-07-26T12:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:35 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:35 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:51:37.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:37.075+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:37.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:37.075+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:37.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:37.075+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:36 smithi112 ceph-mon[37501]: pgmap v561: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-07-26T12:51:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:36 smithi107 ceph-mon[32659]: pgmap v561: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-07-26T12:51:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:36 smithi132 ceph-mon[37580]: pgmap v561: 129 pgs: 129 active+clean; 38 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 19 KiB/s wr, 13 op/s 2024-07-26T12:51:39.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:38 smithi107 ceph-mon[32659]: pgmap v562: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-07-26T12:51:39.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:39.184+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:39.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:39.185+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:51:39.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:39.185+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:38 smithi112 ceph-mon[37501]: pgmap v562: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-07-26T12:51:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:38 smithi132 ceph-mon[37580]: pgmap v562: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.5 KiB/s wr, 13 op/s 2024-07-26T12:51:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:40 smithi112 ceph-mon[37501]: pgmap v563: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 6.5 KiB/s wr, 16 op/s 2024-07-26T12:51:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:40 smithi107 ceph-mon[32659]: pgmap v563: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 6.5 KiB/s wr, 16 op/s 2024-07-26T12:51:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:40 smithi132 ceph-mon[37580]: pgmap v563: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 6.5 KiB/s wr, 16 op/s 2024-07-26T12:51:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:42 smithi112 ceph-mon[37501]: pgmap v564: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 3.9 KiB/s wr, 16 op/s 2024-07-26T12:51:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:42 smithi107 ceph-mon[32659]: pgmap v564: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 3.9 KiB/s wr, 16 op/s 2024-07-26T12:51:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:42 smithi132 ceph-mon[37580]: pgmap v564: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 3.9 KiB/s wr, 16 op/s 2024-07-26T12:51:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:44 smithi112 ceph-mon[37501]: pgmap v565: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 3.3 KiB/s wr, 17 op/s 2024-07-26T12:51:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:44 smithi107 ceph-mon[32659]: pgmap v565: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 3.3 KiB/s wr, 17 op/s 2024-07-26T12:51:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:44 smithi132 ceph-mon[37580]: pgmap v565: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 3.3 KiB/s wr, 17 op/s 2024-07-26T12:51:46.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:45 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 2. 2024-07-26T12:51:46.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:45 smithi112 systemd[1]: Stopped Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:51:46.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:45 smithi112 systemd[1]: Starting Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:51:46.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:46 smithi112 podman[72422]: 2024-07-26 12:51:46.083690489 +0000 UTC m=+0.017209339 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:51:46.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:46 smithi112 podman[72422]: 2024-07-26 12:51:46.206528956 +0000 UTC m=+0.140047852 container create 504a62cabf87ede2b6504d6e93b537ce3869fbe68a175f466e4fcaab3552a494 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T12:51:46.828 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:46 smithi112 podman[72422]: 2024-07-26 12:51:46.475953081 +0000 UTC m=+0.409471949 container init 504a62cabf87ede2b6504d6e93b537ce3869fbe68a175f466e4fcaab3552a494 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240724) 2024-07-26T12:51:46.828 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:46 smithi112 podman[72422]: 2024-07-26 12:51:46.478744278 +0000 UTC m=+0.412263132 container start 504a62cabf87ede2b6504d6e93b537ce3869fbe68a175f466e4fcaab3552a494 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:51:46.828 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-mds[72441]: -- 172.21.15.112:0/1733027185 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb427bf860 con 0x55fb46961a80 2024-07-26T12:51:46.828 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: starting mds.b at 2024-07-26T12:51:46.828 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:46 smithi112 bash[72422]: 504a62cabf87ede2b6504d6e93b537ce3869fbe68a175f466e4fcaab3552a494 2024-07-26T12:51:46.828 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:51:46 smithi112 systemd[1]: Started Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:51:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-mon[37501]: pgmap v566: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-07-26T12:51:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-mon[37501]: daemon mds.b restarted 2024-07-26T12:51:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:51:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:51:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-mon[37501]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T12:51:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:boot 2024-07-26T12:51:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-mon[37501]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:51:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:51:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:46 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:51:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:46 smithi107 ceph-mon[32659]: pgmap v566: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-07-26T12:51:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:46 smithi107 ceph-mon[32659]: daemon mds.b restarted 2024-07-26T12:51:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:46 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:51:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:51:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:46 smithi107 ceph-mon[32659]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T12:51:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:46 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:boot 2024-07-26T12:51:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:46 smithi107 ceph-mon[32659]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:51:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:46 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:51:47.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:46 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:51:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:46 smithi132 ceph-mon[37580]: pgmap v566: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-07-26T12:51:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:46 smithi132 ceph-mon[37580]: daemon mds.b restarted 2024-07-26T12:51:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:46 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:51:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:51:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:46 smithi132 ceph-mon[37580]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T12:51:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:46 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:boot 2024-07-26T12:51:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:46 smithi132 ceph-mon[37580]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:51:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:46 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:51:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:46 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:51:49.149 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:48 smithi107 ceph-mon[32659]: pgmap v568: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 4.0 KiB/s wr, 13 op/s 2024-07-26T12:51:49.149 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:48 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:resolve 2024-07-26T12:51:49.149 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:48 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:51:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:48 smithi112 ceph-mon[37501]: pgmap v568: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 4.0 KiB/s wr, 13 op/s 2024-07-26T12:51:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:48 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:resolve 2024-07-26T12:51:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:48 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:51:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:48 smithi132 ceph-mon[37580]: pgmap v568: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 4.0 KiB/s wr, 13 op/s 2024-07-26T12:51:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:48 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:resolve 2024-07-26T12:51:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:48 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:51:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:49 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:51:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:49 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:reconnect 2024-07-26T12:51:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:49 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:51:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:49 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:51:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:49 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:51:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:49 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:reconnect 2024-07-26T12:51:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:49 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:51:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:49 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:51:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:49 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:51:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:49 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:reconnect 2024-07-26T12:51:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:49 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:51:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:49 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:51:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-mon[37501]: pgmap v569: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:51:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:rejoin 2024-07-26T12:51:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:51:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-mon[37501]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:51:51.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:51:51.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: *** Caught signal (Aborted) ** 2024-07-26T12:51:51.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: in thread 7ff8f3a07640 thread_name:ms_dispatch 2024-07-26T12:51:51.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff8f8dde6f0] 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff8f8e2b94c] 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 3: raise() 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 4: abort() 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff8f8dc871b] 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 6: /lib64/libc.so.6(+0x37386) [0x7ff8f8dd7386] 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bf83b276b9] 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bf83ce580c] 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55bf83c33cde] 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55bf83bf5763] 2024-07-26T12:51:51.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bf83bf6a1b] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bf83bda752] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bf83b28a56] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bf83b2c637] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bf83b2cf5e] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bf83b1003a] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff8f96ab407] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 18: (DispatchQueue::entry()+0x60f) [0x7ff8f96abcf9] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff8f97885a1] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 20: (Thread::entry_wrapper()+0x43) [0x7ff8f9535cdd] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff8f9535cf9] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff8f8e29c02] 2024-07-26T12:51:51.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff8f8eaec40] 2024-07-26T12:51:51.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2024-07-26T12:51:50.938+0000 7ff8f3a07640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:51.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: in thread 7ff8f3a07640 thread_name:ms_dispatch 2024-07-26T12:51:51.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2024-07-26T12:51:51.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:51.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff8f8dde6f0] 2024-07-26T12:51:51.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff8f8e2b94c] 2024-07-26T12:51:51.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 3: raise() 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 4: abort() 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff8f8dc871b] 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 6: /lib64/libc.so.6(+0x37386) [0x7ff8f8dd7386] 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bf83b276b9] 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bf83ce580c] 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55bf83c33cde] 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55bf83bf5763] 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bf83bf6a1b] 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bf83bda752] 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bf83b28a56] 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bf83b2c637] 2024-07-26T12:51:51.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bf83b2cf5e] 2024-07-26T12:51:51.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bf83b1003a] 2024-07-26T12:51:51.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff8f96ab407] 2024-07-26T12:51:51.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 18: (DispatchQueue::entry()+0x60f) [0x7ff8f96abcf9] 2024-07-26T12:51:51.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff8f97885a1] 2024-07-26T12:51:51.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 20: (Thread::entry_wrapper()+0x43) [0x7ff8f9535cdd] 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff8f9535cf9] 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff8f8e29c02] 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff8f8eaec40] 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 0> 2024-07-26T12:51:50.938+0000 7ff8f3a07640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: in thread 7ff8f3a07640 thread_name:ms_dispatch 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff8f8dde6f0] 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff8f8e2b94c] 2024-07-26T12:51:51.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 3: raise() 2024-07-26T12:51:51.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 4: abort() 2024-07-26T12:51:51.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff8f8dc871b] 2024-07-26T12:51:51.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 6: /lib64/libc.so.6(+0x37386) [0x7ff8f8dd7386] 2024-07-26T12:51:51.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bf83b276b9] 2024-07-26T12:51:51.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bf83ce580c] 2024-07-26T12:51:51.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55bf83c33cde] 2024-07-26T12:51:51.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55bf83bf5763] 2024-07-26T12:51:51.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bf83bf6a1b] 2024-07-26T12:51:51.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bf83bda752] 2024-07-26T12:51:51.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bf83b28a56] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bf83b2c637] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bf83b2cf5e] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bf83b1003a] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff8f96ab407] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 18: (DispatchQueue::entry()+0x60f) [0x7ff8f96abcf9] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff8f97885a1] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 20: (Thread::entry_wrapper()+0x43) [0x7ff8f9535cdd] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff8f9535cf9] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff8f8e29c02] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff8f8eaec40] 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:51.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 0> 2024-07-26T12:51:50.938+0000 7ff8f3a07640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: in thread 7ff8f3a07640 thread_name:ms_dispatch 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7ff8f8dde6f0] 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2: /lib64/libc.so.6(+0x8b94c) [0x7ff8f8e2b94c] 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 3: raise() 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 4: abort() 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 5: /lib64/libc.so.6(+0x2871b) [0x7ff8f8dc871b] 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 6: /lib64/libc.so.6(+0x37386) [0x7ff8f8dd7386] 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bf83b276b9] 2024-07-26T12:51:51.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bf83ce580c] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55bf83c33cde] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55bf83bf5763] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bf83bf6a1b] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bf83bda752] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bf83b28a56] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bf83b2c637] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bf83b2cf5e] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bf83b1003a] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7ff8f96ab407] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 18: (DispatchQueue::entry()+0x60f) [0x7ff8f96abcf9] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7ff8f97885a1] 2024-07-26T12:51:51.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 20: (Thread::entry_wrapper()+0x43) [0x7ff8f9535cdd] 2024-07-26T12:51:51.212 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 21: (Thread::_entry_func(void*)+0xd) [0x7ff8f9535cf9] 2024-07-26T12:51:51.212 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 22: /lib64/libc.so.6(+0x89c02) [0x7ff8f8e29c02] 2024-07-26T12:51:51.212 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 23: /lib64/libc.so.6(+0x10ec40) [0x7ff8f8eaec40] 2024-07-26T12:51:51.212 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:51:51.212 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72290]: 2024-07-26T12:51:51.212 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:50 smithi112 podman[72488]: 2024-07-26 12:51:50.999889222 +0000 UTC m=+0.022397360 container died 1d2b226c9861768e089f904dcca81fe68c4193cb37500ef3f5bba65323a28e50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-07-26T12:51:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:50 smithi107 ceph-mon[32659]: pgmap v569: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:51:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:50 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:rejoin 2024-07-26T12:51:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:50 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:51:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:50 smithi107 ceph-mon[32659]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:51:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:50 smithi132 ceph-mon[37580]: pgmap v569: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:51:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:50 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:rejoin 2024-07-26T12:51:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:50 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:51:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:50 smithi132 ceph-mon[37580]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:51:51.911 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:51 smithi112 podman[72488]: 2024-07-26 12:51:51.656615424 +0000 UTC m=+0.679123604 container remove 1d2b226c9861768e089f904dcca81fe68c4193cb37500ef3f5bba65323a28e50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-07-26T12:51:51.911 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:51 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:51:52.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:52.080+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:52.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:52.082+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T12:51:52.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:52.082+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:52.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:52.082+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:52.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:51:52.082+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:52.200 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:51:51 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T12:51:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:51 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:51:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:51 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:51:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:51 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:active 2024-07-26T12:51:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:51 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:51:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:51 smithi112 ceph-mon[37501]: pgmap v570: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:51:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:51 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:51:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:51 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:51:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:51 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:51:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:51 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:active 2024-07-26T12:51:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:51 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:51:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:51 smithi107 ceph-mon[32659]: pgmap v570: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:51:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:51 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:51:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:51 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:51:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:51 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:51:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:51 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/3220120050,v1:172.21.15.112:6837/3220120050] up:active 2024-07-26T12:51:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:51 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:51:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:51 smithi132 ceph-mon[37580]: pgmap v570: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:51:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:51 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:51:54.189 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:54.188+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:51:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:54.189+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T12:51:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:54.189+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:51:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:54.189+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:51:54.190 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:51:54.189+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:53 smithi107 ceph-mon[32659]: pgmap v571: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-07-26T12:51:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:53 smithi132 ceph-mon[37580]: pgmap v571: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-07-26T12:51:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:53 smithi112 ceph-mon[37501]: pgmap v571: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 2.8 KiB/s wr, 15 op/s 2024-07-26T12:51:56.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:55 smithi107 ceph-mon[32659]: pgmap v572: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2024-07-26T12:51:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:55 smithi132 ceph-mon[37580]: pgmap v572: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2024-07-26T12:51:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:55 smithi112 ceph-mon[37501]: pgmap v572: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 2.8 KiB/s wr, 16 op/s 2024-07-26T12:51:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:51:58 smithi112 ceph-mon[37501]: pgmap v573: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 2.7 KiB/s wr, 16 op/s 2024-07-26T12:51:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:51:58 smithi107 ceph-mon[32659]: pgmap v573: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 2.7 KiB/s wr, 16 op/s 2024-07-26T12:51:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:51:58 smithi132 ceph-mon[37580]: pgmap v573: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 2.7 KiB/s wr, 16 op/s 2024-07-26T12:52:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:00 smithi112 ceph-mon[37501]: pgmap v574: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 2.3 KiB/s wr, 16 op/s 2024-07-26T12:52:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:00 smithi107 ceph-mon[32659]: pgmap v574: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 2.3 KiB/s wr, 16 op/s 2024-07-26T12:52:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:00 smithi132 ceph-mon[37580]: pgmap v574: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 2.3 KiB/s wr, 16 op/s 2024-07-26T12:52:02.308 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:01 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 3. 2024-07-26T12:52:02.308 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:01 smithi112 systemd[1]: Stopped Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:52:02.308 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:01 smithi112 systemd[1]: Starting Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:52:02.308 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:02 smithi112 podman[72580]: 2024-07-26 12:52:02.087272171 +0000 UTC m=+0.017472429 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:52:02.578 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:02 smithi112 podman[72580]: 2024-07-26 12:52:02.307807093 +0000 UTC m=+0.238007379 container create f53bc18a459f244bfff9d4d723c9167c2820b596dfb5c7226496006e508367f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-07-26T12:52:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:02 smithi112 ceph-mon[37501]: pgmap v575: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 2.3 KiB/s wr, 14 op/s 2024-07-26T12:52:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:02 smithi112 ceph-mon[37501]: daemon mds.h restarted 2024-07-26T12:52:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:02 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:52:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:02 smithi112 ceph-mon[37501]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T12:52:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:02 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:boot 2024-07-26T12:52:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:02 smithi112 ceph-mon[37501]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:52:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:02 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:02.951 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:02 smithi112 podman[72580]: 2024-07-26 12:52:02.577688288 +0000 UTC m=+0.507888539 container init f53bc18a459f244bfff9d4d723c9167c2820b596dfb5c7226496006e508367f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-07-26T12:52:02.951 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:02 smithi112 podman[72580]: 2024-07-26 12:52:02.580793297 +0000 UTC m=+0.510993539 container start f53bc18a459f244bfff9d4d723c9167c2820b596dfb5c7226496006e508367f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True) 2024-07-26T12:52:02.951 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:02 smithi112 ceph-mds[72599]: -- 172.21.15.112:0/3320309535 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562145957860 con 0x562149af9a80 2024-07-26T12:52:02.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:02 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: starting mds.h at 2024-07-26T12:52:02.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:02 smithi112 bash[72580]: f53bc18a459f244bfff9d4d723c9167c2820b596dfb5c7226496006e508367f5 2024-07-26T12:52:02.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:02 smithi112 systemd[1]: Started Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:52:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:02 smithi107 ceph-mon[32659]: pgmap v575: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 2.3 KiB/s wr, 14 op/s 2024-07-26T12:52:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:02 smithi107 ceph-mon[32659]: daemon mds.h restarted 2024-07-26T12:52:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:02 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:52:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:02 smithi107 ceph-mon[32659]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T12:52:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:02 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:boot 2024-07-26T12:52:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:02 smithi107 ceph-mon[32659]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:52:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:02 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:02 smithi132 ceph-mon[37580]: pgmap v575: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 2.3 KiB/s wr, 14 op/s 2024-07-26T12:52:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:02 smithi132 ceph-mon[37580]: daemon mds.h restarted 2024-07-26T12:52:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:02 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:52:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:02 smithi132 ceph-mon[37580]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T12:52:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:02 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:boot 2024-07-26T12:52:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:02 smithi132 ceph-mon[37580]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:52:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:02 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:03 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:03 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:04.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:03 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:04.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:04 smithi107 ceph-mon[32659]: pgmap v577: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-07-26T12:52:04.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:04 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:resolve 2024-07-26T12:52:04.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:04 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:04 smithi112 ceph-mon[37501]: pgmap v577: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-07-26T12:52:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:04 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:resolve 2024-07-26T12:52:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:04 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:04 smithi132 ceph-mon[37580]: pgmap v577: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 0 B/s wr, 12 op/s 2024-07-26T12:52:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:04 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:resolve 2024-07-26T12:52:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:04 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:05 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299992 2024-07-26T12:52:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:05 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:reconnect 2024-07-26T12:52:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:05 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:05 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109997 2024-07-26T12:52:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:05 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299992 2024-07-26T12:52:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:05 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:reconnect 2024-07-26T12:52:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:05 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:05 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109997 2024-07-26T12:52:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:05 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299992 2024-07-26T12:52:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:05 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:reconnect 2024-07-26T12:52:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:05 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:05 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109997 2024-07-26T12:52:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-mon[37501]: pgmap v578: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-07-26T12:52:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:rejoin 2024-07-26T12:52:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-mon[37501]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:52:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:52:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:active 2024-07-26T12:52:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:06.952 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:52:06.952 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: *** Caught signal (Aborted) ** 2024-07-26T12:52:06.952 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: in thread 7f6b00b53640 thread_name:ms_dispatch 2024-07-26T12:52:06.952 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:06.952 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6b05f2a6f0] 2024-07-26T12:52:06.952 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6b05f7794c] 2024-07-26T12:52:06.952 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 3: raise() 2024-07-26T12:52:06.952 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 4: abort() 2024-07-26T12:52:06.952 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6b05f1471b] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 6: /lib64/libc.so.6(+0x37386) [0x7f6b05f23386] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fb402a36b9] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fb4046180c] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55fb403afcde] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55fb40371763] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fb40372a1b] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fb40356752] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fb402a4a56] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fb402a8637] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fb402a8f5e] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fb4028c03a] 2024-07-26T12:52:06.953 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6b067f7407] 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 18: (DispatchQueue::entry()+0x60f) [0x7f6b067f7cf9] 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6b068d45a1] 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 20: (Thread::entry_wrapper()+0x43) [0x7f6b06681cdd] 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6b06681cf9] 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6b05f75c02] 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6b05ffac40] 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2024-07-26T12:52:06.649+0000 7f6b00b53640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: in thread 7f6b00b53640 thread_name:ms_dispatch 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6b05f2a6f0] 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6b05f7794c] 2024-07-26T12:52:06.954 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 3: raise() 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 4: abort() 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6b05f1471b] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 6: /lib64/libc.so.6(+0x37386) [0x7f6b05f23386] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fb402a36b9] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fb4046180c] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55fb403afcde] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55fb40371763] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fb40372a1b] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fb40356752] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fb402a4a56] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fb402a8637] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fb402a8f5e] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fb4028c03a] 2024-07-26T12:52:06.955 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6b067f7407] 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 18: (DispatchQueue::entry()+0x60f) [0x7f6b067f7cf9] 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6b068d45a1] 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 20: (Thread::entry_wrapper()+0x43) [0x7f6b06681cdd] 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6b06681cf9] 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6b05f75c02] 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6b05ffac40] 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 0> 2024-07-26T12:52:06.649+0000 7f6b00b53640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: in thread 7f6b00b53640 thread_name:ms_dispatch 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6b05f2a6f0] 2024-07-26T12:52:06.956 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6b05f7794c] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 3: raise() 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 4: abort() 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6b05f1471b] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 6: /lib64/libc.so.6(+0x37386) [0x7f6b05f23386] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fb402a36b9] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fb4046180c] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55fb403afcde] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55fb40371763] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fb40372a1b] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fb40356752] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fb402a4a56] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fb402a8637] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fb402a8f5e] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fb4028c03a] 2024-07-26T12:52:06.957 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6b067f7407] 2024-07-26T12:52:06.958 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 18: (DispatchQueue::entry()+0x60f) [0x7f6b067f7cf9] 2024-07-26T12:52:06.958 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6b068d45a1] 2024-07-26T12:52:06.958 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 20: (Thread::entry_wrapper()+0x43) [0x7f6b06681cdd] 2024-07-26T12:52:06.958 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6b06681cf9] 2024-07-26T12:52:06.958 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6b05f75c02] 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6b05ffac40] 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 0> 2024-07-26T12:52:06.649+0000 7f6b00b53640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: in thread 7f6b00b53640 thread_name:ms_dispatch 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6b05f2a6f0] 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6b05f7794c] 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 3: raise() 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 4: abort() 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6b05f1471b] 2024-07-26T12:52:06.959 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 6: /lib64/libc.so.6(+0x37386) [0x7f6b05f23386] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fb402a36b9] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fb4046180c] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55fb403afcde] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55fb40371763] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fb40372a1b] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fb40356752] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fb402a4a56] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fb402a8637] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fb402a8f5e] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fb4028c03a] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6b067f7407] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 18: (DispatchQueue::entry()+0x60f) [0x7f6b067f7cf9] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6b068d45a1] 2024-07-26T12:52:06.960 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 20: (Thread::entry_wrapper()+0x43) [0x7f6b06681cdd] 2024-07-26T12:52:06.961 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6b06681cf9] 2024-07-26T12:52:06.961 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6b05f75c02] 2024-07-26T12:52:06.961 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6b05ffac40] 2024-07-26T12:52:06.961 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:06.961 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72437]: 2024-07-26T12:52:06.961 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:06 smithi112 podman[72634]: 2024-07-26 12:52:06.712819337 +0000 UTC m=+0.022546870 container died 504a62cabf87ede2b6504d6e93b537ce3869fbe68a175f466e4fcaab3552a494 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240724) 2024-07-26T12:52:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:06 smithi107 ceph-mon[32659]: pgmap v578: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-07-26T12:52:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:06 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:rejoin 2024-07-26T12:52:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:06 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:06 smithi107 ceph-mon[32659]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:52:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:06 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:06 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:52:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:06 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:active 2024-07-26T12:52:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:06 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:06 smithi132 ceph-mon[37580]: pgmap v578: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 0 B/s wr, 11 op/s 2024-07-26T12:52:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:06 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:rejoin 2024-07-26T12:52:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:06 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:06 smithi132 ceph-mon[37580]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:52:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:06 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:06 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:52:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:06 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/4163161701,v1:172.21.15.112:6839/4163161701] up:active 2024-07-26T12:52:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:06 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:07.088+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:52:07.090 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:07.088+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:07.091 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:07.088+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:52:07.700 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:07 smithi112 podman[72634]: 2024-07-26 12:52:07.432712282 +0000 UTC m=+0.742439805 container remove 504a62cabf87ede2b6504d6e93b537ce3869fbe68a175f466e4fcaab3552a494 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:52:07.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:07 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:52:07.701 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:07 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T12:52:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:08 smithi107 ceph-mon[32659]: pgmap v579: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 0 B/s wr, 13 op/s 2024-07-26T12:52:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:08 smithi132 ceph-mon[37580]: pgmap v579: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 0 B/s wr, 13 op/s 2024-07-26T12:52:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:08 smithi112 ceph-mon[37501]: pgmap v579: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 0 B/s wr, 13 op/s 2024-07-26T12:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:09.204+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:09.204+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:09.204+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:10 smithi107 ceph-mon[32659]: pgmap v580: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:52:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:10 smithi132 ceph-mon[37580]: pgmap v580: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:52:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:10 smithi112 ceph-mon[37501]: pgmap v580: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:52:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:12 smithi107 ceph-mon[32659]: pgmap v581: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:52:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:12 smithi132 ceph-mon[37580]: pgmap v581: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:52:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:12 smithi112 ceph-mon[37501]: pgmap v581: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:52:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:14 smithi107 ceph-mon[32659]: pgmap v582: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 198 B/s wr, 16 op/s 2024-07-26T12:52:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:14 smithi132 ceph-mon[37580]: pgmap v582: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 198 B/s wr, 16 op/s 2024-07-26T12:52:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:14 smithi112 ceph-mon[37501]: pgmap v582: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 198 B/s wr, 16 op/s 2024-07-26T12:52:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:16 smithi107 ceph-mon[32659]: pgmap v583: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:52:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:16 smithi132 ceph-mon[37580]: pgmap v583: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:52:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:16 smithi112 ceph-mon[37501]: pgmap v583: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:52:18.072 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:17 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 3. 2024-07-26T12:52:18.073 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:17 smithi112 systemd[1]: Stopped Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:52:18.073 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:17 smithi112 systemd[1]: Starting Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:52:18.073 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:17 smithi112 podman[72727]: 2024-07-26 12:52:17.837014324 +0000 UTC m=+0.017260353 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:52:18.073 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:17 smithi112 podman[72727]: 2024-07-26 12:52:17.950274787 +0000 UTC m=+0.130520861 container create 7a5e51c85408d7d4f59eaabd8cafceca70d74e755e2e4fac0c6ca2a235c40694 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:52:18.450 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:18 smithi112 podman[72727]: 2024-07-26 12:52:18.185966512 +0000 UTC m=+0.366212552 container init 7a5e51c85408d7d4f59eaabd8cafceca70d74e755e2e4fac0c6ca2a235c40694 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:52:18.450 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:18 smithi112 podman[72727]: 2024-07-26 12:52:18.188593929 +0000 UTC m=+0.368839959 container start 7a5e51c85408d7d4f59eaabd8cafceca70d74e755e2e4fac0c6ca2a235c40694 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-26T12:52:18.450 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-mds[72746]: -- 172.21.15.112:0/3394608099 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d7ef039860 con 0x55d7f31db600 2024-07-26T12:52:18.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: starting mds.b at 2024-07-26T12:52:18.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:18 smithi112 bash[72727]: 7a5e51c85408d7d4f59eaabd8cafceca70d74e755e2e4fac0c6ca2a235c40694 2024-07-26T12:52:18.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:18 smithi112 systemd[1]: Started Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:52:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:18 smithi107 ceph-mon[32659]: pgmap v584: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:52:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:18 smithi107 ceph-mon[32659]: daemon mds.b restarted 2024-07-26T12:52:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:18 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:52:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:18 smithi107 ceph-mon[32659]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T12:52:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:18 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:boot 2024-07-26T12:52:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:18 smithi107 ceph-mon[32659]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:52:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:18 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:18 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:18 smithi132 ceph-mon[37580]: pgmap v584: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:52:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:18 smithi132 ceph-mon[37580]: daemon mds.b restarted 2024-07-26T12:52:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:18 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:52:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:18 smithi132 ceph-mon[37580]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T12:52:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:18 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:boot 2024-07-26T12:52:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:18 smithi132 ceph-mon[37580]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:52:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:18 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:18 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-mon[37501]: pgmap v584: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:52:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-mon[37501]: daemon mds.b restarted 2024-07-26T12:52:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:52:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-mon[37501]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T12:52:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:boot 2024-07-26T12:52:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-mon[37501]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:52:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:18 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:21.009 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:20 smithi107 ceph-mon[32659]: pgmap v586: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T12:52:21.009 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:20 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:resolve 2024-07-26T12:52:21.009 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:20 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:20 smithi132 ceph-mon[37580]: pgmap v586: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T12:52:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:20 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:resolve 2024-07-26T12:52:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:20 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:20 smithi112 ceph-mon[37501]: pgmap v586: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T12:52:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:20 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:resolve 2024-07-26T12:52:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:20 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:21 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299992 2024-07-26T12:52:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:21 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:reconnect 2024-07-26T12:52:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:21 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:21 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:52:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:21 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299992 2024-07-26T12:52:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:21 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:reconnect 2024-07-26T12:52:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:21 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:21 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:52:22.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:22.100+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:22.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:22.102+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T12:52:22.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:22.102+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:22.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:22.102+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:22.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:22.102+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:21 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299992 2024-07-26T12:52:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:21 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:reconnect 2024-07-26T12:52:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:21 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:21 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:52:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:22 smithi107 ceph-mon[32659]: pgmap v587: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T12:52:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:22 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:rejoin 2024-07-26T12:52:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:22 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:22 smithi107 ceph-mon[32659]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:52:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:22 smithi132 ceph-mon[37580]: pgmap v587: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T12:52:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:22 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:rejoin 2024-07-26T12:52:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:22 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:22 smithi132 ceph-mon[37580]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:52:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-mon[37501]: pgmap v587: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T12:52:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:rejoin 2024-07-26T12:52:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-mon[37501]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:52:23.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:52:23.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: *** Caught signal (Aborted) ** 2024-07-26T12:52:23.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: in thread 7fdceb6b3640 thread_name:ms_dispatch 2024-07-26T12:52:23.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:23.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdcf0a8a6f0] 2024-07-26T12:52:23.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdcf0ad794c] 2024-07-26T12:52:23.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 3: raise() 2024-07-26T12:52:23.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 4: abort() 2024-07-26T12:52:23.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdcf0a7471b] 2024-07-26T12:52:23.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 6: /lib64/libc.so.6(+0x37386) [0x7fdcf0a83386] 2024-07-26T12:52:23.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5621447c46b9] 2024-07-26T12:52:23.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56214498280c] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5621448d0cde] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x562144892763] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562144893a1b] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562144877752] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5621447c5a56] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5621447c9637] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5621447c9f5e] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5621447ad03a] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdcf1357407] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 18: (DispatchQueue::entry()+0x60f) [0x7fdcf1357cf9] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdcf14345a1] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 20: (Thread::entry_wrapper()+0x43) [0x7fdcf11e1cdd] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdcf11e1cf9] 2024-07-26T12:52:23.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdcf0ad5c02] 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdcf0b5ac40] 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2024-07-26T12:52:22.795+0000 7fdceb6b3640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: in thread 7fdceb6b3640 thread_name:ms_dispatch 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdcf0a8a6f0] 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdcf0ad794c] 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 3: raise() 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 4: abort() 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdcf0a7471b] 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 6: /lib64/libc.so.6(+0x37386) [0x7fdcf0a83386] 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5621447c46b9] 2024-07-26T12:52:23.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56214498280c] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5621448d0cde] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x562144892763] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562144893a1b] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562144877752] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5621447c5a56] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5621447c9637] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5621447c9f5e] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5621447ad03a] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdcf1357407] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 18: (DispatchQueue::entry()+0x60f) [0x7fdcf1357cf9] 2024-07-26T12:52:23.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdcf14345a1] 2024-07-26T12:52:23.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 20: (Thread::entry_wrapper()+0x43) [0x7fdcf11e1cdd] 2024-07-26T12:52:23.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdcf11e1cf9] 2024-07-26T12:52:23.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdcf0ad5c02] 2024-07-26T12:52:23.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdcf0b5ac40] 2024-07-26T12:52:23.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:23.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2024-07-26T12:52:23.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 0> 2024-07-26T12:52:22.795+0000 7fdceb6b3640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:23.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: in thread 7fdceb6b3640 thread_name:ms_dispatch 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdcf0a8a6f0] 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdcf0ad794c] 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 3: raise() 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 4: abort() 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdcf0a7471b] 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 6: /lib64/libc.so.6(+0x37386) [0x7fdcf0a83386] 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5621447c46b9] 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56214498280c] 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5621448d0cde] 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x562144892763] 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562144893a1b] 2024-07-26T12:52:23.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562144877752] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5621447c5a56] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5621447c9637] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5621447c9f5e] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5621447ad03a] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdcf1357407] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 18: (DispatchQueue::entry()+0x60f) [0x7fdcf1357cf9] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdcf14345a1] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 20: (Thread::entry_wrapper()+0x43) [0x7fdcf11e1cdd] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdcf11e1cf9] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdcf0ad5c02] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdcf0b5ac40] 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:23.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 0> 2024-07-26T12:52:22.795+0000 7fdceb6b3640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: in thread 7fdceb6b3640 thread_name:ms_dispatch 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fdcf0a8a6f0] 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fdcf0ad794c] 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 3: raise() 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 4: abort() 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 5: /lib64/libc.so.6(+0x2871b) [0x7fdcf0a7471b] 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 6: /lib64/libc.so.6(+0x37386) [0x7fdcf0a83386] 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5621447c46b9] 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56214498280c] 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5621448d0cde] 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x562144892763] 2024-07-26T12:52:23.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562144893a1b] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562144877752] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5621447c5a56] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5621447c9637] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5621447c9f5e] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5621447ad03a] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fdcf1357407] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 18: (DispatchQueue::entry()+0x60f) [0x7fdcf1357cf9] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fdcf14345a1] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 20: (Thread::entry_wrapper()+0x43) [0x7fdcf11e1cdd] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 21: (Thread::_entry_func(void*)+0xd) [0x7fdcf11e1cf9] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 22: /lib64/libc.so.6(+0x89c02) [0x7fdcf0ad5c02] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fdcf0b5ac40] 2024-07-26T12:52:23.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:23.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[72594]: 2024-07-26T12:52:23.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:22 smithi112 podman[72781]: 2024-07-26 12:52:22.863742868 +0000 UTC m=+0.024625199 container died f53bc18a459f244bfff9d4d723c9167c2820b596dfb5c7226496006e508367f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:52:23.801 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:23 smithi112 podman[72781]: 2024-07-26 12:52:23.549960341 +0000 UTC m=+0.710842673 container remove f53bc18a459f244bfff9d4d723c9167c2820b596dfb5c7226496006e508367f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-07-26T12:52:23.802 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:23 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:52:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:23 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:23 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:52:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:23 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:active 2024-07-26T12:52:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:23 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:52:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:23 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:24.200 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:23 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T12:52:24.224 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:23 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:24.224 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:23 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:52:24.225 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:23 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:active 2024-07-26T12:52:24.225 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:23 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:52:24.225 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:23 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:24.217+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:24.219+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T12:52:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:24.219+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:24.219+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:24.219+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:23 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:23 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:52:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:23 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/1839304426,v1:172.21.15.112:6837/1839304426] up:active 2024-07-26T12:52:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:23 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:52:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:23 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:24 smithi132 ceph-mon[37580]: pgmap v588: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T12:52:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:24 smithi112 ceph-mon[37501]: pgmap v588: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T12:52:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:24 smithi107 ceph-mon[32659]: pgmap v588: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T12:52:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: pgmap v589: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: pgmap v589: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:52:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: pgmap v589: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:52:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:27 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T12:52:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:27 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T12:52:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:27 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T12:52:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:27 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T12:52:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:27 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T12:52:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:27 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T12:52:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:27 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T12:52:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:27 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T12:52:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:27 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T12:52:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:27 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T12:52:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:27 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T12:52:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:27 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T12:52:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:52:29.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:28 smithi107 ceph-mon[32659]: pgmap v590: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:52:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:28 smithi112 ceph-mon[37501]: pgmap v590: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:52:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:28 smithi132 ceph-mon[37580]: pgmap v590: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:52:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:30 smithi112 ceph-mon[37501]: pgmap v591: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 190 B/s wr, 15 op/s 2024-07-26T12:52:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:30 smithi107 ceph-mon[32659]: pgmap v591: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 190 B/s wr, 15 op/s 2024-07-26T12:52:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:30 smithi132 ceph-mon[37580]: pgmap v591: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 190 B/s wr, 15 op/s 2024-07-26T12:52:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:32 smithi112 ceph-mon[37501]: pgmap v592: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:52:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:32 smithi107 ceph-mon[32659]: pgmap v592: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:52:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:32 smithi132 ceph-mon[37580]: pgmap v592: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:52:34.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:33 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 4. 2024-07-26T12:52:34.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:33 smithi112 systemd[1]: Stopped Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:52:34.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:33 smithi112 systemd[1]: Starting Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:52:34.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:34 smithi112 podman[73000]: 2024-07-26 12:52:34.094828183 +0000 UTC m=+0.017777429 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:52:34.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:34 smithi112 podman[73000]: 2024-07-26 12:52:34.209426198 +0000 UTC m=+0.132375470 container create 67dbe1ef11012bc8495deef798f814961c70c9360041b7037cad5c70efbeaf77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-07-26T12:52:34.876 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:34 smithi112 podman[73000]: 2024-07-26 12:52:34.470425474 +0000 UTC m=+0.393374721 container init 67dbe1ef11012bc8495deef798f814961c70c9360041b7037cad5c70efbeaf77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-07-26T12:52:34.876 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:34 smithi112 podman[73000]: 2024-07-26 12:52:34.472944879 +0000 UTC m=+0.395894118 container start 67dbe1ef11012bc8495deef798f814961c70c9360041b7037cad5c70efbeaf77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-07-26T12:52:34.876 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-mds[73019]: -- 172.21.15.112:0/754866552 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559a71ae9860 con 0x559a75c8ba80 2024-07-26T12:52:34.876 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: starting mds.h at 2024-07-26T12:52:34.876 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:34 smithi112 bash[73000]: 67dbe1ef11012bc8495deef798f814961c70c9360041b7037cad5c70efbeaf77 2024-07-26T12:52:34.876 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:34 smithi112 systemd[1]: Started Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:52:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-mon[37501]: pgmap v593: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:52:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-mon[37501]: daemon mds.h restarted 2024-07-26T12:52:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:52:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-mon[37501]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T12:52:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:boot 2024-07-26T12:52:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-mon[37501]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:52:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:34 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:34 smithi107 ceph-mon[32659]: pgmap v593: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:52:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:34 smithi107 ceph-mon[32659]: daemon mds.h restarted 2024-07-26T12:52:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:34 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:52:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:34 smithi107 ceph-mon[32659]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T12:52:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:34 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:boot 2024-07-26T12:52:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:34 smithi107 ceph-mon[32659]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:52:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:34 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:34 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:34 smithi132 ceph-mon[37580]: pgmap v593: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:52:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:34 smithi132 ceph-mon[37580]: daemon mds.h restarted 2024-07-26T12:52:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:34 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:52:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:34 smithi132 ceph-mon[37580]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T12:52:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:34 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:boot 2024-07-26T12:52:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:34 smithi132 ceph-mon[37580]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:52:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:34 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:34 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:37.118+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:37.120+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T12:52:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:37.120+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:37.120+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:37.121 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:37.120+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:37.144 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:36 smithi107 ceph-mon[32659]: pgmap v595: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 13 op/s 2024-07-26T12:52:37.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:36 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:resolve 2024-07-26T12:52:37.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:36 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:36 smithi112 ceph-mon[37501]: pgmap v595: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 13 op/s 2024-07-26T12:52:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:36 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:resolve 2024-07-26T12:52:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:36 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:36 smithi132 ceph-mon[37580]: pgmap v595: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 13 op/s 2024-07-26T12:52:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:36 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:resolve 2024-07-26T12:52:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:36 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:37 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.000999974 2024-07-26T12:52:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:37 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:reconnect 2024-07-26T12:52:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:37 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:37 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00899976 2024-07-26T12:52:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:37 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.000999974 2024-07-26T12:52:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:37 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:reconnect 2024-07-26T12:52:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:37 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:37 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00899976 2024-07-26T12:52:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:37 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.000999974 2024-07-26T12:52:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:37 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:reconnect 2024-07-26T12:52:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:37 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:37 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00899976 2024-07-26T12:52:39.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:38 smithi107 ceph-mon[32659]: pgmap v596: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 10 op/s 2024-07-26T12:52:39.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:38 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:rejoin 2024-07-26T12:52:39.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:38 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:39.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:38 smithi107 ceph-mon[32659]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:52:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-mon[37501]: pgmap v596: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 10 op/s 2024-07-26T12:52:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:rejoin 2024-07-26T12:52:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-mon[37501]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:52:39.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:52:39.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: *** Caught signal (Aborted) ** 2024-07-26T12:52:39.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: in thread 7fabd3725640 thread_name:ms_dispatch 2024-07-26T12:52:39.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:39.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fabd8afc6f0] 2024-07-26T12:52:39.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fabd8b4994c] 2024-07-26T12:52:39.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 3: raise() 2024-07-26T12:52:39.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 4: abort() 2024-07-26T12:52:39.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 5: /lib64/libc.so.6(+0x2871b) [0x7fabd8ae671b] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 6: /lib64/libc.so.6(+0x37386) [0x7fabd8af5386] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d7ec3f76b9] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d7ec5b580c] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d7ec503cde] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d7ec4c5763] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d7ec4c6a1b] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d7ec4aa752] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d7ec3f8a56] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d7ec3fc637] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d7ec3fcf5e] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d7ec3e003a] 2024-07-26T12:52:39.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fabd93c9407] 2024-07-26T12:52:39.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 18: (DispatchQueue::entry()+0x60f) [0x7fabd93c9cf9] 2024-07-26T12:52:39.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fabd94a65a1] 2024-07-26T12:52:39.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 20: (Thread::entry_wrapper()+0x43) [0x7fabd9253cdd] 2024-07-26T12:52:39.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 21: (Thread::_entry_func(void*)+0xd) [0x7fabd9253cf9] 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 22: /lib64/libc.so.6(+0x89c02) [0x7fabd8b47c02] 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fabd8bccc40] 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2024-07-26T12:52:38.934+0000 7fabd3725640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: in thread 7fabd3725640 thread_name:ms_dispatch 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fabd8afc6f0] 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fabd8b4994c] 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 3: raise() 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 4: abort() 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 5: /lib64/libc.so.6(+0x2871b) [0x7fabd8ae671b] 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 6: /lib64/libc.so.6(+0x37386) [0x7fabd8af5386] 2024-07-26T12:52:39.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d7ec3f76b9] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d7ec5b580c] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d7ec503cde] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d7ec4c5763] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d7ec4c6a1b] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d7ec4aa752] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d7ec3f8a56] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d7ec3fc637] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d7ec3fcf5e] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d7ec3e003a] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fabd93c9407] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 18: (DispatchQueue::entry()+0x60f) [0x7fabd93c9cf9] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fabd94a65a1] 2024-07-26T12:52:39.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 20: (Thread::entry_wrapper()+0x43) [0x7fabd9253cdd] 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 21: (Thread::_entry_func(void*)+0xd) [0x7fabd9253cf9] 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 22: /lib64/libc.so.6(+0x89c02) [0x7fabd8b47c02] 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fabd8bccc40] 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 0> 2024-07-26T12:52:38.934+0000 7fabd3725640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: in thread 7fabd3725640 thread_name:ms_dispatch 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fabd8afc6f0] 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fabd8b4994c] 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 3: raise() 2024-07-26T12:52:39.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 4: abort() 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 5: /lib64/libc.so.6(+0x2871b) [0x7fabd8ae671b] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 6: /lib64/libc.so.6(+0x37386) [0x7fabd8af5386] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d7ec3f76b9] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d7ec5b580c] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d7ec503cde] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d7ec4c5763] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d7ec4c6a1b] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d7ec4aa752] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d7ec3f8a56] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d7ec3fc637] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d7ec3fcf5e] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d7ec3e003a] 2024-07-26T12:52:39.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fabd93c9407] 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 18: (DispatchQueue::entry()+0x60f) [0x7fabd93c9cf9] 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fabd94a65a1] 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 20: (Thread::entry_wrapper()+0x43) [0x7fabd9253cdd] 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 21: (Thread::_entry_func(void*)+0xd) [0x7fabd9253cf9] 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 22: /lib64/libc.so.6(+0x89c02) [0x7fabd8b47c02] 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fabd8bccc40] 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 0> 2024-07-26T12:52:38.934+0000 7fabd3725640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: in thread 7fabd3725640 thread_name:ms_dispatch 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fabd8afc6f0] 2024-07-26T12:52:39.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fabd8b4994c] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 3: raise() 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 4: abort() 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 5: /lib64/libc.so.6(+0x2871b) [0x7fabd8ae671b] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 6: /lib64/libc.so.6(+0x37386) [0x7fabd8af5386] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d7ec3f76b9] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d7ec5b580c] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d7ec503cde] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d7ec4c5763] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d7ec4c6a1b] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d7ec4aa752] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d7ec3f8a56] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d7ec3fc637] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d7ec3fcf5e] 2024-07-26T12:52:39.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d7ec3e003a] 2024-07-26T12:52:39.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fabd93c9407] 2024-07-26T12:52:39.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 18: (DispatchQueue::entry()+0x60f) [0x7fabd93c9cf9] 2024-07-26T12:52:39.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fabd94a65a1] 2024-07-26T12:52:39.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 20: (Thread::entry_wrapper()+0x43) [0x7fabd9253cdd] 2024-07-26T12:52:39.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 21: (Thread::_entry_func(void*)+0xd) [0x7fabd9253cf9] 2024-07-26T12:52:39.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 22: /lib64/libc.so.6(+0x89c02) [0x7fabd8b47c02] 2024-07-26T12:52:39.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fabd8bccc40] 2024-07-26T12:52:39.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:39.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[72742]: 2024-07-26T12:52:39.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:38 smithi112 podman[73054]: 2024-07-26 12:52:38.998004986 +0000 UTC m=+0.025592026 container died 7a5e51c85408d7d4f59eaabd8cafceca70d74e755e2e4fac0c6ca2a235c40694 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-07-26T12:52:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:39.235+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:52:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:39.235+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:52:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:39.235+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:38 smithi132 ceph-mon[37580]: pgmap v596: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 10 op/s 2024-07-26T12:52:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:38 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:rejoin 2024-07-26T12:52:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:38 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:38 smithi132 ceph-mon[37580]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:52:39.945 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:39 smithi112 podman[73054]: 2024-07-26 12:52:39.667475846 +0000 UTC m=+0.695062878 container remove 7a5e51c85408d7d4f59eaabd8cafceca70d74e755e2e4fac0c6ca2a235c40694 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-07-26T12:52:39.945 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:39 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:52:39.945 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:39 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T12:52:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:39 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:39 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:52:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:39 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:active 2024-07-26T12:52:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:39 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:52:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:39 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:39 smithi112 ceph-mon[37501]: pgmap v597: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-07-26T12:52:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:39 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:39 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:52:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:39 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:active 2024-07-26T12:52:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:39 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:52:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:39 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:39 smithi107 ceph-mon[32659]: pgmap v597: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-07-26T12:52:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:39 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:39 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:52:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:39 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/2274412191,v1:172.21.15.112:6839/2274412191] up:active 2024-07-26T12:52:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:39 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:52:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:39 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:39 smithi132 ceph-mon[37580]: pgmap v597: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-07-26T12:52:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:42 smithi107 ceph-mon[32659]: pgmap v598: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:52:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:42 smithi132 ceph-mon[37580]: pgmap v598: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:52:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:42 smithi112 ceph-mon[37501]: pgmap v598: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:52:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:44 smithi107 ceph-mon[32659]: pgmap v599: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:52:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:44 smithi132 ceph-mon[37580]: pgmap v599: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:52:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:44 smithi112 ceph-mon[37501]: pgmap v599: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:52:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:46 smithi132 ceph-mon[37580]: pgmap v600: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 195 B/s wr, 15 op/s 2024-07-26T12:52:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:46 smithi112 ceph-mon[37501]: pgmap v600: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 195 B/s wr, 15 op/s 2024-07-26T12:52:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:46 smithi107 ceph-mon[32659]: pgmap v600: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 195 B/s wr, 15 op/s 2024-07-26T12:52:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:48 smithi107 ceph-mon[32659]: pgmap v601: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:52:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:48 smithi112 ceph-mon[37501]: pgmap v601: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:52:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:48 smithi132 ceph-mon[37580]: pgmap v601: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:52:50.210 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:50 smithi112 ceph-mon[37501]: pgmap v602: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:52:50.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:49 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 4. 2024-07-26T12:52:50.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:49 smithi112 systemd[1]: Stopped Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:52:50.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:49 smithi112 systemd[1]: Starting Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:52:50.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:50 smithi112 podman[73144]: 2024-07-26 12:52:50.085019864 +0000 UTC m=+0.017465814 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:52:50.475 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:50 smithi112 podman[73144]: 2024-07-26 12:52:50.210290251 +0000 UTC m=+0.142736200 container create 31ef795d81423331b8fae2fa2faf2065d78626e22a3c9cac562eb4b53fe2e6f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:52:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:50 smithi107 ceph-mon[32659]: pgmap v602: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:52:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:50 smithi132 ceph-mon[37580]: pgmap v602: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:52:50.950 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:50 smithi112 podman[73144]: 2024-07-26 12:52:50.50817266 +0000 UTC m=+0.440618610 container init 31ef795d81423331b8fae2fa2faf2065d78626e22a3c9cac562eb4b53fe2e6f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-26T12:52:50.950 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:50 smithi112 podman[73144]: 2024-07-26 12:52:50.51095653 +0000 UTC m=+0.443402481 container start 31ef795d81423331b8fae2fa2faf2065d78626e22a3c9cac562eb4b53fe2e6f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-07-26T12:52:50.951 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:50 smithi112 ceph-mds[73164]: -- 172.21.15.112:0/1072685886 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56457e637860 con 0x5645827d9a80 2024-07-26T12:52:50.951 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:50 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: starting mds.b at 2024-07-26T12:52:50.951 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:50 smithi112 bash[73144]: 31ef795d81423331b8fae2fa2faf2065d78626e22a3c9cac562eb4b53fe2e6f0 2024-07-26T12:52:50.951 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:52:50 smithi112 systemd[1]: Started Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:52:51.421 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:51 smithi112 ceph-mon[37501]: daemon mds.b restarted 2024-07-26T12:52:51.421 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:51 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:51.421 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:52:51.421 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:51 smithi112 ceph-mon[37501]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T12:52:51.422 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:51 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:boot 2024-07-26T12:52:51.422 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:51 smithi112 ceph-mon[37501]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:52:51.422 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:51 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:51.422 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:51 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:51 smithi107 ceph-mon[32659]: daemon mds.b restarted 2024-07-26T12:52:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:51 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:52:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:51 smithi107 ceph-mon[32659]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T12:52:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:51 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:boot 2024-07-26T12:52:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:51 smithi107 ceph-mon[32659]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:52:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:51 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:51 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:51 smithi132 ceph-mon[37580]: daemon mds.b restarted 2024-07-26T12:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:51 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T12:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:51 smithi132 ceph-mon[37580]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T12:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:51 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:boot 2024-07-26T12:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:51 smithi132 ceph-mon[37580]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-07-26T12:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:51 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:51 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:52:52.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:52.136+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:52.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:52.137+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T12:52:52.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:52.137+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:52.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:52.137+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:52.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:52:52.138+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:52.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:52 smithi107 ceph-mon[32659]: pgmap v604: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:52:52.852 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:52 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:resolve 2024-07-26T12:52:52.852 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:52 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:52 smithi112 ceph-mon[37501]: pgmap v604: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:52:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:52 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:resolve 2024-07-26T12:52:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:52 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:52 smithi132 ceph-mon[37580]: pgmap v604: 129 pgs: 129 active+clean; 38 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:52:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:52 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:resolve 2024-07-26T12:52:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:52 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:52:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:53 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:52:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:53 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:reconnect 2024-07-26T12:52:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:53 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:53 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109997 2024-07-26T12:52:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:53 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:52:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:53 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:reconnect 2024-07-26T12:52:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:53 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:53 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109997 2024-07-26T12:52:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:53 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:52:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:53 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:reconnect 2024-07-26T12:52:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:53 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:52:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:53 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109997 2024-07-26T12:52:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:54.250+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:52:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:54.251+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T12:52:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:54.252+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:52:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:54.252+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:52:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:52:54.252+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-mon[37501]: pgmap v605: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T12:52:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:rejoin 2024-07-26T12:52:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-mon[37501]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:52:54.951 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:52:54.951 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: *** Caught signal (Aborted) ** 2024-07-26T12:52:54.951 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: in thread 7f0ab949b640 thread_name:ms_dispatch 2024-07-26T12:52:54.951 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:54.951 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0abe8726f0] 2024-07-26T12:52:54.951 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0abe8bf94c] 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 3: raise() 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 4: abort() 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0abe85c71b] 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 6: /lib64/libc.so.6(+0x37386) [0x7f0abe86b386] 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559a6f5ba6b9] 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559a6f77880c] 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559a6f6c6cde] 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559a6f688763] 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559a6f689a1b] 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559a6f66d752] 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559a6f5bba56] 2024-07-26T12:52:54.952 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559a6f5bf637] 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559a6f5bff5e] 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559a6f5a303a] 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0abf13f407] 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 18: (DispatchQueue::entry()+0x60f) [0x7f0abf13fcf9] 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0abf21c5a1] 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 20: (Thread::entry_wrapper()+0x43) [0x7f0abefc9cdd] 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0abefc9cf9] 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0abe8bdc02] 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0abe942c40] 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2024-07-26T12:52:54.644+0000 7f0ab949b640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: in thread 7f0ab949b640 thread_name:ms_dispatch 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2024-07-26T12:52:54.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0abe8726f0] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0abe8bf94c] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 3: raise() 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 4: abort() 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0abe85c71b] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 6: /lib64/libc.so.6(+0x37386) [0x7f0abe86b386] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559a6f5ba6b9] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559a6f77880c] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559a6f6c6cde] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559a6f688763] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559a6f689a1b] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559a6f66d752] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559a6f5bba56] 2024-07-26T12:52:54.954 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559a6f5bf637] 2024-07-26T12:52:54.955 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559a6f5bff5e] 2024-07-26T12:52:54.955 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559a6f5a303a] 2024-07-26T12:52:54.955 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0abf13f407] 2024-07-26T12:52:54.955 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 18: (DispatchQueue::entry()+0x60f) [0x7f0abf13fcf9] 2024-07-26T12:52:54.955 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0abf21c5a1] 2024-07-26T12:52:54.955 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 20: (Thread::entry_wrapper()+0x43) [0x7f0abefc9cdd] 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0abefc9cf9] 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0abe8bdc02] 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0abe942c40] 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 0> 2024-07-26T12:52:54.644+0000 7f0ab949b640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: in thread 7f0ab949b640 thread_name:ms_dispatch 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0abe8726f0] 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0abe8bf94c] 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 3: raise() 2024-07-26T12:52:54.956 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 4: abort() 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0abe85c71b] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 6: /lib64/libc.so.6(+0x37386) [0x7f0abe86b386] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559a6f5ba6b9] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559a6f77880c] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559a6f6c6cde] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559a6f688763] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559a6f689a1b] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559a6f66d752] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559a6f5bba56] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559a6f5bf637] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559a6f5bff5e] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559a6f5a303a] 2024-07-26T12:52:54.957 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0abf13f407] 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 18: (DispatchQueue::entry()+0x60f) [0x7f0abf13fcf9] 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0abf21c5a1] 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 20: (Thread::entry_wrapper()+0x43) [0x7f0abefc9cdd] 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0abefc9cf9] 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0abe8bdc02] 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0abe942c40] 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 0> 2024-07-26T12:52:54.644+0000 7f0ab949b640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: in thread 7f0ab949b640 thread_name:ms_dispatch 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0abe8726f0] 2024-07-26T12:52:54.958 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0abe8bf94c] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 3: raise() 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 4: abort() 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0abe85c71b] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 6: /lib64/libc.so.6(+0x37386) [0x7f0abe86b386] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559a6f5ba6b9] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559a6f77880c] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559a6f6c6cde] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559a6f688763] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559a6f689a1b] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559a6f66d752] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559a6f5bba56] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559a6f5bf637] 2024-07-26T12:52:54.959 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559a6f5bff5e] 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559a6f5a303a] 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0abf13f407] 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 18: (DispatchQueue::entry()+0x60f) [0x7f0abf13fcf9] 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0abf21c5a1] 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 20: (Thread::entry_wrapper()+0x43) [0x7f0abefc9cdd] 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0abefc9cf9] 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0abe8bdc02] 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0abe942c40] 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73015]: 2024-07-26T12:52:54.960 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:54 smithi112 podman[73210]: 2024-07-26 12:52:54.704116064 +0000 UTC m=+0.019490422 container died 67dbe1ef11012bc8495deef798f814961c70c9360041b7037cad5c70efbeaf77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:52:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:54 smithi107 ceph-mon[32659]: pgmap v605: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T12:52:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:54 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:rejoin 2024-07-26T12:52:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:54 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:54 smithi107 ceph-mon[32659]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:52:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:54 smithi132 ceph-mon[37580]: pgmap v605: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T12:52:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:54 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:rejoin 2024-07-26T12:52:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:54 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:52:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:54 smithi132 ceph-mon[37580]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:52:55.622 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:55 smithi112 podman[73210]: 2024-07-26 12:52:55.343582408 +0000 UTC m=+0.658956765 container remove 67dbe1ef11012bc8495deef798f814961c70c9360041b7037cad5c70efbeaf77 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-26T12:52:55.622 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:55 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:52:55.622 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:52:55 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T12:52:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:55 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:55 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:52:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:55 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:active 2024-07-26T12:52:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:55 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:52:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:55 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:55 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:55 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:52:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:55 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:active 2024-07-26T12:52:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:55 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:52:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:55 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:55 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:52:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:55 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:52:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:55 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/1036393836,v1:172.21.15.112:6837/1036393836] up:active 2024-07-26T12:52:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:55 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:52:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:55 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:52:56.924 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:56 smithi107 ceph-mon[32659]: pgmap v606: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:52:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:56 smithi112 ceph-mon[37501]: pgmap v606: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:52:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:56 smithi132 ceph-mon[37580]: pgmap v606: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:52:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:52:58 smithi112 ceph-mon[37501]: pgmap v607: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 13 op/s 2024-07-26T12:52:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:52:58 smithi107 ceph-mon[32659]: pgmap v607: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 13 op/s 2024-07-26T12:52:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:52:58 smithi132 ceph-mon[37580]: pgmap v607: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 13 op/s 2024-07-26T12:53:00.914 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:00 smithi112 ceph-mon[37501]: pgmap v608: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:00 smithi107 ceph-mon[32659]: pgmap v608: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:00 smithi132 ceph-mon[37580]: pgmap v608: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:02 smithi112 ceph-mon[37501]: pgmap v609: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 196 B/s wr, 15 op/s 2024-07-26T12:53:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:02 smithi107 ceph-mon[32659]: pgmap v609: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 196 B/s wr, 15 op/s 2024-07-26T12:53:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:02 smithi132 ceph-mon[37580]: pgmap v609: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 196 B/s wr, 15 op/s 2024-07-26T12:53:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:04 smithi112 ceph-mon[37501]: pgmap v610: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:04 smithi107 ceph-mon[32659]: pgmap v610: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:04 smithi132 ceph-mon[37580]: pgmap v610: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:05.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:05 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 5. 2024-07-26T12:53:05.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:05 smithi112 systemd[1]: Stopped Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:53:05.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:05 smithi112 systemd[1]: Starting Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:53:05.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:05 smithi112 podman[73301]: 2024-07-26 12:53:05.837791191 +0000 UTC m=+0.017483448 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:53:05.953 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:05 smithi112 podman[73301]: 2024-07-26 12:53:05.952776772 +0000 UTC m=+0.132469019 container create 17b8e915547c306d2387a55a89c19165a2109cc9236a57598e9fbb1cac81906d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:53:06.232 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:06 smithi112 podman[73301]: 2024-07-26 12:53:06.197518193 +0000 UTC m=+0.377210449 container init 17b8e915547c306d2387a55a89c19165a2109cc9236a57598e9fbb1cac81906d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-07-26T12:53:06.232 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:06 smithi112 podman[73301]: 2024-07-26 12:53:06.200344012 +0000 UTC m=+0.380036259 container start 17b8e915547c306d2387a55a89c19165a2109cc9236a57598e9fbb1cac81906d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-07-26T12:53:06.681 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-mds[73320]: -- 172.21.15.112:0/1153955479 <== mon.2 v2:172.21.15.112:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558e0966d860 con 0x558e0d80fa80 2024-07-26T12:53:06.681 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: starting mds.h at 2024-07-26T12:53:06.681 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:06 smithi112 bash[73301]: 17b8e915547c306d2387a55a89c19165a2109cc9236a57598e9fbb1cac81906d 2024-07-26T12:53:06.681 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:06 smithi112 systemd[1]: Started Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:53:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-mon[37501]: pgmap v611: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:53:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-mon[37501]: daemon mds.h restarted 2024-07-26T12:53:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:53:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:53:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-mon[37501]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T12:53:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:boot 2024-07-26T12:53:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-mon[37501]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:53:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:53:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:06 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:53:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:06 smithi107 ceph-mon[32659]: pgmap v611: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:53:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:06 smithi107 ceph-mon[32659]: daemon mds.h restarted 2024-07-26T12:53:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:06 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:53:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:53:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:06 smithi107 ceph-mon[32659]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T12:53:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:06 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:boot 2024-07-26T12:53:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:06 smithi107 ceph-mon[32659]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:53:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:06 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:53:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:06 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:53:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:06 smithi132 ceph-mon[37580]: pgmap v611: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:53:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:06 smithi132 ceph-mon[37580]: daemon mds.h restarted 2024-07-26T12:53:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:06 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:53:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T12:53:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:06 smithi132 ceph-mon[37580]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T12:53:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:06 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:boot 2024-07-26T12:53:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:06 smithi132 ceph-mon[37580]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-07-26T12:53:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:06 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T12:53:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:06 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T12:53:07.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:07.147+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:53:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:07.149+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T12:53:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:07.149+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:53:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:07.149+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:53:07.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:07.149+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:08.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:08 smithi107 ceph-mon[32659]: pgmap v613: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:53:08.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:08 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:resolve 2024-07-26T12:53:08.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:08 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:53:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:08 smithi132 ceph-mon[37580]: pgmap v613: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:53:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:08 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:resolve 2024-07-26T12:53:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:08 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:53:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:08 smithi112 ceph-mon[37501]: pgmap v613: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:53:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:08 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:resolve 2024-07-26T12:53:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:08 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T12:53:09.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:09.265+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:53:09.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:09.267+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T12:53:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:09.267+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T12:53:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:09.267+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:53:09.269 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:09.267+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:09 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:53:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:09 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:reconnect 2024-07-26T12:53:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:09 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:53:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:09 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:53:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:09 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:53:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:09 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:reconnect 2024-07-26T12:53:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:09 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:53:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:09 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:53:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:09 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:53:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:09 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:reconnect 2024-07-26T12:53:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:09 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:53:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:09 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:53:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:10 smithi107 ceph-mon[32659]: pgmap v614: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:10 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:rejoin 2024-07-26T12:53:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:10 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:53:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:10 smithi107 ceph-mon[32659]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:53:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:10 smithi132 ceph-mon[37580]: pgmap v614: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:10 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:rejoin 2024-07-26T12:53:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:10 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:53:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:10 smithi132 ceph-mon[37580]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:53:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-mon[37501]: pgmap v614: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:rejoin 2024-07-26T12:53:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:53:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-mon[37501]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-07-26T12:53:11.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:53:11.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: *** Caught signal (Aborted) ** 2024-07-26T12:53:11.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: in thread 7f41da422640 thread_name:ms_dispatch 2024-07-26T12:53:11.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:11.201 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f41df7f96f0] 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f41df84694c] 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 3: raise() 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 4: abort() 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 5: /lib64/libc.so.6(+0x2871b) [0x7f41df7e371b] 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 6: /lib64/libc.so.6(+0x37386) [0x7f41df7f2386] 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56457d3606b9] 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56457d51e80c] 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56457d46ccde] 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56457d42e763] 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56457d42fa1b] 2024-07-26T12:53:11.202 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56457d413752] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56457d361a56] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56457d365637] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56457d365f5e] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56457d34903a] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f41e00c6407] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 18: (DispatchQueue::entry()+0x60f) [0x7f41e00c6cf9] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f41e01a35a1] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 20: (Thread::entry_wrapper()+0x43) [0x7f41dff50cdd] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 21: (Thread::_entry_func(void*)+0xd) [0x7f41dff50cf9] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 22: /lib64/libc.so.6(+0x89c02) [0x7f41df844c02] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f41df8c9c40] 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2024-07-26T12:53:10.748+0000 7f41da422640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:53:11.203 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: in thread 7f41da422640 thread_name:ms_dispatch 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f41df7f96f0] 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f41df84694c] 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 3: raise() 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 4: abort() 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 5: /lib64/libc.so.6(+0x2871b) [0x7f41df7e371b] 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 6: /lib64/libc.so.6(+0x37386) [0x7f41df7f2386] 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56457d3606b9] 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56457d51e80c] 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56457d46ccde] 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56457d42e763] 2024-07-26T12:53:11.204 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56457d42fa1b] 2024-07-26T12:53:11.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56457d413752] 2024-07-26T12:53:11.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56457d361a56] 2024-07-26T12:53:11.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56457d365637] 2024-07-26T12:53:11.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56457d365f5e] 2024-07-26T12:53:11.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56457d34903a] 2024-07-26T12:53:11.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f41e00c6407] 2024-07-26T12:53:11.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 18: (DispatchQueue::entry()+0x60f) [0x7f41e00c6cf9] 2024-07-26T12:53:11.205 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f41e01a35a1] 2024-07-26T12:53:11.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 20: (Thread::entry_wrapper()+0x43) [0x7f41dff50cdd] 2024-07-26T12:53:11.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 21: (Thread::_entry_func(void*)+0xd) [0x7f41dff50cf9] 2024-07-26T12:53:11.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 22: /lib64/libc.so.6(+0x89c02) [0x7f41df844c02] 2024-07-26T12:53:11.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f41df8c9c40] 2024-07-26T12:53:11.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:53:11.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2024-07-26T12:53:11.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 0> 2024-07-26T12:53:10.748+0000 7f41da422640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:53:11.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: in thread 7f41da422640 thread_name:ms_dispatch 2024-07-26T12:53:11.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2024-07-26T12:53:11.206 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f41df7f96f0] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f41df84694c] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 3: raise() 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 4: abort() 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 5: /lib64/libc.so.6(+0x2871b) [0x7f41df7e371b] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 6: /lib64/libc.so.6(+0x37386) [0x7f41df7f2386] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56457d3606b9] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56457d51e80c] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56457d46ccde] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56457d42e763] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56457d42fa1b] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56457d413752] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56457d361a56] 2024-07-26T12:53:11.207 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56457d365637] 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56457d365f5e] 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56457d34903a] 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f41e00c6407] 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 18: (DispatchQueue::entry()+0x60f) [0x7f41e00c6cf9] 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f41e01a35a1] 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 20: (Thread::entry_wrapper()+0x43) [0x7f41dff50cdd] 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 21: (Thread::_entry_func(void*)+0xd) [0x7f41dff50cf9] 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 22: /lib64/libc.so.6(+0x89c02) [0x7f41df844c02] 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f41df8c9c40] 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 0> 2024-07-26T12:53:10.748+0000 7f41da422640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:53:11.208 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: in thread 7f41da422640 thread_name:ms_dispatch 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f41df7f96f0] 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f41df84694c] 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 3: raise() 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 4: abort() 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 5: /lib64/libc.so.6(+0x2871b) [0x7f41df7e371b] 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 6: /lib64/libc.so.6(+0x37386) [0x7f41df7f2386] 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56457d3606b9] 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56457d51e80c] 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56457d46ccde] 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56457d42e763] 2024-07-26T12:53:11.209 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56457d42fa1b] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56457d413752] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56457d361a56] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56457d365637] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56457d365f5e] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56457d34903a] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f41e00c6407] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 18: (DispatchQueue::entry()+0x60f) [0x7f41e00c6cf9] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f41e01a35a1] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 20: (Thread::entry_wrapper()+0x43) [0x7f41dff50cdd] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 21: (Thread::_entry_func(void*)+0xd) [0x7f41dff50cf9] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 22: /lib64/libc.so.6(+0x89c02) [0x7f41df844c02] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f41df8c9c40] 2024-07-26T12:53:11.210 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:53:11.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b[73160]: 2024-07-26T12:53:11.211 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:10 smithi112 podman[73355]: 2024-07-26 12:53:10.807602022 +0000 UTC m=+0.019663052 container died 31ef795d81423331b8fae2fa2faf2065d78626e22a3c9cac562eb4b53fe2e6f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-07-26T12:53:11.928 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:11 smithi112 podman[73355]: 2024-07-26 12:53:11.669561889 +0000 UTC m=+0.881622902 container remove 31ef795d81423331b8fae2fa2faf2065d78626e22a3c9cac562eb4b53fe2e6f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-b, org.label-schema.build-date=20240724, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-07-26T12:53:11.929 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:11 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:53:11.929 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:11 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:53:11.929 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:11 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:53:11.929 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:11 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:active 2024-07-26T12:53:11.929 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:11 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:53:11.929 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:11 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:53:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:11 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:53:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:11 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:53:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:11 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:active 2024-07-26T12:53:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:11 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:53:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:11 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:53:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:11 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:53:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:11 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:53:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:11 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6838/504240926,v1:172.21.15.112:6839/504240926] up:active 2024-07-26T12:53:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:11 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T12:53:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:11 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:53:12.200 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:11 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T12:53:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:12 smithi107 ceph-mon[32659]: pgmap v615: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:12 smithi132 ceph-mon[37580]: pgmap v615: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:12 smithi112 ceph-mon[37501]: pgmap v615: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:14 smithi107 ceph-mon[32659]: pgmap v616: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:14 smithi132 ceph-mon[37580]: pgmap v616: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:14 smithi112 ceph-mon[37501]: pgmap v616: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:16 smithi107 ceph-mon[32659]: pgmap v617: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:53:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:16 smithi132 ceph-mon[37580]: pgmap v617: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:53:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:16 smithi112 ceph-mon[37501]: pgmap v617: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:53:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:18 smithi107 ceph-mon[32659]: pgmap v618: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 190 B/s wr, 16 op/s 2024-07-26T12:53:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:18 smithi132 ceph-mon[37580]: pgmap v618: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 190 B/s wr, 16 op/s 2024-07-26T12:53:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:18 smithi112 ceph-mon[37501]: pgmap v618: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 190 B/s wr, 16 op/s 2024-07-26T12:53:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:20 smithi107 ceph-mon[32659]: pgmap v619: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:20 smithi132 ceph-mon[37580]: pgmap v619: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:20 smithi112 ceph-mon[37501]: pgmap v619: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:22.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:22.154+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:22.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:22.154+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:22.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:22.154+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:22.450 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:21 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 5. 2024-07-26T12:53:22.450 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:21 smithi112 systemd[1]: Stopped Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:53:22.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:21 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Start request repeated too quickly. 2024-07-26T12:53:22.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:21 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T12:53:22.451 INFO:journalctl@ceph.mds.b.smithi112.stdout:Jul 26 12:53:21 smithi112 systemd[1]: Failed to start Ceph mds.b for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:53:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:22 smithi107 ceph-mon[32659]: pgmap v620: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:53:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:22 smithi132 ceph-mon[37580]: pgmap v620: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:53:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:22 smithi112 ceph-mon[37501]: pgmap v620: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:53:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:24.275+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:24.275+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:24.277 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:24.275+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:24 smithi107 ceph-mon[32659]: pgmap v621: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:24 smithi132 ceph-mon[37580]: pgmap v621: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:24 smithi112 ceph-mon[37501]: pgmap v621: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:26 smithi132 ceph-mon[37580]: pgmap v622: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-07-26T12:53:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:26 smithi132 ceph-mon[37580]: Replacing daemon mds.b as rank 0 with standby daemon mds.h 2024-07-26T12:53:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:26 smithi132 ceph-mon[37580]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T12:53:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:26 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:53:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:26 smithi132 ceph-mon[37580]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T12:53:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:26 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T12:53:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:26 smithi112 ceph-mon[37501]: pgmap v622: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-07-26T12:53:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:26 smithi112 ceph-mon[37501]: Replacing daemon mds.b as rank 0 with standby daemon mds.h 2024-07-26T12:53:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:26 smithi112 ceph-mon[37501]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T12:53:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:26 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:53:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:26 smithi112 ceph-mon[37501]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T12:53:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:26 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T12:53:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:26 smithi107 ceph-mon[32659]: pgmap v622: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-07-26T12:53:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:26 smithi107 ceph-mon[32659]: Replacing daemon mds.b as rank 0 with standby daemon mds.h 2024-07-26T12:53:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:26 smithi107 ceph-mon[32659]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T12:53:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:26 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:53:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:26 smithi107 ceph-mon[32659]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T12:53:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:26 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T12:53:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:29.069 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:28 smithi107 ceph-mon[32659]: pgmap v624: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:29.069 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:28 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:resolve 2024-07-26T12:53:29.069 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:28 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T12:53:29.069 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:29.069 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:29.069 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:28 smithi112 ceph-mon[37501]: pgmap v624: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:28 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:resolve 2024-07-26T12:53:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:28 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T12:53:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:28 smithi132 ceph-mon[37580]: pgmap v624: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:28 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:resolve 2024-07-26T12:53:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:28 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T12:53:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:53:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:29 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:53:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:29 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:reconnect 2024-07-26T12:53:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:29 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:53:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:29 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:53:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:29 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:53:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:29 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:reconnect 2024-07-26T12:53:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:29 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:53:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:29 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:53:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:29 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199995 2024-07-26T12:53:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:29 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:reconnect 2024-07-26T12:53:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:29 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:53:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:29 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999974 2024-07-26T12:53:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-mon[37501]: pgmap v625: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:53:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:rejoin 2024-07-26T12:53:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:53:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-mon[37501]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:53:31.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:53:31.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: *** Caught signal (Aborted) ** 2024-07-26T12:53:31.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: in thread 7fec80605640 thread_name:ms_dispatch 2024-07-26T12:53:31.201 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fec859dc6f0] 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fec85a2994c] 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 3: raise() 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 4: abort() 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 5: /lib64/libc.so.6(+0x2871b) [0x7fec859c671b] 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 6: /lib64/libc.so.6(+0x37386) [0x7fec859d5386] 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558e06a176b9] 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558e06bd580c] 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x558e06b23cde] 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x558e06ae5763] 2024-07-26T12:53:31.202 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558e06ae6a1b] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558e06aca752] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558e06a18a56] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558e06a1c637] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558e06a1cf5e] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558e06a0003a] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fec862a9407] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 18: (DispatchQueue::entry()+0x60f) [0x7fec862a9cf9] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fec863865a1] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 20: (Thread::entry_wrapper()+0x43) [0x7fec86133cdd] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 21: (Thread::_entry_func(void*)+0xd) [0x7fec86133cf9] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 22: /lib64/libc.so.6(+0x89c02) [0x7fec85a27c02] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fec85aacc40] 2024-07-26T12:53:31.203 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2024-07-26T12:53:30.850+0000 7fec80605640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: in thread 7fec80605640 thread_name:ms_dispatch 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fec859dc6f0] 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fec85a2994c] 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 3: raise() 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 4: abort() 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 5: /lib64/libc.so.6(+0x2871b) [0x7fec859c671b] 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 6: /lib64/libc.so.6(+0x37386) [0x7fec859d5386] 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558e06a176b9] 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558e06bd580c] 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x558e06b23cde] 2024-07-26T12:53:31.204 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x558e06ae5763] 2024-07-26T12:53:31.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558e06ae6a1b] 2024-07-26T12:53:31.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558e06aca752] 2024-07-26T12:53:31.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558e06a18a56] 2024-07-26T12:53:31.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558e06a1c637] 2024-07-26T12:53:31.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558e06a1cf5e] 2024-07-26T12:53:31.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558e06a0003a] 2024-07-26T12:53:31.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fec862a9407] 2024-07-26T12:53:31.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 18: (DispatchQueue::entry()+0x60f) [0x7fec862a9cf9] 2024-07-26T12:53:31.205 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fec863865a1] 2024-07-26T12:53:31.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 20: (Thread::entry_wrapper()+0x43) [0x7fec86133cdd] 2024-07-26T12:53:31.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 21: (Thread::_entry_func(void*)+0xd) [0x7fec86133cf9] 2024-07-26T12:53:31.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 22: /lib64/libc.so.6(+0x89c02) [0x7fec85a27c02] 2024-07-26T12:53:31.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fec85aacc40] 2024-07-26T12:53:31.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:53:31.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2024-07-26T12:53:31.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 0> 2024-07-26T12:53:30.850+0000 7fec80605640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:53:31.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: in thread 7fec80605640 thread_name:ms_dispatch 2024-07-26T12:53:31.206 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fec859dc6f0] 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fec85a2994c] 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 3: raise() 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 4: abort() 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 5: /lib64/libc.so.6(+0x2871b) [0x7fec859c671b] 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 6: /lib64/libc.so.6(+0x37386) [0x7fec859d5386] 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558e06a176b9] 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558e06bd580c] 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x558e06b23cde] 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x558e06ae5763] 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558e06ae6a1b] 2024-07-26T12:53:31.207 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558e06aca752] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558e06a18a56] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558e06a1c637] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558e06a1cf5e] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558e06a0003a] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fec862a9407] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 18: (DispatchQueue::entry()+0x60f) [0x7fec862a9cf9] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fec863865a1] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 20: (Thread::entry_wrapper()+0x43) [0x7fec86133cdd] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 21: (Thread::_entry_func(void*)+0xd) [0x7fec86133cf9] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 22: /lib64/libc.so.6(+0x89c02) [0x7fec85a27c02] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fec85aacc40] 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:53:31.208 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2024-07-26T12:53:31.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 0> 2024-07-26T12:53:30.850+0000 7fec80605640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:53:31.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: in thread 7fec80605640 thread_name:ms_dispatch 2024-07-26T12:53:31.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2024-07-26T12:53:31.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:31.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fec859dc6f0] 2024-07-26T12:53:31.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fec85a2994c] 2024-07-26T12:53:31.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 3: raise() 2024-07-26T12:53:31.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 4: abort() 2024-07-26T12:53:31.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 5: /lib64/libc.so.6(+0x2871b) [0x7fec859c671b] 2024-07-26T12:53:31.209 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 6: /lib64/libc.so.6(+0x37386) [0x7fec859d5386] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558e06a176b9] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558e06bd580c] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x558e06b23cde] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x558e06ae5763] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558e06ae6a1b] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558e06aca752] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558e06a18a56] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558e06a1c637] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558e06a1cf5e] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558e06a0003a] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fec862a9407] 2024-07-26T12:53:31.210 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 18: (DispatchQueue::entry()+0x60f) [0x7fec862a9cf9] 2024-07-26T12:53:31.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fec863865a1] 2024-07-26T12:53:31.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 20: (Thread::entry_wrapper()+0x43) [0x7fec86133cdd] 2024-07-26T12:53:31.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 21: (Thread::_entry_func(void*)+0xd) [0x7fec86133cf9] 2024-07-26T12:53:31.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 22: /lib64/libc.so.6(+0x89c02) [0x7fec85a27c02] 2024-07-26T12:53:31.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fec85aacc40] 2024-07-26T12:53:31.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:53:31.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h[73316]: 2024-07-26T12:53:31.211 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:30 smithi112 podman[73530]: 2024-07-26 12:53:30.909364623 +0000 UTC m=+0.019496942 container died 17b8e915547c306d2387a55a89c19165a2109cc9236a57598e9fbb1cac81906d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-07-26T12:53:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:30 smithi107 ceph-mon[32659]: pgmap v625: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:53:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:30 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:rejoin 2024-07-26T12:53:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:30 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:53:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:30 smithi107 ceph-mon[32659]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:53:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:30 smithi132 ceph-mon[37580]: pgmap v625: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:53:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:30 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:rejoin 2024-07-26T12:53:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:30 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:53:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:30 smithi132 ceph-mon[37580]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-07-26T12:53:31.816 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:31 smithi112 podman[73530]: 2024-07-26 12:53:31.55779747 +0000 UTC m=+0.667929820 container remove 17b8e915547c306d2387a55a89c19165a2109cc9236a57598e9fbb1cac81906d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:53:31.816 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:31 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:53:32.200 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:31 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T12:53:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:31 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:53:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:31 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:53:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:31 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:active 2024-07-26T12:53:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:31 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:53:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:31 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:53:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:31 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:53:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:31 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:53:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:31 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:active 2024-07-26T12:53:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:31 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:53:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:31 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:53:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:31 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:53:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:31 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:53:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:31 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.112:6836/422754403,v1:172.21.15.112:6837/422754403] up:active 2024-07-26T12:53:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:31 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T12:53:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:31 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:53:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:32 smithi112 ceph-mon[37501]: pgmap v626: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:32 smithi107 ceph-mon[32659]: pgmap v626: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:32 smithi132 ceph-mon[37580]: pgmap v626: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:34 smithi112 ceph-mon[37501]: pgmap v627: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:34 smithi107 ceph-mon[32659]: pgmap v627: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:34 smithi132 ceph-mon[37580]: pgmap v627: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:37.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:37.170+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:37.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:37.170+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:37.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:37.170+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:37.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:36 smithi107 ceph-mon[32659]: pgmap v628: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:53:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:36 smithi112 ceph-mon[37501]: pgmap v628: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:53:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:36 smithi132 ceph-mon[37580]: pgmap v628: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:53:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:38 smithi112 ceph-mon[37501]: pgmap v629: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 198 B/s wr, 17 op/s 2024-07-26T12:53:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:38 smithi107 ceph-mon[32659]: pgmap v629: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 198 B/s wr, 17 op/s 2024-07-26T12:53:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:39.292+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:39.292+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:39.292+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:38 smithi132 ceph-mon[37580]: pgmap v629: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 198 B/s wr, 17 op/s 2024-07-26T12:53:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:40 smithi112 ceph-mon[37501]: pgmap v630: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:40 smithi107 ceph-mon[32659]: pgmap v630: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:40 smithi132 ceph-mon[37580]: pgmap v630: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:42.450 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:41 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 6. 2024-07-26T12:53:42.451 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:41 smithi112 systemd[1]: Stopped Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:53:42.451 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:41 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Start request repeated too quickly. 2024-07-26T12:53:42.451 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:41 smithi112 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T12:53:42.451 INFO:journalctl@ceph.mds.h.smithi112.stdout:Jul 26 12:53:41 smithi112 systemd[1]: Failed to start Ceph mds.h for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:53:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:42 smithi112 ceph-mon[37501]: pgmap v631: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:53:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:42 smithi107 ceph-mon[32659]: pgmap v631: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:53:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:42 smithi132 ceph-mon[37580]: pgmap v631: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:53:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:44 smithi112 ceph-mon[37501]: pgmap v632: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:44 smithi107 ceph-mon[32659]: pgmap v632: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:44 smithi132 ceph-mon[37580]: pgmap v632: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:53:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:46 smithi112 ceph-mon[37501]: pgmap v633: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-07-26T12:53:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:46 smithi112 ceph-mon[37501]: Replacing daemon mds.h as rank 0 with standby daemon mds.a 2024-07-26T12:53:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:46 smithi112 ceph-mon[37501]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T12:53:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:46 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:53:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:46 smithi112 ceph-mon[37501]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T12:53:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:46 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:53:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:46 smithi107 ceph-mon[32659]: pgmap v633: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-07-26T12:53:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:46 smithi107 ceph-mon[32659]: Replacing daemon mds.h as rank 0 with standby daemon mds.a 2024-07-26T12:53:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:46 smithi107 ceph-mon[32659]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T12:53:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:46 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:53:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:46 smithi107 ceph-mon[32659]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T12:53:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:46 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:53:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:46 smithi132 ceph-mon[37580]: pgmap v633: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-07-26T12:53:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:46 smithi132 ceph-mon[37580]: Replacing daemon mds.h as rank 0 with standby daemon mds.a 2024-07-26T12:53:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:46 smithi132 ceph-mon[37580]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T12:53:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:46 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:53:47.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:46 smithi132 ceph-mon[37580]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T12:53:47.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:46 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:53:49.174 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:48 smithi107 ceph-mon[32659]: pgmap v635: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:49.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:48 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:resolve 2024-07-26T12:53:49.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:48 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:53:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:48 smithi112 ceph-mon[37501]: pgmap v635: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:48 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:resolve 2024-07-26T12:53:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:48 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:53:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:48 smithi132 ceph-mon[37580]: pgmap v635: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:48 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:resolve 2024-07-26T12:53:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:48 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:53:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:49 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:53:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:49 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:reconnect 2024-07-26T12:53:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:49 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:53:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:49 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:53:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:49 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:53:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:49 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:reconnect 2024-07-26T12:53:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:49 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:53:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:49 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:53:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:49 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:53:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:49 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:reconnect 2024-07-26T12:53:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:49 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:53:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:49 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:53:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:50 smithi112 ceph-mon[37501]: pgmap v636: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:53:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:50 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:rejoin 2024-07-26T12:53:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:50 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:53:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:50 smithi112 ceph-mon[37501]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:53:51.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-mon[32659]: pgmap v636: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:53:51.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:rejoin 2024-07-26T12:53:51.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:53:51.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-mon[32659]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:53:51.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:53:51.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: *** Caught signal (Aborted) ** 2024-07-26T12:53:51.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: in thread 7f651b600640 thread_name:ms_dispatch 2024-07-26T12:53:51.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:51.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f652183e6f0] 2024-07-26T12:53:51.246 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f652188b94c] 2024-07-26T12:53:51.246 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 3: raise() 2024-07-26T12:53:51.246 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 4: abort() 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 5: /lib64/libc.so.6(+0x2871b) [0x7f652182871b] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 6: /lib64/libc.so.6(+0x37386) [0x7f6521837386] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5555e28436b9] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5555e2a0180c] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5555e294fcde] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5555e2911763] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5555e2912a1b] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5555e28f6752] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5555e2844a56] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5555e2848637] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5555e2848f5e] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5555e282c03a] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f652239b407] 2024-07-26T12:53:51.247 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 18: (DispatchQueue::entry()+0x60f) [0x7f652239bcf9] 2024-07-26T12:53:51.248 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f65224785a1] 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 20: (Thread::entry_wrapper()+0x43) [0x7f6522225cdd] 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6522225cf9] 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6521889c02] 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f652190ec40] 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2024-07-26T12:53:50.986+0000 7f651b600640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: in thread 7f651b600640 thread_name:ms_dispatch 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f652183e6f0] 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f652188b94c] 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 3: raise() 2024-07-26T12:53:51.249 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 4: abort() 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 5: /lib64/libc.so.6(+0x2871b) [0x7f652182871b] 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 6: /lib64/libc.so.6(+0x37386) [0x7f6521837386] 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5555e28436b9] 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5555e2a0180c] 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5555e294fcde] 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5555e2911763] 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5555e2912a1b] 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5555e28f6752] 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5555e2844a56] 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5555e2848637] 2024-07-26T12:53:51.250 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5555e2848f5e] 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5555e282c03a] 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f652239b407] 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 18: (DispatchQueue::entry()+0x60f) [0x7f652239bcf9] 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f65224785a1] 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 20: (Thread::entry_wrapper()+0x43) [0x7f6522225cdd] 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6522225cf9] 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6521889c02] 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f652190ec40] 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:50 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 0> 2024-07-26T12:53:50.986+0000 7f651b600640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: in thread 7f651b600640 thread_name:ms_dispatch 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2024-07-26T12:53:51.251 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f652183e6f0] 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f652188b94c] 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 3: raise() 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 4: abort() 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 5: /lib64/libc.so.6(+0x2871b) [0x7f652182871b] 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 6: /lib64/libc.so.6(+0x37386) [0x7f6521837386] 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5555e28436b9] 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5555e2a0180c] 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5555e294fcde] 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5555e2911763] 2024-07-26T12:53:51.252 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5555e2912a1b] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5555e28f6752] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5555e2844a56] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5555e2848637] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5555e2848f5e] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5555e282c03a] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f652239b407] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 18: (DispatchQueue::entry()+0x60f) [0x7f652239bcf9] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f65224785a1] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 20: (Thread::entry_wrapper()+0x43) [0x7f6522225cdd] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6522225cf9] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6521889c02] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f652190ec40] 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:53:51.253 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2024-07-26T12:53:51.254 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 0> 2024-07-26T12:53:50.986+0000 7f651b600640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:53:51.254 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: in thread 7f651b600640 thread_name:ms_dispatch 2024-07-26T12:53:51.254 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2024-07-26T12:53:51.254 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:53:51.254 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f652183e6f0] 2024-07-26T12:53:51.254 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f652188b94c] 2024-07-26T12:53:51.254 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 3: raise() 2024-07-26T12:53:51.254 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 4: abort() 2024-07-26T12:53:51.254 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 5: /lib64/libc.so.6(+0x2871b) [0x7f652182871b] 2024-07-26T12:53:51.254 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 6: /lib64/libc.so.6(+0x37386) [0x7f6521837386] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5555e28436b9] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5555e2a0180c] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5555e294fcde] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5555e2911763] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5555e2912a1b] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5555e28f6752] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5555e2844a56] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5555e2848637] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5555e2848f5e] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5555e282c03a] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f652239b407] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 18: (DispatchQueue::entry()+0x60f) [0x7f652239bcf9] 2024-07-26T12:53:51.255 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f65224785a1] 2024-07-26T12:53:51.256 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 20: (Thread::entry_wrapper()+0x43) [0x7f6522225cdd] 2024-07-26T12:53:51.256 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6522225cf9] 2024-07-26T12:53:51.256 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6521889c02] 2024-07-26T12:53:51.256 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f652190ec40] 2024-07-26T12:53:51.256 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:53:51.256 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[77554]: 2024-07-26T12:53:51.256 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 podman[103805]: 2024-07-26 12:53:51.113313939 +0000 UTC m=+0.032552144 container died a91deb8200a253abaae47b4ed76dc2ca433b31ed0dd4752d8119c7f3cfda8cd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:53:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:50 smithi132 ceph-mon[37580]: pgmap v636: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:53:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:50 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:rejoin 2024-07-26T12:53:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:50 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:53:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:50 smithi132 ceph-mon[37580]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:53:52.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:52.186+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:52.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:52.186+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:52.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:53:52.186+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:51 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:53:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:51 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:53:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:51 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:active 2024-07-26T12:53:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:51 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:53:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:51 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:53:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:51 smithi112 ceph-mon[37501]: pgmap v637: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:52.292 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 podman[103805]: 2024-07-26 12:53:51.967956247 +0000 UTC m=+0.887194494 container remove a91deb8200a253abaae47b4ed76dc2ca433b31ed0dd4752d8119c7f3cfda8cd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-07-26T12:53:52.292 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:51 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:53:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:53:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:53:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:active 2024-07-26T12:53:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:53:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:53:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:51 smithi107 ceph-mon[32659]: pgmap v637: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:51 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:53:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:51 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:53:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:51 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/300465136,v1:172.21.15.107:6841/300465136] up:active 2024-07-26T12:53:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:51 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:53:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:51 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:53:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:51 smithi132 ceph-mon[37580]: pgmap v637: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:53:52.792 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:52 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T12:53:52.793 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:53:52 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Consumed 2.350s CPU time. 2024-07-26T12:53:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:54.307+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:54.307+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:53:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:53:54.307+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:54 smithi112 ceph-mon[37501]: pgmap v638: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:54 smithi107 ceph-mon[32659]: pgmap v638: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:54 smithi132 ceph-mon[37580]: pgmap v638: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:53:56.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:56 smithi112 ceph-mon[37501]: pgmap v639: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:53:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:56 smithi107 ceph-mon[32659]: pgmap v639: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:53:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:56 smithi132 ceph-mon[37580]: pgmap v639: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:53:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:53:58 smithi112 ceph-mon[37501]: pgmap v640: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 198 B/s wr, 16 op/s 2024-07-26T12:53:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:53:58 smithi107 ceph-mon[32659]: pgmap v640: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 198 B/s wr, 16 op/s 2024-07-26T12:53:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:53:58 smithi132 ceph-mon[37580]: pgmap v640: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 198 B/s wr, 16 op/s 2024-07-26T12:54:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:00 smithi112 ceph-mon[37501]: pgmap v641: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:00 smithi107 ceph-mon[32659]: pgmap v641: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:00.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:00 smithi132 ceph-mon[37580]: pgmap v641: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:02.377 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:02 smithi112 ceph-mon[37501]: pgmap v642: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:54:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:02 smithi107 ceph-mon[32659]: pgmap v642: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:54:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:02 smithi132 ceph-mon[37580]: pgmap v642: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:54:02.926 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:02 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-07-26T12:54:02.926 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:02 smithi107 systemd[1]: Stopped Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:54:02.926 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:02 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Consumed 2.350s CPU time. 2024-07-26T12:54:02.926 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:02 smithi107 systemd[1]: Starting Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:54:03.183 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:02 smithi107 podman[103898]: 2024-07-26 12:54:02.828903733 +0000 UTC m=+0.031244500 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:54:03.184 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:02 smithi107 podman[103898]: 2024-07-26 12:54:02.963376174 +0000 UTC m=+0.165716996 container create d0cb3de7bf040e7d6b8e67bd8bc7e486c3d9a027b712bbb7e8ab425caa1ebb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, org.label-schema.build-date=20240724, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-26T12:54:03.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:03 smithi107 podman[103898]: 2024-07-26 12:54:03.288130179 +0000 UTC m=+0.490470988 container init d0cb3de7bf040e7d6b8e67bd8bc7e486c3d9a027b712bbb7e8ab425caa1ebb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240724, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-07-26T12:54:03.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:03 smithi107 podman[103898]: 2024-07-26 12:54:03.298174285 +0000 UTC m=+0.500515069 container start d0cb3de7bf040e7d6b8e67bd8bc7e486c3d9a027b712bbb7e8ab425caa1ebb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:54:03.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:03 smithi107 bash[103898]: d0cb3de7bf040e7d6b8e67bd8bc7e486c3d9a027b712bbb7e8ab425caa1ebb2c 2024-07-26T12:54:03.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:03 smithi107 systemd[1]: Started Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:54:03.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:03 smithi107 ceph-mds[103917]: -- 172.21.15.107:0/3375450014 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5612a10df860 con 0x5612a5281180 2024-07-26T12:54:03.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:03 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: starting mds.a at 2024-07-26T12:54:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:04 smithi112 ceph-mon[37501]: pgmap v643: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:04 smithi112 ceph-mon[37501]: daemon mds.a restarted 2024-07-26T12:54:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:04 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:04 smithi112 ceph-mon[37501]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T12:54:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:04 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:boot 2024-07-26T12:54:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:04 smithi112 ceph-mon[37501]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:54:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:04 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:54:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:04 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:04 smithi107 ceph-mon[32659]: pgmap v643: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:04 smithi107 ceph-mon[32659]: daemon mds.a restarted 2024-07-26T12:54:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:04 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:04 smithi107 ceph-mon[32659]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T12:54:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:04 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:boot 2024-07-26T12:54:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:04 smithi107 ceph-mon[32659]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:54:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:04 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:54:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:04 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:04 smithi132 ceph-mon[37580]: pgmap v643: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:04 smithi132 ceph-mon[37580]: daemon mds.a restarted 2024-07-26T12:54:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:04 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:04 smithi132 ceph-mon[37580]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T12:54:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:04 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:boot 2024-07-26T12:54:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:04 smithi132 ceph-mon[37580]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:54:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:04 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:54:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:04 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:05.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:05 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:resolve 2024-07-26T12:54:05.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:05 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:05.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:05 smithi107 ceph-mon[32659]: 1 slow requests, 1 included below; oldest blocked for > 181.745899 secs 2024-07-26T12:54:05.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:05 smithi107 ceph-mon[32659]: slow request 181.745897 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T12:54:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:05 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:resolve 2024-07-26T12:54:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:05 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:05 smithi112 ceph-mon[37501]: 1 slow requests, 1 included below; oldest blocked for > 181.745899 secs 2024-07-26T12:54:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:05 smithi112 ceph-mon[37501]: slow request 181.745897 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T12:54:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:05 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:resolve 2024-07-26T12:54:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:05 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:05 smithi132 ceph-mon[37580]: 1 slow requests, 1 included below; oldest blocked for > 181.745899 secs 2024-07-26T12:54:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:05 smithi132 ceph-mon[37580]: slow request 181.745897 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T12:54:06.729 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-mon[32659]: pgmap v645: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:54:06.730 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00499992 2024-07-26T12:54:06.730 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:reconnect 2024-07-26T12:54:06.730 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:06.730 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:54:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:06 smithi132 ceph-mon[37580]: pgmap v645: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:54:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:06 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00499992 2024-07-26T12:54:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:06 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:reconnect 2024-07-26T12:54:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:06 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:06 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:54:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:06 smithi112 ceph-mon[37501]: pgmap v645: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:54:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:06 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00499992 2024-07-26T12:54:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:06 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:reconnect 2024-07-26T12:54:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:06 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:06 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:54:07.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:54:07.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: *** Caught signal (Aborted) ** 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: in thread 7f233d200640 thread_name:ms_dispatch 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f234363e6f0] 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f234368b94c] 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 3: raise() 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 4: abort() 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 5: /lib64/libc.so.6(+0x2871b) [0x7f234362871b] 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 6: /lib64/libc.so.6(+0x37386) [0x7f2343637386] 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d1ac3d76b9] 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d1ac59580c] 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d1ac4e3cde] 2024-07-26T12:54:07.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d1ac4a5763] 2024-07-26T12:54:07.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d1ac4a6a1b] 2024-07-26T12:54:07.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d1ac48a752] 2024-07-26T12:54:07.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d1ac3d8a56] 2024-07-26T12:54:07.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d1ac3dc637] 2024-07-26T12:54:07.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d1ac3dcf5e] 2024-07-26T12:54:07.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d1ac3c003a] 2024-07-26T12:54:07.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f234419b407] 2024-07-26T12:54:07.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 18: (DispatchQueue::entry()+0x60f) [0x7f234419bcf9] 2024-07-26T12:54:07.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f23442785a1] 2024-07-26T12:54:07.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 20: (Thread::entry_wrapper()+0x43) [0x7f2344025cdd] 2024-07-26T12:54:07.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2344025cf9] 2024-07-26T12:54:07.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2343689c02] 2024-07-26T12:54:07.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f234370ec40] 2024-07-26T12:54:07.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:06.730+0000 7f233d200640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:07.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: in thread 7f233d200640 thread_name:ms_dispatch 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f234363e6f0] 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f234368b94c] 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 3: raise() 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 4: abort() 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 5: /lib64/libc.so.6(+0x2871b) [0x7f234362871b] 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 6: /lib64/libc.so.6(+0x37386) [0x7f2343637386] 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d1ac3d76b9] 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d1ac59580c] 2024-07-26T12:54:07.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d1ac4e3cde] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d1ac4a5763] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d1ac4a6a1b] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d1ac48a752] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d1ac3d8a56] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d1ac3dc637] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d1ac3dcf5e] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d1ac3c003a] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f234419b407] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 18: (DispatchQueue::entry()+0x60f) [0x7f234419bcf9] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f23442785a1] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 20: (Thread::entry_wrapper()+0x43) [0x7f2344025cdd] 2024-07-26T12:54:07.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2344025cf9] 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2343689c02] 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f234370ec40] 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 0> 2024-07-26T12:54:06.730+0000 7f233d200640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: in thread 7f233d200640 thread_name:ms_dispatch 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f234363e6f0] 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f234368b94c] 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 3: raise() 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 4: abort() 2024-07-26T12:54:07.049 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 5: /lib64/libc.so.6(+0x2871b) [0x7f234362871b] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 6: /lib64/libc.so.6(+0x37386) [0x7f2343637386] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d1ac3d76b9] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d1ac59580c] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d1ac4e3cde] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d1ac4a5763] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d1ac4a6a1b] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d1ac48a752] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d1ac3d8a56] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d1ac3dc637] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d1ac3dcf5e] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d1ac3c003a] 2024-07-26T12:54:07.050 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f234419b407] 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 18: (DispatchQueue::entry()+0x60f) [0x7f234419bcf9] 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f23442785a1] 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 20: (Thread::entry_wrapper()+0x43) [0x7f2344025cdd] 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2344025cf9] 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2343689c02] 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f234370ec40] 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 0> 2024-07-26T12:54:06.730+0000 7f233d200640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: in thread 7f233d200640 thread_name:ms_dispatch 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f234363e6f0] 2024-07-26T12:54:07.051 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f234368b94c] 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 3: raise() 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 4: abort() 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 5: /lib64/libc.so.6(+0x2871b) [0x7f234362871b] 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 6: /lib64/libc.so.6(+0x37386) [0x7f2343637386] 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55d1ac3d76b9] 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55d1ac59580c] 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55d1ac4e3cde] 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55d1ac4a5763] 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55d1ac4a6a1b] 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55d1ac48a752] 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55d1ac3d8a56] 2024-07-26T12:54:07.052 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55d1ac3dc637] 2024-07-26T12:54:07.053 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55d1ac3dcf5e] 2024-07-26T12:54:07.053 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55d1ac3c003a] 2024-07-26T12:54:07.053 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f234419b407] 2024-07-26T12:54:07.053 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 18: (DispatchQueue::entry()+0x60f) [0x7f234419bcf9] 2024-07-26T12:54:07.053 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f23442785a1] 2024-07-26T12:54:07.053 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 20: (Thread::entry_wrapper()+0x43) [0x7f2344025cdd] 2024-07-26T12:54:07.053 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2344025cf9] 2024-07-26T12:54:07.054 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2343689c02] 2024-07-26T12:54:07.054 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f234370ec40] 2024-07-26T12:54:07.054 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:07.054 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:07.054 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:06.767+0000 7f233a000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T12:54:07.054 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:06.767+0000 7f233a000640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T12:54:07.054 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:06.767+0000 7f233a000640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-26T12:54:07.054 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:06.767+0000 7f233a000640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-07-26T12:54:07.054 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[76380]: 2024-07-26T12:54:06.767+0000 7f233a000640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-07-26T12:54:07.054 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:06 smithi107 podman[103952]: 2024-07-26 12:54:06.843466339 +0000 UTC m=+0.030869108 container died dab1f478088964ff72b80cf18fa861f82b0792a723586768db672397707b0178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-07-26T12:54:07.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:07.194+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:07.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:07.194+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:07.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:07.194+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:07.709 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:07 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:rejoin 2024-07-26T12:54:07.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:07 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:07.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:07 smithi107 ceph-mon[32659]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:54:07.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:07 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:07.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:07 smithi107 ceph-mon[32659]: Cluster is now healthy 2024-07-26T12:54:07.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:07 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:active 2024-07-26T12:54:07.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:07 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:07 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:rejoin 2024-07-26T12:54:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:07 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:07 smithi132 ceph-mon[37580]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:54:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:07 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:07 smithi132 ceph-mon[37580]: Cluster is now healthy 2024-07-26T12:54:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:07 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:active 2024-07-26T12:54:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:07 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:07 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:rejoin 2024-07-26T12:54:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:07 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:07 smithi112 ceph-mon[37501]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:54:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:07 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:07 smithi112 ceph-mon[37501]: Cluster is now healthy 2024-07-26T12:54:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:07 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/4197833362,v1:172.21.15.107:6837/4197833362] up:active 2024-07-26T12:54:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:07 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:08.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:07 smithi107 podman[103952]: 2024-07-26 12:54:07.707913731 +0000 UTC m=+0.895316541 container remove dab1f478088964ff72b80cf18fa861f82b0792a723586768db672397707b0178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T12:54:08.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:07 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:54:08.542 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:08 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T12:54:08.542 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:08 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 2.167s CPU time. 2024-07-26T12:54:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:08 smithi107 ceph-mon[32659]: pgmap v646: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 11 op/s 2024-07-26T12:54:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:08 smithi132 ceph-mon[37580]: pgmap v646: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 11 op/s 2024-07-26T12:54:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:08 smithi112 ceph-mon[37501]: pgmap v646: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 11 op/s 2024-07-26T12:54:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:09.322+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:09.322+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:09.322+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:09 smithi107 ceph-mon[32659]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T12:54:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:09 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] up:active 2024-07-26T12:54:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:09 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:09 smithi132 ceph-mon[37580]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T12:54:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:09 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] up:active 2024-07-26T12:54:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:09 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:09 smithi112 ceph-mon[37501]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T12:54:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:09 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.132:6838/3489644052,v1:172.21.15.132:6839/3489644052] up:active 2024-07-26T12:54:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:09 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:10 smithi107 ceph-mon[32659]: pgmap v647: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.3 KiB/s wr, 14 op/s 2024-07-26T12:54:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 186.745988 secs 2024-07-26T12:54:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:10 smithi132 ceph-mon[37580]: pgmap v647: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.3 KiB/s wr, 14 op/s 2024-07-26T12:54:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 186.745988 secs 2024-07-26T12:54:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:10 smithi112 ceph-mon[37501]: pgmap v647: 129 pgs: 129 active+clean; 38 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.3 KiB/s wr, 14 op/s 2024-07-26T12:54:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 186.745988 secs 2024-07-26T12:54:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:11.665+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d40034c0 con 0x565421993390 2024-07-26T12:54:11.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:11.666+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d40034c0 con 0x565421993390 2024-07-26T12:54:11.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:11.686+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80034c0 con 0x7ff104102930 2024-07-26T12:54:11.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:11.687+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80034c0 con 0x7ff104102930 2024-07-26T12:54:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:12 smithi107 ceph-mon[32659]: pgmap v648: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-07-26T12:54:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:12 smithi132 ceph-mon[37580]: pgmap v648: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-07-26T12:54:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:12 smithi112 ceph-mon[37501]: pgmap v648: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 4.3 KiB/s wr, 15 op/s 2024-07-26T12:54:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:13.765+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480034c0 con 0x7efc78103340 2024-07-26T12:54:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:13.766+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480034c0 con 0x7efc78103340 2024-07-26T12:54:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:13.771+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc440034c0 con 0x55aecd6b64d0 2024-07-26T12:54:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:13.778+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc440034c0 con 0x55aecd6b64d0 2024-07-26T12:54:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:14 smithi107 ceph-mon[32659]: pgmap v649: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 4.3 KiB/s wr, 17 op/s 2024-07-26T12:54:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:14 smithi132 ceph-mon[37580]: pgmap v649: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 4.3 KiB/s wr, 17 op/s 2024-07-26T12:54:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:14 smithi112 ceph-mon[37501]: pgmap v649: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 4.3 KiB/s wr, 17 op/s 2024-07-26T12:54:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 191.746045 secs 2024-07-26T12:54:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 191.746045 secs 2024-07-26T12:54:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 191.746045 secs 2024-07-26T12:54:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:16 smithi107 ceph-mon[32659]: pgmap v650: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 3.7 KiB/s wr, 15 op/s 2024-07-26T12:54:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:16 smithi132 ceph-mon[37580]: pgmap v650: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 3.7 KiB/s wr, 15 op/s 2024-07-26T12:54:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:16 smithi112 ceph-mon[37501]: pgmap v650: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 3.7 KiB/s wr, 15 op/s 2024-07-26T12:54:18.683 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:18 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-07-26T12:54:18.684 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:18 smithi107 systemd[1]: Stopped Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:54:18.684 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:18 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 2.167s CPU time. 2024-07-26T12:54:18.684 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:18 smithi107 systemd[1]: Starting Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:54:18.971 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:18 smithi107 podman[104044]: 2024-07-26 12:54:18.586939683 +0000 UTC m=+0.030988127 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:54:18.971 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:18 smithi107 podman[104044]: 2024-07-26 12:54:18.707831927 +0000 UTC m=+0.151880402 container create 486a1d699c27de9517645b494783e9ba5ff5f8386c2e3eb86fa88016c89365c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724) 2024-07-26T12:54:18.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:18 smithi107 ceph-mon[32659]: pgmap v651: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-07-26T12:54:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:18 smithi132 ceph-mon[37580]: pgmap v651: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-07-26T12:54:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:18 smithi112 ceph-mon[37501]: pgmap v651: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-07-26T12:54:19.293 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:18 smithi107 podman[104044]: 2024-07-26 12:54:18.969987042 +0000 UTC m=+0.414035521 container init 486a1d699c27de9517645b494783e9ba5ff5f8386c2e3eb86fa88016c89365c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:54:19.293 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:18 smithi107 podman[104044]: 2024-07-26 12:54:18.980198601 +0000 UTC m=+0.424247103 container start 486a1d699c27de9517645b494783e9ba5ff5f8386c2e3eb86fa88016c89365c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-07-26T12:54:19.293 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:19 smithi107 bash[104044]: 486a1d699c27de9517645b494783e9ba5ff5f8386c2e3eb86fa88016c89365c9 2024-07-26T12:54:19.293 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:19 smithi107 systemd[1]: Started Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:54:19.293 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:19 smithi107 ceph-mds[104063]: -- 172.21.15.107:0/2945156397 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5560bd653860 con 0x5560c17f5180 2024-07-26T12:54:19.293 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:19 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: starting mds.d at 2024-07-26T12:54:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:19 smithi132 ceph-mon[37580]: daemon mds.d restarted 2024-07-26T12:54:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:19 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:19 smithi132 ceph-mon[37580]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T12:54:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:19 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:boot 2024-07-26T12:54:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:19 smithi132 ceph-mon[37580]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:54:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:19 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:54:20.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:19 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:19 smithi112 ceph-mon[37501]: daemon mds.d restarted 2024-07-26T12:54:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:19 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:19 smithi112 ceph-mon[37501]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T12:54:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:19 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:boot 2024-07-26T12:54:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:19 smithi112 ceph-mon[37501]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:54:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:19 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:54:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:19 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:19 smithi107 ceph-mon[32659]: daemon mds.d restarted 2024-07-26T12:54:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:19 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:19 smithi107 ceph-mon[32659]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T12:54:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:19 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:boot 2024-07-26T12:54:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:19 smithi107 ceph-mon[32659]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:54:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:19 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:54:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:19 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:21.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:20 smithi112 ceph-mon[37501]: pgmap v652: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-07-26T12:54:21.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 196.746109 secs 2024-07-26T12:54:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:20 smithi107 ceph-mon[32659]: pgmap v652: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-07-26T12:54:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 196.746109 secs 2024-07-26T12:54:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:20 smithi132 ceph-mon[37580]: pgmap v652: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 3.6 KiB/s wr, 15 op/s 2024-07-26T12:54:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 196.746109 secs 2024-07-26T12:54:22.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:21 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:resolve 2024-07-26T12:54:22.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:21 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:21 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:resolve 2024-07-26T12:54:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:21 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:22.208+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:22.208+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:22.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:22.208+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:21 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:resolve 2024-07-26T12:54:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:21 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:22 smithi112 ceph-mon[37501]: pgmap v654: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 op/s 2024-07-26T12:54:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:22 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:54:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:22 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:reconnect 2024-07-26T12:54:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:22 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:22 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00899985 2024-07-26T12:54:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:22 smithi107 ceph-mon[32659]: pgmap v654: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 op/s 2024-07-26T12:54:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:22 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:54:23.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:22 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:reconnect 2024-07-26T12:54:23.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:22 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:23.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:22 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00899985 2024-07-26T12:54:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:22 smithi132 ceph-mon[37580]: pgmap v654: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 op/s 2024-07-26T12:54:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:22 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:54:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:22 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:reconnect 2024-07-26T12:54:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:22 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:22 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00899985 2024-07-26T12:54:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:23 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:rejoin 2024-07-26T12:54:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:23 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:23 smithi112 ceph-mon[37501]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:54:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:rejoin 2024-07-26T12:54:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-mon[32659]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:54:24.294 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:54:24.294 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: *** Caught signal (Aborted) ** 2024-07-26T12:54:24.294 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: in thread 7faac0400640 thread_name:ms_dispatch 2024-07-26T12:54:24.294 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:24.294 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7faac663e6f0] 2024-07-26T12:54:24.294 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2: /lib64/libc.so.6(+0x8b94c) [0x7faac668b94c] 2024-07-26T12:54:24.294 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 3: raise() 2024-07-26T12:54:24.294 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 4: abort() 2024-07-26T12:54:24.294 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 5: /lib64/libc.so.6(+0x2871b) [0x7faac662871b] 2024-07-26T12:54:24.294 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 6: /lib64/libc.so.6(+0x37386) [0x7faac6637386] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56127dc306b9] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56127ddee80c] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56127dd3ccde] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56127dcfe763] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56127dcffa1b] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56127dce3752] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56127dc31a56] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56127dc35637] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56127dc35f5e] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56127dc1903a] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7faac719b407] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 18: (DispatchQueue::entry()+0x60f) [0x7faac719bcf9] 2024-07-26T12:54:24.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7faac72785a1] 2024-07-26T12:54:24.296 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 20: (Thread::entry_wrapper()+0x43) [0x7faac7025cdd] 2024-07-26T12:54:24.296 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 21: (Thread::_entry_func(void*)+0xd) [0x7faac7025cf9] 2024-07-26T12:54:24.296 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 22: /lib64/libc.so.6(+0x89c02) [0x7faac6689c02] 2024-07-26T12:54:24.296 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 23: /lib64/libc.so.6(+0x10ec40) [0x7faac670ec40] 2024-07-26T12:54:24.296 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2024-07-26T12:54:23.893+0000 7faac0400640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:24.296 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: in thread 7faac0400640 thread_name:ms_dispatch 2024-07-26T12:54:24.296 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2024-07-26T12:54:24.296 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7faac663e6f0] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2: /lib64/libc.so.6(+0x8b94c) [0x7faac668b94c] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 3: raise() 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 4: abort() 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 5: /lib64/libc.so.6(+0x2871b) [0x7faac662871b] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 6: /lib64/libc.so.6(+0x37386) [0x7faac6637386] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56127dc306b9] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56127ddee80c] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56127dd3ccde] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56127dcfe763] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56127dcffa1b] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56127dce3752] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56127dc31a56] 2024-07-26T12:54:24.297 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56127dc35637] 2024-07-26T12:54:24.298 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56127dc35f5e] 2024-07-26T12:54:24.298 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56127dc1903a] 2024-07-26T12:54:24.298 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7faac719b407] 2024-07-26T12:54:24.298 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 18: (DispatchQueue::entry()+0x60f) [0x7faac719bcf9] 2024-07-26T12:54:24.298 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7faac72785a1] 2024-07-26T12:54:24.298 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 20: (Thread::entry_wrapper()+0x43) [0x7faac7025cdd] 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 21: (Thread::_entry_func(void*)+0xd) [0x7faac7025cf9] 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 22: /lib64/libc.so.6(+0x89c02) [0x7faac6689c02] 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 23: /lib64/libc.so.6(+0x10ec40) [0x7faac670ec40] 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 0> 2024-07-26T12:54:23.893+0000 7faac0400640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: in thread 7faac0400640 thread_name:ms_dispatch 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7faac663e6f0] 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2: /lib64/libc.so.6(+0x8b94c) [0x7faac668b94c] 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 3: raise() 2024-07-26T12:54:24.299 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 4: abort() 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 5: /lib64/libc.so.6(+0x2871b) [0x7faac662871b] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 6: /lib64/libc.so.6(+0x37386) [0x7faac6637386] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56127dc306b9] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56127ddee80c] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56127dd3ccde] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56127dcfe763] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56127dcffa1b] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56127dce3752] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56127dc31a56] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56127dc35637] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56127dc35f5e] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56127dc1903a] 2024-07-26T12:54:24.300 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7faac719b407] 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 18: (DispatchQueue::entry()+0x60f) [0x7faac719bcf9] 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7faac72785a1] 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 20: (Thread::entry_wrapper()+0x43) [0x7faac7025cdd] 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 21: (Thread::_entry_func(void*)+0xd) [0x7faac7025cf9] 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 22: /lib64/libc.so.6(+0x89c02) [0x7faac6689c02] 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 23: /lib64/libc.so.6(+0x10ec40) [0x7faac670ec40] 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 0> 2024-07-26T12:54:23.893+0000 7faac0400640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: in thread 7faac0400640 thread_name:ms_dispatch 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7faac663e6f0] 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2: /lib64/libc.so.6(+0x8b94c) [0x7faac668b94c] 2024-07-26T12:54:24.301 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 3: raise() 2024-07-26T12:54:24.302 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 4: abort() 2024-07-26T12:54:24.302 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 5: /lib64/libc.so.6(+0x2871b) [0x7faac662871b] 2024-07-26T12:54:24.302 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 6: /lib64/libc.so.6(+0x37386) [0x7faac6637386] 2024-07-26T12:54:24.302 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56127dc306b9] 2024-07-26T12:54:24.302 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56127ddee80c] 2024-07-26T12:54:24.302 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x56127dd3ccde] 2024-07-26T12:54:24.302 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x56127dcfe763] 2024-07-26T12:54:24.302 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56127dcffa1b] 2024-07-26T12:54:24.302 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56127dce3752] 2024-07-26T12:54:24.303 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56127dc31a56] 2024-07-26T12:54:24.303 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56127dc35637] 2024-07-26T12:54:24.303 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56127dc35f5e] 2024-07-26T12:54:24.303 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56127dc1903a] 2024-07-26T12:54:24.303 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7faac719b407] 2024-07-26T12:54:24.303 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 18: (DispatchQueue::entry()+0x60f) [0x7faac719bcf9] 2024-07-26T12:54:24.303 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7faac72785a1] 2024-07-26T12:54:24.303 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 20: (Thread::entry_wrapper()+0x43) [0x7faac7025cdd] 2024-07-26T12:54:24.304 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 21: (Thread::_entry_func(void*)+0xd) [0x7faac7025cf9] 2024-07-26T12:54:24.304 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 22: /lib64/libc.so.6(+0x89c02) [0x7faac6689c02] 2024-07-26T12:54:24.304 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 23: /lib64/libc.so.6(+0x10ec40) [0x7faac670ec40] 2024-07-26T12:54:24.304 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:24.304 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:23 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[103913]: 2024-07-26T12:54:24.304 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:24 smithi107 podman[104099]: 2024-07-26 12:54:24.017283495 +0000 UTC m=+0.031070062 container died d0cb3de7bf040e7d6b8e67bd8bc7e486c3d9a027b712bbb7e8ab425caa1ebb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:54:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:23 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:rejoin 2024-07-26T12:54:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:23 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:23 smithi132 ceph-mon[37580]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:54:24.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:24.337+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:24.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:24.338+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:24.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:24.338+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:24 smithi107 ceph-mon[32659]: pgmap v655: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 9 op/s 2024-07-26T12:54:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:24 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:25.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:24 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:active 2024-07-26T12:54:25.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:24 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:54:25.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:24 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:25.043 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:24 smithi107 podman[104099]: 2024-07-26 12:54:24.713944687 +0000 UTC m=+0.727731289 container remove d0cb3de7bf040e7d6b8e67bd8bc7e486c3d9a027b712bbb7e8ab425caa1ebb2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:54:25.043 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:24 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:54:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:24 smithi112 ceph-mon[37501]: pgmap v655: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 9 op/s 2024-07-26T12:54:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:24 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:24 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:active 2024-07-26T12:54:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:24 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:54:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:24 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:24 smithi132 ceph-mon[37580]: pgmap v655: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 9 op/s 2024-07-26T12:54:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:24 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:24 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/2346044639,v1:172.21.15.107:6841/2346044639] up:active 2024-07-26T12:54:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:24 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:54:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:24 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:25.542 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:25 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T12:54:25.542 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:25 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.853s CPU time. 2024-07-26T12:54:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 201.746167 secs 2024-07-26T12:54:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 201.746167 secs 2024-07-26T12:54:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 201.746167 secs 2024-07-26T12:54:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:26 smithi112 ceph-mon[37501]: pgmap v656: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:54:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:26 smithi107 ceph-mon[32659]: pgmap v656: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:54:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:26 smithi132 ceph-mon[37580]: pgmap v656: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:54:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:28 smithi112 ceph-mon[37501]: pgmap v657: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 14 op/s 2024-07-26T12:54:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:28 smithi107 ceph-mon[32659]: pgmap v657: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 14 op/s 2024-07-26T12:54:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:28 smithi132 ceph-mon[37580]: pgmap v657: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 14 op/s 2024-07-26T12:54:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:30 smithi112 ceph-mon[37501]: pgmap v658: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:54:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 206.746261 secs 2024-07-26T12:54:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:30 smithi107 ceph-mon[32659]: pgmap v658: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:54:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 206.746261 secs 2024-07-26T12:54:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:30 smithi132 ceph-mon[37580]: pgmap v658: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:54:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:54:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 206.746261 secs 2024-07-26T12:54:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:32 smithi112 ceph-mon[37501]: pgmap v659: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 171 B/s wr, 14 op/s 2024-07-26T12:54:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:32 smithi107 ceph-mon[32659]: pgmap v659: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 171 B/s wr, 14 op/s 2024-07-26T12:54:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:32 smithi132 ceph-mon[37580]: pgmap v659: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 171 B/s wr, 14 op/s 2024-07-26T12:54:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:34 smithi112 ceph-mon[37501]: pgmap v660: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:34 smithi107 ceph-mon[32659]: pgmap v660: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:34 smithi132 ceph-mon[37580]: pgmap v660: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:35.706 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:35 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-07-26T12:54:35.706 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:35 smithi107 systemd[1]: Stopped Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:54:35.706 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:35 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.853s CPU time. 2024-07-26T12:54:35.706 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:35 smithi107 systemd[1]: Starting Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:54:36.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 211.746362 secs 2024-07-26T12:54:36.036 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:35 smithi107 podman[104417]: 2024-07-26 12:54:35.608582137 +0000 UTC m=+0.031249365 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:54:36.036 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:35 smithi107 podman[104417]: 2024-07-26 12:54:35.739009504 +0000 UTC m=+0.161676767 container create 4d4903a8ee689ba8bad4848a70bc61e4d8b8b29aacbc0de791d200782b607899 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0) 2024-07-26T12:54:36.036 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:36 smithi107 podman[104417]: 2024-07-26 12:54:36.03514585 +0000 UTC m=+0.457813120 container init 4d4903a8ee689ba8bad4848a70bc61e4d8b8b29aacbc0de791d200782b607899 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240724, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:54:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 211.746362 secs 2024-07-26T12:54:36.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:36 smithi107 podman[104417]: 2024-07-26 12:54:36.046381771 +0000 UTC m=+0.469049037 container start 4d4903a8ee689ba8bad4848a70bc61e4d8b8b29aacbc0de791d200782b607899 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T12:54:36.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:36 smithi107 bash[104417]: 4d4903a8ee689ba8bad4848a70bc61e4d8b8b29aacbc0de791d200782b607899 2024-07-26T12:54:36.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-mds[104436]: -- 172.21.15.107:0/2771926970 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562e51423860 con 0x562e555c5180 2024-07-26T12:54:36.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:36 smithi107 systemd[1]: Started Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:54:36.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: starting mds.a at 2024-07-26T12:54:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 211.746362 secs 2024-07-26T12:54:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:36 smithi112 ceph-mon[37501]: pgmap v661: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:54:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:36 smithi112 ceph-mon[37501]: daemon mds.a restarted 2024-07-26T12:54:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:36 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:54:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:36 smithi112 ceph-mon[37501]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T12:54:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:36 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:boot 2024-07-26T12:54:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:36 smithi112 ceph-mon[37501]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:54:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:36 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:36 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:37.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-mon[32659]: pgmap v661: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:54:37.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-mon[32659]: daemon mds.a restarted 2024-07-26T12:54:37.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:37.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:54:37.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-mon[32659]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T12:54:37.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:boot 2024-07-26T12:54:37.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-mon[32659]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:54:37.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:37.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:36 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:37.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:37.218+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:37.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:37.218+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:37.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:37.218+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:36 smithi132 ceph-mon[37580]: pgmap v661: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:54:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:36 smithi132 ceph-mon[37580]: daemon mds.a restarted 2024-07-26T12:54:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:36 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:54:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:36 smithi132 ceph-mon[37580]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T12:54:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:36 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:boot 2024-07-26T12:54:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:36 smithi132 ceph-mon[37580]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:54:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:36 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:36 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:39.204 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:38 smithi107 ceph-mon[32659]: pgmap v663: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:54:39.204 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:38 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:resolve 2024-07-26T12:54:39.205 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:38 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:38 smithi132 ceph-mon[37580]: pgmap v663: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:54:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:38 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:resolve 2024-07-26T12:54:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:38 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:39.354+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:39.354+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:39.354+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:38 smithi112 ceph-mon[37501]: pgmap v663: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:54:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:38 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:resolve 2024-07-26T12:54:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:38 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:39 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199997 2024-07-26T12:54:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:39 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:reconnect 2024-07-26T12:54:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:39 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:39 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:54:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:39 smithi107 ceph-mon[32659]: pgmap v664: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:54:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:39 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199997 2024-07-26T12:54:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:39 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:reconnect 2024-07-26T12:54:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:39 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:39 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:54:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:39 smithi132 ceph-mon[37580]: pgmap v664: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:54:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:39 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00199997 2024-07-26T12:54:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:39 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:reconnect 2024-07-26T12:54:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:39 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:39 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:54:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:39 smithi112 ceph-mon[37501]: pgmap v664: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:54:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:40 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:rejoin 2024-07-26T12:54:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:40 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:40 smithi107 ceph-mon[32659]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:54:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 216.746474 secs 2024-07-26T12:54:41.293 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: *** Caught signal (Aborted) ** 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: in thread 7fd747200640 thread_name:ms_dispatch 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd74d43e6f0] 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd74d48b94c] 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 3: raise() 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 4: abort() 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd74d42871b] 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 6: /lib64/libc.so.6(+0x37386) [0x7fd74d437386] 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5560ad2b96b9] 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5560ad47780c] 2024-07-26T12:54:41.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5560ad3c5cde] 2024-07-26T12:54:41.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5560ad387763] 2024-07-26T12:54:41.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5560ad388a1b] 2024-07-26T12:54:41.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5560ad36c752] 2024-07-26T12:54:41.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5560ad2baa56] 2024-07-26T12:54:41.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5560ad2be637] 2024-07-26T12:54:41.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5560ad2bef5e] 2024-07-26T12:54:41.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5560ad2a203a] 2024-07-26T12:54:41.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd74df9b407] 2024-07-26T12:54:41.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 18: (DispatchQueue::entry()+0x60f) [0x7fd74df9bcf9] 2024-07-26T12:54:41.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd74e0785a1] 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 20: (Thread::entry_wrapper()+0x43) [0x7fd74de25cdd] 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd74de25cf9] 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd74d489c02] 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd74d50ec40] 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2024-07-26T12:54:41.006+0000 7fd747200640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: in thread 7fd747200640 thread_name:ms_dispatch 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd74d43e6f0] 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd74d48b94c] 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 3: raise() 2024-07-26T12:54:41.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 4: abort() 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd74d42871b] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 6: /lib64/libc.so.6(+0x37386) [0x7fd74d437386] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5560ad2b96b9] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5560ad47780c] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5560ad3c5cde] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5560ad387763] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5560ad388a1b] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5560ad36c752] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5560ad2baa56] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5560ad2be637] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5560ad2bef5e] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5560ad2a203a] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd74df9b407] 2024-07-26T12:54:41.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 18: (DispatchQueue::entry()+0x60f) [0x7fd74df9bcf9] 2024-07-26T12:54:41.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd74e0785a1] 2024-07-26T12:54:41.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 20: (Thread::entry_wrapper()+0x43) [0x7fd74de25cdd] 2024-07-26T12:54:41.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd74de25cf9] 2024-07-26T12:54:41.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd74d489c02] 2024-07-26T12:54:41.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd74d50ec40] 2024-07-26T12:54:41.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:41.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2024-07-26T12:54:41.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 0> 2024-07-26T12:54:41.006+0000 7fd747200640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: in thread 7fd747200640 thread_name:ms_dispatch 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd74d43e6f0] 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd74d48b94c] 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 3: raise() 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 4: abort() 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd74d42871b] 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 6: /lib64/libc.so.6(+0x37386) [0x7fd74d437386] 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5560ad2b96b9] 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5560ad47780c] 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5560ad3c5cde] 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5560ad387763] 2024-07-26T12:54:41.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5560ad388a1b] 2024-07-26T12:54:41.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5560ad36c752] 2024-07-26T12:54:41.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5560ad2baa56] 2024-07-26T12:54:41.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5560ad2be637] 2024-07-26T12:54:41.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5560ad2bef5e] 2024-07-26T12:54:41.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5560ad2a203a] 2024-07-26T12:54:41.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd74df9b407] 2024-07-26T12:54:41.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 18: (DispatchQueue::entry()+0x60f) [0x7fd74df9bcf9] 2024-07-26T12:54:41.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd74e0785a1] 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 20: (Thread::entry_wrapper()+0x43) [0x7fd74de25cdd] 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd74de25cf9] 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd74d489c02] 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd74d50ec40] 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 0> 2024-07-26T12:54:41.006+0000 7fd747200640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: in thread 7fd747200640 thread_name:ms_dispatch 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd74d43e6f0] 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fd74d48b94c] 2024-07-26T12:54:41.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 3: raise() 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 4: abort() 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 5: /lib64/libc.so.6(+0x2871b) [0x7fd74d42871b] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 6: /lib64/libc.so.6(+0x37386) [0x7fd74d437386] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5560ad2b96b9] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5560ad47780c] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5560ad3c5cde] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5560ad387763] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5560ad388a1b] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5560ad36c752] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5560ad2baa56] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5560ad2be637] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5560ad2bef5e] 2024-07-26T12:54:41.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5560ad2a203a] 2024-07-26T12:54:41.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fd74df9b407] 2024-07-26T12:54:41.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 18: (DispatchQueue::entry()+0x60f) [0x7fd74df9bcf9] 2024-07-26T12:54:41.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fd74e0785a1] 2024-07-26T12:54:41.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 20: (Thread::entry_wrapper()+0x43) [0x7fd74de25cdd] 2024-07-26T12:54:41.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 21: (Thread::_entry_func(void*)+0xd) [0x7fd74de25cf9] 2024-07-26T12:54:41.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 22: /lib64/libc.so.6(+0x89c02) [0x7fd74d489c02] 2024-07-26T12:54:41.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fd74d50ec40] 2024-07-26T12:54:41.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:41.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104059]: 2024-07-26T12:54:41.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 podman[104471]: 2024-07-26 12:54:41.125261947 +0000 UTC m=+0.030911413 container died 486a1d699c27de9517645b494783e9ba5ff5f8386c2e3eb86fa88016c89365c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-07-26T12:54:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:40 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:rejoin 2024-07-26T12:54:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:40 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:40 smithi132 ceph-mon[37580]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:54:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 216.746474 secs 2024-07-26T12:54:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:40 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:rejoin 2024-07-26T12:54:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:40 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:40 smithi112 ceph-mon[37501]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:54:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 216.746474 secs 2024-07-26T12:54:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:42 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:42 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:active 2024-07-26T12:54:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:42 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:54:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:42 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:42.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:42 smithi107 ceph-mon[32659]: pgmap v665: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:54:42.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 podman[104471]: 2024-07-26 12:54:41.904437599 +0000 UTC m=+0.810087099 container remove 486a1d699c27de9517645b494783e9ba5ff5f8386c2e3eb86fa88016c89365c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-07-26T12:54:42.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:41 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:54:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:42 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:42 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:active 2024-07-26T12:54:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:42 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:54:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:42 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:42 smithi132 ceph-mon[37580]: pgmap v665: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:54:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:42 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:42 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/3728710780,v1:172.21.15.107:6837/3728710780] up:active 2024-07-26T12:54:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:42 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:54:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:42 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:42 smithi112 ceph-mon[37501]: pgmap v665: 129 pgs: 129 active+clean; 38 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:54:42.792 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:42 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T12:54:42.792 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:42 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.770s CPU time. 2024-07-26T12:54:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:44 smithi112 ceph-mon[37501]: pgmap v666: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:54:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:44 smithi107 ceph-mon[32659]: pgmap v666: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:54:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:44 smithi132 ceph-mon[37580]: pgmap v666: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:54:46.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:46 smithi107 ceph-mon[32659]: pgmap v667: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:54:46.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 221.746585 secs 2024-07-26T12:54:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:46 smithi112 ceph-mon[37501]: pgmap v667: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:54:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 221.746585 secs 2024-07-26T12:54:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:46 smithi132 ceph-mon[37580]: pgmap v667: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:54:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 221.746585 secs 2024-07-26T12:54:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:48 smithi112 ceph-mon[37501]: pgmap v668: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 188 B/s wr, 15 op/s 2024-07-26T12:54:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:48 smithi107 ceph-mon[32659]: pgmap v668: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 188 B/s wr, 15 op/s 2024-07-26T12:54:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:48 smithi132 ceph-mon[37580]: pgmap v668: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 188 B/s wr, 15 op/s 2024-07-26T12:54:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:50 smithi112 ceph-mon[37501]: pgmap v669: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:50 smithi107 ceph-mon[32659]: pgmap v669: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:50 smithi132 ceph-mon[37580]: pgmap v669: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 226.746648 secs 2024-07-26T12:54:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 226.746648 secs 2024-07-26T12:54:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 226.746648 secs 2024-07-26T12:54:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:52.222+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:52.223+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:52.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:54:52.223+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:52 smithi112 ceph-mon[37501]: pgmap v670: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:54:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:52 smithi107 ceph-mon[32659]: pgmap v670: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:54:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:52 smithi132 ceph-mon[37580]: pgmap v670: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:54:52.924 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:52 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-07-26T12:54:52.925 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:52 smithi107 systemd[1]: Stopped Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:54:52.925 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:52 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.770s CPU time. 2024-07-26T12:54:52.925 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:52 smithi107 systemd[1]: Starting Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:54:53.218 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:52 smithi107 podman[104560]: 2024-07-26 12:54:52.828153239 +0000 UTC m=+0.031804676 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:54:53.218 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:52 smithi107 podman[104560]: 2024-07-26 12:54:52.95486069 +0000 UTC m=+0.158512140 container create 0e80046ab5000bf5d012b48c8d8cb6ff9de9aacfe7bbb0a102b886724b421211 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-07-26T12:54:53.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:53 smithi107 podman[104560]: 2024-07-26 12:54:53.216761582 +0000 UTC m=+0.420413079 container init 0e80046ab5000bf5d012b48c8d8cb6ff9de9aacfe7bbb0a102b886724b421211 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:54:53.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:53 smithi107 podman[104560]: 2024-07-26 12:54:53.226049521 +0000 UTC m=+0.429700954 container start 0e80046ab5000bf5d012b48c8d8cb6ff9de9aacfe7bbb0a102b886724b421211 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:54:53.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:53 smithi107 bash[104560]: 0e80046ab5000bf5d012b48c8d8cb6ff9de9aacfe7bbb0a102b886724b421211 2024-07-26T12:54:53.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:53 smithi107 systemd[1]: Started Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:54:53.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:53 smithi107 ceph-mds[104579]: -- 172.21.15.107:0/3662362928 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5567f0723860 con 0x5567f48c5180 2024-07-26T12:54:53.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:54:53 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: starting mds.d at 2024-07-26T12:54:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:54.361+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:54.361+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:54:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:54:54.361+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:54 smithi112 ceph-mon[37501]: pgmap v671: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:54 smithi112 ceph-mon[37501]: daemon mds.d restarted 2024-07-26T12:54:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:54 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:54 smithi112 ceph-mon[37501]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T12:54:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:54 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:boot 2024-07-26T12:54:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:54 smithi112 ceph-mon[37501]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:54:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:54 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:54:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:54 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:54 smithi107 ceph-mon[32659]: pgmap v671: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:54 smithi107 ceph-mon[32659]: daemon mds.d restarted 2024-07-26T12:54:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:54 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:54 smithi107 ceph-mon[32659]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T12:54:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:54 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:boot 2024-07-26T12:54:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:54 smithi107 ceph-mon[32659]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:54:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:54 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:54:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:54 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:54 smithi132 ceph-mon[37580]: pgmap v671: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:54:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:54 smithi132 ceph-mon[37580]: daemon mds.d restarted 2024-07-26T12:54:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:54 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:54:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:54 smithi132 ceph-mon[37580]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T12:54:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:54 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:boot 2024-07-26T12:54:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:54 smithi132 ceph-mon[37580]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:54:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:54 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:54:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:54:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:54 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:54:55.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:55 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:resolve 2024-07-26T12:54:55.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:55 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:55.628 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 231.746689 secs 2024-07-26T12:54:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:55 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:resolve 2024-07-26T12:54:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:55 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 231.746689 secs 2024-07-26T12:54:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:55 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:resolve 2024-07-26T12:54:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:55 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:54:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 231.746689 secs 2024-07-26T12:54:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:56 smithi112 ceph-mon[37501]: pgmap v673: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:56 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:54:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:56 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:reconnect 2024-07-26T12:54:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:56 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:56 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:54:56.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-mon[32659]: pgmap v673: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:56.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:54:56.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:reconnect 2024-07-26T12:54:56.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:56.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:54:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:56 smithi132 ceph-mon[37580]: pgmap v673: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:56 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:54:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:56 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:reconnect 2024-07-26T12:54:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:56 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:54:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:56 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:54:57.044 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:54:57.044 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: *** Caught signal (Aborted) ** 2024-07-26T12:54:57.044 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: in thread 7fa185c00640 thread_name:ms_dispatch 2024-07-26T12:54:57.044 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:57.044 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa18be3e6f0] 2024-07-26T12:54:57.044 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa18be8b94c] 2024-07-26T12:54:57.044 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 3: raise() 2024-07-26T12:54:57.044 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 4: abort() 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa18be2871b] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 6: /lib64/libc.so.6(+0x37386) [0x7fa18be37386] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562e13a2c6b9] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562e13bea80c] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x562e13b38cde] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x562e13afa763] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562e13afba1b] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562e13adf752] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562e13a2da56] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562e13a31637] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562e13a31f5e] 2024-07-26T12:54:57.045 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562e13a1503a] 2024-07-26T12:54:57.046 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa18c99b407] 2024-07-26T12:54:57.046 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 18: (DispatchQueue::entry()+0x60f) [0x7fa18c99bcf9] 2024-07-26T12:54:57.046 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa18ca785a1] 2024-07-26T12:54:57.046 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 20: (Thread::entry_wrapper()+0x43) [0x7fa18c825cdd] 2024-07-26T12:54:57.046 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa18c825cf9] 2024-07-26T12:54:57.046 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa18be89c02] 2024-07-26T12:54:57.046 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa18bf0ec40] 2024-07-26T12:54:57.046 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2024-07-26T12:54:56.743+0000 7fa185c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: in thread 7fa185c00640 thread_name:ms_dispatch 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa18be3e6f0] 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa18be8b94c] 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 3: raise() 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 4: abort() 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa18be2871b] 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 6: /lib64/libc.so.6(+0x37386) [0x7fa18be37386] 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562e13a2c6b9] 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562e13bea80c] 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x562e13b38cde] 2024-07-26T12:54:57.047 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x562e13afa763] 2024-07-26T12:54:57.048 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562e13afba1b] 2024-07-26T12:54:57.048 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562e13adf752] 2024-07-26T12:54:57.048 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562e13a2da56] 2024-07-26T12:54:57.048 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562e13a31637] 2024-07-26T12:54:57.048 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562e13a31f5e] 2024-07-26T12:54:57.048 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562e13a1503a] 2024-07-26T12:54:57.048 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa18c99b407] 2024-07-26T12:54:57.048 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 18: (DispatchQueue::entry()+0x60f) [0x7fa18c99bcf9] 2024-07-26T12:54:57.048 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa18ca785a1] 2024-07-26T12:54:57.049 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 20: (Thread::entry_wrapper()+0x43) [0x7fa18c825cdd] 2024-07-26T12:54:57.049 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa18c825cf9] 2024-07-26T12:54:57.049 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa18be89c02] 2024-07-26T12:54:57.049 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa18bf0ec40] 2024-07-26T12:54:57.049 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:57.049 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2024-07-26T12:54:57.049 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 0> 2024-07-26T12:54:56.743+0000 7fa185c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:57.049 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: in thread 7fa185c00640 thread_name:ms_dispatch 2024-07-26T12:54:57.049 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2024-07-26T12:54:57.049 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa18be3e6f0] 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa18be8b94c] 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 3: raise() 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 4: abort() 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa18be2871b] 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 6: /lib64/libc.so.6(+0x37386) [0x7fa18be37386] 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562e13a2c6b9] 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562e13bea80c] 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x562e13b38cde] 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x562e13afa763] 2024-07-26T12:54:57.050 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562e13afba1b] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562e13adf752] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562e13a2da56] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562e13a31637] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562e13a31f5e] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562e13a1503a] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa18c99b407] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 18: (DispatchQueue::entry()+0x60f) [0x7fa18c99bcf9] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa18ca785a1] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 20: (Thread::entry_wrapper()+0x43) [0x7fa18c825cdd] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa18c825cf9] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa18be89c02] 2024-07-26T12:54:57.051 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa18bf0ec40] 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 0> 2024-07-26T12:54:56.743+0000 7fa185c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: in thread 7fa185c00640 thread_name:ms_dispatch 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa18be3e6f0] 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa18be8b94c] 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 3: raise() 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 4: abort() 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa18be2871b] 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 6: /lib64/libc.so.6(+0x37386) [0x7fa18be37386] 2024-07-26T12:54:57.052 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x562e13a2c6b9] 2024-07-26T12:54:57.053 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x562e13bea80c] 2024-07-26T12:54:57.053 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x562e13b38cde] 2024-07-26T12:54:57.053 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x562e13afa763] 2024-07-26T12:54:57.053 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x562e13afba1b] 2024-07-26T12:54:57.053 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x562e13adf752] 2024-07-26T12:54:57.053 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x562e13a2da56] 2024-07-26T12:54:57.053 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x562e13a31637] 2024-07-26T12:54:57.053 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x562e13a31f5e] 2024-07-26T12:54:57.053 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x562e13a1503a] 2024-07-26T12:54:57.054 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa18c99b407] 2024-07-26T12:54:57.054 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 18: (DispatchQueue::entry()+0x60f) [0x7fa18c99bcf9] 2024-07-26T12:54:57.054 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa18ca785a1] 2024-07-26T12:54:57.054 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 20: (Thread::entry_wrapper()+0x43) [0x7fa18c825cdd] 2024-07-26T12:54:57.054 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa18c825cf9] 2024-07-26T12:54:57.054 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa18be89c02] 2024-07-26T12:54:57.054 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa18bf0ec40] 2024-07-26T12:54:57.054 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:54:57.054 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104432]: 2024-07-26T12:54:57.054 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:56 smithi107 podman[104614]: 2024-07-26 12:54:56.853277542 +0000 UTC m=+0.030840304 container died 4d4903a8ee689ba8bad4848a70bc61e4d8b8b29aacbc0de791d200782b607899 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-07-26T12:54:57.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:57 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:rejoin 2024-07-26T12:54:57.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:57 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:57.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:57 smithi107 ceph-mon[32659]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:54:57.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:57 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:57.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:57 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:active 2024-07-26T12:54:57.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:57 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:57 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:rejoin 2024-07-26T12:54:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:57 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:57 smithi112 ceph-mon[37501]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:54:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:57 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:57 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:active 2024-07-26T12:54:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:57 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:57.792 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:57 smithi107 podman[104614]: 2024-07-26 12:54:57.637297363 +0000 UTC m=+0.814860149 container remove 4d4903a8ee689ba8bad4848a70bc61e4d8b8b29aacbc0de791d200782b607899 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-07-26T12:54:57.804 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:57 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:54:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:57 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:rejoin 2024-07-26T12:54:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:57 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:54:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:57 smithi132 ceph-mon[37580]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:54:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:57 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:54:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:57 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/2976013350,v1:172.21.15.107:6841/2976013350] up:active 2024-07-26T12:54:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:57 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:54:58.542 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:58 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T12:54:58.542 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:54:58 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.759s CPU time. 2024-07-26T12:54:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:54:58 smithi107 ceph-mon[32659]: pgmap v674: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-07-26T12:54:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:54:58 smithi132 ceph-mon[37580]: pgmap v674: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-07-26T12:54:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:54:58 smithi112 ceph-mon[37501]: pgmap v674: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 11 op/s 2024-07-26T12:55:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:00 smithi107 ceph-mon[32659]: pgmap v675: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T12:55:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 236.746807 secs 2024-07-26T12:55:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:00 smithi132 ceph-mon[37580]: pgmap v675: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T12:55:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 236.746807 secs 2024-07-26T12:55:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:00 smithi112 ceph-mon[37501]: pgmap v675: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T12:55:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 236.746807 secs 2024-07-26T12:55:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:02 smithi107 ceph-mon[32659]: pgmap v676: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:55:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:02 smithi132 ceph-mon[37580]: pgmap v676: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:55:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:02 smithi112 ceph-mon[37501]: pgmap v676: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:55:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:04 smithi107 ceph-mon[32659]: pgmap v677: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:55:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:04 smithi132 ceph-mon[37580]: pgmap v677: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:55:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:04 smithi112 ceph-mon[37501]: pgmap v677: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:55:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 241.746954 secs 2024-07-26T12:55:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 241.746954 secs 2024-07-26T12:55:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 241.746954 secs 2024-07-26T12:55:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:06 smithi107 ceph-mon[32659]: pgmap v678: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 174 B/s wr, 15 op/s 2024-07-26T12:55:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:06 smithi132 ceph-mon[37580]: pgmap v678: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 174 B/s wr, 15 op/s 2024-07-26T12:55:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:06 smithi112 ceph-mon[37501]: pgmap v678: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 174 B/s wr, 15 op/s 2024-07-26T12:55:07.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:07.238+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:07.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:07.238+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:07.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:07.238+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:08.692 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:08 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-07-26T12:55:08.693 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:08 smithi107 systemd[1]: Stopped Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:55:08.693 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:08 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.759s CPU time. 2024-07-26T12:55:08.693 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:08 smithi107 systemd[1]: Starting Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:55:09.033 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:08 smithi107 ceph-mon[32659]: pgmap v679: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:55:09.033 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:08 smithi107 podman[104703]: 2024-07-26 12:55:08.59572829 +0000 UTC m=+0.031563653 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:55:09.033 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:08 smithi107 podman[104703]: 2024-07-26 12:55:08.770568383 +0000 UTC m=+0.206403781 container create caf296812e7ae0e9ec4641d15eef0359f1ef07aa6e2071851c6821755ae0b3cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:55:09.034 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:09 smithi107 podman[104703]: 2024-07-26 12:55:09.032250416 +0000 UTC m=+0.468085820 container init caf296812e7ae0e9ec4641d15eef0359f1ef07aa6e2071851c6821755ae0b3cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-07-26T12:55:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:08 smithi132 ceph-mon[37580]: pgmap v679: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:55:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:08 smithi112 ceph-mon[37501]: pgmap v679: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:55:09.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:09 smithi107 podman[104703]: 2024-07-26 12:55:09.041974255 +0000 UTC m=+0.477809621 container start caf296812e7ae0e9ec4641d15eef0359f1ef07aa6e2071851c6821755ae0b3cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0) 2024-07-26T12:55:09.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:09 smithi107 bash[104703]: caf296812e7ae0e9ec4641d15eef0359f1ef07aa6e2071851c6821755ae0b3cb 2024-07-26T12:55:09.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:09 smithi107 systemd[1]: Started Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:55:09.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:09 smithi107 ceph-mds[104722]: -- 172.21.15.107:0/1515153418 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af66217c20 con 0x55af6a3b9600 2024-07-26T12:55:09.295 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:09 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: starting mds.a at 2024-07-26T12:55:09.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:09.377+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:09.377 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:09.377+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:09.378 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:09.377+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:09 smithi107 ceph-mon[32659]: daemon mds.a restarted 2024-07-26T12:55:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:09 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:55:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:09 smithi107 ceph-mon[32659]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T12:55:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:09 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:boot 2024-07-26T12:55:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:09 smithi107 ceph-mon[32659]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:55:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:09 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:55:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:55:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:09 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:55:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:09 smithi132 ceph-mon[37580]: daemon mds.a restarted 2024-07-26T12:55:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:09 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:55:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:09 smithi132 ceph-mon[37580]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T12:55:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:09 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:boot 2024-07-26T12:55:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:09 smithi132 ceph-mon[37580]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:55:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:09 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:55:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:55:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:09 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:55:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:09 smithi112 ceph-mon[37501]: daemon mds.a restarted 2024-07-26T12:55:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:09 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:55:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:09 smithi112 ceph-mon[37501]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T12:55:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:09 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:boot 2024-07-26T12:55:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:09 smithi112 ceph-mon[37501]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:55:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:09 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:55:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:55:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:09 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:55:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:10 smithi132 ceph-mon[37580]: pgmap v680: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:55:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 246.747103 secs 2024-07-26T12:55:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:10 smithi112 ceph-mon[37501]: pgmap v680: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:55:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 246.747103 secs 2024-07-26T12:55:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:10 smithi107 ceph-mon[32659]: pgmap v680: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:55:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 246.747103 secs 2024-07-26T12:55:12.067 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:11 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:resolve 2024-07-26T12:55:12.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:11 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:55:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:11 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:resolve 2024-07-26T12:55:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:11 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:55:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:11 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:resolve 2024-07-26T12:55:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:11 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:55:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:12 smithi112 ceph-mon[37501]: pgmap v682: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T12:55:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:12 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:55:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:12 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:reconnect 2024-07-26T12:55:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:12 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:55:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:12 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:55:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:12 smithi107 ceph-mon[32659]: pgmap v682: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T12:55:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:12 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:55:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:12 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:reconnect 2024-07-26T12:55:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:12 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:55:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:12 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:55:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:12 smithi132 ceph-mon[37580]: pgmap v682: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T12:55:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:12 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:55:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:12 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:reconnect 2024-07-26T12:55:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:12 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:55:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:12 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.0109998 2024-07-26T12:55:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:13 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:rejoin 2024-07-26T12:55:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:13 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:55:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:13 smithi112 ceph-mon[37501]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:55:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:rejoin 2024-07-26T12:55:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:55:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-mon[32659]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:55:14.293 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: *** Caught signal (Aborted) ** 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: in thread 7f5aeda00640 thread_name:ms_dispatch 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5af3c3e6f0] 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5af3c8b94c] 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 3: raise() 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 4: abort() 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5af3c2871b] 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 6: /lib64/libc.so.6(+0x37386) [0x7f5af3c37386] 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5567e291c6b9] 2024-07-26T12:55:14.294 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5567e2ada80c] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5567e2a28cde] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5567e29ea763] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5567e29eba1b] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5567e29cf752] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5567e291da56] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5567e2921637] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5567e2921f5e] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5567e290503a] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5af479b407] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 18: (DispatchQueue::entry()+0x60f) [0x7f5af479bcf9] 2024-07-26T12:55:14.295 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5af48785a1] 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 20: (Thread::entry_wrapper()+0x43) [0x7f5af4625cdd] 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5af4625cf9] 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5af3c89c02] 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5af3d0ec40] 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2024-07-26T12:55:13.864+0000 7f5aeda00640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: in thread 7f5aeda00640 thread_name:ms_dispatch 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5af3c3e6f0] 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5af3c8b94c] 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 3: raise() 2024-07-26T12:55:14.296 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 4: abort() 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5af3c2871b] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 6: /lib64/libc.so.6(+0x37386) [0x7f5af3c37386] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5567e291c6b9] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5567e2ada80c] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5567e2a28cde] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5567e29ea763] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5567e29eba1b] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5567e29cf752] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5567e291da56] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5567e2921637] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5567e2921f5e] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5567e290503a] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5af479b407] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 18: (DispatchQueue::entry()+0x60f) [0x7f5af479bcf9] 2024-07-26T12:55:14.297 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5af48785a1] 2024-07-26T12:55:14.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 20: (Thread::entry_wrapper()+0x43) [0x7f5af4625cdd] 2024-07-26T12:55:14.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5af4625cf9] 2024-07-26T12:55:14.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5af3c89c02] 2024-07-26T12:55:14.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5af3d0ec40] 2024-07-26T12:55:14.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:55:14.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2024-07-26T12:55:14.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 0> 2024-07-26T12:55:13.864+0000 7f5aeda00640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:55:14.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: in thread 7f5aeda00640 thread_name:ms_dispatch 2024-07-26T12:55:14.298 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5af3c3e6f0] 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5af3c8b94c] 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 3: raise() 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 4: abort() 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5af3c2871b] 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 6: /lib64/libc.so.6(+0x37386) [0x7f5af3c37386] 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5567e291c6b9] 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5567e2ada80c] 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5567e2a28cde] 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5567e29ea763] 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5567e29eba1b] 2024-07-26T12:55:14.299 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5567e29cf752] 2024-07-26T12:55:14.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5567e291da56] 2024-07-26T12:55:14.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5567e2921637] 2024-07-26T12:55:14.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5567e2921f5e] 2024-07-26T12:55:14.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5567e290503a] 2024-07-26T12:55:14.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5af479b407] 2024-07-26T12:55:14.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 18: (DispatchQueue::entry()+0x60f) [0x7f5af479bcf9] 2024-07-26T12:55:14.300 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5af48785a1] 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 20: (Thread::entry_wrapper()+0x43) [0x7f5af4625cdd] 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5af4625cf9] 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5af3c89c02] 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5af3d0ec40] 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 0> 2024-07-26T12:55:13.864+0000 7f5aeda00640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: in thread 7f5aeda00640 thread_name:ms_dispatch 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5af3c3e6f0] 2024-07-26T12:55:14.301 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5af3c8b94c] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 3: raise() 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 4: abort() 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5af3c2871b] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 6: /lib64/libc.so.6(+0x37386) [0x7f5af3c37386] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5567e291c6b9] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5567e2ada80c] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x5567e2a28cde] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x5567e29ea763] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5567e29eba1b] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5567e29cf752] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5567e291da56] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5567e2921637] 2024-07-26T12:55:14.302 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5567e2921f5e] 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5567e290503a] 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f5af479b407] 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 18: (DispatchQueue::entry()+0x60f) [0x7f5af479bcf9] 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f5af48785a1] 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 20: (Thread::entry_wrapper()+0x43) [0x7f5af4625cdd] 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5af4625cf9] 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5af3c89c02] 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5af3d0ec40] 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104575]: 2024-07-26T12:55:14.303 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:13 smithi107 podman[104757]: 2024-07-26 12:55:13.983716341 +0000 UTC m=+0.031802810 container died 0e80046ab5000bf5d012b48c8d8cb6ff9de9aacfe7bbb0a102b886724b421211 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:55:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:13 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:rejoin 2024-07-26T12:55:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:13 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:55:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:13 smithi132 ceph-mon[37580]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:55:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:14 smithi112 ceph-mon[37501]: pgmap v683: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 9 op/s 2024-07-26T12:55:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:14 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:55:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:14 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:active 2024-07-26T12:55:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:14 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:55:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:14 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:55:15.292 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:14 smithi107 podman[104757]: 2024-07-26 12:55:14.806728441 +0000 UTC m=+0.854814935 container remove 0e80046ab5000bf5d012b48c8d8cb6ff9de9aacfe7bbb0a102b886724b421211 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-07-26T12:55:15.293 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:14 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:55:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:14 smithi107 ceph-mon[32659]: pgmap v683: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 9 op/s 2024-07-26T12:55:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:14 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:55:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:14 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:active 2024-07-26T12:55:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:14 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:55:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:14 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:55:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:14 smithi132 ceph-mon[37580]: pgmap v683: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 9 op/s 2024-07-26T12:55:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:14 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:55:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:14 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/2018139236,v1:172.21.15.107:6837/2018139236] up:active 2024-07-26T12:55:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:14 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:55:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:14 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:55:15.792 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:15 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T12:55:15.792 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:15 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.850s CPU time. 2024-07-26T12:55:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 251.747218 secs 2024-07-26T12:55:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 251.747218 secs 2024-07-26T12:55:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 251.747218 secs 2024-07-26T12:55:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:16 smithi112 ceph-mon[37501]: pgmap v684: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:55:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:16 smithi107 ceph-mon[32659]: pgmap v684: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:55:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:16 smithi132 ceph-mon[37580]: pgmap v684: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:55:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:18 smithi112 ceph-mon[37501]: pgmap v685: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 14 op/s 2024-07-26T12:55:19.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:18 smithi107 ceph-mon[32659]: pgmap v685: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 14 op/s 2024-07-26T12:55:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:18 smithi132 ceph-mon[37580]: pgmap v685: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 14 op/s 2024-07-26T12:55:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:20 smithi112 ceph-mon[37501]: pgmap v686: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:55:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 256.747336 secs 2024-07-26T12:55:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:20 smithi107 ceph-mon[32659]: pgmap v686: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:55:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 256.747336 secs 2024-07-26T12:55:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:20 smithi132 ceph-mon[37580]: pgmap v686: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:55:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 256.747336 secs 2024-07-26T12:55:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:22.253+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:22.253+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:22.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:22.253+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:22 smithi112 ceph-mon[37501]: pgmap v687: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 171 B/s wr, 14 op/s 2024-07-26T12:55:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:22 smithi107 ceph-mon[32659]: pgmap v687: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 171 B/s wr, 14 op/s 2024-07-26T12:55:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:22 smithi132 ceph-mon[37580]: pgmap v687: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 171 B/s wr, 14 op/s 2024-07-26T12:55:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:24.391+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:24.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:24.391+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:24.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:24.391+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:24 smithi112 ceph-mon[37501]: pgmap v688: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:55:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:24 smithi107 ceph-mon[32659]: pgmap v688: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:55:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:24 smithi132 ceph-mon[37580]: pgmap v688: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:55:25.887 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:25 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-07-26T12:55:25.888 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:25 smithi107 systemd[1]: Stopped Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:55:25.888 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:25 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.850s CPU time. 2024-07-26T12:55:25.888 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:25 smithi107 systemd[1]: Starting Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:55:26.140 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 261.747428 secs 2024-07-26T12:55:26.141 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:25 smithi107 podman[104852]: 2024-07-26 12:55:25.829311551 +0000 UTC m=+0.031322978 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:55:26.141 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:25 smithi107 podman[104852]: 2024-07-26 12:55:25.943998514 +0000 UTC m=+0.146009959 container create 0c79b28821d2ca0855ea8dcb71e881a56fadac8e04a2e9783b6b54cdd4522c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:55:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 261.747428 secs 2024-07-26T12:55:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 261.747428 secs 2024-07-26T12:55:26.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:26 smithi107 podman[104852]: 2024-07-26 12:55:26.255805221 +0000 UTC m=+0.457816669 container init 0c79b28821d2ca0855ea8dcb71e881a56fadac8e04a2e9783b6b54cdd4522c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:55:26.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:26 smithi107 podman[104852]: 2024-07-26 12:55:26.265818597 +0000 UTC m=+0.467830075 container start 0c79b28821d2ca0855ea8dcb71e881a56fadac8e04a2e9783b6b54cdd4522c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-07-26T12:55:26.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:26 smithi107 bash[104852]: 0c79b28821d2ca0855ea8dcb71e881a56fadac8e04a2e9783b6b54cdd4522c6b 2024-07-26T12:55:26.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:26 smithi107 systemd[1]: Started Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:55:26.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-mds[104871]: -- 172.21.15.107:0/3811385101 <== mon.1 v2:172.21.15.132:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559956dbf860 con 0x55995af61180 2024-07-26T12:55:26.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: starting mds.d at 2024-07-26T12:55:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:26 smithi112 ceph-mon[37501]: pgmap v689: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:55:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:26 smithi112 ceph-mon[37501]: daemon mds.d restarted 2024-07-26T12:55:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:26 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:55:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:26 smithi112 ceph-mon[37501]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T12:55:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:26 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:boot 2024-07-26T12:55:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:26 smithi112 ceph-mon[37501]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:55:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:26 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:55:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:55:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:26 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:55:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-mon[32659]: pgmap v689: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:55:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-mon[32659]: daemon mds.d restarted 2024-07-26T12:55:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:55:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-mon[32659]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T12:55:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:boot 2024-07-26T12:55:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-mon[32659]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:55:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:55:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:55:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:26 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:55:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:26 smithi132 ceph-mon[37580]: pgmap v689: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 15 op/s 2024-07-26T12:55:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:26 smithi132 ceph-mon[37580]: daemon mds.d restarted 2024-07-26T12:55:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:26 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:55:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:26 smithi132 ceph-mon[37580]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T12:55:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:26 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:boot 2024-07-26T12:55:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:26 smithi132 ceph-mon[37580]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:55:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:26 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:55:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:55:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:26 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:55:29.162 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:28 smithi107 ceph-mon[32659]: pgmap v691: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:55:29.162 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:28 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:resolve 2024-07-26T12:55:29.162 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:28 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:55:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:28 smithi112 ceph-mon[37501]: pgmap v691: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:55:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:28 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:resolve 2024-07-26T12:55:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:28 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:55:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:28 smithi132 ceph-mon[37580]: pgmap v691: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:55:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:28 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:resolve 2024-07-26T12:55:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:28 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:55:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:29 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:55:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:29 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:reconnect 2024-07-26T12:55:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:29 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:55:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:29 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:55:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:29 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:55:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:29 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:reconnect 2024-07-26T12:55:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:29 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:55:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:29 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:55:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:29 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:55:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:29 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:reconnect 2024-07-26T12:55:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:29 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:55:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:29 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:55:31.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-mon[32659]: pgmap v692: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:55:31.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:rejoin 2024-07-26T12:55:31.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:55:31.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-mon[32659]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:55:31.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 266.747551 secs 2024-07-26T12:55:31.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:31.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.196 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.196 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.196 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.196 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:55:31.196 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: *** Caught signal (Aborted) ** 2024-07-26T12:55:31.196 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: in thread 7f3034c00640 thread_name:ms_dispatch 2024-07-26T12:55:31.196 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:31.196 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f303ae3e6f0] 2024-07-26T12:55:31.196 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f303ae8b94c] 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 3: raise() 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 4: abort() 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 5: /lib64/libc.so.6(+0x2871b) [0x7f303ae2871b] 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 6: /lib64/libc.so.6(+0x37386) [0x7f303ae37386] 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55af3c28a6b9] 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55af3c44880c] 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55af3c396cde] 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55af3c358763] 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55af3c359a1b] 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55af3c33d752] 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55af3c28ba56] 2024-07-26T12:55:31.197 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55af3c28f637] 2024-07-26T12:55:31.198 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55af3c28ff5e] 2024-07-26T12:55:31.198 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55af3c27303a] 2024-07-26T12:55:31.198 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f303b99b407] 2024-07-26T12:55:31.198 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 18: (DispatchQueue::entry()+0x60f) [0x7f303b99bcf9] 2024-07-26T12:55:31.198 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f303ba785a1] 2024-07-26T12:55:31.198 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 20: (Thread::entry_wrapper()+0x43) [0x7f303b825cdd] 2024-07-26T12:55:31.198 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 21: (Thread::_entry_func(void*)+0xd) [0x7f303b825cf9] 2024-07-26T12:55:31.198 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 22: /lib64/libc.so.6(+0x89c02) [0x7f303ae89c02] 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f303af0ec40] 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2024-07-26T12:55:30.962+0000 7f3034c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: in thread 7f3034c00640 thread_name:ms_dispatch 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f303ae3e6f0] 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f303ae8b94c] 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 3: raise() 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 4: abort() 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 5: /lib64/libc.so.6(+0x2871b) [0x7f303ae2871b] 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 6: /lib64/libc.so.6(+0x37386) [0x7f303ae37386] 2024-07-26T12:55:31.199 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55af3c28a6b9] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55af3c44880c] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55af3c396cde] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55af3c358763] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55af3c359a1b] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55af3c33d752] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55af3c28ba56] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55af3c28f637] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55af3c28ff5e] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55af3c27303a] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f303b99b407] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 18: (DispatchQueue::entry()+0x60f) [0x7f303b99bcf9] 2024-07-26T12:55:31.200 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f303ba785a1] 2024-07-26T12:55:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:30 smithi112 ceph-mon[37501]: pgmap v692: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:55:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:30 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:rejoin 2024-07-26T12:55:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:30 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:55:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:30 smithi112 ceph-mon[37501]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:55:31.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 266.747551 secs 2024-07-26T12:55:31.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:31.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.202 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 20: (Thread::entry_wrapper()+0x43) [0x7f303b825cdd] 2024-07-26T12:55:31.202 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 21: (Thread::_entry_func(void*)+0xd) [0x7f303b825cf9] 2024-07-26T12:55:31.202 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 22: /lib64/libc.so.6(+0x89c02) [0x7f303ae89c02] 2024-07-26T12:55:31.202 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f303af0ec40] 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 0> 2024-07-26T12:55:30.962+0000 7f3034c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: in thread 7f3034c00640 thread_name:ms_dispatch 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f303ae3e6f0] 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f303ae8b94c] 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 3: raise() 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 4: abort() 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 5: /lib64/libc.so.6(+0x2871b) [0x7f303ae2871b] 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 6: /lib64/libc.so.6(+0x37386) [0x7f303ae37386] 2024-07-26T12:55:31.203 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55af3c28a6b9] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55af3c44880c] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55af3c396cde] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55af3c358763] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55af3c359a1b] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55af3c33d752] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55af3c28ba56] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55af3c28f637] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55af3c28ff5e] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55af3c27303a] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f303b99b407] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 18: (DispatchQueue::entry()+0x60f) [0x7f303b99bcf9] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f303ba785a1] 2024-07-26T12:55:31.204 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 20: (Thread::entry_wrapper()+0x43) [0x7f303b825cdd] 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 21: (Thread::_entry_func(void*)+0xd) [0x7f303b825cf9] 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 22: /lib64/libc.so.6(+0x89c02) [0x7f303ae89c02] 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f303af0ec40] 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:30 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 0> 2024-07-26T12:55:30.962+0000 7f3034c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: in thread 7f3034c00640 thread_name:ms_dispatch 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f303ae3e6f0] 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f303ae8b94c] 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 3: raise() 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 4: abort() 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 5: /lib64/libc.so.6(+0x2871b) [0x7f303ae2871b] 2024-07-26T12:55:31.205 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 6: /lib64/libc.so.6(+0x37386) [0x7f303ae37386] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55af3c28a6b9] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55af3c44880c] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55af3c396cde] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55af3c358763] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55af3c359a1b] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55af3c33d752] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55af3c28ba56] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55af3c28f637] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55af3c28ff5e] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55af3c27303a] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f303b99b407] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 18: (DispatchQueue::entry()+0x60f) [0x7f303b99bcf9] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f303ba785a1] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 20: (Thread::entry_wrapper()+0x43) [0x7f303b825cdd] 2024-07-26T12:55:31.206 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 21: (Thread::_entry_func(void*)+0xd) [0x7f303b825cf9] 2024-07-26T12:55:31.207 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 22: /lib64/libc.so.6(+0x89c02) [0x7f303ae89c02] 2024-07-26T12:55:31.207 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f303af0ec40] 2024-07-26T12:55:31.207 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:55:31.207 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[104718]: 2024-07-26T12:55:31.305 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:30 smithi132 ceph-mon[37580]: pgmap v692: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:55:31.306 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:30 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:rejoin 2024-07-26T12:55:31.306 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:30 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:55:31.306 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:30 smithi132 ceph-mon[37580]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:55:31.306 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 266.747551 secs 2024-07-26T12:55:31.306 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:31.306 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.306 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.306 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.306 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:31.793 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:31 smithi107 podman[105054]: 2024-07-26 12:55:31.523882702 +0000 UTC m=+0.468313939 container died caf296812e7ae0e9ec4641d15eef0359f1ef07aa6e2071851c6821755ae0b3cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-07-26T12:55:32.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:55:32.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:active 2024-07-26T12:55:32.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:55:32.180 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:31 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:55:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:31 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:55:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:31 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:active 2024-07-26T12:55:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:31 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:55:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:31 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:55:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:31 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:55:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:31 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/654390554,v1:172.21.15.107:6841/654390554] up:active 2024-07-26T12:55:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:31 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:55:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:31 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:55:32.542 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:32 smithi107 podman[105054]: 2024-07-26 12:55:32.178064775 +0000 UTC m=+1.122496003 container remove caf296812e7ae0e9ec4641d15eef0359f1ef07aa6e2071851c6821755ae0b3cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-07-26T12:55:32.542 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:32 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:55:33.292 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:32 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T12:55:33.292 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:32 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.695s CPU time. 2024-07-26T12:55:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:32 smithi107 ceph-mon[32659]: pgmap v693: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:55:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:32 smithi132 ceph-mon[37580]: pgmap v693: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:55:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:32 smithi112 ceph-mon[37501]: pgmap v693: 129 pgs: 129 active+clean; 38 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:55:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:34 smithi107 ceph-mon[32659]: pgmap v694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:55:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:34 smithi132 ceph-mon[37580]: pgmap v694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:55:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:34 smithi112 ceph-mon[37501]: pgmap v694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:55:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:35 smithi107 ceph-mon[32659]: pgmap v695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 207 B/s wr, 16 op/s 2024-07-26T12:55:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 271.747673 secs 2024-07-26T12:55:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:35 smithi132 ceph-mon[37580]: pgmap v695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 207 B/s wr, 16 op/s 2024-07-26T12:55:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 271.747673 secs 2024-07-26T12:55:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:35 smithi112 ceph-mon[37501]: pgmap v695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 207 B/s wr, 16 op/s 2024-07-26T12:55:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:55:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 271.747673 secs 2024-07-26T12:55:37.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:36 smithi107 ceph-mon[32659]: Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T12:55:37.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:37.266+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:37.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:37.266+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:37.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:37.266+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:36 smithi132 ceph-mon[37580]: Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T12:55:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:36 smithi112 ceph-mon[37501]: Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T12:55:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:37 smithi107 ceph-mon[32659]: pgmap v696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 193 B/s wr, 16 op/s 2024-07-26T12:55:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:37 smithi132 ceph-mon[37580]: pgmap v696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 193 B/s wr, 16 op/s 2024-07-26T12:55:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:37 smithi112 ceph-mon[37501]: pgmap v696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 193 B/s wr, 16 op/s 2024-07-26T12:55:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:38 smithi107 ceph-mon[32659]: pgmap v697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 172 B/s wr, 16 op/s 2024-07-26T12:55:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:38 smithi132 ceph-mon[37580]: pgmap v697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 172 B/s wr, 16 op/s 2024-07-26T12:55:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:39.405+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:39.405+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:39.405+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:38 smithi112 ceph-mon[37501]: pgmap v697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 172 B/s wr, 16 op/s 2024-07-26T12:55:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 276.747795 secs 2024-07-26T12:55:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 276.747795 secs 2024-07-26T12:55:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 276.747795 secs 2024-07-26T12:55:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:41 smithi107 ceph-mon[32659]: pgmap v698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 172 B/s wr, 14 op/s 2024-07-26T12:55:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:41 smithi132 ceph-mon[37580]: pgmap v698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 172 B/s wr, 14 op/s 2024-07-26T12:55:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:41 smithi112 ceph-mon[37501]: pgmap v698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 172 B/s wr, 14 op/s 2024-07-26T12:55:43.066 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:43 smithi107 ceph-mon[32659]: pgmap v699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 172 B/s wr, 16 op/s 2024-07-26T12:55:43.067 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-07-26T12:55:43.067 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 systemd[1]: Stopped Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:55:43.067 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.695s CPU time. 2024-07-26T12:55:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:43 smithi132 ceph-mon[37580]: pgmap v699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 172 B/s wr, 16 op/s 2024-07-26T12:55:43.423 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 systemd[1]: Starting Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:55:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:43 smithi112 ceph-mon[37501]: pgmap v699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 172 B/s wr, 16 op/s 2024-07-26T12:55:43.737 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 podman[105480]: 2024-07-26 12:55:43.327620262 +0000 UTC m=+0.035631775 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:55:43.738 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 podman[105480]: 2024-07-26 12:55:43.450709968 +0000 UTC m=+0.158721499 container create f7c493a44ede64a0e8e66cf44d3cb832b7dcf251a3cbabe8ebb560a22e505ea3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, org.label-schema.build-date=20240724, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=) 2024-07-26T12:55:44.039 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 podman[105480]: 2024-07-26 12:55:43.734454179 +0000 UTC m=+0.442465714 container init f7c493a44ede64a0e8e66cf44d3cb832b7dcf251a3cbabe8ebb560a22e505ea3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-07-26T12:55:44.039 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 podman[105480]: 2024-07-26 12:55:43.747070533 +0000 UTC m=+0.455082091 container start f7c493a44ede64a0e8e66cf44d3cb832b7dcf251a3cbabe8ebb560a22e505ea3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, io.buildah.version=1.36.0) 2024-07-26T12:55:44.039 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 bash[105480]: f7c493a44ede64a0e8e66cf44d3cb832b7dcf251a3cbabe8ebb560a22e505ea3 2024-07-26T12:55:44.039 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 systemd[1]: Started Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:55:44.039 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 ceph-mds[105499]: -- 172.21.15.107:0/1476663172 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559b72153520 con 0x559b762eba80 2024-07-26T12:55:44.039 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 ceph-mds[105499]: -- 172.21.15.107:0/1476663172 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b72149860 con 0x559b762eba80 2024-07-26T12:55:44.039 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:55:43 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: starting mds.a at 2024-07-26T12:55:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:44 smithi107 ceph-mon[32659]: daemon mds.a restarted 2024-07-26T12:55:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:44 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:55:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:44 smithi107 ceph-mon[32659]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T12:55:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:44 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:boot 2024-07-26T12:55:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:44 smithi107 ceph-mon[32659]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:55:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:44 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:55:44.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:55:44.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:44 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:55:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:44 smithi132 ceph-mon[37580]: daemon mds.a restarted 2024-07-26T12:55:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:44 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:55:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:44 smithi132 ceph-mon[37580]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T12:55:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:44 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:boot 2024-07-26T12:55:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:44 smithi132 ceph-mon[37580]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:55:44.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:44 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:55:44.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:55:44.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:44 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:55:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:44 smithi112 ceph-mon[37501]: daemon mds.a restarted 2024-07-26T12:55:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:44 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:55:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:44 smithi112 ceph-mon[37501]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T12:55:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:44 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:boot 2024-07-26T12:55:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:44 smithi112 ceph-mon[37501]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:55:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:44 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:55:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T12:55:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:44 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:55:46.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:46 smithi107 ceph-mon[32659]: pgmap v701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:55:46.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:46 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:resolve 2024-07-26T12:55:46.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:46 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:55:46.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 281.747932 secs 2024-07-26T12:55:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:46 smithi132 ceph-mon[37580]: pgmap v701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:55:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:46 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:resolve 2024-07-26T12:55:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:46 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:55:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 281.747932 secs 2024-07-26T12:55:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:46 smithi112 ceph-mon[37501]: pgmap v701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:55:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:46 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:resolve 2024-07-26T12:55:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:46 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:55:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 281.747932 secs 2024-07-26T12:55:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:47 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:55:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:47 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:reconnect 2024-07-26T12:55:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:47 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:55:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:47 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:55:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:47 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:55:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:47 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:reconnect 2024-07-26T12:55:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:47 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:55:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:47 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:55:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:47 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:55:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:47 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:reconnect 2024-07-26T12:55:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:47 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:55:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:47 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:55:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:48 smithi112 ceph-mon[37501]: pgmap v702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:55:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:48 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:rejoin 2024-07-26T12:55:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:48 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:55:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:48 smithi112 ceph-mon[37501]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:55:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-mon[32659]: pgmap v702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:55:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:rejoin 2024-07-26T12:55:48.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:55:48.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-mon[32659]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:55:48.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:55:48.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: *** Caught signal (Aborted) ** 2024-07-26T12:55:48.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: in thread 7f3815400640 thread_name:ms_dispatch 2024-07-26T12:55:48.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:48.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f381b63e6f0] 2024-07-26T12:55:48.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f381b68b94c] 2024-07-26T12:55:48.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 3: raise() 2024-07-26T12:55:48.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 4: abort() 2024-07-26T12:55:48.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 5: /lib64/libc.so.6(+0x2871b) [0x7f381b62871b] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 6: /lib64/libc.so.6(+0x37386) [0x7f381b637386] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55994a7236b9] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55994a8e180c] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55994a82fcde] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55994a7f1763] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55994a7f2a1b] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55994a7d6752] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55994a724a56] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55994a728637] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55994a728f5e] 2024-07-26T12:55:48.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55994a70c03a] 2024-07-26T12:55:48.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f381c19b407] 2024-07-26T12:55:48.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 18: (DispatchQueue::entry()+0x60f) [0x7f381c19bcf9] 2024-07-26T12:55:48.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f381c2785a1] 2024-07-26T12:55:48.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 20: (Thread::entry_wrapper()+0x43) [0x7f381c025cdd] 2024-07-26T12:55:48.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 21: (Thread::_entry_func(void*)+0xd) [0x7f381c025cf9] 2024-07-26T12:55:48.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 22: /lib64/libc.so.6(+0x89c02) [0x7f381b689c02] 2024-07-26T12:55:48.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f381b70ec40] 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2024-07-26T12:55:48.125+0000 7f3815400640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: in thread 7f3815400640 thread_name:ms_dispatch 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f381b63e6f0] 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f381b68b94c] 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 3: raise() 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 4: abort() 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 5: /lib64/libc.so.6(+0x2871b) [0x7f381b62871b] 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 6: /lib64/libc.so.6(+0x37386) [0x7f381b637386] 2024-07-26T12:55:48.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55994a7236b9] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55994a8e180c] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55994a82fcde] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55994a7f1763] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55994a7f2a1b] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55994a7d6752] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55994a724a56] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55994a728637] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55994a728f5e] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55994a70c03a] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f381c19b407] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 18: (DispatchQueue::entry()+0x60f) [0x7f381c19bcf9] 2024-07-26T12:55:48.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f381c2785a1] 2024-07-26T12:55:48.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 20: (Thread::entry_wrapper()+0x43) [0x7f381c025cdd] 2024-07-26T12:55:48.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 21: (Thread::_entry_func(void*)+0xd) [0x7f381c025cf9] 2024-07-26T12:55:48.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 22: /lib64/libc.so.6(+0x89c02) [0x7f381b689c02] 2024-07-26T12:55:48.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f381b70ec40] 2024-07-26T12:55:48.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:55:48.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2024-07-26T12:55:48.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 0> 2024-07-26T12:55:48.125+0000 7f3815400640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:55:48.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: in thread 7f3815400640 thread_name:ms_dispatch 2024-07-26T12:55:48.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2024-07-26T12:55:48.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:48.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f381b63e6f0] 2024-07-26T12:55:48.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f381b68b94c] 2024-07-26T12:55:48.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 3: raise() 2024-07-26T12:55:48.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 4: abort() 2024-07-26T12:55:48.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 5: /lib64/libc.so.6(+0x2871b) [0x7f381b62871b] 2024-07-26T12:55:48.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 6: /lib64/libc.so.6(+0x37386) [0x7f381b637386] 2024-07-26T12:55:48.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55994a7236b9] 2024-07-26T12:55:48.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55994a8e180c] 2024-07-26T12:55:48.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55994a82fcde] 2024-07-26T12:55:48.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55994a7f1763] 2024-07-26T12:55:48.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55994a7f2a1b] 2024-07-26T12:55:48.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55994a7d6752] 2024-07-26T12:55:48.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55994a724a56] 2024-07-26T12:55:48.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55994a728637] 2024-07-26T12:55:48.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55994a728f5e] 2024-07-26T12:55:48.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55994a70c03a] 2024-07-26T12:55:48.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f381c19b407] 2024-07-26T12:55:48.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 18: (DispatchQueue::entry()+0x60f) [0x7f381c19bcf9] 2024-07-26T12:55:48.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f381c2785a1] 2024-07-26T12:55:48.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 20: (Thread::entry_wrapper()+0x43) [0x7f381c025cdd] 2024-07-26T12:55:48.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 21: (Thread::_entry_func(void*)+0xd) [0x7f381c025cf9] 2024-07-26T12:55:48.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 22: /lib64/libc.so.6(+0x89c02) [0x7f381b689c02] 2024-07-26T12:55:48.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f381b70ec40] 2024-07-26T12:55:48.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:55:48.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 0> 2024-07-26T12:55:48.125+0000 7f3815400640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: in thread 7f3815400640 thread_name:ms_dispatch 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f381b63e6f0] 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f381b68b94c] 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 3: raise() 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 4: abort() 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 5: /lib64/libc.so.6(+0x2871b) [0x7f381b62871b] 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 6: /lib64/libc.so.6(+0x37386) [0x7f381b637386] 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55994a7236b9] 2024-07-26T12:55:48.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55994a8e180c] 2024-07-26T12:55:48.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55994a82fcde] 2024-07-26T12:55:48.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55994a7f1763] 2024-07-26T12:55:48.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55994a7f2a1b] 2024-07-26T12:55:48.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55994a7d6752] 2024-07-26T12:55:48.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55994a724a56] 2024-07-26T12:55:48.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55994a728637] 2024-07-26T12:55:48.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55994a728f5e] 2024-07-26T12:55:48.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55994a70c03a] 2024-07-26T12:55:48.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f381c19b407] 2024-07-26T12:55:48.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 18: (DispatchQueue::entry()+0x60f) [0x7f381c19bcf9] 2024-07-26T12:55:48.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f381c2785a1] 2024-07-26T12:55:48.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 20: (Thread::entry_wrapper()+0x43) [0x7f381c025cdd] 2024-07-26T12:55:48.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 21: (Thread::_entry_func(void*)+0xd) [0x7f381c025cf9] 2024-07-26T12:55:48.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 22: /lib64/libc.so.6(+0x89c02) [0x7f381b689c02] 2024-07-26T12:55:48.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f381b70ec40] 2024-07-26T12:55:48.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:55:48.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[104866]: 2024-07-26T12:55:48.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:48 smithi107 podman[105534]: 2024-07-26 12:55:48.245613051 +0000 UTC m=+0.033158995 container died 0c79b28821d2ca0855ea8dcb71e881a56fadac8e04a2e9783b6b54cdd4522c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T12:55:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:48 smithi132 ceph-mon[37580]: pgmap v702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 10 op/s 2024-07-26T12:55:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:48 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:rejoin 2024-07-26T12:55:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:48 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:55:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:48 smithi132 ceph-mon[37580]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:55:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:49 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:55:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:49 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:active 2024-07-26T12:55:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:49 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:55:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:49 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:55:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:49 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:55:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:49 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:active 2024-07-26T12:55:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:49 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:55:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:49 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:55:49.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:49 smithi107 podman[105534]: 2024-07-26 12:55:49.104190941 +0000 UTC m=+0.891736927 container remove 0c79b28821d2ca0855ea8dcb71e881a56fadac8e04a2e9783b6b54cdd4522c6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T12:55:49.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:49 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:55:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:49 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:55:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:49 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/4178570490,v1:172.21.15.107:6837/4178570490] up:active 2024-07-26T12:55:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:49 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:55:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:49 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:55:50.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:49 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T12:55:50.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:49 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.878s CPU time. 2024-07-26T12:55:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:50 smithi112 ceph-mon[37501]: pgmap v703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T12:55:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:50 smithi107 ceph-mon[32659]: pgmap v703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T12:55:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:50 smithi132 ceph-mon[37580]: pgmap v703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T12:55:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 286.748052 secs 2024-07-26T12:55:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 286.748052 secs 2024-07-26T12:55:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 286.748052 secs 2024-07-26T12:55:52.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:52 smithi107 ceph-mon[32659]: pgmap v704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:55:52.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:52.282+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:52.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:52.282+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:52.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:55:52.282+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:52 smithi112 ceph-mon[37501]: pgmap v704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:55:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:52 smithi132 ceph-mon[37580]: pgmap v704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:55:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:54.415+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:54.415+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:55:54.415+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:55:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:54 smithi112 ceph-mon[37501]: pgmap v705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:55:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:54 smithi107 ceph-mon[32659]: pgmap v705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:55:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:54 smithi132 ceph-mon[37580]: pgmap v705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 16 op/s 2024-07-26T12:55:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:56 smithi112 ceph-mon[37501]: pgmap v706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 184 B/s wr, 15 op/s 2024-07-26T12:55:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 291.748167 secs 2024-07-26T12:55:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:56 smithi107 ceph-mon[32659]: pgmap v706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 184 B/s wr, 15 op/s 2024-07-26T12:55:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 291.748167 secs 2024-07-26T12:55:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:56 smithi132 ceph-mon[37580]: pgmap v706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 184 B/s wr, 15 op/s 2024-07-26T12:55:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 291.748167 secs 2024-07-26T12:55:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:55:58 smithi112 ceph-mon[37501]: pgmap v707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:55:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:55:58 smithi107 ceph-mon[32659]: pgmap v707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:55:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:55:58 smithi132 ceph-mon[37580]: pgmap v707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:56:00.153 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:59 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-07-26T12:56:00.153 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:59 smithi107 systemd[1]: Stopped Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:56:00.153 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:59 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.878s CPU time. 2024-07-26T12:56:00.153 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:55:59 smithi107 systemd[1]: Starting Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:56:00.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:00 smithi107 ceph-mon[32659]: pgmap v708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:56:00.435 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:00 smithi107 podman[105626]: 2024-07-26 12:56:00.10387694 +0000 UTC m=+0.033933339 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:56:00.435 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:00 smithi107 podman[105626]: 2024-07-26 12:56:00.237370114 +0000 UTC m=+0.167426547 container create 3d3f4874b7a6c789481befc5e31db3f6262557036e2d5fd3261eeb049eb675be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_CLEAN=True) 2024-07-26T12:56:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:00 smithi112 ceph-mon[37501]: pgmap v708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:56:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:00 smithi132 ceph-mon[37580]: pgmap v708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:56:00.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:00 smithi107 podman[105626]: 2024-07-26 12:56:00.533032035 +0000 UTC m=+0.463088471 container init 3d3f4874b7a6c789481befc5e31db3f6262557036e2d5fd3261eeb049eb675be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:56:00.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:00 smithi107 podman[105626]: 2024-07-26 12:56:00.543221958 +0000 UTC m=+0.473278364 container start 3d3f4874b7a6c789481befc5e31db3f6262557036e2d5fd3261eeb049eb675be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:56:00.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:00 smithi107 bash[105626]: 3d3f4874b7a6c789481befc5e31db3f6262557036e2d5fd3261eeb049eb675be 2024-07-26T12:56:00.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:00 smithi107 ceph-mds[105645]: -- 172.21.15.107:0/1131085943 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ee6fa6d860 con 0x55ee73c0f180 2024-07-26T12:56:00.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:00 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: starting mds.d at 2024-07-26T12:56:00.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:00 smithi107 systemd[1]: Started Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:56:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 296.748251 secs 2024-07-26T12:56:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:01 smithi112 ceph-mon[37501]: daemon mds.d restarted 2024-07-26T12:56:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:01 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:56:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:56:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:01 smithi112 ceph-mon[37501]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T12:56:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:01 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:boot 2024-07-26T12:56:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:01 smithi112 ceph-mon[37501]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:56:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:01 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:56:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:01 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 296.748251 secs 2024-07-26T12:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:01 smithi107 ceph-mon[32659]: daemon mds.d restarted 2024-07-26T12:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:01 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:01 smithi107 ceph-mon[32659]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T12:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:01 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:boot 2024-07-26T12:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:01 smithi107 ceph-mon[32659]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:01 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:01 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:56:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 296.748251 secs 2024-07-26T12:56:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:01 smithi132 ceph-mon[37580]: daemon mds.d restarted 2024-07-26T12:56:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:01 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:56:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:56:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:01 smithi132 ceph-mon[37580]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T12:56:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:01 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:boot 2024-07-26T12:56:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:01 smithi132 ceph-mon[37580]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T12:56:01.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:01 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:56:01.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:01 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T12:56:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:02 smithi112 ceph-mon[37501]: pgmap v710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 14 op/s 2024-07-26T12:56:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:02 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:resolve 2024-07-26T12:56:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:02 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:56:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:02 smithi107 ceph-mon[32659]: pgmap v710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 14 op/s 2024-07-26T12:56:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:02 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:resolve 2024-07-26T12:56:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:02 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:56:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:02 smithi132 ceph-mon[37580]: pgmap v710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 14 op/s 2024-07-26T12:56:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:02 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:resolve 2024-07-26T12:56:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:02 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T12:56:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:04 smithi112 ceph-mon[37501]: pgmap v711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-07-26T12:56:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:04 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:56:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:04 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:reconnect 2024-07-26T12:56:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:04 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:56:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:04 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:56:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:04 smithi107 ceph-mon[32659]: pgmap v711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-07-26T12:56:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:04 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:56:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:04 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:reconnect 2024-07-26T12:56:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:04 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:56:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:04 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:56:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:04 smithi132 ceph-mon[37580]: pgmap v711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-07-26T12:56:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:04 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:56:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:04 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:reconnect 2024-07-26T12:56:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:04 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T12:56:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:04 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:56:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:05 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:rejoin 2024-07-26T12:56:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:05 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:56:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:05 smithi112 ceph-mon[37501]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:56:05.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:05 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:rejoin 2024-07-26T12:56:05.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:05 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:56:05.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:05 smithi132 ceph-mon[37580]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:56:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:rejoin 2024-07-26T12:56:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T12:56:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-mon[32659]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T12:56:05.543 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: *** Caught signal (Aborted) ** 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: in thread 7fe5f5400640 thread_name:ms_dispatch 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe5fb63e6f0] 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe5fb68b94c] 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 3: raise() 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 4: abort() 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe5fb62871b] 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 6: /lib64/libc.so.6(+0x37386) [0x7fe5fb637386] 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559b39f2e6b9] 2024-07-26T12:56:05.544 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559b3a0ec80c] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559b3a03acde] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559b39ffc763] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559b39ffda1b] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559b39fe1752] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559b39f2fa56] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559b39f33637] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559b39f33f5e] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559b39f1703a] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe5fc19b407] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 18: (DispatchQueue::entry()+0x60f) [0x7fe5fc19bcf9] 2024-07-26T12:56:05.545 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe5fc2785a1] 2024-07-26T12:56:05.546 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 20: (Thread::entry_wrapper()+0x43) [0x7fe5fc025cdd] 2024-07-26T12:56:05.546 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe5fc025cf9] 2024-07-26T12:56:05.546 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe5fb689c02] 2024-07-26T12:56:05.546 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe5fb70ec40] 2024-07-26T12:56:05.546 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2024-07-26T12:56:05.208+0000 7fe5f5400640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:56:05.546 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: in thread 7fe5f5400640 thread_name:ms_dispatch 2024-07-26T12:56:05.546 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2024-07-26T12:56:05.546 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe5fb63e6f0] 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe5fb68b94c] 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 3: raise() 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 4: abort() 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe5fb62871b] 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 6: /lib64/libc.so.6(+0x37386) [0x7fe5fb637386] 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559b39f2e6b9] 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559b3a0ec80c] 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559b3a03acde] 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559b39ffc763] 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559b39ffda1b] 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559b39fe1752] 2024-07-26T12:56:05.547 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559b39f2fa56] 2024-07-26T12:56:05.548 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559b39f33637] 2024-07-26T12:56:05.548 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559b39f33f5e] 2024-07-26T12:56:05.548 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559b39f1703a] 2024-07-26T12:56:05.548 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe5fc19b407] 2024-07-26T12:56:05.548 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 18: (DispatchQueue::entry()+0x60f) [0x7fe5fc19bcf9] 2024-07-26T12:56:05.548 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe5fc2785a1] 2024-07-26T12:56:05.548 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 20: (Thread::entry_wrapper()+0x43) [0x7fe5fc025cdd] 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe5fc025cf9] 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe5fb689c02] 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe5fb70ec40] 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 0> 2024-07-26T12:56:05.208+0000 7fe5f5400640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: in thread 7fe5f5400640 thread_name:ms_dispatch 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe5fb63e6f0] 2024-07-26T12:56:05.549 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe5fb68b94c] 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 3: raise() 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 4: abort() 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe5fb62871b] 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 6: /lib64/libc.so.6(+0x37386) [0x7fe5fb637386] 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559b39f2e6b9] 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559b3a0ec80c] 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559b3a03acde] 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559b39ffc763] 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559b39ffda1b] 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559b39fe1752] 2024-07-26T12:56:05.550 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559b39f2fa56] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559b39f33637] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559b39f33f5e] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559b39f1703a] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe5fc19b407] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 18: (DispatchQueue::entry()+0x60f) [0x7fe5fc19bcf9] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe5fc2785a1] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 20: (Thread::entry_wrapper()+0x43) [0x7fe5fc025cdd] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe5fc025cf9] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe5fb689c02] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe5fb70ec40] 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2024-07-26T12:56:05.551 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 0> 2024-07-26T12:56:05.208+0000 7fe5f5400640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: in thread 7fe5f5400640 thread_name:ms_dispatch 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe5fb63e6f0] 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe5fb68b94c] 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 3: raise() 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 4: abort() 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe5fb62871b] 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 6: /lib64/libc.so.6(+0x37386) [0x7fe5fb637386] 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559b39f2e6b9] 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559b3a0ec80c] 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x559b3a03acde] 2024-07-26T12:56:05.552 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x559b39ffc763] 2024-07-26T12:56:05.553 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559b39ffda1b] 2024-07-26T12:56:05.553 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559b39fe1752] 2024-07-26T12:56:05.553 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559b39f2fa56] 2024-07-26T12:56:05.553 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559b39f33637] 2024-07-26T12:56:05.553 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559b39f33f5e] 2024-07-26T12:56:05.553 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559b39f1703a] 2024-07-26T12:56:05.553 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe5fc19b407] 2024-07-26T12:56:05.554 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 18: (DispatchQueue::entry()+0x60f) [0x7fe5fc19bcf9] 2024-07-26T12:56:05.554 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe5fc2785a1] 2024-07-26T12:56:05.554 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 20: (Thread::entry_wrapper()+0x43) [0x7fe5fc025cdd] 2024-07-26T12:56:05.554 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe5fc025cf9] 2024-07-26T12:56:05.554 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe5fb689c02] 2024-07-26T12:56:05.554 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe5fb70ec40] 2024-07-26T12:56:05.554 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:56:05.554 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a[105494]: 2024-07-26T12:56:05.554 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:05 smithi107 podman[105680]: 2024-07-26 12:56:05.371873935 +0000 UTC m=+0.034770873 container died f7c493a44ede64a0e8e66cf44d3cb832b7dcf251a3cbabe8ebb560a22e505ea3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=) 2024-07-26T12:56:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:06 smithi112 ceph-mon[37501]: pgmap v712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T12:56:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 301.748329 secs 2024-07-26T12:56:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:06 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:56:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:06 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:active 2024-07-26T12:56:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:06 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:56:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:06 smithi112 ceph-mon[37501]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:56:06.542 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:06 smithi107 podman[105680]: 2024-07-26 12:56:06.210993827 +0000 UTC m=+0.873890826 container remove f7c493a44ede64a0e8e66cf44d3cb832b7dcf251a3cbabe8ebb560a22e505ea3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=) 2024-07-26T12:56:06.542 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:06 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:56:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:06 smithi107 ceph-mon[32659]: pgmap v712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T12:56:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 301.748329 secs 2024-07-26T12:56:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:06 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:56:06.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:06 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:active 2024-07-26T12:56:06.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:06 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:56:06.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:06 smithi107 ceph-mon[32659]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:56:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:06 smithi132 ceph-mon[37580]: pgmap v712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T12:56:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 301.748329 secs 2024-07-26T12:56:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:06 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:56:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:06 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6840/160144645,v1:172.21.15.107:6841/160144645] up:active 2024-07-26T12:56:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:06 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T12:56:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:06 smithi132 ceph-mon[37580]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T12:56:07.042 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:06 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T12:56:07.042 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:06 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.939s CPU time. 2024-07-26T12:56:07.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:07.297+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:07.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:07.297+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:07.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:07.297+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:08 smithi112 ceph-mon[37501]: pgmap v713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 13 op/s 2024-07-26T12:56:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:08 smithi107 ceph-mon[32659]: pgmap v713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 13 op/s 2024-07-26T12:56:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:08 smithi132 ceph-mon[37580]: pgmap v713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 13 op/s 2024-07-26T12:56:09.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:09.418+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:09.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:09.418+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:09.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:09.418+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:10 smithi107 ceph-mon[32659]: pgmap v714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:56:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 306.748450 secs 2024-07-26T12:56:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:10 smithi132 ceph-mon[37580]: pgmap v714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:56:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 306.748450 secs 2024-07-26T12:56:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:10 smithi112 ceph-mon[37501]: pgmap v714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:56:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 306.748450 secs 2024-07-26T12:56:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:12 smithi107 ceph-mon[32659]: pgmap v715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 198 B/s wr, 15 op/s 2024-07-26T12:56:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:12 smithi132 ceph-mon[37580]: pgmap v715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 198 B/s wr, 15 op/s 2024-07-26T12:56:12.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:12 smithi112 ceph-mon[37501]: pgmap v715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 198 B/s wr, 15 op/s 2024-07-26T12:56:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:14 smithi107 ceph-mon[32659]: pgmap v716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:56:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:14 smithi132 ceph-mon[37580]: pgmap v716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:56:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:14 smithi112 ceph-mon[37501]: pgmap v716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 170 B/s wr, 16 op/s 2024-07-26T12:56:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 311.748563 secs 2024-07-26T12:56:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 311.748563 secs 2024-07-26T12:56:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 311.748563 secs 2024-07-26T12:56:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:16 smithi107 ceph-mon[32659]: pgmap v717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:56:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:16 smithi132 ceph-mon[37580]: pgmap v717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:56:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:16 smithi112 ceph-mon[37501]: pgmap v717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:56:17.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:16 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-07-26T12:56:17.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:16 smithi107 systemd[1]: Stopped Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:56:17.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:16 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.939s CPU time. 2024-07-26T12:56:17.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:16 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-07-26T12:56:17.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:16 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T12:56:17.293 INFO:journalctl@ceph.mds.a.smithi107.stdout:Jul 26 12:56:16 smithi107 systemd[1]: Failed to start Ceph mds.a for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:56:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:18 smithi107 ceph-mon[32659]: pgmap v718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:56:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:18 smithi132 ceph-mon[37580]: pgmap v718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:56:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:18 smithi112 ceph-mon[37501]: pgmap v718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 14 op/s 2024-07-26T12:56:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:20 smithi107 ceph-mon[32659]: pgmap v719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T12:56:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 316.748644 secs 2024-07-26T12:56:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:20 smithi132 ceph-mon[37580]: pgmap v719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T12:56:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 316.748644 secs 2024-07-26T12:56:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:20 smithi112 ceph-mon[37501]: pgmap v719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T12:56:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 316.748644 secs 2024-07-26T12:56:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:22.311+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:22.311+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:22.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:22.311+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:22 smithi112 ceph-mon[37501]: pgmap v720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:22 smithi112 ceph-mon[37501]: Replacing daemon mds.a as rank 0 with standby daemon mds.d 2024-07-26T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:22 smithi112 ceph-mon[37501]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:22 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:22 smithi112 ceph-mon[37501]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T12:56:22.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:22 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T12:56:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:22 smithi107 ceph-mon[32659]: pgmap v720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:56:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:22 smithi107 ceph-mon[32659]: Replacing daemon mds.a as rank 0 with standby daemon mds.d 2024-07-26T12:56:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:22 smithi107 ceph-mon[32659]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T12:56:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:22 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:56:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:22 smithi107 ceph-mon[32659]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T12:56:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:22 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T12:56:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:22 smithi132 ceph-mon[37580]: pgmap v720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:56:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:22 smithi132 ceph-mon[37580]: Replacing daemon mds.a as rank 0 with standby daemon mds.d 2024-07-26T12:56:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:22 smithi132 ceph-mon[37580]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T12:56:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:22 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:56:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:22 smithi132 ceph-mon[37580]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T12:56:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:22 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T12:56:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:24.430+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:24.430+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:24.432 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:24.430+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:24.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:24 smithi107 ceph-mon[32659]: pgmap v722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:56:24.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:24 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:resolve 2024-07-26T12:56:24.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:24 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T12:56:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:24 smithi132 ceph-mon[37580]: pgmap v722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:56:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:24 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:resolve 2024-07-26T12:56:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:24 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T12:56:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:24 smithi112 ceph-mon[37501]: pgmap v722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T12:56:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:24 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:resolve 2024-07-26T12:56:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:24 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T12:56:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:25 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:56:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:25 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:reconnect 2024-07-26T12:56:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:25 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T12:56:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:25 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:56:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 321.748751 secs 2024-07-26T12:56:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:25 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:56:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:25 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:reconnect 2024-07-26T12:56:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:25 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T12:56:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:25 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:56:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 321.748751 secs 2024-07-26T12:56:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:25 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00299995 2024-07-26T12:56:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:25 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:reconnect 2024-07-26T12:56:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:25 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T12:56:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:25 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:56:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 321.748751 secs 2024-07-26T12:56:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-mon[32659]: pgmap v723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:56:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:rejoin 2024-07-26T12:56:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T12:56:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-mon[32659]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:56:26.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:56:26.543 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: *** Caught signal (Aborted) ** 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: in thread 7f4ce3000640 thread_name:ms_dispatch 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4ce923e6f0] 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4ce928b94c] 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 3: raise() 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 4: abort() 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4ce922871b] 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 6: /lib64/libc.so.6(+0x37386) [0x7f4ce9237386] 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ee4e0826b9] 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ee4e24080c] 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ee4e18ecde] 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ee4e150763] 2024-07-26T12:56:26.544 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ee4e151a1b] 2024-07-26T12:56:26.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ee4e135752] 2024-07-26T12:56:26.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ee4e083a56] 2024-07-26T12:56:26.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ee4e087637] 2024-07-26T12:56:26.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ee4e087f5e] 2024-07-26T12:56:26.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ee4e06b03a] 2024-07-26T12:56:26.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4ce9d9b407] 2024-07-26T12:56:26.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 18: (DispatchQueue::entry()+0x60f) [0x7f4ce9d9bcf9] 2024-07-26T12:56:26.545 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4ce9e785a1] 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 20: (Thread::entry_wrapper()+0x43) [0x7f4ce9c25cdd] 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4ce9c25cf9] 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4ce9289c02] 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4ce930ec40] 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2024-07-26T12:56:26.326+0000 7f4ce3000640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: in thread 7f4ce3000640 thread_name:ms_dispatch 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4ce923e6f0] 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4ce928b94c] 2024-07-26T12:56:26.546 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 3: raise() 2024-07-26T12:56:26.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 4: abort() 2024-07-26T12:56:26.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4ce922871b] 2024-07-26T12:56:26.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 6: /lib64/libc.so.6(+0x37386) [0x7f4ce9237386] 2024-07-26T12:56:26.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ee4e0826b9] 2024-07-26T12:56:26.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ee4e24080c] 2024-07-26T12:56:26.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ee4e18ecde] 2024-07-26T12:56:26.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ee4e150763] 2024-07-26T12:56:26.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ee4e151a1b] 2024-07-26T12:56:26.547 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ee4e135752] 2024-07-26T12:56:26.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ee4e083a56] 2024-07-26T12:56:26.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ee4e087637] 2024-07-26T12:56:26.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ee4e087f5e] 2024-07-26T12:56:26.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ee4e06b03a] 2024-07-26T12:56:26.548 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4ce9d9b407] 2024-07-26T12:56:26.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 18: (DispatchQueue::entry()+0x60f) [0x7f4ce9d9bcf9] 2024-07-26T12:56:26.549 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4ce9e785a1] 2024-07-26T12:56:26.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 20: (Thread::entry_wrapper()+0x43) [0x7f4ce9c25cdd] 2024-07-26T12:56:26.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4ce9c25cf9] 2024-07-26T12:56:26.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4ce9289c02] 2024-07-26T12:56:26.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4ce930ec40] 2024-07-26T12:56:26.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:56:26.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2024-07-26T12:56:26.550 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 0> 2024-07-26T12:56:26.326+0000 7f4ce3000640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:56:26.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: in thread 7f4ce3000640 thread_name:ms_dispatch 2024-07-26T12:56:26.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2024-07-26T12:56:26.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:26.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4ce923e6f0] 2024-07-26T12:56:26.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4ce928b94c] 2024-07-26T12:56:26.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 3: raise() 2024-07-26T12:56:26.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 4: abort() 2024-07-26T12:56:26.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4ce922871b] 2024-07-26T12:56:26.551 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 6: /lib64/libc.so.6(+0x37386) [0x7f4ce9237386] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ee4e0826b9] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ee4e24080c] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ee4e18ecde] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ee4e150763] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ee4e151a1b] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ee4e135752] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ee4e083a56] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ee4e087637] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ee4e087f5e] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ee4e06b03a] 2024-07-26T12:56:26.552 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4ce9d9b407] 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 18: (DispatchQueue::entry()+0x60f) [0x7f4ce9d9bcf9] 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4ce9e785a1] 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 20: (Thread::entry_wrapper()+0x43) [0x7f4ce9c25cdd] 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4ce9c25cf9] 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4ce9289c02] 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4ce930ec40] 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 0> 2024-07-26T12:56:26.326+0000 7f4ce3000640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: in thread 7f4ce3000640 thread_name:ms_dispatch 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4ce923e6f0] 2024-07-26T12:56:26.553 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4ce928b94c] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 3: raise() 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 4: abort() 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4ce922871b] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 6: /lib64/libc.so.6(+0x37386) [0x7f4ce9237386] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ee4e0826b9] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ee4e24080c] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55ee4e18ecde] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55ee4e150763] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ee4e151a1b] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ee4e135752] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ee4e083a56] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ee4e087637] 2024-07-26T12:56:26.554 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ee4e087f5e] 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ee4e06b03a] 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4ce9d9b407] 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 18: (DispatchQueue::entry()+0x60f) [0x7f4ce9d9bcf9] 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4ce9e785a1] 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 20: (Thread::entry_wrapper()+0x43) [0x7f4ce9c25cdd] 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4ce9c25cf9] 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4ce9289c02] 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4ce930ec40] 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105641]: 2024-07-26T12:56:26.555 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:26 smithi107 podman[105727]: 2024-07-26 12:56:26.468730928 +0000 UTC m=+0.034136242 container died 3d3f4874b7a6c789481befc5e31db3f6262557036e2d5fd3261eeb049eb675be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-07-26T12:56:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:26 smithi132 ceph-mon[37580]: pgmap v723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:56:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:26 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:rejoin 2024-07-26T12:56:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:26 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T12:56:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:26 smithi132 ceph-mon[37580]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:56:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:26 smithi112 ceph-mon[37501]: pgmap v723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T12:56:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:26 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:rejoin 2024-07-26T12:56:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:26 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T12:56:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:26 smithi112 ceph-mon[37501]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:56:27.542 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:27 smithi107 podman[105727]: 2024-07-26 12:56:27.23862817 +0000 UTC m=+0.804033515 container remove 3d3f4874b7a6c789481befc5e31db3f6262557036e2d5fd3261eeb049eb675be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:56:27.542 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:27 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:56:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:27 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:56:27.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:27 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:active 2024-07-26T12:56:27.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:27 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T12:56:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:27 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:56:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:27 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:active 2024-07-26T12:56:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:27 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T12:56:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:27 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:56:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:27 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/1888877963,v1:172.21.15.107:6837/1888877963] up:active 2024-07-26T12:56:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:27 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T12:56:28.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:27 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T12:56:28.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:27 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.911s CPU time. 2024-07-26T12:56:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:28 smithi132 ceph-mon[37580]: pgmap v724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-07-26T12:56:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:28 smithi112 ceph-mon[37501]: pgmap v724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-07-26T12:56:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:28 smithi107 ceph-mon[32659]: pgmap v724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 13 op/s 2024-07-26T12:56:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:30 smithi112 ceph-mon[37501]: pgmap v725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T12:56:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 326.748837 secs 2024-07-26T12:56:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:30 smithi107 ceph-mon[32659]: pgmap v725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T12:56:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 326.748837 secs 2024-07-26T12:56:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:30 smithi132 ceph-mon[37580]: pgmap v725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 204 B/s wr, 14 op/s 2024-07-26T12:56:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 326.748837 secs 2024-07-26T12:56:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:32 smithi112 ceph-mon[37501]: pgmap v726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:56:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:32 smithi107 ceph-mon[32659]: pgmap v726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:56:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:32 smithi132 ceph-mon[37580]: pgmap v726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:56:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:34 smithi112 ceph-mon[37501]: pgmap v727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 182 B/s wr, 15 op/s 2024-07-26T12:56:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:34 smithi107 ceph-mon[32659]: pgmap v727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 182 B/s wr, 15 op/s 2024-07-26T12:56:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:34 smithi132 ceph-mon[37580]: pgmap v727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 182 B/s wr, 15 op/s 2024-07-26T12:56:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 331.748953 secs 2024-07-26T12:56:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:35.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 331.748953 secs 2024-07-26T12:56:35.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 331.748953 secs 2024-07-26T12:56:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:36 smithi112 ceph-mon[37501]: pgmap v728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T12:56:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:56:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:36 smithi107 ceph-mon[32659]: pgmap v728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T12:56:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:56:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:36 smithi132 ceph-mon[37580]: pgmap v728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 170 B/s wr, 12 op/s 2024-07-26T12:56:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:56:37.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:37.327+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:37.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:37.327+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:37.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:37.327+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:37 smithi112 ceph-mon[37501]: pgmap v729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 182 B/s wr, 12 op/s 2024-07-26T12:56:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:37 smithi112 ceph-mon[37501]: pgmap v730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 222 B/s wr, 10 op/s 2024-07-26T12:56:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:37 smithi107 ceph-mon[32659]: pgmap v729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 182 B/s wr, 12 op/s 2024-07-26T12:56:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:37 smithi107 ceph-mon[32659]: pgmap v730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 222 B/s wr, 10 op/s 2024-07-26T12:56:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:37 smithi132 ceph-mon[37580]: pgmap v729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 182 B/s wr, 12 op/s 2024-07-26T12:56:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:37 smithi132 ceph-mon[37580]: pgmap v730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 222 B/s wr, 10 op/s 2024-07-26T12:56:38.188 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:37 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-07-26T12:56:38.188 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:37 smithi107 systemd[1]: Stopped Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:56:38.188 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:37 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.911s CPU time. 2024-07-26T12:56:38.188 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:37 smithi107 systemd[1]: Starting Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968... 2024-07-26T12:56:38.510 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:38 smithi107 podman[105949]: 2024-07-26 12:56:38.092781073 +0000 UTC m=+0.036198833 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T12:56:38.510 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:38 smithi107 podman[105949]: 2024-07-26 12:56:38.221920981 +0000 UTC m=+0.165338753 container create 2c8b6da2bd7ad4e7474c36a2cff915bd15be7dd0446f96e4055a6f662de72d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-07-26T12:56:38.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:38 smithi107 podman[105949]: 2024-07-26 12:56:38.509243837 +0000 UTC m=+0.452661656 container init 2c8b6da2bd7ad4e7474c36a2cff915bd15be7dd0446f96e4055a6f662de72d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-07-26T12:56:38.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:38 smithi107 podman[105949]: 2024-07-26 12:56:38.519158517 +0000 UTC m=+0.462576262 container start 2c8b6da2bd7ad4e7474c36a2cff915bd15be7dd0446f96e4055a6f662de72d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T12:56:38.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:38 smithi107 ceph-mds[105968]: -- 172.21.15.107:0/2839256727 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af3cedf860 con 0x55af41081180 2024-07-26T12:56:38.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:38 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: starting mds.d at 2024-07-26T12:56:38.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:38 smithi107 bash[105949]: 2c8b6da2bd7ad4e7474c36a2cff915bd15be7dd0446f96e4055a6f662de72d66 2024-07-26T12:56:38.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:38 smithi107 systemd[1]: Started Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:56:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:39.446+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:39.446+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:39.446+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:39 smithi112 ceph-mon[37501]: pgmap v731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:56:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:39 smithi112 ceph-mon[37501]: daemon mds.d restarted 2024-07-26T12:56:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:39 smithi112 ceph-mon[37501]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:56:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:56:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:39 smithi112 ceph-mon[37501]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T12:56:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:39 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:boot 2024-07-26T12:56:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:39 smithi112 ceph-mon[37501]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:56:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:39 smithi112 ceph-mon[37501]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:56:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:39 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T12:56:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:39 smithi107 ceph-mon[32659]: pgmap v731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:56:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:39 smithi107 ceph-mon[32659]: daemon mds.d restarted 2024-07-26T12:56:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:39 smithi107 ceph-mon[32659]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:56:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:56:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:39 smithi107 ceph-mon[32659]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T12:56:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:39 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:boot 2024-07-26T12:56:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:39 smithi107 ceph-mon[32659]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:56:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:39 smithi107 ceph-mon[32659]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:56:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:39 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T12:56:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:39 smithi132 ceph-mon[37580]: pgmap v731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:56:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:39 smithi132 ceph-mon[37580]: daemon mds.d restarted 2024-07-26T12:56:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:39 smithi132 ceph-mon[37580]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T12:56:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T12:56:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:39 smithi132 ceph-mon[37580]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T12:56:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:39 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:boot 2024-07-26T12:56:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:39 smithi132 ceph-mon[37580]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-07-26T12:56:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:39 smithi132 ceph-mon[37580]: fsmap cephfs:4/5 {1=g=up:active,2=i=up:active,3=c=up:active,4=f=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T12:56:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:39 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T12:56:40.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:40 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:resolve 2024-07-26T12:56:40.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:40 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T12:56:40.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 336.749040 secs 2024-07-26T12:56:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:40 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:resolve 2024-07-26T12:56:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:40 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T12:56:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 336.749040 secs 2024-07-26T12:56:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:40 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:resolve 2024-07-26T12:56:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:40 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T12:56:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 336.749040 secs 2024-07-26T12:56:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:41 smithi112 ceph-mon[37501]: pgmap v733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:41 smithi112 ceph-mon[37501]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:56:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:41 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:reconnect 2024-07-26T12:56:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:41 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T12:56:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:41 smithi112 ceph-mon[37501]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:56:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:41 smithi107 ceph-mon[32659]: pgmap v733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:41 smithi107 ceph-mon[32659]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:56:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:41 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:reconnect 2024-07-26T12:56:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:41 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T12:56:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:41 smithi107 ceph-mon[32659]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:56:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:41 smithi132 ceph-mon[37580]: pgmap v733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:41 smithi132 ceph-mon[37580]: reconnect by client.15393 172.21.15.107:0/1594965163 after 0.00399993 2024-07-26T12:56:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:41 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:reconnect 2024-07-26T12:56:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:41 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T12:56:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:41 smithi132 ceph-mon[37580]: reconnect by client.15267 192.168.0.1:0/1025124677 after 0.00999984 2024-07-26T12:56:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:42 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:rejoin 2024-07-26T12:56:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:42 smithi112 ceph-mon[37501]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T12:56:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:42 smithi112 ceph-mon[37501]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:56:43.038 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:rejoin 2024-07-26T12:56:43.038 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-mon[32659]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T12:56:43.038 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-mon[32659]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:56:43.039 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T12:56:43.039 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: *** Caught signal (Aborted) ** 2024-07-26T12:56:43.039 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: in thread 7f9c68600640 thread_name:ms_dispatch 2024-07-26T12:56:43.039 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9c6e83e6f0] 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9c6e88b94c] 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 3: raise() 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 4: abort() 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9c6e82871b] 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 6: /lib64/libc.so.6(+0x37386) [0x7f9c6e837386] 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55af21b746b9] 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55af21d3280c] 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55af21c80cde] 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55af21c42763] 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55af21c43a1b] 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55af21c27752] 2024-07-26T12:56:43.040 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55af21b75a56] 2024-07-26T12:56:43.041 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55af21b79637] 2024-07-26T12:56:43.041 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55af21b79f5e] 2024-07-26T12:56:43.041 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55af21b5d03a] 2024-07-26T12:56:43.041 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9c6f39b407] 2024-07-26T12:56:43.041 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 18: (DispatchQueue::entry()+0x60f) [0x7f9c6f39bcf9] 2024-07-26T12:56:43.041 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9c6f4785a1] 2024-07-26T12:56:43.041 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 20: (Thread::entry_wrapper()+0x43) [0x7f9c6f225cdd] 2024-07-26T12:56:43.041 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9c6f225cf9] 2024-07-26T12:56:43.041 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9c6e889c02] 2024-07-26T12:56:43.041 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9c6e90ec40] 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2024-07-26T12:56:42.723+0000 7f9c68600640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: in thread 7f9c68600640 thread_name:ms_dispatch 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9c6e83e6f0] 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9c6e88b94c] 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 3: raise() 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 4: abort() 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9c6e82871b] 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 6: /lib64/libc.so.6(+0x37386) [0x7f9c6e837386] 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55af21b746b9] 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55af21d3280c] 2024-07-26T12:56:43.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55af21c80cde] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55af21c42763] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55af21c43a1b] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55af21c27752] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55af21b75a56] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55af21b79637] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55af21b79f5e] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55af21b5d03a] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9c6f39b407] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 18: (DispatchQueue::entry()+0x60f) [0x7f9c6f39bcf9] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9c6f4785a1] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 20: (Thread::entry_wrapper()+0x43) [0x7f9c6f225cdd] 2024-07-26T12:56:43.043 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9c6f225cf9] 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9c6e889c02] 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9c6e90ec40] 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 0> 2024-07-26T12:56:42.723+0000 7f9c68600640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: in thread 7f9c68600640 thread_name:ms_dispatch 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9c6e83e6f0] 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9c6e88b94c] 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 3: raise() 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 4: abort() 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9c6e82871b] 2024-07-26T12:56:43.044 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 6: /lib64/libc.so.6(+0x37386) [0x7f9c6e837386] 2024-07-26T12:56:43.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55af21b746b9] 2024-07-26T12:56:43.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55af21d3280c] 2024-07-26T12:56:43.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55af21c80cde] 2024-07-26T12:56:43.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55af21c42763] 2024-07-26T12:56:43.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55af21c43a1b] 2024-07-26T12:56:43.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55af21c27752] 2024-07-26T12:56:43.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55af21b75a56] 2024-07-26T12:56:43.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55af21b79637] 2024-07-26T12:56:43.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55af21b79f5e] 2024-07-26T12:56:43.045 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55af21b5d03a] 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9c6f39b407] 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 18: (DispatchQueue::entry()+0x60f) [0x7f9c6f39bcf9] 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9c6f4785a1] 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 20: (Thread::entry_wrapper()+0x43) [0x7f9c6f225cdd] 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9c6f225cf9] 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9c6e889c02] 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9c6e90ec40] 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 0> 2024-07-26T12:56:42.723+0000 7f9c68600640 -1 *** Caught signal (Aborted) ** 2024-07-26T12:56:43.046 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: in thread 7f9c68600640 thread_name:ms_dispatch 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9c6e83e6f0] 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9c6e88b94c] 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 3: raise() 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 4: abort() 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9c6e82871b] 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 6: /lib64/libc.so.6(+0x37386) [0x7f9c6e837386] 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55af21b746b9] 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55af21d3280c] 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 9: (Server::handle_peer_rename_prep(boost::intrusive_ptr const&)+0x4fe) [0x55af21c80cde] 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x933) [0x55af21c42763] 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55af21c43a1b] 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55af21c27752] 2024-07-26T12:56:43.047 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55af21b75a56] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55af21b79637] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55af21b79f5e] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55af21b5d03a] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9c6f39b407] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 18: (DispatchQueue::entry()+0x60f) [0x7f9c6f39bcf9] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9c6f4785a1] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 20: (Thread::entry_wrapper()+0x43) [0x7f9c6f225cdd] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9c6f225cf9] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9c6e889c02] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9c6e90ec40] 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d[105964]: 2024-07-26T12:56:43.048 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:42 smithi107 podman[106003]: 2024-07-26 12:56:42.841410571 +0000 UTC m=+0.032439101 container died 2c8b6da2bd7ad4e7474c36a2cff915bd15be7dd0446f96e4055a6f662de72d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T12:56:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:42 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:rejoin 2024-07-26T12:56:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:42 smithi132 ceph-mon[37580]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T12:56:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:42 smithi132 ceph-mon[37580]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-07-26T12:56:44.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:43 smithi107 podman[106003]: 2024-07-26 12:56:43.698000692 +0000 UTC m=+0.889029260 container remove 2c8b6da2bd7ad4e7474c36a2cff915bd15be7dd0446f96e4055a6f662de72d66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968-mds-d, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-07-26T12:56:44.042 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:43 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T12:56:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:43 smithi107 ceph-mon[32659]: pgmap v734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 8 op/s 2024-07-26T12:56:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:43 smithi107 ceph-mon[32659]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:56:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:43 smithi107 ceph-mon[32659]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:active 2024-07-26T12:56:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:43 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T12:56:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:43 smithi132 ceph-mon[37580]: pgmap v734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 8 op/s 2024-07-26T12:56:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:43 smithi132 ceph-mon[37580]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:56:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:43 smithi132 ceph-mon[37580]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:active 2024-07-26T12:56:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:43 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T12:56:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:43 smithi112 ceph-mon[37501]: pgmap v734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 8 op/s 2024-07-26T12:56:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:43 smithi112 ceph-mon[37501]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T12:56:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:43 smithi112 ceph-mon[37501]: mds.? [v2:172.21.15.107:6836/2097724764,v1:172.21.15.107:6837/2097724764] up:active 2024-07-26T12:56:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:43 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T12:56:44.471 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:44 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T12:56:44.471 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:44 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.737s CPU time. 2024-07-26T12:56:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:45 smithi107 ceph-mon[32659]: pgmap v735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 255 B/s wr, 14 op/s 2024-07-26T12:56:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 341.749161 secs 2024-07-26T12:56:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:45 smithi132 ceph-mon[37580]: pgmap v735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 255 B/s wr, 14 op/s 2024-07-26T12:56:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 341.749161 secs 2024-07-26T12:56:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:45 smithi112 ceph-mon[37501]: pgmap v735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 255 B/s wr, 14 op/s 2024-07-26T12:56:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 341.749161 secs 2024-07-26T12:56:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:47 smithi107 ceph-mon[32659]: pgmap v736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:56:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:47 smithi132 ceph-mon[37580]: pgmap v736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:56:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:47 smithi112 ceph-mon[37501]: pgmap v736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 204 B/s wr, 12 op/s 2024-07-26T12:56:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:49 smithi107 ceph-mon[32659]: pgmap v737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:56:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:49 smithi132 ceph-mon[37580]: pgmap v737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:56:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:49 smithi112 ceph-mon[37501]: pgmap v737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 204 B/s wr, 15 op/s 2024-07-26T12:56:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 346.749263 secs 2024-07-26T12:56:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 346.749263 secs 2024-07-26T12:56:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 346.749263 secs 2024-07-26T12:56:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:51 smithi107 ceph-mon[32659]: pgmap v738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 176 B/s wr, 13 op/s 2024-07-26T12:56:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:51 smithi132 ceph-mon[37580]: pgmap v738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 176 B/s wr, 13 op/s 2024-07-26T12:56:52.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:51 smithi112 ceph-mon[37501]: pgmap v738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 176 B/s wr, 13 op/s 2024-07-26T12:56:52.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:52.343+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:52.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:52.344+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:52.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:56:52.344+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:53 smithi107 ceph-mon[32659]: pgmap v739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T12:56:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:53 smithi132 ceph-mon[37580]: pgmap v739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T12:56:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:53 smithi112 ceph-mon[37501]: pgmap v739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T12:56:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:54.455+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:54.456+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:56:54.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:56:54.456+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:54.792 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:54 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-07-26T12:56:54.792 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:54 smithi107 systemd[1]: Stopped Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:56:54.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:54 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.737s CPU time. 2024-07-26T12:56:54.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:54 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-07-26T12:56:54.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:54 smithi107 systemd[1]: ceph-14de5e40-4b4b-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T12:56:54.793 INFO:journalctl@ceph.mds.d.smithi107.stdout:Jul 26 12:56:54 smithi107 systemd[1]: Failed to start Ceph mds.d for 14de5e40-4b4b-11ef-bcb7-c7b262605968. 2024-07-26T12:56:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:55 smithi107 ceph-mon[32659]: pgmap v740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T12:56:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 351.749384 secs 2024-07-26T12:56:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:55 smithi132 ceph-mon[37580]: pgmap v740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T12:56:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 351.749384 secs 2024-07-26T12:56:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:55 smithi112 ceph-mon[37501]: pgmap v740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 170 B/s wr, 13 op/s 2024-07-26T12:56:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 351.749384 secs 2024-07-26T12:56:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:57 smithi107 ceph-mon[32659]: pgmap v741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:56:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:57 smithi107 ceph-mon[32659]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T12:56:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:57 smithi132 ceph-mon[37580]: pgmap v741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:56:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:57 smithi132 ceph-mon[37580]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T12:56:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:57 smithi112 ceph-mon[37501]: pgmap v741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T12:56:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:57 smithi112 ceph-mon[37501]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T12:57:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:56:59 smithi107 ceph-mon[32659]: pgmap v742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-07-26T12:57:00.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:56:59 smithi132 ceph-mon[37580]: pgmap v742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-07-26T12:57:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:56:59 smithi112 ceph-mon[37501]: pgmap v742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-07-26T12:57:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 356.749539 secs 2024-07-26T12:57:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 356.749539 secs 2024-07-26T12:57:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 356.749539 secs 2024-07-26T12:57:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:01 smithi132 ceph-mon[37580]: pgmap v743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:02.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:01 smithi107 ceph-mon[32659]: pgmap v743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:01 smithi112 ceph-mon[37501]: pgmap v743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:03 smithi112 ceph-mon[37501]: pgmap v744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:04.219 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:03 smithi107 ceph-mon[32659]: pgmap v744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:03 smithi132 ceph-mon[37580]: pgmap v744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:06.111 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:05 smithi132 ceph-mon[37580]: pgmap v745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:06.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:05 smithi132 ceph-mon[37580]: 1 slow requests, 1 included below; oldest blocked for > 361.749650 secs 2024-07-26T12:57:06.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:05 smithi132 ceph-mon[37580]: slow request 361.749649 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T12:57:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:05 smithi112 ceph-mon[37501]: pgmap v745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:05 smithi112 ceph-mon[37501]: 1 slow requests, 1 included below; oldest blocked for > 361.749650 secs 2024-07-26T12:57:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:05 smithi112 ceph-mon[37501]: slow request 361.749649 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T12:57:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:05 smithi107 ceph-mon[32659]: pgmap v745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:05 smithi107 ceph-mon[32659]: 1 slow requests, 1 included below; oldest blocked for > 361.749650 secs 2024-07-26T12:57:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:05 smithi107 ceph-mon[32659]: slow request 361.749649 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T12:57:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:07.359+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:07.359+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:07.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:07.359+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:07 smithi112 ceph-mon[37501]: pgmap v746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:07 smithi107 ceph-mon[32659]: pgmap v746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:07 smithi132 ceph-mon[37580]: pgmap v746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:09.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:09.458+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:09.458+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:09.460 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:09.458+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:09 smithi112 ceph-mon[37501]: pgmap v747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:57:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:09 smithi107 ceph-mon[32659]: pgmap v747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:57:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:09 smithi132 ceph-mon[37580]: pgmap v747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:57:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 366.749773 secs 2024-07-26T12:57:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 366.749773 secs 2024-07-26T12:57:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 366.749773 secs 2024-07-26T12:57:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:11.698+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4005850 con 0x565421993390 2024-07-26T12:57:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:11.699+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4005850 con 0x565421993390 2024-07-26T12:57:11.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:11.759+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d8005820 con 0x7ff104102930 2024-07-26T12:57:11.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:11.759+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d8005820 con 0x7ff104102930 2024-07-26T12:57:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:11 smithi107 ceph-mon[32659]: pgmap v748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:11 smithi112 ceph-mon[37501]: pgmap v748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:11 smithi132 ceph-mon[37580]: pgmap v748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:13.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:13.797+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48005850 con 0x7efc78103340 2024-07-26T12:57:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:13.798+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48005850 con 0x7efc78103340 2024-07-26T12:57:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:13.799+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44005850 con 0x55aecd6b64d0 2024-07-26T12:57:13.801 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:13.800+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44005850 con 0x55aecd6b64d0 2024-07-26T12:57:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:13 smithi112 ceph-mon[37501]: pgmap v749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:13 smithi107 ceph-mon[32659]: pgmap v749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:13 smithi132 ceph-mon[37580]: pgmap v749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:15 smithi112 ceph-mon[37501]: pgmap v750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:57:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 371.749910 secs 2024-07-26T12:57:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:15 smithi107 ceph-mon[32659]: pgmap v750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:57:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 371.749910 secs 2024-07-26T12:57:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:15 smithi132 ceph-mon[37580]: pgmap v750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:57:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 371.749910 secs 2024-07-26T12:57:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:17 smithi112 ceph-mon[37501]: pgmap v751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:57:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:17 smithi107 ceph-mon[32659]: pgmap v751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:57:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:17 smithi132 ceph-mon[37580]: pgmap v751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:57:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:19 smithi112 ceph-mon[37501]: pgmap v752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:57:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:19 smithi107 ceph-mon[32659]: pgmap v752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:57:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:19 smithi132 ceph-mon[37580]: pgmap v752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:57:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 376.750021 secs 2024-07-26T12:57:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 376.750021 secs 2024-07-26T12:57:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 376.750021 secs 2024-07-26T12:57:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:21 smithi112 ceph-mon[37501]: pgmap v753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:21 smithi107 ceph-mon[32659]: pgmap v753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:21 smithi132 ceph-mon[37580]: pgmap v753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:22.370+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:22.370+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:22.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:22.370+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:23 smithi112 ceph-mon[37501]: pgmap v754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:23 smithi107 ceph-mon[32659]: pgmap v754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:23 smithi132 ceph-mon[37580]: pgmap v754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:24.470+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:24.470+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:24.472 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:24.470+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:25 smithi112 ceph-mon[37501]: pgmap v755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 381.750145 secs 2024-07-26T12:57:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:25 smithi107 ceph-mon[32659]: pgmap v755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 381.750145 secs 2024-07-26T12:57:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:25 smithi132 ceph-mon[37580]: pgmap v755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 381.750145 secs 2024-07-26T12:57:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:27 smithi112 ceph-mon[37501]: pgmap v756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:27 smithi107 ceph-mon[32659]: pgmap v756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:27 smithi132 ceph-mon[37580]: pgmap v756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:29 smithi112 ceph-mon[37501]: pgmap v757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:29 smithi107 ceph-mon[32659]: pgmap v757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:29 smithi132 ceph-mon[37580]: pgmap v757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 386.750213 secs 2024-07-26T12:57:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 386.750213 secs 2024-07-26T12:57:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 386.750213 secs 2024-07-26T12:57:32.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:31 smithi107 ceph-mon[32659]: pgmap v758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:31 smithi112 ceph-mon[37501]: pgmap v758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:31 smithi132 ceph-mon[37580]: pgmap v758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:33 smithi112 ceph-mon[37501]: pgmap v759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:33 smithi107 ceph-mon[32659]: pgmap v759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:33 smithi132 ceph-mon[37580]: pgmap v759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:35 smithi112 ceph-mon[37501]: pgmap v760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 391.750293 secs 2024-07-26T12:57:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:35 smithi107 ceph-mon[32659]: pgmap v760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 391.750293 secs 2024-07-26T12:57:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:35 smithi132 ceph-mon[37580]: pgmap v760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 391.750293 secs 2024-07-26T12:57:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:37.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:37.385+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:37.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:37.385+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:37.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:37.385+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:37 smithi107 ceph-mon[32659]: pgmap v761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:57:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:37 smithi132 ceph-mon[37580]: pgmap v761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:57:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:37 smithi112 ceph-mon[37501]: pgmap v761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:57:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:38 smithi107 ceph-mon[32659]: pgmap v762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:38 smithi107 ceph-mon[32659]: pgmap v763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:57:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:38 smithi132 ceph-mon[37580]: pgmap v762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:38 smithi132 ceph-mon[37580]: pgmap v763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:57:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:38 smithi112 ceph-mon[37501]: pgmap v762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:38 smithi112 ceph-mon[37501]: pgmap v763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:57:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:39.486+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:39.486+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:39.486+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:40 smithi107 ceph-mon[32659]: pgmap v764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 396.750371 secs 2024-07-26T12:57:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:40 smithi132 ceph-mon[37580]: pgmap v764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 396.750371 secs 2024-07-26T12:57:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:40 smithi112 ceph-mon[37501]: pgmap v764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 396.750371 secs 2024-07-26T12:57:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:42 smithi107 ceph-mon[32659]: pgmap v765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:57:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:42 smithi132 ceph-mon[37580]: pgmap v765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:57:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:42 smithi112 ceph-mon[37501]: pgmap v765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:57:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:44 smithi107 ceph-mon[32659]: pgmap v766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:44 smithi132 ceph-mon[37580]: pgmap v766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:44 smithi112 ceph-mon[37501]: pgmap v766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 401.750434 secs 2024-07-26T12:57:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 401.750434 secs 2024-07-26T12:57:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 401.750434 secs 2024-07-26T12:57:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:46 smithi107 ceph-mon[32659]: pgmap v767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:46 smithi132 ceph-mon[37580]: pgmap v767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:46 smithi112 ceph-mon[37501]: pgmap v767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:49 smithi107 ceph-mon[32659]: pgmap v768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:49 smithi132 ceph-mon[37580]: pgmap v768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:49 smithi112 ceph-mon[37501]: pgmap v768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:50 smithi107 ceph-mon[32659]: pgmap v769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:57:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:50 smithi132 ceph-mon[37580]: pgmap v769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:57:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:50 smithi112 ceph-mon[37501]: pgmap v769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:57:51.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 406.750520 secs 2024-07-26T12:57:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 406.750520 secs 2024-07-26T12:57:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 406.750520 secs 2024-07-26T12:57:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:52 smithi107 ceph-mon[32659]: pgmap v770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:52 smithi132 ceph-mon[37580]: pgmap v770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:52.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:52 smithi112 ceph-mon[37501]: pgmap v770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:57:52.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:52.393+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:52.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:52.393+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:52.394 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:57:52.393+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:54.502+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:54.502+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:57:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:57:54.502+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:54 smithi107 ceph-mon[32659]: pgmap v771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:54 smithi132 ceph-mon[37580]: pgmap v771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:54 smithi112 ceph-mon[37501]: pgmap v771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 411.750636 secs 2024-07-26T12:57:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 411.750636 secs 2024-07-26T12:57:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 411.750636 secs 2024-07-26T12:57:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:56 smithi107 ceph-mon[32659]: pgmap v772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:56 smithi132 ceph-mon[37580]: pgmap v772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:56 smithi112 ceph-mon[37501]: pgmap v772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:57:58 smithi107 ceph-mon[32659]: pgmap v773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:57:58 smithi132 ceph-mon[37580]: pgmap v773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:57:58 smithi112 ceph-mon[37501]: pgmap v773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:00 smithi107 ceph-mon[32659]: pgmap v774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 416.750758 secs 2024-07-26T12:58:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:00 smithi132 ceph-mon[37580]: pgmap v774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 416.750758 secs 2024-07-26T12:58:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:00 smithi112 ceph-mon[37501]: pgmap v774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 416.750758 secs 2024-07-26T12:58:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:02 smithi107 ceph-mon[32659]: pgmap v775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:02 smithi132 ceph-mon[37580]: pgmap v775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:02 smithi112 ceph-mon[37501]: pgmap v775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:04 smithi107 ceph-mon[32659]: pgmap v776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:04 smithi132 ceph-mon[37580]: pgmap v776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:04 smithi112 ceph-mon[37501]: pgmap v776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 421.750877 secs 2024-07-26T12:58:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 421.750877 secs 2024-07-26T12:58:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 421.750877 secs 2024-07-26T12:58:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:06 smithi132 ceph-mon[37580]: pgmap v777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:06 smithi112 ceph-mon[37501]: pgmap v777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:06 smithi107 ceph-mon[32659]: pgmap v777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:07.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:07.406+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:07.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:07.406+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:07.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:07.406+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:08 smithi132 ceph-mon[37580]: pgmap v778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:08 smithi112 ceph-mon[37501]: pgmap v778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:08 smithi107 ceph-mon[32659]: pgmap v778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:09.515+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:09.515+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:09.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:09.515+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:10 smithi132 ceph-mon[37580]: pgmap v779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 426.751021 secs 2024-07-26T12:58:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:10 smithi112 ceph-mon[37501]: pgmap v779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 426.751021 secs 2024-07-26T12:58:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:10 smithi107 ceph-mon[32659]: pgmap v779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 426.751021 secs 2024-07-26T12:58:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:12 smithi112 ceph-mon[37501]: pgmap v780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:12 smithi107 ceph-mon[32659]: pgmap v780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:12 smithi132 ceph-mon[37580]: pgmap v780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:14 smithi112 ceph-mon[37501]: pgmap v781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:14 smithi107 ceph-mon[32659]: pgmap v781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:14 smithi132 ceph-mon[37580]: pgmap v781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 431.751142 secs 2024-07-26T12:58:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 431.751142 secs 2024-07-26T12:58:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 431.751142 secs 2024-07-26T12:58:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:16 smithi112 ceph-mon[37501]: pgmap v782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:16 smithi107 ceph-mon[32659]: pgmap v782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:16 smithi132 ceph-mon[37580]: pgmap v782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:18 smithi112 ceph-mon[37501]: pgmap v783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:18 smithi107 ceph-mon[32659]: pgmap v783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:18 smithi132 ceph-mon[37580]: pgmap v783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:20 smithi112 ceph-mon[37501]: pgmap v784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 436.751261 secs 2024-07-26T12:58:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:20 smithi107 ceph-mon[32659]: pgmap v784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 436.751261 secs 2024-07-26T12:58:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:20 smithi132 ceph-mon[37580]: pgmap v784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 436.751261 secs 2024-07-26T12:58:22.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:22.420+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:22.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:22.420+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:22.423 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:22.421+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:22 smithi112 ceph-mon[37501]: pgmap v785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:22 smithi107 ceph-mon[32659]: pgmap v785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:22 smithi132 ceph-mon[37580]: pgmap v785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:24.527+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:24.527+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:24.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:24.527+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:24 smithi112 ceph-mon[37501]: pgmap v786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:24 smithi107 ceph-mon[32659]: pgmap v786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:24 smithi132 ceph-mon[37580]: pgmap v786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 441.751371 secs 2024-07-26T12:58:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 441.751371 secs 2024-07-26T12:58:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 441.751371 secs 2024-07-26T12:58:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:26 smithi112 ceph-mon[37501]: pgmap v787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:26 smithi107 ceph-mon[32659]: pgmap v787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:26 smithi132 ceph-mon[37580]: pgmap v787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:28 smithi112 ceph-mon[37501]: pgmap v788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:28 smithi107 ceph-mon[32659]: pgmap v788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:28 smithi132 ceph-mon[37580]: pgmap v788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:30 smithi112 ceph-mon[37501]: pgmap v789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 446.751495 secs 2024-07-26T12:58:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:30 smithi107 ceph-mon[32659]: pgmap v789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 446.751495 secs 2024-07-26T12:58:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:30 smithi132 ceph-mon[37580]: pgmap v789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T12:58:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 446.751495 secs 2024-07-26T12:58:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:32 smithi112 ceph-mon[37501]: pgmap v790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:32 smithi107 ceph-mon[32659]: pgmap v790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:32 smithi132 ceph-mon[37580]: pgmap v790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:34 smithi112 ceph-mon[37501]: pgmap v791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:34 smithi107 ceph-mon[32659]: pgmap v791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:34 smithi132 ceph-mon[37580]: pgmap v791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 451.751631 secs 2024-07-26T12:58:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 451.751631 secs 2024-07-26T12:58:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 451.751631 secs 2024-07-26T12:58:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:36 smithi112 ceph-mon[37501]: pgmap v792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:36.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:36 smithi132 ceph-mon[37580]: pgmap v792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:36 smithi107 ceph-mon[32659]: pgmap v792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:37.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:37.436+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:37.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:37.436+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:37.439 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:37.436+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:38 smithi112 ceph-mon[37501]: pgmap v793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:58:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:38 smithi107 ceph-mon[32659]: pgmap v793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:58:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:38 smithi132 ceph-mon[37580]: pgmap v793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:58:39.544 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:39.543+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:39.543+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:39.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:39.543+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:39 smithi112 ceph-mon[37501]: pgmap v794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:58:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:39 smithi107 ceph-mon[32659]: pgmap v794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:58:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:39 smithi132 ceph-mon[37580]: pgmap v794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:58:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 456.751755 secs 2024-07-26T12:58:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 456.751755 secs 2024-07-26T12:58:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 456.751755 secs 2024-07-26T12:58:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:41 smithi112 ceph-mon[37501]: pgmap v795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:58:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:41 smithi107 ceph-mon[32659]: pgmap v795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:58:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:41 smithi132 ceph-mon[37580]: pgmap v795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:58:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:43 smithi107 ceph-mon[32659]: pgmap v796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:58:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:43 smithi132 ceph-mon[37580]: pgmap v796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:58:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:43 smithi112 ceph-mon[37501]: pgmap v796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T12:58:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:45 smithi107 ceph-mon[32659]: pgmap v797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:58:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 461.751869 secs 2024-07-26T12:58:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:45 smithi132 ceph-mon[37580]: pgmap v797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:58:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 461.751869 secs 2024-07-26T12:58:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:45 smithi112 ceph-mon[37501]: pgmap v797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:58:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 461.751869 secs 2024-07-26T12:58:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:47 smithi107 ceph-mon[32659]: pgmap v798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:58:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:47 smithi132 ceph-mon[37580]: pgmap v798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:58:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:47 smithi112 ceph-mon[37501]: pgmap v798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:58:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:49 smithi107 ceph-mon[32659]: pgmap v799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:58:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:49 smithi132 ceph-mon[37580]: pgmap v799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:58:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:49 smithi112 ceph-mon[37501]: pgmap v799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:58:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 466.752008 secs 2024-07-26T12:58:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 466.752008 secs 2024-07-26T12:58:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 466.752008 secs 2024-07-26T12:58:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:51 smithi107 ceph-mon[32659]: pgmap v800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:58:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:51 smithi132 ceph-mon[37580]: pgmap v800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:58:52.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:51 smithi112 ceph-mon[37501]: pgmap v800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:58:52.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:52.453+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:52.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:52.453+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:52.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:58:52.453+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:53 smithi112 ceph-mon[37501]: pgmap v801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:58:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:53 smithi107 ceph-mon[32659]: pgmap v801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:58:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:53 smithi132 ceph-mon[37580]: pgmap v801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:58:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:54.559+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:54.559+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:58:54.559+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:58:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:55 smithi112 ceph-mon[37501]: pgmap v802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:58:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 471.752157 secs 2024-07-26T12:58:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:55 smithi107 ceph-mon[32659]: pgmap v802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:58:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 471.752157 secs 2024-07-26T12:58:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:55 smithi132 ceph-mon[37580]: pgmap v802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:58:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 471.752157 secs 2024-07-26T12:58:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:57 smithi112 ceph-mon[37501]: pgmap v803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:58:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:57 smithi107 ceph-mon[32659]: pgmap v803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:58:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:57 smithi132 ceph-mon[37580]: pgmap v803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:59:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:58:59 smithi112 ceph-mon[37501]: pgmap v804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:59:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:58:59 smithi107 ceph-mon[32659]: pgmap v804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:59:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:58:59 smithi132 ceph-mon[37580]: pgmap v804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:59:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 476.752291 secs 2024-07-26T12:59:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 476.752291 secs 2024-07-26T12:59:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 476.752291 secs 2024-07-26T12:59:02.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:01 smithi107 ceph-mon[32659]: pgmap v805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:01 smithi112 ceph-mon[37501]: pgmap v805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:01 smithi132 ceph-mon[37580]: pgmap v805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:03 smithi112 ceph-mon[37501]: pgmap v806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:03 smithi107 ceph-mon[32659]: pgmap v806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:03 smithi132 ceph-mon[37580]: pgmap v806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:05 smithi112 ceph-mon[37501]: pgmap v807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 481.752402 secs 2024-07-26T12:59:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:05 smithi107 ceph-mon[32659]: pgmap v807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 481.752402 secs 2024-07-26T12:59:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:05 smithi132 ceph-mon[37580]: pgmap v807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 481.752402 secs 2024-07-26T12:59:07.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:07.468+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:07.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:07.469+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:07.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:07.469+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:07 smithi112 ceph-mon[37501]: pgmap v808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:07 smithi107 ceph-mon[32659]: pgmap v808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:07 smithi132 ceph-mon[37580]: pgmap v808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:09.569+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:09.569+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:09.569+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:09 smithi112 ceph-mon[37501]: pgmap v809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:09 smithi107 ceph-mon[32659]: pgmap v809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:09 smithi132 ceph-mon[37580]: pgmap v809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 486.752523 secs 2024-07-26T12:59:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 486.752523 secs 2024-07-26T12:59:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 486.752523 secs 2024-07-26T12:59:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:11 smithi112 ceph-mon[37501]: pgmap v810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:11 smithi107 ceph-mon[32659]: pgmap v810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:11 smithi132 ceph-mon[37580]: pgmap v810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:13 smithi112 ceph-mon[37501]: pgmap v811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:13 smithi107 ceph-mon[32659]: pgmap v811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:13 smithi132 ceph-mon[37580]: pgmap v811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:15 smithi112 ceph-mon[37501]: pgmap v812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 491.752642 secs 2024-07-26T12:59:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:15 smithi107 ceph-mon[32659]: pgmap v812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 491.752642 secs 2024-07-26T12:59:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:15 smithi132 ceph-mon[37580]: pgmap v812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 491.752642 secs 2024-07-26T12:59:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:17 smithi112 ceph-mon[37501]: pgmap v813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:17 smithi107 ceph-mon[32659]: pgmap v813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:17 smithi132 ceph-mon[37580]: pgmap v813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:19 smithi112 ceph-mon[37501]: pgmap v814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:19 smithi107 ceph-mon[32659]: pgmap v814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:19 smithi132 ceph-mon[37580]: pgmap v814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 496.752770 secs 2024-07-26T12:59:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 496.752770 secs 2024-07-26T12:59:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 496.752770 secs 2024-07-26T12:59:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:21 smithi107 ceph-mon[32659]: pgmap v815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:21 smithi132 ceph-mon[37580]: pgmap v815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:21 smithi112 ceph-mon[37501]: pgmap v815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:22.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:22.482+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:22.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:22.482+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:22.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:22.482+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:23 smithi107 ceph-mon[32659]: pgmap v816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:23 smithi132 ceph-mon[37580]: pgmap v816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:23 smithi112 ceph-mon[37501]: pgmap v816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:24.577+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:24.577+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:24.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:24.577+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:25 smithi107 ceph-mon[32659]: pgmap v817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 501.752911 secs 2024-07-26T12:59:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:25 smithi132 ceph-mon[37580]: pgmap v817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 501.752911 secs 2024-07-26T12:59:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:25 smithi112 ceph-mon[37501]: pgmap v817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 501.752911 secs 2024-07-26T12:59:26.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:26.720+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4005b10 con 0x565421993390 2024-07-26T12:59:26.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:26.720+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4005b10 con 0x565421993390 2024-07-26T12:59:26.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:26.779+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d8005b00 con 0x7ff104102930 2024-07-26T12:59:26.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:26.780+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d8005b00 con 0x7ff104102930 2024-07-26T12:59:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:27 smithi107 ceph-mon[32659]: pgmap v818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:27 smithi132 ceph-mon[37580]: pgmap v818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:27 smithi112 ceph-mon[37501]: pgmap v818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:28.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:28.822+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48005b70 con 0x7efc78103340 2024-07-26T12:59:28.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:28.822+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44005b50 con 0x55aecd6b64d0 2024-07-26T12:59:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:28.822+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44005b50 con 0x55aecd6b64d0 2024-07-26T12:59:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:28.822+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc48005b70 con 0x7efc78103340 2024-07-26T12:59:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:29 smithi107 ceph-mon[32659]: pgmap v819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:29 smithi132 ceph-mon[37580]: pgmap v819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:29 smithi112 ceph-mon[37501]: pgmap v819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 506.753002 secs 2024-07-26T12:59:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 506.753002 secs 2024-07-26T12:59:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 506.753002 secs 2024-07-26T12:59:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:31 smithi107 ceph-mon[32659]: pgmap v820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:31 smithi132 ceph-mon[37580]: pgmap v820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:31 smithi112 ceph-mon[37501]: pgmap v820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:33 smithi107 ceph-mon[32659]: pgmap v821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:33 smithi132 ceph-mon[37580]: pgmap v821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:33 smithi112 ceph-mon[37501]: pgmap v821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:35 smithi107 ceph-mon[32659]: pgmap v822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 511.753121 secs 2024-07-26T12:59:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:35 smithi132 ceph-mon[37580]: pgmap v822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 511.753121 secs 2024-07-26T12:59:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:35 smithi112 ceph-mon[37501]: pgmap v822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:59:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 511.753121 secs 2024-07-26T12:59:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:37.494+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:37.494+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:37.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:37.494+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:38 smithi107 ceph-mon[32659]: pgmap v823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:38 smithi132 ceph-mon[37580]: pgmap v823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:38 smithi112 ceph-mon[37501]: pgmap v823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:39.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:39.589+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:39.589+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:39.589+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:40.791 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:40 smithi112 ceph-mon[37501]: pgmap v824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:40.791 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:40.791 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:40.791 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:59:40.792 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:40 smithi132 ceph-mon[37580]: pgmap v824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:40.792 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:40.792 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:40.792 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:59:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:40 smithi107 ceph-mon[32659]: pgmap v824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T12:59:41.031 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:41 smithi112 ceph-mon[37501]: pgmap v825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:59:41.031 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:41 smithi112 ceph-mon[37501]: pgmap v826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:41.031 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 516.753215 secs 2024-07-26T12:59:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:41 smithi107 ceph-mon[32659]: pgmap v825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:59:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:41 smithi107 ceph-mon[32659]: pgmap v826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 516.753215 secs 2024-07-26T12:59:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:41 smithi132 ceph-mon[37580]: pgmap v825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:59:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:41 smithi132 ceph-mon[37580]: pgmap v826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 516.753215 secs 2024-07-26T12:59:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:42 smithi132 ceph-mon[37580]: pgmap v827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:42 smithi112 ceph-mon[37501]: pgmap v827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:42 smithi107 ceph-mon[32659]: pgmap v827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:44 smithi112 ceph-mon[37501]: pgmap v828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:59:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:44 smithi107 ceph-mon[32659]: pgmap v828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:59:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:44 smithi132 ceph-mon[37580]: pgmap v828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:59:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 521.753331 secs 2024-07-26T12:59:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 521.753331 secs 2024-07-26T12:59:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 521.753331 secs 2024-07-26T12:59:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:46 smithi112 ceph-mon[37501]: pgmap v829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:46 smithi107 ceph-mon[32659]: pgmap v829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:46 smithi132 ceph-mon[37580]: pgmap v829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:48 smithi112 ceph-mon[37501]: pgmap v830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:59:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:48 smithi107 ceph-mon[32659]: pgmap v830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:59:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:48 smithi132 ceph-mon[37580]: pgmap v830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:59:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:50 smithi112 ceph-mon[37501]: pgmap v831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:59:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 526.753459 secs 2024-07-26T12:59:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:50 smithi107 ceph-mon[32659]: pgmap v831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:59:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 526.753459 secs 2024-07-26T12:59:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:50 smithi132 ceph-mon[37580]: pgmap v831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:59:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 526.753459 secs 2024-07-26T12:59:52.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:52.498+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:52.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:52.498+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:52.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T12:59:52.498+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:52 smithi112 ceph-mon[37501]: pgmap v832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:52 smithi107 ceph-mon[32659]: pgmap v832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:52 smithi132 ceph-mon[37580]: pgmap v832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:59:54.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:54.606+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:54.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:54.606+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T12:59:54.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T12:59:54.606+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:54 smithi112 ceph-mon[37501]: pgmap v833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:55.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:54 smithi107 ceph-mon[32659]: pgmap v833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:54 smithi132 ceph-mon[37580]: pgmap v833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 531.753583 secs 2024-07-26T12:59:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 531.753583 secs 2024-07-26T12:59:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 531.753583 secs 2024-07-26T12:59:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:56 smithi107 ceph-mon[32659]: pgmap v834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:56 smithi132 ceph-mon[37580]: pgmap v834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:56 smithi112 ceph-mon[37501]: pgmap v834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 12:59:58 smithi107 ceph-mon[32659]: pgmap v835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 12:59:58 smithi132 ceph-mon[37580]: pgmap v835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:59:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 12:59:58 smithi112 ceph-mon[37501]: pgmap v835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:00 smithi107 ceph-mon[32659]: pgmap v836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:00 smithi107 ceph-mon[32659]: Health detail: HEALTH_WARN 3 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:00 smithi107 ceph-mon[32659]: [WRN] CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s) 2024-07-26T13:00:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:00 smithi107 ceph-mon[32659]: daemon mds.a on smithi107 is in unknown state 2024-07-26T13:00:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:00 smithi107 ceph-mon[32659]: daemon mds.h on smithi112 is in error state 2024-07-26T13:00:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:00 smithi107 ceph-mon[32659]: daemon mds.b on smithi112 is in error state 2024-07-26T13:00:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:00 smithi107 ceph-mon[32659]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T13:00:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:00 smithi107 ceph-mon[32659]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2024-07-26T13:00:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 536.753728 secs 2024-07-26T13:00:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:00 smithi132 ceph-mon[37580]: pgmap v836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:00 smithi132 ceph-mon[37580]: Health detail: HEALTH_WARN 3 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:00 smithi132 ceph-mon[37580]: [WRN] CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s) 2024-07-26T13:00:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:00 smithi132 ceph-mon[37580]: daemon mds.a on smithi107 is in unknown state 2024-07-26T13:00:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:00 smithi132 ceph-mon[37580]: daemon mds.h on smithi112 is in error state 2024-07-26T13:00:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:00 smithi132 ceph-mon[37580]: daemon mds.b on smithi112 is in error state 2024-07-26T13:00:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:00 smithi132 ceph-mon[37580]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T13:00:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:00 smithi132 ceph-mon[37580]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2024-07-26T13:00:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 536.753728 secs 2024-07-26T13:00:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:00 smithi112 ceph-mon[37501]: pgmap v836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:00 smithi112 ceph-mon[37501]: Health detail: HEALTH_WARN 3 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:00:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:00 smithi112 ceph-mon[37501]: [WRN] CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s) 2024-07-26T13:00:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:00 smithi112 ceph-mon[37501]: daemon mds.a on smithi107 is in unknown state 2024-07-26T13:00:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:00 smithi112 ceph-mon[37501]: daemon mds.h on smithi112 is in error state 2024-07-26T13:00:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:00 smithi112 ceph-mon[37501]: daemon mds.b on smithi112 is in error state 2024-07-26T13:00:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:00 smithi112 ceph-mon[37501]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T13:00:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:00 smithi112 ceph-mon[37501]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2024-07-26T13:00:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 536.753728 secs 2024-07-26T13:00:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:02 smithi107 ceph-mon[32659]: pgmap v837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:02 smithi132 ceph-mon[37580]: pgmap v837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:02 smithi112 ceph-mon[37501]: pgmap v837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:04 smithi107 ceph-mon[32659]: pgmap v838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:04 smithi132 ceph-mon[37580]: pgmap v838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:04 smithi112 ceph-mon[37501]: pgmap v838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 541.753879 secs 2024-07-26T13:00:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 541.753879 secs 2024-07-26T13:00:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 541.753879 secs 2024-07-26T13:00:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:06 smithi107 ceph-mon[32659]: pgmap v839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:06 smithi132 ceph-mon[37580]: pgmap v839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:06 smithi112 ceph-mon[37501]: pgmap v839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:07.514+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:07.514+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:07.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:07.514+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:08 smithi107 ceph-mon[32659]: pgmap v840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:08 smithi132 ceph-mon[37580]: pgmap v840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:08 smithi112 ceph-mon[37501]: pgmap v840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:09.618+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:09.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:09.618+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:09.620 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:09.618+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:11 smithi107 ceph-mon[32659]: pgmap v841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 546.754027 secs 2024-07-26T13:00:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:11 smithi132 ceph-mon[37580]: pgmap v841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 546.754027 secs 2024-07-26T13:00:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:11 smithi112 ceph-mon[37501]: pgmap v841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 546.754027 secs 2024-07-26T13:00:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:12.729+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4005e10 con 0x565421993390 2024-07-26T13:00:12.730 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:12.729+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4005e10 con 0x565421993390 2024-07-26T13:00:12.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:12.788+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d8005d20 con 0x7ff104102930 2024-07-26T13:00:12.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:12.788+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d8005d20 con 0x7ff104102930 2024-07-26T13:00:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:13 smithi107 ceph-mon[32659]: pgmap v842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:13 smithi132 ceph-mon[37580]: pgmap v842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:13 smithi112 ceph-mon[37501]: pgmap v842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:13.831+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44005e30 con 0x55aecd6b64d0 2024-07-26T13:00:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:13.831+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48005ef0 con 0x7efc78103340 2024-07-26T13:00:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:13.832+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44005e30 con 0x55aecd6b64d0 2024-07-26T13:00:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:13.832+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48005ef0 con 0x7efc78103340 2024-07-26T13:00:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:15 smithi107 ceph-mon[32659]: pgmap v843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:15 smithi132 ceph-mon[37580]: pgmap v843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:15 smithi112 ceph-mon[37501]: pgmap v843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 551.754143 secs 2024-07-26T13:00:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 551.754143 secs 2024-07-26T13:00:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 551.754143 secs 2024-07-26T13:00:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:17 smithi107 ceph-mon[32659]: pgmap v844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:17 smithi132 ceph-mon[37580]: pgmap v844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:17 smithi112 ceph-mon[37501]: pgmap v844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:19 smithi107 ceph-mon[32659]: pgmap v845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:19 smithi132 ceph-mon[37580]: pgmap v845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:19 smithi112 ceph-mon[37501]: pgmap v845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:20 smithi112 ceph-mon[37501]: pgmap v846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:20 smithi107 ceph-mon[32659]: pgmap v846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:20 smithi132 ceph-mon[37580]: pgmap v846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 556.754212 secs 2024-07-26T13:00:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 556.754212 secs 2024-07-26T13:00:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 556.754212 secs 2024-07-26T13:00:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:22 smithi112 ceph-mon[37501]: pgmap v847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:22.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:22.528+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:22.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:22.528+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:22.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:22.528+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:22.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:22 smithi107 ceph-mon[32659]: pgmap v847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:22 smithi132 ceph-mon[37580]: pgmap v847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:24.633+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:24.633+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:24.635 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:24.633+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:24 smithi112 ceph-mon[37501]: pgmap v848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:24 smithi107 ceph-mon[32659]: pgmap v848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:24 smithi132 ceph-mon[37580]: pgmap v848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 561.754327 secs 2024-07-26T13:00:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 561.754327 secs 2024-07-26T13:00:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 561.754327 secs 2024-07-26T13:00:27.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:26 smithi107 ceph-mon[32659]: pgmap v849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:26 smithi112 ceph-mon[37501]: pgmap v849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:26 smithi132 ceph-mon[37580]: pgmap v849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:28 smithi112 ceph-mon[37501]: pgmap v850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:28 smithi107 ceph-mon[32659]: pgmap v850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:28 smithi132 ceph-mon[37580]: pgmap v850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:30 smithi107 ceph-mon[32659]: pgmap v851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 566.754467 secs 2024-07-26T13:00:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:30 smithi132 ceph-mon[37580]: pgmap v851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 566.754467 secs 2024-07-26T13:00:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:30 smithi112 ceph-mon[37501]: pgmap v851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 566.754467 secs 2024-07-26T13:00:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:32 smithi107 ceph-mon[32659]: pgmap v852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:32 smithi132 ceph-mon[37580]: pgmap v852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:32 smithi112 ceph-mon[37501]: pgmap v852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:34 smithi107 ceph-mon[32659]: pgmap v853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:34 smithi132 ceph-mon[37580]: pgmap v853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:34 smithi112 ceph-mon[37501]: pgmap v853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 571.754598 secs 2024-07-26T13:00:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 571.754598 secs 2024-07-26T13:00:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 571.754598 secs 2024-07-26T13:00:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:36 smithi107 ceph-mon[32659]: pgmap v854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:36 smithi132 ceph-mon[37580]: pgmap v854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:36 smithi112 ceph-mon[37501]: pgmap v854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:37.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:37.539+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:37.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:37.539+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:37.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:37.539+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:39.267 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:38 smithi107 ceph-mon[32659]: pgmap v855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:38 smithi132 ceph-mon[37580]: pgmap v855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:38 smithi112 ceph-mon[37501]: pgmap v855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:39.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:39.648+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:39.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:39.648+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:39.651 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:39.649+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:40 smithi107 ceph-mon[32659]: pgmap v856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 576.754669 secs 2024-07-26T13:00:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:40 smithi132 ceph-mon[37580]: pgmap v856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 576.754669 secs 2024-07-26T13:00:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:40 smithi112 ceph-mon[37501]: pgmap v856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 576.754669 secs 2024-07-26T13:00:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:41 smithi107 ceph-mon[32659]: pgmap v857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:00:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:00:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:41 smithi132 ceph-mon[37580]: pgmap v857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:00:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:00:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:41 smithi112 ceph-mon[37501]: pgmap v857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:00:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:00:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:43 smithi112 ceph-mon[37501]: pgmap v858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:00:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:43 smithi107 ceph-mon[32659]: pgmap v858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:00:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:43 smithi132 ceph-mon[37580]: pgmap v858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:00:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:45 smithi112 ceph-mon[37501]: pgmap v859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:00:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:45 smithi107 ceph-mon[32659]: pgmap v859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:00:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:45 smithi132 ceph-mon[37580]: pgmap v859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:00:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 581.754754 secs 2024-07-26T13:00:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 581.754754 secs 2024-07-26T13:00:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 581.754754 secs 2024-07-26T13:00:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:47 smithi112 ceph-mon[37501]: pgmap v860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:47 smithi107 ceph-mon[32659]: pgmap v860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:47 smithi132 ceph-mon[37580]: pgmap v860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:00:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:49 smithi112 ceph-mon[37501]: pgmap v861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:49 smithi107 ceph-mon[32659]: pgmap v861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:49 smithi132 ceph-mon[37580]: pgmap v861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:50.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 586.754885 secs 2024-07-26T13:00:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 586.754885 secs 2024-07-26T13:00:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 586.754885 secs 2024-07-26T13:00:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:51 smithi132 ceph-mon[37580]: pgmap v862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:00:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:51 smithi112 ceph-mon[37501]: pgmap v862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:00:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:51 smithi107 ceph-mon[32659]: pgmap v862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:00:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:52.555+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:52.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:52.555+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:52.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:00:52.555+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:53 smithi112 ceph-mon[37501]: pgmap v863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:53 smithi107 ceph-mon[32659]: pgmap v863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:53 smithi132 ceph-mon[37580]: pgmap v863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:54.661+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:54.661+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:00:54.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:00:54.661+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:55 smithi112 ceph-mon[37501]: pgmap v864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:55 smithi107 ceph-mon[32659]: pgmap v864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:55 smithi132 ceph-mon[37580]: pgmap v864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 591.754984 secs 2024-07-26T13:00:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 591.754984 secs 2024-07-26T13:00:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 591.754984 secs 2024-07-26T13:00:57.443 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:57 smithi107 ceph-mon[32659]: pgmap v865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:57 smithi112 ceph-mon[37501]: pgmap v865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:57 smithi132 ceph-mon[37580]: pgmap v865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:00:59 smithi112 ceph-mon[37501]: pgmap v866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:00:59 smithi107 ceph-mon[32659]: pgmap v866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:00:59 smithi132 ceph-mon[37580]: pgmap v866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 596.755098 secs 2024-07-26T13:01:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 596.755098 secs 2024-07-26T13:01:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 596.755098 secs 2024-07-26T13:01:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:01 smithi107 ceph-mon[32659]: pgmap v867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:01.572 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:01 smithi132 ceph-mon[37580]: pgmap v867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:01.589 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:01 smithi112 ceph-mon[37501]: pgmap v867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:03.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:03 smithi107 ceph-mon[32659]: pgmap v868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:03 smithi112 ceph-mon[37501]: pgmap v868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:03 smithi132 ceph-mon[37580]: pgmap v868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:05 smithi112 ceph-mon[37501]: pgmap v869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:05 smithi107 ceph-mon[32659]: pgmap v869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:05 smithi132 ceph-mon[37580]: pgmap v869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 601.755187 secs 2024-07-26T13:01:06.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 601.755187 secs 2024-07-26T13:01:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 601.755187 secs 2024-07-26T13:01:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:07 smithi112 ceph-mon[37501]: pgmap v870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:07 smithi107 ceph-mon[32659]: pgmap v870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:07 smithi132 ceph-mon[37580]: pgmap v870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:07.571+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:07.571+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:07.571+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:09 smithi112 ceph-mon[37501]: pgmap v871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:09 smithi107 ceph-mon[32659]: pgmap v871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:09 smithi132 ceph-mon[37580]: pgmap v871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:09.677+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:09.677+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:09.677+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 606.755334 secs 2024-07-26T13:01:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 606.755334 secs 2024-07-26T13:01:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 606.755334 secs 2024-07-26T13:01:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:11 smithi107 ceph-mon[32659]: pgmap v872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:11 smithi132 ceph-mon[37580]: pgmap v872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:11 smithi112 ceph-mon[37501]: pgmap v872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:13 smithi112 ceph-mon[37501]: pgmap v873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:13 smithi107 ceph-mon[32659]: pgmap v873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:13 smithi132 ceph-mon[37580]: pgmap v873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:15 smithi112 ceph-mon[37501]: pgmap v874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:15 smithi107 ceph-mon[32659]: pgmap v874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:15 smithi132 ceph-mon[37580]: pgmap v874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 611.755399 secs 2024-07-26T13:01:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 611.755399 secs 2024-07-26T13:01:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 611.755399 secs 2024-07-26T13:01:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:17 smithi112 ceph-mon[37501]: pgmap v875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:17 smithi107 ceph-mon[32659]: pgmap v875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:17 smithi132 ceph-mon[37580]: pgmap v875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:19 smithi112 ceph-mon[37501]: pgmap v876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:19 smithi107 ceph-mon[32659]: pgmap v876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:19 smithi132 ceph-mon[37580]: pgmap v876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 616.755534 secs 2024-07-26T13:01:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 616.755534 secs 2024-07-26T13:01:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 616.755534 secs 2024-07-26T13:01:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:21 smithi132 ceph-mon[37580]: pgmap v877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:21 smithi112 ceph-mon[37501]: pgmap v877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:21 smithi107 ceph-mon[32659]: pgmap v877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:22.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:22.576+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:22.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:22.576+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:22.578 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:22.576+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:23 smithi112 ceph-mon[37501]: pgmap v878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:23 smithi107 ceph-mon[32659]: pgmap v878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:23 smithi132 ceph-mon[37580]: pgmap v878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:24.693 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:24.692+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:24.692+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:24.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:24.692+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:25 smithi112 ceph-mon[37501]: pgmap v879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:25 smithi107 ceph-mon[32659]: pgmap v879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:25 smithi132 ceph-mon[37580]: pgmap v879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 621.755671 secs 2024-07-26T13:01:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 621.755671 secs 2024-07-26T13:01:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 621.755671 secs 2024-07-26T13:01:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:27 smithi112 ceph-mon[37501]: pgmap v880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:27 smithi107 ceph-mon[32659]: pgmap v880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:27 smithi132 ceph-mon[37580]: pgmap v880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:29 smithi107 ceph-mon[32659]: pgmap v881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:29 smithi112 ceph-mon[37501]: pgmap v881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:29 smithi132 ceph-mon[37580]: pgmap v881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 626.755793 secs 2024-07-26T13:01:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 626.755793 secs 2024-07-26T13:01:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 626.755793 secs 2024-07-26T13:01:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:31 smithi107 ceph-mon[32659]: pgmap v882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:31 smithi132 ceph-mon[37580]: pgmap v882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:31 smithi112 ceph-mon[37501]: pgmap v882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:33 smithi112 ceph-mon[37501]: pgmap v883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:33 smithi107 ceph-mon[32659]: pgmap v883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:33 smithi132 ceph-mon[37580]: pgmap v883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:36 smithi112 ceph-mon[37501]: pgmap v884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 631.755911 secs 2024-07-26T13:01:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:36 smithi107 ceph-mon[32659]: pgmap v884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 631.755911 secs 2024-07-26T13:01:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:36 smithi132 ceph-mon[37580]: pgmap v884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 631.755911 secs 2024-07-26T13:01:37.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:37.585+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:37.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:37.585+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:37.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:37.585+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:38 smithi112 ceph-mon[37501]: pgmap v885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:38 smithi107 ceph-mon[32659]: pgmap v885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:38 smithi132 ceph-mon[37580]: pgmap v885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:39.707+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:39.707+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:39.709 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:39.707+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:40 smithi112 ceph-mon[37501]: pgmap v886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:40 smithi107 ceph-mon[32659]: pgmap v886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:40 smithi132 ceph-mon[37580]: pgmap v886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 636.756051 secs 2024-07-26T13:01:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 636.756051 secs 2024-07-26T13:01:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 636.756051 secs 2024-07-26T13:01:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:42 smithi112 ceph-mon[37501]: pgmap v887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:42 smithi107 ceph-mon[32659]: pgmap v887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:42 smithi132 ceph-mon[37580]: pgmap v887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:01:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:01:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: pgmap v888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:01:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: pgmap v889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:01:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: pgmap v888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:01:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: pgmap v889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:01:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:01:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:01:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:01:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: pgmap v888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:01:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: pgmap v889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:01:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:45 smithi132 ceph-mon[37580]: pgmap v890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 641.756161 secs 2024-07-26T13:01:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:45 smithi112 ceph-mon[37501]: pgmap v890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 641.756161 secs 2024-07-26T13:01:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:45 smithi107 ceph-mon[32659]: pgmap v890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:01:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 641.756161 secs 2024-07-26T13:01:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:47 smithi132 ceph-mon[37580]: pgmap v891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:01:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:47 smithi112 ceph-mon[37501]: pgmap v891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:01:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:47 smithi107 ceph-mon[32659]: pgmap v891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:01:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:49 smithi112 ceph-mon[37501]: pgmap v892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:01:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:49 smithi107 ceph-mon[32659]: pgmap v892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:01:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:49 smithi132 ceph-mon[37580]: pgmap v892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:01:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 646.756278 secs 2024-07-26T13:01:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 646.756278 secs 2024-07-26T13:01:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 646.756278 secs 2024-07-26T13:01:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:51 smithi112 ceph-mon[37501]: pgmap v893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:51 smithi107 ceph-mon[32659]: pgmap v893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:51 smithi132 ceph-mon[37580]: pgmap v893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:01:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:52.594+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:52.594+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:52.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:01:52.594+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:53 smithi112 ceph-mon[37501]: pgmap v894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:01:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:53 smithi107 ceph-mon[32659]: pgmap v894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:01:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:53 smithi132 ceph-mon[37580]: pgmap v894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:01:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:54.724+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:54.724+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:01:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:01:54.724+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:55 smithi112 ceph-mon[37501]: pgmap v895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:01:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 651.756395 secs 2024-07-26T13:01:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:55 smithi107 ceph-mon[32659]: pgmap v895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:01:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 651.756395 secs 2024-07-26T13:01:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:55 smithi132 ceph-mon[37580]: pgmap v895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:01:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 651.756395 secs 2024-07-26T13:01:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:57 smithi112 ceph-mon[37501]: pgmap v896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:01:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:57 smithi107 ceph-mon[32659]: pgmap v896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:01:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:57 smithi132 ceph-mon[37580]: pgmap v896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:01:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:01:59 smithi112 ceph-mon[37501]: pgmap v897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:01:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:01:59 smithi107 ceph-mon[32659]: pgmap v897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:01:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:01:59 smithi132 ceph-mon[37580]: pgmap v897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:02:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 656.756493 secs 2024-07-26T13:02:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 656.756493 secs 2024-07-26T13:02:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 656.756493 secs 2024-07-26T13:02:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:01 smithi112 ceph-mon[37501]: pgmap v898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:01 smithi107 ceph-mon[32659]: pgmap v898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:01 smithi132 ceph-mon[37580]: pgmap v898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:03 smithi112 ceph-mon[37501]: pgmap v899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:03 smithi107 ceph-mon[32659]: pgmap v899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:03 smithi132 ceph-mon[37580]: pgmap v899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:05 smithi112 ceph-mon[37501]: pgmap v900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 661.756623 secs 2024-07-26T13:02:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:05 smithi107 ceph-mon[32659]: pgmap v900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 661.756623 secs 2024-07-26T13:02:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:05 smithi132 ceph-mon[37580]: pgmap v900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 661.756623 secs 2024-07-26T13:02:07.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:07.602+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:07.602+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:07.602+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:07 smithi112 ceph-mon[37501]: pgmap v901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:07 smithi107 ceph-mon[32659]: pgmap v901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:07 smithi132 ceph-mon[37580]: pgmap v901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:09 smithi112 ceph-mon[37501]: pgmap v902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:09.732+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:09.732+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:09.732+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:09 smithi107 ceph-mon[32659]: pgmap v902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:09 smithi132 ceph-mon[37580]: pgmap v902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 666.756719 secs 2024-07-26T13:02:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 666.756719 secs 2024-07-26T13:02:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 666.756719 secs 2024-07-26T13:02:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:11 smithi112 ceph-mon[37501]: pgmap v903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:11 smithi107 ceph-mon[32659]: pgmap v903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:11 smithi132 ceph-mon[37580]: pgmap v903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:13 smithi112 ceph-mon[37501]: pgmap v904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:13 smithi107 ceph-mon[32659]: pgmap v904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:13 smithi132 ceph-mon[37580]: pgmap v904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:15 smithi112 ceph-mon[37501]: pgmap v905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 671.756858 secs 2024-07-26T13:02:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:15 smithi107 ceph-mon[32659]: pgmap v905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 671.756858 secs 2024-07-26T13:02:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:15 smithi132 ceph-mon[37580]: pgmap v905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 671.756858 secs 2024-07-26T13:02:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:17 smithi112 ceph-mon[37501]: pgmap v906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:17 smithi107 ceph-mon[32659]: pgmap v906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:17 smithi132 ceph-mon[37580]: pgmap v906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:19 smithi112 ceph-mon[37501]: pgmap v907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:19 smithi107 ceph-mon[32659]: pgmap v907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:19 smithi132 ceph-mon[37580]: pgmap v907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 676.756994 secs 2024-07-26T13:02:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 676.756994 secs 2024-07-26T13:02:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 676.756994 secs 2024-07-26T13:02:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:21 smithi112 ceph-mon[37501]: pgmap v908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:21 smithi107 ceph-mon[32659]: pgmap v908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:21 smithi132 ceph-mon[37580]: pgmap v908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:22.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:22.602+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:22.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:22.602+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:22.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:22.602+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:23 smithi112 ceph-mon[37501]: pgmap v909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:23 smithi107 ceph-mon[32659]: pgmap v909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:23 smithi132 ceph-mon[37580]: pgmap v909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:24.746+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:24.746+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:24.746+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:25 smithi112 ceph-mon[37501]: pgmap v910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 681.757116 secs 2024-07-26T13:02:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:25 smithi107 ceph-mon[32659]: pgmap v910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 681.757116 secs 2024-07-26T13:02:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:25 smithi132 ceph-mon[37580]: pgmap v910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 681.757116 secs 2024-07-26T13:02:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:27 smithi107 ceph-mon[32659]: pgmap v911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:27 smithi132 ceph-mon[37580]: pgmap v911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:27 smithi112 ceph-mon[37501]: pgmap v911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:29 smithi107 ceph-mon[32659]: pgmap v912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:29 smithi132 ceph-mon[37580]: pgmap v912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:29 smithi112 ceph-mon[37501]: pgmap v912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 686.757270 secs 2024-07-26T13:02:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 686.757270 secs 2024-07-26T13:02:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 686.757270 secs 2024-07-26T13:02:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:31 smithi107 ceph-mon[32659]: pgmap v913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:31 smithi132 ceph-mon[37580]: pgmap v913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:31 smithi112 ceph-mon[37501]: pgmap v913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:33 smithi107 ceph-mon[32659]: pgmap v914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:33 smithi132 ceph-mon[37580]: pgmap v914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:33 smithi112 ceph-mon[37501]: pgmap v914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:35 smithi107 ceph-mon[32659]: pgmap v915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 691.757388 secs 2024-07-26T13:02:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:35 smithi132 ceph-mon[37580]: pgmap v915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 691.757388 secs 2024-07-26T13:02:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:35 smithi112 ceph-mon[37501]: pgmap v915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 691.757388 secs 2024-07-26T13:02:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:37.612+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:37.612+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:37.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:37.612+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:37 smithi107 ceph-mon[32659]: pgmap v916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:37 smithi132 ceph-mon[37580]: pgmap v916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:37 smithi112 ceph-mon[37501]: pgmap v916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:39.762+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:39.762+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:39.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:39.762+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:39 smithi107 ceph-mon[32659]: pgmap v917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:39 smithi132 ceph-mon[37580]: pgmap v917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:39 smithi112 ceph-mon[37501]: pgmap v917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 696.757511 secs 2024-07-26T13:02:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 696.757511 secs 2024-07-26T13:02:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 696.757511 secs 2024-07-26T13:02:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:41 smithi107 ceph-mon[32659]: pgmap v918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:41 smithi132 ceph-mon[37580]: pgmap v918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:41 smithi112 ceph-mon[37501]: pgmap v918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: pgmap v919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:02:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:02:43.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:43.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:43.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: pgmap v919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:02:43.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:02:43.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:43.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:43.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: pgmap v919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:43.981 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.981 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.982 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.983 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.983 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:43.983 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:02:43.983 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:02:43.983 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:02:43.983 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:02:43.983 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:43.983 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:43.983 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:02:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:44 smithi107 ceph-mon[32659]: pgmap v920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:02:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:44 smithi107 ceph-mon[32659]: pgmap v921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:02:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:44 smithi132 ceph-mon[37580]: pgmap v920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:02:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:44 smithi132 ceph-mon[37580]: pgmap v921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:02:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:44 smithi112 ceph-mon[37501]: pgmap v920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:02:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:44 smithi112 ceph-mon[37501]: pgmap v921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:02:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 701.757649 secs 2024-07-26T13:02:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 701.757649 secs 2024-07-26T13:02:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 701.757649 secs 2024-07-26T13:02:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:46 smithi132 ceph-mon[37580]: pgmap v922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:02:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:46 smithi112 ceph-mon[37501]: pgmap v922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:02:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:46 smithi107 ceph-mon[32659]: pgmap v922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:02:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:48 smithi132 ceph-mon[37580]: pgmap v923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:48 smithi112 ceph-mon[37501]: pgmap v923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:48 smithi107 ceph-mon[32659]: pgmap v923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:50 smithi132 ceph-mon[37580]: pgmap v924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:02:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 706.757786 secs 2024-07-26T13:02:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:50 smithi112 ceph-mon[37501]: pgmap v924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:02:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 706.757786 secs 2024-07-26T13:02:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:50 smithi107 ceph-mon[32659]: pgmap v924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:02:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 706.757786 secs 2024-07-26T13:02:52.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:52.628+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:52.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:52.628+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:52.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:02:52.628+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:52 smithi112 ceph-mon[37501]: pgmap v925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:02:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:52 smithi107 ceph-mon[32659]: pgmap v925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:02:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:52 smithi132 ceph-mon[37580]: pgmap v925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:02:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:54.770+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:54.770+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:02:54.771 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:02:54.770+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:54 smithi112 ceph-mon[37501]: pgmap v926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:02:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:54 smithi107 ceph-mon[32659]: pgmap v926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:02:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:54 smithi132 ceph-mon[37580]: pgmap v926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:02:55.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 711.757902 secs 2024-07-26T13:02:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 711.757902 secs 2024-07-26T13:02:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 711.757902 secs 2024-07-26T13:02:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:56 smithi112 ceph-mon[37501]: pgmap v927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:56 smithi107 ceph-mon[32659]: pgmap v927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:56 smithi132 ceph-mon[37580]: pgmap v927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:02:58 smithi112 ceph-mon[37501]: pgmap v928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:02:58 smithi107 ceph-mon[32659]: pgmap v928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:02:58 smithi132 ceph-mon[37580]: pgmap v928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:00 smithi112 ceph-mon[37501]: pgmap v929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 716.758023 secs 2024-07-26T13:03:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:00 smithi107 ceph-mon[32659]: pgmap v929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 716.758023 secs 2024-07-26T13:03:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:00 smithi132 ceph-mon[37580]: pgmap v929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 716.758023 secs 2024-07-26T13:03:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:02 smithi112 ceph-mon[37501]: pgmap v930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:02 smithi107 ceph-mon[32659]: pgmap v930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:02 smithi132 ceph-mon[37580]: pgmap v930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:04 smithi112 ceph-mon[37501]: pgmap v931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:04 smithi107 ceph-mon[32659]: pgmap v931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:04 smithi132 ceph-mon[37580]: pgmap v931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:05 smithi112 ceph-mon[37501]: 1 slow requests, 1 included below; oldest blocked for > 721.758120 secs 2024-07-26T13:03:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:05 smithi112 ceph-mon[37501]: slow request 721.758120 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T13:03:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:05 smithi107 ceph-mon[32659]: 1 slow requests, 1 included below; oldest blocked for > 721.758120 secs 2024-07-26T13:03:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:05 smithi107 ceph-mon[32659]: slow request 721.758120 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T13:03:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:05 smithi132 ceph-mon[37580]: 1 slow requests, 1 included below; oldest blocked for > 721.758120 secs 2024-07-26T13:03:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:05 smithi132 ceph-mon[37580]: slow request 721.758120 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T13:03:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:06 smithi112 ceph-mon[37501]: pgmap v932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:06 smithi107 ceph-mon[32659]: pgmap v932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:06 smithi132 ceph-mon[37580]: pgmap v932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:07.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:07.642+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:07.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:07.642+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:07.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:07.642+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:08 smithi112 ceph-mon[37501]: pgmap v933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:08 smithi107 ceph-mon[32659]: pgmap v933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:08 smithi132 ceph-mon[37580]: pgmap v933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:09.776+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:09.776+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:09.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:09.776+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:10.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:10 smithi112 ceph-mon[37501]: pgmap v934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:10.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 726.758233 secs 2024-07-26T13:03:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:10 smithi107 ceph-mon[32659]: pgmap v934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 726.758233 secs 2024-07-26T13:03:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:10 smithi132 ceph-mon[37580]: pgmap v934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 726.758233 secs 2024-07-26T13:03:12.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:12.760+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006030 con 0x565421993390 2024-07-26T13:03:12.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:12.760+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006030 con 0x565421993390 2024-07-26T13:03:12.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:12.817+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d8006000 con 0x7ff104102930 2024-07-26T13:03:12.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:12.818+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d8006000 con 0x7ff104102930 2024-07-26T13:03:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:12 smithi112 ceph-mon[37501]: pgmap v935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:12 smithi107 ceph-mon[32659]: pgmap v935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:12 smithi132 ceph-mon[37580]: pgmap v935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:13.865 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:13.864+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006050 con 0x55aecd6b64d0 2024-07-26T13:03:13.865 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:13.864+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006110 con 0x7efc78103340 2024-07-26T13:03:13.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:13.865+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006050 con 0x55aecd6b64d0 2024-07-26T13:03:13.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:13.865+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006110 con 0x7efc78103340 2024-07-26T13:03:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:14 smithi112 ceph-mon[37501]: pgmap v936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:14 smithi107 ceph-mon[32659]: pgmap v936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:14 smithi132 ceph-mon[37580]: pgmap v936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 731.758322 secs 2024-07-26T13:03:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 731.758322 secs 2024-07-26T13:03:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 731.758322 secs 2024-07-26T13:03:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:16 smithi112 ceph-mon[37501]: pgmap v937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:16 smithi107 ceph-mon[32659]: pgmap v937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:16 smithi132 ceph-mon[37580]: pgmap v937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:18 smithi112 ceph-mon[37501]: pgmap v938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:18 smithi107 ceph-mon[32659]: pgmap v938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:18 smithi132 ceph-mon[37580]: pgmap v938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:20 smithi112 ceph-mon[37501]: pgmap v939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 736.758438 secs 2024-07-26T13:03:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:20 smithi107 ceph-mon[32659]: pgmap v939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 736.758438 secs 2024-07-26T13:03:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:20 smithi132 ceph-mon[37580]: pgmap v939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 736.758438 secs 2024-07-26T13:03:22.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:22.650+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:22.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:22.650+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:22.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:22.650+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:22 smithi112 ceph-mon[37501]: pgmap v940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:22 smithi107 ceph-mon[32659]: pgmap v940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:22 smithi132 ceph-mon[37580]: pgmap v940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:24.788+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:24.788+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:24.788+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:24 smithi112 ceph-mon[37501]: pgmap v941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:24 smithi107 ceph-mon[32659]: pgmap v941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:24 smithi132 ceph-mon[37580]: pgmap v941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 741.758561 secs 2024-07-26T13:03:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 741.758561 secs 2024-07-26T13:03:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 741.758561 secs 2024-07-26T13:03:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:26 smithi112 ceph-mon[37501]: pgmap v942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:26 smithi107 ceph-mon[32659]: pgmap v942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:26 smithi132 ceph-mon[37580]: pgmap v942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:28 smithi112 ceph-mon[37501]: pgmap v943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:28 smithi107 ceph-mon[32659]: pgmap v943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:28 smithi132 ceph-mon[37580]: pgmap v943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:30 smithi107 ceph-mon[32659]: pgmap v944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 746.758692 secs 2024-07-26T13:03:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:30 smithi132 ceph-mon[37580]: pgmap v944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 746.758692 secs 2024-07-26T13:03:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:30 smithi112 ceph-mon[37501]: pgmap v944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 746.758692 secs 2024-07-26T13:03:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:32 smithi107 ceph-mon[32659]: pgmap v945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:32 smithi132 ceph-mon[37580]: pgmap v945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:32 smithi112 ceph-mon[37501]: pgmap v945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:34 smithi107 ceph-mon[32659]: pgmap v946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:34 smithi132 ceph-mon[37580]: pgmap v946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:34 smithi112 ceph-mon[37501]: pgmap v946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 751.758837 secs 2024-07-26T13:03:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 751.758837 secs 2024-07-26T13:03:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 751.758837 secs 2024-07-26T13:03:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:36 smithi107 ceph-mon[32659]: pgmap v947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:36 smithi132 ceph-mon[37580]: pgmap v947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:36 smithi112 ceph-mon[37501]: pgmap v947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:37.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:37.659+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:37.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:37.659+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:37.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:37.659+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:38 smithi107 ceph-mon[32659]: pgmap v948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:38 smithi132 ceph-mon[37580]: pgmap v948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:38 smithi112 ceph-mon[37501]: pgmap v948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:39.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:39.793+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:39.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:39.793+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:39.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:39.793+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:40 smithi107 ceph-mon[32659]: pgmap v949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 756.758955 secs 2024-07-26T13:03:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:40 smithi132 ceph-mon[37580]: pgmap v949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 756.758955 secs 2024-07-26T13:03:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:40 smithi112 ceph-mon[37501]: pgmap v949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:03:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 756.758955 secs 2024-07-26T13:03:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:42 smithi107 ceph-mon[32659]: pgmap v950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:42 smithi132 ceph-mon[37580]: pgmap v950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:42 smithi112 ceph-mon[37501]: pgmap v950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:44 smithi107 ceph-mon[32659]: pgmap v951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:44 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:44 smithi132 ceph-mon[37580]: pgmap v951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:44 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:44 smithi112 ceph-mon[37501]: pgmap v951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:44 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:03:45.766 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:45 smithi132 ceph-mon[37580]: pgmap v952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:03:45.766 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:45 smithi132 ceph-mon[37580]: pgmap v953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:45 smithi107 ceph-mon[32659]: pgmap v952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:03:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:45 smithi107 ceph-mon[32659]: pgmap v953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 761.759081 secs 2024-07-26T13:03:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 761.759081 secs 2024-07-26T13:03:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:45 smithi112 ceph-mon[37501]: pgmap v952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:03:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:45 smithi112 ceph-mon[37501]: pgmap v953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 761.759081 secs 2024-07-26T13:03:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:47 smithi107 ceph-mon[32659]: pgmap v954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:03:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:47 smithi132 ceph-mon[37580]: pgmap v954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:03:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:47 smithi112 ceph-mon[37501]: pgmap v954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:03:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:49 smithi107 ceph-mon[32659]: pgmap v955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:03:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:49 smithi132 ceph-mon[37580]: pgmap v955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:03:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:49 smithi112 ceph-mon[37501]: pgmap v955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:03:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 766.759157 secs 2024-07-26T13:03:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 766.759157 secs 2024-07-26T13:03:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 766.759157 secs 2024-07-26T13:03:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:51 smithi107 ceph-mon[32659]: pgmap v956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:51 smithi132 ceph-mon[37580]: pgmap v956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:51 smithi112 ceph-mon[37501]: pgmap v956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:52.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:52.665+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:52.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:52.665+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:52.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:03:52.665+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:53 smithi132 ceph-mon[37580]: pgmap v957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:03:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:53 smithi112 ceph-mon[37501]: pgmap v957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:03:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:53 smithi107 ceph-mon[32659]: pgmap v957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:03:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:54.810+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:54.810+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:03:54.811 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:03:54.810+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:55 smithi132 ceph-mon[37580]: pgmap v958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 771.759274 secs 2024-07-26T13:03:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:55 smithi112 ceph-mon[37501]: pgmap v958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 771.759274 secs 2024-07-26T13:03:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:55 smithi107 ceph-mon[32659]: pgmap v958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 771.759274 secs 2024-07-26T13:03:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:57 smithi132 ceph-mon[37580]: pgmap v959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:57 smithi112 ceph-mon[37501]: pgmap v959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:57 smithi107 ceph-mon[32659]: pgmap v959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:03:59 smithi112 ceph-mon[37501]: pgmap v960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:03:59 smithi107 ceph-mon[32659]: pgmap v960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:03:59 smithi132 ceph-mon[37580]: pgmap v960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 776.759413 secs 2024-07-26T13:04:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 776.759413 secs 2024-07-26T13:04:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 776.759413 secs 2024-07-26T13:04:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:01 smithi112 ceph-mon[37501]: pgmap v961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:01 smithi107 ceph-mon[32659]: pgmap v961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:01 smithi132 ceph-mon[37580]: pgmap v961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:04.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:03 smithi107 ceph-mon[32659]: pgmap v962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:03 smithi112 ceph-mon[37501]: pgmap v962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:03 smithi132 ceph-mon[37580]: pgmap v962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:05 smithi112 ceph-mon[37501]: pgmap v963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 781.759500 secs 2024-07-26T13:04:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:05 smithi107 ceph-mon[32659]: pgmap v963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 781.759500 secs 2024-07-26T13:04:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:05 smithi132 ceph-mon[37580]: pgmap v963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 781.759500 secs 2024-07-26T13:04:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:07.675+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:07.675+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:07.675+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:07 smithi107 ceph-mon[32659]: pgmap v964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:08.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:07 smithi132 ceph-mon[37580]: pgmap v964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:07 smithi112 ceph-mon[37501]: pgmap v964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:09.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:09.826+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:09.826+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:09.826+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:09 smithi112 ceph-mon[37501]: pgmap v965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:09 smithi107 ceph-mon[32659]: pgmap v965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:09 smithi132 ceph-mon[37580]: pgmap v965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 786.759636 secs 2024-07-26T13:04:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 786.759636 secs 2024-07-26T13:04:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 786.759636 secs 2024-07-26T13:04:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:11 smithi112 ceph-mon[37501]: pgmap v966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:11 smithi107 ceph-mon[32659]: pgmap v966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:11 smithi132 ceph-mon[37580]: pgmap v966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:13 smithi112 ceph-mon[37501]: pgmap v967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:13 smithi107 ceph-mon[32659]: pgmap v967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:13 smithi132 ceph-mon[37580]: pgmap v967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:16.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:15 smithi112 ceph-mon[37501]: pgmap v968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:16.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 791.759773 secs 2024-07-26T13:04:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:15 smithi107 ceph-mon[32659]: pgmap v968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 791.759773 secs 2024-07-26T13:04:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:15 smithi132 ceph-mon[37580]: pgmap v968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 791.759773 secs 2024-07-26T13:04:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:17 smithi112 ceph-mon[37501]: pgmap v969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:17 smithi107 ceph-mon[32659]: pgmap v969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:17 smithi132 ceph-mon[37580]: pgmap v969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:19 smithi112 ceph-mon[37501]: pgmap v970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:19 smithi107 ceph-mon[32659]: pgmap v970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:19 smithi132 ceph-mon[37580]: pgmap v970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 796.759899 secs 2024-07-26T13:04:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 796.759899 secs 2024-07-26T13:04:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 796.759899 secs 2024-07-26T13:04:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:21 smithi112 ceph-mon[37501]: pgmap v971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:21 smithi107 ceph-mon[32659]: pgmap v971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:21 smithi132 ceph-mon[37580]: pgmap v971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:22.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:22.681+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:22.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:22.681+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:22.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:22.681+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:23 smithi112 ceph-mon[37501]: pgmap v972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:23 smithi107 ceph-mon[32659]: pgmap v972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:23 smithi132 ceph-mon[37580]: pgmap v972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:24.840+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:24.840+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:24.840+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:25 smithi112 ceph-mon[37501]: pgmap v973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 801.760012 secs 2024-07-26T13:04:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:25 smithi107 ceph-mon[32659]: pgmap v973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 801.760012 secs 2024-07-26T13:04:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:25 smithi132 ceph-mon[37580]: pgmap v973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 801.760012 secs 2024-07-26T13:04:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:27 smithi112 ceph-mon[37501]: pgmap v974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:27 smithi107 ceph-mon[32659]: pgmap v974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:27 smithi132 ceph-mon[37580]: pgmap v974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:29 smithi112 ceph-mon[37501]: pgmap v975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:29 smithi107 ceph-mon[32659]: pgmap v975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:29 smithi132 ceph-mon[37580]: pgmap v975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 806.760132 secs 2024-07-26T13:04:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 806.760132 secs 2024-07-26T13:04:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 806.760132 secs 2024-07-26T13:04:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:31 smithi112 ceph-mon[37501]: pgmap v976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:31 smithi107 ceph-mon[32659]: pgmap v976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:31 smithi132 ceph-mon[37580]: pgmap v976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:33 smithi107 ceph-mon[32659]: pgmap v977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:33 smithi132 ceph-mon[37580]: pgmap v977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:33 smithi112 ceph-mon[37501]: pgmap v977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:35 smithi107 ceph-mon[32659]: pgmap v978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 811.760195 secs 2024-07-26T13:04:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:35 smithi132 ceph-mon[37580]: pgmap v978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 811.760195 secs 2024-07-26T13:04:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:35 smithi112 ceph-mon[37501]: pgmap v978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 811.760195 secs 2024-07-26T13:04:37.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:37.689+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:37.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:37.689+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:37.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:37.689+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:37 smithi107 ceph-mon[32659]: pgmap v979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:37 smithi132 ceph-mon[37580]: pgmap v979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:37 smithi112 ceph-mon[37501]: pgmap v979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:39.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:39.851+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:39.851+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:39.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:39.852+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:39 smithi107 ceph-mon[32659]: pgmap v980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:39 smithi132 ceph-mon[37580]: pgmap v980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:39 smithi112 ceph-mon[37501]: pgmap v980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 816.760339 secs 2024-07-26T13:04:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 816.760339 secs 2024-07-26T13:04:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 816.760339 secs 2024-07-26T13:04:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:41 smithi107 ceph-mon[32659]: pgmap v981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:41 smithi132 ceph-mon[37580]: pgmap v981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:41 smithi112 ceph-mon[37501]: pgmap v981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:43 smithi107 ceph-mon[32659]: pgmap v982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:43 smithi132 ceph-mon[37580]: pgmap v982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:43 smithi112 ceph-mon[37501]: pgmap v982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:04:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:45 smithi107 ceph-mon[32659]: pgmap v983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 821.760428 secs 2024-07-26T13:04:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:45 smithi132 ceph-mon[37580]: pgmap v983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 821.760428 secs 2024-07-26T13:04:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:45 smithi112 ceph-mon[37501]: pgmap v983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 821.760428 secs 2024-07-26T13:04:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:04:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:48 smithi107 ceph-mon[32659]: pgmap v984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:04:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:48 smithi132 ceph-mon[37580]: pgmap v984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:04:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:48 smithi112 ceph-mon[37501]: pgmap v984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:04:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:50 smithi107 ceph-mon[32659]: pgmap v985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:50 smithi132 ceph-mon[37580]: pgmap v985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:50 smithi112 ceph-mon[37501]: pgmap v985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 826.760512 secs 2024-07-26T13:04:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 826.760512 secs 2024-07-26T13:04:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 826.760512 secs 2024-07-26T13:04:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:52 smithi107 ceph-mon[32659]: pgmap v986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:52 smithi132 ceph-mon[37580]: pgmap v986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:52 smithi112 ceph-mon[37501]: pgmap v986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:52.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:52.700+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:52.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:52.700+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:52.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:04:52.700+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:54 smithi107 ceph-mon[32659]: pgmap v987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:54 smithi132 ceph-mon[37580]: pgmap v987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:54 smithi112 ceph-mon[37501]: pgmap v987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:04:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:54.868+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:54.868+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:04:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:04:54.868+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:56 smithi132 ceph-mon[37580]: pgmap v988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 831.760630 secs 2024-07-26T13:04:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:56 smithi112 ceph-mon[37501]: pgmap v988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 831.760630 secs 2024-07-26T13:04:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:56 smithi107 ceph-mon[32659]: pgmap v988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:04:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 831.760630 secs 2024-07-26T13:04:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:04:58 smithi132 ceph-mon[37580]: pgmap v989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:04:58 smithi112 ceph-mon[37501]: pgmap v989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:04:58 smithi107 ceph-mon[32659]: pgmap v989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:00 smithi132 ceph-mon[37580]: pgmap v990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:05:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:00 smithi112 ceph-mon[37501]: pgmap v990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:05:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:00 smithi107 ceph-mon[32659]: pgmap v990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:05:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 836.760769 secs 2024-07-26T13:05:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 836.760769 secs 2024-07-26T13:05:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 836.760769 secs 2024-07-26T13:05:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:02 smithi112 ceph-mon[37501]: pgmap v991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:02 smithi107 ceph-mon[32659]: pgmap v991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:02 smithi132 ceph-mon[37580]: pgmap v991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:04.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:04 smithi107 ceph-mon[32659]: pgmap v992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:04 smithi112 ceph-mon[37501]: pgmap v992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:04 smithi132 ceph-mon[37580]: pgmap v992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:06 smithi112 ceph-mon[37501]: pgmap v993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 841.760867 secs 2024-07-26T13:05:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:06 smithi107 ceph-mon[32659]: pgmap v993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 841.760867 secs 2024-07-26T13:05:06.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:06 smithi132 ceph-mon[37580]: pgmap v993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:06.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 841.760867 secs 2024-07-26T13:05:07.717 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:07.716+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:07.716+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:07.718 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:07.716+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:08.361 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:08 smithi132 ceph-mon[37580]: pgmap v994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:08 smithi112 ceph-mon[37501]: pgmap v994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:08 smithi107 ceph-mon[32659]: pgmap v994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:09.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:09.884+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:09.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:09.884+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:09.885 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:09.884+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:10 smithi112 ceph-mon[37501]: pgmap v995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:10 smithi107 ceph-mon[32659]: pgmap v995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:10 smithi132 ceph-mon[37580]: pgmap v995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 846.760959 secs 2024-07-26T13:05:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:11 smithi112 ceph-mon[37501]: pgmap v996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 846.760959 secs 2024-07-26T13:05:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:11 smithi107 ceph-mon[32659]: pgmap v996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 846.760959 secs 2024-07-26T13:05:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:11 smithi132 ceph-mon[37580]: pgmap v996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:14 smithi107 ceph-mon[32659]: pgmap v997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:14 smithi132 ceph-mon[37580]: pgmap v997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:14 smithi112 ceph-mon[37501]: pgmap v997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:16 smithi107 ceph-mon[32659]: pgmap v998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 851.761087 secs 2024-07-26T13:05:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:16 smithi132 ceph-mon[37580]: pgmap v998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 851.761087 secs 2024-07-26T13:05:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:16 smithi112 ceph-mon[37501]: pgmap v998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 851.761087 secs 2024-07-26T13:05:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:18 smithi107 ceph-mon[32659]: pgmap v999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:18 smithi132 ceph-mon[37580]: pgmap v999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:18 smithi112 ceph-mon[37501]: pgmap v999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:20 smithi107 ceph-mon[32659]: pgmap v1000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:20 smithi132 ceph-mon[37580]: pgmap v1000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:20 smithi112 ceph-mon[37501]: pgmap v1000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 856.761211 secs 2024-07-26T13:05:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 856.761211 secs 2024-07-26T13:05:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 856.761211 secs 2024-07-26T13:05:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:22 smithi132 ceph-mon[37580]: pgmap v1001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:22 smithi112 ceph-mon[37501]: pgmap v1001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:22 smithi107 ceph-mon[32659]: pgmap v1001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:22.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:22.730+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:22.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:22.730+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:22.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:22.730+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:24 smithi112 ceph-mon[37501]: pgmap v1002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:24 smithi107 ceph-mon[32659]: pgmap v1002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:24 smithi132 ceph-mon[37580]: pgmap v1002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:24.900 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:24.898+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:24.898+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:24.901 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:24.898+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:26 smithi112 ceph-mon[37501]: pgmap v1003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 861.761329 secs 2024-07-26T13:05:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:26 smithi107 ceph-mon[32659]: pgmap v1003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 861.761329 secs 2024-07-26T13:05:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:26 smithi132 ceph-mon[37580]: pgmap v1003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 861.761329 secs 2024-07-26T13:05:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:28 smithi112 ceph-mon[37501]: pgmap v1004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:28 smithi107 ceph-mon[32659]: pgmap v1004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:28 smithi132 ceph-mon[37580]: pgmap v1004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:30 smithi112 ceph-mon[37501]: pgmap v1005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:30 smithi107 ceph-mon[32659]: pgmap v1005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:30 smithi132 ceph-mon[37580]: pgmap v1005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 866.761453 secs 2024-07-26T13:05:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 866.761453 secs 2024-07-26T13:05:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 866.761453 secs 2024-07-26T13:05:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:32 smithi112 ceph-mon[37501]: pgmap v1006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:32 smithi107 ceph-mon[32659]: pgmap v1006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:32.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:32 smithi132 ceph-mon[37580]: pgmap v1006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:34 smithi112 ceph-mon[37501]: pgmap v1007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:34 smithi107 ceph-mon[32659]: pgmap v1007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:34 smithi132 ceph-mon[37580]: pgmap v1007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:35 smithi112 ceph-mon[37501]: pgmap v1008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:35 smithi107 ceph-mon[32659]: pgmap v1008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:35 smithi132 ceph-mon[37580]: pgmap v1008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 871.761590 secs 2024-07-26T13:05:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 871.761590 secs 2024-07-26T13:05:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 871.761590 secs 2024-07-26T13:05:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:37 smithi112 ceph-mon[37501]: pgmap v1009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:37 smithi107 ceph-mon[32659]: pgmap v1009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:37 smithi132 ceph-mon[37580]: pgmap v1009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:37.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:37.746+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:37.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:37.746+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:37.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:37.746+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:39.900+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:39.900+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:39.902 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:39.900+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:39 smithi112 ceph-mon[37501]: pgmap v1010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:39 smithi107 ceph-mon[32659]: pgmap v1010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:39 smithi132 ceph-mon[37580]: pgmap v1010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 876.761725 secs 2024-07-26T13:05:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 876.761725 secs 2024-07-26T13:05:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 876.761725 secs 2024-07-26T13:05:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:41 smithi112 ceph-mon[37501]: pgmap v1011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:41 smithi107 ceph-mon[32659]: pgmap v1011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:41 smithi132 ceph-mon[37580]: pgmap v1011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:43 smithi112 ceph-mon[37501]: pgmap v1012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:43 smithi107 ceph-mon[32659]: pgmap v1012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:43 smithi132 ceph-mon[37580]: pgmap v1012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:05:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:45 smithi112 ceph-mon[37501]: pgmap v1013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 881.761829 secs 2024-07-26T13:05:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:45 smithi107 ceph-mon[32659]: pgmap v1013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 881.761829 secs 2024-07-26T13:05:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:45 smithi132 ceph-mon[37580]: pgmap v1013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:05:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 881.761829 secs 2024-07-26T13:05:47.871 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:47 smithi132 ceph-mon[37580]: pgmap v1014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:47.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:47.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.872 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.924 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:47 smithi112 ceph-mon[37501]: pgmap v1014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:47.924 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:47.924 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.924 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.924 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.924 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.997 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:47 smithi107 ceph-mon[32659]: pgmap v1014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:47.997 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:47.998 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.998 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.998 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:47.998 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:49 smithi112 ceph-mon[37501]: pgmap v1015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:49 smithi107 ceph-mon[32659]: pgmap v1015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:49 smithi132 ceph-mon[37580]: pgmap v1015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 886.761900 secs 2024-07-26T13:05:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 886.761900 secs 2024-07-26T13:05:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 886.761900 secs 2024-07-26T13:05:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:51 smithi112 ceph-mon[37501]: pgmap v1016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:51 smithi107 ceph-mon[32659]: pgmap v1016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:51 smithi132 ceph-mon[37580]: pgmap v1016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:52.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:52.763+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:52.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:52.763+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:52.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:05:52.763+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:52 smithi112 ceph-mon[37501]: pgmap v1017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:05:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:52 smithi112 ceph-mon[37501]: pgmap v1018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:05:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:52 smithi107 ceph-mon[32659]: pgmap v1017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:05:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:52 smithi107 ceph-mon[32659]: pgmap v1018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:05:53.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:52 smithi132 ceph-mon[37580]: pgmap v1017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:05:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:52 smithi132 ceph-mon[37580]: pgmap v1018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:05:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:05:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:53 smithi112 ceph-mon[37501]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T13:05:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:53 smithi107 ceph-mon[32659]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T13:05:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:53 smithi132 ceph-mon[37580]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T13:05:54.917 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:54.916+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:54.916+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:05:54.916+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:05:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:54 smithi107 ceph-mon[32659]: pgmap v1019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:05:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:54 smithi132 ceph-mon[37580]: pgmap v1019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:05:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:54 smithi112 ceph-mon[37501]: pgmap v1019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:05:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 891.762030 secs 2024-07-26T13:05:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 891.762030 secs 2024-07-26T13:05:56.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 891.762030 secs 2024-07-26T13:05:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:56 smithi107 ceph-mon[32659]: pgmap v1020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:05:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:56 smithi132 ceph-mon[37580]: pgmap v1020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:05:57.119 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:56 smithi112 ceph-mon[37501]: pgmap v1020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:05:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:05:58 smithi107 ceph-mon[32659]: pgmap v1021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:05:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:05:58 smithi132 ceph-mon[37580]: pgmap v1021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:05:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:05:58 smithi112 ceph-mon[37501]: pgmap v1021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:06:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:00 smithi132 ceph-mon[37580]: pgmap v1022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 896.762161 secs 2024-07-26T13:06:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:00 smithi112 ceph-mon[37501]: pgmap v1022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 896.762161 secs 2024-07-26T13:06:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:00 smithi107 ceph-mon[32659]: pgmap v1022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 896.762161 secs 2024-07-26T13:06:03.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:02 smithi132 ceph-mon[37580]: pgmap v1023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:02 smithi112 ceph-mon[37501]: pgmap v1023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:02 smithi107 ceph-mon[32659]: pgmap v1023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:04 smithi132 ceph-mon[37580]: pgmap v1024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:05.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:04 smithi107 ceph-mon[32659]: pgmap v1024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:04 smithi112 ceph-mon[37501]: pgmap v1024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 901.762283 secs 2024-07-26T13:06:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 901.762283 secs 2024-07-26T13:06:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 901.762283 secs 2024-07-26T13:06:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:06 smithi107 ceph-mon[32659]: pgmap v1025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:06 smithi112 ceph-mon[37501]: pgmap v1025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:06 smithi132 ceph-mon[37580]: pgmap v1025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:07.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:07.778+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:07.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:07.779+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:07.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:07.779+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:08 smithi112 ceph-mon[37501]: pgmap v1026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:08 smithi107 ceph-mon[32659]: pgmap v1026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:08 smithi132 ceph-mon[37580]: pgmap v1026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:09.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:09.923+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:09.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:09.923+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:09.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:09.923+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:11.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:10 smithi132 ceph-mon[37580]: pgmap v1027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:11.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 906.762393 secs 2024-07-26T13:06:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:10 smithi112 ceph-mon[37501]: pgmap v1027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 906.762393 secs 2024-07-26T13:06:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:10 smithi107 ceph-mon[32659]: pgmap v1027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 906.762393 secs 2024-07-26T13:06:11.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:11.636+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T13:06:11.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:11.636+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:06:11.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:11.636+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0b6250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x565421d57c90 tx=0x7ff0f8009260 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:06:11.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:11.636+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0b6250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x565421d57c90 tx=0x7ff0f8009260 comp rx=0 tx=0).stop 2024-07-26T13:06:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:11.636+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0b6250 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T13:06:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:11.636+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0b6250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:06:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:11.636+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0b6250 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:06:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:11.636+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:06:11.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:11.637+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:06:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:11.639+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c1830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80126f0 tx=0x7ff0f8014040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:06:12.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:12.789+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d40062a0 con 0x565421993390 2024-07-26T13:06:12.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:12.790+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d40062a0 con 0x565421993390 2024-07-26T13:06:12.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:12.850+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80061d0 con 0x7ff104102930 2024-07-26T13:06:12.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:12.851+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80061d0 con 0x7ff104102930 2024-07-26T13:06:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:13 smithi132 ceph-mon[37580]: pgmap v1028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:13 smithi112 ceph-mon[37501]: pgmap v1028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:13 smithi107 ceph-mon[32659]: pgmap v1028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.745+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 msgr2=0x7efc5c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000296) for more than 900000000 us, fault. 2024-07-26T13:06:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.745+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0b6250 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7efc680e1b10 tx=0x7efc680ea550 comp rx=0 tx=0).stop 2024-07-26T13:06:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.745+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 msgr2=0x7efc5c0b6250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:06:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.745+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0b6250 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:06:13.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.745+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:06:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.746+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:06:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.747+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c1830 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efc680a7050 tx=0x7efc680e6000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:06:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.896+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480063d0 con 0x7efc78103340 2024-07-26T13:06:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.896+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc440062c0 con 0x55aecd6b64d0 2024-07-26T13:06:13.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.897+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc440062c0 con 0x55aecd6b64d0 2024-07-26T13:06:13.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:13.897+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480063d0 con 0x7efc78103340 2024-07-26T13:06:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:15 smithi112 ceph-mon[37501]: pgmap v1029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:15 smithi107 ceph-mon[32659]: pgmap v1029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:15 smithi132 ceph-mon[37580]: pgmap v1029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 911.762509 secs 2024-07-26T13:06:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 911.762509 secs 2024-07-26T13:06:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 911.762509 secs 2024-07-26T13:06:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:17 smithi107 ceph-mon[32659]: pgmap v1030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:17 smithi112 ceph-mon[37501]: pgmap v1030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:17 smithi132 ceph-mon[37580]: pgmap v1030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:19 smithi112 ceph-mon[37501]: pgmap v1031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:19 smithi107 ceph-mon[32659]: pgmap v1031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:19.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:19 smithi132 ceph-mon[37580]: pgmap v1031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:21 smithi112 ceph-mon[37501]: pgmap v1032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 916.762629 secs 2024-07-26T13:06:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:21 smithi107 ceph-mon[32659]: pgmap v1032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 916.762629 secs 2024-07-26T13:06:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:21 smithi132 ceph-mon[37580]: pgmap v1032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 916.762629 secs 2024-07-26T13:06:22.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:22.789+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:22.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:22.789+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:22.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:22.789+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:23.382 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:23 smithi132 ceph-mon[37580]: pgmap v1033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:23 smithi112 ceph-mon[37501]: pgmap v1033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:23 smithi107 ceph-mon[32659]: pgmap v1033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:24.923+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:24.923+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:24.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:24.923+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:25 smithi107 ceph-mon[32659]: pgmap v1034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:25 smithi112 ceph-mon[37501]: pgmap v1034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:25 smithi132 ceph-mon[37580]: pgmap v1034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 921.762718 secs 2024-07-26T13:06:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 921.762718 secs 2024-07-26T13:06:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 921.762718 secs 2024-07-26T13:06:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:27 smithi112 ceph-mon[37501]: pgmap v1035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:27 smithi107 ceph-mon[32659]: pgmap v1035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:27 smithi132 ceph-mon[37580]: pgmap v1035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:28 smithi112 ceph-mon[37501]: pgmap v1036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:28 smithi107 ceph-mon[32659]: pgmap v1036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:28 smithi132 ceph-mon[37580]: pgmap v1036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:30 smithi132 ceph-mon[37580]: pgmap v1037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 926.762840 secs 2024-07-26T13:06:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:30 smithi112 ceph-mon[37501]: pgmap v1037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 926.762840 secs 2024-07-26T13:06:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:30 smithi107 ceph-mon[32659]: pgmap v1037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 926.762840 secs 2024-07-26T13:06:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:32 smithi112 ceph-mon[37501]: pgmap v1038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:32 smithi107 ceph-mon[32659]: pgmap v1038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:32 smithi132 ceph-mon[37580]: pgmap v1038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:35.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:34 smithi107 ceph-mon[32659]: pgmap v1039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:34 smithi112 ceph-mon[37501]: pgmap v1039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:34 smithi132 ceph-mon[37580]: pgmap v1039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 931.762977 secs 2024-07-26T13:06:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 931.762977 secs 2024-07-26T13:06:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 931.762977 secs 2024-07-26T13:06:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:36 smithi112 ceph-mon[37501]: pgmap v1040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:36 smithi107 ceph-mon[32659]: pgmap v1040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:36 smithi132 ceph-mon[37580]: pgmap v1040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:37.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:37.805+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:37.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:37.805+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:37.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:37.805+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:38 smithi112 ceph-mon[37501]: pgmap v1041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:38 smithi107 ceph-mon[32659]: pgmap v1041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:38 smithi132 ceph-mon[37580]: pgmap v1041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:39.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:39.928+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:39.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:39.928+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:39.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:39.928+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:41.111 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:40 smithi132 ceph-mon[37580]: pgmap v1042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:41.111 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 936.763096 secs 2024-07-26T13:06:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:40 smithi112 ceph-mon[37501]: pgmap v1042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 936.763096 secs 2024-07-26T13:06:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:40 smithi107 ceph-mon[32659]: pgmap v1042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 936.763096 secs 2024-07-26T13:06:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:42 smithi112 ceph-mon[37501]: pgmap v1043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:42 smithi107 ceph-mon[32659]: pgmap v1043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:42 smithi132 ceph-mon[37580]: pgmap v1043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:44 smithi112 ceph-mon[37501]: pgmap v1044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:44 smithi107 ceph-mon[32659]: pgmap v1044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:44 smithi132 ceph-mon[37580]: pgmap v1044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 941.763145 secs 2024-07-26T13:06:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 941.763145 secs 2024-07-26T13:06:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 941.763145 secs 2024-07-26T13:06:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:46 smithi112 ceph-mon[37501]: pgmap v1045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:46 smithi107 ceph-mon[32659]: pgmap v1045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:46 smithi132 ceph-mon[37580]: pgmap v1045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:48 smithi112 ceph-mon[37501]: pgmap v1046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:48 smithi107 ceph-mon[32659]: pgmap v1046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:48 smithi132 ceph-mon[37580]: pgmap v1046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:50 smithi112 ceph-mon[37501]: pgmap v1047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 946.763273 secs 2024-07-26T13:06:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:50 smithi107 ceph-mon[32659]: pgmap v1047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 946.763273 secs 2024-07-26T13:06:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:50 smithi132 ceph-mon[37580]: pgmap v1047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:06:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 946.763273 secs 2024-07-26T13:06:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:52.822+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:52.822+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:52.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:06:52.822+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:52 smithi112 ceph-mon[37501]: pgmap v1048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:52 smithi107 ceph-mon[32659]: pgmap v1048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:52 smithi132 ceph-mon[37580]: pgmap v1048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:53 smithi112 ceph-mon[37501]: pgmap v1049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:06:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:53 smithi112 ceph-mon[37501]: pgmap v1050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:06:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:06:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:53 smithi107 ceph-mon[32659]: pgmap v1049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:06:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:53 smithi107 ceph-mon[32659]: pgmap v1050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:06:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:06:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:53 smithi132 ceph-mon[37580]: pgmap v1049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:06:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:53 smithi132 ceph-mon[37580]: pgmap v1050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:06:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:06:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:54.940+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:54.940+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:06:54.940+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:06:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:56 smithi107 ceph-mon[32659]: pgmap v1051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 951.763393 secs 2024-07-26T13:06:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:56 smithi132 ceph-mon[37580]: pgmap v1051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 951.763393 secs 2024-07-26T13:06:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:56 smithi112 ceph-mon[37501]: pgmap v1051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:06:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 951.763393 secs 2024-07-26T13:06:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:06:58 smithi107 ceph-mon[32659]: pgmap v1052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:06:58 smithi132 ceph-mon[37580]: pgmap v1052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:06:58 smithi112 ceph-mon[37501]: pgmap v1052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:00 smithi132 ceph-mon[37580]: pgmap v1053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:07:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:00 smithi112 ceph-mon[37501]: pgmap v1053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:07:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:00 smithi107 ceph-mon[32659]: pgmap v1053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:07:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 956.763509 secs 2024-07-26T13:07:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 956.763509 secs 2024-07-26T13:07:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 956.763509 secs 2024-07-26T13:07:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:02 smithi132 ceph-mon[37580]: pgmap v1054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:02 smithi112 ceph-mon[37501]: pgmap v1054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:02 smithi107 ceph-mon[32659]: pgmap v1054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:07:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:04 smithi112 ceph-mon[37501]: pgmap v1055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:04 smithi107 ceph-mon[32659]: pgmap v1055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:04 smithi132 ceph-mon[37580]: pgmap v1055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:06.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:06 smithi107 ceph-mon[32659]: pgmap v1056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:06.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 961.763645 secs 2024-07-26T13:07:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:06 smithi112 ceph-mon[37501]: pgmap v1056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 961.763645 secs 2024-07-26T13:07:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:06 smithi132 ceph-mon[37580]: pgmap v1056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 961.763645 secs 2024-07-26T13:07:07.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:07.835+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:07.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:07.835+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:07.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:07.835+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:08 smithi107 ceph-mon[32659]: pgmap v1057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:08 smithi112 ceph-mon[37501]: pgmap v1057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:08 smithi132 ceph-mon[37580]: pgmap v1057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:09.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:09.956+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:09.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:09.956+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:09.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:09.956+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:10 smithi112 ceph-mon[37501]: pgmap v1058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:10 smithi107 ceph-mon[32659]: pgmap v1058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:10 smithi132 ceph-mon[37580]: pgmap v1058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 966.763789 secs 2024-07-26T13:07:11.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 966.763789 secs 2024-07-26T13:07:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 966.763789 secs 2024-07-26T13:07:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:12 smithi107 ceph-mon[32659]: pgmap v1059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:12 smithi112 ceph-mon[37501]: pgmap v1059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:12 smithi132 ceph-mon[37580]: pgmap v1059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:14 smithi112 ceph-mon[37501]: pgmap v1060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:14 smithi107 ceph-mon[32659]: pgmap v1060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:14 smithi132 ceph-mon[37580]: pgmap v1060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:16 smithi112 ceph-mon[37501]: pgmap v1061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 971.763851 secs 2024-07-26T13:07:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:16 smithi107 ceph-mon[32659]: pgmap v1061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 971.763851 secs 2024-07-26T13:07:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:16 smithi132 ceph-mon[37580]: pgmap v1061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 971.763851 secs 2024-07-26T13:07:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:18 smithi112 ceph-mon[37501]: pgmap v1062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:18 smithi107 ceph-mon[32659]: pgmap v1062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:18 smithi132 ceph-mon[37580]: pgmap v1062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:20 smithi112 ceph-mon[37501]: pgmap v1063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:20 smithi107 ceph-mon[32659]: pgmap v1063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:20 smithi132 ceph-mon[37580]: pgmap v1063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 976.763974 secs 2024-07-26T13:07:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 976.763974 secs 2024-07-26T13:07:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 976.763974 secs 2024-07-26T13:07:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:22 smithi112 ceph-mon[37501]: pgmap v1064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:22 smithi107 ceph-mon[32659]: pgmap v1064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:22 smithi132 ceph-mon[37580]: pgmap v1064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:22.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:22.850+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:22.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:22.850+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:22.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:22.850+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:23 smithi112 ceph-mon[37501]: pgmap v1065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:23 smithi107 ceph-mon[32659]: pgmap v1065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:23 smithi132 ceph-mon[37580]: pgmap v1065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:24.969+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:24.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:24.969+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:24.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:24.969+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:26 smithi132 ceph-mon[37580]: pgmap v1066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 981.764077 secs 2024-07-26T13:07:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:26 smithi112 ceph-mon[37501]: pgmap v1066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 981.764077 secs 2024-07-26T13:07:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:26 smithi107 ceph-mon[32659]: pgmap v1066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 981.764077 secs 2024-07-26T13:07:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:28 smithi132 ceph-mon[37580]: pgmap v1067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:28 smithi112 ceph-mon[37501]: pgmap v1067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:28 smithi107 ceph-mon[32659]: pgmap v1067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:30 smithi112 ceph-mon[37501]: pgmap v1068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:30 smithi107 ceph-mon[32659]: pgmap v1068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:30 smithi132 ceph-mon[37580]: pgmap v1068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 986.764179 secs 2024-07-26T13:07:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 986.764179 secs 2024-07-26T13:07:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 986.764179 secs 2024-07-26T13:07:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:32 smithi112 ceph-mon[37501]: pgmap v1069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:32 smithi107 ceph-mon[32659]: pgmap v1069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:32 smithi132 ceph-mon[37580]: pgmap v1069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:34 smithi112 ceph-mon[37501]: pgmap v1070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:34 smithi107 ceph-mon[32659]: pgmap v1070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:34 smithi132 ceph-mon[37580]: pgmap v1070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:36.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:36 smithi107 ceph-mon[32659]: pgmap v1071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:36.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 991.764293 secs 2024-07-26T13:07:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:36 smithi112 ceph-mon[37501]: pgmap v1071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 991.764293 secs 2024-07-26T13:07:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:36 smithi132 ceph-mon[37580]: pgmap v1071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 991.764293 secs 2024-07-26T13:07:37.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:37.865+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:37.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:37.865+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:37.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:37.865+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:38 smithi107 ceph-mon[32659]: pgmap v1072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:38 smithi112 ceph-mon[37501]: pgmap v1072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:38 smithi132 ceph-mon[37580]: pgmap v1072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:39.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:39.977+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:39.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:39.977+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:39.979 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:39.977+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:40 smithi112 ceph-mon[37501]: pgmap v1073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:40 smithi107 ceph-mon[32659]: pgmap v1073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:40 smithi132 ceph-mon[37580]: pgmap v1073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 996.764387 secs 2024-07-26T13:07:41.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 996.764387 secs 2024-07-26T13:07:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 996.764387 secs 2024-07-26T13:07:41.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:41.804+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006470 con 0x565421993390 2024-07-26T13:07:41.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:41.804+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006470 con 0x565421993390 2024-07-26T13:07:41.865 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:41.864+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80063f0 con 0x7ff104102930 2024-07-26T13:07:41.866 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:41.864+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80063f0 con 0x7ff104102930 2024-07-26T13:07:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:42 smithi112 ceph-mon[37501]: pgmap v1074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:42 smithi107 ceph-mon[32659]: pgmap v1074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:42 smithi132 ceph-mon[37580]: pgmap v1074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:43.912+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:07:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:43.912+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:07:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:43.912+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:07:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:43.912+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:07:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:44 smithi112 ceph-mon[37501]: pgmap v1075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:44 smithi107 ceph-mon[32659]: pgmap v1075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:44 smithi132 ceph-mon[37580]: pgmap v1075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:46 smithi112 ceph-mon[37501]: pgmap v1076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1001.764521 secs 2024-07-26T13:07:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:46 smithi107 ceph-mon[32659]: pgmap v1076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1001.764521 secs 2024-07-26T13:07:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:46 smithi132 ceph-mon[37580]: pgmap v1076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1001.764521 secs 2024-07-26T13:07:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:48 smithi112 ceph-mon[37501]: pgmap v1077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:48 smithi107 ceph-mon[32659]: pgmap v1077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:48 smithi132 ceph-mon[37580]: pgmap v1077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:49 smithi112 ceph-mon[37501]: pgmap v1078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:49 smithi107 ceph-mon[32659]: pgmap v1078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:49 smithi132 ceph-mon[37580]: pgmap v1078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1006.764640 secs 2024-07-26T13:07:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1006.764640 secs 2024-07-26T13:07:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1006.764640 secs 2024-07-26T13:07:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:51 smithi112 ceph-mon[37501]: pgmap v1079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:51 smithi107 ceph-mon[32659]: pgmap v1079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:51 smithi132 ceph-mon[37580]: pgmap v1079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:52.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:52.881+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:52.881+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:07:52.881+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:54 smithi112 ceph-mon[37501]: pgmap v1080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:54 smithi107 ceph-mon[32659]: pgmap v1080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:54 smithi132 ceph-mon[37580]: pgmap v1080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:54.994 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:54.994+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:54.994+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:07:54.995 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:07:54.994+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:07:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:07:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:07:56.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:56 smithi112 ceph-mon[37501]: pgmap v1081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:07:56.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1011.764780 secs 2024-07-26T13:07:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:56 smithi107 ceph-mon[32659]: pgmap v1081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:07:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1011.764780 secs 2024-07-26T13:07:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:56 smithi132 ceph-mon[37580]: pgmap v1081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:07:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1011.764780 secs 2024-07-26T13:07:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:07:58 smithi107 ceph-mon[32659]: pgmap v1082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:07:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:07:58 smithi112 ceph-mon[37501]: pgmap v1082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:07:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:07:58 smithi132 ceph-mon[37580]: pgmap v1082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:08:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:00 smithi112 ceph-mon[37501]: pgmap v1083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:08:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:00 smithi107 ceph-mon[32659]: pgmap v1083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:08:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:00 smithi132 ceph-mon[37580]: pgmap v1083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:08:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1016.764874 secs 2024-07-26T13:08:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1016.764874 secs 2024-07-26T13:08:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1016.764874 secs 2024-07-26T13:08:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:02 smithi112 ceph-mon[37501]: pgmap v1084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:08:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:02 smithi107 ceph-mon[32659]: pgmap v1084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:08:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:02 smithi132 ceph-mon[37580]: pgmap v1084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:08:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:04 smithi112 ceph-mon[37501]: pgmap v1085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:04 smithi107 ceph-mon[32659]: pgmap v1085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:04 smithi132 ceph-mon[37580]: pgmap v1085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:08:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:06 smithi112 ceph-mon[37501]: pgmap v1086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1021.764969 secs 2024-07-26T13:08:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:06 smithi107 ceph-mon[32659]: pgmap v1086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1021.764969 secs 2024-07-26T13:08:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:06 smithi132 ceph-mon[37580]: pgmap v1086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1021.764969 secs 2024-07-26T13:08:07.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:07.896+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:07.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:07.896+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:07.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:07.896+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:08 smithi107 ceph-mon[32659]: pgmap v1087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:08 smithi112 ceph-mon[37501]: pgmap v1087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:08 smithi132 ceph-mon[37580]: pgmap v1087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:10.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:10.009+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:10.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:10.009+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:10.010 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:10.009+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:10 smithi112 ceph-mon[37501]: pgmap v1088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:10 smithi107 ceph-mon[32659]: pgmap v1088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:10 smithi132 ceph-mon[37580]: pgmap v1088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1026.765100 secs 2024-07-26T13:08:11.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1026.765100 secs 2024-07-26T13:08:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1026.765100 secs 2024-07-26T13:08:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:12 smithi112 ceph-mon[37501]: pgmap v1089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:12 smithi107 ceph-mon[32659]: pgmap v1089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:12 smithi132 ceph-mon[37580]: pgmap v1089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:13 smithi112 ceph-mon[37501]: pgmap v1090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:13 smithi107 ceph-mon[32659]: pgmap v1090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:13 smithi132 ceph-mon[37580]: pgmap v1090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:15 smithi107 ceph-mon[32659]: pgmap v1091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:15.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:15 smithi132 ceph-mon[37580]: pgmap v1091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:15 smithi112 ceph-mon[37501]: pgmap v1091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1031.765221 secs 2024-07-26T13:08:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1031.765221 secs 2024-07-26T13:08:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1031.765221 secs 2024-07-26T13:08:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:17 smithi107 ceph-mon[32659]: pgmap v1092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:17 smithi132 ceph-mon[37580]: pgmap v1092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:17 smithi112 ceph-mon[37501]: pgmap v1092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:19 smithi107 ceph-mon[32659]: pgmap v1093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:19 smithi132 ceph-mon[37580]: pgmap v1093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:19 smithi112 ceph-mon[37501]: pgmap v1093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1036.765335 secs 2024-07-26T13:08:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:21 smithi107 ceph-mon[32659]: pgmap v1094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1036.765335 secs 2024-07-26T13:08:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:21 smithi132 ceph-mon[37580]: pgmap v1094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1036.765335 secs 2024-07-26T13:08:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:21 smithi112 ceph-mon[37501]: pgmap v1094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:22.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:22.911+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:22.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:22.911+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:22.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:22.911+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:23 smithi107 ceph-mon[32659]: pgmap v1095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:23 smithi132 ceph-mon[37580]: pgmap v1095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:23 smithi112 ceph-mon[37501]: pgmap v1095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:25.026 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:25.024+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:25.026 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:25.024+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:25.026 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:25.024+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:25 smithi107 ceph-mon[32659]: pgmap v1096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:25 smithi132 ceph-mon[37580]: pgmap v1096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:25 smithi112 ceph-mon[37501]: pgmap v1096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1041.765469 secs 2024-07-26T13:08:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1041.765469 secs 2024-07-26T13:08:26.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1041.765469 secs 2024-07-26T13:08:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:27 smithi107 ceph-mon[32659]: pgmap v1097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:27 smithi132 ceph-mon[37580]: pgmap v1097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:27 smithi112 ceph-mon[37501]: pgmap v1097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:29 smithi107 ceph-mon[32659]: pgmap v1098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:29 smithi132 ceph-mon[37580]: pgmap v1098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:29 smithi112 ceph-mon[37501]: pgmap v1098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1046.765582 secs 2024-07-26T13:08:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:31 smithi107 ceph-mon[32659]: pgmap v1099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1046.765582 secs 2024-07-26T13:08:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:31 smithi132 ceph-mon[37580]: pgmap v1099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1046.765582 secs 2024-07-26T13:08:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:31 smithi112 ceph-mon[37501]: pgmap v1099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:34 smithi112 ceph-mon[37501]: pgmap v1100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:34 smithi107 ceph-mon[32659]: pgmap v1100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:34 smithi132 ceph-mon[37580]: pgmap v1100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:36 smithi112 ceph-mon[37501]: pgmap v1101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1051.765694 secs 2024-07-26T13:08:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:36 smithi107 ceph-mon[32659]: pgmap v1101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1051.765694 secs 2024-07-26T13:08:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:36 smithi132 ceph-mon[37580]: pgmap v1101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1051.765694 secs 2024-07-26T13:08:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:37 smithi112 ceph-mon[37501]: pgmap v1102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:37 smithi107 ceph-mon[32659]: pgmap v1102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:37 smithi132 ceph-mon[37580]: pgmap v1102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:37.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:37.913+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:37.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:37.913+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:37.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:37.913+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:39 smithi107 ceph-mon[32659]: pgmap v1103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:39 smithi132 ceph-mon[37580]: pgmap v1103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:39 smithi112 ceph-mon[37501]: pgmap v1103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:40.039+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:40.040+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:40.040+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1056.765822 secs 2024-07-26T13:08:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:41 smithi107 ceph-mon[32659]: pgmap v1104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1056.765822 secs 2024-07-26T13:08:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:41 smithi132 ceph-mon[37580]: pgmap v1104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1056.765822 secs 2024-07-26T13:08:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:41 smithi112 ceph-mon[37501]: pgmap v1104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:43 smithi107 ceph-mon[32659]: pgmap v1105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:43 smithi132 ceph-mon[37580]: pgmap v1105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:43 smithi112 ceph-mon[37501]: pgmap v1105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:45 smithi107 ceph-mon[32659]: pgmap v1106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:45 smithi132 ceph-mon[37580]: pgmap v1106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:45 smithi112 ceph-mon[37501]: pgmap v1106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1061.765936 secs 2024-07-26T13:08:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1061.765936 secs 2024-07-26T13:08:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1061.765936 secs 2024-07-26T13:08:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:47 smithi107 ceph-mon[32659]: pgmap v1107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:47 smithi132 ceph-mon[37580]: pgmap v1107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:47 smithi112 ceph-mon[37501]: pgmap v1107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:49 smithi107 ceph-mon[32659]: pgmap v1108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:49 smithi132 ceph-mon[37580]: pgmap v1108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:49 smithi112 ceph-mon[37501]: pgmap v1108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:08:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1066.766095 secs 2024-07-26T13:08:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:51 smithi107 ceph-mon[32659]: pgmap v1109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1066.766095 secs 2024-07-26T13:08:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:51 smithi132 ceph-mon[37580]: pgmap v1109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1066.766095 secs 2024-07-26T13:08:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:51 smithi112 ceph-mon[37501]: pgmap v1109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:52.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:52.930+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:52.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:52.930+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:52.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:08:52.930+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:53 smithi107 ceph-mon[32659]: pgmap v1110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:53 smithi132 ceph-mon[37580]: pgmap v1110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:53 smithi112 ceph-mon[37501]: pgmap v1110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:55.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:55.056+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:55.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:55.056+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:08:55.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:08:55.056+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:55 smithi107 ceph-mon[32659]: pgmap v1111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:08:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:55 smithi132 ceph-mon[37580]: pgmap v1111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:08:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:55 smithi112 ceph-mon[37501]: pgmap v1111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:08:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1071.766206 secs 2024-07-26T13:08:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:56 smithi132 ceph-mon[37580]: pgmap v1112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:08:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:08:56.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1071.766206 secs 2024-07-26T13:08:56.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:56.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:56.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:56 smithi112 ceph-mon[37501]: pgmap v1112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:08:56.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:08:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1071.766206 secs 2024-07-26T13:08:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:56 smithi107 ceph-mon[32659]: pgmap v1112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:08:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:08:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:08:58 smithi112 ceph-mon[37501]: pgmap v1113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:08:58 smithi107 ceph-mon[32659]: pgmap v1113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:08:58 smithi132 ceph-mon[37580]: pgmap v1113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:09:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:00 smithi112 ceph-mon[37501]: pgmap v1114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:09:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1076.766301 secs 2024-07-26T13:09:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:00 smithi107 ceph-mon[32659]: pgmap v1114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:09:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1076.766301 secs 2024-07-26T13:09:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:00 smithi132 ceph-mon[37580]: pgmap v1114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:09:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1076.766301 secs 2024-07-26T13:09:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:02 smithi112 ceph-mon[37501]: pgmap v1115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:02 smithi107 ceph-mon[32659]: pgmap v1115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:02 smithi132 ceph-mon[37580]: pgmap v1115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:04 smithi112 ceph-mon[37501]: pgmap v1116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:04 smithi107 ceph-mon[32659]: pgmap v1116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:04 smithi132 ceph-mon[37580]: pgmap v1116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:09:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1081.766422 secs 2024-07-26T13:09:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1081.766422 secs 2024-07-26T13:09:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1081.766422 secs 2024-07-26T13:09:06.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:06 smithi107 ceph-mon[32659]: pgmap v1117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:09:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:06 smithi112 ceph-mon[37501]: pgmap v1117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:09:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:06 smithi132 ceph-mon[37580]: pgmap v1117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:09:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:07.945+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:07.945+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:07.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:07.945+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:08 smithi112 ceph-mon[37501]: pgmap v1118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:08 smithi107 ceph-mon[32659]: pgmap v1118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:08 smithi132 ceph-mon[37580]: pgmap v1118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:10.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:10.072+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:10.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:10.072+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:10.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:10.072+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:10 smithi112 ceph-mon[37501]: pgmap v1119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1086.766563 secs 2024-07-26T13:09:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:10 smithi107 ceph-mon[32659]: pgmap v1119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1086.766563 secs 2024-07-26T13:09:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:10 smithi132 ceph-mon[37580]: pgmap v1119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1086.766563 secs 2024-07-26T13:09:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:12 smithi112 ceph-mon[37501]: pgmap v1120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:12 smithi107 ceph-mon[32659]: pgmap v1120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:12 smithi132 ceph-mon[37580]: pgmap v1120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:12.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:12.820+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4003690 con 0x565421993390 2024-07-26T13:09:12.822 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:12.821+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4003690 con 0x565421993390 2024-07-26T13:09:12.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:12.878+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d8003690 con 0x7ff104102930 2024-07-26T13:09:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:12.879+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d8003690 con 0x7ff104102930 2024-07-26T13:09:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:13.929+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44003690 con 0x55aecd6b64d0 2024-07-26T13:09:13.929 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:13.929+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:09:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:13.929+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:09:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:13.929+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44003690 con 0x55aecd6b64d0 2024-07-26T13:09:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:14 smithi112 ceph-mon[37501]: pgmap v1121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:14 smithi107 ceph-mon[32659]: pgmap v1121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:14 smithi132 ceph-mon[37580]: pgmap v1121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1091.766695 secs 2024-07-26T13:09:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1091.766695 secs 2024-07-26T13:09:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1091.766695 secs 2024-07-26T13:09:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:16 smithi112 ceph-mon[37501]: pgmap v1122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:16 smithi107 ceph-mon[32659]: pgmap v1122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:16 smithi132 ceph-mon[37580]: pgmap v1122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:18 smithi112 ceph-mon[37501]: pgmap v1123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:18 smithi107 ceph-mon[32659]: pgmap v1123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:18 smithi132 ceph-mon[37580]: pgmap v1123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:20 smithi112 ceph-mon[37501]: pgmap v1124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1096.766774 secs 2024-07-26T13:09:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:20 smithi107 ceph-mon[32659]: pgmap v1124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1096.766774 secs 2024-07-26T13:09:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:20 smithi132 ceph-mon[37580]: pgmap v1124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1096.766774 secs 2024-07-26T13:09:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:22 smithi112 ceph-mon[37501]: pgmap v1125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:22 smithi107 ceph-mon[32659]: pgmap v1125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:22 smithi132 ceph-mon[37580]: pgmap v1125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:22.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:22.960+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:22.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:22.960+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:22.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:22.960+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:24 smithi107 ceph-mon[32659]: pgmap v1126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:24.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:24 smithi132 ceph-mon[37580]: pgmap v1126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:24 smithi112 ceph-mon[37501]: pgmap v1126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:25.073+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:25.073+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:25.073+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1101.766898 secs 2024-07-26T13:09:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1101.766898 secs 2024-07-26T13:09:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1101.766898 secs 2024-07-26T13:09:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:26 smithi107 ceph-mon[32659]: pgmap v1127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:26 smithi132 ceph-mon[37580]: pgmap v1127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:26.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:26 smithi112 ceph-mon[37501]: pgmap v1127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:28 smithi107 ceph-mon[32659]: pgmap v1128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:28 smithi132 ceph-mon[37580]: pgmap v1128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:28 smithi112 ceph-mon[37501]: pgmap v1128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:30 smithi107 ceph-mon[32659]: pgmap v1129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1106.767019 secs 2024-07-26T13:09:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:30 smithi132 ceph-mon[37580]: pgmap v1129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1106.767019 secs 2024-07-26T13:09:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:30 smithi112 ceph-mon[37501]: pgmap v1129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1106.767019 secs 2024-07-26T13:09:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:32 smithi107 ceph-mon[32659]: pgmap v1130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:32 smithi132 ceph-mon[37580]: pgmap v1130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:32 smithi112 ceph-mon[37501]: pgmap v1130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:34 smithi107 ceph-mon[32659]: pgmap v1131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:34 smithi132 ceph-mon[37580]: pgmap v1131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:34 smithi112 ceph-mon[37501]: pgmap v1131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1111.767143 secs 2024-07-26T13:09:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1111.767143 secs 2024-07-26T13:09:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1111.767143 secs 2024-07-26T13:09:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:36 smithi107 ceph-mon[32659]: pgmap v1132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:36 smithi132 ceph-mon[37580]: pgmap v1132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:36 smithi112 ceph-mon[37501]: pgmap v1132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:37.976+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:37.976+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:37.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:37.976+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:38 smithi107 ceph-mon[32659]: pgmap v1133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:38 smithi132 ceph-mon[37580]: pgmap v1133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:38 smithi112 ceph-mon[37501]: pgmap v1133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:40.090 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:40.089+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:40.089+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:40.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:40.089+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:40 smithi107 ceph-mon[32659]: pgmap v1134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1116.767263 secs 2024-07-26T13:09:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:40 smithi132 ceph-mon[37580]: pgmap v1134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1116.767263 secs 2024-07-26T13:09:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:40 smithi112 ceph-mon[37501]: pgmap v1134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1116.767263 secs 2024-07-26T13:09:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:42 smithi107 ceph-mon[32659]: pgmap v1135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:42 smithi132 ceph-mon[37580]: pgmap v1135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:42 smithi112 ceph-mon[37501]: pgmap v1135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:44 smithi107 ceph-mon[32659]: pgmap v1136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:44 smithi132 ceph-mon[37580]: pgmap v1136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:44 smithi112 ceph-mon[37501]: pgmap v1136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:09:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1121.767409 secs 2024-07-26T13:09:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1121.767409 secs 2024-07-26T13:09:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1121.767409 secs 2024-07-26T13:09:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:46 smithi107 ceph-mon[32659]: pgmap v1137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:46 smithi132 ceph-mon[37580]: pgmap v1137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:46 smithi112 ceph-mon[37501]: pgmap v1137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:48 smithi132 ceph-mon[37580]: pgmap v1138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:48 smithi112 ceph-mon[37501]: pgmap v1138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:48 smithi107 ceph-mon[32659]: pgmap v1138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:50 smithi132 ceph-mon[37580]: pgmap v1139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1126.767526 secs 2024-07-26T13:09:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:50 smithi112 ceph-mon[37501]: pgmap v1139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1126.767526 secs 2024-07-26T13:09:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:50 smithi107 ceph-mon[32659]: pgmap v1139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:09:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1126.767526 secs 2024-07-26T13:09:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:52 smithi132 ceph-mon[37580]: pgmap v1140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:52 smithi112 ceph-mon[37501]: pgmap v1140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:52.992+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:52.992+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:52.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:09:52.992+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:52.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:52 smithi107 ceph-mon[32659]: pgmap v1140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:54 smithi112 ceph-mon[37501]: pgmap v1141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:54 smithi107 ceph-mon[32659]: pgmap v1141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:54 smithi132 ceph-mon[37580]: pgmap v1141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:55.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:55.105+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:55.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:55.105+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:55.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:09:55.105+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:09:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1131.767642 secs 2024-07-26T13:09:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1131.767642 secs 2024-07-26T13:09:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1131.767642 secs 2024-07-26T13:09:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:56.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:56 smithi112 ceph-mon[37501]: pgmap v1142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:56.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:56.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:56.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:56 smithi107 ceph-mon[32659]: pgmap v1142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:56 smithi132 ceph-mon[37580]: pgmap v1142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:09:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:09:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:57 smithi112 ceph-mon[37501]: pgmap v1143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:09:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:57 smithi112 ceph-mon[37501]: pgmap v1144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:57 smithi107 ceph-mon[32659]: pgmap v1143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:09:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:57 smithi107 ceph-mon[32659]: pgmap v1144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:57 smithi132 ceph-mon[37580]: pgmap v1143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:09:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:57 smithi132 ceph-mon[37580]: pgmap v1144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:09:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:09:59 smithi112 ceph-mon[37501]: pgmap v1145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:10:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:09:59 smithi107 ceph-mon[32659]: pgmap v1145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:10:00.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:09:59 smithi132 ceph-mon[37580]: pgmap v1145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:10:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:00 smithi112 ceph-mon[37501]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:00 smithi112 ceph-mon[37501]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T13:10:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:00 smithi112 ceph-mon[37501]: daemon mds.d on smithi107 is in error state 2024-07-26T13:10:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:00 smithi112 ceph-mon[37501]: daemon mds.a on smithi107 is in error state 2024-07-26T13:10:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:00 smithi112 ceph-mon[37501]: daemon mds.h on smithi112 is in error state 2024-07-26T13:10:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:00 smithi112 ceph-mon[37501]: daemon mds.b on smithi112 is in error state 2024-07-26T13:10:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:00 smithi112 ceph-mon[37501]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T13:10:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:00 smithi112 ceph-mon[37501]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2024-07-26T13:10:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1136.767762 secs 2024-07-26T13:10:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:00 smithi107 ceph-mon[32659]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:00 smithi107 ceph-mon[32659]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T13:10:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:00 smithi107 ceph-mon[32659]: daemon mds.d on smithi107 is in error state 2024-07-26T13:10:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:00 smithi107 ceph-mon[32659]: daemon mds.a on smithi107 is in error state 2024-07-26T13:10:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:00 smithi107 ceph-mon[32659]: daemon mds.h on smithi112 is in error state 2024-07-26T13:10:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:00 smithi107 ceph-mon[32659]: daemon mds.b on smithi112 is in error state 2024-07-26T13:10:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:00 smithi107 ceph-mon[32659]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T13:10:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:00 smithi107 ceph-mon[32659]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2024-07-26T13:10:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1136.767762 secs 2024-07-26T13:10:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:00 smithi132 ceph-mon[37580]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:10:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:00 smithi132 ceph-mon[37580]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T13:10:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:00 smithi132 ceph-mon[37580]: daemon mds.d on smithi107 is in error state 2024-07-26T13:10:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:00 smithi132 ceph-mon[37580]: daemon mds.a on smithi107 is in error state 2024-07-26T13:10:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:00 smithi132 ceph-mon[37580]: daemon mds.h on smithi112 is in error state 2024-07-26T13:10:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:00 smithi132 ceph-mon[37580]: daemon mds.b on smithi112 is in error state 2024-07-26T13:10:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:00 smithi132 ceph-mon[37580]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-07-26T13:10:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:00 smithi132 ceph-mon[37580]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2024-07-26T13:10:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1136.767762 secs 2024-07-26T13:10:01.946 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:01 smithi107 ceph-mon[32659]: pgmap v1146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:01 smithi112 ceph-mon[37501]: pgmap v1146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:01 smithi132 ceph-mon[37580]: pgmap v1146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:03 smithi112 ceph-mon[37501]: pgmap v1147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:10:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:03 smithi107 ceph-mon[32659]: pgmap v1147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:10:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:03 smithi132 ceph-mon[37580]: pgmap v1147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:10:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:05 smithi112 ceph-mon[37501]: pgmap v1148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1141.767856 secs 2024-07-26T13:10:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:05 smithi107 ceph-mon[32659]: pgmap v1148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1141.767856 secs 2024-07-26T13:10:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:05 smithi132 ceph-mon[37580]: pgmap v1148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1141.767856 secs 2024-07-26T13:10:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:07 smithi112 ceph-mon[37501]: pgmap v1149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:08.009 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:07 smithi107 ceph-mon[32659]: pgmap v1149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:08.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:08.008+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:08.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:08.008+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:08.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:08.008+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:07 smithi132 ceph-mon[37580]: pgmap v1149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:09 smithi112 ceph-mon[37501]: pgmap v1150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:10.022 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:09 smithi132 ceph-mon[37580]: pgmap v1150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:09 smithi107 ceph-mon[32659]: pgmap v1150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:10.115+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:10.115+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:10.116 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:10.115+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1146.767984 secs 2024-07-26T13:10:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1146.767984 secs 2024-07-26T13:10:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1146.767984 secs 2024-07-26T13:10:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:11 smithi112 ceph-mon[37501]: pgmap v1151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:11 smithi107 ceph-mon[32659]: pgmap v1151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:11 smithi132 ceph-mon[37580]: pgmap v1151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:13 smithi112 ceph-mon[37501]: pgmap v1152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:13 smithi107 ceph-mon[32659]: pgmap v1152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:13 smithi132 ceph-mon[37580]: pgmap v1152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:15 smithi112 ceph-mon[37501]: pgmap v1153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1151.768099 secs 2024-07-26T13:10:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:15 smithi107 ceph-mon[32659]: pgmap v1153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1151.768099 secs 2024-07-26T13:10:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:15 smithi132 ceph-mon[37580]: pgmap v1153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1151.768099 secs 2024-07-26T13:10:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:17 smithi112 ceph-mon[37501]: pgmap v1154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:17 smithi107 ceph-mon[32659]: pgmap v1154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:17 smithi132 ceph-mon[37580]: pgmap v1154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:19 smithi112 ceph-mon[37501]: pgmap v1155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:19 smithi107 ceph-mon[32659]: pgmap v1155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:19 smithi132 ceph-mon[37580]: pgmap v1155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1156.768216 secs 2024-07-26T13:10:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1156.768216 secs 2024-07-26T13:10:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1156.768216 secs 2024-07-26T13:10:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:21 smithi112 ceph-mon[37501]: pgmap v1156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:21 smithi107 ceph-mon[32659]: pgmap v1156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:21 smithi132 ceph-mon[37580]: pgmap v1156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:23.024+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:23.024+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:23.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:23.024+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:23 smithi112 ceph-mon[37501]: pgmap v1157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:23 smithi107 ceph-mon[32659]: pgmap v1157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:23 smithi132 ceph-mon[37580]: pgmap v1157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:25.130+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:25.130+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:25.132 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:25.130+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:25 smithi112 ceph-mon[37501]: pgmap v1158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1161.768279 secs 2024-07-26T13:10:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:25 smithi107 ceph-mon[32659]: pgmap v1158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1161.768279 secs 2024-07-26T13:10:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:25 smithi132 ceph-mon[37580]: pgmap v1158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1161.768279 secs 2024-07-26T13:10:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:27 smithi112 ceph-mon[37501]: pgmap v1159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:27 smithi107 ceph-mon[32659]: pgmap v1159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:27 smithi132 ceph-mon[37580]: pgmap v1159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:29 smithi112 ceph-mon[37501]: pgmap v1160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:29 smithi107 ceph-mon[32659]: pgmap v1160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:29 smithi132 ceph-mon[37580]: pgmap v1160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1166.768363 secs 2024-07-26T13:10:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1166.768363 secs 2024-07-26T13:10:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1166.768363 secs 2024-07-26T13:10:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:31 smithi107 ceph-mon[32659]: pgmap v1161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:31 smithi132 ceph-mon[37580]: pgmap v1161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:31 smithi112 ceph-mon[37501]: pgmap v1161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:33 smithi107 ceph-mon[32659]: pgmap v1162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:33 smithi132 ceph-mon[37580]: pgmap v1162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:33 smithi112 ceph-mon[37501]: pgmap v1162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:35 smithi107 ceph-mon[32659]: pgmap v1163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1171.768497 secs 2024-07-26T13:10:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:35 smithi132 ceph-mon[37580]: pgmap v1163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1171.768497 secs 2024-07-26T13:10:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:35 smithi112 ceph-mon[37501]: pgmap v1163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1171.768497 secs 2024-07-26T13:10:38.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:38.032+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:38.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:38.032+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:38.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:38.032+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:38.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:37 smithi107 ceph-mon[32659]: pgmap v1164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:37 smithi132 ceph-mon[37580]: pgmap v1164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:37 smithi112 ceph-mon[37501]: pgmap v1164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:39 smithi107 ceph-mon[32659]: pgmap v1165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:39 smithi132 ceph-mon[37580]: pgmap v1165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:40.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:40.146+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:40.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:40.146+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:40.148 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:40.146+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:39 smithi112 ceph-mon[37501]: pgmap v1165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1176.768594 secs 2024-07-26T13:10:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1176.768594 secs 2024-07-26T13:10:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1176.768594 secs 2024-07-26T13:10:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:41 smithi107 ceph-mon[32659]: pgmap v1166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:41 smithi132 ceph-mon[37580]: pgmap v1166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:41 smithi112 ceph-mon[37501]: pgmap v1166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:43 smithi107 ceph-mon[32659]: pgmap v1167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:43 smithi132 ceph-mon[37580]: pgmap v1167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:43 smithi112 ceph-mon[37501]: pgmap v1167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:45 smithi107 ceph-mon[32659]: pgmap v1168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1181.768707 secs 2024-07-26T13:10:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:45 smithi132 ceph-mon[37580]: pgmap v1168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1181.768707 secs 2024-07-26T13:10:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:45 smithi112 ceph-mon[37501]: pgmap v1168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1181.768707 secs 2024-07-26T13:10:48.026 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:47 smithi107 ceph-mon[32659]: pgmap v1169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:47 smithi132 ceph-mon[37580]: pgmap v1169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:47 smithi112 ceph-mon[37501]: pgmap v1169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:49 smithi107 ceph-mon[32659]: pgmap v1170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:49 smithi132 ceph-mon[37580]: pgmap v1170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:49 smithi112 ceph-mon[37501]: pgmap v1170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1186.768826 secs 2024-07-26T13:10:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1186.768826 secs 2024-07-26T13:10:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1186.768826 secs 2024-07-26T13:10:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:51 smithi107 ceph-mon[32659]: pgmap v1171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:51 smithi132 ceph-mon[37580]: pgmap v1171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:51 smithi112 ceph-mon[37501]: pgmap v1171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:53.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:53.048+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:53.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:53.048+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:53.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:10:53.048+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:53 smithi132 ceph-mon[37580]: pgmap v1172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:53 smithi112 ceph-mon[37501]: pgmap v1172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:53 smithi107 ceph-mon[32659]: pgmap v1172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:55.154+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:55.154+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:10:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:10:55.154+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:55 smithi132 ceph-mon[37580]: pgmap v1173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1191.768948 secs 2024-07-26T13:10:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:55 smithi112 ceph-mon[37501]: pgmap v1173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1191.768948 secs 2024-07-26T13:10:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:55 smithi107 ceph-mon[32659]: pgmap v1173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:10:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1191.768948 secs 2024-07-26T13:10:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:57.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:57 smithi112 ceph-mon[37501]: pgmap v1174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:57 smithi107 ceph-mon[32659]: pgmap v1174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:57 smithi132 ceph-mon[37580]: pgmap v1174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:10:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:58 smithi112 ceph-mon[37501]: pgmap v1175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:10:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:10:58 smithi112 ceph-mon[37501]: pgmap v1176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:10:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:58 smithi107 ceph-mon[32659]: pgmap v1175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:10:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:10:58 smithi107 ceph-mon[32659]: pgmap v1176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:10:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:58 smithi132 ceph-mon[37580]: pgmap v1175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:10:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:10:58 smithi132 ceph-mon[37580]: pgmap v1176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:11:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:00 smithi112 ceph-mon[37501]: pgmap v1177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:11:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1196.769080 secs 2024-07-26T13:11:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:00 smithi107 ceph-mon[32659]: pgmap v1177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:11:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1196.769080 secs 2024-07-26T13:11:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:00 smithi132 ceph-mon[37580]: pgmap v1177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:11:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1196.769080 secs 2024-07-26T13:11:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:02 smithi112 ceph-mon[37501]: pgmap v1178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:11:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:02 smithi107 ceph-mon[32659]: pgmap v1178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:11:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:02 smithi132 ceph-mon[37580]: pgmap v1178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:11:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:04 smithi112 ceph-mon[37501]: pgmap v1179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:11:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:04 smithi107 ceph-mon[32659]: pgmap v1179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:11:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:04 smithi132 ceph-mon[37580]: pgmap v1179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:11:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1201.769238 secs 2024-07-26T13:11:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1201.769238 secs 2024-07-26T13:11:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1201.769238 secs 2024-07-26T13:11:07.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:06 smithi107 ceph-mon[32659]: pgmap v1180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:11:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:06 smithi112 ceph-mon[37501]: pgmap v1180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:11:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:06 smithi132 ceph-mon[37580]: pgmap v1180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:11:08.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:08.050+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:08.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:08.050+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:08.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:08.050+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:08 smithi112 ceph-mon[37501]: pgmap v1181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:08 smithi107 ceph-mon[32659]: pgmap v1181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:08 smithi132 ceph-mon[37580]: pgmap v1181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:10.170+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:10.170+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:10.170+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:10 smithi112 ceph-mon[37501]: pgmap v1182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1206.769363 secs 2024-07-26T13:11:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:10 smithi107 ceph-mon[32659]: pgmap v1182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1206.769363 secs 2024-07-26T13:11:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:10 smithi132 ceph-mon[37580]: pgmap v1182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1206.769363 secs 2024-07-26T13:11:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:12 smithi112 ceph-mon[37501]: pgmap v1183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:11:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:12 smithi107 ceph-mon[32659]: pgmap v1183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:11:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:12 smithi132 ceph-mon[37580]: pgmap v1183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:11:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:14 smithi112 ceph-mon[37501]: pgmap v1184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:14 smithi107 ceph-mon[32659]: pgmap v1184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:14 smithi132 ceph-mon[37580]: pgmap v1184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1211.769471 secs 2024-07-26T13:11:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1211.769471 secs 2024-07-26T13:11:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1211.769471 secs 2024-07-26T13:11:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:16 smithi112 ceph-mon[37501]: pgmap v1185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:16 smithi107 ceph-mon[32659]: pgmap v1185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:16 smithi132 ceph-mon[37580]: pgmap v1185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:18 smithi112 ceph-mon[37501]: pgmap v1186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:18 smithi107 ceph-mon[32659]: pgmap v1186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:18 smithi132 ceph-mon[37580]: pgmap v1186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:20 smithi112 ceph-mon[37501]: pgmap v1187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1216.769534 secs 2024-07-26T13:11:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:20 smithi107 ceph-mon[32659]: pgmap v1187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1216.769534 secs 2024-07-26T13:11:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:20 smithi132 ceph-mon[37580]: pgmap v1187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1216.769534 secs 2024-07-26T13:11:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:23.056+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:23.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:23.056+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:23.058 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:23.056+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:22 smithi112 ceph-mon[37501]: pgmap v1188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:11:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:22 smithi107 ceph-mon[32659]: pgmap v1188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:11:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:22 smithi132 ceph-mon[37580]: pgmap v1188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:11:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:25.177+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:25.177+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:25.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:25.177+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:24 smithi112 ceph-mon[37501]: pgmap v1189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:11:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:24 smithi107 ceph-mon[32659]: pgmap v1189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:11:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:24 smithi132 ceph-mon[37580]: pgmap v1189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:11:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1221.769648 secs 2024-07-26T13:11:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1221.769648 secs 2024-07-26T13:11:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1221.769648 secs 2024-07-26T13:11:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:26 smithi112 ceph-mon[37501]: pgmap v1190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:11:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:26 smithi107 ceph-mon[32659]: pgmap v1190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:11:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:26 smithi132 ceph-mon[37580]: pgmap v1190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:11:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:28 smithi112 ceph-mon[37501]: pgmap v1191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:28 smithi107 ceph-mon[32659]: pgmap v1191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:28 smithi132 ceph-mon[37580]: pgmap v1191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:30 smithi112 ceph-mon[37501]: pgmap v1192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1226.769760 secs 2024-07-26T13:11:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:30 smithi107 ceph-mon[32659]: pgmap v1192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1226.769760 secs 2024-07-26T13:11:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:30 smithi132 ceph-mon[37580]: pgmap v1192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1226.769760 secs 2024-07-26T13:11:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:32 smithi107 ceph-mon[32659]: pgmap v1193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:11:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:32 smithi132 ceph-mon[37580]: pgmap v1193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:11:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:32 smithi112 ceph-mon[37501]: pgmap v1193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:11:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:34 smithi107 ceph-mon[32659]: pgmap v1194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:34 smithi132 ceph-mon[37580]: pgmap v1194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:34 smithi112 ceph-mon[37501]: pgmap v1194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1231.769853 secs 2024-07-26T13:11:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1231.769853 secs 2024-07-26T13:11:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1231.769853 secs 2024-07-26T13:11:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:36 smithi107 ceph-mon[32659]: pgmap v1195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:11:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:36 smithi132 ceph-mon[37580]: pgmap v1195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:11:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:36 smithi112 ceph-mon[37501]: pgmap v1195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:11:38.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:38.071+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:38.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:38.071+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:38.073 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:38.071+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:38 smithi107 ceph-mon[32659]: pgmap v1196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:38 smithi132 ceph-mon[37580]: pgmap v1196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:38 smithi112 ceph-mon[37501]: pgmap v1196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:11:40.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:40.193+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:40.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:40.193+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:40.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:40.193+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:40 smithi107 ceph-mon[32659]: pgmap v1197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1236.770011 secs 2024-07-26T13:11:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:40 smithi132 ceph-mon[37580]: pgmap v1197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1236.770011 secs 2024-07-26T13:11:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:40 smithi112 ceph-mon[37501]: pgmap v1197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1236.770011 secs 2024-07-26T13:11:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:42 smithi107 ceph-mon[32659]: pgmap v1198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:42 smithi132 ceph-mon[37580]: pgmap v1198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:42 smithi112 ceph-mon[37501]: pgmap v1198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:44 smithi107 ceph-mon[32659]: pgmap v1199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:44 smithi132 ceph-mon[37580]: pgmap v1199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:44 smithi112 ceph-mon[37501]: pgmap v1199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1241.770109 secs 2024-07-26T13:11:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1241.770109 secs 2024-07-26T13:11:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1241.770109 secs 2024-07-26T13:11:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:47 smithi107 ceph-mon[32659]: pgmap v1200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:47 smithi132 ceph-mon[37580]: pgmap v1200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:47 smithi112 ceph-mon[37501]: pgmap v1200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:49 smithi107 ceph-mon[32659]: pgmap v1201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:49 smithi132 ceph-mon[37580]: pgmap v1201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:49 smithi112 ceph-mon[37501]: pgmap v1201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:51 smithi107 ceph-mon[32659]: pgmap v1202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1246.770232 secs 2024-07-26T13:11:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:51 smithi132 ceph-mon[37580]: pgmap v1202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1246.770232 secs 2024-07-26T13:11:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:51 smithi112 ceph-mon[37501]: pgmap v1202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1246.770232 secs 2024-07-26T13:11:53.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:53.084+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:53.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:53.084+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:53.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:11:53.084+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:53 smithi107 ceph-mon[32659]: pgmap v1203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:53 smithi132 ceph-mon[37580]: pgmap v1203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:53 smithi112 ceph-mon[37501]: pgmap v1203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:55.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:55.209+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:55.209+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:55.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:11:55.209+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:11:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:55 smithi107 ceph-mon[32659]: pgmap v1204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:55 smithi132 ceph-mon[37580]: pgmap v1204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:55 smithi112 ceph-mon[37501]: pgmap v1204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1251.770371 secs 2024-07-26T13:11:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1251.770371 secs 2024-07-26T13:11:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1251.770371 secs 2024-07-26T13:11:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:57 smithi132 ceph-mon[37580]: pgmap v1205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:57.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:57 smithi112 ceph-mon[37501]: pgmap v1205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:57 smithi107 ceph-mon[32659]: pgmap v1205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:11:59 smithi132 ceph-mon[37580]: pgmap v1206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:11:59 smithi112 ceph-mon[37501]: pgmap v1206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:59.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:11:59 smithi107 ceph-mon[32659]: pgmap v1206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:01 smithi112 ceph-mon[37501]: pgmap v1207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1256.770499 secs 2024-07-26T13:12:01.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:01 smithi107 ceph-mon[32659]: pgmap v1207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:01.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1256.770499 secs 2024-07-26T13:12:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:01 smithi132 ceph-mon[37580]: pgmap v1207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1256.770499 secs 2024-07-26T13:12:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:02 smithi107 ceph-mon[32659]: pgmap v1208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:02 smithi132 ceph-mon[37580]: pgmap v1208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:02 smithi112 ceph-mon[37501]: pgmap v1208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: pgmap v1209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:05.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: pgmap v1209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: pgmap v1209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:12:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:05 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:12:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:05 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:12:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:05 smithi107 ceph-mon[32659]: pgmap v1210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:05 smithi107 ceph-mon[32659]: pgmap v1211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T13:12:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1261.770619 secs 2024-07-26T13:12:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:05 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:12:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:05 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:12:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:05 smithi132 ceph-mon[37580]: pgmap v1210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:05 smithi132 ceph-mon[37580]: pgmap v1211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T13:12:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1261.770619 secs 2024-07-26T13:12:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:05 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:12:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:05 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:12:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:05 smithi112 ceph-mon[37501]: pgmap v1210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:05 smithi112 ceph-mon[37501]: pgmap v1211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T13:12:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1261.770619 secs 2024-07-26T13:12:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:07 smithi107 ceph-mon[32659]: pgmap v1212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:12:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:07 smithi132 ceph-mon[37580]: pgmap v1212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:12:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:08.099+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:08.099+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:08.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:08.099+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:07 smithi112 ceph-mon[37501]: pgmap v1212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:12:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:09 smithi107 ceph-mon[32659]: pgmap v1213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T13:12:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:09 smithi132 ceph-mon[37580]: pgmap v1213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T13:12:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:09 smithi112 ceph-mon[37501]: pgmap v1213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T13:12:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:10.225+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:10.225+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:10.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:10.225+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1266.770742 secs 2024-07-26T13:12:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1266.770742 secs 2024-07-26T13:12:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1266.770742 secs 2024-07-26T13:12:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:11 smithi132 ceph-mon[37580]: pgmap v1214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:11 smithi112 ceph-mon[37501]: pgmap v1214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:11 smithi107 ceph-mon[32659]: pgmap v1214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:12.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:12.851+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:12:12.853 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:12.852+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:12:12.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:12.906+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d8003690 con 0x7ff104102930 2024-07-26T13:12:12.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:12.907+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d8003690 con 0x7ff104102930 2024-07-26T13:12:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:13.960+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:12:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:13.960+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44003690 con 0x55aecd6b64d0 2024-07-26T13:12:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:13.961+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:12:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:13.961+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44003690 con 0x55aecd6b64d0 2024-07-26T13:12:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:13 smithi112 ceph-mon[37501]: pgmap v1215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:13 smithi107 ceph-mon[32659]: pgmap v1215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:13 smithi132 ceph-mon[37580]: pgmap v1215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:15 smithi112 ceph-mon[37501]: pgmap v1216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1271.770859 secs 2024-07-26T13:12:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:15 smithi107 ceph-mon[32659]: pgmap v1216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1271.770859 secs 2024-07-26T13:12:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:15 smithi132 ceph-mon[37580]: pgmap v1216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:12:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1271.770859 secs 2024-07-26T13:12:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:17 smithi112 ceph-mon[37501]: pgmap v1217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:12:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:17 smithi107 ceph-mon[32659]: pgmap v1217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:12:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:17 smithi132 ceph-mon[37580]: pgmap v1217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:12:20.186 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:19 smithi107 ceph-mon[32659]: pgmap v1218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:19 smithi112 ceph-mon[37501]: pgmap v1218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:19 smithi132 ceph-mon[37580]: pgmap v1218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1276.770973 secs 2024-07-26T13:12:21.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1276.770973 secs 2024-07-26T13:12:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1276.770973 secs 2024-07-26T13:12:22.163 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:21 smithi107 ceph-mon[32659]: pgmap v1219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:21 smithi112 ceph-mon[37501]: pgmap v1219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:21 smithi132 ceph-mon[37580]: pgmap v1219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:23.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:23.115+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:23.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:23.115+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:23.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:23.115+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:23 smithi107 ceph-mon[32659]: pgmap v1220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:23 smithi132 ceph-mon[37580]: pgmap v1220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:23 smithi112 ceph-mon[37501]: pgmap v1220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:25.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:25.238+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:25.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:25.238+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:25.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:25.238+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:25 smithi112 ceph-mon[37501]: pgmap v1221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1281.771075 secs 2024-07-26T13:12:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:25 smithi107 ceph-mon[32659]: pgmap v1221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1281.771075 secs 2024-07-26T13:12:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:25 smithi132 ceph-mon[37580]: pgmap v1221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1281.771075 secs 2024-07-26T13:12:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:27 smithi112 ceph-mon[37501]: pgmap v1222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:27 smithi107 ceph-mon[32659]: pgmap v1222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:27 smithi132 ceph-mon[37580]: pgmap v1222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:29 smithi112 ceph-mon[37501]: pgmap v1223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:29 smithi107 ceph-mon[32659]: pgmap v1223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:29 smithi132 ceph-mon[37580]: pgmap v1223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1286.771190 secs 2024-07-26T13:12:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1286.771190 secs 2024-07-26T13:12:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1286.771190 secs 2024-07-26T13:12:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:31 smithi112 ceph-mon[37501]: pgmap v1224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:31 smithi107 ceph-mon[32659]: pgmap v1224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:31 smithi132 ceph-mon[37580]: pgmap v1224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:33 smithi112 ceph-mon[37501]: pgmap v1225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:33 smithi107 ceph-mon[32659]: pgmap v1225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:33 smithi132 ceph-mon[37580]: pgmap v1225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:35 smithi112 ceph-mon[37501]: pgmap v1226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1291.771279 secs 2024-07-26T13:12:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:35 smithi107 ceph-mon[32659]: pgmap v1226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1291.771279 secs 2024-07-26T13:12:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:35 smithi132 ceph-mon[37580]: pgmap v1226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1291.771279 secs 2024-07-26T13:12:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:37 smithi112 ceph-mon[37501]: pgmap v1227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:37 smithi107 ceph-mon[32659]: pgmap v1227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:37 smithi132 ceph-mon[37580]: pgmap v1227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:38.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:38.120+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:38.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:38.121+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:38.123 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:38.121+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:39 smithi112 ceph-mon[37501]: pgmap v1228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:39 smithi107 ceph-mon[32659]: pgmap v1228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:39 smithi132 ceph-mon[37580]: pgmap v1228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:40.245+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:40.245+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:40.245+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1296.771360 secs 2024-07-26T13:12:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1296.771360 secs 2024-07-26T13:12:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1296.771360 secs 2024-07-26T13:12:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:41 smithi112 ceph-mon[37501]: pgmap v1229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:41 smithi107 ceph-mon[32659]: pgmap v1229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:41 smithi132 ceph-mon[37580]: pgmap v1229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:43 smithi112 ceph-mon[37501]: pgmap v1230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:43 smithi107 ceph-mon[32659]: pgmap v1230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:43 smithi132 ceph-mon[37580]: pgmap v1230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:45 smithi112 ceph-mon[37501]: pgmap v1231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1301.771458 secs 2024-07-26T13:12:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:45 smithi107 ceph-mon[32659]: pgmap v1231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1301.771458 secs 2024-07-26T13:12:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:45 smithi132 ceph-mon[37580]: pgmap v1231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1301.771458 secs 2024-07-26T13:12:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:47 smithi112 ceph-mon[37501]: pgmap v1232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:47 smithi107 ceph-mon[32659]: pgmap v1232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:47 smithi132 ceph-mon[37580]: pgmap v1232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:49 smithi112 ceph-mon[37501]: pgmap v1233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:49 smithi107 ceph-mon[32659]: pgmap v1233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:49 smithi132 ceph-mon[37580]: pgmap v1233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1306.771569 secs 2024-07-26T13:12:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1306.771569 secs 2024-07-26T13:12:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1306.771569 secs 2024-07-26T13:12:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:51 smithi107 ceph-mon[32659]: pgmap v1234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:51 smithi132 ceph-mon[37580]: pgmap v1234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:51 smithi112 ceph-mon[37501]: pgmap v1234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:53.137+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:53.137+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:53.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:12:53.137+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:53 smithi107 ceph-mon[32659]: pgmap v1235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:53 smithi132 ceph-mon[37580]: pgmap v1235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:53 smithi112 ceph-mon[37501]: pgmap v1235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:12:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:55.262+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:55.262+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:12:55.262+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:12:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:55 smithi107 ceph-mon[32659]: pgmap v1236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1311.771684 secs 2024-07-26T13:12:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:55 smithi132 ceph-mon[37580]: pgmap v1236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1311.771684 secs 2024-07-26T13:12:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:55 smithi112 ceph-mon[37501]: pgmap v1236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:12:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1311.771684 secs 2024-07-26T13:12:58.002 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:57 smithi112 ceph-mon[37501]: pgmap v1237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:57 smithi107 ceph-mon[32659]: pgmap v1237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:12:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:57 smithi132 ceph-mon[37580]: pgmap v1237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:12:59 smithi107 ceph-mon[32659]: pgmap v1238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:12:59 smithi132 ceph-mon[37580]: pgmap v1238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:12:59 smithi112 ceph-mon[37501]: pgmap v1238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1316.771805 secs 2024-07-26T13:13:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1316.771805 secs 2024-07-26T13:13:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1316.771805 secs 2024-07-26T13:13:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:01 smithi107 ceph-mon[32659]: pgmap v1239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:01 smithi132 ceph-mon[37580]: pgmap v1239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:01 smithi112 ceph-mon[37501]: pgmap v1239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:03 smithi107 ceph-mon[32659]: pgmap v1240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:04.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:03 smithi132 ceph-mon[37580]: pgmap v1240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:03 smithi112 ceph-mon[37501]: pgmap v1240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:05.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: pgmap v1241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:05.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:05.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1321.771900 secs 2024-07-26T13:13:05.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:05.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:05.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:05.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:05.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:05.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:05.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:05.978 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: pgmap v1241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1321.771900 secs 2024-07-26T13:13:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: pgmap v1241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1321.771900 secs 2024-07-26T13:13:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:06 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:13:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:06 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:13:07.044 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:06 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:13:07.044 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:06 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:13:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:06 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:13:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:06 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:13:07.821 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:07 smithi132 ceph-mon[37580]: pgmap v1242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:07 smithi107 ceph-mon[32659]: pgmap v1242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:08.147+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:08.147+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:08.147+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:07 smithi112 ceph-mon[37501]: pgmap v1242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: pgmap v1243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:09.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:13:09.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:13:09.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:09.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:09.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: pgmap v1244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:09.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: pgmap v1245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:13:09.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: pgmap v1243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: pgmap v1244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: pgmap v1245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:13:09.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: pgmap v1243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:13:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:13:09.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:09.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:09.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: pgmap v1244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:09.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: pgmap v1245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T13:13:09.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:13:10.278 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:10.278+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:10.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:10.278+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:10.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:10.278+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1326.772038 secs 2024-07-26T13:13:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1326.772038 secs 2024-07-26T13:13:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1326.772038 secs 2024-07-26T13:13:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:11 smithi112 ceph-mon[37501]: pgmap v1246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:13:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:11 smithi107 ceph-mon[32659]: pgmap v1246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:13:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:11 smithi132 ceph-mon[37580]: pgmap v1246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:13:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:13 smithi107 ceph-mon[32659]: pgmap v1247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:13:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:13 smithi132 ceph-mon[37580]: pgmap v1247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:13:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:13 smithi112 ceph-mon[37501]: pgmap v1247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:13:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:15 smithi112 ceph-mon[37501]: pgmap v1248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-26T13:13:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1331.772167 secs 2024-07-26T13:13:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:15 smithi107 ceph-mon[32659]: pgmap v1248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-26T13:13:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1331.772167 secs 2024-07-26T13:13:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:15 smithi132 ceph-mon[37580]: pgmap v1248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-07-26T13:13:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1331.772167 secs 2024-07-26T13:13:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:17 smithi112 ceph-mon[37501]: pgmap v1249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:13:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:17 smithi107 ceph-mon[32659]: pgmap v1249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:13:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:17 smithi132 ceph-mon[37580]: pgmap v1249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:13:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:19 smithi112 ceph-mon[37501]: pgmap v1250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:19 smithi107 ceph-mon[32659]: pgmap v1250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:19 smithi132 ceph-mon[37580]: pgmap v1250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1336.772256 secs 2024-07-26T13:13:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1336.772256 secs 2024-07-26T13:13:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1336.772256 secs 2024-07-26T13:13:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:21 smithi112 ceph-mon[37501]: pgmap v1251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:21 smithi107 ceph-mon[32659]: pgmap v1251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:21 smithi132 ceph-mon[37580]: pgmap v1251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:23.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:23.159+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:23.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:23.159+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:23.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:23.159+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:23 smithi112 ceph-mon[37501]: pgmap v1252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:23 smithi107 ceph-mon[32659]: pgmap v1252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:23 smithi132 ceph-mon[37580]: pgmap v1252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:25.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:25.293+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:25.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:25.293+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:25.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:25.293+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:25 smithi112 ceph-mon[37501]: pgmap v1253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1341.772367 secs 2024-07-26T13:13:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:25 smithi107 ceph-mon[32659]: pgmap v1253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1341.772367 secs 2024-07-26T13:13:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:25 smithi132 ceph-mon[37580]: pgmap v1253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1341.772367 secs 2024-07-26T13:13:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:27 smithi112 ceph-mon[37501]: pgmap v1254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:27 smithi107 ceph-mon[32659]: pgmap v1254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:27 smithi132 ceph-mon[37580]: pgmap v1254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:29 smithi112 ceph-mon[37501]: pgmap v1255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:29 smithi107 ceph-mon[32659]: pgmap v1255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:29 smithi132 ceph-mon[37580]: pgmap v1255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1346.772463 secs 2024-07-26T13:13:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1346.772463 secs 2024-07-26T13:13:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1346.772463 secs 2024-07-26T13:13:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:31 smithi107 ceph-mon[32659]: pgmap v1256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:31 smithi132 ceph-mon[37580]: pgmap v1256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:31 smithi112 ceph-mon[37501]: pgmap v1256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:33 smithi107 ceph-mon[32659]: pgmap v1257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:33 smithi132 ceph-mon[37580]: pgmap v1257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:33 smithi112 ceph-mon[37501]: pgmap v1257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:35 smithi107 ceph-mon[32659]: pgmap v1258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1351.772533 secs 2024-07-26T13:13:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:35 smithi132 ceph-mon[37580]: pgmap v1258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1351.772533 secs 2024-07-26T13:13:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:35 smithi112 ceph-mon[37501]: pgmap v1258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1351.772533 secs 2024-07-26T13:13:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:37 smithi107 ceph-mon[32659]: pgmap v1259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:38.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:37 smithi132 ceph-mon[37580]: pgmap v1259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:38.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:38.174+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:38.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:38.174+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:38.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:38.174+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:37 smithi112 ceph-mon[37501]: pgmap v1259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:39 smithi107 ceph-mon[32659]: pgmap v1260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:39 smithi132 ceph-mon[37580]: pgmap v1260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:39 smithi112 ceph-mon[37501]: pgmap v1260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:40.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:40.308+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:40.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:40.308+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:40.310 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:40.308+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1356.772658 secs 2024-07-26T13:13:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1356.772658 secs 2024-07-26T13:13:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1356.772658 secs 2024-07-26T13:13:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:41 smithi107 ceph-mon[32659]: pgmap v1261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:41 smithi132 ceph-mon[37580]: pgmap v1261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:41 smithi112 ceph-mon[37501]: pgmap v1261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:43 smithi107 ceph-mon[32659]: pgmap v1262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:43 smithi132 ceph-mon[37580]: pgmap v1262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:44.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:43 smithi112 ceph-mon[37501]: pgmap v1262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:45 smithi107 ceph-mon[32659]: pgmap v1263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1361.772718 secs 2024-07-26T13:13:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:45 smithi132 ceph-mon[37580]: pgmap v1263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1361.772718 secs 2024-07-26T13:13:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:45 smithi112 ceph-mon[37501]: pgmap v1263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1361.772718 secs 2024-07-26T13:13:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:47 smithi107 ceph-mon[32659]: pgmap v1264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:47 smithi132 ceph-mon[37580]: pgmap v1264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:47 smithi112 ceph-mon[37501]: pgmap v1264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:49 smithi107 ceph-mon[32659]: pgmap v1265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:49 smithi132 ceph-mon[37580]: pgmap v1265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:49 smithi112 ceph-mon[37501]: pgmap v1265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1366.772828 secs 2024-07-26T13:13:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1366.772828 secs 2024-07-26T13:13:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1366.772828 secs 2024-07-26T13:13:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:51 smithi107 ceph-mon[32659]: pgmap v1266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:51 smithi132 ceph-mon[37580]: pgmap v1266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:51 smithi112 ceph-mon[37501]: pgmap v1266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:53.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:53.176+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:53.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:53.176+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:53.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:13:53.176+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:53 smithi107 ceph-mon[32659]: pgmap v1267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:53 smithi132 ceph-mon[37580]: pgmap v1267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:54.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:53 smithi112 ceph-mon[37501]: pgmap v1267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:13:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:55.316+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:55.316+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:55.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:13:55.316+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:13:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:55 smithi132 ceph-mon[37580]: pgmap v1268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1371.772961 secs 2024-07-26T13:13:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:55 smithi112 ceph-mon[37501]: pgmap v1268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1371.772961 secs 2024-07-26T13:13:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:55 smithi107 ceph-mon[32659]: pgmap v1268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1371.772961 secs 2024-07-26T13:13:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:57 smithi132 ceph-mon[37580]: pgmap v1269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:57 smithi112 ceph-mon[37501]: pgmap v1269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:13:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:57 smithi107 ceph-mon[32659]: pgmap v1269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:13:59 smithi132 ceph-mon[37580]: pgmap v1270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:13:59 smithi112 ceph-mon[37501]: pgmap v1270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:13:59 smithi107 ceph-mon[32659]: pgmap v1270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1376.773102 secs 2024-07-26T13:14:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1376.773102 secs 2024-07-26T13:14:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1376.773102 secs 2024-07-26T13:14:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:01 smithi112 ceph-mon[37501]: pgmap v1271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:01 smithi107 ceph-mon[32659]: pgmap v1271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:01 smithi132 ceph-mon[37580]: pgmap v1271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:03 smithi112 ceph-mon[37501]: pgmap v1272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:03 smithi107 ceph-mon[32659]: pgmap v1272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:03 smithi132 ceph-mon[37580]: pgmap v1272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:05 smithi112 ceph-mon[37501]: pgmap v1273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1381.773238 secs 2024-07-26T13:14:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:05 smithi107 ceph-mon[32659]: pgmap v1273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1381.773238 secs 2024-07-26T13:14:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:05 smithi132 ceph-mon[37580]: pgmap v1273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1381.773238 secs 2024-07-26T13:14:08.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:07 smithi107 ceph-mon[32659]: pgmap v1274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:08.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:08.186+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:08.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:08.186+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:08.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:08.186+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:07 smithi112 ceph-mon[37501]: pgmap v1274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:07 smithi132 ceph-mon[37580]: pgmap v1274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:09 smithi112 ceph-mon[37501]: pgmap v1275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:10.266 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:09 smithi107 ceph-mon[32659]: pgmap v1275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:10.266 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:10.266 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:10.266 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:09 smithi132 ceph-mon[37580]: pgmap v1275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:14:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:10.321+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:10.321+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:10.321 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:10.321+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:10 smithi112 ceph-mon[37501]: pgmap v1276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:14:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:10 smithi112 ceph-mon[37501]: pgmap v1277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1386.773360 secs 2024-07-26T13:14:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:10 smithi107 ceph-mon[32659]: pgmap v1276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:14:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:10 smithi107 ceph-mon[32659]: pgmap v1277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1386.773360 secs 2024-07-26T13:14:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:10 smithi132 ceph-mon[37580]: pgmap v1276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:14:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:10 smithi132 ceph-mon[37580]: pgmap v1277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1386.773360 secs 2024-07-26T13:14:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:12 smithi112 ceph-mon[37501]: pgmap v1278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:12 smithi107 ceph-mon[32659]: pgmap v1278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:12 smithi132 ceph-mon[37580]: pgmap v1278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:14 smithi112 ceph-mon[37501]: pgmap v1279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:14:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:14 smithi107 ceph-mon[32659]: pgmap v1279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:14:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:14 smithi132 ceph-mon[37580]: pgmap v1279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:14:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1391.773473 secs 2024-07-26T13:14:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1391.773473 secs 2024-07-26T13:14:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1391.773473 secs 2024-07-26T13:14:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:16 smithi112 ceph-mon[37501]: pgmap v1280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:14:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:16 smithi107 ceph-mon[32659]: pgmap v1280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:14:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:16 smithi132 ceph-mon[37580]: pgmap v1280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:14:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:18 smithi112 ceph-mon[37501]: pgmap v1281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:14:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:18 smithi107 ceph-mon[32659]: pgmap v1281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:14:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:18 smithi132 ceph-mon[37580]: pgmap v1281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:14:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:20 smithi112 ceph-mon[37501]: pgmap v1282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:14:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1396.773592 secs 2024-07-26T13:14:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:20 smithi107 ceph-mon[32659]: pgmap v1282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:14:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1396.773592 secs 2024-07-26T13:14:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:20 smithi132 ceph-mon[37580]: pgmap v1282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:14:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1396.773592 secs 2024-07-26T13:14:23.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:23.198+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:23.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:23.198+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:23.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:23.198+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:23.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:22 smithi107 ceph-mon[32659]: pgmap v1283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:23.204 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:22 smithi112 ceph-mon[37501]: pgmap v1283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:22 smithi132 ceph-mon[37580]: pgmap v1283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:24 smithi112 ceph-mon[37501]: pgmap v1284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:24 smithi107 ceph-mon[32659]: pgmap v1284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:24 smithi132 ceph-mon[37580]: pgmap v1284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:25.336+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:25.336+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:25.337 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:25.336+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:26.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1401.773729 secs 2024-07-26T13:14:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1401.773729 secs 2024-07-26T13:14:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1401.773729 secs 2024-07-26T13:14:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:26 smithi112 ceph-mon[37501]: pgmap v1285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:26 smithi107 ceph-mon[32659]: pgmap v1285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:26 smithi132 ceph-mon[37580]: pgmap v1285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:28 smithi112 ceph-mon[37501]: pgmap v1286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:28 smithi107 ceph-mon[32659]: pgmap v1286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:28 smithi132 ceph-mon[37580]: pgmap v1286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:30 smithi112 ceph-mon[37501]: pgmap v1287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1406.773801 secs 2024-07-26T13:14:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:30 smithi107 ceph-mon[32659]: pgmap v1287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1406.773801 secs 2024-07-26T13:14:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:30 smithi132 ceph-mon[37580]: pgmap v1287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1406.773801 secs 2024-07-26T13:14:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:32 smithi112 ceph-mon[37501]: pgmap v1288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:32 smithi107 ceph-mon[32659]: pgmap v1288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:32 smithi132 ceph-mon[37580]: pgmap v1288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:34 smithi112 ceph-mon[37501]: pgmap v1289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:34 smithi107 ceph-mon[32659]: pgmap v1289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:34 smithi132 ceph-mon[37580]: pgmap v1289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1411.773929 secs 2024-07-26T13:14:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1411.773929 secs 2024-07-26T13:14:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1411.773929 secs 2024-07-26T13:14:37.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:36 smithi107 ceph-mon[32659]: pgmap v1290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:36 smithi132 ceph-mon[37580]: pgmap v1290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:36 smithi112 ceph-mon[37501]: pgmap v1290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:38.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:38.212+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:38.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:38.212+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:38.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:38.212+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:38 smithi107 ceph-mon[32659]: pgmap v1291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:38 smithi132 ceph-mon[37580]: pgmap v1291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:38 smithi112 ceph-mon[37501]: pgmap v1291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:40.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:40.351+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:40.351+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:40.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:40.351+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:40 smithi107 ceph-mon[32659]: pgmap v1292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1416.774027 secs 2024-07-26T13:14:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:40 smithi132 ceph-mon[37580]: pgmap v1292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1416.774027 secs 2024-07-26T13:14:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:40 smithi112 ceph-mon[37501]: pgmap v1292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1416.774027 secs 2024-07-26T13:14:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:42 smithi107 ceph-mon[32659]: pgmap v1293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:42 smithi132 ceph-mon[37580]: pgmap v1293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:42 smithi112 ceph-mon[37501]: pgmap v1293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:44 smithi107 ceph-mon[32659]: pgmap v1294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:44 smithi132 ceph-mon[37580]: pgmap v1294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:44 smithi112 ceph-mon[37501]: pgmap v1294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1421.774152 secs 2024-07-26T13:14:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1421.774152 secs 2024-07-26T13:14:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1421.774152 secs 2024-07-26T13:14:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:47 smithi107 ceph-mon[32659]: pgmap v1295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:47 smithi132 ceph-mon[37580]: pgmap v1295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:47 smithi112 ceph-mon[37501]: pgmap v1295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:49 smithi107 ceph-mon[32659]: pgmap v1296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:49 smithi132 ceph-mon[37580]: pgmap v1296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:49 smithi112 ceph-mon[37501]: pgmap v1296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:51 smithi107 ceph-mon[32659]: pgmap v1297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1426.774234 secs 2024-07-26T13:14:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:51 smithi132 ceph-mon[37580]: pgmap v1297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1426.774234 secs 2024-07-26T13:14:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:51 smithi112 ceph-mon[37501]: pgmap v1297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1426.774234 secs 2024-07-26T13:14:53.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:53.217+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:53.217+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:53.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:14:53.217+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:53 smithi107 ceph-mon[32659]: pgmap v1298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:53.337 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:53 smithi132 ceph-mon[37580]: pgmap v1298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:53 smithi112 ceph-mon[37501]: pgmap v1298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:14:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:55 smithi107 ceph-mon[32659]: pgmap v1299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:55 smithi132 ceph-mon[37580]: pgmap v1299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:55.367+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:55.367+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:14:55.368 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:14:55.367+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:55 smithi112 ceph-mon[37501]: pgmap v1299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1431.774350 secs 2024-07-26T13:14:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1431.774350 secs 2024-07-26T13:14:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1431.774350 secs 2024-07-26T13:14:57.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:57 smithi107 ceph-mon[32659]: pgmap v1300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:57 smithi132 ceph-mon[37580]: pgmap v1300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:57 smithi112 ceph-mon[37501]: pgmap v1300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:14:59 smithi132 ceph-mon[37580]: pgmap v1301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:14:59 smithi112 ceph-mon[37501]: pgmap v1301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:14:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:14:59 smithi107 ceph-mon[32659]: pgmap v1301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:01 smithi112 ceph-mon[37501]: pgmap v1302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1436.774475 secs 2024-07-26T13:15:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:01 smithi107 ceph-mon[32659]: pgmap v1302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1436.774475 secs 2024-07-26T13:15:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:01 smithi132 ceph-mon[37580]: pgmap v1302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1436.774475 secs 2024-07-26T13:15:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:03 smithi112 ceph-mon[37501]: pgmap v1303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:03.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:03 smithi107 ceph-mon[32659]: pgmap v1303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:03 smithi132 ceph-mon[37580]: pgmap v1303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:05 smithi112 ceph-mon[37501]: pgmap v1304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:15:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:05 smithi107 ceph-mon[32659]: pgmap v1304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:15:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:05 smithi132 ceph-mon[37580]: pgmap v1304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:15:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:06 smithi112 ceph-mon[37501]: 1 slow requests, 1 included below; oldest blocked for > 1441.774593 secs 2024-07-26T13:15:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:06 smithi112 ceph-mon[37501]: slow request 1441.774592 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T13:15:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:06 smithi107 ceph-mon[32659]: 1 slow requests, 1 included below; oldest blocked for > 1441.774593 secs 2024-07-26T13:15:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:06 smithi107 ceph-mon[32659]: slow request 1441.774592 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T13:15:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:06 smithi132 ceph-mon[37580]: 1 slow requests, 1 included below; oldest blocked for > 1441.774593 secs 2024-07-26T13:15:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:06 smithi132 ceph-mon[37580]: slow request 1441.774592 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T13:15:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:07 smithi112 ceph-mon[37501]: pgmap v1305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:07 smithi107 ceph-mon[32659]: pgmap v1305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:07 smithi132 ceph-mon[37580]: pgmap v1305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:08.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:08.234+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:08.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:08.234+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:08.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:08.234+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:09 smithi112 ceph-mon[37501]: pgmap v1306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:15:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:09 smithi107 ceph-mon[32659]: pgmap v1306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:15:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:09 smithi132 ceph-mon[37580]: pgmap v1306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:15:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:10.383+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:10.383+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:10.383+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:11 smithi112 ceph-mon[37501]: pgmap v1307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:15:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1446.774664 secs 2024-07-26T13:15:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:11 smithi107 ceph-mon[32659]: pgmap v1307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:15:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1446.774664 secs 2024-07-26T13:15:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:11 smithi132 ceph-mon[37580]: pgmap v1307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:15:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1446.774664 secs 2024-07-26T13:15:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:15:12.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:12.881+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:15:12.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:12.882+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:15:12.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:12.935+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d8003690 con 0x7ff104102930 2024-07-26T13:15:12.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:12.935+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d8003690 con 0x7ff104102930 2024-07-26T13:15:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:13 smithi107 ceph-mon[32659]: pgmap v1308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:15:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:13 smithi107 ceph-mon[32659]: pgmap v1309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:13 smithi112 ceph-mon[37501]: pgmap v1308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:15:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:13 smithi112 ceph-mon[37501]: pgmap v1309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:13.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:13 smithi132 ceph-mon[37580]: pgmap v1308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:15:13.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:13 smithi132 ceph-mon[37580]: pgmap v1309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:13.993+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44003690 con 0x55aecd6b64d0 2024-07-26T13:15:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:13.993+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:15:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:13.994+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44003690 con 0x55aecd6b64d0 2024-07-26T13:15:13.994 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:13.994+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:15:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:15 smithi112 ceph-mon[37501]: pgmap v1310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:15 smithi107 ceph-mon[32659]: pgmap v1310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:15 smithi132 ceph-mon[37580]: pgmap v1310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1451.774782 secs 2024-07-26T13:15:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1451.774782 secs 2024-07-26T13:15:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1451.774782 secs 2024-07-26T13:15:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:17 smithi112 ceph-mon[37501]: pgmap v1311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:17 smithi107 ceph-mon[32659]: pgmap v1311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:17 smithi132 ceph-mon[37580]: pgmap v1311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:19 smithi112 ceph-mon[37501]: pgmap v1312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:19 smithi107 ceph-mon[32659]: pgmap v1312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:19 smithi132 ceph-mon[37580]: pgmap v1312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:21 smithi112 ceph-mon[37501]: pgmap v1313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1456.774926 secs 2024-07-26T13:15:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:21 smithi107 ceph-mon[32659]: pgmap v1313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1456.774926 secs 2024-07-26T13:15:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:21 smithi132 ceph-mon[37580]: pgmap v1313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1456.774926 secs 2024-07-26T13:15:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:23.242+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:23.242+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:23.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:23.242+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:23 smithi112 ceph-mon[37501]: pgmap v1314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:23 smithi107 ceph-mon[32659]: pgmap v1314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:23 smithi132 ceph-mon[37580]: pgmap v1314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:25.393+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:25.393+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:25.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:25.393+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:25 smithi112 ceph-mon[37501]: pgmap v1315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:25 smithi107 ceph-mon[32659]: pgmap v1315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:25 smithi132 ceph-mon[37580]: pgmap v1315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1461.775044 secs 2024-07-26T13:15:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1461.775044 secs 2024-07-26T13:15:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1461.775044 secs 2024-07-26T13:15:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:27 smithi112 ceph-mon[37501]: pgmap v1316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:27 smithi107 ceph-mon[32659]: pgmap v1316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:27 smithi132 ceph-mon[37580]: pgmap v1316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:29 smithi112 ceph-mon[37501]: pgmap v1317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:29 smithi107 ceph-mon[32659]: pgmap v1317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:29 smithi132 ceph-mon[37580]: pgmap v1317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:31 smithi112 ceph-mon[37501]: pgmap v1318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1466.775159 secs 2024-07-26T13:15:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:31 smithi107 ceph-mon[32659]: pgmap v1318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1466.775159 secs 2024-07-26T13:15:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:31 smithi132 ceph-mon[37580]: pgmap v1318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1466.775159 secs 2024-07-26T13:15:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:33 smithi112 ceph-mon[37501]: pgmap v1319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:33 smithi107 ceph-mon[32659]: pgmap v1319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:33 smithi132 ceph-mon[37580]: pgmap v1319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:35 smithi112 ceph-mon[37501]: pgmap v1320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:35 smithi107 ceph-mon[32659]: pgmap v1320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:35 smithi132 ceph-mon[37580]: pgmap v1320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1471.775242 secs 2024-07-26T13:15:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1471.775242 secs 2024-07-26T13:15:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1471.775242 secs 2024-07-26T13:15:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:37 smithi107 ceph-mon[32659]: pgmap v1321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:37 smithi132 ceph-mon[37580]: pgmap v1321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:37 smithi112 ceph-mon[37501]: pgmap v1321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:38.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:38.251+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:38.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:38.251+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:38.254 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:38.252+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:39 smithi107 ceph-mon[32659]: pgmap v1322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:39 smithi132 ceph-mon[37580]: pgmap v1322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:39 smithi112 ceph-mon[37501]: pgmap v1322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:40.407+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:40.407+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:40.407+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:41 smithi107 ceph-mon[32659]: pgmap v1323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1476.775361 secs 2024-07-26T13:15:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:41 smithi132 ceph-mon[37580]: pgmap v1323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1476.775361 secs 2024-07-26T13:15:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:41 smithi112 ceph-mon[37501]: pgmap v1323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1476.775361 secs 2024-07-26T13:15:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:43 smithi107 ceph-mon[32659]: pgmap v1324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:15:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:43 smithi132 ceph-mon[37580]: pgmap v1324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:15:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:43 smithi112 ceph-mon[37501]: pgmap v1324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:15:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:45 smithi107 ceph-mon[32659]: pgmap v1325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:45 smithi132 ceph-mon[37580]: pgmap v1325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:45 smithi112 ceph-mon[37501]: pgmap v1325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:15:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1481.775485 secs 2024-07-26T13:15:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1481.775485 secs 2024-07-26T13:15:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1481.775485 secs 2024-07-26T13:15:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:47 smithi107 ceph-mon[32659]: pgmap v1326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:47 smithi132 ceph-mon[37580]: pgmap v1326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:47 smithi112 ceph-mon[37501]: pgmap v1326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:48 smithi107 ceph-mon[32659]: pgmap v1327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:15:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:48 smithi132 ceph-mon[37580]: pgmap v1327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:15:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:48 smithi112 ceph-mon[37501]: pgmap v1327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:15:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:50 smithi107 ceph-mon[32659]: pgmap v1328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1486.775554 secs 2024-07-26T13:15:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:50 smithi132 ceph-mon[37580]: pgmap v1328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1486.775554 secs 2024-07-26T13:15:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:50 smithi112 ceph-mon[37501]: pgmap v1328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:15:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1486.775554 secs 2024-07-26T13:15:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:52 smithi107 ceph-mon[32659]: pgmap v1329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:15:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:52 smithi132 ceph-mon[37580]: pgmap v1329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:15:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:52 smithi112 ceph-mon[37501]: pgmap v1329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:15:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:53.267+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:53.267+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:53.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:53.267+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:54 smithi107 ceph-mon[32659]: pgmap v1330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:15:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:54 smithi132 ceph-mon[37580]: pgmap v1330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:15:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:54 smithi112 ceph-mon[37501]: pgmap v1330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:15:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:55.415+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:55.415+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:15:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:55.415+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1491.775691 secs 2024-07-26T13:15:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1491.775691 secs 2024-07-26T13:15:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1491.775691 secs 2024-07-26T13:15:56.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:56.889+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:15:56.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:56.890+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:15:56.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:56.942+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:15:56.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:15:56.942+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:15:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:56 smithi107 ceph-mon[32659]: pgmap v1331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:56 smithi132 ceph-mon[37580]: pgmap v1331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:56 smithi112 ceph-mon[37501]: pgmap v1331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:15:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:59.001+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44003690 con 0x55aecd6b64d0 2024-07-26T13:15:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:59.001+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:15:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:59.002+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44003690 con 0x55aecd6b64d0 2024-07-26T13:15:59.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:15:59.003+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:15:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:15:58 smithi107 ceph-mon[32659]: pgmap v1332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:15:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:15:58 smithi132 ceph-mon[37580]: pgmap v1332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:15:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:15:58 smithi112 ceph-mon[37501]: pgmap v1332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:16:01.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:00 smithi112 ceph-mon[37501]: pgmap v1333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:01.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1496.775823 secs 2024-07-26T13:16:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:00 smithi107 ceph-mon[32659]: pgmap v1333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1496.775823 secs 2024-07-26T13:16:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:00 smithi132 ceph-mon[37580]: pgmap v1333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1496.775823 secs 2024-07-26T13:16:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:02 smithi107 ceph-mon[32659]: pgmap v1334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:02 smithi132 ceph-mon[37580]: pgmap v1334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:02 smithi112 ceph-mon[37501]: pgmap v1334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:04 smithi107 ceph-mon[32659]: pgmap v1335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:04 smithi132 ceph-mon[37580]: pgmap v1335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:04 smithi112 ceph-mon[37501]: pgmap v1335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1501.775967 secs 2024-07-26T13:16:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1501.775967 secs 2024-07-26T13:16:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1501.775967 secs 2024-07-26T13:16:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:06 smithi107 ceph-mon[32659]: pgmap v1336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:06 smithi132 ceph-mon[37580]: pgmap v1336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:06 smithi112 ceph-mon[37501]: pgmap v1336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:08.283+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:08.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:08.283+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:08.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:08.283+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:08 smithi107 ceph-mon[32659]: pgmap v1337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:08 smithi132 ceph-mon[37580]: pgmap v1337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:08 smithi112 ceph-mon[37501]: pgmap v1337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:10.421+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:10.421+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:10.421+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:10 smithi107 ceph-mon[32659]: pgmap v1338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1506.776109 secs 2024-07-26T13:16:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:10 smithi132 ceph-mon[37580]: pgmap v1338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1506.776109 secs 2024-07-26T13:16:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:10 smithi112 ceph-mon[37501]: pgmap v1338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1506.776109 secs 2024-07-26T13:16:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:12 smithi132 ceph-mon[37580]: pgmap v1339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.065 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:12 smithi107 ceph-mon[32659]: pgmap v1339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:13.065 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:13.065 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.066 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.066 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.066 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.083 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:12 smithi112 ceph-mon[37501]: pgmap v1339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:13.083 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:13.083 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.083 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.083 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:13.083 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:14 smithi107 ceph-mon[32659]: pgmap v1340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:14 smithi132 ceph-mon[37580]: pgmap v1340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:14 smithi112 ceph-mon[37501]: pgmap v1340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1511.776174 secs 2024-07-26T13:16:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:16 smithi112 ceph-mon[37501]: pgmap v1341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1511.776174 secs 2024-07-26T13:16:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:16 smithi107 ceph-mon[32659]: pgmap v1341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1511.776174 secs 2024-07-26T13:16:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:16 smithi132 ceph-mon[37580]: pgmap v1341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:17 smithi112 ceph-mon[37501]: pgmap v1342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:17 smithi112 ceph-mon[37501]: pgmap v1343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:16:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:17 smithi107 ceph-mon[32659]: pgmap v1342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:17 smithi107 ceph-mon[32659]: pgmap v1343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:16:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:16:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:16:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:17 smithi132 ceph-mon[37580]: pgmap v1342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:16:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:17 smithi132 ceph-mon[37580]: pgmap v1343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T13:16:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:16:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:19 smithi112 ceph-mon[37501]: pgmap v1344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:19 smithi107 ceph-mon[32659]: pgmap v1344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:19 smithi132 ceph-mon[37580]: pgmap v1344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1516.776242 secs 2024-07-26T13:16:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1516.776242 secs 2024-07-26T13:16:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1516.776242 secs 2024-07-26T13:16:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:21 smithi107 ceph-mon[32659]: pgmap v1345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:21 smithi132 ceph-mon[37580]: pgmap v1345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:21 smithi112 ceph-mon[37501]: pgmap v1345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:23.298+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:23.298+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:23.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:23.298+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:23 smithi107 ceph-mon[32659]: pgmap v1346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:23 smithi132 ceph-mon[37580]: pgmap v1346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:23 smithi112 ceph-mon[37501]: pgmap v1346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:16:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:25.433+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:25.433+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:25.433+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:25 smithi107 ceph-mon[32659]: pgmap v1347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:16:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1521.776321 secs 2024-07-26T13:16:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:25 smithi132 ceph-mon[37580]: pgmap v1347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:16:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1521.776321 secs 2024-07-26T13:16:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:25 smithi112 ceph-mon[37501]: pgmap v1347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T13:16:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1521.776321 secs 2024-07-26T13:16:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:27 smithi107 ceph-mon[32659]: pgmap v1348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:27 smithi132 ceph-mon[37580]: pgmap v1348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:27 smithi112 ceph-mon[37501]: pgmap v1348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:29 smithi107 ceph-mon[32659]: pgmap v1349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:29 smithi132 ceph-mon[37580]: pgmap v1349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:29 smithi112 ceph-mon[37501]: pgmap v1349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:30.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1526.776406 secs 2024-07-26T13:16:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1526.776406 secs 2024-07-26T13:16:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1526.776406 secs 2024-07-26T13:16:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:31 smithi107 ceph-mon[32659]: pgmap v1350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:31 smithi132 ceph-mon[37580]: pgmap v1350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:31.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:31 smithi112 ceph-mon[37501]: pgmap v1350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:33 smithi107 ceph-mon[32659]: pgmap v1351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:33 smithi132 ceph-mon[37580]: pgmap v1351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:33 smithi112 ceph-mon[37501]: pgmap v1351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:35 smithi107 ceph-mon[32659]: pgmap v1352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:16:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1531.776487 secs 2024-07-26T13:16:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:35 smithi132 ceph-mon[37580]: pgmap v1352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:16:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1531.776487 secs 2024-07-26T13:16:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:35 smithi112 ceph-mon[37501]: pgmap v1352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:16:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1531.776487 secs 2024-07-26T13:16:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:37 smithi107 ceph-mon[32659]: pgmap v1353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:37 smithi132 ceph-mon[37580]: pgmap v1353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:37 smithi112 ceph-mon[37501]: pgmap v1353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:38.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:38.313+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:38.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:38.313+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:38.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:38.313+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:39 smithi107 ceph-mon[32659]: pgmap v1354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:39 smithi132 ceph-mon[37580]: pgmap v1354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:39 smithi112 ceph-mon[37501]: pgmap v1354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:40.446+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:40.446+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:40.446+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1536.776570 secs 2024-07-26T13:16:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1536.776570 secs 2024-07-26T13:16:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1536.776570 secs 2024-07-26T13:16:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:41 smithi107 ceph-mon[32659]: pgmap v1355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:41 smithi132 ceph-mon[37580]: pgmap v1355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:41 smithi112 ceph-mon[37501]: pgmap v1355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:43 smithi107 ceph-mon[32659]: pgmap v1356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:43 smithi132 ceph-mon[37580]: pgmap v1356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:43 smithi112 ceph-mon[37501]: pgmap v1356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:45 smithi132 ceph-mon[37580]: pgmap v1357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:16:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1541.776649 secs 2024-07-26T13:16:45.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:45 smithi107 ceph-mon[32659]: pgmap v1357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:16:45.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1541.776649 secs 2024-07-26T13:16:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:45 smithi112 ceph-mon[37501]: pgmap v1357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:16:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1541.776649 secs 2024-07-26T13:16:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:47 smithi132 ceph-mon[37580]: pgmap v1358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:47 smithi112 ceph-mon[37501]: pgmap v1358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:47 smithi107 ceph-mon[32659]: pgmap v1358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:49 smithi132 ceph-mon[37580]: pgmap v1359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:49 smithi112 ceph-mon[37501]: pgmap v1359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:49 smithi107 ceph-mon[32659]: pgmap v1359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1546.776792 secs 2024-07-26T13:16:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1546.776792 secs 2024-07-26T13:16:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1546.776792 secs 2024-07-26T13:16:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:51 smithi112 ceph-mon[37501]: pgmap v1360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:51 smithi107 ceph-mon[32659]: pgmap v1360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:51 smithi132 ceph-mon[37580]: pgmap v1360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:53.315+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:53.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:53.315+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:16:53.315+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:53 smithi112 ceph-mon[37501]: pgmap v1361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:53 smithi107 ceph-mon[32659]: pgmap v1361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:53 smithi132 ceph-mon[37580]: pgmap v1361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:16:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:55.458+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:55.458+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:16:55.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:16:55.458+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:55 smithi112 ceph-mon[37501]: pgmap v1362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:16:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1551.776925 secs 2024-07-26T13:16:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:55 smithi107 ceph-mon[32659]: pgmap v1362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:16:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1551.776925 secs 2024-07-26T13:16:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:55 smithi132 ceph-mon[37580]: pgmap v1362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:16:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1551.776925 secs 2024-07-26T13:16:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:57 smithi112 ceph-mon[37501]: pgmap v1363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:57 smithi107 ceph-mon[32659]: pgmap v1363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:57 smithi132 ceph-mon[37580]: pgmap v1363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:16:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:16:59 smithi112 ceph-mon[37501]: pgmap v1364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:16:59 smithi107 ceph-mon[32659]: pgmap v1364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:00.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:16:59 smithi132 ceph-mon[37580]: pgmap v1364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1556.777040 secs 2024-07-26T13:17:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1556.777040 secs 2024-07-26T13:17:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1556.777040 secs 2024-07-26T13:17:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:01 smithi112 ceph-mon[37501]: pgmap v1365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:01 smithi107 ceph-mon[32659]: pgmap v1365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:01 smithi132 ceph-mon[37580]: pgmap v1365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:03 smithi112 ceph-mon[37501]: pgmap v1366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:03 smithi107 ceph-mon[32659]: pgmap v1366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:04.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:03 smithi132 ceph-mon[37580]: pgmap v1366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:05 smithi112 ceph-mon[37501]: pgmap v1367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:17:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1561.777147 secs 2024-07-26T13:17:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:05 smithi107 ceph-mon[32659]: pgmap v1367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:17:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1561.777147 secs 2024-07-26T13:17:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:05 smithi132 ceph-mon[37580]: pgmap v1367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:17:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1561.777147 secs 2024-07-26T13:17:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:07 smithi112 ceph-mon[37501]: pgmap v1368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:07 smithi107 ceph-mon[32659]: pgmap v1368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:07 smithi132 ceph-mon[37580]: pgmap v1368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:08.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:08.319+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:08.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:08.319+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:08.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:08.319+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:09 smithi112 ceph-mon[37501]: pgmap v1369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:09 smithi107 ceph-mon[32659]: pgmap v1369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:09 smithi132 ceph-mon[37580]: pgmap v1369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:10.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:10.465+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:10.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:10.465+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:10.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:10.465+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1566.777213 secs 2024-07-26T13:17:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1566.777213 secs 2024-07-26T13:17:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1566.777213 secs 2024-07-26T13:17:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:11 smithi112 ceph-mon[37501]: pgmap v1370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:11 smithi107 ceph-mon[32659]: pgmap v1370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:11 smithi132 ceph-mon[37580]: pgmap v1370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:13 smithi112 ceph-mon[37501]: pgmap v1371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:13 smithi107 ceph-mon[32659]: pgmap v1371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:13 smithi132 ceph-mon[37580]: pgmap v1371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:15 smithi107 ceph-mon[32659]: pgmap v1372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:17:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1571.777277 secs 2024-07-26T13:17:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:15 smithi132 ceph-mon[37580]: pgmap v1372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:17:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1571.777277 secs 2024-07-26T13:17:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:15 smithi112 ceph-mon[37501]: pgmap v1372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:17:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1571.777277 secs 2024-07-26T13:17:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:17 smithi107 ceph-mon[32659]: pgmap v1373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:17 smithi132 ceph-mon[37580]: pgmap v1373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:17 smithi112 ceph-mon[37501]: pgmap v1373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:17:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:18 smithi107 ceph-mon[32659]: pgmap v1374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:17:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:18 smithi107 ceph-mon[32659]: pgmap v1375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:17:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:17:18.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:18.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:18 smithi132 ceph-mon[37580]: pgmap v1374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:17:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:18 smithi132 ceph-mon[37580]: pgmap v1375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:17:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:17:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:18 smithi112 ceph-mon[37501]: pgmap v1374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:17:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:18 smithi112 ceph-mon[37501]: pgmap v1375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:17:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:17:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:20 smithi112 ceph-mon[37501]: pgmap v1376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1576.777384 secs 2024-07-26T13:17:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:20 smithi107 ceph-mon[32659]: pgmap v1376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1576.777384 secs 2024-07-26T13:17:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:20 smithi132 ceph-mon[37580]: pgmap v1376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1576.777384 secs 2024-07-26T13:17:22.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:22 smithi112 ceph-mon[37501]: pgmap v1377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:22 smithi107 ceph-mon[32659]: pgmap v1377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:22 smithi132 ceph-mon[37580]: pgmap v1377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:23.334+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:23.335+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:23.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:23.335+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:24 smithi112 ceph-mon[37501]: pgmap v1378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:17:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:24 smithi107 ceph-mon[32659]: pgmap v1378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:17:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:24 smithi132 ceph-mon[37580]: pgmap v1378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:17:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:25.481+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:25.481+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:25.481+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1581.777457 secs 2024-07-26T13:17:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1581.777457 secs 2024-07-26T13:17:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1581.777457 secs 2024-07-26T13:17:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:26 smithi112 ceph-mon[37501]: pgmap v1379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:17:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:26 smithi107 ceph-mon[32659]: pgmap v1379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:17:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:26 smithi132 ceph-mon[37580]: pgmap v1379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:17:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:28 smithi112 ceph-mon[37501]: pgmap v1380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:28 smithi107 ceph-mon[32659]: pgmap v1380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:28 smithi132 ceph-mon[37580]: pgmap v1380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:30 smithi112 ceph-mon[37501]: pgmap v1381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1586.777518 secs 2024-07-26T13:17:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:30 smithi107 ceph-mon[32659]: pgmap v1381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1586.777518 secs 2024-07-26T13:17:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:30 smithi132 ceph-mon[37580]: pgmap v1381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1586.777518 secs 2024-07-26T13:17:33.018 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:32 smithi107 ceph-mon[32659]: pgmap v1382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:17:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:32 smithi132 ceph-mon[37580]: pgmap v1382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:17:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:32 smithi112 ceph-mon[37501]: pgmap v1382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:17:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:34 smithi107 ceph-mon[32659]: pgmap v1383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:17:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:34 smithi132 ceph-mon[37580]: pgmap v1383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:17:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:34 smithi112 ceph-mon[37501]: pgmap v1383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:17:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1591.777599 secs 2024-07-26T13:17:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1591.777599 secs 2024-07-26T13:17:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1591.777599 secs 2024-07-26T13:17:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:36 smithi107 ceph-mon[32659]: pgmap v1384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:36 smithi132 ceph-mon[37580]: pgmap v1384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:36 smithi112 ceph-mon[37501]: pgmap v1384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:17:38.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:38.349+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:38.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:38.349+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:38.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:38.349+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:38 smithi107 ceph-mon[32659]: pgmap v1385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:17:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:38 smithi132 ceph-mon[37580]: pgmap v1385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:17:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:38 smithi112 ceph-mon[37501]: pgmap v1385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:17:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:40.495+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:40.495+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:40.495+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:40 smithi107 ceph-mon[32659]: pgmap v1386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1596.777718 secs 2024-07-26T13:17:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:40 smithi132 ceph-mon[37580]: pgmap v1386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1596.777718 secs 2024-07-26T13:17:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:40 smithi112 ceph-mon[37501]: pgmap v1386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1596.777718 secs 2024-07-26T13:17:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:42 smithi107 ceph-mon[32659]: pgmap v1387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:42 smithi132 ceph-mon[37580]: pgmap v1387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:42 smithi112 ceph-mon[37501]: pgmap v1387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:44 smithi107 ceph-mon[32659]: pgmap v1388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:44 smithi132 ceph-mon[37580]: pgmap v1388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:44 smithi112 ceph-mon[37501]: pgmap v1388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1601.777832 secs 2024-07-26T13:17:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1601.777832 secs 2024-07-26T13:17:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1601.777832 secs 2024-07-26T13:17:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:46 smithi107 ceph-mon[32659]: pgmap v1389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:46 smithi132 ceph-mon[37580]: pgmap v1389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:46 smithi112 ceph-mon[37501]: pgmap v1389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:48 smithi107 ceph-mon[32659]: pgmap v1390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:48 smithi132 ceph-mon[37580]: pgmap v1390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:48 smithi112 ceph-mon[37501]: pgmap v1390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:50 smithi107 ceph-mon[32659]: pgmap v1391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1606.777958 secs 2024-07-26T13:17:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:50 smithi132 ceph-mon[37580]: pgmap v1391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1606.777958 secs 2024-07-26T13:17:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:50 smithi112 ceph-mon[37501]: pgmap v1391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1606.777958 secs 2024-07-26T13:17:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:52 smithi107 ceph-mon[32659]: pgmap v1392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:52 smithi132 ceph-mon[37580]: pgmap v1392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:52 smithi112 ceph-mon[37501]: pgmap v1392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:17:53.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:53.364+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:53.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:53.364+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:53.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:17:53.364+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:54 smithi107 ceph-mon[32659]: pgmap v1393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:54 smithi132 ceph-mon[37580]: pgmap v1393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:54 smithi112 ceph-mon[37501]: pgmap v1393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:55.511+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:55.511+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:17:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:17:55.511+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1611.778078 secs 2024-07-26T13:17:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1611.778078 secs 2024-07-26T13:17:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1611.778078 secs 2024-07-26T13:17:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:56 smithi132 ceph-mon[37580]: pgmap v1394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:56 smithi112 ceph-mon[37501]: pgmap v1394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:56 smithi107 ceph-mon[32659]: pgmap v1394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:17:58 smithi132 ceph-mon[37580]: pgmap v1395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:17:58 smithi112 ceph-mon[37501]: pgmap v1395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:17:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:17:58 smithi107 ceph-mon[32659]: pgmap v1395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:00 smithi132 ceph-mon[37580]: pgmap v1396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1616.778195 secs 2024-07-26T13:18:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:00 smithi112 ceph-mon[37501]: pgmap v1396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1616.778195 secs 2024-07-26T13:18:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:00 smithi107 ceph-mon[32659]: pgmap v1396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1616.778195 secs 2024-07-26T13:18:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:02 smithi112 ceph-mon[37501]: pgmap v1397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:03.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:02 smithi107 ceph-mon[32659]: pgmap v1397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:02 smithi132 ceph-mon[37580]: pgmap v1397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:04 smithi112 ceph-mon[37501]: pgmap v1398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:04 smithi107 ceph-mon[32659]: pgmap v1398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:04 smithi132 ceph-mon[37580]: pgmap v1398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1621.778261 secs 2024-07-26T13:18:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1621.778261 secs 2024-07-26T13:18:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1621.778261 secs 2024-07-26T13:18:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:06 smithi112 ceph-mon[37501]: pgmap v1399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:06 smithi107 ceph-mon[32659]: pgmap v1399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:06 smithi132 ceph-mon[37580]: pgmap v1399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:08.380+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:08.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:08.380+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:08.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:08.380+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:08 smithi112 ceph-mon[37501]: pgmap v1400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:08 smithi107 ceph-mon[32659]: pgmap v1400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:08 smithi132 ceph-mon[37580]: pgmap v1400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:10.528+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:10.528+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:10.529 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:10.528+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:10 smithi112 ceph-mon[37501]: pgmap v1401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1626.778380 secs 2024-07-26T13:18:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:10 smithi107 ceph-mon[32659]: pgmap v1401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1626.778380 secs 2024-07-26T13:18:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:10 smithi132 ceph-mon[37580]: pgmap v1401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1626.778380 secs 2024-07-26T13:18:12.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:12.912+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:18:12.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:12.913+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:18:12.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:12.963+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:18:12.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:12.964+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:18:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:12 smithi112 ceph-mon[37501]: pgmap v1402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:12 smithi107 ceph-mon[32659]: pgmap v1402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:12 smithi132 ceph-mon[37580]: pgmap v1402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:14.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:14.025+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:18:14.025 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:14.025+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44003690 con 0x55aecd6b64d0 2024-07-26T13:18:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:14.025+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:18:14.026 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:14.025+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44003690 con 0x55aecd6b64d0 2024-07-26T13:18:15.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:14 smithi132 ceph-mon[37580]: pgmap v1403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:14 smithi112 ceph-mon[37501]: pgmap v1403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:14 smithi107 ceph-mon[32659]: pgmap v1403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1631.778472 secs 2024-07-26T13:18:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1631.778472 secs 2024-07-26T13:18:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1631.778472 secs 2024-07-26T13:18:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:16 smithi112 ceph-mon[37501]: pgmap v1404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:16 smithi107 ceph-mon[32659]: pgmap v1404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:16 smithi132 ceph-mon[37580]: pgmap v1404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:18.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:18 smithi112 ceph-mon[37501]: pgmap v1405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:18:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:18 smithi107 ceph-mon[32659]: pgmap v1405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:18:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:18 smithi132 ceph-mon[37580]: pgmap v1405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:18:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:19 smithi112 ceph-mon[37501]: pgmap v1406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:19 smithi112 ceph-mon[37501]: pgmap v1407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:18:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:19 smithi107 ceph-mon[32659]: pgmap v1406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:19 smithi107 ceph-mon[32659]: pgmap v1407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:18:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:19 smithi132 ceph-mon[37580]: pgmap v1406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:19 smithi132 ceph-mon[37580]: pgmap v1407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:18:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1636.778593 secs 2024-07-26T13:18:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1636.778593 secs 2024-07-26T13:18:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1636.778593 secs 2024-07-26T13:18:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:21 smithi112 ceph-mon[37501]: pgmap v1408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:18:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:21 smithi107 ceph-mon[32659]: pgmap v1408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:18:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:21 smithi132 ceph-mon[37580]: pgmap v1408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:18:23.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:23.391+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:23.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:23.391+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:23.393 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:23.391+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:23 smithi112 ceph-mon[37501]: pgmap v1409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:18:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:23 smithi107 ceph-mon[32659]: pgmap v1409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:18:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:23 smithi132 ceph-mon[37580]: pgmap v1409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:18:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:25.541+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:25.541+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:25.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:25.541+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:25 smithi112 ceph-mon[37501]: pgmap v1410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:18:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1641.778722 secs 2024-07-26T13:18:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:25 smithi107 ceph-mon[32659]: pgmap v1410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:18:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1641.778722 secs 2024-07-26T13:18:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:25 smithi132 ceph-mon[37580]: pgmap v1410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:18:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1641.778722 secs 2024-07-26T13:18:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:27 smithi112 ceph-mon[37501]: pgmap v1411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:27 smithi107 ceph-mon[32659]: pgmap v1411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:27 smithi132 ceph-mon[37580]: pgmap v1411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:29 smithi112 ceph-mon[37501]: pgmap v1412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:29 smithi107 ceph-mon[32659]: pgmap v1412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:29 smithi132 ceph-mon[37580]: pgmap v1412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1646.778870 secs 2024-07-26T13:18:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1646.778870 secs 2024-07-26T13:18:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1646.778870 secs 2024-07-26T13:18:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:31 smithi107 ceph-mon[32659]: pgmap v1413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:31 smithi132 ceph-mon[37580]: pgmap v1413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:31 smithi112 ceph-mon[37501]: pgmap v1413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:34.252 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:33 smithi112 ceph-mon[37501]: pgmap v1414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:33 smithi107 ceph-mon[32659]: pgmap v1414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:33 smithi132 ceph-mon[37580]: pgmap v1414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:35 smithi107 ceph-mon[32659]: pgmap v1415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1651.779017 secs 2024-07-26T13:18:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:35 smithi132 ceph-mon[37580]: pgmap v1415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1651.779017 secs 2024-07-26T13:18:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:35 smithi112 ceph-mon[37501]: pgmap v1415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1651.779017 secs 2024-07-26T13:18:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:37 smithi107 ceph-mon[32659]: pgmap v1416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:37 smithi132 ceph-mon[37580]: pgmap v1416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:38.396+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:38.396+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:38.398 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:38.396+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:37 smithi112 ceph-mon[37501]: pgmap v1416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:39 smithi107 ceph-mon[32659]: pgmap v1417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:39 smithi132 ceph-mon[37580]: pgmap v1417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:39 smithi112 ceph-mon[37501]: pgmap v1417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:40.555+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:40.556+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:40.558 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:40.556+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1656.779117 secs 2024-07-26T13:18:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1656.779117 secs 2024-07-26T13:18:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1656.779117 secs 2024-07-26T13:18:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:41 smithi107 ceph-mon[32659]: pgmap v1418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:41 smithi132 ceph-mon[37580]: pgmap v1418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:41 smithi112 ceph-mon[37501]: pgmap v1418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:43 smithi107 ceph-mon[32659]: pgmap v1419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:43 smithi132 ceph-mon[37580]: pgmap v1419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:43 smithi112 ceph-mon[37501]: pgmap v1419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:45 smithi107 ceph-mon[32659]: pgmap v1420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1661.779239 secs 2024-07-26T13:18:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:45 smithi132 ceph-mon[37580]: pgmap v1420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1661.779239 secs 2024-07-26T13:18:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:45 smithi112 ceph-mon[37501]: pgmap v1420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1661.779239 secs 2024-07-26T13:18:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:48 smithi107 ceph-mon[32659]: pgmap v1421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:48 smithi132 ceph-mon[37580]: pgmap v1421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:48 smithi112 ceph-mon[37501]: pgmap v1421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:50 smithi107 ceph-mon[32659]: pgmap v1422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:50 smithi132 ceph-mon[37580]: pgmap v1422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:50 smithi112 ceph-mon[37501]: pgmap v1422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1666.779330 secs 2024-07-26T13:18:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1666.779330 secs 2024-07-26T13:18:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1666.779330 secs 2024-07-26T13:18:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:52 smithi107 ceph-mon[32659]: pgmap v1423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:52 smithi132 ceph-mon[37580]: pgmap v1423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:52 smithi112 ceph-mon[37501]: pgmap v1423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:53.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:53.412+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:53.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:53.412+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:53.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:18:53.412+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:54 smithi107 ceph-mon[32659]: pgmap v1424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:54 smithi132 ceph-mon[37580]: pgmap v1424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:54 smithi112 ceph-mon[37501]: pgmap v1424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:55.572+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:55.572+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:18:55.573 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:18:55.572+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:56 smithi107 ceph-mon[32659]: pgmap v1425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1671.779445 secs 2024-07-26T13:18:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:56 smithi132 ceph-mon[37580]: pgmap v1425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1671.779445 secs 2024-07-26T13:18:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:56 smithi112 ceph-mon[37501]: pgmap v1425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1671.779445 secs 2024-07-26T13:18:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:18:58 smithi107 ceph-mon[32659]: pgmap v1426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:18:58 smithi132 ceph-mon[37580]: pgmap v1426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:18:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:18:58 smithi112 ceph-mon[37501]: pgmap v1426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:00 smithi132 ceph-mon[37580]: pgmap v1427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:00 smithi112 ceph-mon[37501]: pgmap v1427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:00 smithi107 ceph-mon[32659]: pgmap v1427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1676.779527 secs 2024-07-26T13:19:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1676.779527 secs 2024-07-26T13:19:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1676.779527 secs 2024-07-26T13:19:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:02 smithi112 ceph-mon[37501]: pgmap v1428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:02 smithi107 ceph-mon[32659]: pgmap v1428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:02 smithi132 ceph-mon[37580]: pgmap v1428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:04 smithi112 ceph-mon[37501]: pgmap v1429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:04 smithi107 ceph-mon[32659]: pgmap v1429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:04 smithi132 ceph-mon[37580]: pgmap v1429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:06 smithi112 ceph-mon[37501]: pgmap v1430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1681.779663 secs 2024-07-26T13:19:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:06 smithi107 ceph-mon[32659]: pgmap v1430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1681.779663 secs 2024-07-26T13:19:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:06 smithi132 ceph-mon[37580]: pgmap v1430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1681.779663 secs 2024-07-26T13:19:08.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:08 smithi107 ceph-mon[32659]: pgmap v1431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:08.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:08.426+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:08.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:08.426+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:08.428 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:08.426+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:08 smithi112 ceph-mon[37501]: pgmap v1431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:08 smithi132 ceph-mon[37580]: pgmap v1431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:10 smithi112 ceph-mon[37501]: pgmap v1432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:10 smithi107 ceph-mon[32659]: pgmap v1432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:10 smithi132 ceph-mon[37580]: pgmap v1432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:10.588+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:10.588+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:10.589 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:10.588+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1686.779795 secs 2024-07-26T13:19:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1686.779795 secs 2024-07-26T13:19:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1686.779795 secs 2024-07-26T13:19:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:12 smithi112 ceph-mon[37501]: pgmap v1433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:12 smithi107 ceph-mon[32659]: pgmap v1433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:12 smithi132 ceph-mon[37580]: pgmap v1433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:13 smithi112 ceph-mon[37501]: pgmap v1434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:13 smithi107 ceph-mon[32659]: pgmap v1434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:13 smithi132 ceph-mon[37580]: pgmap v1434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:15 smithi112 ceph-mon[37501]: pgmap v1435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1691.779917 secs 2024-07-26T13:19:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:15 smithi107 ceph-mon[32659]: pgmap v1435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1691.779917 secs 2024-07-26T13:19:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:15 smithi132 ceph-mon[37580]: pgmap v1435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1691.779917 secs 2024-07-26T13:19:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:17 smithi112 ceph-mon[37501]: pgmap v1436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:17 smithi107 ceph-mon[32659]: pgmap v1436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:17 smithi132 ceph-mon[37580]: pgmap v1436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:19 smithi112 ceph-mon[37501]: pgmap v1437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:19 smithi107 ceph-mon[32659]: pgmap v1437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:19 smithi132 ceph-mon[37580]: pgmap v1437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:20 smithi112 ceph-mon[37501]: pgmap v1438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:19:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:20 smithi112 ceph-mon[37501]: pgmap v1439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:19:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:19:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1696.780008 secs 2024-07-26T13:19:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:20 smithi107 ceph-mon[32659]: pgmap v1438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:19:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:20 smithi107 ceph-mon[32659]: pgmap v1439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:19:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:19:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1696.780008 secs 2024-07-26T13:19:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:20 smithi132 ceph-mon[37580]: pgmap v1438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:19:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:20 smithi132 ceph-mon[37580]: pgmap v1439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:19:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:19:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1696.780008 secs 2024-07-26T13:19:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:23 smithi107 ceph-mon[32659]: pgmap v1440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:23 smithi132 ceph-mon[37580]: pgmap v1440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:23.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:23.435+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:23.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:23.435+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:23.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:23.435+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:23 smithi112 ceph-mon[37501]: pgmap v1440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:25 smithi107 ceph-mon[32659]: pgmap v1441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:19:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:25 smithi132 ceph-mon[37580]: pgmap v1441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:19:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:25 smithi112 ceph-mon[37501]: pgmap v1441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:19:25.604 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:25.604+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:25.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:25.604+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:25.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:25.604+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1701.780127 secs 2024-07-26T13:19:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1701.780127 secs 2024-07-26T13:19:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1701.780127 secs 2024-07-26T13:19:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:27 smithi132 ceph-mon[37580]: pgmap v1442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:19:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:27 smithi112 ceph-mon[37501]: pgmap v1442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:19:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:27 smithi107 ceph-mon[32659]: pgmap v1442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:19:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:29 smithi132 ceph-mon[37580]: pgmap v1443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:19:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:29 smithi112 ceph-mon[37501]: pgmap v1443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:19:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:29 smithi107 ceph-mon[32659]: pgmap v1443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:19:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:31 smithi112 ceph-mon[37501]: pgmap v1444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:19:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1706.780248 secs 2024-07-26T13:19:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:31 smithi107 ceph-mon[32659]: pgmap v1444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:19:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1706.780248 secs 2024-07-26T13:19:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:31 smithi132 ceph-mon[37580]: pgmap v1444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:19:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1706.780248 secs 2024-07-26T13:19:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:33 smithi112 ceph-mon[37501]: pgmap v1445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:33 smithi107 ceph-mon[32659]: pgmap v1445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:33 smithi132 ceph-mon[37580]: pgmap v1445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:35 smithi112 ceph-mon[37501]: pgmap v1446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:35 smithi107 ceph-mon[32659]: pgmap v1446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:35 smithi132 ceph-mon[37580]: pgmap v1446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1711.780370 secs 2024-07-26T13:19:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1711.780370 secs 2024-07-26T13:19:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1711.780370 secs 2024-07-26T13:19:37.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:37 smithi107 ceph-mon[32659]: pgmap v1447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:37 smithi112 ceph-mon[37501]: pgmap v1447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:37 smithi132 ceph-mon[37580]: pgmap v1447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:38.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:38.449+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:38.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:38.449+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:38.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:38.449+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:39 smithi112 ceph-mon[37501]: pgmap v1448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:39 smithi107 ceph-mon[32659]: pgmap v1448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:39 smithi132 ceph-mon[37580]: pgmap v1448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:40.617+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:40.617+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:40.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:40.617+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:41 smithi112 ceph-mon[37501]: pgmap v1449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1716.780508 secs 2024-07-26T13:19:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:41 smithi107 ceph-mon[32659]: pgmap v1449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1716.780508 secs 2024-07-26T13:19:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:41 smithi132 ceph-mon[37580]: pgmap v1449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1716.780508 secs 2024-07-26T13:19:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:43 smithi112 ceph-mon[37501]: pgmap v1450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:43 smithi107 ceph-mon[32659]: pgmap v1450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:43 smithi132 ceph-mon[37580]: pgmap v1450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:45 smithi112 ceph-mon[37501]: pgmap v1451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:45 smithi107 ceph-mon[32659]: pgmap v1451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:45 smithi132 ceph-mon[37580]: pgmap v1451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1721.780569 secs 2024-07-26T13:19:46.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1721.780569 secs 2024-07-26T13:19:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1721.780569 secs 2024-07-26T13:19:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:47 smithi112 ceph-mon[37501]: pgmap v1452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:47 smithi107 ceph-mon[32659]: pgmap v1452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:47 smithi132 ceph-mon[37580]: pgmap v1452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:49 smithi112 ceph-mon[37501]: pgmap v1453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:49 smithi107 ceph-mon[32659]: pgmap v1453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:49 smithi132 ceph-mon[37580]: pgmap v1453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:51 smithi112 ceph-mon[37501]: pgmap v1454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1726.780689 secs 2024-07-26T13:19:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:51 smithi107 ceph-mon[32659]: pgmap v1454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1726.780689 secs 2024-07-26T13:19:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:51 smithi132 ceph-mon[37580]: pgmap v1454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1726.780689 secs 2024-07-26T13:19:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:53 smithi112 ceph-mon[37501]: pgmap v1455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:53.455+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:53.455+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:53.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:19:53.455+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:53.457 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:53 smithi107 ceph-mon[32659]: pgmap v1455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:53 smithi132 ceph-mon[37580]: pgmap v1455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:19:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:55 smithi112 ceph-mon[37501]: pgmap v1456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:55 smithi107 ceph-mon[32659]: pgmap v1456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:55 smithi132 ceph-mon[37580]: pgmap v1456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:55.634 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:55.633+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:55.634 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:55.633+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:19:55.634 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:19:55.633+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1731.780809 secs 2024-07-26T13:19:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1731.780809 secs 2024-07-26T13:19:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1731.780809 secs 2024-07-26T13:19:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:57 smithi112 ceph-mon[37501]: pgmap v1457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:57 smithi107 ceph-mon[32659]: pgmap v1457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:57 smithi132 ceph-mon[37580]: pgmap v1457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:19:59 smithi112 ceph-mon[37501]: pgmap v1458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:19:59 smithi107 ceph-mon[32659]: pgmap v1458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:19:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:19:59 smithi132 ceph-mon[37580]: pgmap v1458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:20:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:01 smithi112 ceph-mon[37501]: pgmap v1459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1736.780922 secs 2024-07-26T13:20:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:01 smithi107 ceph-mon[32659]: pgmap v1459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1736.780922 secs 2024-07-26T13:20:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:01 smithi132 ceph-mon[37580]: pgmap v1459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1736.780922 secs 2024-07-26T13:20:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:03 smithi112 ceph-mon[37501]: pgmap v1460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:03 smithi107 ceph-mon[32659]: pgmap v1460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:03 smithi132 ceph-mon[37580]: pgmap v1460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:05 smithi112 ceph-mon[37501]: pgmap v1461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:05 smithi107 ceph-mon[32659]: pgmap v1461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:05 smithi132 ceph-mon[37580]: pgmap v1461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:06.461 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1741.781033 secs 2024-07-26T13:20:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1741.781033 secs 2024-07-26T13:20:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1741.781033 secs 2024-07-26T13:20:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:07 smithi107 ceph-mon[32659]: pgmap v1462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:07 smithi132 ceph-mon[37580]: pgmap v1462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:07 smithi112 ceph-mon[37501]: pgmap v1462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:08.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:08.459+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:08.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:08.459+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:08.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:08.459+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:09 smithi107 ceph-mon[32659]: pgmap v1463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:09 smithi132 ceph-mon[37580]: pgmap v1463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:09 smithi112 ceph-mon[37501]: pgmap v1463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:10.640+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:10.640+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:10.640+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:11 smithi107 ceph-mon[32659]: pgmap v1464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1746.781165 secs 2024-07-26T13:20:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:11 smithi132 ceph-mon[37580]: pgmap v1464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1746.781165 secs 2024-07-26T13:20:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:11 smithi112 ceph-mon[37501]: pgmap v1464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1746.781165 secs 2024-07-26T13:20:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:13 smithi107 ceph-mon[32659]: pgmap v1465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:13 smithi132 ceph-mon[37580]: pgmap v1465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:13 smithi112 ceph-mon[37501]: pgmap v1465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:15 smithi107 ceph-mon[32659]: pgmap v1466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:15 smithi132 ceph-mon[37580]: pgmap v1466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:15 smithi112 ceph-mon[37501]: pgmap v1466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:16.523 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1751.781308 secs 2024-07-26T13:20:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1751.781308 secs 2024-07-26T13:20:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1751.781308 secs 2024-07-26T13:20:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:17 smithi107 ceph-mon[32659]: pgmap v1467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:17 smithi132 ceph-mon[37580]: pgmap v1467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:17 smithi112 ceph-mon[37501]: pgmap v1467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:19 smithi107 ceph-mon[32659]: pgmap v1468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:19.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:19 smithi132 ceph-mon[37580]: pgmap v1468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:19 smithi112 ceph-mon[37501]: pgmap v1468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:21 smithi107 ceph-mon[32659]: pgmap v1469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1756.781423 secs 2024-07-26T13:20:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:20:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:21 smithi132 ceph-mon[37580]: pgmap v1469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1756.781423 secs 2024-07-26T13:20:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:20:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:21 smithi112 ceph-mon[37501]: pgmap v1469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1756.781423 secs 2024-07-26T13:20:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:20:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:22 smithi107 ceph-mon[32659]: pgmap v1470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:20:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:22 smithi107 ceph-mon[32659]: pgmap v1471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:20:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:22 smithi132 ceph-mon[37580]: pgmap v1470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:20:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:22 smithi132 ceph-mon[37580]: pgmap v1471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:20:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:22 smithi112 ceph-mon[37501]: pgmap v1470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:20:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:22 smithi112 ceph-mon[37501]: pgmap v1471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:20:23.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:23.475+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:23.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:23.475+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:23.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:23.475+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:24 smithi107 ceph-mon[32659]: pgmap v1472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:24 smithi132 ceph-mon[37580]: pgmap v1472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:24 smithi112 ceph-mon[37501]: pgmap v1472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:25.655+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:25.656 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:25.655+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:25.657 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:25.655+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1761.781536 secs 2024-07-26T13:20:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:26 smithi107 ceph-mon[32659]: pgmap v1473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1761.781536 secs 2024-07-26T13:20:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:26 smithi132 ceph-mon[37580]: pgmap v1473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1761.781536 secs 2024-07-26T13:20:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:26 smithi112 ceph-mon[37501]: pgmap v1473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:28 smithi132 ceph-mon[37580]: pgmap v1474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:28 smithi112 ceph-mon[37501]: pgmap v1474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:28 smithi107 ceph-mon[32659]: pgmap v1474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:29 smithi112 ceph-mon[37501]: pgmap v1475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:29 smithi107 ceph-mon[32659]: pgmap v1475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:29 smithi132 ceph-mon[37580]: pgmap v1475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:20:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1766.781657 secs 2024-07-26T13:20:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1766.781657 secs 2024-07-26T13:20:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1766.781657 secs 2024-07-26T13:20:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:31 smithi107 ceph-mon[32659]: pgmap v1476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:31 smithi132 ceph-mon[37580]: pgmap v1476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:31 smithi112 ceph-mon[37501]: pgmap v1476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:34 smithi107 ceph-mon[32659]: pgmap v1477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:34 smithi132 ceph-mon[37580]: pgmap v1477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:34 smithi112 ceph-mon[37501]: pgmap v1477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1771.781792 secs 2024-07-26T13:20:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:36 smithi107 ceph-mon[32659]: pgmap v1478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1771.781792 secs 2024-07-26T13:20:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:36 smithi132 ceph-mon[37580]: pgmap v1478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1771.781792 secs 2024-07-26T13:20:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:36 smithi112 ceph-mon[37501]: pgmap v1478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:38.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:38.490+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:38.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:38.490+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:38.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:38.490+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:38 smithi107 ceph-mon[32659]: pgmap v1479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:38 smithi132 ceph-mon[37580]: pgmap v1479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:38 smithi112 ceph-mon[37501]: pgmap v1479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:40 smithi107 ceph-mon[32659]: pgmap v1480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:40 smithi132 ceph-mon[37580]: pgmap v1480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:40.670+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:40.670+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:40.670+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:40.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:40 smithi112 ceph-mon[37501]: pgmap v1480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1776.781927 secs 2024-07-26T13:20:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1776.781927 secs 2024-07-26T13:20:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1776.781927 secs 2024-07-26T13:20:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:42 smithi107 ceph-mon[32659]: pgmap v1481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:42 smithi132 ceph-mon[37580]: pgmap v1481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:42 smithi112 ceph-mon[37501]: pgmap v1481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:44 smithi132 ceph-mon[37580]: pgmap v1482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:44 smithi112 ceph-mon[37501]: pgmap v1482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:44 smithi107 ceph-mon[32659]: pgmap v1482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1781.782046 secs 2024-07-26T13:20:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:46 smithi132 ceph-mon[37580]: pgmap v1483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1781.782046 secs 2024-07-26T13:20:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:46 smithi112 ceph-mon[37501]: pgmap v1483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1781.782046 secs 2024-07-26T13:20:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:46 smithi107 ceph-mon[32659]: pgmap v1483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:47.566 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:47 smithi132 ceph-mon[37580]: pgmap v1484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:47 smithi112 ceph-mon[37501]: pgmap v1484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:47 smithi107 ceph-mon[32659]: pgmap v1484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:50 smithi107 ceph-mon[32659]: pgmap v1485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:50 smithi132 ceph-mon[37580]: pgmap v1485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:50 smithi112 ceph-mon[37501]: pgmap v1485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1786.782165 secs 2024-07-26T13:20:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1786.782165 secs 2024-07-26T13:20:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1786.782165 secs 2024-07-26T13:20:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:52 smithi107 ceph-mon[32659]: pgmap v1486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:52 smithi132 ceph-mon[37580]: pgmap v1486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:52 smithi112 ceph-mon[37501]: pgmap v1486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:53.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:53.505+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:53.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:53.505+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:53.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:20:53.505+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:54 smithi107 ceph-mon[32659]: pgmap v1487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:54 smithi132 ceph-mon[37580]: pgmap v1487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:54 smithi112 ceph-mon[37501]: pgmap v1487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:55.687 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:55.686+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:55.686+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:20:55.688 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:20:55.686+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1791.782291 secs 2024-07-26T13:20:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:56 smithi132 ceph-mon[37580]: pgmap v1488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1791.782291 secs 2024-07-26T13:20:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:56 smithi112 ceph-mon[37501]: pgmap v1488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1791.782291 secs 2024-07-26T13:20:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:56 smithi107 ceph-mon[32659]: pgmap v1488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:58 smithi132 ceph-mon[37580]: pgmap v1489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:58.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:58 smithi112 ceph-mon[37501]: pgmap v1489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:58 smithi107 ceph-mon[32659]: pgmap v1489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:20:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:20:59 smithi112 ceph-mon[37501]: pgmap v1490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:20:59 smithi107 ceph-mon[32659]: pgmap v1490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:20:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:20:59 smithi132 ceph-mon[37580]: pgmap v1490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1796.782409 secs 2024-07-26T13:21:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1796.782409 secs 2024-07-26T13:21:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1796.782409 secs 2024-07-26T13:21:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:01 smithi107 ceph-mon[32659]: pgmap v1491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:01 smithi132 ceph-mon[37580]: pgmap v1491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:01 smithi112 ceph-mon[37501]: pgmap v1491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:04 smithi107 ceph-mon[32659]: pgmap v1492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:04 smithi132 ceph-mon[37580]: pgmap v1492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:04 smithi112 ceph-mon[37501]: pgmap v1492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1801.782526 secs 2024-07-26T13:21:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:06 smithi107 ceph-mon[32659]: pgmap v1493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1801.782526 secs 2024-07-26T13:21:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:06 smithi132 ceph-mon[37580]: pgmap v1493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1801.782526 secs 2024-07-26T13:21:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:06 smithi112 ceph-mon[37501]: pgmap v1493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:08.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:08.522+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:08.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:08.522+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:08.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:08.522+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:08 smithi107 ceph-mon[32659]: pgmap v1494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:08 smithi132 ceph-mon[37580]: pgmap v1494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:08 smithi112 ceph-mon[37501]: pgmap v1494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:10 smithi107 ceph-mon[32659]: pgmap v1495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:10 smithi132 ceph-mon[37580]: pgmap v1495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:10 smithi112 ceph-mon[37501]: pgmap v1495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:10.702+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:10.702+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:10.702+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1806.782646 secs 2024-07-26T13:21:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:11.639+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 msgr2=0x7ff0ec0c1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T13:21:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:11.639+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 msgr2=0x7ff0ec0c1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:21:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:11.639+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c1830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80126f0 tx=0x7ff0f8014040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:21:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:11.639+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c1830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80126f0 tx=0x7ff0f8014040 comp rx=0 tx=0).stop 2024-07-26T13:21:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:11.639+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 msgr2=0x7ff0ec0c1830 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T13:21:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:11.639+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 msgr2=0x7ff0ec0c1830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:21:11.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:11.639+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c1830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:21:11.642 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1806.782646 secs 2024-07-26T13:21:11.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:11.640+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:21:11.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:11.641+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:21:11.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:11.643+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c1e10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f8047040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:21:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1806.782646 secs 2024-07-26T13:21:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:12 smithi132 ceph-mon[37580]: pgmap v1496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:12 smithi112 ceph-mon[37501]: pgmap v1496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:12 smithi107 ceph-mon[32659]: pgmap v1496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:12.943+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:21:12.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:12.943+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:21:12.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:12.992+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:21:12.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:12.993+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:21:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:13.747+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 msgr2=0x7efc5c0c1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T13:21:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:13.747+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 msgr2=0x7efc5c0c1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:21:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:13.747+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c1830 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efc680a7050 tx=0x7efc680e6000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:21:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:13.747+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c1830 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efc680a7050 tx=0x7efc680e6000 comp rx=0 tx=0).stop 2024-07-26T13:21:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:13.747+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 msgr2=0x7efc5c0c1830 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T13:21:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:13.747+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 msgr2=0x7efc5c0c1830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:21:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:13.747+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c1830 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:21:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:13.748+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0c1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:21:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:13.748+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0c1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:21:13.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:13.750+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0c1d70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efc680aacd0 tx=0x7efc680e5000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:21:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:13 smithi107 ceph-mon[32659]: pgmap v1497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:13 smithi132 ceph-mon[37580]: pgmap v1497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:13 smithi112 ceph-mon[37501]: pgmap v1497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:14.056+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:21:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:14.056+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T13:21:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:14.057+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T13:21:14.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:14.057+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:21:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1811.782742 secs 2024-07-26T13:21:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:16 smithi107 ceph-mon[32659]: pgmap v1498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1811.782742 secs 2024-07-26T13:21:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:16 smithi132 ceph-mon[37580]: pgmap v1498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1811.782742 secs 2024-07-26T13:21:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:16 smithi112 ceph-mon[37501]: pgmap v1498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:18 smithi107 ceph-mon[32659]: pgmap v1499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:18 smithi132 ceph-mon[37580]: pgmap v1499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:18 smithi112 ceph-mon[37501]: pgmap v1499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:20 smithi132 ceph-mon[37580]: pgmap v1500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:20 smithi112 ceph-mon[37501]: pgmap v1500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:20 smithi107 ceph-mon[32659]: pgmap v1500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1816.782857 secs 2024-07-26T13:21:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1816.782857 secs 2024-07-26T13:21:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1816.782857 secs 2024-07-26T13:21:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:22 smithi132 ceph-mon[37580]: pgmap v1501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:22 smithi112 ceph-mon[37501]: pgmap v1501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:22 smithi107 ceph-mon[32659]: pgmap v1501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:23.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:23.524+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:23.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:23.524+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:23.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:23.524+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:23 smithi112 ceph-mon[37501]: pgmap v1502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:21:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:23 smithi112 ceph-mon[37501]: pgmap v1503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:21:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:21:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:23 smithi107 ceph-mon[32659]: pgmap v1502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:21:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:23 smithi107 ceph-mon[32659]: pgmap v1503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:21:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:21:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:23 smithi132 ceph-mon[37580]: pgmap v1502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:21:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:23 smithi132 ceph-mon[37580]: pgmap v1503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:21:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:21:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:25.714+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:25.714+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:25.714+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:25 smithi107 ceph-mon[32659]: pgmap v1504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1821.782973 secs 2024-07-26T13:21:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:25 smithi132 ceph-mon[37580]: pgmap v1504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1821.782973 secs 2024-07-26T13:21:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:25 smithi112 ceph-mon[37501]: pgmap v1504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1821.782973 secs 2024-07-26T13:21:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:27 smithi107 ceph-mon[32659]: pgmap v1505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:27 smithi132 ceph-mon[37580]: pgmap v1505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:27 smithi112 ceph-mon[37501]: pgmap v1505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:29 smithi107 ceph-mon[32659]: pgmap v1506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:29 smithi132 ceph-mon[37580]: pgmap v1506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:29 smithi112 ceph-mon[37501]: pgmap v1506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1826.783109 secs 2024-07-26T13:21:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1826.783109 secs 2024-07-26T13:21:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1826.783109 secs 2024-07-26T13:21:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:31 smithi107 ceph-mon[32659]: pgmap v1507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:31 smithi132 ceph-mon[37580]: pgmap v1507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:31 smithi112 ceph-mon[37501]: pgmap v1507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:33 smithi107 ceph-mon[32659]: pgmap v1508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:21:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:33 smithi132 ceph-mon[37580]: pgmap v1508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:21:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:33 smithi112 ceph-mon[37501]: pgmap v1508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:21:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:35 smithi107 ceph-mon[32659]: pgmap v1509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:21:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1831.783242 secs 2024-07-26T13:21:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:35 smithi132 ceph-mon[37580]: pgmap v1509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:21:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1831.783242 secs 2024-07-26T13:21:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:35 smithi112 ceph-mon[37501]: pgmap v1509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:21:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1831.783242 secs 2024-07-26T13:21:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:37 smithi107 ceph-mon[32659]: pgmap v1510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:37 smithi132 ceph-mon[37580]: pgmap v1510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:37 smithi112 ceph-mon[37501]: pgmap v1510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:38.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:38.539+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:38.539+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:38.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:38.539+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:39 smithi107 ceph-mon[32659]: pgmap v1511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:39.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:39 smithi132 ceph-mon[37580]: pgmap v1511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:39 smithi112 ceph-mon[37501]: pgmap v1511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:40.727+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:40.727+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:40.729 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:40.727+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1836.783354 secs 2024-07-26T13:21:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1836.783354 secs 2024-07-26T13:21:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1836.783354 secs 2024-07-26T13:21:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:41 smithi107 ceph-mon[32659]: pgmap v1512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:41 smithi132 ceph-mon[37580]: pgmap v1512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:41 smithi112 ceph-mon[37501]: pgmap v1512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:43 smithi107 ceph-mon[32659]: pgmap v1513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:43 smithi132 ceph-mon[37580]: pgmap v1513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:43 smithi112 ceph-mon[37501]: pgmap v1513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:45 smithi107 ceph-mon[32659]: pgmap v1514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:21:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1841.783449 secs 2024-07-26T13:21:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:45 smithi132 ceph-mon[37580]: pgmap v1514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:21:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1841.783449 secs 2024-07-26T13:21:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:45 smithi112 ceph-mon[37501]: pgmap v1514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:21:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1841.783449 secs 2024-07-26T13:21:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:47 smithi132 ceph-mon[37580]: pgmap v1515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:47 smithi112 ceph-mon[37501]: pgmap v1515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:47 smithi107 ceph-mon[32659]: pgmap v1515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:49 smithi132 ceph-mon[37580]: pgmap v1516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:49 smithi112 ceph-mon[37501]: pgmap v1516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:49 smithi107 ceph-mon[32659]: pgmap v1516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1846.783529 secs 2024-07-26T13:21:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1846.783529 secs 2024-07-26T13:21:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1846.783529 secs 2024-07-26T13:21:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:51 smithi112 ceph-mon[37501]: pgmap v1517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:51 smithi107 ceph-mon[32659]: pgmap v1517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:51 smithi132 ceph-mon[37580]: pgmap v1517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:53.555+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:53.555+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:53.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:21:53.555+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:53 smithi112 ceph-mon[37501]: pgmap v1518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:53 smithi107 ceph-mon[32659]: pgmap v1518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:53 smithi132 ceph-mon[37580]: pgmap v1518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:55.743+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:55.743+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:21:55.743+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:55 smithi112 ceph-mon[37501]: pgmap v1519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:21:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1851.783641 secs 2024-07-26T13:21:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:55 smithi107 ceph-mon[32659]: pgmap v1519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:21:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1851.783641 secs 2024-07-26T13:21:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:55 smithi132 ceph-mon[37580]: pgmap v1519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:21:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1851.783641 secs 2024-07-26T13:21:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:57 smithi112 ceph-mon[37501]: pgmap v1520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:57 smithi107 ceph-mon[32659]: pgmap v1520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:57 smithi132 ceph-mon[37580]: pgmap v1520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:21:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:21:59 smithi112 ceph-mon[37501]: pgmap v1521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:21:59 smithi107 ceph-mon[32659]: pgmap v1521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:00.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:21:59 smithi132 ceph-mon[37580]: pgmap v1521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1856.783756 secs 2024-07-26T13:22:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1856.783756 secs 2024-07-26T13:22:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1856.783756 secs 2024-07-26T13:22:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:01 smithi112 ceph-mon[37501]: pgmap v1522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:01 smithi107 ceph-mon[32659]: pgmap v1522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:01 smithi132 ceph-mon[37580]: pgmap v1522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:03 smithi112 ceph-mon[37501]: pgmap v1523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:03 smithi107 ceph-mon[32659]: pgmap v1523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:04.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:03 smithi132 ceph-mon[37580]: pgmap v1523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:05 smithi112 ceph-mon[37501]: pgmap v1524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:22:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1861.783873 secs 2024-07-26T13:22:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:05 smithi107 ceph-mon[32659]: pgmap v1524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:22:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1861.783873 secs 2024-07-26T13:22:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:05 smithi132 ceph-mon[37580]: pgmap v1524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:22:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1861.783873 secs 2024-07-26T13:22:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:07 smithi112 ceph-mon[37501]: pgmap v1525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:07 smithi107 ceph-mon[32659]: pgmap v1525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:07 smithi132 ceph-mon[37580]: pgmap v1525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:08.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:08.563+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:08.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:08.563+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:08.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:08.563+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:09 smithi112 ceph-mon[37501]: pgmap v1526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:09 smithi107 ceph-mon[32659]: pgmap v1526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:09 smithi132 ceph-mon[37580]: pgmap v1526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:10.745+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:10.745+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:10.745+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1866.783995 secs 2024-07-26T13:22:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1866.783995 secs 2024-07-26T13:22:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1866.783995 secs 2024-07-26T13:22:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:11 smithi112 ceph-mon[37501]: pgmap v1527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:11 smithi107 ceph-mon[32659]: pgmap v1527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:11 smithi132 ceph-mon[37580]: pgmap v1527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:13 smithi112 ceph-mon[37501]: pgmap v1528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:13 smithi107 ceph-mon[32659]: pgmap v1528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:13 smithi132 ceph-mon[37580]: pgmap v1528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:15 smithi112 ceph-mon[37501]: pgmap v1529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:22:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1871.784112 secs 2024-07-26T13:22:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:15 smithi107 ceph-mon[32659]: pgmap v1529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:22:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1871.784112 secs 2024-07-26T13:22:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:15 smithi132 ceph-mon[37580]: pgmap v1529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:22:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1871.784112 secs 2024-07-26T13:22:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:17 smithi112 ceph-mon[37501]: pgmap v1530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:17 smithi107 ceph-mon[32659]: pgmap v1530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:17 smithi132 ceph-mon[37580]: pgmap v1530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:19 smithi112 ceph-mon[37501]: pgmap v1531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:19 smithi107 ceph-mon[32659]: pgmap v1531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:19 smithi132 ceph-mon[37580]: pgmap v1531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1876.784179 secs 2024-07-26T13:22:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1876.784179 secs 2024-07-26T13:22:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1876.784179 secs 2024-07-26T13:22:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:21 smithi112 ceph-mon[37501]: pgmap v1532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:21 smithi107 ceph-mon[32659]: pgmap v1532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:21 smithi132 ceph-mon[37580]: pgmap v1532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:23.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:23.565+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:23.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:23.565+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:23.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:23.565+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: pgmap v1533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:22:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: pgmap v1533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:22:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: pgmap v1533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:22:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:22:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:24 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:22:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:24 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:22:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:24 smithi107 ceph-mon[32659]: pgmap v1534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:22:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:24 smithi107 ceph-mon[32659]: pgmap v1535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:22:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:24 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:22:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:24 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:22:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:24 smithi132 ceph-mon[37580]: pgmap v1534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:22:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:24 smithi132 ceph-mon[37580]: pgmap v1535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:22:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:24 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:22:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:24 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:22:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:24 smithi112 ceph-mon[37501]: pgmap v1534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:22:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:24 smithi112 ceph-mon[37501]: pgmap v1535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:22:25.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:25.753+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:25.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:25.753+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:25.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:25.753+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1881.784319 secs 2024-07-26T13:22:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1881.784319 secs 2024-07-26T13:22:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1881.784319 secs 2024-07-26T13:22:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:26 smithi107 ceph-mon[32659]: pgmap v1536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:26 smithi132 ceph-mon[37580]: pgmap v1536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:26 smithi112 ceph-mon[37501]: pgmap v1536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:28 smithi107 ceph-mon[32659]: pgmap v1537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:28 smithi132 ceph-mon[37580]: pgmap v1537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:28 smithi112 ceph-mon[37501]: pgmap v1537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:30 smithi107 ceph-mon[32659]: pgmap v1538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1886.784418 secs 2024-07-26T13:22:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:30 smithi132 ceph-mon[37580]: pgmap v1538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1886.784418 secs 2024-07-26T13:22:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:30 smithi112 ceph-mon[37501]: pgmap v1538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1886.784418 secs 2024-07-26T13:22:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:32 smithi107 ceph-mon[32659]: pgmap v1539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:32 smithi132 ceph-mon[37580]: pgmap v1539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:32 smithi112 ceph-mon[37501]: pgmap v1539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:34 smithi107 ceph-mon[32659]: pgmap v1540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:34 smithi132 ceph-mon[37580]: pgmap v1540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:34 smithi112 ceph-mon[37501]: pgmap v1540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1891.784529 secs 2024-07-26T13:22:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1891.784529 secs 2024-07-26T13:22:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1891.784529 secs 2024-07-26T13:22:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:36 smithi107 ceph-mon[32659]: pgmap v1541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:36 smithi132 ceph-mon[37580]: pgmap v1541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:37.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:36 smithi112 ceph-mon[37501]: pgmap v1541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:38.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:38.568+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:38.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:38.568+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:38.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:38.568+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:38.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:38 smithi107 ceph-mon[32659]: pgmap v1542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:38 smithi132 ceph-mon[37580]: pgmap v1542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:38 smithi112 ceph-mon[37501]: pgmap v1542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:40.760+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:40.760+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:40.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:40.760+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:40 smithi107 ceph-mon[32659]: pgmap v1543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1896.784655 secs 2024-07-26T13:22:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:40 smithi132 ceph-mon[37580]: pgmap v1543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1896.784655 secs 2024-07-26T13:22:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:40 smithi112 ceph-mon[37501]: pgmap v1543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1896.784655 secs 2024-07-26T13:22:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:42 smithi107 ceph-mon[32659]: pgmap v1544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:42 smithi132 ceph-mon[37580]: pgmap v1544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:42 smithi112 ceph-mon[37501]: pgmap v1544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:44 smithi107 ceph-mon[32659]: pgmap v1545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:44 smithi132 ceph-mon[37580]: pgmap v1545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:44 smithi112 ceph-mon[37501]: pgmap v1545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1901.784721 secs 2024-07-26T13:22:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1901.784721 secs 2024-07-26T13:22:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1901.784721 secs 2024-07-26T13:22:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:46 smithi132 ceph-mon[37580]: pgmap v1546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:46 smithi112 ceph-mon[37501]: pgmap v1546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:46 smithi107 ceph-mon[32659]: pgmap v1546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:48 smithi132 ceph-mon[37580]: pgmap v1547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:48 smithi112 ceph-mon[37501]: pgmap v1547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:48 smithi107 ceph-mon[32659]: pgmap v1547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:50 smithi132 ceph-mon[37580]: pgmap v1548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1906.784844 secs 2024-07-26T13:22:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:50 smithi112 ceph-mon[37501]: pgmap v1548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1906.784844 secs 2024-07-26T13:22:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:50 smithi107 ceph-mon[32659]: pgmap v1548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1906.784844 secs 2024-07-26T13:22:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:52 smithi132 ceph-mon[37580]: pgmap v1549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:52 smithi112 ceph-mon[37501]: pgmap v1549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:52 smithi107 ceph-mon[32659]: pgmap v1549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:53.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:53.579+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:53.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:53.579+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:53.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:22:53.579+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:54 smithi112 ceph-mon[37501]: pgmap v1550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:54 smithi107 ceph-mon[32659]: pgmap v1550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:54 smithi132 ceph-mon[37580]: pgmap v1550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:55.776+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:55.776+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:22:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:22:55.776+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1911.784967 secs 2024-07-26T13:22:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1911.784967 secs 2024-07-26T13:22:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1911.784967 secs 2024-07-26T13:22:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:56 smithi112 ceph-mon[37501]: pgmap v1551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:56 smithi107 ceph-mon[32659]: pgmap v1551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:56 smithi132 ceph-mon[37580]: pgmap v1551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:59.162 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:22:58 smithi107 ceph-mon[32659]: pgmap v1552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:22:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:22:58 smithi112 ceph-mon[37501]: pgmap v1552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:22:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:22:58 smithi132 ceph-mon[37580]: pgmap v1552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:23:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:00 smithi112 ceph-mon[37501]: pgmap v1553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1916.785052 secs 2024-07-26T13:23:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:00 smithi107 ceph-mon[32659]: pgmap v1553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1916.785052 secs 2024-07-26T13:23:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:00 smithi132 ceph-mon[37580]: pgmap v1553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1916.785052 secs 2024-07-26T13:23:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:02 smithi112 ceph-mon[37501]: pgmap v1554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:23:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:02 smithi107 ceph-mon[32659]: pgmap v1554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:23:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:02 smithi132 ceph-mon[37580]: pgmap v1554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:23:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:04 smithi112 ceph-mon[37501]: pgmap v1555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:23:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:04 smithi107 ceph-mon[32659]: pgmap v1555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:23:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:04 smithi132 ceph-mon[37580]: pgmap v1555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:23:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1921.785164 secs 2024-07-26T13:23:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1921.785164 secs 2024-07-26T13:23:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1921.785164 secs 2024-07-26T13:23:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:06 smithi112 ceph-mon[37501]: pgmap v1556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:23:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:06 smithi107 ceph-mon[32659]: pgmap v1556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:23:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:06 smithi132 ceph-mon[37580]: pgmap v1556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:23:08.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:08.596+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:08.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:08.596+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:08.597 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:08.596+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:08 smithi112 ceph-mon[37501]: pgmap v1557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:23:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:08 smithi107 ceph-mon[32659]: pgmap v1557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:23:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:08 smithi132 ceph-mon[37580]: pgmap v1557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:23:10.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:10.793+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:10.793+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:10.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:10.793+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:10 smithi112 ceph-mon[37501]: pgmap v1558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1926.785278 secs 2024-07-26T13:23:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:10 smithi107 ceph-mon[32659]: pgmap v1558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1926.785278 secs 2024-07-26T13:23:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:10 smithi132 ceph-mon[37580]: pgmap v1558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1926.785278 secs 2024-07-26T13:23:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:12 smithi112 ceph-mon[37501]: pgmap v1559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:12 smithi107 ceph-mon[32659]: pgmap v1559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:12 smithi132 ceph-mon[37580]: pgmap v1559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:14 smithi112 ceph-mon[37501]: pgmap v1560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:14 smithi107 ceph-mon[32659]: pgmap v1560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:14 smithi132 ceph-mon[37580]: pgmap v1560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1931.785393 secs 2024-07-26T13:23:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1931.785393 secs 2024-07-26T13:23:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1931.785393 secs 2024-07-26T13:23:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:16 smithi112 ceph-mon[37501]: pgmap v1561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:16 smithi107 ceph-mon[32659]: pgmap v1561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:16 smithi132 ceph-mon[37580]: pgmap v1561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:18 smithi112 ceph-mon[37501]: pgmap v1562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:18 smithi107 ceph-mon[32659]: pgmap v1562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:18 smithi132 ceph-mon[37580]: pgmap v1562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:20 smithi112 ceph-mon[37501]: pgmap v1563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1936.785516 secs 2024-07-26T13:23:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:20 smithi107 ceph-mon[32659]: pgmap v1563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1936.785516 secs 2024-07-26T13:23:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:20 smithi132 ceph-mon[37580]: pgmap v1563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1936.785516 secs 2024-07-26T13:23:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:22 smithi112 ceph-mon[37501]: pgmap v1564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:23:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:22 smithi107 ceph-mon[32659]: pgmap v1564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:23:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:22 smithi132 ceph-mon[37580]: pgmap v1564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:23:23.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:23.611+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:23.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:23.611+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:23.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:23.611+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:24.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: pgmap v1565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:23:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.204 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:25.204 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:25.204 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: pgmap v1565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:23:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:25.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:25.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: pgmap v1565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:25.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:25.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:25.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:23:25.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:25.807+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:25.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:25.807+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:25.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:25.807+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:25 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:23:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:25 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:23:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:25 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:23:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:25 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:23:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:25 smithi112 ceph-mon[37501]: pgmap v1566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:23:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:25 smithi112 ceph-mon[37501]: pgmap v1567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:23:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1941.785606 secs 2024-07-26T13:23:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:25 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:23:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:25 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:23:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:25 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:23:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:25 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:23:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:25 smithi107 ceph-mon[32659]: pgmap v1566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:23:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:25 smithi107 ceph-mon[32659]: pgmap v1567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:23:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1941.785606 secs 2024-07-26T13:23:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:25 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:23:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:25 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:23:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:25 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:23:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:25 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:23:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:25 smithi132 ceph-mon[37580]: pgmap v1566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:23:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:25 smithi132 ceph-mon[37580]: pgmap v1567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:23:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1941.785606 secs 2024-07-26T13:23:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:27 smithi112 ceph-mon[37501]: pgmap v1568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:23:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:27 smithi107 ceph-mon[32659]: pgmap v1568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:23:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:27 smithi132 ceph-mon[37580]: pgmap v1568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:23:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:29 smithi112 ceph-mon[37501]: pgmap v1569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:23:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:29 smithi107 ceph-mon[32659]: pgmap v1569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:23:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:29 smithi132 ceph-mon[37580]: pgmap v1569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:23:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1946.785731 secs 2024-07-26T13:23:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1946.785731 secs 2024-07-26T13:23:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1946.785731 secs 2024-07-26T13:23:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:31 smithi107 ceph-mon[32659]: pgmap v1570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:23:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:31 smithi132 ceph-mon[37580]: pgmap v1570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:23:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:31 smithi112 ceph-mon[37501]: pgmap v1570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:23:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:33 smithi107 ceph-mon[32659]: pgmap v1571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:23:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:33 smithi132 ceph-mon[37580]: pgmap v1571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:23:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:33 smithi112 ceph-mon[37501]: pgmap v1571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:23:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:35 smithi107 ceph-mon[32659]: pgmap v1572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:23:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1951.785818 secs 2024-07-26T13:23:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:35 smithi132 ceph-mon[37580]: pgmap v1572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:23:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1951.785818 secs 2024-07-26T13:23:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:35 smithi112 ceph-mon[37501]: pgmap v1572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:23:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1951.785818 secs 2024-07-26T13:23:38.252 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:37 smithi112 ceph-mon[37501]: pgmap v1573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:37 smithi107 ceph-mon[32659]: pgmap v1573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:37 smithi132 ceph-mon[37580]: pgmap v1573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:38.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:38.624+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:38.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:38.624+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:38.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:38.624+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:39 smithi107 ceph-mon[32659]: pgmap v1574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:39 smithi132 ceph-mon[37580]: pgmap v1574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:39 smithi112 ceph-mon[37501]: pgmap v1574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:40.822+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:40.822+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:40.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:40.822+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1956.785959 secs 2024-07-26T13:23:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1956.785959 secs 2024-07-26T13:23:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1956.785959 secs 2024-07-26T13:23:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:41 smithi107 ceph-mon[32659]: pgmap v1575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:41 smithi132 ceph-mon[37580]: pgmap v1575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:41 smithi112 ceph-mon[37501]: pgmap v1575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:44 smithi107 ceph-mon[32659]: pgmap v1576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:44 smithi132 ceph-mon[37580]: pgmap v1576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:44 smithi112 ceph-mon[37501]: pgmap v1576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:46 smithi107 ceph-mon[32659]: pgmap v1577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1961.786026 secs 2024-07-26T13:23:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:46 smithi132 ceph-mon[37580]: pgmap v1577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:46.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1961.786026 secs 2024-07-26T13:23:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:46 smithi112 ceph-mon[37501]: pgmap v1577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1961.786026 secs 2024-07-26T13:23:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:48 smithi107 ceph-mon[32659]: pgmap v1578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:48 smithi132 ceph-mon[37580]: pgmap v1578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:48 smithi112 ceph-mon[37501]: pgmap v1578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:50 smithi107 ceph-mon[32659]: pgmap v1579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:50 smithi132 ceph-mon[37580]: pgmap v1579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:50 smithi112 ceph-mon[37501]: pgmap v1579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1966.786142 secs 2024-07-26T13:23:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1966.786142 secs 2024-07-26T13:23:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1966.786142 secs 2024-07-26T13:23:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:52 smithi107 ceph-mon[32659]: pgmap v1580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:52 smithi132 ceph-mon[37580]: pgmap v1580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:52 smithi112 ceph-mon[37501]: pgmap v1580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:53.632+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:53.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:53.632+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:53.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:23:53.632+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:54 smithi107 ceph-mon[32659]: pgmap v1581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:54.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:54 smithi132 ceph-mon[37580]: pgmap v1581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:54 smithi112 ceph-mon[37501]: pgmap v1581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:55.831+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:55.831+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:55.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:23:55.831+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:23:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:56 smithi107 ceph-mon[32659]: pgmap v1582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1971.786263 secs 2024-07-26T13:23:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:56 smithi132 ceph-mon[37580]: pgmap v1582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1971.786263 secs 2024-07-26T13:23:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:56 smithi112 ceph-mon[37501]: pgmap v1582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1971.786263 secs 2024-07-26T13:23:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:23:58 smithi132 ceph-mon[37580]: pgmap v1583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:23:58 smithi112 ceph-mon[37501]: pgmap v1583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:23:58 smithi107 ceph-mon[32659]: pgmap v1583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:00 smithi132 ceph-mon[37580]: pgmap v1584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:00 smithi112 ceph-mon[37501]: pgmap v1584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:00 smithi107 ceph-mon[32659]: pgmap v1584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1976.786410 secs 2024-07-26T13:24:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1976.786410 secs 2024-07-26T13:24:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1976.786410 secs 2024-07-26T13:24:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:02 smithi112 ceph-mon[37501]: pgmap v1585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:02 smithi107 ceph-mon[32659]: pgmap v1585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:02 smithi132 ceph-mon[37580]: pgmap v1585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:04 smithi112 ceph-mon[37501]: pgmap v1586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:04 smithi107 ceph-mon[32659]: pgmap v1586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:04 smithi132 ceph-mon[37580]: pgmap v1586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:06 smithi112 ceph-mon[37501]: pgmap v1587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1981.786540 secs 2024-07-26T13:24:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:06 smithi107 ceph-mon[32659]: pgmap v1587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1981.786540 secs 2024-07-26T13:24:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:06 smithi132 ceph-mon[37580]: pgmap v1587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1981.786540 secs 2024-07-26T13:24:08.344 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:08 smithi107 ceph-mon[32659]: pgmap v1588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:08 smithi112 ceph-mon[37501]: pgmap v1588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:08 smithi132 ceph-mon[37580]: pgmap v1588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:08.634+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:08.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:08.634+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:08.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:08.634+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:10 smithi112 ceph-mon[37501]: pgmap v1589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:10 smithi107 ceph-mon[32659]: pgmap v1589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:10 smithi132 ceph-mon[37580]: pgmap v1589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:10.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:10.834+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:10.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:10.834+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:10.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:10.834+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1986.786661 secs 2024-07-26T13:24:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1986.786661 secs 2024-07-26T13:24:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1986.786661 secs 2024-07-26T13:24:11.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:11.972+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:24:11.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:11.973+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:24:12.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:12.022+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:24:12.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:12.023+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:24:12.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:12 smithi107 ceph-mon[32659]: pgmap v1590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:12 smithi112 ceph-mon[37501]: pgmap v1590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:12 smithi132 ceph-mon[37580]: pgmap v1590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:12.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:12.973+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:24:12.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:12.973+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:24:13.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:13.022+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:24:13.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:13.023+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:24:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:14.088+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:24:14.089 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:14.088+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T13:24:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:14.089+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T13:24:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:14.089+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:24:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:14.089+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:24:14.091 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:14.089+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:24:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:14.090+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:24:14.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:14.090+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:24:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:14 smithi112 ceph-mon[37501]: pgmap v1591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:14 smithi107 ceph-mon[32659]: pgmap v1591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:14 smithi132 ceph-mon[37580]: pgmap v1591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:16 smithi112 ceph-mon[37501]: pgmap v1592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1991.786799 secs 2024-07-26T13:24:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:16 smithi107 ceph-mon[32659]: pgmap v1592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1991.786799 secs 2024-07-26T13:24:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:16 smithi132 ceph-mon[37580]: pgmap v1592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1991.786799 secs 2024-07-26T13:24:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:18 smithi112 ceph-mon[37501]: pgmap v1593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:18 smithi107 ceph-mon[32659]: pgmap v1593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:18 smithi132 ceph-mon[37580]: pgmap v1593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:20 smithi112 ceph-mon[37501]: pgmap v1594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:20 smithi107 ceph-mon[32659]: pgmap v1594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:20 smithi132 ceph-mon[37580]: pgmap v1594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 1996.786934 secs 2024-07-26T13:24:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 1996.786934 secs 2024-07-26T13:24:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 1996.786934 secs 2024-07-26T13:24:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:22 smithi112 ceph-mon[37501]: pgmap v1595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:22 smithi107 ceph-mon[32659]: pgmap v1595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:22 smithi132 ceph-mon[37580]: pgmap v1595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:23.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:23.650+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:23.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:23.650+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:23.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:23.650+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:24 smithi112 ceph-mon[37501]: pgmap v1596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:24 smithi107 ceph-mon[32659]: pgmap v1596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:24 smithi132 ceph-mon[37580]: pgmap v1596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:25.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:25.849+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:25.849+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:25.849+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:26 smithi112 ceph-mon[37501]: pgmap v1597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2001.787056 secs 2024-07-26T13:24:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:26 smithi107 ceph-mon[32659]: pgmap v1597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2001.787056 secs 2024-07-26T13:24:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:26 smithi132 ceph-mon[37580]: pgmap v1597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2001.787056 secs 2024-07-26T13:24:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:24:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:27 smithi112 ceph-mon[37501]: pgmap v1598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:24:27.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:27 smithi112 ceph-mon[37501]: pgmap v1599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:24:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:27 smithi107 ceph-mon[32659]: pgmap v1598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:24:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:27 smithi107 ceph-mon[32659]: pgmap v1599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:24:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:27 smithi132 ceph-mon[37580]: pgmap v1598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:24:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:27 smithi132 ceph-mon[37580]: pgmap v1599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:24:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:29 smithi112 ceph-mon[37501]: pgmap v1600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:24:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:29 smithi107 ceph-mon[32659]: pgmap v1600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:24:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:29 smithi132 ceph-mon[37580]: pgmap v1600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:24:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:31 smithi112 ceph-mon[37501]: pgmap v1601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2006.787194 secs 2024-07-26T13:24:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:31 smithi107 ceph-mon[32659]: pgmap v1601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2006.787194 secs 2024-07-26T13:24:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:31 smithi132 ceph-mon[37580]: pgmap v1601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:24:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2006.787194 secs 2024-07-26T13:24:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:33 smithi112 ceph-mon[37501]: pgmap v1602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:33 smithi107 ceph-mon[32659]: pgmap v1602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:33 smithi132 ceph-mon[37580]: pgmap v1602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:35 smithi112 ceph-mon[37501]: pgmap v1603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:24:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:35 smithi107 ceph-mon[32659]: pgmap v1603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:24:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:35 smithi132 ceph-mon[37580]: pgmap v1603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:24:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2011.787307 secs 2024-07-26T13:24:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2011.787307 secs 2024-07-26T13:24:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2011.787307 secs 2024-07-26T13:24:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:37 smithi112 ceph-mon[37501]: pgmap v1604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:37 smithi107 ceph-mon[32659]: pgmap v1604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:37 smithi132 ceph-mon[37580]: pgmap v1604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:24:38.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:38.664+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:38.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:38.664+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:38.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:38.664+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:39 smithi107 ceph-mon[32659]: pgmap v1605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:39 smithi132 ceph-mon[37580]: pgmap v1605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:39 smithi112 ceph-mon[37501]: pgmap v1605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:40.849+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:40.849+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:40.849+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:41 smithi107 ceph-mon[32659]: pgmap v1606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2016.787390 secs 2024-07-26T13:24:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:41 smithi132 ceph-mon[37580]: pgmap v1606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2016.787390 secs 2024-07-26T13:24:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:41 smithi112 ceph-mon[37501]: pgmap v1606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2016.787390 secs 2024-07-26T13:24:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:43 smithi107 ceph-mon[32659]: pgmap v1607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:43 smithi132 ceph-mon[37580]: pgmap v1607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:43 smithi112 ceph-mon[37501]: pgmap v1607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:45 smithi107 ceph-mon[32659]: pgmap v1608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:45 smithi132 ceph-mon[37580]: pgmap v1608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:45 smithi112 ceph-mon[37501]: pgmap v1608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2021.787507 secs 2024-07-26T13:24:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2021.787507 secs 2024-07-26T13:24:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2021.787507 secs 2024-07-26T13:24:47.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:47 smithi107 ceph-mon[32659]: pgmap v1609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:47 smithi132 ceph-mon[37580]: pgmap v1609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:47 smithi112 ceph-mon[37501]: pgmap v1609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:49 smithi107 ceph-mon[32659]: pgmap v1610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:49 smithi132 ceph-mon[37580]: pgmap v1610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:49 smithi112 ceph-mon[37501]: pgmap v1610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:51 smithi107 ceph-mon[32659]: pgmap v1611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2026.787623 secs 2024-07-26T13:24:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:51 smithi132 ceph-mon[37580]: pgmap v1611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2026.787623 secs 2024-07-26T13:24:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:51 smithi112 ceph-mon[37501]: pgmap v1611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2026.787623 secs 2024-07-26T13:24:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:53 smithi107 ceph-mon[32659]: pgmap v1612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:53 smithi132 ceph-mon[37580]: pgmap v1612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:53.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:53.678+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:53.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:53.679+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:53.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:24:53.679+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:53 smithi112 ceph-mon[37501]: pgmap v1612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:55 smithi107 ceph-mon[32659]: pgmap v1613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:55 smithi132 ceph-mon[37580]: pgmap v1613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:55 smithi112 ceph-mon[37501]: pgmap v1613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:55.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:55.852+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:55.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:55.852+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:24:55.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:24:55.852+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2031.787762 secs 2024-07-26T13:24:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2031.787762 secs 2024-07-26T13:24:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2031.787762 secs 2024-07-26T13:24:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:57 smithi107 ceph-mon[32659]: pgmap v1614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:57 smithi132 ceph-mon[37580]: pgmap v1614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:57 smithi112 ceph-mon[37501]: pgmap v1614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:24:59 smithi107 ceph-mon[32659]: pgmap v1615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:24:59 smithi132 ceph-mon[37580]: pgmap v1615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:24:59 smithi112 ceph-mon[37501]: pgmap v1615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:01 smithi107 ceph-mon[32659]: pgmap v1616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2036.787894 secs 2024-07-26T13:25:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:01 smithi132 ceph-mon[37580]: pgmap v1616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2036.787894 secs 2024-07-26T13:25:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:01 smithi112 ceph-mon[37501]: pgmap v1616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2036.787894 secs 2024-07-26T13:25:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:03 smithi107 ceph-mon[32659]: pgmap v1617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:03 smithi132 ceph-mon[37580]: pgmap v1617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:03 smithi112 ceph-mon[37501]: pgmap v1617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:05 smithi132 ceph-mon[37580]: pgmap v1618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:05 smithi112 ceph-mon[37501]: pgmap v1618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:05 smithi107 ceph-mon[32659]: pgmap v1618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2041.788016 secs 2024-07-26T13:25:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2041.788016 secs 2024-07-26T13:25:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2041.788016 secs 2024-07-26T13:25:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:07 smithi132 ceph-mon[37580]: pgmap v1619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:07 smithi112 ceph-mon[37501]: pgmap v1619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:07 smithi107 ceph-mon[32659]: pgmap v1619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:08.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:08.693+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:08.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:08.693+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:08.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:08.693+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:09 smithi112 ceph-mon[37501]: pgmap v1620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:09 smithi107 ceph-mon[32659]: pgmap v1620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:09 smithi132 ceph-mon[37580]: pgmap v1620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:10.869+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:10.869+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:10.869+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:11 smithi112 ceph-mon[37501]: pgmap v1621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2046.788136 secs 2024-07-26T13:25:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:11 smithi107 ceph-mon[32659]: pgmap v1621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2046.788136 secs 2024-07-26T13:25:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:11 smithi132 ceph-mon[37580]: pgmap v1621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2046.788136 secs 2024-07-26T13:25:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:13 smithi112 ceph-mon[37501]: pgmap v1622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:13 smithi107 ceph-mon[32659]: pgmap v1622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:13 smithi132 ceph-mon[37580]: pgmap v1622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:15 smithi112 ceph-mon[37501]: pgmap v1623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:15 smithi107 ceph-mon[32659]: pgmap v1623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:15 smithi132 ceph-mon[37580]: pgmap v1623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2051.788239 secs 2024-07-26T13:25:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2051.788239 secs 2024-07-26T13:25:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2051.788239 secs 2024-07-26T13:25:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:17 smithi112 ceph-mon[37501]: pgmap v1624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:17 smithi107 ceph-mon[32659]: pgmap v1624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:17 smithi132 ceph-mon[37580]: pgmap v1624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:19 smithi112 ceph-mon[37501]: pgmap v1625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:19 smithi107 ceph-mon[32659]: pgmap v1625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:19 smithi132 ceph-mon[37580]: pgmap v1625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:21.614 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:21 smithi132 ceph-mon[37580]: pgmap v1626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:21.614 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2056.788360 secs 2024-07-26T13:25:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:21 smithi112 ceph-mon[37501]: pgmap v1626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2056.788360 secs 2024-07-26T13:25:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:21 smithi107 ceph-mon[32659]: pgmap v1626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2056.788360 secs 2024-07-26T13:25:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:23 smithi112 ceph-mon[37501]: pgmap v1627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:23.708+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:23.708+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:23.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:23.708+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:23.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:23 smithi107 ceph-mon[32659]: pgmap v1627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:23 smithi132 ceph-mon[37580]: pgmap v1627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:24 smithi112 ceph-mon[37501]: pgmap v1628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:24 smithi107 ceph-mon[32659]: pgmap v1628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:24.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:24 smithi132 ceph-mon[37580]: pgmap v1628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2061.788479 secs 2024-07-26T13:25:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2061.788479 secs 2024-07-26T13:25:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:25.874+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:25.874+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:25.874+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2061.788479 secs 2024-07-26T13:25:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:26 smithi107 ceph-mon[32659]: pgmap v1629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:26 smithi132 ceph-mon[37580]: pgmap v1629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:26 smithi112 ceph-mon[37501]: pgmap v1629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:28 smithi112 ceph-mon[37501]: pgmap v1630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:28 smithi112 ceph-mon[37501]: pgmap v1631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:28 smithi107 ceph-mon[32659]: pgmap v1630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:28 smithi107 ceph-mon[32659]: pgmap v1631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:28 smithi132 ceph-mon[37580]: pgmap v1630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:28 smithi132 ceph-mon[37580]: pgmap v1631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:25:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:30 smithi107 ceph-mon[32659]: pgmap v1632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2066.788616 secs 2024-07-26T13:25:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:30 smithi132 ceph-mon[37580]: pgmap v1632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2066.788616 secs 2024-07-26T13:25:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:30 smithi112 ceph-mon[37501]: pgmap v1632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2066.788616 secs 2024-07-26T13:25:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:32 smithi107 ceph-mon[32659]: pgmap v1633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:32 smithi132 ceph-mon[37580]: pgmap v1633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:32 smithi112 ceph-mon[37501]: pgmap v1633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:34 smithi107 ceph-mon[32659]: pgmap v1634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:25:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:34 smithi132 ceph-mon[37580]: pgmap v1634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:25:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:34 smithi112 ceph-mon[37501]: pgmap v1634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:25:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2071.788711 secs 2024-07-26T13:25:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2071.788711 secs 2024-07-26T13:25:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2071.788711 secs 2024-07-26T13:25:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:37 smithi107 ceph-mon[32659]: pgmap v1635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:37 smithi132 ceph-mon[37580]: pgmap v1635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:37 smithi112 ceph-mon[37501]: pgmap v1635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:25:38.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:38.721+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:38.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:38.721+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:38.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:38.721+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:39 smithi107 ceph-mon[32659]: pgmap v1636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:25:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:39 smithi132 ceph-mon[37580]: pgmap v1636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:25:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:39 smithi112 ceph-mon[37501]: pgmap v1636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:25:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:40.889+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:40.889+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:40.889+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:41 smithi107 ceph-mon[32659]: pgmap v1637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2076.788865 secs 2024-07-26T13:25:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:41 smithi132 ceph-mon[37580]: pgmap v1637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2076.788865 secs 2024-07-26T13:25:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:41 smithi112 ceph-mon[37501]: pgmap v1637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2076.788865 secs 2024-07-26T13:25:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:43 smithi107 ceph-mon[32659]: pgmap v1638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:43 smithi132 ceph-mon[37580]: pgmap v1638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:43 smithi112 ceph-mon[37501]: pgmap v1638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:45 smithi132 ceph-mon[37580]: pgmap v1639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:45 smithi112 ceph-mon[37501]: pgmap v1639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:45 smithi107 ceph-mon[32659]: pgmap v1639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2081.788990 secs 2024-07-26T13:25:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2081.788990 secs 2024-07-26T13:25:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2081.788990 secs 2024-07-26T13:25:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:47 smithi132 ceph-mon[37580]: pgmap v1640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:47 smithi112 ceph-mon[37501]: pgmap v1640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:47 smithi107 ceph-mon[32659]: pgmap v1640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:49 smithi112 ceph-mon[37501]: pgmap v1641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:49 smithi107 ceph-mon[32659]: pgmap v1641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:49 smithi132 ceph-mon[37580]: pgmap v1641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:25:51.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:51 smithi132 ceph-mon[37580]: pgmap v1642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:51.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2086.789082 secs 2024-07-26T13:25:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:51 smithi112 ceph-mon[37501]: pgmap v1642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2086.789082 secs 2024-07-26T13:25:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:51 smithi107 ceph-mon[32659]: pgmap v1642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2086.789082 secs 2024-07-26T13:25:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:53 smithi112 ceph-mon[37501]: pgmap v1643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:53 smithi107 ceph-mon[32659]: pgmap v1643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:53 smithi132 ceph-mon[37580]: pgmap v1643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:25:53.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:53.737+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:53.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:53.737+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:53.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:25:53.737+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:55 smithi112 ceph-mon[37501]: pgmap v1644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:55 smithi107 ceph-mon[32659]: pgmap v1644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:55 smithi132 ceph-mon[37580]: pgmap v1644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:55.905+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:55.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:55.905+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:55.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:25:55.905+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:25:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2091.789155 secs 2024-07-26T13:25:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2091.789155 secs 2024-07-26T13:25:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2091.789155 secs 2024-07-26T13:25:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:57 smithi112 ceph-mon[37501]: pgmap v1645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:25:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:57 smithi107 ceph-mon[32659]: pgmap v1645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:25:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:57 smithi132 ceph-mon[37580]: pgmap v1645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:25:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:25:58 smithi112 ceph-mon[37501]: pgmap v1646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:25:58 smithi107 ceph-mon[32659]: pgmap v1646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:25:58 smithi132 ceph-mon[37580]: pgmap v1646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:26:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:01 smithi107 ceph-mon[32659]: pgmap v1647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:26:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2096.789223 secs 2024-07-26T13:26:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:01 smithi132 ceph-mon[37580]: pgmap v1647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:26:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2096.789223 secs 2024-07-26T13:26:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:01 smithi112 ceph-mon[37501]: pgmap v1647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:26:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2096.789223 secs 2024-07-26T13:26:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:03 smithi107 ceph-mon[32659]: pgmap v1648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:26:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:03 smithi132 ceph-mon[37580]: pgmap v1648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:26:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:03 smithi112 ceph-mon[37501]: pgmap v1648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:26:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:05 smithi107 ceph-mon[32659]: pgmap v1649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:26:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:05 smithi132 ceph-mon[37580]: pgmap v1649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:26:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:05 smithi112 ceph-mon[37501]: pgmap v1649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:26:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2101.789321 secs 2024-07-26T13:26:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2101.789321 secs 2024-07-26T13:26:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2101.789321 secs 2024-07-26T13:26:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:07 smithi107 ceph-mon[32659]: pgmap v1650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:07 smithi132 ceph-mon[37580]: pgmap v1650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:07 smithi112 ceph-mon[37501]: pgmap v1650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:08.746+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:08.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:08.746+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:08.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:08.746+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:09 smithi107 ceph-mon[32659]: pgmap v1651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:09 smithi132 ceph-mon[37580]: pgmap v1651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:09 smithi112 ceph-mon[37501]: pgmap v1651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:10.914+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:10.914+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:10.914+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:11 smithi107 ceph-mon[32659]: pgmap v1652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2106.789455 secs 2024-07-26T13:26:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:11 smithi132 ceph-mon[37580]: pgmap v1652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2106.789455 secs 2024-07-26T13:26:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:11 smithi112 ceph-mon[37501]: pgmap v1652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2106.789455 secs 2024-07-26T13:26:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:13 smithi132 ceph-mon[37580]: pgmap v1653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:13 smithi112 ceph-mon[37501]: pgmap v1653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:13 smithi107 ceph-mon[32659]: pgmap v1653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:15 smithi132 ceph-mon[37580]: pgmap v1654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:15 smithi112 ceph-mon[37501]: pgmap v1654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:15 smithi107 ceph-mon[32659]: pgmap v1654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2111.789574 secs 2024-07-26T13:26:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2111.789574 secs 2024-07-26T13:26:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2111.789574 secs 2024-07-26T13:26:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:17 smithi112 ceph-mon[37501]: pgmap v1655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:17 smithi107 ceph-mon[32659]: pgmap v1655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:17 smithi132 ceph-mon[37580]: pgmap v1655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:19 smithi112 ceph-mon[37501]: pgmap v1656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:19 smithi107 ceph-mon[32659]: pgmap v1656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:19.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:19 smithi132 ceph-mon[37580]: pgmap v1656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:21.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:21 smithi132 ceph-mon[37580]: pgmap v1657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:21.363 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2116.789686 secs 2024-07-26T13:26:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:21 smithi112 ceph-mon[37501]: pgmap v1657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2116.789686 secs 2024-07-26T13:26:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:21 smithi107 ceph-mon[32659]: pgmap v1657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2116.789686 secs 2024-07-26T13:26:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:23 smithi112 ceph-mon[37501]: pgmap v1658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:23 smithi107 ceph-mon[32659]: pgmap v1658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:23 smithi132 ceph-mon[37580]: pgmap v1658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:23.748+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:23.748+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:23.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:23.748+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:25 smithi112 ceph-mon[37501]: pgmap v1659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:25 smithi107 ceph-mon[32659]: pgmap v1659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:25 smithi132 ceph-mon[37580]: pgmap v1659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:25.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:25.922+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:25.922+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:25.922+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2121.789805 secs 2024-07-26T13:26:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2121.789805 secs 2024-07-26T13:26:26.604 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2121.789805 secs 2024-07-26T13:26:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:27 smithi112 ceph-mon[37501]: pgmap v1660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:27 smithi107 ceph-mon[32659]: pgmap v1660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:27 smithi132 ceph-mon[37580]: pgmap v1660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:29.375 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:29 smithi132 ceph-mon[37580]: pgmap v1661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:29.375 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:29.375 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.375 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.375 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.375 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:29 smithi107 ceph-mon[32659]: pgmap v1661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:29.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:29.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:29 smithi112 ceph-mon[37501]: pgmap v1661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:30 smithi107 ceph-mon[32659]: pgmap v1662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2126.789875 secs 2024-07-26T13:26:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:30 smithi132 ceph-mon[37580]: pgmap v1662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2126.789875 secs 2024-07-26T13:26:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:30 smithi112 ceph-mon[37501]: pgmap v1662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2126.789875 secs 2024-07-26T13:26:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:32 smithi112 ceph-mon[37501]: pgmap v1663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:32 smithi107 ceph-mon[32659]: pgmap v1663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:32 smithi132 ceph-mon[37580]: pgmap v1663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:26:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:26:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:33 smithi112 ceph-mon[37501]: pgmap v1664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:26:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:26:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:33 smithi107 ceph-mon[32659]: pgmap v1664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:26:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:26:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:33 smithi132 ceph-mon[37580]: pgmap v1664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:26:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:35 smithi107 ceph-mon[32659]: pgmap v1665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:26:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2131.789959 secs 2024-07-26T13:26:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:35 smithi132 ceph-mon[37580]: pgmap v1665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:26:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2131.789959 secs 2024-07-26T13:26:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:35 smithi112 ceph-mon[37501]: pgmap v1665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:26:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2131.789959 secs 2024-07-26T13:26:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:37 smithi107 ceph-mon[32659]: pgmap v1666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:37 smithi132 ceph-mon[37580]: pgmap v1666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:37 smithi112 ceph-mon[37501]: pgmap v1666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:38.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:38.762+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:38.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:38.762+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:38.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:38.762+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:39 smithi107 ceph-mon[32659]: pgmap v1667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:39 smithi132 ceph-mon[37580]: pgmap v1667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:39 smithi112 ceph-mon[37501]: pgmap v1667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:26:40.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:40.929+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:40.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:40.929+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:40.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:40.929+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2136.790098 secs 2024-07-26T13:26:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2136.790098 secs 2024-07-26T13:26:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2136.790098 secs 2024-07-26T13:26:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:41 smithi107 ceph-mon[32659]: pgmap v1668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:41 smithi132 ceph-mon[37580]: pgmap v1668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:41 smithi112 ceph-mon[37501]: pgmap v1668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:43 smithi132 ceph-mon[37580]: pgmap v1669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:43 smithi112 ceph-mon[37501]: pgmap v1669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:43 smithi107 ceph-mon[32659]: pgmap v1669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:45 smithi112 ceph-mon[37501]: pgmap v1670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2141.790238 secs 2024-07-26T13:26:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:45 smithi107 ceph-mon[32659]: pgmap v1670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2141.790238 secs 2024-07-26T13:26:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:45 smithi132 ceph-mon[37580]: pgmap v1670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2141.790238 secs 2024-07-26T13:26:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:47 smithi112 ceph-mon[37501]: pgmap v1671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:47 smithi107 ceph-mon[32659]: pgmap v1671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:47 smithi132 ceph-mon[37580]: pgmap v1671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:49 smithi112 ceph-mon[37501]: pgmap v1672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:49 smithi107 ceph-mon[32659]: pgmap v1672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:49 smithi132 ceph-mon[37580]: pgmap v1672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2146.790368 secs 2024-07-26T13:26:51.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2146.790368 secs 2024-07-26T13:26:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2146.790368 secs 2024-07-26T13:26:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:51 smithi112 ceph-mon[37501]: pgmap v1673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:51 smithi107 ceph-mon[32659]: pgmap v1673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:51 smithi132 ceph-mon[37580]: pgmap v1673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:53.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:53.778+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:53.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:53.778+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:53.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:26:53.778+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:53 smithi112 ceph-mon[37501]: pgmap v1674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:53 smithi107 ceph-mon[32659]: pgmap v1674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:54.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:53 smithi132 ceph-mon[37580]: pgmap v1674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:26:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:55.944+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:55.944+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:26:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:26:55.944+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:55 smithi112 ceph-mon[37501]: pgmap v1675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2151.790514 secs 2024-07-26T13:26:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:55 smithi107 ceph-mon[32659]: pgmap v1675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2151.790514 secs 2024-07-26T13:26:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:55 smithi132 ceph-mon[37580]: pgmap v1675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2151.790514 secs 2024-07-26T13:26:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:57 smithi112 ceph-mon[37501]: pgmap v1676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:57 smithi107 ceph-mon[32659]: pgmap v1676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:26:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:57 smithi132 ceph-mon[37580]: pgmap v1676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:26:59 smithi112 ceph-mon[37501]: pgmap v1677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:00.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:26:59 smithi107 ceph-mon[32659]: pgmap v1677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:26:59 smithi132 ceph-mon[37580]: pgmap v1677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2156.790638 secs 2024-07-26T13:27:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2156.790638 secs 2024-07-26T13:27:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2156.790638 secs 2024-07-26T13:27:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:01 smithi112 ceph-mon[37501]: pgmap v1678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:01 smithi107 ceph-mon[32659]: pgmap v1678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:01 smithi132 ceph-mon[37580]: pgmap v1678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:03 smithi112 ceph-mon[37501]: pgmap v1679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:03 smithi107 ceph-mon[32659]: pgmap v1679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:03 smithi132 ceph-mon[37580]: pgmap v1679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:05 smithi112 ceph-mon[37501]: pgmap v1680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2161.790759 secs 2024-07-26T13:27:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:05 smithi107 ceph-mon[32659]: pgmap v1680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2161.790759 secs 2024-07-26T13:27:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:05 smithi132 ceph-mon[37580]: pgmap v1680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2161.790759 secs 2024-07-26T13:27:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:07 smithi112 ceph-mon[37501]: pgmap v1681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:27:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:07 smithi107 ceph-mon[32659]: pgmap v1681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:27:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:07 smithi132 ceph-mon[37580]: pgmap v1681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:27:08.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:08.790+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:08.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:08.791+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:08.792 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:08.791+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:09 smithi112 ceph-mon[37501]: pgmap v1682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:09 smithi107 ceph-mon[32659]: pgmap v1682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:09 smithi132 ceph-mon[37580]: pgmap v1682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:10.960+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:10.960+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:10.961 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:10.960+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2166.790881 secs 2024-07-26T13:27:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2166.790881 secs 2024-07-26T13:27:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2166.790881 secs 2024-07-26T13:27:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:11 smithi112 ceph-mon[37501]: pgmap v1683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:11 smithi107 ceph-mon[32659]: pgmap v1683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:11 smithi132 ceph-mon[37580]: pgmap v1683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:13.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:13.003+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:27:13.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:13.004+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:27:13.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:13.051+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:27:13.052 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:13.052+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:27:14.120 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:14.119+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:27:14.120 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:14.119+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:27:14.121 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:14.120+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:27:14.121 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:14.120+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:27:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:13 smithi112 ceph-mon[37501]: pgmap v1684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:13 smithi107 ceph-mon[32659]: pgmap v1684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:13 smithi132 ceph-mon[37580]: pgmap v1684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:15 smithi112 ceph-mon[37501]: pgmap v1685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2171.791006 secs 2024-07-26T13:27:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:15 smithi107 ceph-mon[32659]: pgmap v1685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2171.791006 secs 2024-07-26T13:27:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:15 smithi132 ceph-mon[37580]: pgmap v1685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2171.791006 secs 2024-07-26T13:27:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:17 smithi112 ceph-mon[37501]: pgmap v1686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:17 smithi107 ceph-mon[32659]: pgmap v1686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:17 smithi132 ceph-mon[37580]: pgmap v1686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:19 smithi112 ceph-mon[37501]: pgmap v1687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:19 smithi107 ceph-mon[32659]: pgmap v1687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:19 smithi132 ceph-mon[37580]: pgmap v1687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2176.791139 secs 2024-07-26T13:27:21.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2176.791139 secs 2024-07-26T13:27:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2176.791139 secs 2024-07-26T13:27:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:21 smithi112 ceph-mon[37501]: pgmap v1688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:21 smithi107 ceph-mon[32659]: pgmap v1688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:21 smithi132 ceph-mon[37580]: pgmap v1688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:23.807 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:23.806+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:23.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:23.806+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:23.808 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:23.806+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:24.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:23 smithi107 ceph-mon[32659]: pgmap v1689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:23 smithi132 ceph-mon[37580]: pgmap v1689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:23 smithi112 ceph-mon[37501]: pgmap v1689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:25.976+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:25.976+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:25.976+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:25 smithi107 ceph-mon[32659]: pgmap v1690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2181.791214 secs 2024-07-26T13:27:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:25 smithi132 ceph-mon[37580]: pgmap v1690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2181.791214 secs 2024-07-26T13:27:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:25 smithi112 ceph-mon[37501]: pgmap v1690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2181.791214 secs 2024-07-26T13:27:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:27 smithi107 ceph-mon[32659]: pgmap v1691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:27 smithi132 ceph-mon[37580]: pgmap v1691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:27 smithi112 ceph-mon[37501]: pgmap v1691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:29 smithi107 ceph-mon[32659]: pgmap v1692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:29 smithi132 ceph-mon[37580]: pgmap v1692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:29 smithi112 ceph-mon[37501]: pgmap v1692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2186.791361 secs 2024-07-26T13:27:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2186.791361 secs 2024-07-26T13:27:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2186.791361 secs 2024-07-26T13:27:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:31 smithi107 ceph-mon[32659]: pgmap v1693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:31 smithi132 ceph-mon[37580]: pgmap v1693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:31 smithi112 ceph-mon[37501]: pgmap v1693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:34 smithi107 ceph-mon[32659]: pgmap v1694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:27:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:34 smithi132 ceph-mon[37580]: pgmap v1694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:27:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:34 smithi112 ceph-mon[37501]: pgmap v1694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:27:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:35 smithi107 ceph-mon[32659]: pgmap v1695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:35 smithi132 ceph-mon[37580]: pgmap v1695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:35 smithi112 ceph-mon[37501]: pgmap v1695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2191.791522 secs 2024-07-26T13:27:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2191.791522 secs 2024-07-26T13:27:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2191.791522 secs 2024-07-26T13:27:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:37 smithi107 ceph-mon[32659]: pgmap v1696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:27:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:37 smithi132 ceph-mon[37580]: pgmap v1696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:27:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:37 smithi112 ceph-mon[37501]: pgmap v1696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:27:38.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:38.821+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:38.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:38.821+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:38.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:38.821+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:39.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:39 smithi107 ceph-mon[32659]: pgmap v1697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:27:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:39 smithi132 ceph-mon[37580]: pgmap v1697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:27:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:39 smithi112 ceph-mon[37501]: pgmap v1697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:27:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:40.980+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:40.980+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:40.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:40.980+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:41 smithi107 ceph-mon[32659]: pgmap v1698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:27:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2196.791650 secs 2024-07-26T13:27:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:41 smithi132 ceph-mon[37580]: pgmap v1698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:27:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2196.791650 secs 2024-07-26T13:27:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:41 smithi112 ceph-mon[37501]: pgmap v1698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:27:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2196.791650 secs 2024-07-26T13:27:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:43 smithi107 ceph-mon[32659]: pgmap v1699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:27:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:43 smithi132 ceph-mon[37580]: pgmap v1699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:27:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:43 smithi112 ceph-mon[37501]: pgmap v1699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:27:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:45 smithi107 ceph-mon[32659]: pgmap v1700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:45 smithi132 ceph-mon[37580]: pgmap v1700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:45 smithi112 ceph-mon[37501]: pgmap v1700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2201.791767 secs 2024-07-26T13:27:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2201.791767 secs 2024-07-26T13:27:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2201.791767 secs 2024-07-26T13:27:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:47 smithi132 ceph-mon[37580]: pgmap v1701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:47 smithi112 ceph-mon[37501]: pgmap v1701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:47 smithi107 ceph-mon[32659]: pgmap v1701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:49 smithi112 ceph-mon[37501]: pgmap v1702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:49 smithi107 ceph-mon[32659]: pgmap v1702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:49 smithi132 ceph-mon[37580]: pgmap v1702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:51.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:51 smithi132 ceph-mon[37580]: pgmap v1703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:51.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2206.791885 secs 2024-07-26T13:27:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:51 smithi112 ceph-mon[37501]: pgmap v1703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2206.791885 secs 2024-07-26T13:27:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:51 smithi107 ceph-mon[32659]: pgmap v1703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2206.791885 secs 2024-07-26T13:27:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:53 smithi112 ceph-mon[37501]: pgmap v1704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:53 smithi107 ceph-mon[32659]: pgmap v1704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:53 smithi132 ceph-mon[37580]: pgmap v1704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:27:53.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:53.837+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:53.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:53.837+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:53.839 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:27:53.837+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:55 smithi112 ceph-mon[37501]: pgmap v1705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:55 smithi107 ceph-mon[32659]: pgmap v1705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:55 smithi132 ceph-mon[37580]: pgmap v1705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:55.996+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:55.996+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:55.998 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:27:55.996+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:27:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2211.792012 secs 2024-07-26T13:27:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2211.792012 secs 2024-07-26T13:27:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2211.792012 secs 2024-07-26T13:27:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:57 smithi112 ceph-mon[37501]: pgmap v1706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:57 smithi107 ceph-mon[32659]: pgmap v1706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:57 smithi132 ceph-mon[37580]: pgmap v1706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:27:59 smithi112 ceph-mon[37501]: pgmap v1707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:27:59 smithi107 ceph-mon[32659]: pgmap v1707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:27:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:27:59 smithi132 ceph-mon[37580]: pgmap v1707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:01 smithi112 ceph-mon[37501]: pgmap v1708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2216.792164 secs 2024-07-26T13:28:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:01 smithi107 ceph-mon[32659]: pgmap v1708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2216.792164 secs 2024-07-26T13:28:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:01 smithi132 ceph-mon[37580]: pgmap v1708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2216.792164 secs 2024-07-26T13:28:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:03 smithi112 ceph-mon[37501]: pgmap v1709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:03 smithi107 ceph-mon[32659]: pgmap v1709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:03 smithi132 ceph-mon[37580]: pgmap v1709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:05 smithi112 ceph-mon[37501]: pgmap v1710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:05 smithi107 ceph-mon[32659]: pgmap v1710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:05 smithi132 ceph-mon[37580]: pgmap v1710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2221.792277 secs 2024-07-26T13:28:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2221.792277 secs 2024-07-26T13:28:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2221.792277 secs 2024-07-26T13:28:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:07 smithi112 ceph-mon[37501]: pgmap v1711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:07 smithi107 ceph-mon[32659]: pgmap v1711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:07 smithi132 ceph-mon[37580]: pgmap v1711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:08.853+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:08.853+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:08.854 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:08.853+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:09 smithi112 ceph-mon[37501]: pgmap v1712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:09 smithi107 ceph-mon[32659]: pgmap v1712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:09 smithi132 ceph-mon[37580]: pgmap v1712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:11.012+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:11.013 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:11.013+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:11.014 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:11.013+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:11 smithi107 ceph-mon[32659]: pgmap v1713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2226.792343 secs 2024-07-26T13:28:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:11 smithi112 ceph-mon[37501]: pgmap v1713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2226.792343 secs 2024-07-26T13:28:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:11 smithi132 ceph-mon[37580]: pgmap v1713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2226.792343 secs 2024-07-26T13:28:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:13 smithi112 ceph-mon[37501]: pgmap v1714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:13 smithi107 ceph-mon[32659]: pgmap v1714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:13 smithi132 ceph-mon[37580]: pgmap v1714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:15 smithi112 ceph-mon[37501]: pgmap v1715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:15 smithi107 ceph-mon[32659]: pgmap v1715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:15 smithi132 ceph-mon[37580]: pgmap v1715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2231.792471 secs 2024-07-26T13:28:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2231.792471 secs 2024-07-26T13:28:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2231.792471 secs 2024-07-26T13:28:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:17 smithi112 ceph-mon[37501]: pgmap v1716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:17 smithi107 ceph-mon[32659]: pgmap v1716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:17 smithi132 ceph-mon[37580]: pgmap v1716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:19 smithi112 ceph-mon[37501]: pgmap v1717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:19 smithi107 ceph-mon[32659]: pgmap v1717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:19.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:19 smithi132 ceph-mon[37580]: pgmap v1717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:21 smithi112 ceph-mon[37501]: pgmap v1718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2236.792586 secs 2024-07-26T13:28:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:21 smithi107 ceph-mon[32659]: pgmap v1718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2236.792586 secs 2024-07-26T13:28:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:21 smithi132 ceph-mon[37580]: pgmap v1718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2236.792586 secs 2024-07-26T13:28:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:23 smithi112 ceph-mon[37501]: pgmap v1719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:23 smithi107 ceph-mon[32659]: pgmap v1719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:23 smithi132 ceph-mon[37580]: pgmap v1719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:23.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:23.856+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:23.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:23.856+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:23.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:23.856+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:25 smithi112 ceph-mon[37501]: pgmap v1720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:25 smithi107 ceph-mon[32659]: pgmap v1720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:25 smithi132 ceph-mon[37580]: pgmap v1720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:26.018+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:26.018+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:26.019 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:26.018+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2241.792671 secs 2024-07-26T13:28:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2241.792671 secs 2024-07-26T13:28:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2241.792671 secs 2024-07-26T13:28:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:27 smithi107 ceph-mon[32659]: pgmap v1721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:27 smithi132 ceph-mon[37580]: pgmap v1721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:27 smithi112 ceph-mon[37501]: pgmap v1721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:29 smithi107 ceph-mon[32659]: pgmap v1722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:29 smithi132 ceph-mon[37580]: pgmap v1722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:29 smithi112 ceph-mon[37501]: pgmap v1722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:31 smithi107 ceph-mon[32659]: pgmap v1723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2246.792792 secs 2024-07-26T13:28:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:31 smithi132 ceph-mon[37580]: pgmap v1723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2246.792792 secs 2024-07-26T13:28:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:31 smithi112 ceph-mon[37501]: pgmap v1723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2246.792792 secs 2024-07-26T13:28:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:33 smithi107 ceph-mon[32659]: pgmap v1724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:33 smithi132 ceph-mon[37580]: pgmap v1724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:33 smithi112 ceph-mon[37501]: pgmap v1724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:34.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:28:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:28:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:28:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:35 smithi107 ceph-mon[32659]: pgmap v1725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:28:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:35 smithi132 ceph-mon[37580]: pgmap v1725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:28:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:35 smithi112 ceph-mon[37501]: pgmap v1725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:28:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:36 smithi107 ceph-mon[32659]: pgmap v1726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:28:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:36 smithi107 ceph-mon[32659]: pgmap v1727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:28:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2251.792914 secs 2024-07-26T13:28:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:36 smithi132 ceph-mon[37580]: pgmap v1726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:28:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:36 smithi132 ceph-mon[37580]: pgmap v1727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:28:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2251.792914 secs 2024-07-26T13:28:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:36 smithi112 ceph-mon[37501]: pgmap v1726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:28:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:36 smithi112 ceph-mon[37501]: pgmap v1727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:28:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2251.792914 secs 2024-07-26T13:28:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:38 smithi107 ceph-mon[32659]: pgmap v1728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:38 smithi132 ceph-mon[37580]: pgmap v1728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:38 smithi112 ceph-mon[37501]: pgmap v1728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:38.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:38.872+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:38.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:38.872+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:38.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:38.872+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:40 smithi107 ceph-mon[32659]: pgmap v1729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:28:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:40 smithi132 ceph-mon[37580]: pgmap v1729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:28:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:40 smithi112 ceph-mon[37501]: pgmap v1729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:28:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:41.033+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:41.033+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:41.033+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2256.792998 secs 2024-07-26T13:28:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2256.792998 secs 2024-07-26T13:28:41.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2256.792998 secs 2024-07-26T13:28:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:42 smithi107 ceph-mon[32659]: pgmap v1730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:28:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:42 smithi132 ceph-mon[37580]: pgmap v1730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:28:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:42 smithi112 ceph-mon[37501]: pgmap v1730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:28:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:44 smithi107 ceph-mon[32659]: pgmap v1731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:28:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:44 smithi132 ceph-mon[37580]: pgmap v1731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:28:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:44 smithi112 ceph-mon[37501]: pgmap v1731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:28:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:46 smithi107 ceph-mon[32659]: pgmap v1732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2261.793121 secs 2024-07-26T13:28:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:46 smithi132 ceph-mon[37580]: pgmap v1732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2261.793121 secs 2024-07-26T13:28:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:46 smithi112 ceph-mon[37501]: pgmap v1732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2261.793121 secs 2024-07-26T13:28:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:48 smithi107 ceph-mon[32659]: pgmap v1733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:48 smithi132 ceph-mon[37580]: pgmap v1733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:48 smithi112 ceph-mon[37501]: pgmap v1733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:50 smithi132 ceph-mon[37580]: pgmap v1734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:50 smithi112 ceph-mon[37501]: pgmap v1734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:50 smithi107 ceph-mon[32659]: pgmap v1734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2266.793187 secs 2024-07-26T13:28:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2266.793187 secs 2024-07-26T13:28:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2266.793187 secs 2024-07-26T13:28:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:52 smithi132 ceph-mon[37580]: pgmap v1735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:52 smithi112 ceph-mon[37501]: pgmap v1735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:52 smithi107 ceph-mon[32659]: pgmap v1735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:53.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:53.876+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:53.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:53.876+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:53.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:28:53.876+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:54 smithi112 ceph-mon[37501]: pgmap v1736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:54 smithi107 ceph-mon[32659]: pgmap v1736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:54 smithi132 ceph-mon[37580]: pgmap v1736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:56.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:56.049+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:56.050 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:56.049+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:56.051 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:28:56.049+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:28:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:56 smithi112 ceph-mon[37501]: pgmap v1737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2271.793307 secs 2024-07-26T13:28:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:56 smithi107 ceph-mon[32659]: pgmap v1737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2271.793307 secs 2024-07-26T13:28:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:56 smithi132 ceph-mon[37580]: pgmap v1737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2271.793307 secs 2024-07-26T13:28:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:28:58 smithi112 ceph-mon[37501]: pgmap v1738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:28:58 smithi107 ceph-mon[32659]: pgmap v1738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:28:58 smithi132 ceph-mon[37580]: pgmap v1738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:00 smithi112 ceph-mon[37501]: pgmap v1739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:00 smithi107 ceph-mon[32659]: pgmap v1739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:00 smithi132 ceph-mon[37580]: pgmap v1739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2276.793431 secs 2024-07-26T13:29:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2276.793431 secs 2024-07-26T13:29:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2276.793431 secs 2024-07-26T13:29:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:02 smithi112 ceph-mon[37501]: pgmap v1740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:02 smithi107 ceph-mon[32659]: pgmap v1740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:02 smithi132 ceph-mon[37580]: pgmap v1740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:04 smithi112 ceph-mon[37501]: pgmap v1741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:04 smithi107 ceph-mon[32659]: pgmap v1741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:04 smithi132 ceph-mon[37580]: pgmap v1741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:06 smithi112 ceph-mon[37501]: pgmap v1742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2281.793543 secs 2024-07-26T13:29:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:06 smithi107 ceph-mon[32659]: pgmap v1742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2281.793543 secs 2024-07-26T13:29:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:06 smithi132 ceph-mon[37580]: pgmap v1742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2281.793543 secs 2024-07-26T13:29:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:08 smithi112 ceph-mon[37501]: pgmap v1743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:08 smithi107 ceph-mon[32659]: pgmap v1743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:08 smithi132 ceph-mon[37580]: pgmap v1743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:08.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:08.893+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:08.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:08.893+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:08.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:08.893+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:10 smithi112 ceph-mon[37501]: pgmap v1744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:10 smithi107 ceph-mon[32659]: pgmap v1744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:10 smithi132 ceph-mon[37580]: pgmap v1744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:11.064+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:11.064+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:11.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:11.064+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2286.793679 secs 2024-07-26T13:29:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2286.793679 secs 2024-07-26T13:29:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2286.793679 secs 2024-07-26T13:29:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:12 smithi112 ceph-mon[37501]: pgmap v1745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:12 smithi107 ceph-mon[32659]: pgmap v1745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:12 smithi132 ceph-mon[37580]: pgmap v1745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:13 smithi107 ceph-mon[32659]: pgmap v1746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:13 smithi132 ceph-mon[37580]: pgmap v1746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:13 smithi112 ceph-mon[37501]: pgmap v1746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:16 smithi112 ceph-mon[37501]: pgmap v1747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2291.793801 secs 2024-07-26T13:29:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:16 smithi107 ceph-mon[32659]: pgmap v1747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2291.793801 secs 2024-07-26T13:29:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:16 smithi132 ceph-mon[37580]: pgmap v1747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2291.793801 secs 2024-07-26T13:29:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:18 smithi112 ceph-mon[37501]: pgmap v1748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:18 smithi107 ceph-mon[32659]: pgmap v1748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:18 smithi132 ceph-mon[37580]: pgmap v1748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:20 smithi112 ceph-mon[37501]: pgmap v1749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:29:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:20 smithi107 ceph-mon[32659]: pgmap v1749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:29:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:20 smithi132 ceph-mon[37580]: pgmap v1749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:29:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2296.793942 secs 2024-07-26T13:29:21.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2296.793942 secs 2024-07-26T13:29:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2296.793942 secs 2024-07-26T13:29:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:22 smithi112 ceph-mon[37501]: pgmap v1750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:29:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:22 smithi107 ceph-mon[32659]: pgmap v1750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:29:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:22 smithi132 ceph-mon[37580]: pgmap v1750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:29:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:23.905+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:23.905+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:23.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:23.905+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:24 smithi112 ceph-mon[37501]: pgmap v1751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:29:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:24 smithi107 ceph-mon[32659]: pgmap v1751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:29:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:24 smithi132 ceph-mon[37580]: pgmap v1751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:29:26.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:26.080+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:26.080+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:26.080+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:26 smithi107 ceph-mon[32659]: pgmap v1752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2301.794072 secs 2024-07-26T13:29:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:26 smithi132 ceph-mon[37580]: pgmap v1752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2301.794072 secs 2024-07-26T13:29:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:26 smithi112 ceph-mon[37501]: pgmap v1752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2301.794072 secs 2024-07-26T13:29:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:28 smithi107 ceph-mon[32659]: pgmap v1753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:28 smithi132 ceph-mon[37580]: pgmap v1753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:28 smithi112 ceph-mon[37501]: pgmap v1753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:30 smithi107 ceph-mon[32659]: pgmap v1754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:29:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:30 smithi132 ceph-mon[37580]: pgmap v1754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:29:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:30 smithi112 ceph-mon[37501]: pgmap v1754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:29:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2306.794191 secs 2024-07-26T13:29:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2306.794191 secs 2024-07-26T13:29:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2306.794191 secs 2024-07-26T13:29:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:32 smithi107 ceph-mon[32659]: pgmap v1755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:32.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:32 smithi132 ceph-mon[37580]: pgmap v1755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:32 smithi112 ceph-mon[37501]: pgmap v1755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:34 smithi107 ceph-mon[32659]: pgmap v1756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:34 smithi132 ceph-mon[37580]: pgmap v1756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:34 smithi112 ceph-mon[37501]: pgmap v1756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:36 smithi107 ceph-mon[32659]: pgmap v1757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2311.794313 secs 2024-07-26T13:29:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:36 smithi132 ceph-mon[37580]: pgmap v1757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2311.794313 secs 2024-07-26T13:29:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:36 smithi112 ceph-mon[37501]: pgmap v1757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2311.794313 secs 2024-07-26T13:29:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:37 smithi107 ceph-mon[32659]: pgmap v1758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:29:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:37 smithi107 ceph-mon[32659]: pgmap v1759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:29:37.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:29:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:37 smithi132 ceph-mon[37580]: pgmap v1758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:29:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:37 smithi132 ceph-mon[37580]: pgmap v1759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:29:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:29:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:37 smithi112 ceph-mon[37501]: pgmap v1758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:29:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:37 smithi112 ceph-mon[37501]: pgmap v1759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:29:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:29:38.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:38.921+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:38.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:38.921+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:38.922 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:38.921+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:39 smithi112 ceph-mon[37501]: pgmap v1760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:39 smithi107 ceph-mon[32659]: pgmap v1760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:39.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:39 smithi132 ceph-mon[37580]: pgmap v1760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:41.094+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:41.094+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:41.097 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:41.094+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2316.794421 secs 2024-07-26T13:29:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:41 smithi112 ceph-mon[37501]: pgmap v1761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2316.794421 secs 2024-07-26T13:29:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:41 smithi107 ceph-mon[32659]: pgmap v1761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2316.794421 secs 2024-07-26T13:29:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:41 smithi132 ceph-mon[37580]: pgmap v1761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:43 smithi112 ceph-mon[37501]: pgmap v1762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:43 smithi107 ceph-mon[32659]: pgmap v1762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:43 smithi132 ceph-mon[37580]: pgmap v1762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:45 smithi112 ceph-mon[37501]: pgmap v1763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:45 smithi107 ceph-mon[32659]: pgmap v1763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:45 smithi132 ceph-mon[37580]: pgmap v1763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2321.794548 secs 2024-07-26T13:29:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2321.794548 secs 2024-07-26T13:29:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2321.794548 secs 2024-07-26T13:29:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:47 smithi112 ceph-mon[37501]: pgmap v1764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:29:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:47 smithi107 ceph-mon[32659]: pgmap v1764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:29:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:47 smithi132 ceph-mon[37580]: pgmap v1764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:29:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:49 smithi112 ceph-mon[37501]: pgmap v1765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:49 smithi107 ceph-mon[32659]: pgmap v1765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:49 smithi132 ceph-mon[37580]: pgmap v1765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2326.794697 secs 2024-07-26T13:29:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:51 smithi112 ceph-mon[37501]: pgmap v1766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2326.794697 secs 2024-07-26T13:29:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:51 smithi107 ceph-mon[32659]: pgmap v1766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2326.794697 secs 2024-07-26T13:29:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:51 smithi132 ceph-mon[37580]: pgmap v1766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:53 smithi112 ceph-mon[37501]: pgmap v1767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:53 smithi107 ceph-mon[32659]: pgmap v1767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:53 smithi132 ceph-mon[37580]: pgmap v1767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:53.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:53.936+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:53.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:53.936+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:53.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:29:53.936+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:55 smithi112 ceph-mon[37501]: pgmap v1768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:55 smithi107 ceph-mon[32659]: pgmap v1768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:55 smithi132 ceph-mon[37580]: pgmap v1768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:56.108+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:56.108+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:29:56.108+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:29:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2331.794835 secs 2024-07-26T13:29:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2331.794835 secs 2024-07-26T13:29:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2331.794835 secs 2024-07-26T13:29:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:57 smithi112 ceph-mon[37501]: pgmap v1769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:57 smithi107 ceph-mon[32659]: pgmap v1769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:57 smithi132 ceph-mon[37580]: pgmap v1769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:29:59 smithi112 ceph-mon[37501]: pgmap v1770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:29:59 smithi107 ceph-mon[32659]: pgmap v1770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:29:59 smithi132 ceph-mon[37580]: pgmap v1770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:30:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:30:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:30:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2336.794956 secs 2024-07-26T13:30:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:01 smithi112 ceph-mon[37501]: pgmap v1771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2336.794956 secs 2024-07-26T13:30:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:01 smithi107 ceph-mon[32659]: pgmap v1771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2336.794956 secs 2024-07-26T13:30:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:01 smithi132 ceph-mon[37580]: pgmap v1771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:02 smithi112 ceph-mon[37501]: pgmap v1772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:02 smithi107 ceph-mon[32659]: pgmap v1772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:02 smithi132 ceph-mon[37580]: pgmap v1772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:05 smithi112 ceph-mon[37501]: pgmap v1773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:05 smithi107 ceph-mon[32659]: pgmap v1773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:05 smithi132 ceph-mon[37580]: pgmap v1773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2341.795093 secs 2024-07-26T13:30:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2341.795093 secs 2024-07-26T13:30:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2341.795093 secs 2024-07-26T13:30:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:07 smithi112 ceph-mon[37501]: pgmap v1774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:07 smithi107 ceph-mon[32659]: pgmap v1774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:07 smithi132 ceph-mon[37580]: pgmap v1774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:08.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:08.950+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:08.951 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:08.950+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:08.952 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:08.950+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:09 smithi112 ceph-mon[37501]: pgmap v1775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:09 smithi107 ceph-mon[32659]: pgmap v1775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:09 smithi132 ceph-mon[37580]: pgmap v1775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:11.124 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:11.123+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:11.124+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:11.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:11.124+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2346.795210 secs 2024-07-26T13:30:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:11 smithi112 ceph-mon[37501]: pgmap v1776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2346.795210 secs 2024-07-26T13:30:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:11 smithi107 ceph-mon[32659]: pgmap v1776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2346.795210 secs 2024-07-26T13:30:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:11 smithi132 ceph-mon[37580]: pgmap v1776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:13.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:13.031+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:30:13.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:13.032+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:30:13.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:13.080+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:30:13.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:13.081+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:30:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:13 smithi112 ceph-mon[37501]: pgmap v1777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:13 smithi107 ceph-mon[32659]: pgmap v1777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:13 smithi132 ceph-mon[37580]: pgmap v1777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:14.151+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:30:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:14.151+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:30:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:14.152+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:30:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:14.152+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:30:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:15 smithi112 ceph-mon[37501]: pgmap v1778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:15 smithi107 ceph-mon[32659]: pgmap v1778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:15 smithi132 ceph-mon[37580]: pgmap v1778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2351.795319 secs 2024-07-26T13:30:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2351.795319 secs 2024-07-26T13:30:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2351.795319 secs 2024-07-26T13:30:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:17 smithi112 ceph-mon[37501]: pgmap v1779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:17 smithi107 ceph-mon[32659]: pgmap v1779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:17 smithi132 ceph-mon[37580]: pgmap v1779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:19 smithi112 ceph-mon[37501]: pgmap v1780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:19 smithi107 ceph-mon[32659]: pgmap v1780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:19 smithi132 ceph-mon[37580]: pgmap v1780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2356.795410 secs 2024-07-26T13:30:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:21 smithi112 ceph-mon[37501]: pgmap v1781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2356.795410 secs 2024-07-26T13:30:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:21 smithi107 ceph-mon[32659]: pgmap v1781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2356.795410 secs 2024-07-26T13:30:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:21 smithi132 ceph-mon[37580]: pgmap v1781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:23 smithi112 ceph-mon[37501]: pgmap v1782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:23 smithi107 ceph-mon[32659]: pgmap v1782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:23 smithi132 ceph-mon[37580]: pgmap v1782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:23.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:23.954+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:23.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:23.954+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:23.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:23.954+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:25 smithi112 ceph-mon[37501]: pgmap v1783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:25 smithi107 ceph-mon[32659]: pgmap v1783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:25 smithi132 ceph-mon[37580]: pgmap v1783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:26.130+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:26.130+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:26.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:26.130+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2361.795508 secs 2024-07-26T13:30:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:26 smithi112 ceph-mon[37501]: pgmap v1784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2361.795508 secs 2024-07-26T13:30:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:26 smithi107 ceph-mon[32659]: pgmap v1784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2361.795508 secs 2024-07-26T13:30:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:26 smithi132 ceph-mon[37580]: pgmap v1784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:29 smithi112 ceph-mon[37501]: pgmap v1785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:29 smithi107 ceph-mon[32659]: pgmap v1785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:29 smithi132 ceph-mon[37580]: pgmap v1785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2366.795651 secs 2024-07-26T13:30:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:31 smithi112 ceph-mon[37501]: pgmap v1786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2366.795651 secs 2024-07-26T13:30:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:31 smithi107 ceph-mon[32659]: pgmap v1786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2366.795651 secs 2024-07-26T13:30:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:31 smithi132 ceph-mon[37580]: pgmap v1786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:33 smithi112 ceph-mon[37501]: pgmap v1787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:33 smithi107 ceph-mon[32659]: pgmap v1787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:33 smithi132 ceph-mon[37580]: pgmap v1787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:35 smithi112 ceph-mon[37501]: pgmap v1788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:35 smithi107 ceph-mon[32659]: pgmap v1788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:35 smithi132 ceph-mon[37580]: pgmap v1788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2371.795767 secs 2024-07-26T13:30:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2371.795767 secs 2024-07-26T13:30:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2371.795767 secs 2024-07-26T13:30:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:37 smithi112 ceph-mon[37501]: pgmap v1789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:37 smithi107 ceph-mon[32659]: pgmap v1789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:37 smithi132 ceph-mon[37580]: pgmap v1789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:30:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:30:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:30:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:38.969+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:38.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:38.969+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:38.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:38.969+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:39 smithi112 ceph-mon[37501]: pgmap v1790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:30:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:39 smithi107 ceph-mon[32659]: pgmap v1790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:30:39.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:39 smithi132 ceph-mon[37580]: pgmap v1790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:30:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:41.146+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:41.146+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:41.146+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:41 smithi112 ceph-mon[37501]: pgmap v1791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:30:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2376.795890 secs 2024-07-26T13:30:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:41 smithi107 ceph-mon[32659]: pgmap v1791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:30:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2376.795890 secs 2024-07-26T13:30:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:41 smithi132 ceph-mon[37580]: pgmap v1791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:30:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2376.795890 secs 2024-07-26T13:30:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:43 smithi112 ceph-mon[37501]: pgmap v1792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:30:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:43 smithi107 ceph-mon[32659]: pgmap v1792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:30:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:43 smithi132 ceph-mon[37580]: pgmap v1792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:30:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:44 smithi107 ceph-mon[32659]: pgmap v1793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:30:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:44 smithi132 ceph-mon[37580]: pgmap v1793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:30:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:44 smithi112 ceph-mon[37501]: pgmap v1793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:30:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2381.796027 secs 2024-07-26T13:30:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:46 smithi107 ceph-mon[32659]: pgmap v1794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:30:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2381.796027 secs 2024-07-26T13:30:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:46 smithi132 ceph-mon[37580]: pgmap v1794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:30:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2381.796027 secs 2024-07-26T13:30:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:46 smithi112 ceph-mon[37501]: pgmap v1794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:30:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:48 smithi107 ceph-mon[32659]: pgmap v1795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:30:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:48 smithi132 ceph-mon[37580]: pgmap v1795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:30:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:48 smithi112 ceph-mon[37501]: pgmap v1795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:30:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:50 smithi107 ceph-mon[32659]: pgmap v1796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:30:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:50 smithi132 ceph-mon[37580]: pgmap v1796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:30:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:50 smithi112 ceph-mon[37501]: pgmap v1796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:30:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2386.796168 secs 2024-07-26T13:30:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2386.796168 secs 2024-07-26T13:30:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2386.796168 secs 2024-07-26T13:30:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:52 smithi107 ceph-mon[32659]: pgmap v1797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:52 smithi132 ceph-mon[37580]: pgmap v1797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:52 smithi112 ceph-mon[37501]: pgmap v1797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:53.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:53.970+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:53.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:53.970+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:53.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:30:53.970+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:54 smithi107 ceph-mon[32659]: pgmap v1798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:54 smithi132 ceph-mon[37580]: pgmap v1798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:54 smithi112 ceph-mon[37501]: pgmap v1798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2391.796294 secs 2024-07-26T13:30:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2391.796294 secs 2024-07-26T13:30:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2391.796294 secs 2024-07-26T13:30:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:56.161+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:56.161+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:56.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:30:56.161+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:30:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:56 smithi107 ceph-mon[32659]: pgmap v1799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:56 smithi132 ceph-mon[37580]: pgmap v1799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:56 smithi112 ceph-mon[37501]: pgmap v1799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:30:58 smithi107 ceph-mon[32659]: pgmap v1800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:30:58 smithi132 ceph-mon[37580]: pgmap v1800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:30:58 smithi112 ceph-mon[37501]: pgmap v1800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:00 smithi107 ceph-mon[32659]: pgmap v1801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2396.796414 secs 2024-07-26T13:31:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:00 smithi132 ceph-mon[37580]: pgmap v1801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2396.796414 secs 2024-07-26T13:31:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:00 smithi112 ceph-mon[37501]: pgmap v1801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2396.796414 secs 2024-07-26T13:31:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:02 smithi132 ceph-mon[37580]: pgmap v1802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:02 smithi112 ceph-mon[37501]: pgmap v1802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:02 smithi107 ceph-mon[32659]: pgmap v1802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:04 smithi132 ceph-mon[37580]: pgmap v1803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:04 smithi112 ceph-mon[37501]: pgmap v1803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:04 smithi107 ceph-mon[32659]: pgmap v1803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2401.796516 secs 2024-07-26T13:31:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2401.796516 secs 2024-07-26T13:31:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2401.796516 secs 2024-07-26T13:31:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:06 smithi112 ceph-mon[37501]: pgmap v1804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:06 smithi107 ceph-mon[32659]: pgmap v1804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:06 smithi132 ceph-mon[37580]: pgmap v1804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:08 smithi112 ceph-mon[37501]: pgmap v1805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:08.986+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:08.986+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:08.986+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:08.988 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:08 smithi107 ceph-mon[32659]: pgmap v1805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:08 smithi132 ceph-mon[37580]: pgmap v1805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:10 smithi112 ceph-mon[37501]: pgmap v1806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2406.796637 secs 2024-07-26T13:31:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:10 smithi107 ceph-mon[32659]: pgmap v1806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2406.796637 secs 2024-07-26T13:31:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:10 smithi132 ceph-mon[37580]: pgmap v1806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2406.796637 secs 2024-07-26T13:31:11.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:11.177+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:11.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:11.177+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:11.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:11.177+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:12 smithi112 ceph-mon[37501]: pgmap v1807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:12 smithi107 ceph-mon[32659]: pgmap v1807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:12 smithi132 ceph-mon[37580]: pgmap v1807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:14 smithi112 ceph-mon[37501]: pgmap v1808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:14 smithi107 ceph-mon[32659]: pgmap v1808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:14 smithi132 ceph-mon[37580]: pgmap v1808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2411.796753 secs 2024-07-26T13:31:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2411.796753 secs 2024-07-26T13:31:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2411.796753 secs 2024-07-26T13:31:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:16 smithi112 ceph-mon[37501]: pgmap v1809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:16 smithi107 ceph-mon[32659]: pgmap v1809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:16 smithi132 ceph-mon[37580]: pgmap v1809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:18 smithi112 ceph-mon[37501]: pgmap v1810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:18 smithi107 ceph-mon[32659]: pgmap v1810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:18 smithi132 ceph-mon[37580]: pgmap v1810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:20 smithi112 ceph-mon[37501]: pgmap v1811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2416.796871 secs 2024-07-26T13:31:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:20 smithi107 ceph-mon[32659]: pgmap v1811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2416.796871 secs 2024-07-26T13:31:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:20 smithi132 ceph-mon[37580]: pgmap v1811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2416.796871 secs 2024-07-26T13:31:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:22 smithi112 ceph-mon[37501]: pgmap v1812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:22 smithi107 ceph-mon[32659]: pgmap v1812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:22 smithi132 ceph-mon[37580]: pgmap v1812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:23.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:23.994+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:23.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:23.994+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:23.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:23.994+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:24 smithi112 ceph-mon[37501]: pgmap v1813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:24 smithi107 ceph-mon[32659]: pgmap v1813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:24 smithi132 ceph-mon[37580]: pgmap v1813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2421.797007 secs 2024-07-26T13:31:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2421.797007 secs 2024-07-26T13:31:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2421.797007 secs 2024-07-26T13:31:26.193 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:26.193+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:26.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:26.193+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:26.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:26.193+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:26 smithi112 ceph-mon[37501]: pgmap v1814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:26 smithi107 ceph-mon[32659]: pgmap v1814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:26 smithi132 ceph-mon[37580]: pgmap v1814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:28 smithi112 ceph-mon[37501]: pgmap v1815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:28 smithi107 ceph-mon[32659]: pgmap v1815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:28 smithi132 ceph-mon[37580]: pgmap v1815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:30 smithi112 ceph-mon[37501]: pgmap v1816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2426.797136 secs 2024-07-26T13:31:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:30 smithi107 ceph-mon[32659]: pgmap v1816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2426.797136 secs 2024-07-26T13:31:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:30 smithi132 ceph-mon[37580]: pgmap v1816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:31:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2426.797136 secs 2024-07-26T13:31:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:32 smithi112 ceph-mon[37501]: pgmap v1817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:31:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:32 smithi107 ceph-mon[32659]: pgmap v1817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:31:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:32 smithi132 ceph-mon[37580]: pgmap v1817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:31:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:34 smithi112 ceph-mon[37501]: pgmap v1818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:34 smithi107 ceph-mon[32659]: pgmap v1818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:34 smithi132 ceph-mon[37580]: pgmap v1818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2431.797256 secs 2024-07-26T13:31:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2431.797256 secs 2024-07-26T13:31:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2431.797256 secs 2024-07-26T13:31:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:36 smithi112 ceph-mon[37501]: pgmap v1819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:36 smithi107 ceph-mon[32659]: pgmap v1819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:36 smithi132 ceph-mon[37580]: pgmap v1819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:39.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:39.007+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:39.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:39.007+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:39.008 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:39.007+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:39.009 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:38 smithi107 ceph-mon[32659]: pgmap v1820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:39.009 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:39.009 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:39.009 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:31:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:38 smithi132 ceph-mon[37580]: pgmap v1820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:31:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:38 smithi112 ceph-mon[37501]: pgmap v1820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:31:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:39 smithi107 ceph-mon[32659]: pgmap v1821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:39 smithi107 ceph-mon[32659]: pgmap v1822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:31:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:39 smithi132 ceph-mon[37580]: pgmap v1821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:39 smithi132 ceph-mon[37580]: pgmap v1822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:31:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:39 smithi112 ceph-mon[37501]: pgmap v1821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:39 smithi112 ceph-mon[37501]: pgmap v1822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:31:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2436.797321 secs 2024-07-26T13:31:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2436.797321 secs 2024-07-26T13:31:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2436.797321 secs 2024-07-26T13:31:41.209 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:41.208+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:41.209 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:41.208+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:41.209 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:41.208+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:41 smithi107 ceph-mon[32659]: pgmap v1823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:31:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:41 smithi132 ceph-mon[37580]: pgmap v1823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:31:42.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:41 smithi112 ceph-mon[37501]: pgmap v1823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:31:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:43 smithi107 ceph-mon[32659]: pgmap v1824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:31:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:43 smithi132 ceph-mon[37580]: pgmap v1824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:31:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:43 smithi112 ceph-mon[37501]: pgmap v1824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:31:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:45 smithi107 ceph-mon[32659]: pgmap v1825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2441.797478 secs 2024-07-26T13:31:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:45 smithi132 ceph-mon[37580]: pgmap v1825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2441.797478 secs 2024-07-26T13:31:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:45 smithi112 ceph-mon[37501]: pgmap v1825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:31:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2441.797478 secs 2024-07-26T13:31:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:47 smithi107 ceph-mon[32659]: pgmap v1826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:47 smithi132 ceph-mon[37580]: pgmap v1826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:47 smithi112 ceph-mon[37501]: pgmap v1826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:49 smithi107 ceph-mon[32659]: pgmap v1827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:49 smithi132 ceph-mon[37580]: pgmap v1827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:49 smithi112 ceph-mon[37501]: pgmap v1827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2446.797609 secs 2024-07-26T13:31:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2446.797609 secs 2024-07-26T13:31:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2446.797609 secs 2024-07-26T13:31:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:51 smithi132 ceph-mon[37580]: pgmap v1828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:51 smithi112 ceph-mon[37501]: pgmap v1828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:51 smithi107 ceph-mon[32659]: pgmap v1828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:54.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:54.022+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:54.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:54.022+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:54.024 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:31:54.022+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:53 smithi132 ceph-mon[37580]: pgmap v1829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:53 smithi112 ceph-mon[37501]: pgmap v1829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:53 smithi107 ceph-mon[32659]: pgmap v1829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:55 smithi132 ceph-mon[37580]: pgmap v1830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2451.797763 secs 2024-07-26T13:31:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:55 smithi112 ceph-mon[37501]: pgmap v1830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2451.797763 secs 2024-07-26T13:31:56.224 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:56.223+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:56.223+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:56.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:31:56.223+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:31:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:55 smithi107 ceph-mon[32659]: pgmap v1830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2451.797763 secs 2024-07-26T13:31:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:57 smithi132 ceph-mon[37580]: pgmap v1831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:57 smithi112 ceph-mon[37501]: pgmap v1831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:57 smithi107 ceph-mon[32659]: pgmap v1831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:31:59 smithi112 ceph-mon[37501]: pgmap v1832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:00.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:31:59 smithi107 ceph-mon[32659]: pgmap v1832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:31:59 smithi132 ceph-mon[37580]: pgmap v1832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2456.797900 secs 2024-07-26T13:32:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2456.797900 secs 2024-07-26T13:32:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2456.797900 secs 2024-07-26T13:32:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:01 smithi112 ceph-mon[37501]: pgmap v1833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:01 smithi107 ceph-mon[32659]: pgmap v1833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:01 smithi132 ceph-mon[37580]: pgmap v1833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:04.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:03 smithi107 ceph-mon[32659]: pgmap v1834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:03 smithi112 ceph-mon[37501]: pgmap v1834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:03 smithi132 ceph-mon[37580]: pgmap v1834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:05 smithi112 ceph-mon[37501]: pgmap v1835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2461.798018 secs 2024-07-26T13:32:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:05 smithi107 ceph-mon[32659]: pgmap v1835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2461.798018 secs 2024-07-26T13:32:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:05 smithi132 ceph-mon[37580]: pgmap v1835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2461.798018 secs 2024-07-26T13:32:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:07 smithi112 ceph-mon[37501]: pgmap v1836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:07 smithi107 ceph-mon[32659]: pgmap v1836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:07 smithi132 ceph-mon[37580]: pgmap v1836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:09.038+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:09.038+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:09.039 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:09.038+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:09 smithi112 ceph-mon[37501]: pgmap v1837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:09 smithi107 ceph-mon[32659]: pgmap v1837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:09 smithi132 ceph-mon[37580]: pgmap v1837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2466.798136 secs 2024-07-26T13:32:11.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:11.231+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:11.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:11.231+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:11.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:11.231+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:11.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2466.798136 secs 2024-07-26T13:32:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2466.798136 secs 2024-07-26T13:32:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:11 smithi112 ceph-mon[37501]: pgmap v1838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:11 smithi107 ceph-mon[32659]: pgmap v1838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:11 smithi132 ceph-mon[37580]: pgmap v1838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:13 smithi112 ceph-mon[37501]: pgmap v1839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:13 smithi107 ceph-mon[32659]: pgmap v1839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:13 smithi132 ceph-mon[37580]: pgmap v1839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:16.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:15 smithi112 ceph-mon[37501]: pgmap v1840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:16.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2471.798256 secs 2024-07-26T13:32:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:15 smithi107 ceph-mon[32659]: pgmap v1840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2471.798256 secs 2024-07-26T13:32:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:15 smithi132 ceph-mon[37580]: pgmap v1840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2471.798256 secs 2024-07-26T13:32:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:17 smithi112 ceph-mon[37501]: pgmap v1841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:17 smithi107 ceph-mon[32659]: pgmap v1841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:17 smithi132 ceph-mon[37580]: pgmap v1841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:19 smithi112 ceph-mon[37501]: pgmap v1842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:19 smithi107 ceph-mon[32659]: pgmap v1842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:19 smithi132 ceph-mon[37580]: pgmap v1842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2476.798339 secs 2024-07-26T13:32:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2476.798339 secs 2024-07-26T13:32:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2476.798339 secs 2024-07-26T13:32:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:21 smithi112 ceph-mon[37501]: pgmap v1843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:21 smithi107 ceph-mon[32659]: pgmap v1843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:21 smithi132 ceph-mon[37580]: pgmap v1843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:24.054+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:24.054+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:24.054+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:23 smithi112 ceph-mon[37501]: pgmap v1844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:23 smithi107 ceph-mon[32659]: pgmap v1844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:23 smithi132 ceph-mon[37580]: pgmap v1844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:25 smithi112 ceph-mon[37501]: pgmap v1845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2481.798465 secs 2024-07-26T13:32:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:26.234+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:26.234+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:26.234+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:25 smithi107 ceph-mon[32659]: pgmap v1845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2481.798465 secs 2024-07-26T13:32:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:25 smithi132 ceph-mon[37580]: pgmap v1845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2481.798465 secs 2024-07-26T13:32:27.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:27.054+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:32:27.055 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:27.055+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:32:27.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:27.102+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:32:27.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:27.103+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:32:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:27 smithi112 ceph-mon[37501]: pgmap v1846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:27 smithi107 ceph-mon[32659]: pgmap v1846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:27 smithi132 ceph-mon[37580]: pgmap v1846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:29.177 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:29.176+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:32:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:29.176+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:32:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:29.177+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:32:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:29.177+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:32:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:29 smithi112 ceph-mon[37501]: pgmap v1847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:29 smithi107 ceph-mon[32659]: pgmap v1847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:29 smithi132 ceph-mon[37580]: pgmap v1847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2486.798587 secs 2024-07-26T13:32:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2486.798587 secs 2024-07-26T13:32:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2486.798587 secs 2024-07-26T13:32:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:31 smithi112 ceph-mon[37501]: pgmap v1848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:31 smithi107 ceph-mon[32659]: pgmap v1848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:31 smithi132 ceph-mon[37580]: pgmap v1848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:33 smithi112 ceph-mon[37501]: pgmap v1849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:33 smithi107 ceph-mon[32659]: pgmap v1849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:33 smithi132 ceph-mon[37580]: pgmap v1849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:35 smithi107 ceph-mon[32659]: pgmap v1850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2491.798685 secs 2024-07-26T13:32:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:35 smithi132 ceph-mon[37580]: pgmap v1850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2491.798685 secs 2024-07-26T13:32:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:35 smithi112 ceph-mon[37501]: pgmap v1850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2491.798685 secs 2024-07-26T13:32:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:37 smithi107 ceph-mon[32659]: pgmap v1851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:37 smithi132 ceph-mon[37580]: pgmap v1851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:37 smithi112 ceph-mon[37501]: pgmap v1851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:39.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:39.069+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:39.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:39.069+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:39.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:39.069+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: pgmap v1852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:32:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: pgmap v1852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:32:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: pgmap v1852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:32:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:32:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:41.250+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:41.250+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:41.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:41.250+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:40 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:32:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:40 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:32:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:40 smithi107 ceph-mon[32659]: pgmap v1853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:32:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:40 smithi107 ceph-mon[32659]: pgmap v1854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:32:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2496.798751 secs 2024-07-26T13:32:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:40 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:32:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:40 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:32:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:40 smithi132 ceph-mon[37580]: pgmap v1853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:32:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:40 smithi132 ceph-mon[37580]: pgmap v1854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:32:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2496.798751 secs 2024-07-26T13:32:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:40 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:32:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:40 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:32:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:40 smithi112 ceph-mon[37501]: pgmap v1853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:32:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:40 smithi112 ceph-mon[37501]: pgmap v1854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:32:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2496.798751 secs 2024-07-26T13:32:43.252 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:42 smithi112 ceph-mon[37501]: pgmap v1855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:42 smithi107 ceph-mon[32659]: pgmap v1855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:42 smithi132 ceph-mon[37580]: pgmap v1855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:44 smithi107 ceph-mon[32659]: pgmap v1856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:32:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:44 smithi132 ceph-mon[37580]: pgmap v1856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:32:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:44 smithi112 ceph-mon[37501]: pgmap v1856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:32:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2501.798885 secs 2024-07-26T13:32:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2501.798885 secs 2024-07-26T13:32:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2501.798885 secs 2024-07-26T13:32:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:47 smithi107 ceph-mon[32659]: pgmap v1857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:32:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:47 smithi132 ceph-mon[37580]: pgmap v1857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:32:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:47 smithi112 ceph-mon[37501]: pgmap v1857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:32:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:48 smithi132 ceph-mon[37580]: pgmap v1858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:32:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:48 smithi112 ceph-mon[37501]: pgmap v1858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:32:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:48 smithi107 ceph-mon[32659]: pgmap v1858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:32:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:51 smithi107 ceph-mon[32659]: pgmap v1859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:32:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2506.799024 secs 2024-07-26T13:32:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:51 smithi132 ceph-mon[37580]: pgmap v1859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:32:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2506.799024 secs 2024-07-26T13:32:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:51 smithi112 ceph-mon[37501]: pgmap v1859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:32:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2506.799024 secs 2024-07-26T13:32:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:53 smithi107 ceph-mon[32659]: pgmap v1860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:53 smithi132 ceph-mon[37580]: pgmap v1860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:53 smithi112 ceph-mon[37501]: pgmap v1860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:32:54.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:54.083+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:54.083+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:54.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:32:54.083+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:55 smithi107 ceph-mon[32659]: pgmap v1861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:55 smithi132 ceph-mon[37580]: pgmap v1861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:55 smithi112 ceph-mon[37501]: pgmap v1861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:56.269 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:56.265+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:56.265+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:32:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:32:56.265+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2511.799141 secs 2024-07-26T13:32:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2511.799141 secs 2024-07-26T13:32:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2511.799141 secs 2024-07-26T13:32:57.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:57 smithi107 ceph-mon[32659]: pgmap v1862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:57 smithi132 ceph-mon[37580]: pgmap v1862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:57 smithi112 ceph-mon[37501]: pgmap v1862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:32:59 smithi107 ceph-mon[32659]: pgmap v1863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:32:59 smithi132 ceph-mon[37580]: pgmap v1863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:32:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:32:59 smithi112 ceph-mon[37501]: pgmap v1863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:01 smithi107 ceph-mon[32659]: pgmap v1864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2516.799261 secs 2024-07-26T13:33:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:01 smithi132 ceph-mon[37580]: pgmap v1864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2516.799261 secs 2024-07-26T13:33:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:01 smithi112 ceph-mon[37501]: pgmap v1864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2516.799261 secs 2024-07-26T13:33:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:03 smithi132 ceph-mon[37580]: pgmap v1865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:03 smithi112 ceph-mon[37501]: pgmap v1865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:03 smithi107 ceph-mon[32659]: pgmap v1865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:05 smithi132 ceph-mon[37580]: pgmap v1866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:05 smithi112 ceph-mon[37501]: pgmap v1866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:05 smithi107 ceph-mon[32659]: pgmap v1866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2521.799340 secs 2024-07-26T13:33:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2521.799340 secs 2024-07-26T13:33:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2521.799340 secs 2024-07-26T13:33:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:07 smithi112 ceph-mon[37501]: pgmap v1867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:07 smithi107 ceph-mon[32659]: pgmap v1867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:07 smithi132 ceph-mon[37580]: pgmap v1867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:09.099+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:09.099+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:09.101 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:09.099+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:09.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:09 smithi107 ceph-mon[32659]: pgmap v1868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:09 smithi112 ceph-mon[37501]: pgmap v1868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:09 smithi132 ceph-mon[37580]: pgmap v1868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:11.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:11.278+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:11.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:11.278+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:11.280 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:11.278+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:11 smithi112 ceph-mon[37501]: pgmap v1869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2526.799460 secs 2024-07-26T13:33:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:11 smithi107 ceph-mon[32659]: pgmap v1869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2526.799460 secs 2024-07-26T13:33:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:11 smithi132 ceph-mon[37580]: pgmap v1869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2526.799460 secs 2024-07-26T13:33:13.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:13.062+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:33:13.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:13.063+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:33:13.110 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:13.109+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:33:13.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:13.110+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:33:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:13 smithi112 ceph-mon[37501]: pgmap v1870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:13 smithi107 ceph-mon[32659]: pgmap v1870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:13 smithi132 ceph-mon[37580]: pgmap v1870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:14.185+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:33:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:14.185+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:33:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:14.186+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:33:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:14.186+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006670 con 0x7efc78103340 2024-07-26T13:33:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:15 smithi112 ceph-mon[37501]: pgmap v1871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:15 smithi107 ceph-mon[32659]: pgmap v1871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:15 smithi132 ceph-mon[37580]: pgmap v1871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2531.799535 secs 2024-07-26T13:33:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2531.799535 secs 2024-07-26T13:33:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2531.799535 secs 2024-07-26T13:33:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:17 smithi112 ceph-mon[37501]: pgmap v1872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:17 smithi107 ceph-mon[32659]: pgmap v1872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:17 smithi132 ceph-mon[37580]: pgmap v1872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:18 smithi112 ceph-mon[37501]: pgmap v1873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:18 smithi107 ceph-mon[32659]: pgmap v1873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:18 smithi132 ceph-mon[37580]: pgmap v1873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:21 smithi107 ceph-mon[32659]: pgmap v1874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2536.799616 secs 2024-07-26T13:33:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:21 smithi132 ceph-mon[37580]: pgmap v1874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2536.799616 secs 2024-07-26T13:33:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:21 smithi112 ceph-mon[37501]: pgmap v1874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2536.799616 secs 2024-07-26T13:33:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:23 smithi107 ceph-mon[32659]: pgmap v1875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:23 smithi132 ceph-mon[37580]: pgmap v1875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:23 smithi112 ceph-mon[37501]: pgmap v1875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:24.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:24.115+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:24.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:24.115+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:24.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:24.115+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:25 smithi107 ceph-mon[32659]: pgmap v1876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:25 smithi132 ceph-mon[37580]: pgmap v1876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:25 smithi112 ceph-mon[37501]: pgmap v1876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:26.294+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:26.294+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:26.295 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:26.294+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2541.799728 secs 2024-07-26T13:33:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2541.799728 secs 2024-07-26T13:33:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2541.799728 secs 2024-07-26T13:33:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:27 smithi132 ceph-mon[37580]: pgmap v1877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:27 smithi112 ceph-mon[37501]: pgmap v1877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:27 smithi107 ceph-mon[32659]: pgmap v1877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:29 smithi132 ceph-mon[37580]: pgmap v1878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:29 smithi112 ceph-mon[37501]: pgmap v1878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:29 smithi107 ceph-mon[32659]: pgmap v1878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:31 smithi132 ceph-mon[37580]: pgmap v1879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2546.799846 secs 2024-07-26T13:33:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:31 smithi112 ceph-mon[37501]: pgmap v1879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2546.799846 secs 2024-07-26T13:33:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:31 smithi107 ceph-mon[32659]: pgmap v1879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2546.799846 secs 2024-07-26T13:33:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:33 smithi112 ceph-mon[37501]: pgmap v1880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:33 smithi107 ceph-mon[32659]: pgmap v1880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:33 smithi132 ceph-mon[37580]: pgmap v1880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:35 smithi112 ceph-mon[37501]: pgmap v1881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:35 smithi107 ceph-mon[32659]: pgmap v1881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:35 smithi132 ceph-mon[37580]: pgmap v1881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2551.799996 secs 2024-07-26T13:33:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2551.799996 secs 2024-07-26T13:33:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2551.799996 secs 2024-07-26T13:33:37.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:37 smithi107 ceph-mon[32659]: pgmap v1882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:37 smithi112 ceph-mon[37501]: pgmap v1882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:37 smithi132 ceph-mon[37580]: pgmap v1882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:39.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:39.131+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:39.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:39.131+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:39.132 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:39.131+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:39 smithi112 ceph-mon[37501]: pgmap v1883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:39 smithi107 ceph-mon[32659]: pgmap v1883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:39 smithi132 ceph-mon[37580]: pgmap v1883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:41.310+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:41.310+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:41.310+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: pgmap v1884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2556.800113 secs 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: pgmap v1884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2556.800113 secs 2024-07-26T13:33:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: pgmap v1884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2556.800113 secs 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:42 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:33:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:42 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:33:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:42 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:33:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:42 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:33:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:42 smithi112 ceph-mon[37501]: pgmap v1885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:33:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:42 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:33:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:42 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:33:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:42 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:33:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:42 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:33:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:42 smithi107 ceph-mon[32659]: pgmap v1885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:33:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:42 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:33:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:42 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:33:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:42 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:33:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:42 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:33:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:42 smithi132 ceph-mon[37580]: pgmap v1885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:33:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:33:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:44 smithi107 ceph-mon[32659]: pgmap v1886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:33:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:44 smithi132 ceph-mon[37580]: pgmap v1886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:33:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:44 smithi112 ceph-mon[37501]: pgmap v1886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:33:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2561.800215 secs 2024-07-26T13:33:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:46 smithi107 ceph-mon[32659]: pgmap v1887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:33:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2561.800215 secs 2024-07-26T13:33:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:46 smithi132 ceph-mon[37580]: pgmap v1887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:33:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2561.800215 secs 2024-07-26T13:33:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:46 smithi112 ceph-mon[37501]: pgmap v1887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:33:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:48 smithi107 ceph-mon[32659]: pgmap v1888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:48 smithi132 ceph-mon[37580]: pgmap v1888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:48 smithi112 ceph-mon[37501]: pgmap v1888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:50 smithi107 ceph-mon[32659]: pgmap v1889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:50 smithi132 ceph-mon[37580]: pgmap v1889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:50 smithi112 ceph-mon[37501]: pgmap v1889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2566.800330 secs 2024-07-26T13:33:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2566.800330 secs 2024-07-26T13:33:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2566.800330 secs 2024-07-26T13:33:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:52 smithi107 ceph-mon[32659]: pgmap v1890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:33:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:52 smithi132 ceph-mon[37580]: pgmap v1890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:33:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:52 smithi112 ceph-mon[37501]: pgmap v1890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:33:54.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:54.143+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:54.143+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:54.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:33:54.143+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:54 smithi107 ceph-mon[32659]: pgmap v1891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:54.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:54 smithi132 ceph-mon[37580]: pgmap v1891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:54 smithi112 ceph-mon[37501]: pgmap v1891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:33:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:56.324+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:56.325+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:33:56.325+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:33:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2571.800408 secs 2024-07-26T13:33:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:56 smithi107 ceph-mon[32659]: pgmap v1892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2571.800408 secs 2024-07-26T13:33:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:56 smithi132 ceph-mon[37580]: pgmap v1892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2571.800408 secs 2024-07-26T13:33:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:56 smithi112 ceph-mon[37501]: pgmap v1892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:33:58 smithi107 ceph-mon[32659]: pgmap v1893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:33:58 smithi132 ceph-mon[37580]: pgmap v1893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:33:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:33:58 smithi112 ceph-mon[37501]: pgmap v1893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:00 smithi107 ceph-mon[32659]: pgmap v1894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:00 smithi132 ceph-mon[37580]: pgmap v1894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:00 smithi112 ceph-mon[37501]: pgmap v1894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2576.800546 secs 2024-07-26T13:34:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2576.800546 secs 2024-07-26T13:34:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2576.800546 secs 2024-07-26T13:34:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:02 smithi107 ceph-mon[32659]: pgmap v1895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:02 smithi132 ceph-mon[37580]: pgmap v1895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:02 smithi112 ceph-mon[37501]: pgmap v1895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:04 smithi107 ceph-mon[32659]: pgmap v1896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:04 smithi132 ceph-mon[37580]: pgmap v1896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:04 smithi112 ceph-mon[37501]: pgmap v1896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2581.800684 secs 2024-07-26T13:34:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:06 smithi107 ceph-mon[32659]: pgmap v1897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2581.800684 secs 2024-07-26T13:34:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:06 smithi132 ceph-mon[37580]: pgmap v1897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2581.800684 secs 2024-07-26T13:34:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:06 smithi112 ceph-mon[37501]: pgmap v1897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:08 smithi107 ceph-mon[32659]: pgmap v1898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:08 smithi132 ceph-mon[37580]: pgmap v1898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:08 smithi112 ceph-mon[37501]: pgmap v1898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:09.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:09.154+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:09.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:09.154+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:09.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:09.154+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:10 smithi132 ceph-mon[37580]: pgmap v1899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:10 smithi112 ceph-mon[37501]: pgmap v1899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:10 smithi107 ceph-mon[32659]: pgmap v1899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:11.341+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:11.341+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:11.341+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2586.800824 secs 2024-07-26T13:34:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2586.800824 secs 2024-07-26T13:34:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2586.800824 secs 2024-07-26T13:34:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:12 smithi132 ceph-mon[37580]: pgmap v1900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:12 smithi112 ceph-mon[37501]: pgmap v1900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:12 smithi107 ceph-mon[32659]: pgmap v1900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:14 smithi112 ceph-mon[37501]: pgmap v1901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:14 smithi107 ceph-mon[32659]: pgmap v1901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:14 smithi132 ceph-mon[37580]: pgmap v1901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2591.800898 secs 2024-07-26T13:34:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:16 smithi112 ceph-mon[37501]: pgmap v1902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2591.800898 secs 2024-07-26T13:34:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:16 smithi107 ceph-mon[32659]: pgmap v1902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2591.800898 secs 2024-07-26T13:34:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:16 smithi132 ceph-mon[37580]: pgmap v1902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:18 smithi112 ceph-mon[37501]: pgmap v1903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:18 smithi107 ceph-mon[32659]: pgmap v1903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:18.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:18 smithi132 ceph-mon[37580]: pgmap v1903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:20 smithi112 ceph-mon[37501]: pgmap v1904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:20 smithi107 ceph-mon[32659]: pgmap v1904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:20 smithi132 ceph-mon[37580]: pgmap v1904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2596.801016 secs 2024-07-26T13:34:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2596.801016 secs 2024-07-26T13:34:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2596.801016 secs 2024-07-26T13:34:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:22 smithi112 ceph-mon[37501]: pgmap v1905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:22 smithi107 ceph-mon[32659]: pgmap v1905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:22.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:22 smithi132 ceph-mon[37580]: pgmap v1905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:24.165+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:24.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:24.165+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:24.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:24.165+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:24 smithi107 ceph-mon[32659]: pgmap v1906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:24 smithi112 ceph-mon[37501]: pgmap v1906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:24.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:24 smithi132 ceph-mon[37580]: pgmap v1906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:26.357+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:26.357 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:26.357+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:26.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:26.357+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:26.611 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2601.801135 secs 2024-07-26T13:34:26.611 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:26 smithi132 ceph-mon[37580]: pgmap v1907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:26.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2601.801135 secs 2024-07-26T13:34:26.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:26 smithi112 ceph-mon[37501]: pgmap v1907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2601.801135 secs 2024-07-26T13:34:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:26 smithi107 ceph-mon[32659]: pgmap v1907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:28 smithi112 ceph-mon[37501]: pgmap v1908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:28 smithi107 ceph-mon[32659]: pgmap v1908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:28 smithi132 ceph-mon[37580]: pgmap v1908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:30 smithi112 ceph-mon[37501]: pgmap v1909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:30 smithi107 ceph-mon[32659]: pgmap v1909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:30 smithi132 ceph-mon[37580]: pgmap v1909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2606.801260 secs 2024-07-26T13:34:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2606.801260 secs 2024-07-26T13:34:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2606.801260 secs 2024-07-26T13:34:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:32 smithi112 ceph-mon[37501]: pgmap v1910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:32 smithi107 ceph-mon[32659]: pgmap v1910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:32 smithi132 ceph-mon[37580]: pgmap v1910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:34 smithi112 ceph-mon[37501]: pgmap v1911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:34 smithi107 ceph-mon[32659]: pgmap v1911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:34 smithi132 ceph-mon[37580]: pgmap v1911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2611.801377 secs 2024-07-26T13:34:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:36 smithi112 ceph-mon[37501]: pgmap v1912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2611.801377 secs 2024-07-26T13:34:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:36 smithi107 ceph-mon[32659]: pgmap v1912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2611.801377 secs 2024-07-26T13:34:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:36 smithi132 ceph-mon[37580]: pgmap v1912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:38 smithi112 ceph-mon[37501]: pgmap v1913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:38 smithi107 ceph-mon[32659]: pgmap v1913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:38 smithi132 ceph-mon[37580]: pgmap v1913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:39.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:39.178+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:39.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:39.178+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:39.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:39.178+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:40 smithi112 ceph-mon[37501]: pgmap v1914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:40 smithi107 ceph-mon[32659]: pgmap v1914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:40 smithi132 ceph-mon[37580]: pgmap v1914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:41.372+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:41.372+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:41.372+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2616.801494 secs 2024-07-26T13:34:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2616.801494 secs 2024-07-26T13:34:41.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2616.801494 secs 2024-07-26T13:34:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:42 smithi112 ceph-mon[37501]: pgmap v1915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:42 smithi107 ceph-mon[32659]: pgmap v1915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:42 smithi132 ceph-mon[37580]: pgmap v1915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:34:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:43 smithi112 ceph-mon[37501]: pgmap v1916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:34:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:43 smithi112 ceph-mon[37501]: pgmap v1917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:43 smithi107 ceph-mon[32659]: pgmap v1916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:34:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:43 smithi107 ceph-mon[32659]: pgmap v1917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:43 smithi132 ceph-mon[37580]: pgmap v1916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:34:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:43 smithi132 ceph-mon[37580]: pgmap v1917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:45 smithi112 ceph-mon[37501]: pgmap v1918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:34:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:45 smithi107 ceph-mon[32659]: pgmap v1918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:34:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:45 smithi132 ceph-mon[37580]: pgmap v1918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:34:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2621.801613 secs 2024-07-26T13:34:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2621.801613 secs 2024-07-26T13:34:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2621.801613 secs 2024-07-26T13:34:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:47 smithi112 ceph-mon[37501]: pgmap v1919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:34:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:47 smithi107 ceph-mon[32659]: pgmap v1919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:34:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:47 smithi132 ceph-mon[37580]: pgmap v1919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:34:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:49 smithi112 ceph-mon[37501]: pgmap v1920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:34:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:49 smithi107 ceph-mon[32659]: pgmap v1920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:34:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:49 smithi132 ceph-mon[37580]: pgmap v1920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:34:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2626.801731 secs 2024-07-26T13:34:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:51 smithi107 ceph-mon[32659]: pgmap v1921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:34:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2626.801731 secs 2024-07-26T13:34:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:51 smithi132 ceph-mon[37580]: pgmap v1921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:34:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2626.801731 secs 2024-07-26T13:34:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:51 smithi112 ceph-mon[37501]: pgmap v1921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:34:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:53 smithi107 ceph-mon[32659]: pgmap v1922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:34:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:53 smithi132 ceph-mon[37580]: pgmap v1922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:34:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:53 smithi112 ceph-mon[37501]: pgmap v1922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:34:54.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:54.193+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:54.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:54.193+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:54.195 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:34:54.193+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:55 smithi107 ceph-mon[32659]: pgmap v1923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:55 smithi132 ceph-mon[37580]: pgmap v1923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:55 smithi112 ceph-mon[37501]: pgmap v1923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:56.385+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:56.385+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:34:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:34:56.385+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:56.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2631.801849 secs 2024-07-26T13:34:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2631.801849 secs 2024-07-26T13:34:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2631.801849 secs 2024-07-26T13:34:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:57 smithi107 ceph-mon[32659]: pgmap v1924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:57 smithi132 ceph-mon[37580]: pgmap v1924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:57 smithi112 ceph-mon[37501]: pgmap v1924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:34:59 smithi107 ceph-mon[32659]: pgmap v1925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:34:59 smithi132 ceph-mon[37580]: pgmap v1925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:34:59 smithi112 ceph-mon[37501]: pgmap v1925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:00 smithi107 ceph-mon[32659]: pgmap v1926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:00 smithi132 ceph-mon[37580]: pgmap v1926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:00 smithi112 ceph-mon[37501]: pgmap v1926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2636.801975 secs 2024-07-26T13:35:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2636.801975 secs 2024-07-26T13:35:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2636.801975 secs 2024-07-26T13:35:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:02 smithi107 ceph-mon[32659]: pgmap v1927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:02 smithi132 ceph-mon[37580]: pgmap v1927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:02 smithi112 ceph-mon[37501]: pgmap v1927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:05 smithi112 ceph-mon[37501]: pgmap v1928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:05 smithi107 ceph-mon[32659]: pgmap v1928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:05 smithi132 ceph-mon[37580]: pgmap v1928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2641.802095 secs 2024-07-26T13:35:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2641.802095 secs 2024-07-26T13:35:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2641.802095 secs 2024-07-26T13:35:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:07 smithi112 ceph-mon[37501]: pgmap v1929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:07 smithi107 ceph-mon[32659]: pgmap v1929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:07 smithi132 ceph-mon[37580]: pgmap v1929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:09.208+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:09.208+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:09.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:09.208+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:09 smithi107 ceph-mon[32659]: pgmap v1930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:09 smithi112 ceph-mon[37501]: pgmap v1930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:09 smithi132 ceph-mon[37580]: pgmap v1930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:11.402 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:11.401+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:11.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:11.401+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:11.403 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:11.401+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2646.802142 secs 2024-07-26T13:35:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:11 smithi112 ceph-mon[37501]: pgmap v1931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2646.802142 secs 2024-07-26T13:35:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:11 smithi107 ceph-mon[32659]: pgmap v1931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2646.802142 secs 2024-07-26T13:35:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:11 smithi132 ceph-mon[37580]: pgmap v1931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:13 smithi112 ceph-mon[37501]: pgmap v1932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:13 smithi107 ceph-mon[32659]: pgmap v1932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:13 smithi132 ceph-mon[37580]: pgmap v1932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:15 smithi112 ceph-mon[37501]: pgmap v1933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:15 smithi107 ceph-mon[32659]: pgmap v1933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:15 smithi132 ceph-mon[37580]: pgmap v1933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2651.802281 secs 2024-07-26T13:35:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2651.802281 secs 2024-07-26T13:35:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2651.802281 secs 2024-07-26T13:35:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:17 smithi112 ceph-mon[37501]: pgmap v1934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:35:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:17 smithi107 ceph-mon[32659]: pgmap v1934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:35:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:17 smithi132 ceph-mon[37580]: pgmap v1934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:35:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:19 smithi112 ceph-mon[37501]: pgmap v1935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:19 smithi107 ceph-mon[32659]: pgmap v1935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:19 smithi132 ceph-mon[37580]: pgmap v1935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2656.802401 secs 2024-07-26T13:35:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:21 smithi112 ceph-mon[37501]: pgmap v1936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2656.802401 secs 2024-07-26T13:35:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:21 smithi107 ceph-mon[32659]: pgmap v1936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2656.802401 secs 2024-07-26T13:35:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:21 smithi132 ceph-mon[37580]: pgmap v1936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:23 smithi112 ceph-mon[37501]: pgmap v1937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:23.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:23 smithi107 ceph-mon[32659]: pgmap v1937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:23 smithi132 ceph-mon[37580]: pgmap v1937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:24.223+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:24.223+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:24.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:24.223+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:25 smithi107 ceph-mon[32659]: pgmap v1938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:25 smithi132 ceph-mon[37580]: pgmap v1938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:25 smithi112 ceph-mon[37501]: pgmap v1938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:26.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:26.417+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:26.417+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:26.417+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2661.802549 secs 2024-07-26T13:35:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2661.802549 secs 2024-07-26T13:35:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2661.802549 secs 2024-07-26T13:35:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:27 smithi107 ceph-mon[32659]: pgmap v1939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:27 smithi132 ceph-mon[37580]: pgmap v1939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:27 smithi112 ceph-mon[37501]: pgmap v1939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:29 smithi107 ceph-mon[32659]: pgmap v1940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:29 smithi132 ceph-mon[37580]: pgmap v1940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:29 smithi112 ceph-mon[37501]: pgmap v1940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2666.802608 secs 2024-07-26T13:35:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:31 smithi107 ceph-mon[32659]: pgmap v1941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2666.802608 secs 2024-07-26T13:35:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:31 smithi132 ceph-mon[37580]: pgmap v1941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2666.802608 secs 2024-07-26T13:35:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:31 smithi112 ceph-mon[37501]: pgmap v1941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:33 smithi107 ceph-mon[32659]: pgmap v1942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:33 smithi132 ceph-mon[37580]: pgmap v1942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:33 smithi112 ceph-mon[37501]: pgmap v1942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:34 smithi107 ceph-mon[32659]: pgmap v1943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:34 smithi132 ceph-mon[37580]: pgmap v1943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:34 smithi112 ceph-mon[37501]: pgmap v1943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2671.802735 secs 2024-07-26T13:35:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2671.802735 secs 2024-07-26T13:35:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2671.802735 secs 2024-07-26T13:35:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:36 smithi132 ceph-mon[37580]: pgmap v1944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:36 smithi112 ceph-mon[37501]: pgmap v1944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:36 smithi107 ceph-mon[32659]: pgmap v1944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:39.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:39.239+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:39.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:39.239+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:39.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:39.239+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:39 smithi107 ceph-mon[32659]: pgmap v1945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:35:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:39 smithi112 ceph-mon[37501]: pgmap v1945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:35:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:39 smithi132 ceph-mon[37580]: pgmap v1945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:35:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:41.433+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:41.433+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:41.434 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:41.433+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2676.802849 secs 2024-07-26T13:35:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:41 smithi112 ceph-mon[37501]: pgmap v1946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2676.802849 secs 2024-07-26T13:35:41.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:41 smithi107 ceph-mon[32659]: pgmap v1946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2676.802849 secs 2024-07-26T13:35:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:41 smithi132 ceph-mon[37580]: pgmap v1946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:43 smithi112 ceph-mon[37501]: pgmap v1947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:43 smithi107 ceph-mon[32659]: pgmap v1947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:43 smithi132 ceph-mon[37580]: pgmap v1947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:44 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:44 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:44 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:35:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:45 smithi112 ceph-mon[37501]: pgmap v1948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:35:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:45 smithi112 ceph-mon[37501]: pgmap v1949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:35:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:45 smithi107 ceph-mon[32659]: pgmap v1948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:35:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:45 smithi107 ceph-mon[32659]: pgmap v1949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:35:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:45 smithi132 ceph-mon[37580]: pgmap v1948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:35:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:45 smithi132 ceph-mon[37580]: pgmap v1949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:35:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2681.802965 secs 2024-07-26T13:35:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2681.802965 secs 2024-07-26T13:35:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2681.802965 secs 2024-07-26T13:35:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:47 smithi112 ceph-mon[37501]: pgmap v1950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:47 smithi107 ceph-mon[32659]: pgmap v1950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:47 smithi132 ceph-mon[37580]: pgmap v1950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:49 smithi112 ceph-mon[37501]: pgmap v1951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:35:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:49 smithi107 ceph-mon[32659]: pgmap v1951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:35:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:49 smithi132 ceph-mon[37580]: pgmap v1951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:35:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2686.803106 secs 2024-07-26T13:35:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:51 smithi107 ceph-mon[32659]: pgmap v1952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2686.803106 secs 2024-07-26T13:35:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:51 smithi132 ceph-mon[37580]: pgmap v1952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2686.803106 secs 2024-07-26T13:35:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:51 smithi112 ceph-mon[37501]: pgmap v1952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:53 smithi107 ceph-mon[32659]: pgmap v1953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:53 smithi132 ceph-mon[37580]: pgmap v1953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:53 smithi112 ceph-mon[37501]: pgmap v1953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:35:54.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:54.253+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:54.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:54.253+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:54.256 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:35:54.253+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:55 smithi107 ceph-mon[32659]: pgmap v1954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:55 smithi132 ceph-mon[37580]: pgmap v1954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:55 smithi112 ceph-mon[37501]: pgmap v1954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:56.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:56.448+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:56.448+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:35:56.448+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:35:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2691.803190 secs 2024-07-26T13:35:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2691.803190 secs 2024-07-26T13:35:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2691.803190 secs 2024-07-26T13:35:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:57 smithi107 ceph-mon[32659]: pgmap v1955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:57 smithi132 ceph-mon[37580]: pgmap v1955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:57 smithi112 ceph-mon[37501]: pgmap v1955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:35:59 smithi107 ceph-mon[32659]: pgmap v1956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:35:59 smithi132 ceph-mon[37580]: pgmap v1956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:35:59 smithi112 ceph-mon[37501]: pgmap v1956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2696.803305 secs 2024-07-26T13:36:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:01 smithi107 ceph-mon[32659]: pgmap v1957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2696.803305 secs 2024-07-26T13:36:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:01 smithi132 ceph-mon[37580]: pgmap v1957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2696.803305 secs 2024-07-26T13:36:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:01 smithi112 ceph-mon[37501]: pgmap v1957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:03 smithi107 ceph-mon[32659]: pgmap v1958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:03 smithi132 ceph-mon[37580]: pgmap v1958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:03 smithi112 ceph-mon[37501]: pgmap v1958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:04 smithi107 ceph-mon[32659]: pgmap v1959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:04 smithi132 ceph-mon[37580]: pgmap v1959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:04 smithi112 ceph-mon[37501]: pgmap v1959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2701.803383 secs 2024-07-26T13:36:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2701.803383 secs 2024-07-26T13:36:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2701.803383 secs 2024-07-26T13:36:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:06 smithi112 ceph-mon[37501]: pgmap v1960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:06 smithi107 ceph-mon[32659]: pgmap v1960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:06 smithi132 ceph-mon[37580]: pgmap v1960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:09.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:09.269+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:09.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:09.269+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:09.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:09.269+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:09 smithi107 ceph-mon[32659]: pgmap v1961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:09 smithi132 ceph-mon[37580]: pgmap v1961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:09 smithi112 ceph-mon[37501]: pgmap v1961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:11.457+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:11.457+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:11.457+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2706.803459 secs 2024-07-26T13:36:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:11 smithi107 ceph-mon[32659]: pgmap v1962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:36:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2706.803459 secs 2024-07-26T13:36:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:11 smithi132 ceph-mon[37580]: pgmap v1962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:36:11.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:11.643+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0c1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000233) for more than 900000000 us, fault. 2024-07-26T13:36:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:11.643+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c1e10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f8047040 comp rx=0 tx=0).stop 2024-07-26T13:36:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:11.643+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0c1e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:36:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:11.643+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c1e10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:36:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:11.643+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:36:11.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:11.643+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:36:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:11.646+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c2350 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fa00 tx=0x7ff0f801f040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:36:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2706.803459 secs 2024-07-26T13:36:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:11 smithi112 ceph-mon[37501]: pgmap v1962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:36:13.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:13.093+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:36:13.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:13.093+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:36:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:13.139+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:36:13.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:13.139+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:36:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:13 smithi107 ceph-mon[32659]: pgmap v1963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:36:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:13 smithi132 ceph-mon[37580]: pgmap v1963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:36:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:13 smithi112 ceph-mon[37501]: pgmap v1963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:36:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:13.751+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 msgr2=0x7efc5c0c1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T13:36:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:13.751+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 msgr2=0x7efc5c0c1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:36:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:13.751+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0c1d70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efc680aacd0 tx=0x7efc680e5000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:36:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:13.751+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0c1d70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7efc680aacd0 tx=0x7efc680e5000 comp rx=0 tx=0).stop 2024-07-26T13:36:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:13.751+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 msgr2=0x7efc5c0c1d70 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T13:36:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:13.751+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 msgr2=0x7efc5c0c1d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:36:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:13.751+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0b3d40 0x7efc5c0c1d70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:36:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:13.751+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:36:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:13.752+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:36:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:13.754+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c22b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efc48005820 tx=0x7efc680c1000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:36:14.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:14.217+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:36:14.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:14.217+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:36:14.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:14.218+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:36:14.219 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:14.218+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:36:15.502 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:15 smithi112 ceph-mon[37501]: pgmap v1964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:15 smithi107 ceph-mon[32659]: pgmap v1964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:15 smithi132 ceph-mon[37580]: pgmap v1964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2711.803539 secs 2024-07-26T13:36:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2711.803539 secs 2024-07-26T13:36:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2711.803539 secs 2024-07-26T13:36:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:17 smithi107 ceph-mon[32659]: pgmap v1965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:36:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:17 smithi132 ceph-mon[37580]: pgmap v1965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:36:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:17 smithi112 ceph-mon[37501]: pgmap v1965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:36:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:19 smithi107 ceph-mon[32659]: pgmap v1966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:19.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:19 smithi132 ceph-mon[37580]: pgmap v1966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:19 smithi112 ceph-mon[37501]: pgmap v1966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2716.803676 secs 2024-07-26T13:36:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:21 smithi107 ceph-mon[32659]: pgmap v1967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2716.803676 secs 2024-07-26T13:36:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:21 smithi132 ceph-mon[37580]: pgmap v1967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2716.803676 secs 2024-07-26T13:36:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:21 smithi112 ceph-mon[37501]: pgmap v1967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:23 smithi107 ceph-mon[32659]: pgmap v1968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:23 smithi132 ceph-mon[37580]: pgmap v1968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:23 smithi112 ceph-mon[37501]: pgmap v1968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:24.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:24.283+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:24.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:24.283+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:24.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:24.283+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:25 smithi132 ceph-mon[37580]: pgmap v1969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:25 smithi112 ceph-mon[37501]: pgmap v1969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:25 smithi107 ceph-mon[32659]: pgmap v1969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:26.474 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:26.474+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:26.474+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:26.474+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2721.803826 secs 2024-07-26T13:36:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2721.803826 secs 2024-07-26T13:36:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2721.803826 secs 2024-07-26T13:36:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:27 smithi112 ceph-mon[37501]: pgmap v1970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:27 smithi107 ceph-mon[32659]: pgmap v1970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:27 smithi132 ceph-mon[37580]: pgmap v1970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:29 smithi107 ceph-mon[32659]: pgmap v1971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:29 smithi112 ceph-mon[37501]: pgmap v1971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:29 smithi132 ceph-mon[37580]: pgmap v1971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2726.803946 secs 2024-07-26T13:36:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:31 smithi112 ceph-mon[37501]: pgmap v1972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2726.803946 secs 2024-07-26T13:36:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:31 smithi107 ceph-mon[32659]: pgmap v1972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2726.803946 secs 2024-07-26T13:36:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:31 smithi132 ceph-mon[37580]: pgmap v1972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:33 smithi112 ceph-mon[37501]: pgmap v1973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:33 smithi107 ceph-mon[32659]: pgmap v1973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:33 smithi132 ceph-mon[37580]: pgmap v1973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:35 smithi112 ceph-mon[37501]: pgmap v1974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:35 smithi107 ceph-mon[32659]: pgmap v1974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:35 smithi132 ceph-mon[37580]: pgmap v1974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2731.804090 secs 2024-07-26T13:36:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2731.804090 secs 2024-07-26T13:36:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2731.804090 secs 2024-07-26T13:36:37.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:37 smithi112 ceph-mon[37501]: pgmap v1975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:37 smithi107 ceph-mon[32659]: pgmap v1975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:37 smithi132 ceph-mon[37580]: pgmap v1975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:39.298+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:39.298+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:39.300 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:39.298+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:39.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:39 smithi107 ceph-mon[32659]: pgmap v1976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:39 smithi112 ceph-mon[37501]: pgmap v1976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:39 smithi132 ceph-mon[37580]: pgmap v1976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:41.489+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:41.489+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:41.489+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2736.804212 secs 2024-07-26T13:36:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:41 smithi112 ceph-mon[37501]: pgmap v1977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2736.804212 secs 2024-07-26T13:36:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:41 smithi107 ceph-mon[32659]: pgmap v1977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2736.804212 secs 2024-07-26T13:36:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:41 smithi132 ceph-mon[37580]: pgmap v1977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:43 smithi112 ceph-mon[37501]: pgmap v1978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:43 smithi107 ceph-mon[32659]: pgmap v1978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:43 smithi132 ceph-mon[37580]: pgmap v1978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:45.646 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:45 smithi132 ceph-mon[37580]: pgmap v1979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:45.647 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:45.647 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.647 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.647 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.647 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:45 smithi107 ceph-mon[32659]: pgmap v1979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:45.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:45.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.680 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:45 smithi112 ceph-mon[37501]: pgmap v1979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:45.681 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:45.681 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.681 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.681 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:45.681 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2741.804294 secs 2024-07-26T13:36:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2741.804294 secs 2024-07-26T13:36:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2741.804294 secs 2024-07-26T13:36:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:47 smithi112 ceph-mon[37501]: pgmap v1980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:47 smithi107 ceph-mon[32659]: pgmap v1980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:47 smithi132 ceph-mon[37580]: pgmap v1980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:48 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:48 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:48 smithi107 ceph-mon[32659]: pgmap v1981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:48 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:48 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:48 smithi132 ceph-mon[37580]: pgmap v1981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:48 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:48 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:48 smithi112 ceph-mon[37501]: pgmap v1981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:50 smithi112 ceph-mon[37501]: pgmap v1982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:36:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:50 smithi112 ceph-mon[37501]: pgmap v1983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:50 smithi107 ceph-mon[32659]: pgmap v1982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:36:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:50 smithi107 ceph-mon[32659]: pgmap v1983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:50 smithi132 ceph-mon[37580]: pgmap v1982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:36:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:50 smithi132 ceph-mon[37580]: pgmap v1983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:36:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2746.804419 secs 2024-07-26T13:36:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2746.804419 secs 2024-07-26T13:36:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2746.804419 secs 2024-07-26T13:36:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:52 smithi112 ceph-mon[37501]: pgmap v1984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:36:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:52 smithi107 ceph-mon[32659]: pgmap v1984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:36:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:52 smithi132 ceph-mon[37580]: pgmap v1984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:36:54.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:54.313+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:54.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:54.313+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:54.316 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:36:54.313+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:54 smithi112 ceph-mon[37501]: pgmap v1985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:54 smithi107 ceph-mon[32659]: pgmap v1985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:54 smithi132 ceph-mon[37580]: pgmap v1985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:56.504+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:56.504+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:56.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:36:56.504+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:36:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2751.804557 secs 2024-07-26T13:36:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:56 smithi112 ceph-mon[37501]: pgmap v1986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2751.804557 secs 2024-07-26T13:36:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:56 smithi107 ceph-mon[32659]: pgmap v1986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2751.804557 secs 2024-07-26T13:36:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:56 smithi132 ceph-mon[37580]: pgmap v1986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:36:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:36:58 smithi112 ceph-mon[37501]: pgmap v1987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:36:58 smithi107 ceph-mon[32659]: pgmap v1987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:36:58 smithi132 ceph-mon[37580]: pgmap v1987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:00 smithi112 ceph-mon[37501]: pgmap v1988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:37:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:00 smithi107 ceph-mon[32659]: pgmap v1988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:37:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:00 smithi132 ceph-mon[37580]: pgmap v1988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:37:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2756.804694 secs 2024-07-26T13:37:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2756.804694 secs 2024-07-26T13:37:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2756.804694 secs 2024-07-26T13:37:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:02 smithi107 ceph-mon[32659]: pgmap v1989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:02 smithi132 ceph-mon[37580]: pgmap v1989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:02 smithi112 ceph-mon[37501]: pgmap v1989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:04 smithi107 ceph-mon[32659]: pgmap v1990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:04 smithi132 ceph-mon[37580]: pgmap v1990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:04 smithi112 ceph-mon[37501]: pgmap v1990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2761.804803 secs 2024-07-26T13:37:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:06 smithi107 ceph-mon[32659]: pgmap v1991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2761.804803 secs 2024-07-26T13:37:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:06 smithi132 ceph-mon[37580]: pgmap v1991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2761.804803 secs 2024-07-26T13:37:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:06 smithi112 ceph-mon[37501]: pgmap v1991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:08 smithi107 ceph-mon[32659]: pgmap v1992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:08 smithi132 ceph-mon[37580]: pgmap v1992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:08 smithi112 ceph-mon[37501]: pgmap v1992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:09.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:09.329+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:09.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:09.329+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:09.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:09.329+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:10 smithi107 ceph-mon[32659]: pgmap v1993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:10 smithi132 ceph-mon[37580]: pgmap v1993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:10 smithi112 ceph-mon[37501]: pgmap v1993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:11.520+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:11.520+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:11.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:11.520+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2766.804925 secs 2024-07-26T13:37:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2766.804925 secs 2024-07-26T13:37:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2766.804925 secs 2024-07-26T13:37:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:12 smithi107 ceph-mon[32659]: pgmap v1994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:12 smithi132 ceph-mon[37580]: pgmap v1994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:12 smithi112 ceph-mon[37501]: pgmap v1994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:13 smithi112 ceph-mon[37501]: pgmap v1995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:13 smithi107 ceph-mon[32659]: pgmap v1995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:13 smithi132 ceph-mon[37580]: pgmap v1995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2771.805001 secs 2024-07-26T13:37:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:16 smithi112 ceph-mon[37501]: pgmap v1996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2771.805001 secs 2024-07-26T13:37:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:16 smithi107 ceph-mon[32659]: pgmap v1996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2771.805001 secs 2024-07-26T13:37:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:16 smithi132 ceph-mon[37580]: pgmap v1996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:18 smithi112 ceph-mon[37501]: pgmap v1997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:18 smithi107 ceph-mon[32659]: pgmap v1997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:18.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:18 smithi132 ceph-mon[37580]: pgmap v1997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:20 smithi112 ceph-mon[37501]: pgmap v1998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:20 smithi107 ceph-mon[32659]: pgmap v1998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:20 smithi132 ceph-mon[37580]: pgmap v1998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:21.739 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2776.805103 secs 2024-07-26T13:37:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2776.805103 secs 2024-07-26T13:37:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2776.805103 secs 2024-07-26T13:37:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:22 smithi107 ceph-mon[32659]: pgmap v1999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:22.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:22 smithi132 ceph-mon[37580]: pgmap v1999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:22 smithi112 ceph-mon[37501]: pgmap v1999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:24.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:24.334+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:24.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:24.335+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:24.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:24.335+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:24 smithi107 ceph-mon[32659]: pgmap v2000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:24 smithi132 ceph-mon[37580]: pgmap v2000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:24 smithi112 ceph-mon[37501]: pgmap v2000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:26.536+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:26.536+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:26.537 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:26.536+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2781.805226 secs 2024-07-26T13:37:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:26 smithi107 ceph-mon[32659]: pgmap v2001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2781.805226 secs 2024-07-26T13:37:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:26 smithi132 ceph-mon[37580]: pgmap v2001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2781.805226 secs 2024-07-26T13:37:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:26 smithi112 ceph-mon[37501]: pgmap v2001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:28 smithi107 ceph-mon[32659]: pgmap v2002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:28 smithi132 ceph-mon[37580]: pgmap v2002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:28 smithi112 ceph-mon[37501]: pgmap v2002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:30 smithi107 ceph-mon[32659]: pgmap v2003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:30 smithi132 ceph-mon[37580]: pgmap v2003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:30 smithi112 ceph-mon[37501]: pgmap v2003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2786.805346 secs 2024-07-26T13:37:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2786.805346 secs 2024-07-26T13:37:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2786.805346 secs 2024-07-26T13:37:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:32 smithi107 ceph-mon[32659]: pgmap v2004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:32 smithi132 ceph-mon[37580]: pgmap v2004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:32 smithi112 ceph-mon[37501]: pgmap v2004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:34 smithi107 ceph-mon[32659]: pgmap v2005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:34 smithi132 ceph-mon[37580]: pgmap v2005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:34 smithi112 ceph-mon[37501]: pgmap v2005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2791.805450 secs 2024-07-26T13:37:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:36 smithi107 ceph-mon[32659]: pgmap v2006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2791.805450 secs 2024-07-26T13:37:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:36 smithi132 ceph-mon[37580]: pgmap v2006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2791.805450 secs 2024-07-26T13:37:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:36 smithi112 ceph-mon[37501]: pgmap v2006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:38 smithi132 ceph-mon[37580]: pgmap v2007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:38 smithi107 ceph-mon[32659]: pgmap v2007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:38 smithi112 ceph-mon[37501]: pgmap v2007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:39.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:39.350+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:39.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:39.350+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:39.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:39.350+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:39.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:39 smithi107 ceph-mon[32659]: pgmap v2008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:39 smithi132 ceph-mon[37580]: pgmap v2008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:39 smithi112 ceph-mon[37501]: pgmap v2008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2796.805570 secs 2024-07-26T13:37:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2796.805570 secs 2024-07-26T13:37:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2796.805570 secs 2024-07-26T13:37:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:41.552+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:41.552+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:41.552+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:41 smithi112 ceph-mon[37501]: pgmap v2009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:41 smithi107 ceph-mon[32659]: pgmap v2009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:41 smithi132 ceph-mon[37580]: pgmap v2009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:44 smithi107 ceph-mon[32659]: pgmap v2010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:44 smithi132 ceph-mon[37580]: pgmap v2010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:44 smithi112 ceph-mon[37501]: pgmap v2010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:46.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2801.805666 secs 2024-07-26T13:37:46.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:46 smithi112 ceph-mon[37501]: pgmap v2011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2801.805666 secs 2024-07-26T13:37:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:46 smithi107 ceph-mon[32659]: pgmap v2011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2801.805666 secs 2024-07-26T13:37:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:46 smithi132 ceph-mon[37580]: pgmap v2011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:48 smithi107 ceph-mon[32659]: pgmap v2012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:48 smithi132 ceph-mon[37580]: pgmap v2012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:48 smithi112 ceph-mon[37501]: pgmap v2012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:50 smithi107 ceph-mon[32659]: pgmap v2013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:50 smithi132 ceph-mon[37580]: pgmap v2013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:50 smithi112 ceph-mon[37501]: pgmap v2013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2806.805813 secs 2024-07-26T13:37:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:37:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2806.805813 secs 2024-07-26T13:37:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:37:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2806.805813 secs 2024-07-26T13:37:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:37:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:52 smithi107 ceph-mon[32659]: pgmap v2014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:37:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:52 smithi107 ceph-mon[32659]: pgmap v2015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:37:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:52 smithi132 ceph-mon[37580]: pgmap v2014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:37:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:52 smithi132 ceph-mon[37580]: pgmap v2015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:37:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:52 smithi112 ceph-mon[37501]: pgmap v2014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:37:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:52 smithi112 ceph-mon[37501]: pgmap v2015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:37:54.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:54.365+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:54.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:54.365+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:54.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:37:54.365+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:54 smithi107 ceph-mon[32659]: pgmap v2016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:37:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:54 smithi132 ceph-mon[37580]: pgmap v2016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:37:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:54 smithi112 ceph-mon[37501]: pgmap v2016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:37:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:56.566+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:56.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:56.566+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:56.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:37:56.566+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:37:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:56 smithi107 ceph-mon[32659]: pgmap v2017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:37:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2811.805872 secs 2024-07-26T13:37:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:56 smithi132 ceph-mon[37580]: pgmap v2017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:37:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2811.805872 secs 2024-07-26T13:37:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:56 smithi112 ceph-mon[37501]: pgmap v2017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:37:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2811.805872 secs 2024-07-26T13:37:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:37:58 smithi107 ceph-mon[32659]: pgmap v2018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:37:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:37:58 smithi132 ceph-mon[37580]: pgmap v2018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:37:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:37:58 smithi112 ceph-mon[37501]: pgmap v2018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:00 smithi107 ceph-mon[32659]: pgmap v2019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:00 smithi132 ceph-mon[37580]: pgmap v2019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:00 smithi112 ceph-mon[37501]: pgmap v2019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2816.806013 secs 2024-07-26T13:38:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2816.806013 secs 2024-07-26T13:38:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2816.806013 secs 2024-07-26T13:38:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:02 smithi107 ceph-mon[32659]: pgmap v2020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:02 smithi132 ceph-mon[37580]: pgmap v2020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:02 smithi112 ceph-mon[37501]: pgmap v2020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:03 smithi107 ceph-mon[32659]: pgmap v2021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:03 smithi132 ceph-mon[37580]: pgmap v2021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:03 smithi112 ceph-mon[37501]: pgmap v2021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:05 smithi112 ceph-mon[37501]: pgmap v2022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2821.806138 secs 2024-07-26T13:38:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:05 smithi107 ceph-mon[32659]: pgmap v2022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2821.806138 secs 2024-07-26T13:38:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:05 smithi132 ceph-mon[37580]: pgmap v2022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2821.806138 secs 2024-07-26T13:38:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:07 smithi112 ceph-mon[37501]: pgmap v2023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:38:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:07 smithi107 ceph-mon[32659]: pgmap v2023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:38:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:07 smithi132 ceph-mon[37580]: pgmap v2023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:38:09.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:09.381+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:09.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:09.381+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:09.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:09.381+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:09 smithi112 ceph-mon[37501]: pgmap v2024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:09 smithi107 ceph-mon[32659]: pgmap v2024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:09 smithi132 ceph-mon[37580]: pgmap v2024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2826.806234 secs 2024-07-26T13:38:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2826.806234 secs 2024-07-26T13:38:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2826.806234 secs 2024-07-26T13:38:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:11.582+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:11.582+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:11.582+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:11 smithi112 ceph-mon[37501]: pgmap v2025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:11 smithi107 ceph-mon[32659]: pgmap v2025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:11 smithi132 ceph-mon[37580]: pgmap v2025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:13 smithi112 ceph-mon[37501]: pgmap v2026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:38:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:13 smithi107 ceph-mon[32659]: pgmap v2026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:38:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:13 smithi132 ceph-mon[37580]: pgmap v2026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:38:15.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:15 smithi112 ceph-mon[37501]: pgmap v2027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:15.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2831.806326 secs 2024-07-26T13:38:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:15 smithi107 ceph-mon[32659]: pgmap v2027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2831.806326 secs 2024-07-26T13:38:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:15 smithi132 ceph-mon[37580]: pgmap v2027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2831.806326 secs 2024-07-26T13:38:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:17 smithi112 ceph-mon[37501]: pgmap v2028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:38:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:17 smithi107 ceph-mon[32659]: pgmap v2028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:38:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:17 smithi132 ceph-mon[37580]: pgmap v2028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:38:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:19 smithi112 ceph-mon[37501]: pgmap v2029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:38:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:19 smithi107 ceph-mon[32659]: pgmap v2029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:38:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:19 smithi132 ceph-mon[37580]: pgmap v2029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:38:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2836.806430 secs 2024-07-26T13:38:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2836.806430 secs 2024-07-26T13:38:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2836.806430 secs 2024-07-26T13:38:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:21 smithi112 ceph-mon[37501]: pgmap v2030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:38:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:21 smithi107 ceph-mon[32659]: pgmap v2030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:38:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:21 smithi132 ceph-mon[37580]: pgmap v2030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:38:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:23 smithi107 ceph-mon[32659]: pgmap v2031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:38:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:23 smithi132 ceph-mon[37580]: pgmap v2031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:38:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:23 smithi112 ceph-mon[37501]: pgmap v2031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:38:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:24.390+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:24.390+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:24.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:24.390+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:25 smithi107 ceph-mon[32659]: pgmap v2032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2841.806554 secs 2024-07-26T13:38:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:25 smithi132 ceph-mon[37580]: pgmap v2032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2841.806554 secs 2024-07-26T13:38:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:25 smithi112 ceph-mon[37501]: pgmap v2032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2841.806554 secs 2024-07-26T13:38:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:26.592+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:26.592+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:26.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:26.592+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:27 smithi107 ceph-mon[32659]: pgmap v2033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:38:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:27 smithi132 ceph-mon[37580]: pgmap v2033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:38:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:27 smithi112 ceph-mon[37501]: pgmap v2033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:38:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:29 smithi107 ceph-mon[32659]: pgmap v2034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:29 smithi132 ceph-mon[37580]: pgmap v2034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:29 smithi112 ceph-mon[37501]: pgmap v2034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2846.806651 secs 2024-07-26T13:38:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2846.806651 secs 2024-07-26T13:38:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2846.806651 secs 2024-07-26T13:38:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:31 smithi107 ceph-mon[32659]: pgmap v2035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:31 smithi132 ceph-mon[37580]: pgmap v2035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:31 smithi112 ceph-mon[37501]: pgmap v2035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:33 smithi107 ceph-mon[32659]: pgmap v2036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:33 smithi132 ceph-mon[37580]: pgmap v2036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:33 smithi112 ceph-mon[37501]: pgmap v2036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:35 smithi107 ceph-mon[32659]: pgmap v2037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2851.806746 secs 2024-07-26T13:38:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:35 smithi132 ceph-mon[37580]: pgmap v2037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2851.806746 secs 2024-07-26T13:38:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:35 smithi112 ceph-mon[37501]: pgmap v2037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2851.806746 secs 2024-07-26T13:38:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:37 smithi107 ceph-mon[32659]: pgmap v2038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:38:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:37 smithi132 ceph-mon[37580]: pgmap v2038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:38:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:37 smithi112 ceph-mon[37501]: pgmap v2038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:38:39.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:39.402+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:39.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:39.402+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:39.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:39.402+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:39 smithi107 ceph-mon[32659]: pgmap v2039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:39 smithi132 ceph-mon[37580]: pgmap v2039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:39 smithi112 ceph-mon[37501]: pgmap v2039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2856.806846 secs 2024-07-26T13:38:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2856.806846 secs 2024-07-26T13:38:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2856.806846 secs 2024-07-26T13:38:41.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:41.605+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:41.606+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:41.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:41.606+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:41 smithi107 ceph-mon[32659]: pgmap v2040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:41 smithi132 ceph-mon[37580]: pgmap v2040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:41 smithi112 ceph-mon[37501]: pgmap v2040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:43 smithi107 ceph-mon[32659]: pgmap v2041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:43 smithi132 ceph-mon[37580]: pgmap v2041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:43 smithi112 ceph-mon[37501]: pgmap v2041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:45 smithi107 ceph-mon[32659]: pgmap v2042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2861.806966 secs 2024-07-26T13:38:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:45 smithi132 ceph-mon[37580]: pgmap v2042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2861.806966 secs 2024-07-26T13:38:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:45 smithi112 ceph-mon[37501]: pgmap v2042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:38:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2861.806966 secs 2024-07-26T13:38:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:47 smithi107 ceph-mon[32659]: pgmap v2043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:38:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:47 smithi132 ceph-mon[37580]: pgmap v2043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:38:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:47 smithi112 ceph-mon[37501]: pgmap v2043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:38:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:49 smithi132 ceph-mon[37580]: pgmap v2044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:49 smithi112 ceph-mon[37501]: pgmap v2044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:49 smithi107 ceph-mon[32659]: pgmap v2044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:38:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2866.807098 secs 2024-07-26T13:38:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2866.807098 secs 2024-07-26T13:38:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:51.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2866.807098 secs 2024-07-26T13:38:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:51 smithi112 ceph-mon[37501]: pgmap v2045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:38:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:51 smithi107 ceph-mon[32659]: pgmap v2045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:38:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:51 smithi132 ceph-mon[37580]: pgmap v2045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:38:53.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:52 smithi112 ceph-mon[37501]: pgmap v2046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:53.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:52 smithi112 ceph-mon[37501]: pgmap v2047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:52 smithi107 ceph-mon[32659]: pgmap v2046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:52 smithi107 ceph-mon[32659]: pgmap v2047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:52 smithi132 ceph-mon[37580]: pgmap v2046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:52 smithi132 ceph-mon[37580]: pgmap v2047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:54.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:54.417+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:54.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:54.417+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:54.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:38:54.417+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:54 smithi112 ceph-mon[37501]: pgmap v2048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:38:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:54 smithi107 ceph-mon[32659]: pgmap v2048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:38:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:54 smithi132 ceph-mon[37580]: pgmap v2048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:38:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2871.807196 secs 2024-07-26T13:38:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2871.807196 secs 2024-07-26T13:38:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2871.807196 secs 2024-07-26T13:38:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:56.617+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:56.619 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:56.617+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:38:56.620 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:38:56.617+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:56 smithi112 ceph-mon[37501]: pgmap v2049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:38:57.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:56 smithi132 ceph-mon[37580]: pgmap v2049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:38:57.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:56 smithi107 ceph-mon[32659]: pgmap v2049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:38:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:38:58 smithi112 ceph-mon[37501]: pgmap v2050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:38:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:38:58 smithi107 ceph-mon[32659]: pgmap v2050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:38:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:38:58 smithi132 ceph-mon[37580]: pgmap v2050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:39:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:00 smithi112 ceph-mon[37501]: pgmap v2051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2876.807297 secs 2024-07-26T13:39:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:00 smithi107 ceph-mon[32659]: pgmap v2051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2876.807297 secs 2024-07-26T13:39:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:00 smithi132 ceph-mon[37580]: pgmap v2051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2876.807297 secs 2024-07-26T13:39:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:02 smithi112 ceph-mon[37501]: pgmap v2052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:39:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:02 smithi107 ceph-mon[32659]: pgmap v2052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:39:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:02 smithi132 ceph-mon[37580]: pgmap v2052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:39:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:04 smithi112 ceph-mon[37501]: pgmap v2053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:04 smithi107 ceph-mon[32659]: pgmap v2053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:04 smithi132 ceph-mon[37580]: pgmap v2053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:05 smithi112 ceph-mon[37501]: 1 slow requests, 1 included below; oldest blocked for > 2881.807415 secs 2024-07-26T13:39:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:05 smithi112 ceph-mon[37501]: slow request 2881.807414 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T13:39:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:05 smithi107 ceph-mon[32659]: 1 slow requests, 1 included below; oldest blocked for > 2881.807415 secs 2024-07-26T13:39:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:05 smithi107 ceph-mon[32659]: slow request 2881.807414 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T13:39:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:05 smithi132 ceph-mon[37580]: 1 slow requests, 1 included below; oldest blocked for > 2881.807415 secs 2024-07-26T13:39:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:05 smithi132 ceph-mon[37580]: slow request 2881.807414 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T13:39:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:06 smithi112 ceph-mon[37501]: pgmap v2054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:06 smithi107 ceph-mon[32659]: pgmap v2054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:06 smithi132 ceph-mon[37580]: pgmap v2054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:08 smithi112 ceph-mon[37501]: pgmap v2055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:08 smithi107 ceph-mon[32659]: pgmap v2055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:08 smithi132 ceph-mon[37580]: pgmap v2055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:09.433+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:09.433+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:09.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:09.433+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:10 smithi112 ceph-mon[37501]: pgmap v2056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2886.807530 secs 2024-07-26T13:39:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:10 smithi107 ceph-mon[32659]: pgmap v2056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2886.807530 secs 2024-07-26T13:39:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:10 smithi132 ceph-mon[37580]: pgmap v2056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2886.807530 secs 2024-07-26T13:39:11.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:11.626+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:11.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:11.626+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:11.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:11.626+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:13.123+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:39:13.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:13.123+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:39:13.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:13.168+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:39:13.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:12 smithi107 ceph-mon[32659]: pgmap v2057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:13.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:13.168+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:39:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:12 smithi112 ceph-mon[37501]: pgmap v2057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:12 smithi132 ceph-mon[37580]: pgmap v2057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:14.250+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:39:14.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:14.250+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:39:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:14.250+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:39:14.252 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:14.250+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:39:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:14 smithi107 ceph-mon[32659]: pgmap v2058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:14 smithi132 ceph-mon[37580]: pgmap v2058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:14 smithi112 ceph-mon[37501]: pgmap v2058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2891.807610 secs 2024-07-26T13:39:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2891.807610 secs 2024-07-26T13:39:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2891.807610 secs 2024-07-26T13:39:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:16 smithi107 ceph-mon[32659]: pgmap v2059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:16 smithi132 ceph-mon[37580]: pgmap v2059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:16 smithi112 ceph-mon[37501]: pgmap v2059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:19.252 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:18 smithi112 ceph-mon[37501]: pgmap v2060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:18 smithi107 ceph-mon[32659]: pgmap v2060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:18 smithi132 ceph-mon[37580]: pgmap v2060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:20 smithi107 ceph-mon[32659]: pgmap v2061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2896.807730 secs 2024-07-26T13:39:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:20 smithi132 ceph-mon[37580]: pgmap v2061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2896.807730 secs 2024-07-26T13:39:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:20 smithi112 ceph-mon[37501]: pgmap v2061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2896.807730 secs 2024-07-26T13:39:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:22 smithi107 ceph-mon[32659]: pgmap v2062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:22 smithi132 ceph-mon[37580]: pgmap v2062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:22 smithi112 ceph-mon[37501]: pgmap v2062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:24.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:24.449+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:24.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:24.449+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:24.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:24.449+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:25 smithi107 ceph-mon[32659]: pgmap v2063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:25 smithi132 ceph-mon[37580]: pgmap v2063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:25 smithi112 ceph-mon[37501]: pgmap v2063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2901.807847 secs 2024-07-26T13:39:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2901.807847 secs 2024-07-26T13:39:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2901.807847 secs 2024-07-26T13:39:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:26.630+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:26.630+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:26.630+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:27 smithi107 ceph-mon[32659]: pgmap v2064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:27 smithi132 ceph-mon[37580]: pgmap v2064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:27 smithi112 ceph-mon[37501]: pgmap v2064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:29 smithi107 ceph-mon[32659]: pgmap v2065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:29 smithi132 ceph-mon[37580]: pgmap v2065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:29 smithi112 ceph-mon[37501]: pgmap v2065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:31 smithi107 ceph-mon[32659]: pgmap v2066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2906.807965 secs 2024-07-26T13:39:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:31 smithi132 ceph-mon[37580]: pgmap v2066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2906.807965 secs 2024-07-26T13:39:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:31 smithi112 ceph-mon[37501]: pgmap v2066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2906.807965 secs 2024-07-26T13:39:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:33 smithi107 ceph-mon[32659]: pgmap v2067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:33 smithi132 ceph-mon[37580]: pgmap v2067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:33 smithi112 ceph-mon[37501]: pgmap v2067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:35 smithi107 ceph-mon[32659]: pgmap v2068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:35 smithi132 ceph-mon[37580]: pgmap v2068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:35 smithi112 ceph-mon[37501]: pgmap v2068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2911.808093 secs 2024-07-26T13:39:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2911.808093 secs 2024-07-26T13:39:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2911.808093 secs 2024-07-26T13:39:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:37 smithi132 ceph-mon[37580]: pgmap v2069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:37 smithi112 ceph-mon[37501]: pgmap v2069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:37 smithi107 ceph-mon[32659]: pgmap v2069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:39.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:39 smithi107 ceph-mon[32659]: pgmap v2070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:39 smithi112 ceph-mon[37501]: pgmap v2070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:39.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:39.464+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:39.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:39.464+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:39.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:39.465+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:39 smithi132 ceph-mon[37580]: pgmap v2070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:41 smithi112 ceph-mon[37501]: pgmap v2071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2916.808194 secs 2024-07-26T13:39:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:41 smithi107 ceph-mon[32659]: pgmap v2071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2916.808194 secs 2024-07-26T13:39:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:41 smithi132 ceph-mon[37580]: pgmap v2071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2916.808194 secs 2024-07-26T13:39:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:41.638+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:41.638+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:41.638+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:43 smithi112 ceph-mon[37501]: pgmap v2072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:43 smithi107 ceph-mon[32659]: pgmap v2072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:43 smithi132 ceph-mon[37580]: pgmap v2072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:45 smithi112 ceph-mon[37501]: pgmap v2073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:45 smithi107 ceph-mon[32659]: pgmap v2073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:45 smithi132 ceph-mon[37580]: pgmap v2073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2921.808323 secs 2024-07-26T13:39:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2921.808323 secs 2024-07-26T13:39:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2921.808323 secs 2024-07-26T13:39:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:47 smithi112 ceph-mon[37501]: pgmap v2074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:47 smithi107 ceph-mon[32659]: pgmap v2074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:47 smithi132 ceph-mon[37580]: pgmap v2074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:49 smithi112 ceph-mon[37501]: pgmap v2075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:49 smithi107 ceph-mon[32659]: pgmap v2075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:49 smithi132 ceph-mon[37580]: pgmap v2075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:51 smithi107 ceph-mon[32659]: pgmap v2076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2926.808430 secs 2024-07-26T13:39:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:51 smithi112 ceph-mon[37501]: pgmap v2076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2926.808430 secs 2024-07-26T13:39:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:51 smithi132 ceph-mon[37580]: pgmap v2076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2926.808430 secs 2024-07-26T13:39:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:53 smithi112 ceph-mon[37501]: pgmap v2077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:39:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:53 smithi107 ceph-mon[32659]: pgmap v2077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:39:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:53 smithi132 ceph-mon[37580]: pgmap v2077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:39:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:54 smithi112 ceph-mon[37501]: pgmap v2078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:39:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:54 smithi112 ceph-mon[37501]: pgmap v2079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:39:54.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:54.479+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:54.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:54.479+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:54.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:39:54.479+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:54.482 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:54 smithi107 ceph-mon[32659]: pgmap v2078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:39:54.482 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:54 smithi107 ceph-mon[32659]: pgmap v2079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:39:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:54 smithi132 ceph-mon[37580]: pgmap v2078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:39:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:54 smithi132 ceph-mon[37580]: pgmap v2079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:39:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:56 smithi112 ceph-mon[37501]: pgmap v2080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2931.808494 secs 2024-07-26T13:39:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:56 smithi107 ceph-mon[32659]: pgmap v2080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2931.808494 secs 2024-07-26T13:39:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:56 smithi132 ceph-mon[37580]: pgmap v2080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:39:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2931.808494 secs 2024-07-26T13:39:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:56.653+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:56.654 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:56.653+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:39:56.655 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:39:56.653+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:39:58 smithi112 ceph-mon[37501]: pgmap v2081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:39:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:39:58 smithi107 ceph-mon[32659]: pgmap v2081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:39:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:39:58 smithi132 ceph-mon[37580]: pgmap v2081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:40:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:00 smithi112 ceph-mon[37501]: pgmap v2082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:40:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:40:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:00 smithi107 ceph-mon[32659]: pgmap v2082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:40:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:40:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:00 smithi132 ceph-mon[37580]: pgmap v2082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:40:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:40:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2936.808614 secs 2024-07-26T13:40:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2936.808614 secs 2024-07-26T13:40:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2936.808614 secs 2024-07-26T13:40:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:02 smithi112 ceph-mon[37501]: pgmap v2083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:02 smithi107 ceph-mon[32659]: pgmap v2083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:02 smithi132 ceph-mon[37580]: pgmap v2083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:04 smithi112 ceph-mon[37501]: pgmap v2084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:40:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:04 smithi107 ceph-mon[32659]: pgmap v2084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:40:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:04 smithi132 ceph-mon[37580]: pgmap v2084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:40:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:06 smithi112 ceph-mon[37501]: pgmap v2085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2941.808751 secs 2024-07-26T13:40:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:06 smithi107 ceph-mon[32659]: pgmap v2085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2941.808751 secs 2024-07-26T13:40:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:06 smithi132 ceph-mon[37580]: pgmap v2085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2941.808751 secs 2024-07-26T13:40:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:08 smithi112 ceph-mon[37501]: pgmap v2086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:08 smithi107 ceph-mon[32659]: pgmap v2086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:08 smithi132 ceph-mon[37580]: pgmap v2086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:09.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:09.495+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:09.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:09.495+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:09.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:09.495+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:10 smithi132 ceph-mon[37580]: pgmap v2087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:10 smithi112 ceph-mon[37501]: pgmap v2087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:10 smithi107 ceph-mon[32659]: pgmap v2087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2946.808847 secs 2024-07-26T13:40:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2946.808847 secs 2024-07-26T13:40:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:11.669+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:11.669+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:11.669+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2946.808847 secs 2024-07-26T13:40:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:12 smithi107 ceph-mon[32659]: pgmap v2088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:12 smithi132 ceph-mon[37580]: pgmap v2088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:12 smithi112 ceph-mon[37501]: pgmap v2088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:14 smithi107 ceph-mon[32659]: pgmap v2089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:14 smithi132 ceph-mon[37580]: pgmap v2089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:14 smithi112 ceph-mon[37501]: pgmap v2089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:16 smithi107 ceph-mon[32659]: pgmap v2090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2951.808958 secs 2024-07-26T13:40:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:16 smithi132 ceph-mon[37580]: pgmap v2090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2951.808958 secs 2024-07-26T13:40:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:16 smithi112 ceph-mon[37501]: pgmap v2090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2951.808958 secs 2024-07-26T13:40:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:18 smithi107 ceph-mon[32659]: pgmap v2091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:18 smithi132 ceph-mon[37580]: pgmap v2091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:18 smithi112 ceph-mon[37501]: pgmap v2091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:20.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:20 smithi112 ceph-mon[37501]: pgmap v2092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:20 smithi107 ceph-mon[32659]: pgmap v2092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:20 smithi132 ceph-mon[37580]: pgmap v2092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:21.518 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2956.809093 secs 2024-07-26T13:40:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2956.809093 secs 2024-07-26T13:40:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2956.809093 secs 2024-07-26T13:40:22.443 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:22 smithi107 ceph-mon[32659]: pgmap v2093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:22 smithi132 ceph-mon[37580]: pgmap v2093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:22 smithi112 ceph-mon[37501]: pgmap v2093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:24.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:24.511+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:24.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:24.511+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:24.513 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:24.511+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:24.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:24 smithi107 ceph-mon[32659]: pgmap v2094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:24 smithi132 ceph-mon[37580]: pgmap v2094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:24 smithi112 ceph-mon[37501]: pgmap v2094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:26 smithi107 ceph-mon[32659]: pgmap v2095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2961.809176 secs 2024-07-26T13:40:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:26 smithi132 ceph-mon[37580]: pgmap v2095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2961.809176 secs 2024-07-26T13:40:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:26.685+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:26.685+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:26.685+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:26 smithi112 ceph-mon[37501]: pgmap v2095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2961.809176 secs 2024-07-26T13:40:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:28 smithi107 ceph-mon[32659]: pgmap v2096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:28 smithi132 ceph-mon[37580]: pgmap v2096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:28 smithi112 ceph-mon[37501]: pgmap v2096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:30 smithi107 ceph-mon[32659]: pgmap v2097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:30.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:30 smithi132 ceph-mon[37580]: pgmap v2097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:30 smithi112 ceph-mon[37501]: pgmap v2097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2966.809269 secs 2024-07-26T13:40:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2966.809269 secs 2024-07-26T13:40:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2966.809269 secs 2024-07-26T13:40:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:32 smithi107 ceph-mon[32659]: pgmap v2098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:32 smithi132 ceph-mon[37580]: pgmap v2098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:32 smithi112 ceph-mon[37501]: pgmap v2098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:34 smithi107 ceph-mon[32659]: pgmap v2099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:34 smithi132 ceph-mon[37580]: pgmap v2099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:34 smithi112 ceph-mon[37501]: pgmap v2099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:36 smithi107 ceph-mon[32659]: pgmap v2100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2971.809352 secs 2024-07-26T13:40:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:36 smithi132 ceph-mon[37580]: pgmap v2100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2971.809352 secs 2024-07-26T13:40:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:36 smithi112 ceph-mon[37501]: pgmap v2100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2971.809352 secs 2024-07-26T13:40:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:38 smithi132 ceph-mon[37580]: pgmap v2101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:38 smithi112 ceph-mon[37501]: pgmap v2101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:38 smithi107 ceph-mon[32659]: pgmap v2101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:39.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:39.527+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:39.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:39.527+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:39.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:39.527+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:40 smithi132 ceph-mon[37580]: pgmap v2102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:40 smithi112 ceph-mon[37501]: pgmap v2102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:40 smithi107 ceph-mon[32659]: pgmap v2102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2976.809437 secs 2024-07-26T13:40:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:41.701+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:41.701+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:41.701+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2976.809437 secs 2024-07-26T13:40:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2976.809437 secs 2024-07-26T13:40:42.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:42.139+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:40:42.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:42.140+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:40:42.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:42.183+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:40:42.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:42.183+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:40:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:42 smithi107 ceph-mon[32659]: pgmap v2103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:42 smithi112 ceph-mon[37501]: pgmap v2103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:42 smithi132 ceph-mon[37580]: pgmap v2103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:44.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:44.267+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:40:44.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:44.267+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:40:44.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:44.267+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:40:44.268 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:44.268+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:40:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:44 smithi112 ceph-mon[37501]: pgmap v2104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:44 smithi107 ceph-mon[32659]: pgmap v2104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:44 smithi132 ceph-mon[37580]: pgmap v2104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:46 smithi112 ceph-mon[37501]: pgmap v2105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2981.809562 secs 2024-07-26T13:40:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:46 smithi107 ceph-mon[32659]: pgmap v2105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2981.809562 secs 2024-07-26T13:40:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:46 smithi132 ceph-mon[37580]: pgmap v2105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2981.809562 secs 2024-07-26T13:40:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:48 smithi112 ceph-mon[37501]: pgmap v2106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:48 smithi107 ceph-mon[32659]: pgmap v2106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:48 smithi132 ceph-mon[37580]: pgmap v2106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:40:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:50 smithi112 ceph-mon[37501]: pgmap v2107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:50 smithi107 ceph-mon[32659]: pgmap v2107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:50 smithi132 ceph-mon[37580]: pgmap v2107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2986.809648 secs 2024-07-26T13:40:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2986.809648 secs 2024-07-26T13:40:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2986.809648 secs 2024-07-26T13:40:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:52 smithi112 ceph-mon[37501]: pgmap v2108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:52 smithi107 ceph-mon[32659]: pgmap v2108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:52 smithi132 ceph-mon[37580]: pgmap v2108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:54.543 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:54.542+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:54.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:54.542+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:54.544 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:40:54.542+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:54 smithi112 ceph-mon[37501]: pgmap v2109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:54.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:54.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:54 smithi107 ceph-mon[32659]: pgmap v2109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:54 smithi132 ceph-mon[37580]: pgmap v2109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:40:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:55 smithi112 ceph-mon[37501]: pgmap v2110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:40:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:55 smithi112 ceph-mon[37501]: pgmap v2111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:55 smithi107 ceph-mon[32659]: pgmap v2110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:40:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:55 smithi107 ceph-mon[32659]: pgmap v2111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:55 smithi132 ceph-mon[37580]: pgmap v2110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:40:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:55 smithi132 ceph-mon[37580]: pgmap v2111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2991.809786 secs 2024-07-26T13:40:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:56.708+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:56.708+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:40:56.708+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:40:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2991.809786 secs 2024-07-26T13:40:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2991.809786 secs 2024-07-26T13:40:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:57 smithi112 ceph-mon[37501]: pgmap v2112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:40:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:57 smithi107 ceph-mon[32659]: pgmap v2112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:40:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:57 smithi132 ceph-mon[37580]: pgmap v2112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:40:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:40:59 smithi112 ceph-mon[37501]: pgmap v2113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:40:59 smithi107 ceph-mon[32659]: pgmap v2113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:40:59 smithi132 ceph-mon[37580]: pgmap v2113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:41:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:01 smithi112 ceph-mon[37501]: pgmap v2114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:41:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 2996.809930 secs 2024-07-26T13:41:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:01 smithi107 ceph-mon[32659]: pgmap v2114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:41:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 2996.809930 secs 2024-07-26T13:41:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:01 smithi132 ceph-mon[37580]: pgmap v2114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:41:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 2996.809930 secs 2024-07-26T13:41:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:03 smithi112 ceph-mon[37501]: pgmap v2115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:03 smithi107 ceph-mon[32659]: pgmap v2115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:03 smithi132 ceph-mon[37580]: pgmap v2115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:05 smithi112 ceph-mon[37501]: pgmap v2116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:41:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:05 smithi107 ceph-mon[32659]: pgmap v2116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:41:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:05 smithi132 ceph-mon[37580]: pgmap v2116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:41:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3001.810053 secs 2024-07-26T13:41:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3001.810053 secs 2024-07-26T13:41:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3001.810053 secs 2024-07-26T13:41:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:07 smithi112 ceph-mon[37501]: pgmap v2117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:07 smithi107 ceph-mon[32659]: pgmap v2117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:07 smithi132 ceph-mon[37580]: pgmap v2117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:09.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:09.549+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:09.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:09.549+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:09.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:09.549+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:09 smithi112 ceph-mon[37501]: pgmap v2118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:09 smithi107 ceph-mon[32659]: pgmap v2118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:09 smithi132 ceph-mon[37580]: pgmap v2118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:11 smithi112 ceph-mon[37501]: pgmap v2119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3006.810141 secs 2024-07-26T13:41:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:11.724+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:11.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:11.724+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:11.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:11.724+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:11 smithi107 ceph-mon[32659]: pgmap v2119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3006.810141 secs 2024-07-26T13:41:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:11 smithi132 ceph-mon[37580]: pgmap v2119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3006.810141 secs 2024-07-26T13:41:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:13 smithi112 ceph-mon[37501]: pgmap v2120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:13 smithi107 ceph-mon[32659]: pgmap v2120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:13 smithi132 ceph-mon[37580]: pgmap v2120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:15 smithi112 ceph-mon[37501]: pgmap v2121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:15 smithi107 ceph-mon[32659]: pgmap v2121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:15 smithi132 ceph-mon[37580]: pgmap v2121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3011.810245 secs 2024-07-26T13:41:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3011.810245 secs 2024-07-26T13:41:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3011.810245 secs 2024-07-26T13:41:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:17 smithi107 ceph-mon[32659]: pgmap v2122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:17 smithi132 ceph-mon[37580]: pgmap v2122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:17 smithi112 ceph-mon[37501]: pgmap v2122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:19 smithi107 ceph-mon[32659]: pgmap v2123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:19 smithi132 ceph-mon[37580]: pgmap v2123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:19 smithi112 ceph-mon[37501]: pgmap v2123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:21.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:21 smithi112 ceph-mon[37501]: pgmap v2124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:21.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3016.810349 secs 2024-07-26T13:41:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:21 smithi107 ceph-mon[32659]: pgmap v2124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3016.810349 secs 2024-07-26T13:41:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:21 smithi132 ceph-mon[37580]: pgmap v2124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3016.810349 secs 2024-07-26T13:41:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:23 smithi107 ceph-mon[32659]: pgmap v2125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:23 smithi132 ceph-mon[37580]: pgmap v2125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:23 smithi112 ceph-mon[37501]: pgmap v2125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:24.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:24.566+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:24.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:24.566+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:24.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:24.566+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:25 smithi107 ceph-mon[32659]: pgmap v2126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:25 smithi132 ceph-mon[37580]: pgmap v2126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:25 smithi112 ceph-mon[37501]: pgmap v2126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:26.733+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:26.733+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:26.734 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:26.733+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3021.810475 secs 2024-07-26T13:41:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3021.810475 secs 2024-07-26T13:41:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3021.810475 secs 2024-07-26T13:41:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:27 smithi107 ceph-mon[32659]: pgmap v2127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:27 smithi132 ceph-mon[37580]: pgmap v2127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:27 smithi112 ceph-mon[37501]: pgmap v2127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:29 smithi107 ceph-mon[32659]: pgmap v2128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:29 smithi132 ceph-mon[37580]: pgmap v2128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:29 smithi112 ceph-mon[37501]: pgmap v2128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:31 smithi107 ceph-mon[32659]: pgmap v2129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3026.810601 secs 2024-07-26T13:41:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:31 smithi132 ceph-mon[37580]: pgmap v2129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3026.810601 secs 2024-07-26T13:41:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:31 smithi112 ceph-mon[37501]: pgmap v2129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3026.810601 secs 2024-07-26T13:41:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:33 smithi107 ceph-mon[32659]: pgmap v2130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:33 smithi132 ceph-mon[37580]: pgmap v2130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:33 smithi112 ceph-mon[37501]: pgmap v2130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:35 smithi107 ceph-mon[32659]: pgmap v2131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:35 smithi132 ceph-mon[37580]: pgmap v2131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:35 smithi112 ceph-mon[37501]: pgmap v2131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3031.810684 secs 2024-07-26T13:41:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3031.810684 secs 2024-07-26T13:41:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3031.810684 secs 2024-07-26T13:41:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:37 smithi107 ceph-mon[32659]: pgmap v2132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:37 smithi132 ceph-mon[37580]: pgmap v2132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:37 smithi112 ceph-mon[37501]: pgmap v2132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:39.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:39.567+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:39.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:39.567+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:39.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:39.567+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:39 smithi107 ceph-mon[32659]: pgmap v2133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:39 smithi132 ceph-mon[37580]: pgmap v2133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:39 smithi112 ceph-mon[37501]: pgmap v2133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:41.746+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:41.746+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:41.748 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:41.746+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:41 smithi132 ceph-mon[37580]: pgmap v2134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3036.810810 secs 2024-07-26T13:41:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:41 smithi112 ceph-mon[37501]: pgmap v2134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3036.810810 secs 2024-07-26T13:41:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:41 smithi107 ceph-mon[32659]: pgmap v2134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3036.810810 secs 2024-07-26T13:41:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:43 smithi132 ceph-mon[37580]: pgmap v2135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:43 smithi112 ceph-mon[37501]: pgmap v2135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:43 smithi107 ceph-mon[32659]: pgmap v2135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:45 smithi132 ceph-mon[37580]: pgmap v2136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:45 smithi112 ceph-mon[37501]: pgmap v2136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:45 smithi107 ceph-mon[32659]: pgmap v2136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3041.810935 secs 2024-07-26T13:41:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3041.810935 secs 2024-07-26T13:41:47.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3041.810935 secs 2024-07-26T13:41:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:47 smithi112 ceph-mon[37501]: pgmap v2137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:47 smithi107 ceph-mon[32659]: pgmap v2137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:47 smithi132 ceph-mon[37580]: pgmap v2137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:48 smithi112 ceph-mon[37501]: pgmap v2138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:48 smithi107 ceph-mon[32659]: pgmap v2138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:48 smithi132 ceph-mon[37580]: pgmap v2138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:51 smithi107 ceph-mon[32659]: pgmap v2139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3046.811007 secs 2024-07-26T13:41:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:51 smithi132 ceph-mon[37580]: pgmap v2139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3046.811007 secs 2024-07-26T13:41:51.653 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:51 smithi112 ceph-mon[37501]: pgmap v2139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:51.654 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3046.811007 secs 2024-07-26T13:41:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:53 smithi107 ceph-mon[32659]: pgmap v2140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:53 smithi132 ceph-mon[37580]: pgmap v2140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:53 smithi112 ceph-mon[37501]: pgmap v2140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:41:54.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:54.581+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:54.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:54.581+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:54.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:41:54.581+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:55 smithi107 ceph-mon[32659]: pgmap v2141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:55 smithi132 ceph-mon[37580]: pgmap v2141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:55 smithi112 ceph-mon[37501]: pgmap v2141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:41:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3051.811139 secs 2024-07-26T13:41:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:56 smithi107 ceph-mon[32659]: pgmap v2142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:41:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:56 smithi107 ceph-mon[32659]: pgmap v2143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3051.811139 secs 2024-07-26T13:41:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:56 smithi132 ceph-mon[37580]: pgmap v2142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:41:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:56 smithi132 ceph-mon[37580]: pgmap v2143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3051.811139 secs 2024-07-26T13:41:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:56 smithi112 ceph-mon[37501]: pgmap v2142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:41:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:56 smithi112 ceph-mon[37501]: pgmap v2143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:41:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:56.761+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:56.761+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:41:56.761+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:41:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:41:58 smithi112 ceph-mon[37501]: pgmap v2144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:41:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:41:58 smithi107 ceph-mon[32659]: pgmap v2144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:41:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:41:58 smithi132 ceph-mon[37580]: pgmap v2144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:42:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:00 smithi112 ceph-mon[37501]: pgmap v2145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:42:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:00 smithi107 ceph-mon[32659]: pgmap v2145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:42:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:00 smithi132 ceph-mon[37580]: pgmap v2145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:42:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3056.811205 secs 2024-07-26T13:42:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3056.811205 secs 2024-07-26T13:42:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3056.811205 secs 2024-07-26T13:42:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:02 smithi112 ceph-mon[37501]: pgmap v2146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:42:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:02 smithi107 ceph-mon[32659]: pgmap v2146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:42:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:02 smithi132 ceph-mon[37580]: pgmap v2146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:42:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:04 smithi112 ceph-mon[37501]: pgmap v2147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:42:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:04 smithi107 ceph-mon[32659]: pgmap v2147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:42:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:04 smithi132 ceph-mon[37580]: pgmap v2147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:42:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3061.811330 secs 2024-07-26T13:42:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:06 smithi112 ceph-mon[37501]: pgmap v2148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:42:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3061.811330 secs 2024-07-26T13:42:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:06 smithi107 ceph-mon[32659]: pgmap v2148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3061.811330 secs 2024-07-26T13:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:06 smithi132 ceph-mon[37580]: pgmap v2148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:42:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:08 smithi112 ceph-mon[37501]: pgmap v2149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:08 smithi107 ceph-mon[32659]: pgmap v2149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:08 smithi132 ceph-mon[37580]: pgmap v2149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:09.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:09.582+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:09.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:09.582+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:09.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:09.582+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:10 smithi112 ceph-mon[37501]: pgmap v2150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:10 smithi107 ceph-mon[32659]: pgmap v2150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:10 smithi132 ceph-mon[37580]: pgmap v2150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3066.811471 secs 2024-07-26T13:42:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:11.777+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:11.777+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:11.777+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3066.811471 secs 2024-07-26T13:42:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3066.811471 secs 2024-07-26T13:42:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:12 smithi107 ceph-mon[32659]: pgmap v2151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:12 smithi132 ceph-mon[37580]: pgmap v2151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:12 smithi112 ceph-mon[37501]: pgmap v2151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:13.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:13.154+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:42:13.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:13.154+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:42:13.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:13.201+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:42:13.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:13.202+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:42:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:14.283+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:42:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:14.283+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:42:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:14.283+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:42:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:14.283+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:42:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:14 smithi107 ceph-mon[32659]: pgmap v2152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:14 smithi132 ceph-mon[37580]: pgmap v2152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:14 smithi112 ceph-mon[37501]: pgmap v2152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3071.811559 secs 2024-07-26T13:42:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:16 smithi107 ceph-mon[32659]: pgmap v2153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3071.811559 secs 2024-07-26T13:42:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:16 smithi132 ceph-mon[37580]: pgmap v2153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3071.811559 secs 2024-07-26T13:42:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:16 smithi112 ceph-mon[37501]: pgmap v2153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:18 smithi107 ceph-mon[32659]: pgmap v2154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:18 smithi132 ceph-mon[37580]: pgmap v2154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:18 smithi112 ceph-mon[37501]: pgmap v2154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:20 smithi107 ceph-mon[32659]: pgmap v2155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:20 smithi132 ceph-mon[37580]: pgmap v2155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:20 smithi112 ceph-mon[37501]: pgmap v2155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3076.811618 secs 2024-07-26T13:42:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3076.811618 secs 2024-07-26T13:42:21.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3076.811618 secs 2024-07-26T13:42:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:22 smithi107 ceph-mon[32659]: pgmap v2156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:22.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:22 smithi132 ceph-mon[37580]: pgmap v2156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:22 smithi112 ceph-mon[37501]: pgmap v2156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:24.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:24.598+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:24.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:24.599+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:24.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:24.599+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:24 smithi107 ceph-mon[32659]: pgmap v2157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:24 smithi132 ceph-mon[37580]: pgmap v2157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:24 smithi112 ceph-mon[37501]: pgmap v2157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3081.811701 secs 2024-07-26T13:42:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:26 smithi107 ceph-mon[32659]: pgmap v2158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:26.793+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:26.793+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:26.794 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:26.793+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3081.811701 secs 2024-07-26T13:42:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:26 smithi132 ceph-mon[37580]: pgmap v2158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3081.811701 secs 2024-07-26T13:42:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:26 smithi112 ceph-mon[37501]: pgmap v2158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:28 smithi107 ceph-mon[32659]: pgmap v2159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:28 smithi132 ceph-mon[37580]: pgmap v2159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:28 smithi112 ceph-mon[37501]: pgmap v2159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:30 smithi107 ceph-mon[32659]: pgmap v2160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:30 smithi132 ceph-mon[37580]: pgmap v2160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:30 smithi112 ceph-mon[37501]: pgmap v2160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3086.811820 secs 2024-07-26T13:42:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3086.811820 secs 2024-07-26T13:42:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3086.811820 secs 2024-07-26T13:42:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:32 smithi107 ceph-mon[32659]: pgmap v2161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:32 smithi132 ceph-mon[37580]: pgmap v2161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:32 smithi112 ceph-mon[37501]: pgmap v2161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:34.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:34 smithi107 ceph-mon[32659]: pgmap v2162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:34 smithi132 ceph-mon[37580]: pgmap v2162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:34 smithi112 ceph-mon[37501]: pgmap v2162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3091.811975 secs 2024-07-26T13:42:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:36 smithi107 ceph-mon[32659]: pgmap v2163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3091.811975 secs 2024-07-26T13:42:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:36 smithi132 ceph-mon[37580]: pgmap v2163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3091.811975 secs 2024-07-26T13:42:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:36 smithi112 ceph-mon[37501]: pgmap v2163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:38 smithi132 ceph-mon[37580]: pgmap v2164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:38 smithi112 ceph-mon[37501]: pgmap v2164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:38 smithi107 ceph-mon[32659]: pgmap v2164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:39.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:39.614+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:39.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:39.614+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:39.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:39.614+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:40 smithi132 ceph-mon[37580]: pgmap v2165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:40 smithi112 ceph-mon[37501]: pgmap v2165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:40 smithi107 ceph-mon[32659]: pgmap v2165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:41.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:41.794+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:41.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:41.794+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:41.795 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:41.794+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3096.812101 secs 2024-07-26T13:42:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3096.812101 secs 2024-07-26T13:42:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3096.812101 secs 2024-07-26T13:42:42.906 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:42 smithi107 ceph-mon[32659]: pgmap v2166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:42 smithi112 ceph-mon[37501]: pgmap v2166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:42 smithi132 ceph-mon[37580]: pgmap v2166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:44 smithi112 ceph-mon[37501]: pgmap v2167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:44 smithi107 ceph-mon[32659]: pgmap v2167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:44 smithi132 ceph-mon[37580]: pgmap v2167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3101.812197 secs 2024-07-26T13:42:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:46 smithi112 ceph-mon[37501]: pgmap v2168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3101.812197 secs 2024-07-26T13:42:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:46 smithi107 ceph-mon[32659]: pgmap v2168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3101.812197 secs 2024-07-26T13:42:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:46 smithi132 ceph-mon[37580]: pgmap v2168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:48 smithi112 ceph-mon[37501]: pgmap v2169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:48 smithi107 ceph-mon[32659]: pgmap v2169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:48 smithi132 ceph-mon[37580]: pgmap v2169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:49 smithi112 ceph-mon[37501]: pgmap v2170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:49 smithi107 ceph-mon[32659]: pgmap v2170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:49 smithi132 ceph-mon[37580]: pgmap v2170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:42:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3106.812285 secs 2024-07-26T13:42:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3106.812285 secs 2024-07-26T13:42:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3106.812285 secs 2024-07-26T13:42:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:51 smithi112 ceph-mon[37501]: pgmap v2171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:51 smithi107 ceph-mon[32659]: pgmap v2171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:51 smithi132 ceph-mon[37580]: pgmap v2171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:42:54.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:54.617+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:54.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:54.617+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:54.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:42:54.617+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:54 smithi112 ceph-mon[37501]: pgmap v2172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:54 smithi107 ceph-mon[32659]: pgmap v2172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:54 smithi132 ceph-mon[37580]: pgmap v2172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3111.812372 secs 2024-07-26T13:42:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:56 smithi112 ceph-mon[37501]: pgmap v2173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3111.812372 secs 2024-07-26T13:42:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:56 smithi107 ceph-mon[32659]: pgmap v2173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3111.812372 secs 2024-07-26T13:42:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:56 smithi132 ceph-mon[37580]: pgmap v2173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:56.808+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:56.809+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:42:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:42:56.809+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:42:58 smithi112 ceph-mon[37501]: pgmap v2174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:42:58 smithi107 ceph-mon[32659]: pgmap v2174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:42:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:42:58 smithi132 ceph-mon[37580]: pgmap v2174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:00 smithi107 ceph-mon[32659]: pgmap v2175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:43:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:00 smithi132 ceph-mon[37580]: pgmap v2175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:43:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:00.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:00 smithi112 ceph-mon[37501]: pgmap v2175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:43:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3116.812490 secs 2024-07-26T13:43:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3116.812490 secs 2024-07-26T13:43:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3116.812490 secs 2024-07-26T13:43:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: pgmap v2176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:43:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: pgmap v2176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: pgmap v2176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:02.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:43:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:03 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:43:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:03 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:43:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:03 smithi132 ceph-mon[37580]: pgmap v2177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:43:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:03 smithi132 ceph-mon[37580]: pgmap v2178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T13:43:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:03 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:43:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:03 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:43:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:03 smithi112 ceph-mon[37501]: pgmap v2177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:43:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:03 smithi112 ceph-mon[37501]: pgmap v2178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T13:43:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:03 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:43:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:03 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:43:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:03 smithi107 ceph-mon[32659]: pgmap v2177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:43:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:03 smithi107 ceph-mon[32659]: pgmap v2178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T13:43:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:05 smithi112 ceph-mon[37501]: pgmap v2179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:43:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:05 smithi107 ceph-mon[32659]: pgmap v2179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:43:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:05 smithi132 ceph-mon[37580]: pgmap v2179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:43:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3121.812573 secs 2024-07-26T13:43:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3121.812573 secs 2024-07-26T13:43:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3121.812573 secs 2024-07-26T13:43:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:07 smithi112 ceph-mon[37501]: pgmap v2180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T13:43:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:07 smithi107 ceph-mon[32659]: pgmap v2180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T13:43:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:07 smithi132 ceph-mon[37580]: pgmap v2180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T13:43:09.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:09 smithi107 ceph-mon[32659]: pgmap v2181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:43:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:09.633+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:09.633+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:09.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:09.633+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:09 smithi112 ceph-mon[37501]: pgmap v2181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:43:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:09 smithi132 ceph-mon[37580]: pgmap v2181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:43:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:11 smithi112 ceph-mon[37501]: pgmap v2182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:43:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3126.812710 secs 2024-07-26T13:43:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:11 smithi107 ceph-mon[32659]: pgmap v2182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:43:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3126.812710 secs 2024-07-26T13:43:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:11 smithi132 ceph-mon[37580]: pgmap v2182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:43:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3126.812710 secs 2024-07-26T13:43:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:11.824+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:11.824+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:11.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:11.824+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:13 smithi112 ceph-mon[37501]: pgmap v2183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:43:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:13 smithi107 ceph-mon[32659]: pgmap v2183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:43:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:13 smithi132 ceph-mon[37580]: pgmap v2183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:43:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:15 smithi112 ceph-mon[37501]: pgmap v2184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:43:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:15 smithi107 ceph-mon[32659]: pgmap v2184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:43:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:15 smithi132 ceph-mon[37580]: pgmap v2184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:43:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3131.812774 secs 2024-07-26T13:43:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3131.812774 secs 2024-07-26T13:43:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3131.812774 secs 2024-07-26T13:43:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:17 smithi112 ceph-mon[37501]: pgmap v2185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:43:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:17 smithi107 ceph-mon[32659]: pgmap v2185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:43:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:17 smithi132 ceph-mon[37580]: pgmap v2185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:43:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:19 smithi112 ceph-mon[37501]: pgmap v2186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:43:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:19 smithi107 ceph-mon[32659]: pgmap v2186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:43:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:19 smithi132 ceph-mon[37580]: pgmap v2186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:43:21.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:21 smithi107 ceph-mon[32659]: pgmap v2187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:21.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3136.812881 secs 2024-07-26T13:43:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:21 smithi112 ceph-mon[37501]: pgmap v2187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3136.812881 secs 2024-07-26T13:43:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:21 smithi132 ceph-mon[37580]: pgmap v2187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3136.812881 secs 2024-07-26T13:43:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:23 smithi112 ceph-mon[37501]: pgmap v2188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:23 smithi107 ceph-mon[32659]: pgmap v2188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:23 smithi132 ceph-mon[37580]: pgmap v2188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:24.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:24.649+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:24.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:24.649+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:24.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:24.649+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:25 smithi112 ceph-mon[37501]: pgmap v2189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:25 smithi107 ceph-mon[32659]: pgmap v2189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:25 smithi132 ceph-mon[37580]: pgmap v2189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3141.813022 secs 2024-07-26T13:43:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3141.813022 secs 2024-07-26T13:43:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3141.813022 secs 2024-07-26T13:43:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:26.841+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:26.841+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:26.841+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:27 smithi112 ceph-mon[37501]: pgmap v2190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:27 smithi107 ceph-mon[32659]: pgmap v2190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:27 smithi132 ceph-mon[37580]: pgmap v2190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:29 smithi112 ceph-mon[37501]: pgmap v2191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:29.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:29 smithi107 ceph-mon[32659]: pgmap v2191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:29 smithi132 ceph-mon[37580]: pgmap v2191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:31 smithi112 ceph-mon[37501]: pgmap v2192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3146.813185 secs 2024-07-26T13:43:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:31 smithi107 ceph-mon[32659]: pgmap v2192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3146.813185 secs 2024-07-26T13:43:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:31 smithi132 ceph-mon[37580]: pgmap v2192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3146.813185 secs 2024-07-26T13:43:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:33 smithi112 ceph-mon[37501]: pgmap v2193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:33 smithi107 ceph-mon[32659]: pgmap v2193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:33 smithi132 ceph-mon[37580]: pgmap v2193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:35 smithi112 ceph-mon[37501]: pgmap v2194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:35 smithi107 ceph-mon[32659]: pgmap v2194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:35 smithi132 ceph-mon[37580]: pgmap v2194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3151.813283 secs 2024-07-26T13:43:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3151.813283 secs 2024-07-26T13:43:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3151.813283 secs 2024-07-26T13:43:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:37 smithi112 ceph-mon[37501]: pgmap v2195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:37 smithi107 ceph-mon[32659]: pgmap v2195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:37 smithi132 ceph-mon[37580]: pgmap v2195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:39.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:39.665+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:39.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:39.665+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:39.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:39.665+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:39 smithi107 ceph-mon[32659]: pgmap v2196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:39.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:39 smithi132 ceph-mon[37580]: pgmap v2196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:39 smithi112 ceph-mon[37501]: pgmap v2196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:41 smithi107 ceph-mon[32659]: pgmap v2197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3156.813342 secs 2024-07-26T13:43:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:41 smithi132 ceph-mon[37580]: pgmap v2197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3156.813342 secs 2024-07-26T13:43:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:41.851+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:41.851+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:41.851+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:41 smithi112 ceph-mon[37501]: pgmap v2197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3156.813342 secs 2024-07-26T13:43:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:43 smithi107 ceph-mon[32659]: pgmap v2198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:43 smithi132 ceph-mon[37580]: pgmap v2198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:43 smithi112 ceph-mon[37501]: pgmap v2198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:45 smithi107 ceph-mon[32659]: pgmap v2199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:45 smithi132 ceph-mon[37580]: pgmap v2199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:45 smithi112 ceph-mon[37501]: pgmap v2199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3161.813408 secs 2024-07-26T13:43:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3161.813408 secs 2024-07-26T13:43:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3161.813408 secs 2024-07-26T13:43:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:47 smithi107 ceph-mon[32659]: pgmap v2200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:47 smithi132 ceph-mon[37580]: pgmap v2200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:47 smithi112 ceph-mon[37501]: pgmap v2200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:49 smithi107 ceph-mon[32659]: pgmap v2201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:49 smithi132 ceph-mon[37580]: pgmap v2201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:49.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:49 smithi112 ceph-mon[37501]: pgmap v2201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:51.752 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:51 smithi112 ceph-mon[37501]: pgmap v2202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:51.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3166.813512 secs 2024-07-26T13:43:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:51 smithi107 ceph-mon[32659]: pgmap v2202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3166.813512 secs 2024-07-26T13:43:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:51 smithi132 ceph-mon[37580]: pgmap v2202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3166.813512 secs 2024-07-26T13:43:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:53 smithi107 ceph-mon[32659]: pgmap v2203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:53 smithi132 ceph-mon[37580]: pgmap v2203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:53 smithi112 ceph-mon[37501]: pgmap v2203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:54.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:54.680+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:54.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:54.680+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:54.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:43:54.680+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:55 smithi107 ceph-mon[32659]: pgmap v2204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:55 smithi132 ceph-mon[37580]: pgmap v2204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:55 smithi112 ceph-mon[37501]: pgmap v2204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3171.813584 secs 2024-07-26T13:43:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3171.813584 secs 2024-07-26T13:43:56.867 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:56.866+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:56.868 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:56.866+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:56.868 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:43:56.866+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:43:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3171.813584 secs 2024-07-26T13:43:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:57 smithi107 ceph-mon[32659]: pgmap v2205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:57 smithi132 ceph-mon[37580]: pgmap v2205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:57 smithi112 ceph-mon[37501]: pgmap v2205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:43:59 smithi107 ceph-mon[32659]: pgmap v2206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:43:59 smithi132 ceph-mon[37580]: pgmap v2206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:43:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:43:59 smithi112 ceph-mon[37501]: pgmap v2206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:01 smithi107 ceph-mon[32659]: pgmap v2207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3176.813728 secs 2024-07-26T13:44:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:01 smithi132 ceph-mon[37580]: pgmap v2207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3176.813728 secs 2024-07-26T13:44:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:01 smithi112 ceph-mon[37501]: pgmap v2207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3176.813728 secs 2024-07-26T13:44:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:44:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:44:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:44:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:03 smithi132 ceph-mon[37580]: pgmap v2208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:03 smithi112 ceph-mon[37501]: pgmap v2208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:03 smithi107 ceph-mon[32659]: pgmap v2208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:04 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:44:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:04 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:44:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:04 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:44:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:04 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:44:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:04 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:44:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:04 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:44:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:05 smithi132 ceph-mon[37580]: pgmap v2209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:05 smithi112 ceph-mon[37501]: pgmap v2209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:05 smithi107 ceph-mon[32659]: pgmap v2209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3181.813787 secs 2024-07-26T13:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:06.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:06.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3181.813787 secs 2024-07-26T13:44:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3181.813787 secs 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:44:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:07 smithi112 ceph-mon[37501]: pgmap v2210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:07 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:44:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:07 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:44:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:07 smithi112 ceph-mon[37501]: pgmap v2211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:44:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:07 smithi112 ceph-mon[37501]: pgmap v2212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:44:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:07 smithi107 ceph-mon[32659]: pgmap v2210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:07 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:44:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:07 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:44:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:07 smithi107 ceph-mon[32659]: pgmap v2211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:44:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:07 smithi107 ceph-mon[32659]: pgmap v2212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:44:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:07 smithi132 ceph-mon[37580]: pgmap v2210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:07 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:44:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:07 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:44:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:07 smithi132 ceph-mon[37580]: pgmap v2211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:44:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:07 smithi132 ceph-mon[37580]: pgmap v2212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:44:09.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:09.696+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:09.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:09.696+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:09.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:09.696+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:09.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:09 smithi107 ceph-mon[32659]: pgmap v2213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:44:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:09 smithi112 ceph-mon[37501]: pgmap v2213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:44:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:09 smithi132 ceph-mon[37580]: pgmap v2213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:44:11.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:11.882+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:11.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:11.882+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:11.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:11.882+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:11 smithi112 ceph-mon[37501]: pgmap v2214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3186.813917 secs 2024-07-26T13:44:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:11 smithi107 ceph-mon[32659]: pgmap v2214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3186.813917 secs 2024-07-26T13:44:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:11 smithi132 ceph-mon[37580]: pgmap v2214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3186.813917 secs 2024-07-26T13:44:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:13 smithi112 ceph-mon[37501]: pgmap v2215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:44:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:13 smithi107 ceph-mon[32659]: pgmap v2215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:44:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:13 smithi132 ceph-mon[37580]: pgmap v2215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:44:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:15 smithi112 ceph-mon[37501]: pgmap v2216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T13:44:15.955 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:15 smithi132 ceph-mon[37580]: pgmap v2216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T13:44:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:15 smithi107 ceph-mon[32659]: pgmap v2216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T13:44:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3191.814039 secs 2024-07-26T13:44:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:16 smithi132 ceph-mon[37580]: pgmap v2217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:44:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3191.814039 secs 2024-07-26T13:44:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:16 smithi112 ceph-mon[37501]: pgmap v2217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:44:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3191.814039 secs 2024-07-26T13:44:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:16 smithi107 ceph-mon[32659]: pgmap v2217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:44:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:19 smithi112 ceph-mon[37501]: pgmap v2218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:44:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:19 smithi107 ceph-mon[32659]: pgmap v2218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:44:19.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:19 smithi132 ceph-mon[37580]: pgmap v2218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:44:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:21 smithi112 ceph-mon[37501]: pgmap v2219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:44:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3196.814086 secs 2024-07-26T13:44:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:21 smithi107 ceph-mon[32659]: pgmap v2219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:44:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3196.814086 secs 2024-07-26T13:44:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:21 smithi132 ceph-mon[37580]: pgmap v2219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:44:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3196.814086 secs 2024-07-26T13:44:23.443 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:23 smithi112 ceph-mon[37501]: pgmap v2220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:44:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:23 smithi107 ceph-mon[32659]: pgmap v2220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:44:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:23 smithi132 ceph-mon[37580]: pgmap v2220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:44:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:24.712+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:24.712+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:24.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:24.712+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:25 smithi112 ceph-mon[37501]: pgmap v2221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:44:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:25 smithi107 ceph-mon[32659]: pgmap v2221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:44:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:25 smithi132 ceph-mon[37580]: pgmap v2221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:44:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3201.814190 secs 2024-07-26T13:44:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3201.814190 secs 2024-07-26T13:44:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3201.814190 secs 2024-07-26T13:44:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:26.898+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:26.898+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:26.898+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:27 smithi112 ceph-mon[37501]: pgmap v2222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:27 smithi107 ceph-mon[32659]: pgmap v2222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:27 smithi132 ceph-mon[37580]: pgmap v2222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:29 smithi112 ceph-mon[37501]: pgmap v2223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:29 smithi107 ceph-mon[32659]: pgmap v2223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:29 smithi132 ceph-mon[37580]: pgmap v2223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:31 smithi112 ceph-mon[37501]: pgmap v2224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3206.814314 secs 2024-07-26T13:44:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:31 smithi107 ceph-mon[32659]: pgmap v2224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3206.814314 secs 2024-07-26T13:44:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:31 smithi132 ceph-mon[37580]: pgmap v2224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3206.814314 secs 2024-07-26T13:44:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:33 smithi112 ceph-mon[37501]: pgmap v2225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:33 smithi107 ceph-mon[32659]: pgmap v2225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:33 smithi132 ceph-mon[37580]: pgmap v2225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:35 smithi112 ceph-mon[37501]: pgmap v2226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:35 smithi107 ceph-mon[32659]: pgmap v2226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:35 smithi132 ceph-mon[37580]: pgmap v2226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3211.814433 secs 2024-07-26T13:44:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3211.814433 secs 2024-07-26T13:44:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3211.814433 secs 2024-07-26T13:44:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:37 smithi112 ceph-mon[37501]: pgmap v2227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:37 smithi107 ceph-mon[32659]: pgmap v2227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:37 smithi132 ceph-mon[37580]: pgmap v2227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:39 smithi112 ceph-mon[37501]: pgmap v2228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:39 smithi107 ceph-mon[32659]: pgmap v2228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:39 smithi132 ceph-mon[37580]: pgmap v2228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:39.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:39.727+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:39.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:39.728+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:39.729 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:39.728+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:41 smithi112 ceph-mon[37501]: pgmap v2229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:41.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3216.814556 secs 2024-07-26T13:44:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:41 smithi107 ceph-mon[32659]: pgmap v2229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3216.814556 secs 2024-07-26T13:44:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:41 smithi132 ceph-mon[37580]: pgmap v2229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3216.814556 secs 2024-07-26T13:44:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:41.906+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:41.906+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:41.906+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:43 smithi112 ceph-mon[37501]: pgmap v2230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:43 smithi107 ceph-mon[32659]: pgmap v2230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:43 smithi132 ceph-mon[37580]: pgmap v2230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:45 smithi107 ceph-mon[32659]: pgmap v2231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:45 smithi132 ceph-mon[37580]: pgmap v2231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:45 smithi112 ceph-mon[37501]: pgmap v2231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3221.814619 secs 2024-07-26T13:44:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3221.814619 secs 2024-07-26T13:44:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3221.814619 secs 2024-07-26T13:44:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:47 smithi107 ceph-mon[32659]: pgmap v2232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:47 smithi132 ceph-mon[37580]: pgmap v2232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:47 smithi112 ceph-mon[37501]: pgmap v2232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:49 smithi107 ceph-mon[32659]: pgmap v2233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:49 smithi132 ceph-mon[37580]: pgmap v2233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:49 smithi112 ceph-mon[37501]: pgmap v2233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:44:51.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:51 smithi112 ceph-mon[37501]: pgmap v2234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:51.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3226.814735 secs 2024-07-26T13:44:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:51 smithi107 ceph-mon[32659]: pgmap v2234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3226.814735 secs 2024-07-26T13:44:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:51 smithi132 ceph-mon[37580]: pgmap v2234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3226.814735 secs 2024-07-26T13:44:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:53 smithi107 ceph-mon[32659]: pgmap v2235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:53 smithi132 ceph-mon[37580]: pgmap v2235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:53 smithi112 ceph-mon[37501]: pgmap v2235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:44:54.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:54.733+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:54.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:54.733+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:54.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:44:54.733+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:55 smithi107 ceph-mon[32659]: pgmap v2236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:44:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:55 smithi132 ceph-mon[37580]: pgmap v2236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:44:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:55 smithi112 ceph-mon[37501]: pgmap v2236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:44:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3231.814796 secs 2024-07-26T13:44:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3231.814796 secs 2024-07-26T13:44:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3231.814796 secs 2024-07-26T13:44:56.922 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:56.921+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:56.921+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:44:56.923 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:44:56.921+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:57 smithi107 ceph-mon[32659]: pgmap v2237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:44:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:57 smithi132 ceph-mon[37580]: pgmap v2237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:44:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:57 smithi112 ceph-mon[37501]: pgmap v2237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:44:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:44:59 smithi107 ceph-mon[32659]: pgmap v2238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:44:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:44:59 smithi132 ceph-mon[37580]: pgmap v2238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:44:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:44:59 smithi112 ceph-mon[37501]: pgmap v2238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:45:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:01 smithi107 ceph-mon[32659]: pgmap v2239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3236.814916 secs 2024-07-26T13:45:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:01 smithi132 ceph-mon[37580]: pgmap v2239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3236.814916 secs 2024-07-26T13:45:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:01 smithi112 ceph-mon[37501]: pgmap v2239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3236.814916 secs 2024-07-26T13:45:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:03 smithi107 ceph-mon[32659]: pgmap v2240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:03 smithi132 ceph-mon[37580]: pgmap v2240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:03 smithi112 ceph-mon[37501]: pgmap v2240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:05 smithi107 ceph-mon[32659]: pgmap v2241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:05 smithi132 ceph-mon[37580]: pgmap v2241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:05 smithi112 ceph-mon[37501]: pgmap v2241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:06.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3241.815044 secs 2024-07-26T13:45:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3241.815044 secs 2024-07-26T13:45:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3241.815044 secs 2024-07-26T13:45:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:07 smithi107 ceph-mon[32659]: pgmap v2242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:07 smithi132 ceph-mon[37580]: pgmap v2242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:07 smithi112 ceph-mon[37501]: pgmap v2242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:08 smithi132 ceph-mon[37580]: pgmap v2243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:45:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:08 smithi132 ceph-mon[37580]: pgmap v2244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:08 smithi112 ceph-mon[37501]: pgmap v2243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:45:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:08 smithi112 ceph-mon[37501]: pgmap v2244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:08 smithi107 ceph-mon[32659]: pgmap v2243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:45:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:08 smithi107 ceph-mon[32659]: pgmap v2244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:45:09.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:09.749+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:09.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:09.749+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:09.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:09.749+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:10 smithi112 ceph-mon[37501]: pgmap v2245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:10 smithi107 ceph-mon[32659]: pgmap v2245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:10 smithi132 ceph-mon[37580]: pgmap v2245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3246.815164 secs 2024-07-26T13:45:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3246.815164 secs 2024-07-26T13:45:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3246.815164 secs 2024-07-26T13:45:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:11.937+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:11.937+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:11.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:11.937+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:12 smithi112 ceph-mon[37501]: pgmap v2246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:12 smithi107 ceph-mon[32659]: pgmap v2246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:12 smithi132 ceph-mon[37580]: pgmap v2246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:13.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:13.183+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:45:13.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:13.184+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:45:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:13.230+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:45:13.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:13.230+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:45:14.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:14.315+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:45:14.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:14.315+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:45:14.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:14.316+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:45:14.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:14.316+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:45:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:14 smithi112 ceph-mon[37501]: pgmap v2247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:14 smithi107 ceph-mon[32659]: pgmap v2247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:14 smithi132 ceph-mon[37580]: pgmap v2247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3251.815307 secs 2024-07-26T13:45:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:16 smithi112 ceph-mon[37501]: pgmap v2248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3251.815307 secs 2024-07-26T13:45:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:16 smithi107 ceph-mon[32659]: pgmap v2248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3251.815307 secs 2024-07-26T13:45:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:16 smithi132 ceph-mon[37580]: pgmap v2248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:45:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:18 smithi112 ceph-mon[37501]: pgmap v2249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:45:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:18 smithi107 ceph-mon[32659]: pgmap v2249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:45:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:18 smithi132 ceph-mon[37580]: pgmap v2249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:45:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:20 smithi112 ceph-mon[37501]: pgmap v2250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:20 smithi107 ceph-mon[32659]: pgmap v2250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:20 smithi132 ceph-mon[37580]: pgmap v2250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3256.815390 secs 2024-07-26T13:45:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3256.815390 secs 2024-07-26T13:45:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3256.815390 secs 2024-07-26T13:45:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:22 smithi112 ceph-mon[37501]: pgmap v2251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:22 smithi107 ceph-mon[32659]: pgmap v2251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:22.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:22 smithi132 ceph-mon[37580]: pgmap v2251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:24 smithi112 ceph-mon[37501]: pgmap v2252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:24.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:24.765+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:24.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:24.765+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:24.767 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:24.765+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:24.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:24 smithi107 ceph-mon[32659]: pgmap v2252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:24 smithi132 ceph-mon[37580]: pgmap v2252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3261.815485 secs 2024-07-26T13:45:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:26 smithi112 ceph-mon[37501]: pgmap v2253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3261.815485 secs 2024-07-26T13:45:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:26 smithi107 ceph-mon[32659]: pgmap v2253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3261.815485 secs 2024-07-26T13:45:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:26 smithi132 ceph-mon[37580]: pgmap v2253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:26.953+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:26.953+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:26.953+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:28 smithi112 ceph-mon[37501]: pgmap v2254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:28 smithi107 ceph-mon[32659]: pgmap v2254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:28 smithi132 ceph-mon[37580]: pgmap v2254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:30 smithi112 ceph-mon[37501]: pgmap v2255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:30 smithi107 ceph-mon[32659]: pgmap v2255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:30 smithi132 ceph-mon[37580]: pgmap v2255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3266.815615 secs 2024-07-26T13:45:31.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3266.815615 secs 2024-07-26T13:45:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3266.815615 secs 2024-07-26T13:45:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:32 smithi112 ceph-mon[37501]: pgmap v2256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:32 smithi107 ceph-mon[32659]: pgmap v2256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:32 smithi132 ceph-mon[37580]: pgmap v2256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:34 smithi112 ceph-mon[37501]: pgmap v2257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:34 smithi107 ceph-mon[32659]: pgmap v2257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:34 smithi132 ceph-mon[37580]: pgmap v2257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3271.815738 secs 2024-07-26T13:45:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:36 smithi112 ceph-mon[37501]: pgmap v2258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3271.815738 secs 2024-07-26T13:45:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:36 smithi107 ceph-mon[32659]: pgmap v2258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3271.815738 secs 2024-07-26T13:45:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:36 smithi132 ceph-mon[37580]: pgmap v2258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:38 smithi112 ceph-mon[37501]: pgmap v2259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:38 smithi107 ceph-mon[32659]: pgmap v2259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:38 smithi132 ceph-mon[37580]: pgmap v2259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:39.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:39.781+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:39.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:39.781+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:39.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:39.781+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:40 smithi107 ceph-mon[32659]: pgmap v2260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:40 smithi132 ceph-mon[37580]: pgmap v2260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:40 smithi112 ceph-mon[37501]: pgmap v2260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3276.815877 secs 2024-07-26T13:45:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3276.815877 secs 2024-07-26T13:45:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3276.815877 secs 2024-07-26T13:45:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:41.968+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:41.969+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:41.969+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:42 smithi107 ceph-mon[32659]: pgmap v2261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:42 smithi132 ceph-mon[37580]: pgmap v2261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:42 smithi112 ceph-mon[37501]: pgmap v2261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:44 smithi107 ceph-mon[32659]: pgmap v2262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:44 smithi132 ceph-mon[37580]: pgmap v2262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:44 smithi112 ceph-mon[37501]: pgmap v2262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3281.815999 secs 2024-07-26T13:45:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:46 smithi107 ceph-mon[32659]: pgmap v2263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3281.815999 secs 2024-07-26T13:45:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:46 smithi132 ceph-mon[37580]: pgmap v2263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3281.815999 secs 2024-07-26T13:45:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:46 smithi112 ceph-mon[37501]: pgmap v2263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:48 smithi107 ceph-mon[32659]: pgmap v2264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:48 smithi132 ceph-mon[37580]: pgmap v2264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:48 smithi112 ceph-mon[37501]: pgmap v2264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:50 smithi107 ceph-mon[32659]: pgmap v2265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:50 smithi132 ceph-mon[37580]: pgmap v2265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:50 smithi112 ceph-mon[37501]: pgmap v2265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:45:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3286.816124 secs 2024-07-26T13:45:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3286.816124 secs 2024-07-26T13:45:51.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3286.816124 secs 2024-07-26T13:45:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:52 smithi107 ceph-mon[32659]: pgmap v2266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:52 smithi132 ceph-mon[37580]: pgmap v2266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:52 smithi112 ceph-mon[37501]: pgmap v2266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:45:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:54 smithi107 ceph-mon[32659]: pgmap v2267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:54.797 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:54.795+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:54.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:54.796+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:54.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:45:54.796+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:54 smithi132 ceph-mon[37580]: pgmap v2267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:54 smithi112 ceph-mon[37501]: pgmap v2267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3291.816208 secs 2024-07-26T13:45:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:56 smithi107 ceph-mon[32659]: pgmap v2268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3291.816208 secs 2024-07-26T13:45:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:56 smithi132 ceph-mon[37580]: pgmap v2268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3291.816208 secs 2024-07-26T13:45:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:56 smithi112 ceph-mon[37501]: pgmap v2268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:56.983+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:56.983+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:45:56.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:45:56.983+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:45:58 smithi107 ceph-mon[32659]: pgmap v2269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:45:58 smithi132 ceph-mon[37580]: pgmap v2269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:45:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:45:58 smithi112 ceph-mon[37501]: pgmap v2269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:00 smithi107 ceph-mon[32659]: pgmap v2270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:46:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:00 smithi132 ceph-mon[37580]: pgmap v2270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:46:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:00 smithi112 ceph-mon[37501]: pgmap v2270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:46:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3296.816274 secs 2024-07-26T13:46:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3296.816274 secs 2024-07-26T13:46:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3296.816274 secs 2024-07-26T13:46:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:02 smithi132 ceph-mon[37580]: pgmap v2271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:02 smithi112 ceph-mon[37501]: pgmap v2271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:02 smithi107 ceph-mon[32659]: pgmap v2271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:04 smithi132 ceph-mon[37580]: pgmap v2272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:04 smithi112 ceph-mon[37501]: pgmap v2272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:04 smithi107 ceph-mon[32659]: pgmap v2272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3301.816397 secs 2024-07-26T13:46:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:06 smithi132 ceph-mon[37580]: pgmap v2273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3301.816397 secs 2024-07-26T13:46:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:06 smithi112 ceph-mon[37501]: pgmap v2273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3301.816397 secs 2024-07-26T13:46:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:06 smithi107 ceph-mon[32659]: pgmap v2273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:08 smithi112 ceph-mon[37501]: pgmap v2274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:08 smithi107 ceph-mon[32659]: pgmap v2274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:08 smithi132 ceph-mon[37580]: pgmap v2274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:09.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:09.811+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:09.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:09.811+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:09.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:09.811+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:46:10.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:10 smithi107 ceph-mon[32659]: pgmap v2275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:46:10.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:10 smithi107 ceph-mon[32659]: pgmap v2276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:46:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:10 smithi112 ceph-mon[37501]: pgmap v2275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:46:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:10 smithi112 ceph-mon[37501]: pgmap v2276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:46:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:10 smithi132 ceph-mon[37580]: pgmap v2275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:46:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:10 smithi132 ceph-mon[37580]: pgmap v2276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:46:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3306.816488 secs 2024-07-26T13:46:12.000 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:11.999+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:12.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:11.999+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:12.001 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:11.999+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3306.816488 secs 2024-07-26T13:46:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3306.816488 secs 2024-07-26T13:46:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:12 smithi112 ceph-mon[37501]: pgmap v2277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:46:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:12 smithi107 ceph-mon[32659]: pgmap v2277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:46:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:12 smithi132 ceph-mon[37580]: pgmap v2277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:46:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:14 smithi112 ceph-mon[37501]: pgmap v2278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:14 smithi107 ceph-mon[32659]: pgmap v2278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:15.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:14 smithi132 ceph-mon[37580]: pgmap v2278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3311.816607 secs 2024-07-26T13:46:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:16 smithi112 ceph-mon[37501]: pgmap v2279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3311.816607 secs 2024-07-26T13:46:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:16 smithi107 ceph-mon[32659]: pgmap v2279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3311.816607 secs 2024-07-26T13:46:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:16 smithi132 ceph-mon[37580]: pgmap v2279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:18 smithi112 ceph-mon[37501]: pgmap v2280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:46:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:18 smithi107 ceph-mon[32659]: pgmap v2280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:46:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:18 smithi132 ceph-mon[37580]: pgmap v2280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:46:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:20 smithi112 ceph-mon[37501]: pgmap v2281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:21.018 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:20 smithi107 ceph-mon[32659]: pgmap v2281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:20 smithi132 ceph-mon[37580]: pgmap v2281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:46:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3316.816743 secs 2024-07-26T13:46:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3316.816743 secs 2024-07-26T13:46:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3316.816743 secs 2024-07-26T13:46:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:22 smithi112 ceph-mon[37501]: pgmap v2282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:22 smithi107 ceph-mon[32659]: pgmap v2282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:22 smithi132 ceph-mon[37580]: pgmap v2282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:24 smithi112 ceph-mon[37501]: pgmap v2283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:24 smithi107 ceph-mon[32659]: pgmap v2283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:24 smithi132 ceph-mon[37580]: pgmap v2283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:24.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:24.826+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:24.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:24.826+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:24.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:24.826+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3321.816867 secs 2024-07-26T13:46:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:26 smithi112 ceph-mon[37501]: pgmap v2284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3321.816867 secs 2024-07-26T13:46:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:26 smithi107 ceph-mon[32659]: pgmap v2284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3321.816867 secs 2024-07-26T13:46:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:26 smithi132 ceph-mon[37580]: pgmap v2284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:27.016 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:27.015+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:27.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:27.015+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:27.017 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:27.015+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:28 smithi112 ceph-mon[37501]: pgmap v2285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:28 smithi107 ceph-mon[32659]: pgmap v2285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:28 smithi132 ceph-mon[37580]: pgmap v2285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:30 smithi112 ceph-mon[37501]: pgmap v2286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:30 smithi107 ceph-mon[32659]: pgmap v2286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:30 smithi132 ceph-mon[37580]: pgmap v2286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3326.817011 secs 2024-07-26T13:46:31.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3326.817011 secs 2024-07-26T13:46:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3326.817011 secs 2024-07-26T13:46:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:32 smithi107 ceph-mon[32659]: pgmap v2287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:32 smithi132 ceph-mon[37580]: pgmap v2287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:32 smithi112 ceph-mon[37501]: pgmap v2287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:34 smithi107 ceph-mon[32659]: pgmap v2288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:34 smithi132 ceph-mon[37580]: pgmap v2288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:34 smithi112 ceph-mon[37501]: pgmap v2288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3331.817129 secs 2024-07-26T13:46:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:36 smithi107 ceph-mon[32659]: pgmap v2289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3331.817129 secs 2024-07-26T13:46:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:36 smithi132 ceph-mon[37580]: pgmap v2289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3331.817129 secs 2024-07-26T13:46:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:36 smithi112 ceph-mon[37501]: pgmap v2289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:38 smithi107 ceph-mon[32659]: pgmap v2290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:38 smithi132 ceph-mon[37580]: pgmap v2290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:38 smithi112 ceph-mon[37501]: pgmap v2290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:39.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:39.842+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:39.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:39.842+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:39.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:39.842+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:40 smithi107 ceph-mon[32659]: pgmap v2291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:40 smithi132 ceph-mon[37580]: pgmap v2291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:40 smithi112 ceph-mon[37501]: pgmap v2291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3336.817255 secs 2024-07-26T13:46:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3336.817255 secs 2024-07-26T13:46:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3336.817255 secs 2024-07-26T13:46:42.020 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:42.020+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:42.020+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:42.021 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:42.020+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:42 smithi107 ceph-mon[32659]: pgmap v2292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:42 smithi132 ceph-mon[37580]: pgmap v2292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:42 smithi112 ceph-mon[37501]: pgmap v2292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:44 smithi107 ceph-mon[32659]: pgmap v2293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:44 smithi132 ceph-mon[37580]: pgmap v2293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:44 smithi112 ceph-mon[37501]: pgmap v2293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3341.817373 secs 2024-07-26T13:46:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:46 smithi107 ceph-mon[32659]: pgmap v2294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3341.817373 secs 2024-07-26T13:46:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:46 smithi132 ceph-mon[37580]: pgmap v2294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3341.817373 secs 2024-07-26T13:46:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:46 smithi112 ceph-mon[37501]: pgmap v2294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:48 smithi107 ceph-mon[32659]: pgmap v2295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:48 smithi132 ceph-mon[37580]: pgmap v2295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:48 smithi112 ceph-mon[37501]: pgmap v2295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:50 smithi107 ceph-mon[32659]: pgmap v2296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:50 smithi132 ceph-mon[37580]: pgmap v2296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:50 smithi112 ceph-mon[37501]: pgmap v2296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:46:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3346.817494 secs 2024-07-26T13:46:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3346.817494 secs 2024-07-26T13:46:51.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3346.817494 secs 2024-07-26T13:46:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:52 smithi107 ceph-mon[32659]: pgmap v2297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:52 smithi132 ceph-mon[37580]: pgmap v2297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:52 smithi112 ceph-mon[37501]: pgmap v2297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:54 smithi107 ceph-mon[32659]: pgmap v2298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:54 smithi132 ceph-mon[37580]: pgmap v2298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:54 smithi112 ceph-mon[37501]: pgmap v2298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:54.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:54.857+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:54.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:54.857+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:54.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:46:54.857+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3351.817631 secs 2024-07-26T13:46:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:56 smithi107 ceph-mon[32659]: pgmap v2299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3351.817631 secs 2024-07-26T13:46:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:56 smithi132 ceph-mon[37580]: pgmap v2299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3351.817631 secs 2024-07-26T13:46:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:56 smithi112 ceph-mon[37501]: pgmap v2299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:57.034+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:57.034+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:46:57.036 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:46:57.034+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:46:58 smithi132 ceph-mon[37580]: pgmap v2300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:46:58 smithi112 ceph-mon[37501]: pgmap v2300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:46:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:46:58 smithi107 ceph-mon[32659]: pgmap v2300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:00 smithi112 ceph-mon[37501]: pgmap v2301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:00 smithi107 ceph-mon[32659]: pgmap v2301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:00 smithi132 ceph-mon[37580]: pgmap v2301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3356.817718 secs 2024-07-26T13:47:01.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3356.817718 secs 2024-07-26T13:47:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3356.817718 secs 2024-07-26T13:47:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:02 smithi112 ceph-mon[37501]: pgmap v2302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:02 smithi107 ceph-mon[32659]: pgmap v2302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:02 smithi132 ceph-mon[37580]: pgmap v2302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:04 smithi112 ceph-mon[37501]: pgmap v2303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:04 smithi107 ceph-mon[32659]: pgmap v2303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:04 smithi132 ceph-mon[37580]: pgmap v2303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3361.817837 secs 2024-07-26T13:47:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:06 smithi112 ceph-mon[37501]: pgmap v2304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3361.817837 secs 2024-07-26T13:47:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:06 smithi107 ceph-mon[32659]: pgmap v2304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3361.817837 secs 2024-07-26T13:47:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:06 smithi132 ceph-mon[37580]: pgmap v2304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:08 smithi112 ceph-mon[37501]: pgmap v2305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:08 smithi107 ceph-mon[32659]: pgmap v2305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:08 smithi132 ceph-mon[37580]: pgmap v2305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:09.871+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:09.871+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:09.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:09.871+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:10.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:10 smithi132 ceph-mon[37580]: pgmap v2306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:10.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:10.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.622 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.623 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.623 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:10 smithi112 ceph-mon[37501]: pgmap v2306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:10.623 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:10.623 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.623 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.623 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.623 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:10 smithi107 ceph-mon[32659]: pgmap v2306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:10.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:10.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:10.625 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3366.817908 secs 2024-07-26T13:47:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3366.817908 secs 2024-07-26T13:47:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3366.817908 secs 2024-07-26T13:47:12.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:12.050+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:12.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:12.050+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:12.052 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:12.050+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:12 smithi112 ceph-mon[37501]: pgmap v2307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:12 smithi107 ceph-mon[32659]: pgmap v2307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:12 smithi132 ceph-mon[37580]: pgmap v2307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:13 smithi112 ceph-mon[37501]: pgmap v2308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:13 smithi107 ceph-mon[32659]: pgmap v2308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:13 smithi132 ceph-mon[37580]: pgmap v2308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:14 smithi112 ceph-mon[37501]: pgmap v2309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:14 smithi107 ceph-mon[32659]: pgmap v2309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:14 smithi132 ceph-mon[37580]: pgmap v2309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:47:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3371.818021 secs 2024-07-26T13:47:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3371.818021 secs 2024-07-26T13:47:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3371.818021 secs 2024-07-26T13:47:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:16 smithi112 ceph-mon[37501]: pgmap v2310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:16 smithi107 ceph-mon[32659]: pgmap v2310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:16 smithi132 ceph-mon[37580]: pgmap v2310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:18 smithi107 ceph-mon[32659]: pgmap v2311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:47:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:18 smithi132 ceph-mon[37580]: pgmap v2311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:47:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:18 smithi112 ceph-mon[37501]: pgmap v2311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:47:21.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:20 smithi112 ceph-mon[37501]: pgmap v2312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:21.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3376.818143 secs 2024-07-26T13:47:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:20 smithi107 ceph-mon[32659]: pgmap v2312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3376.818143 secs 2024-07-26T13:47:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:20 smithi132 ceph-mon[37580]: pgmap v2312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3376.818143 secs 2024-07-26T13:47:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:22 smithi107 ceph-mon[32659]: pgmap v2313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:22 smithi132 ceph-mon[37580]: pgmap v2313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:22 smithi112 ceph-mon[37501]: pgmap v2313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:47:24.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:24.887+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:24.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:24.888+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:24.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:24.888+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:24 smithi107 ceph-mon[32659]: pgmap v2314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:24 smithi132 ceph-mon[37580]: pgmap v2314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:24 smithi112 ceph-mon[37501]: pgmap v2314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3381.818233 secs 2024-07-26T13:47:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3381.818233 secs 2024-07-26T13:47:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3381.818233 secs 2024-07-26T13:47:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:27.052+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:27.052+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:27.052+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:26 smithi107 ceph-mon[32659]: pgmap v2315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:26 smithi132 ceph-mon[37580]: pgmap v2315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:26 smithi112 ceph-mon[37501]: pgmap v2315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:28 smithi107 ceph-mon[32659]: pgmap v2316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:28 smithi132 ceph-mon[37580]: pgmap v2316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:28 smithi112 ceph-mon[37501]: pgmap v2316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:30 smithi107 ceph-mon[32659]: pgmap v2317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3386.818343 secs 2024-07-26T13:47:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:30 smithi132 ceph-mon[37580]: pgmap v2317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3386.818343 secs 2024-07-26T13:47:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:30 smithi112 ceph-mon[37501]: pgmap v2317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3386.818343 secs 2024-07-26T13:47:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:33 smithi107 ceph-mon[32659]: pgmap v2318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:33 smithi132 ceph-mon[37580]: pgmap v2318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:33 smithi112 ceph-mon[37501]: pgmap v2318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:35 smithi107 ceph-mon[32659]: pgmap v2319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:35 smithi132 ceph-mon[37580]: pgmap v2319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:35 smithi112 ceph-mon[37501]: pgmap v2319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3391.818457 secs 2024-07-26T13:47:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3391.818457 secs 2024-07-26T13:47:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3391.818457 secs 2024-07-26T13:47:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:37 smithi107 ceph-mon[32659]: pgmap v2320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:37 smithi132 ceph-mon[37580]: pgmap v2320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:37 smithi112 ceph-mon[37501]: pgmap v2320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:39 smithi107 ceph-mon[32659]: pgmap v2321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:39 smithi132 ceph-mon[37580]: pgmap v2321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:39 smithi112 ceph-mon[37501]: pgmap v2321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:39.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:39.903+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:39.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:39.903+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:39.905 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:39.903+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:41 smithi107 ceph-mon[32659]: pgmap v2322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3396.818580 secs 2024-07-26T13:47:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:41 smithi132 ceph-mon[37580]: pgmap v2322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3396.818580 secs 2024-07-26T13:47:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:41 smithi112 ceph-mon[37501]: pgmap v2322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3396.818580 secs 2024-07-26T13:47:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:42.068+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:42.068+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:42.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:42.068+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:43 smithi107 ceph-mon[32659]: pgmap v2323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:43 smithi132 ceph-mon[37580]: pgmap v2323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:43 smithi112 ceph-mon[37501]: pgmap v2323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:45 smithi132 ceph-mon[37580]: pgmap v2324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:45 smithi112 ceph-mon[37501]: pgmap v2324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:45 smithi107 ceph-mon[32659]: pgmap v2324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3401.818728 secs 2024-07-26T13:47:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3401.818728 secs 2024-07-26T13:47:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3401.818728 secs 2024-07-26T13:47:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:47 smithi112 ceph-mon[37501]: pgmap v2325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:47 smithi107 ceph-mon[32659]: pgmap v2325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:47 smithi132 ceph-mon[37580]: pgmap v2325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:49 smithi112 ceph-mon[37501]: pgmap v2326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:49 smithi107 ceph-mon[32659]: pgmap v2326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:49 smithi132 ceph-mon[37580]: pgmap v2326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:51 smithi112 ceph-mon[37501]: pgmap v2327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3406.818871 secs 2024-07-26T13:47:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:51 smithi107 ceph-mon[32659]: pgmap v2327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3406.818871 secs 2024-07-26T13:47:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:51 smithi132 ceph-mon[37580]: pgmap v2327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3406.818871 secs 2024-07-26T13:47:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:53 smithi112 ceph-mon[37501]: pgmap v2328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:53 smithi107 ceph-mon[32659]: pgmap v2328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:53.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:53 smithi132 ceph-mon[37580]: pgmap v2328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:47:54.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:54.919+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:54.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:54.919+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:54.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:47:54.919+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:55 smithi107 ceph-mon[32659]: pgmap v2329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:55 smithi112 ceph-mon[37501]: pgmap v2329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:55 smithi132 ceph-mon[37580]: pgmap v2329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3411.818979 secs 2024-07-26T13:47:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3411.818979 secs 2024-07-26T13:47:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3411.818979 secs 2024-07-26T13:47:57.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:57.083+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:57.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:57.083+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:57.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:47:57.083+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:47:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:57 smithi112 ceph-mon[37501]: pgmap v2330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:57 smithi107 ceph-mon[32659]: pgmap v2330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:57.655 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:57 smithi132 ceph-mon[37580]: pgmap v2330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:47:59 smithi112 ceph-mon[37501]: pgmap v2331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:47:59 smithi107 ceph-mon[32659]: pgmap v2331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:47:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:47:59 smithi132 ceph-mon[37580]: pgmap v2331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:01.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:01 smithi132 ceph-mon[37580]: pgmap v2332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:01.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3416.819145 secs 2024-07-26T13:48:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:01 smithi112 ceph-mon[37501]: pgmap v2332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3416.819145 secs 2024-07-26T13:48:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:01 smithi107 ceph-mon[32659]: pgmap v2332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3416.819145 secs 2024-07-26T13:48:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:03 smithi112 ceph-mon[37501]: pgmap v2333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:03 smithi107 ceph-mon[32659]: pgmap v2333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:03 smithi132 ceph-mon[37580]: pgmap v2333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:05 smithi112 ceph-mon[37501]: pgmap v2334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:05 smithi107 ceph-mon[32659]: pgmap v2334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:05 smithi132 ceph-mon[37580]: pgmap v2334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3421.819261 secs 2024-07-26T13:48:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3421.819261 secs 2024-07-26T13:48:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3421.819261 secs 2024-07-26T13:48:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:07 smithi112 ceph-mon[37501]: pgmap v2335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:07 smithi107 ceph-mon[32659]: pgmap v2335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:07 smithi132 ceph-mon[37580]: pgmap v2335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:09.344 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:09 smithi107 ceph-mon[32659]: pgmap v2336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:09 smithi112 ceph-mon[37501]: pgmap v2336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:09 smithi132 ceph-mon[37580]: pgmap v2336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:09.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:09.923+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:09.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:09.923+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:09.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:09.923+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:11 smithi112 ceph-mon[37501]: pgmap v2337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3426.819390 secs 2024-07-26T13:48:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:11 smithi107 ceph-mon[32659]: pgmap v2337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3426.819390 secs 2024-07-26T13:48:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:11 smithi132 ceph-mon[37580]: pgmap v2337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3426.819390 secs 2024-07-26T13:48:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:12.098+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:12.098+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:12.098+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:13.215+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:48:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:13.215+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:48:13.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:13.259+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:48:13.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:13.259+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:48:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:13 smithi112 ceph-mon[37501]: pgmap v2338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:13 smithi107 ceph-mon[32659]: pgmap v2338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:13 smithi132 ceph-mon[37580]: pgmap v2338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:14.347+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:48:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:14.347+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:48:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:14.348+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:48:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:14.348+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:48:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:15 smithi112 ceph-mon[37501]: pgmap v2339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:48:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:15 smithi107 ceph-mon[32659]: pgmap v2339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:48:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:15 smithi132 ceph-mon[37580]: pgmap v2339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:48:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:16 smithi112 ceph-mon[37501]: pgmap v2340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:48:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:16 smithi112 ceph-mon[37501]: pgmap v2341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3431.819474 secs 2024-07-26T13:48:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:16 smithi107 ceph-mon[32659]: pgmap v2340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:48:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:16 smithi107 ceph-mon[32659]: pgmap v2341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3431.819474 secs 2024-07-26T13:48:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:16 smithi132 ceph-mon[37580]: pgmap v2340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:48:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:16 smithi132 ceph-mon[37580]: pgmap v2341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3431.819474 secs 2024-07-26T13:48:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:18 smithi112 ceph-mon[37501]: pgmap v2342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:18 smithi107 ceph-mon[32659]: pgmap v2342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:18 smithi132 ceph-mon[37580]: pgmap v2342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:20 smithi112 ceph-mon[37501]: pgmap v2343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:20 smithi107 ceph-mon[32659]: pgmap v2343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:20 smithi132 ceph-mon[37580]: pgmap v2343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3436.819602 secs 2024-07-26T13:48:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3436.819602 secs 2024-07-26T13:48:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3436.819602 secs 2024-07-26T13:48:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:22 smithi112 ceph-mon[37501]: pgmap v2344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:48:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:22 smithi107 ceph-mon[32659]: pgmap v2344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:48:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:22 smithi132 ceph-mon[37580]: pgmap v2344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:48:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:24 smithi107 ceph-mon[32659]: pgmap v2345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:48:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:24 smithi132 ceph-mon[37580]: pgmap v2345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:48:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:24 smithi112 ceph-mon[37501]: pgmap v2345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:48:24.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:24.939+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:24.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:24.939+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:24.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:24.939+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:26 smithi107 ceph-mon[32659]: pgmap v2346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:48:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3441.819733 secs 2024-07-26T13:48:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:26 smithi132 ceph-mon[37580]: pgmap v2346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:48:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3441.819733 secs 2024-07-26T13:48:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:26 smithi112 ceph-mon[37501]: pgmap v2346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:48:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3441.819733 secs 2024-07-26T13:48:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:27.104+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:27.105 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:27.104+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:27.106 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:27.104+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:28 smithi107 ceph-mon[32659]: pgmap v2347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:28 smithi132 ceph-mon[37580]: pgmap v2347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:28 smithi112 ceph-mon[37501]: pgmap v2347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:30 smithi107 ceph-mon[32659]: pgmap v2348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:30 smithi132 ceph-mon[37580]: pgmap v2348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:30 smithi112 ceph-mon[37501]: pgmap v2348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3446.819853 secs 2024-07-26T13:48:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3446.819853 secs 2024-07-26T13:48:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3446.819853 secs 2024-07-26T13:48:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:32 smithi107 ceph-mon[32659]: pgmap v2349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:32 smithi132 ceph-mon[37580]: pgmap v2349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:32 smithi112 ceph-mon[37501]: pgmap v2349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:34 smithi107 ceph-mon[32659]: pgmap v2350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:34 smithi132 ceph-mon[37580]: pgmap v2350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:34 smithi112 ceph-mon[37501]: pgmap v2350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:36 smithi107 ceph-mon[32659]: pgmap v2351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3451.819989 secs 2024-07-26T13:48:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:36 smithi132 ceph-mon[37580]: pgmap v2351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3451.819989 secs 2024-07-26T13:48:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:36 smithi112 ceph-mon[37501]: pgmap v2351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3451.819989 secs 2024-07-26T13:48:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:38 smithi107 ceph-mon[32659]: pgmap v2352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:38 smithi132 ceph-mon[37580]: pgmap v2352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:38 smithi112 ceph-mon[37501]: pgmap v2352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:39.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:39.955+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:39.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:39.955+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:39.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:39.955+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:40 smithi107 ceph-mon[32659]: pgmap v2353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:40 smithi132 ceph-mon[37580]: pgmap v2353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:40 smithi112 ceph-mon[37501]: pgmap v2353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3456.820151 secs 2024-07-26T13:48:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3456.820151 secs 2024-07-26T13:48:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3456.820151 secs 2024-07-26T13:48:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:42.111+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:42.111+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:42.111+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:42 smithi107 ceph-mon[32659]: pgmap v2354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:42 smithi132 ceph-mon[37580]: pgmap v2354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:42 smithi112 ceph-mon[37501]: pgmap v2354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:44 smithi107 ceph-mon[32659]: pgmap v2355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:44 smithi132 ceph-mon[37580]: pgmap v2355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:44 smithi112 ceph-mon[37501]: pgmap v2355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:46 smithi107 ceph-mon[32659]: pgmap v2356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3461.820275 secs 2024-07-26T13:48:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:46 smithi132 ceph-mon[37580]: pgmap v2356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3461.820275 secs 2024-07-26T13:48:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:46 smithi112 ceph-mon[37501]: pgmap v2356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3461.820275 secs 2024-07-26T13:48:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:48 smithi107 ceph-mon[32659]: pgmap v2357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:48 smithi132 ceph-mon[37580]: pgmap v2357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:48 smithi112 ceph-mon[37501]: pgmap v2357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:50 smithi132 ceph-mon[37580]: pgmap v2358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:50 smithi112 ceph-mon[37501]: pgmap v2358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:50 smithi107 ceph-mon[32659]: pgmap v2358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:51.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3466.820408 secs 2024-07-26T13:48:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3466.820408 secs 2024-07-26T13:48:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3466.820408 secs 2024-07-26T13:48:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:52 smithi112 ceph-mon[37501]: pgmap v2359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:52 smithi107 ceph-mon[32659]: pgmap v2359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:52 smithi132 ceph-mon[37580]: pgmap v2359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:54 smithi112 ceph-mon[37501]: pgmap v2360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:54 smithi107 ceph-mon[32659]: pgmap v2360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:54 smithi132 ceph-mon[37580]: pgmap v2360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:48:54.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:54.970+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:54.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:54.970+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:54.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:54.970+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:56 smithi107 ceph-mon[32659]: pgmap v2361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3471.820535 secs 2024-07-26T13:48:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:56 smithi112 ceph-mon[37501]: pgmap v2361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3471.820535 secs 2024-07-26T13:48:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:56 smithi132 ceph-mon[37580]: pgmap v2361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3471.820535 secs 2024-07-26T13:48:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:57.126+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:57.126+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:57.126+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:48:57.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:57.221+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:48:57.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:57.222+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:48:57.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:57.265+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:48:57.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:48:57.266+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:48:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:48:58 smithi112 ceph-mon[37501]: pgmap v2362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:48:58 smithi107 ceph-mon[32659]: pgmap v2362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:48:58 smithi132 ceph-mon[37580]: pgmap v2362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:48:59.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:59.357+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:48:59.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:59.357+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:48:59.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:59.358+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:48:59.359 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:48:59.358+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:49:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:00 smithi112 ceph-mon[37501]: pgmap v2363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:00 smithi107 ceph-mon[32659]: pgmap v2363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:00 smithi132 ceph-mon[37580]: pgmap v2363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3476.820611 secs 2024-07-26T13:49:01.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3476.820611 secs 2024-07-26T13:49:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3476.820611 secs 2024-07-26T13:49:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:02 smithi112 ceph-mon[37501]: pgmap v2364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:02 smithi107 ceph-mon[32659]: pgmap v2364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:02 smithi132 ceph-mon[37580]: pgmap v2364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:04 smithi112 ceph-mon[37501]: pgmap v2365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:04 smithi107 ceph-mon[32659]: pgmap v2365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:04 smithi132 ceph-mon[37580]: pgmap v2365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:06 smithi112 ceph-mon[37501]: pgmap v2366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3481.820728 secs 2024-07-26T13:49:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:06 smithi107 ceph-mon[32659]: pgmap v2366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3481.820728 secs 2024-07-26T13:49:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:06 smithi132 ceph-mon[37580]: pgmap v2366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3481.820728 secs 2024-07-26T13:49:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:08 smithi112 ceph-mon[37501]: pgmap v2367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:08 smithi107 ceph-mon[32659]: pgmap v2367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:08 smithi132 ceph-mon[37580]: pgmap v2367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:09.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:09.985+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:09.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:09.985+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:09.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:09.985+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:10 smithi112 ceph-mon[37501]: pgmap v2368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:10 smithi107 ceph-mon[32659]: pgmap v2368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:10 smithi132 ceph-mon[37580]: pgmap v2368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3486.820825 secs 2024-07-26T13:49:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3486.820825 secs 2024-07-26T13:49:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3486.820825 secs 2024-07-26T13:49:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:12.127+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:12.127+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:12.127+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:12 smithi112 ceph-mon[37501]: pgmap v2369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:12 smithi107 ceph-mon[32659]: pgmap v2369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:12 smithi132 ceph-mon[37580]: pgmap v2369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:14 smithi112 ceph-mon[37501]: pgmap v2370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:14 smithi107 ceph-mon[32659]: pgmap v2370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:14 smithi132 ceph-mon[37580]: pgmap v2370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:49:15.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:49:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:49:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:16 smithi112 ceph-mon[37501]: pgmap v2371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3491.820946 secs 2024-07-26T13:49:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:49:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:49:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:49:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:16 smithi107 ceph-mon[32659]: pgmap v2371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3491.820946 secs 2024-07-26T13:49:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:49:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:49:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:49:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:16 smithi132 ceph-mon[37580]: pgmap v2371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3491.820946 secs 2024-07-26T13:49:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:49:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:49:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:49:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:17 smithi112 ceph-mon[37501]: pgmap v2372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:49:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:17 smithi112 ceph-mon[37501]: pgmap v2373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:49:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:17 smithi107 ceph-mon[32659]: pgmap v2372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:49:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:17 smithi107 ceph-mon[32659]: pgmap v2373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:49:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:17 smithi132 ceph-mon[37580]: pgmap v2372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:49:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:17 smithi132 ceph-mon[37580]: pgmap v2373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:49:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:19 smithi112 ceph-mon[37501]: pgmap v2374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:19 smithi107 ceph-mon[32659]: pgmap v2374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:19 smithi132 ceph-mon[37580]: pgmap v2374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3496.821086 secs 2024-07-26T13:49:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:21 smithi112 ceph-mon[37501]: pgmap v2375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3496.821086 secs 2024-07-26T13:49:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:21 smithi107 ceph-mon[32659]: pgmap v2375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3496.821086 secs 2024-07-26T13:49:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:21 smithi132 ceph-mon[37580]: pgmap v2375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:23 smithi112 ceph-mon[37501]: pgmap v2376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:23 smithi107 ceph-mon[32659]: pgmap v2376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:23 smithi132 ceph-mon[37580]: pgmap v2376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:25.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:25.001+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:25.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:25.001+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:25.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:25.001+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:25 smithi112 ceph-mon[37501]: pgmap v2377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:25 smithi107 ceph-mon[32659]: pgmap v2377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:25 smithi132 ceph-mon[37580]: pgmap v2377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3501.821181 secs 2024-07-26T13:49:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3501.821181 secs 2024-07-26T13:49:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3501.821181 secs 2024-07-26T13:49:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:27.133+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:27.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:27.133+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:27.135 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:27.133+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:27 smithi112 ceph-mon[37501]: pgmap v2378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:49:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:27 smithi107 ceph-mon[32659]: pgmap v2378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:49:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:27 smithi132 ceph-mon[37580]: pgmap v2378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:49:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:29 smithi107 ceph-mon[32659]: pgmap v2379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:29 smithi132 ceph-mon[37580]: pgmap v2379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:29 smithi112 ceph-mon[37501]: pgmap v2379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3506.821264 secs 2024-07-26T13:49:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:31 smithi107 ceph-mon[32659]: pgmap v2380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3506.821264 secs 2024-07-26T13:49:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:31 smithi132 ceph-mon[37580]: pgmap v2380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3506.821264 secs 2024-07-26T13:49:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:31 smithi112 ceph-mon[37501]: pgmap v2380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:33 smithi107 ceph-mon[32659]: pgmap v2381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:33 smithi132 ceph-mon[37580]: pgmap v2381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:33 smithi112 ceph-mon[37501]: pgmap v2381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:35 smithi107 ceph-mon[32659]: pgmap v2382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:35 smithi132 ceph-mon[37580]: pgmap v2382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:35 smithi112 ceph-mon[37501]: pgmap v2382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3511.821341 secs 2024-07-26T13:49:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3511.821341 secs 2024-07-26T13:49:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3511.821341 secs 2024-07-26T13:49:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:37 smithi107 ceph-mon[32659]: pgmap v2383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:37 smithi132 ceph-mon[37580]: pgmap v2383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:37 smithi112 ceph-mon[37501]: pgmap v2383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:39 smithi107 ceph-mon[32659]: pgmap v2384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:39 smithi132 ceph-mon[37580]: pgmap v2384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:39 smithi112 ceph-mon[37501]: pgmap v2384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:40.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:40.012+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:40.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:40.012+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:40.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:40.012+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3516.821455 secs 2024-07-26T13:49:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:41 smithi107 ceph-mon[32659]: pgmap v2385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3516.821455 secs 2024-07-26T13:49:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:41 smithi132 ceph-mon[37580]: pgmap v2385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3516.821455 secs 2024-07-26T13:49:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:41 smithi112 ceph-mon[37501]: pgmap v2385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:42.149+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:42.149+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:42.149+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:43 smithi107 ceph-mon[32659]: pgmap v2386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:43 smithi132 ceph-mon[37580]: pgmap v2386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:43 smithi112 ceph-mon[37501]: pgmap v2386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:45 smithi107 ceph-mon[32659]: pgmap v2387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:45 smithi132 ceph-mon[37580]: pgmap v2387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:45 smithi112 ceph-mon[37501]: pgmap v2387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3521.821536 secs 2024-07-26T13:49:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3521.821536 secs 2024-07-26T13:49:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3521.821536 secs 2024-07-26T13:49:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:47 smithi107 ceph-mon[32659]: pgmap v2388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:47 smithi132 ceph-mon[37580]: pgmap v2388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:47 smithi112 ceph-mon[37501]: pgmap v2388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:49 smithi107 ceph-mon[32659]: pgmap v2389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:49 smithi132 ceph-mon[37580]: pgmap v2389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:49 smithi112 ceph-mon[37501]: pgmap v2389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3526.821658 secs 2024-07-26T13:49:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:51 smithi107 ceph-mon[32659]: pgmap v2390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3526.821658 secs 2024-07-26T13:49:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:51 smithi132 ceph-mon[37580]: pgmap v2390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3526.821658 secs 2024-07-26T13:49:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:51 smithi112 ceph-mon[37501]: pgmap v2390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:53 smithi107 ceph-mon[32659]: pgmap v2391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:53 smithi132 ceph-mon[37580]: pgmap v2391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:53 smithi112 ceph-mon[37501]: pgmap v2391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:49:55.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:55.015+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:55.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:55.015+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:55.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:49:55.015+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:55 smithi132 ceph-mon[37580]: pgmap v2392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:55 smithi112 ceph-mon[37501]: pgmap v2392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:55 smithi107 ceph-mon[32659]: pgmap v2392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:49:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3531.821719 secs 2024-07-26T13:49:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3531.821719 secs 2024-07-26T13:49:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3531.821719 secs 2024-07-26T13:49:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:57.159+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:57.159+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:49:57.159+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:49:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:57 smithi112 ceph-mon[37501]: pgmap v2393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:57 smithi107 ceph-mon[32659]: pgmap v2393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:57 smithi132 ceph-mon[37580]: pgmap v2393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:49:59 smithi112 ceph-mon[37501]: pgmap v2394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:49:59 smithi107 ceph-mon[32659]: pgmap v2394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:00.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:49:59 smithi132 ceph-mon[37580]: pgmap v2394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:50:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:50:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T13:50:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3536.821862 secs 2024-07-26T13:50:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:01 smithi112 ceph-mon[37501]: pgmap v2395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:01.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3536.821862 secs 2024-07-26T13:50:01.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:01 smithi132 ceph-mon[37580]: pgmap v2395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3536.821862 secs 2024-07-26T13:50:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:01 smithi107 ceph-mon[32659]: pgmap v2395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:03 smithi112 ceph-mon[37501]: pgmap v2396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:03 smithi107 ceph-mon[32659]: pgmap v2396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:04.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:03 smithi132 ceph-mon[37580]: pgmap v2396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:05 smithi112 ceph-mon[37501]: pgmap v2397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:05 smithi107 ceph-mon[32659]: pgmap v2397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:05 smithi132 ceph-mon[37580]: pgmap v2397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3541.822012 secs 2024-07-26T13:50:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3541.822012 secs 2024-07-26T13:50:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3541.822012 secs 2024-07-26T13:50:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:07 smithi112 ceph-mon[37501]: pgmap v2398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:07 smithi107 ceph-mon[32659]: pgmap v2398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:07 smithi132 ceph-mon[37580]: pgmap v2398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:09 smithi112 ceph-mon[37501]: pgmap v2399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:10.017 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:09 smithi132 ceph-mon[37580]: pgmap v2399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:10.033 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:09 smithi107 ceph-mon[32659]: pgmap v2399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:10.030+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:10.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:10.030+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:10.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:10.030+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3546.822108 secs 2024-07-26T13:50:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:11 smithi112 ceph-mon[37501]: pgmap v2400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3546.822108 secs 2024-07-26T13:50:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:11 smithi107 ceph-mon[32659]: pgmap v2400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3546.822108 secs 2024-07-26T13:50:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:11 smithi132 ceph-mon[37580]: pgmap v2400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:12.167+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:12.167+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:12.167+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:13 smithi112 ceph-mon[37501]: pgmap v2401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:13.978 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:13 smithi107 ceph-mon[32659]: pgmap v2401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:13 smithi132 ceph-mon[37580]: pgmap v2401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:15 smithi112 ceph-mon[37501]: pgmap v2402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:15 smithi107 ceph-mon[32659]: pgmap v2402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:15 smithi132 ceph-mon[37580]: pgmap v2402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3551.822224 secs 2024-07-26T13:50:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3551.822224 secs 2024-07-26T13:50:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3551.822224 secs 2024-07-26T13:50:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:17 smithi112 ceph-mon[37501]: pgmap v2403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:50:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:17 smithi107 ceph-mon[32659]: pgmap v2403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:50:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:17 smithi132 ceph-mon[37580]: pgmap v2403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:50:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:18 smithi112 ceph-mon[37501]: pgmap v2404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:50:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:18 smithi112 ceph-mon[37501]: pgmap v2405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:50:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:18 smithi107 ceph-mon[32659]: pgmap v2404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:50:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:18 smithi107 ceph-mon[32659]: pgmap v2405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:50:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:18 smithi132 ceph-mon[37580]: pgmap v2404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:50:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:18 smithi132 ceph-mon[37580]: pgmap v2405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:50:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:20 smithi112 ceph-mon[37501]: pgmap v2406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:20 smithi107 ceph-mon[32659]: pgmap v2406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:20 smithi132 ceph-mon[37580]: pgmap v2406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3556.822304 secs 2024-07-26T13:50:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3556.822304 secs 2024-07-26T13:50:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3556.822304 secs 2024-07-26T13:50:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:22 smithi112 ceph-mon[37501]: pgmap v2407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:22 smithi107 ceph-mon[32659]: pgmap v2407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:22 smithi132 ceph-mon[37580]: pgmap v2407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:23 smithi112 ceph-mon[37501]: pgmap v2408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:23 smithi107 ceph-mon[32659]: pgmap v2408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:23 smithi132 ceph-mon[37580]: pgmap v2408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:25.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:25.046+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:25.046+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:25.046+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3561.822420 secs 2024-07-26T13:50:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:26 smithi112 ceph-mon[37501]: pgmap v2409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3561.822420 secs 2024-07-26T13:50:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:26 smithi107 ceph-mon[32659]: pgmap v2409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3561.822420 secs 2024-07-26T13:50:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:26 smithi132 ceph-mon[37580]: pgmap v2409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:50:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:27.184+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:27.184+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:27.184+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:28.826 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:28 smithi107 ceph-mon[32659]: pgmap v2410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:50:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:28 smithi112 ceph-mon[37501]: pgmap v2410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:50:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:28 smithi132 ceph-mon[37580]: pgmap v2410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:50:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:30 smithi112 ceph-mon[37501]: pgmap v2411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:50:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:30 smithi107 ceph-mon[32659]: pgmap v2411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:50:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:30 smithi132 ceph-mon[37580]: pgmap v2411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:50:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3566.822499 secs 2024-07-26T13:50:31.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3566.822499 secs 2024-07-26T13:50:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3566.822499 secs 2024-07-26T13:50:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:32 smithi112 ceph-mon[37501]: pgmap v2412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:32 smithi107 ceph-mon[32659]: pgmap v2412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:32 smithi132 ceph-mon[37580]: pgmap v2412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:34 smithi112 ceph-mon[37501]: pgmap v2413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:34 smithi107 ceph-mon[32659]: pgmap v2413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:34 smithi132 ceph-mon[37580]: pgmap v2413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3571.822616 secs 2024-07-26T13:50:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:36 smithi112 ceph-mon[37501]: pgmap v2414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3571.822616 secs 2024-07-26T13:50:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:36 smithi107 ceph-mon[32659]: pgmap v2414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3571.822616 secs 2024-07-26T13:50:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:36 smithi132 ceph-mon[37580]: pgmap v2414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:38 smithi112 ceph-mon[37501]: pgmap v2415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:38 smithi107 ceph-mon[32659]: pgmap v2415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:38 smithi132 ceph-mon[37580]: pgmap v2415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:40.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:40.062+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:40.062+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:40.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:40.062+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:40 smithi112 ceph-mon[37501]: pgmap v2416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:50:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:40 smithi107 ceph-mon[32659]: pgmap v2416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:50:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:40 smithi132 ceph-mon[37580]: pgmap v2416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:50:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3576.822704 secs 2024-07-26T13:50:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3576.822704 secs 2024-07-26T13:50:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3576.822704 secs 2024-07-26T13:50:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:42.200+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:42.200+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:42.200+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:42 smithi112 ceph-mon[37501]: pgmap v2417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:42 smithi107 ceph-mon[32659]: pgmap v2417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:42 smithi132 ceph-mon[37580]: pgmap v2417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:50:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:44 smithi112 ceph-mon[37501]: pgmap v2418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:44 smithi107 ceph-mon[32659]: pgmap v2418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:44 smithi132 ceph-mon[37580]: pgmap v2418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3581.822796 secs 2024-07-26T13:50:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:46 smithi112 ceph-mon[37501]: pgmap v2419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:50:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3581.822796 secs 2024-07-26T13:50:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:46 smithi107 ceph-mon[32659]: pgmap v2419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:50:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3581.822796 secs 2024-07-26T13:50:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:46 smithi132 ceph-mon[37580]: pgmap v2419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:50:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:48 smithi112 ceph-mon[37501]: pgmap v2420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:48 smithi107 ceph-mon[32659]: pgmap v2420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:48 smithi132 ceph-mon[37580]: pgmap v2420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:50 smithi112 ceph-mon[37501]: pgmap v2421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:50:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:50 smithi107 ceph-mon[32659]: pgmap v2421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:50:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:50 smithi132 ceph-mon[37580]: pgmap v2421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:50:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3586.822917 secs 2024-07-26T13:50:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3586.822917 secs 2024-07-26T13:50:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3586.822917 secs 2024-07-26T13:50:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:52 smithi112 ceph-mon[37501]: pgmap v2422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:50:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:52 smithi107 ceph-mon[32659]: pgmap v2422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:50:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:52 smithi132 ceph-mon[37580]: pgmap v2422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:50:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:54 smithi112 ceph-mon[37501]: pgmap v2423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:54 smithi107 ceph-mon[32659]: pgmap v2423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:54 smithi132 ceph-mon[37580]: pgmap v2423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:50:55.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:55.078+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:55.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:55.078+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:55.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:50:55.078+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3591.823031 secs 2024-07-26T13:50:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3591.823031 secs 2024-07-26T13:50:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3591.823031 secs 2024-07-26T13:50:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:56 smithi112 ceph-mon[37501]: pgmap v2424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:56 smithi107 ceph-mon[32659]: pgmap v2424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:56 smithi132 ceph-mon[37580]: pgmap v2424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:57.215+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:57.215+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:57.216 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:50:57.215+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:50:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:50:57 smithi112 ceph-mon[37501]: pgmap v2425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:50:57 smithi107 ceph-mon[32659]: pgmap v2425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:50:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:50:57 smithi132 ceph-mon[37580]: pgmap v2425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:00 smithi112 ceph-mon[37501]: pgmap v2426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:51:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:00 smithi107 ceph-mon[32659]: pgmap v2426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:51:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:00 smithi132 ceph-mon[37580]: pgmap v2426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:51:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3596.823148 secs 2024-07-26T13:51:01.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3596.823148 secs 2024-07-26T13:51:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3596.823148 secs 2024-07-26T13:51:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:02 smithi112 ceph-mon[37501]: pgmap v2427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:51:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:02 smithi107 ceph-mon[32659]: pgmap v2427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:51:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:02 smithi132 ceph-mon[37580]: pgmap v2427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:51:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:04 smithi112 ceph-mon[37501]: pgmap v2428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:51:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:04 smithi107 ceph-mon[32659]: pgmap v2428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:51:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:04 smithi132 ceph-mon[37580]: pgmap v2428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:51:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3601.823271 secs 2024-07-26T13:51:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:06 smithi112 ceph-mon[37501]: pgmap v2429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3601.823271 secs 2024-07-26T13:51:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:06 smithi107 ceph-mon[32659]: pgmap v2429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3601.823271 secs 2024-07-26T13:51:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:06 smithi132 ceph-mon[37580]: pgmap v2429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:08 smithi112 ceph-mon[37501]: pgmap v2430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:51:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:08 smithi107 ceph-mon[32659]: pgmap v2430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:51:09.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:08 smithi132 ceph-mon[37580]: pgmap v2430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:51:10.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:10.090+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:10.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:10.090+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:10.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:10.090+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:10 smithi112 ceph-mon[37501]: pgmap v2431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:51:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:10 smithi107 ceph-mon[32659]: pgmap v2431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:51:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:10 smithi132 ceph-mon[37580]: pgmap v2431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:11.645+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 msgr2=0x7ff0ec0c2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T13:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:11.645+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 msgr2=0x7ff0ec0c2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:11.645+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c2350 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fa00 tx=0x7ff0f801f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:51:11.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:11.645+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c2350 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fa00 tx=0x7ff0f801f040 comp rx=0 tx=0).stop 2024-07-26T13:51:11.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:11.645+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 msgr2=0x7ff0ec0c2350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:51:11.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:11.645+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bf410 0x7ff0ec0c2350 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:51:11.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:11.645+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:51:11.648 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:11.646+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:51:11.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:11.648+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c1e10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f802d040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:51:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3606.823391 secs 2024-07-26T13:51:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3606.823391 secs 2024-07-26T13:51:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3606.823391 secs 2024-07-26T13:51:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:12.225+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:12.225+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:12.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:12.225+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:12 smithi112 ceph-mon[37501]: pgmap v2432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:51:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:12 smithi107 ceph-mon[32659]: pgmap v2432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:51:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:12 smithi132 ceph-mon[37580]: pgmap v2432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:51:13.246 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:13.245+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:51:13.247 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:13.245+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:51:13.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:13.287+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:51:13.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:13.288+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:51:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:13.754+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 msgr2=0x7efc5c0c22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T13:51:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:13.754+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 msgr2=0x7efc5c0c22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:51:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:13.754+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c22b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efc48005820 tx=0x7efc680c1000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:51:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:13.754+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c22b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7efc48005820 tx=0x7efc680c1000 comp rx=0 tx=0).stop 2024-07-26T13:51:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:13.754+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 msgr2=0x7efc5c0c22b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T13:51:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:13.754+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 msgr2=0x7efc5c0c22b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:51:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:13.754+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bf410 0x7efc5c0c22b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:51:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:13.754+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:51:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:13.755+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:51:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:13.756+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c27f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efc480063a0 tx=0x7efc680e6000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:51:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:14.382+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:51:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:14.382+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:51:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:14.382+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:51:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:14.382+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:51:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:14 smithi112 ceph-mon[37501]: pgmap v2433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:14 smithi107 ceph-mon[32659]: pgmap v2433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:14 smithi132 ceph-mon[37580]: pgmap v2433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3611.823510 secs 2024-07-26T13:51:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:16 smithi112 ceph-mon[37501]: pgmap v2434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3611.823510 secs 2024-07-26T13:51:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:16 smithi107 ceph-mon[32659]: pgmap v2434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3611.823510 secs 2024-07-26T13:51:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:16 smithi132 ceph-mon[37580]: pgmap v2434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:18 smithi112 ceph-mon[37501]: pgmap v2435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:18 smithi107 ceph-mon[32659]: pgmap v2435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:18 smithi132 ceph-mon[37580]: pgmap v2435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:51:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:51:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:51:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:20 smithi112 ceph-mon[37501]: pgmap v2436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:51:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:20 smithi112 ceph-mon[37501]: pgmap v2437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:51:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:20 smithi107 ceph-mon[32659]: pgmap v2436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:51:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:20 smithi107 ceph-mon[32659]: pgmap v2437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:51:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:20 smithi132 ceph-mon[37580]: pgmap v2436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:51:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:20 smithi132 ceph-mon[37580]: pgmap v2437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:51:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3616.823614 secs 2024-07-26T13:51:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3616.823614 secs 2024-07-26T13:51:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3616.823614 secs 2024-07-26T13:51:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:22 smithi112 ceph-mon[37501]: pgmap v2438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:51:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:22 smithi107 ceph-mon[32659]: pgmap v2438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:51:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:22 smithi132 ceph-mon[37580]: pgmap v2438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:51:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:24 smithi112 ceph-mon[37501]: pgmap v2439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:51:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:24 smithi107 ceph-mon[32659]: pgmap v2439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:51:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:24 smithi132 ceph-mon[37580]: pgmap v2439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:51:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:25.105+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:25.105+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:25.106 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:25.105+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:25 smithi112 ceph-mon[37501]: pgmap v2440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3621.823730 secs 2024-07-26T13:51:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:25 smithi107 ceph-mon[32659]: pgmap v2440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3621.823730 secs 2024-07-26T13:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:25 smithi132 ceph-mon[37580]: pgmap v2440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3621.823730 secs 2024-07-26T13:51:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:27.242+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:27.242+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:27.242+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:27 smithi112 ceph-mon[37501]: pgmap v2441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:27 smithi107 ceph-mon[32659]: pgmap v2441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:27 smithi132 ceph-mon[37580]: pgmap v2441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:29 smithi112 ceph-mon[37501]: pgmap v2442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:51:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:29 smithi107 ceph-mon[32659]: pgmap v2442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:51:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:29 smithi132 ceph-mon[37580]: pgmap v2442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:51:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3626.823853 secs 2024-07-26T13:51:31.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3626.823853 secs 2024-07-26T13:51:31.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3626.823853 secs 2024-07-26T13:51:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:31 smithi112 ceph-mon[37501]: pgmap v2443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:31 smithi107 ceph-mon[32659]: pgmap v2443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:31 smithi132 ceph-mon[37580]: pgmap v2443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:33 smithi112 ceph-mon[37501]: pgmap v2444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:33 smithi107 ceph-mon[32659]: pgmap v2444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:33 smithi132 ceph-mon[37580]: pgmap v2444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:35 smithi112 ceph-mon[37501]: pgmap v2445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3631.823971 secs 2024-07-26T13:51:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:35 smithi107 ceph-mon[32659]: pgmap v2445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3631.823971 secs 2024-07-26T13:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:35 smithi132 ceph-mon[37580]: pgmap v2445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3631.823971 secs 2024-07-26T13:51:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:37 smithi112 ceph-mon[37501]: pgmap v2446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:37 smithi107 ceph-mon[32659]: pgmap v2446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:37 smithi132 ceph-mon[37580]: pgmap v2446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:40.121+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:40.121+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:40.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:40.121+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:40.123 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:39 smithi107 ceph-mon[32659]: pgmap v2447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:39 smithi112 ceph-mon[37501]: pgmap v2447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:39 smithi132 ceph-mon[37580]: pgmap v2447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3636.824126 secs 2024-07-26T13:51:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3636.824126 secs 2024-07-26T13:51:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3636.824126 secs 2024-07-26T13:51:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:41 smithi112 ceph-mon[37501]: pgmap v2448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:42.257 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:42.257+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:42.257+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:42.257+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:41 smithi107 ceph-mon[32659]: pgmap v2448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:41 smithi132 ceph-mon[37580]: pgmap v2448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:43 smithi112 ceph-mon[37501]: pgmap v2449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:43 smithi107 ceph-mon[32659]: pgmap v2449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:43 smithi132 ceph-mon[37580]: pgmap v2449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:45 smithi112 ceph-mon[37501]: pgmap v2450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3641.824236 secs 2024-07-26T13:51:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:45 smithi107 ceph-mon[32659]: pgmap v2450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3641.824236 secs 2024-07-26T13:51:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:45 smithi132 ceph-mon[37580]: pgmap v2450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3641.824236 secs 2024-07-26T13:51:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:47 smithi112 ceph-mon[37501]: pgmap v2451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:47 smithi107 ceph-mon[32659]: pgmap v2451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:47 smithi132 ceph-mon[37580]: pgmap v2451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:49 smithi112 ceph-mon[37501]: pgmap v2452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:49 smithi107 ceph-mon[32659]: pgmap v2452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:49 smithi132 ceph-mon[37580]: pgmap v2452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3646.824356 secs 2024-07-26T13:51:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3646.824356 secs 2024-07-26T13:51:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3646.824356 secs 2024-07-26T13:51:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:51 smithi112 ceph-mon[37501]: pgmap v2453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:51 smithi107 ceph-mon[32659]: pgmap v2453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:51 smithi132 ceph-mon[37580]: pgmap v2453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:53 smithi112 ceph-mon[37501]: pgmap v2454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:53 smithi107 ceph-mon[32659]: pgmap v2454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:53 smithi132 ceph-mon[37580]: pgmap v2454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:55.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:55.130+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:55.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:55.130+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:55.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:51:55.130+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:55 smithi112 ceph-mon[37501]: pgmap v2455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3651.824493 secs 2024-07-26T13:51:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:55 smithi107 ceph-mon[32659]: pgmap v2455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3651.824493 secs 2024-07-26T13:51:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:55 smithi132 ceph-mon[37580]: pgmap v2455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3651.824493 secs 2024-07-26T13:51:57.273 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:57.272+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:57.273 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:57.273+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:51:57.273 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:51:57.273+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:57 smithi112 ceph-mon[37501]: pgmap v2456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:57 smithi107 ceph-mon[32659]: pgmap v2456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:57 smithi132 ceph-mon[37580]: pgmap v2456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:51:59 smithi112 ceph-mon[37501]: pgmap v2457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:51:59 smithi107 ceph-mon[32659]: pgmap v2457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:51:59 smithi132 ceph-mon[37580]: pgmap v2457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3656.824634 secs 2024-07-26T13:52:01.210 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3656.824634 secs 2024-07-26T13:52:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3656.824634 secs 2024-07-26T13:52:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:01 smithi112 ceph-mon[37501]: pgmap v2458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:01 smithi107 ceph-mon[32659]: pgmap v2458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:01 smithi132 ceph-mon[37580]: pgmap v2458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:03 smithi112 ceph-mon[37501]: pgmap v2459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:03 smithi107 ceph-mon[32659]: pgmap v2459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:03 smithi132 ceph-mon[37580]: pgmap v2459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:05 smithi107 ceph-mon[32659]: pgmap v2460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3661.824744 secs 2024-07-26T13:52:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:05 smithi132 ceph-mon[37580]: pgmap v2460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3661.824744 secs 2024-07-26T13:52:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:05 smithi112 ceph-mon[37501]: pgmap v2460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3661.824744 secs 2024-07-26T13:52:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:07 smithi107 ceph-mon[32659]: pgmap v2461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:52:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:07 smithi132 ceph-mon[37580]: pgmap v2461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:52:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:07 smithi112 ceph-mon[37501]: pgmap v2461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:52:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:10.145+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:10.145+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:10.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:10.145+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:09 smithi107 ceph-mon[32659]: pgmap v2462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:09 smithi132 ceph-mon[37580]: pgmap v2462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:09 smithi112 ceph-mon[37501]: pgmap v2462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3666.824840 secs 2024-07-26T13:52:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3666.824840 secs 2024-07-26T13:52:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3666.824840 secs 2024-07-26T13:52:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:12.280+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:12.281 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:12.280+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:12.282 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:12.280+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:11 smithi107 ceph-mon[32659]: pgmap v2463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:11 smithi132 ceph-mon[37580]: pgmap v2463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:11 smithi112 ceph-mon[37501]: pgmap v2463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:13 smithi107 ceph-mon[32659]: pgmap v2464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:13 smithi132 ceph-mon[37580]: pgmap v2464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:13 smithi112 ceph-mon[37501]: pgmap v2464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:15 smithi107 ceph-mon[32659]: pgmap v2465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3671.824965 secs 2024-07-26T13:52:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:15 smithi132 ceph-mon[37580]: pgmap v2465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3671.824965 secs 2024-07-26T13:52:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:15 smithi112 ceph-mon[37501]: pgmap v2465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3671.824965 secs 2024-07-26T13:52:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:17 smithi107 ceph-mon[32659]: pgmap v2466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:17 smithi132 ceph-mon[37580]: pgmap v2466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:17 smithi112 ceph-mon[37501]: pgmap v2466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:19.193 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:20 smithi107 ceph-mon[32659]: pgmap v2467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:52:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:20 smithi132 ceph-mon[37580]: pgmap v2467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:52:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:20 smithi112 ceph-mon[37501]: pgmap v2467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:52:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:21 smithi107 ceph-mon[32659]: pgmap v2468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:52:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:21 smithi107 ceph-mon[32659]: pgmap v2469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:52:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3676.825107 secs 2024-07-26T13:52:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:21 smithi132 ceph-mon[37580]: pgmap v2468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:52:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:21 smithi132 ceph-mon[37580]: pgmap v2469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:52:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3676.825107 secs 2024-07-26T13:52:21.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:21 smithi112 ceph-mon[37501]: pgmap v2468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:52:21.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:21 smithi112 ceph-mon[37501]: pgmap v2469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:52:21.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3676.825107 secs 2024-07-26T13:52:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:23 smithi107 ceph-mon[32659]: pgmap v2470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:23 smithi132 ceph-mon[37580]: pgmap v2470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:23 smithi112 ceph-mon[37501]: pgmap v2470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:25.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:25.161+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:25.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:25.161+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:25.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:25.161+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:25 smithi107 ceph-mon[32659]: pgmap v2471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:52:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:25 smithi132 ceph-mon[37580]: pgmap v2471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:52:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:25 smithi112 ceph-mon[37501]: pgmap v2471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:52:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3681.825223 secs 2024-07-26T13:52:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3681.825223 secs 2024-07-26T13:52:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3681.825223 secs 2024-07-26T13:52:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:27.296+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:27.296+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:27.296+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:27 smithi132 ceph-mon[37580]: pgmap v2472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:52:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:27 smithi112 ceph-mon[37501]: pgmap v2472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:52:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:27 smithi107 ceph-mon[32659]: pgmap v2472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:52:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:29 smithi132 ceph-mon[37580]: pgmap v2473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:52:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:29 smithi112 ceph-mon[37501]: pgmap v2473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:52:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:29 smithi107 ceph-mon[32659]: pgmap v2473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:52:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:31 smithi112 ceph-mon[37501]: pgmap v2474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:52:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3686.825334 secs 2024-07-26T13:52:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:31 smithi107 ceph-mon[32659]: pgmap v2474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:52:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3686.825334 secs 2024-07-26T13:52:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:31 smithi132 ceph-mon[37580]: pgmap v2474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:52:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3686.825334 secs 2024-07-26T13:52:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:33 smithi112 ceph-mon[37501]: pgmap v2475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:33 smithi107 ceph-mon[32659]: pgmap v2475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:33 smithi132 ceph-mon[37580]: pgmap v2475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:35 smithi112 ceph-mon[37501]: pgmap v2476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:35 smithi107 ceph-mon[32659]: pgmap v2476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:35 smithi132 ceph-mon[37580]: pgmap v2476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3691.825432 secs 2024-07-26T13:52:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3691.825432 secs 2024-07-26T13:52:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3691.825432 secs 2024-07-26T13:52:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:37 smithi112 ceph-mon[37501]: pgmap v2477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:37 smithi107 ceph-mon[32659]: pgmap v2477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:37 smithi132 ceph-mon[37580]: pgmap v2477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:39 smithi112 ceph-mon[37501]: pgmap v2478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:39 smithi107 ceph-mon[32659]: pgmap v2478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:39 smithi132 ceph-mon[37580]: pgmap v2478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:40.177+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:40.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:40.177+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:40.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:40.177+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:41 smithi112 ceph-mon[37501]: pgmap v2479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3696.825561 secs 2024-07-26T13:52:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:41 smithi107 ceph-mon[32659]: pgmap v2479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3696.825561 secs 2024-07-26T13:52:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:41 smithi132 ceph-mon[37580]: pgmap v2479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3696.825561 secs 2024-07-26T13:52:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:42.312+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:42.312+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:42.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:42.312+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:43 smithi112 ceph-mon[37501]: pgmap v2480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:43 smithi107 ceph-mon[32659]: pgmap v2480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:43 smithi132 ceph-mon[37580]: pgmap v2480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:45 smithi112 ceph-mon[37501]: pgmap v2481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:45 smithi107 ceph-mon[32659]: pgmap v2481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:45 smithi132 ceph-mon[37580]: pgmap v2481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3701.825676 secs 2024-07-26T13:52:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3701.825676 secs 2024-07-26T13:52:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3701.825676 secs 2024-07-26T13:52:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:47 smithi112 ceph-mon[37501]: pgmap v2482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:47 smithi107 ceph-mon[32659]: pgmap v2482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:47 smithi132 ceph-mon[37580]: pgmap v2482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:49 smithi112 ceph-mon[37501]: pgmap v2483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:49 smithi107 ceph-mon[32659]: pgmap v2483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:49 smithi132 ceph-mon[37580]: pgmap v2483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:51 smithi112 ceph-mon[37501]: pgmap v2484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3706.825795 secs 2024-07-26T13:52:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:51 smithi107 ceph-mon[32659]: pgmap v2484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3706.825795 secs 2024-07-26T13:52:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:51 smithi132 ceph-mon[37580]: pgmap v2484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3706.825795 secs 2024-07-26T13:52:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:53 smithi112 ceph-mon[37501]: pgmap v2485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:53 smithi107 ceph-mon[32659]: pgmap v2485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:53 smithi132 ceph-mon[37580]: pgmap v2485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:52:55.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:55.191+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:55.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:55.191+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:55.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:52:55.191+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:55 smithi112 ceph-mon[37501]: pgmap v2486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:55 smithi107 ceph-mon[32659]: pgmap v2486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:55 smithi132 ceph-mon[37580]: pgmap v2486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3711.825921 secs 2024-07-26T13:52:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3711.825921 secs 2024-07-26T13:52:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3711.825921 secs 2024-07-26T13:52:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:57.327+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:57.327+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:52:57.327+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:52:57.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:57 smithi112 ceph-mon[37501]: pgmap v2487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:57 smithi107 ceph-mon[32659]: pgmap v2487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:57 smithi132 ceph-mon[37580]: pgmap v2487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:52:59 smithi112 ceph-mon[37501]: pgmap v2488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:52:59 smithi107 ceph-mon[32659]: pgmap v2488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:52:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:52:59 smithi132 ceph-mon[37580]: pgmap v2488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:01 smithi112 ceph-mon[37501]: pgmap v2489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3716.826037 secs 2024-07-26T13:53:01.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:01 smithi107 ceph-mon[32659]: pgmap v2489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:01.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3716.826037 secs 2024-07-26T13:53:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:01 smithi132 ceph-mon[37580]: pgmap v2489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3716.826037 secs 2024-07-26T13:53:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:03 smithi112 ceph-mon[37501]: pgmap v2490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:03 smithi107 ceph-mon[32659]: pgmap v2490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:03 smithi132 ceph-mon[37580]: pgmap v2490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:05 smithi112 ceph-mon[37501]: pgmap v2491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:05 smithi107 ceph-mon[32659]: pgmap v2491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:05 smithi132 ceph-mon[37580]: pgmap v2491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3721.826161 secs 2024-07-26T13:53:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3721.826161 secs 2024-07-26T13:53:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3721.826161 secs 2024-07-26T13:53:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:07 smithi107 ceph-mon[32659]: pgmap v2492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:07 smithi132 ceph-mon[37580]: pgmap v2492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:07 smithi112 ceph-mon[37501]: pgmap v2492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:09 smithi112 ceph-mon[37501]: pgmap v2493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:09 smithi107 ceph-mon[32659]: pgmap v2493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:09 smithi132 ceph-mon[37580]: pgmap v2493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:10.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:10.205+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:10.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:10.205+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:10.207 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:10.205+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:11 smithi112 ceph-mon[37501]: pgmap v2494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3726.826282 secs 2024-07-26T13:53:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:11 smithi107 ceph-mon[32659]: pgmap v2494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3726.826282 secs 2024-07-26T13:53:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:11 smithi132 ceph-mon[37580]: pgmap v2494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3726.826282 secs 2024-07-26T13:53:12.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:12.341+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:12.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:12.341+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:12.343 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:12.341+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:13 smithi112 ceph-mon[37501]: pgmap v2495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:13 smithi107 ceph-mon[32659]: pgmap v2495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:13 smithi132 ceph-mon[37580]: pgmap v2495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:15 smithi112 ceph-mon[37501]: pgmap v2496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:15 smithi107 ceph-mon[32659]: pgmap v2496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:15 smithi132 ceph-mon[37580]: pgmap v2496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3731.826407 secs 2024-07-26T13:53:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3731.826407 secs 2024-07-26T13:53:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3731.826407 secs 2024-07-26T13:53:17.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:17 smithi112 ceph-mon[37501]: pgmap v2497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:17 smithi107 ceph-mon[32659]: pgmap v2497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:17 smithi132 ceph-mon[37580]: pgmap v2497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:19 smithi112 ceph-mon[37501]: pgmap v2498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:19 smithi107 ceph-mon[32659]: pgmap v2498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:19 smithi132 ceph-mon[37580]: pgmap v2498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: pgmap v2499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3736.826502 secs 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:53:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: pgmap v2499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3736.826502 secs 2024-07-26T13:53:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:53:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: pgmap v2499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3736.826502 secs 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:53:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:22 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:53:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:22 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:53:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:22 smithi112 ceph-mon[37501]: pgmap v2500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:22 smithi112 ceph-mon[37501]: pgmap v2501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:53:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:22 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:53:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:22 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:53:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:22 smithi107 ceph-mon[32659]: pgmap v2500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:22 smithi107 ceph-mon[32659]: pgmap v2501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:53:22.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:22 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T13:53:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:22 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T13:53:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:22 smithi132 ceph-mon[37580]: pgmap v2500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:53:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:22 smithi132 ceph-mon[37580]: pgmap v2501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:53:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:24 smithi112 ceph-mon[37501]: pgmap v2502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:53:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:24 smithi107 ceph-mon[32659]: pgmap v2502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:53:24.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:24 smithi132 ceph-mon[37580]: pgmap v2502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:53:25.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:25.209+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:25.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:25.209+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:25.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:25.209+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:26 smithi112 ceph-mon[37501]: pgmap v2503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3741.826574 secs 2024-07-26T13:53:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:26 smithi107 ceph-mon[32659]: pgmap v2503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3741.826574 secs 2024-07-26T13:53:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:26 smithi132 ceph-mon[37580]: pgmap v2503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3741.826574 secs 2024-07-26T13:53:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:27.357+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:27.357+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:27.357+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:28 smithi107 ceph-mon[32659]: pgmap v2504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:28 smithi132 ceph-mon[37580]: pgmap v2504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:28 smithi112 ceph-mon[37501]: pgmap v2504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:30 smithi107 ceph-mon[32659]: pgmap v2505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:30 smithi132 ceph-mon[37580]: pgmap v2505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:30 smithi112 ceph-mon[37501]: pgmap v2505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3746.826686 secs 2024-07-26T13:53:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3746.826686 secs 2024-07-26T13:53:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3746.826686 secs 2024-07-26T13:53:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:32 smithi107 ceph-mon[32659]: pgmap v2506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:53:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:32 smithi132 ceph-mon[37580]: pgmap v2506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:53:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:32 smithi112 ceph-mon[37501]: pgmap v2506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:53:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:34 smithi107 ceph-mon[32659]: pgmap v2507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:34 smithi132 ceph-mon[37580]: pgmap v2507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:34 smithi112 ceph-mon[37501]: pgmap v2507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:36 smithi107 ceph-mon[32659]: pgmap v2508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3751.826803 secs 2024-07-26T13:53:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:36 smithi132 ceph-mon[37580]: pgmap v2508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3751.826803 secs 2024-07-26T13:53:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:36 smithi112 ceph-mon[37501]: pgmap v2508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3751.826803 secs 2024-07-26T13:53:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:38 smithi107 ceph-mon[32659]: pgmap v2509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:38 smithi132 ceph-mon[37580]: pgmap v2509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:38 smithi112 ceph-mon[37501]: pgmap v2509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:40.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:40.226+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:40.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:40.226+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:40.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:40.226+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:40 smithi107 ceph-mon[32659]: pgmap v2510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:40 smithi132 ceph-mon[37580]: pgmap v2510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:40 smithi112 ceph-mon[37501]: pgmap v2510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3756.826890 secs 2024-07-26T13:53:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3756.826890 secs 2024-07-26T13:53:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3756.826890 secs 2024-07-26T13:53:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:42.373+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:42.373+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:42.374 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:42.373+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:42 smithi107 ceph-mon[32659]: pgmap v2511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:42 smithi132 ceph-mon[37580]: pgmap v2511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:42 smithi112 ceph-mon[37501]: pgmap v2511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:44 smithi107 ceph-mon[32659]: pgmap v2512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:44 smithi132 ceph-mon[37580]: pgmap v2512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:44 smithi112 ceph-mon[37501]: pgmap v2512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:46 smithi107 ceph-mon[32659]: pgmap v2513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3761.827005 secs 2024-07-26T13:53:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:46 smithi132 ceph-mon[37580]: pgmap v2513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3761.827005 secs 2024-07-26T13:53:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:46 smithi112 ceph-mon[37501]: pgmap v2513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3761.827005 secs 2024-07-26T13:53:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:48 smithi107 ceph-mon[32659]: pgmap v2514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:48 smithi132 ceph-mon[37580]: pgmap v2514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:48 smithi112 ceph-mon[37501]: pgmap v2514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:50 smithi107 ceph-mon[32659]: pgmap v2515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:50 smithi132 ceph-mon[37580]: pgmap v2515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:50 smithi112 ceph-mon[37501]: pgmap v2515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3766.827153 secs 2024-07-26T13:53:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3766.827153 secs 2024-07-26T13:53:51.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3766.827153 secs 2024-07-26T13:53:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:52 smithi107 ceph-mon[32659]: pgmap v2516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:52 smithi132 ceph-mon[37580]: pgmap v2516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:52 smithi112 ceph-mon[37501]: pgmap v2516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:54 smithi132 ceph-mon[37580]: pgmap v2517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:54 smithi112 ceph-mon[37501]: pgmap v2517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:54 smithi107 ceph-mon[32659]: pgmap v2517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:53:55.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:55.241+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:55.242 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:55.241+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:55.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:53:55.241+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:56 smithi132 ceph-mon[37580]: pgmap v2518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3771.827274 secs 2024-07-26T13:53:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:56 smithi112 ceph-mon[37501]: pgmap v2518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3771.827274 secs 2024-07-26T13:53:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:56 smithi107 ceph-mon[32659]: pgmap v2518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3771.827274 secs 2024-07-26T13:53:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:57.383+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:57.383+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:53:57.384 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:53:57.383+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:53:58 smithi132 ceph-mon[37580]: pgmap v2519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:53:58 smithi112 ceph-mon[37501]: pgmap v2519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:53:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:53:58 smithi107 ceph-mon[32659]: pgmap v2519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:00 smithi112 ceph-mon[37501]: pgmap v2520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:00 smithi107 ceph-mon[32659]: pgmap v2520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:00 smithi132 ceph-mon[37580]: pgmap v2520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3776.827372 secs 2024-07-26T13:54:01.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3776.827372 secs 2024-07-26T13:54:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3776.827372 secs 2024-07-26T13:54:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:02 smithi112 ceph-mon[37501]: pgmap v2521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:02 smithi107 ceph-mon[32659]: pgmap v2521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:02 smithi132 ceph-mon[37580]: pgmap v2521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:04 smithi112 ceph-mon[37501]: pgmap v2522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:04 smithi107 ceph-mon[32659]: pgmap v2522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:04 smithi132 ceph-mon[37580]: pgmap v2522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:06 smithi112 ceph-mon[37501]: pgmap v2523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3781.827492 secs 2024-07-26T13:54:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:06 smithi107 ceph-mon[32659]: pgmap v2523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3781.827492 secs 2024-07-26T13:54:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:06 smithi132 ceph-mon[37580]: pgmap v2523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3781.827492 secs 2024-07-26T13:54:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:08 smithi112 ceph-mon[37501]: pgmap v2524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:08 smithi107 ceph-mon[32659]: pgmap v2524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:08 smithi132 ceph-mon[37580]: pgmap v2524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:10.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:10.256+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:10.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:10.256+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:10.258 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:10.256+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:10 smithi112 ceph-mon[37501]: pgmap v2525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:10 smithi107 ceph-mon[32659]: pgmap v2525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:10 smithi132 ceph-mon[37580]: pgmap v2525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3786.827636 secs 2024-07-26T13:54:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3786.827636 secs 2024-07-26T13:54:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3786.827636 secs 2024-07-26T13:54:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:12.398+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:12.398+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:12.398+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:12 smithi112 ceph-mon[37501]: pgmap v2526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:12 smithi107 ceph-mon[32659]: pgmap v2526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:12 smithi132 ceph-mon[37580]: pgmap v2526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:13.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:13.276+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:54:13.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:13.276+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:54:13.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:13.316+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:54:13.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:13.317+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:54:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:14.415+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:54:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:14.415+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:54:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:14.415+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:54:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:14.415+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:54:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:14 smithi112 ceph-mon[37501]: pgmap v2527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:14 smithi107 ceph-mon[32659]: pgmap v2527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:14 smithi132 ceph-mon[37580]: pgmap v2527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3791.827756 secs 2024-07-26T13:54:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:16 smithi112 ceph-mon[37501]: pgmap v2528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3791.827756 secs 2024-07-26T13:54:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:16 smithi107 ceph-mon[32659]: pgmap v2528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3791.827756 secs 2024-07-26T13:54:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:16 smithi132 ceph-mon[37580]: pgmap v2528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:18 smithi112 ceph-mon[37501]: pgmap v2529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:18 smithi107 ceph-mon[32659]: pgmap v2529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:18 smithi132 ceph-mon[37580]: pgmap v2529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:20 smithi112 ceph-mon[37501]: pgmap v2530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:20 smithi107 ceph-mon[32659]: pgmap v2530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:20 smithi132 ceph-mon[37580]: pgmap v2530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3796.827862 secs 2024-07-26T13:54:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3796.827862 secs 2024-07-26T13:54:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3796.827862 secs 2024-07-26T13:54:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: pgmap v2531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.948 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:22.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:22.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: pgmap v2531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:22.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:22.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: pgmap v2531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:54:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:23 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:54:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:23 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:54:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:23 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:54:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:23 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:54:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:23 smithi112 ceph-mon[37501]: pgmap v2532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:54:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:23 smithi112 ceph-mon[37501]: pgmap v2533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:23 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:54:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:23 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:54:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:23 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:54:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:23 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:54:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:23 smithi107 ceph-mon[32659]: pgmap v2532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:54:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:23 smithi107 ceph-mon[32659]: pgmap v2533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:23 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T13:54:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:23 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T13:54:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:23 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T13:54:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:23 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T13:54:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:23 smithi132 ceph-mon[37580]: pgmap v2532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:54:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:23 smithi132 ceph-mon[37580]: pgmap v2533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:25.269+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:25.269+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:25.269+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:25 smithi112 ceph-mon[37501]: pgmap v2534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:25 smithi107 ceph-mon[32659]: pgmap v2534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:25 smithi132 ceph-mon[37580]: pgmap v2534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3801.827978 secs 2024-07-26T13:54:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3801.827978 secs 2024-07-26T13:54:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3801.827978 secs 2024-07-26T13:54:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:27.414+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:27.414+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:27.414+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:27 smithi112 ceph-mon[37501]: pgmap v2535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:54:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:27 smithi107 ceph-mon[32659]: pgmap v2535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:54:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:27 smithi132 ceph-mon[37580]: pgmap v2535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:54:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:29 smithi112 ceph-mon[37501]: pgmap v2536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:54:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:29 smithi107 ceph-mon[32659]: pgmap v2536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:54:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:29 smithi132 ceph-mon[37580]: pgmap v2536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:54:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3806.828102 secs 2024-07-26T13:54:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:31 smithi112 ceph-mon[37501]: pgmap v2537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:54:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3806.828102 secs 2024-07-26T13:54:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:31 smithi107 ceph-mon[32659]: pgmap v2537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:54:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3806.828102 secs 2024-07-26T13:54:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:31 smithi132 ceph-mon[37580]: pgmap v2537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:54:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:33 smithi107 ceph-mon[32659]: pgmap v2538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:54:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:33 smithi132 ceph-mon[37580]: pgmap v2538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:54:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:33 smithi112 ceph-mon[37501]: pgmap v2538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:54:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:35 smithi107 ceph-mon[32659]: pgmap v2539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:54:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3811.828137 secs 2024-07-26T13:54:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:35 smithi132 ceph-mon[37580]: pgmap v2539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:54:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3811.828137 secs 2024-07-26T13:54:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:35 smithi112 ceph-mon[37501]: pgmap v2539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:54:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3811.828137 secs 2024-07-26T13:54:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:37 smithi107 ceph-mon[32659]: pgmap v2540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:54:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:37 smithi132 ceph-mon[37580]: pgmap v2540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:54:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:37 smithi112 ceph-mon[37501]: pgmap v2540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:54:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:39 smithi107 ceph-mon[32659]: pgmap v2541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:54:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:39 smithi132 ceph-mon[37580]: pgmap v2541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:54:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:39 smithi112 ceph-mon[37501]: pgmap v2541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:54:40.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:40.274+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:40.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:40.274+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:40.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:40.274+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3816.828252 secs 2024-07-26T13:54:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3816.828252 secs 2024-07-26T13:54:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3816.828252 secs 2024-07-26T13:54:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:41 smithi107 ceph-mon[32659]: pgmap v2542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:41 smithi132 ceph-mon[37580]: pgmap v2542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:41 smithi112 ceph-mon[37501]: pgmap v2542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:42.421+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:42.421+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:42.422 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:42.421+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:42 smithi107 ceph-mon[32659]: pgmap v2543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:42 smithi132 ceph-mon[37580]: pgmap v2543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:42 smithi112 ceph-mon[37501]: pgmap v2543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:45 smithi112 ceph-mon[37501]: pgmap v2544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:54:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:45 smithi107 ceph-mon[32659]: pgmap v2544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:54:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:45 smithi132 ceph-mon[37580]: pgmap v2544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:54:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3821.828370 secs 2024-07-26T13:54:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3821.828370 secs 2024-07-26T13:54:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3821.828370 secs 2024-07-26T13:54:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:47 smithi112 ceph-mon[37501]: pgmap v2545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:47 smithi107 ceph-mon[32659]: pgmap v2545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:47 smithi132 ceph-mon[37580]: pgmap v2545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:49 smithi112 ceph-mon[37501]: pgmap v2546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:49 smithi107 ceph-mon[32659]: pgmap v2546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:49 smithi132 ceph-mon[37580]: pgmap v2546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3826.828454 secs 2024-07-26T13:54:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:51 smithi107 ceph-mon[32659]: pgmap v2547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3826.828454 secs 2024-07-26T13:54:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:51 smithi132 ceph-mon[37580]: pgmap v2547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3826.828454 secs 2024-07-26T13:54:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:51 smithi112 ceph-mon[37501]: pgmap v2547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:53 smithi107 ceph-mon[32659]: pgmap v2548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:53 smithi132 ceph-mon[37580]: pgmap v2548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:53 smithi112 ceph-mon[37501]: pgmap v2548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:55.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:55.290+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:55.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:55.290+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:55.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:54:55.290+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:55 smithi107 ceph-mon[32659]: pgmap v2549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:54:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:55 smithi132 ceph-mon[37580]: pgmap v2549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:54:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:55 smithi112 ceph-mon[37501]: pgmap v2549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:54:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3831.828573 secs 2024-07-26T13:54:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3831.828573 secs 2024-07-26T13:54:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3831.828573 secs 2024-07-26T13:54:57.424 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:57.423+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:57.424+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:54:57.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:54:57.424+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:57 smithi107 ceph-mon[32659]: pgmap v2550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:57 smithi132 ceph-mon[37580]: pgmap v2550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:57 smithi112 ceph-mon[37501]: pgmap v2550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:54:59 smithi107 ceph-mon[32659]: pgmap v2551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:54:59 smithi132 ceph-mon[37580]: pgmap v2551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:54:59 smithi112 ceph-mon[37501]: pgmap v2551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3836.828695 secs 2024-07-26T13:55:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:01 smithi107 ceph-mon[32659]: pgmap v2552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3836.828695 secs 2024-07-26T13:55:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:01 smithi132 ceph-mon[37580]: pgmap v2552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3836.828695 secs 2024-07-26T13:55:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:01 smithi112 ceph-mon[37501]: pgmap v2552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:03 smithi107 ceph-mon[32659]: pgmap v2553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:03 smithi132 ceph-mon[37580]: pgmap v2553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:03 smithi112 ceph-mon[37501]: pgmap v2553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:05 smithi107 ceph-mon[32659]: pgmap v2554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:55:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:05 smithi132 ceph-mon[37580]: pgmap v2554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:55:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:05 smithi112 ceph-mon[37501]: pgmap v2554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:55:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3841.828813 secs 2024-07-26T13:55:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3841.828813 secs 2024-07-26T13:55:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3841.828813 secs 2024-07-26T13:55:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:07 smithi107 ceph-mon[32659]: pgmap v2555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:55:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:07 smithi132 ceph-mon[37580]: pgmap v2555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:55:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:07 smithi112 ceph-mon[37501]: pgmap v2555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:55:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:09 smithi107 ceph-mon[32659]: pgmap v2556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:09 smithi132 ceph-mon[37580]: pgmap v2556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:09 smithi112 ceph-mon[37501]: pgmap v2556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:10.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:10.305+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:10.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:10.305+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:10.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:10.305+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3846.828965 secs 2024-07-26T13:55:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:11 smithi107 ceph-mon[32659]: pgmap v2557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3846.828965 secs 2024-07-26T13:55:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:11 smithi132 ceph-mon[37580]: pgmap v2557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3846.828965 secs 2024-07-26T13:55:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:11 smithi112 ceph-mon[37501]: pgmap v2557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:12.438+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:12.438+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:12.438+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:13 smithi107 ceph-mon[32659]: pgmap v2558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:13 smithi132 ceph-mon[37580]: pgmap v2558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:13 smithi112 ceph-mon[37501]: pgmap v2558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:15 smithi107 ceph-mon[32659]: pgmap v2559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:55:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:15 smithi132 ceph-mon[37580]: pgmap v2559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:55:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:15 smithi112 ceph-mon[37501]: pgmap v2559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:55:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3851.829104 secs 2024-07-26T13:55:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3851.829104 secs 2024-07-26T13:55:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3851.829104 secs 2024-07-26T13:55:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:17 smithi132 ceph-mon[37580]: pgmap v2560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:55:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:17 smithi112 ceph-mon[37501]: pgmap v2560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:55:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:17 smithi107 ceph-mon[32659]: pgmap v2560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:55:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:19 smithi112 ceph-mon[37501]: pgmap v2561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:19 smithi107 ceph-mon[32659]: pgmap v2561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:19 smithi132 ceph-mon[37580]: pgmap v2561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3856.829199 secs 2024-07-26T13:55:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:21 smithi112 ceph-mon[37501]: pgmap v2562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3856.829199 secs 2024-07-26T13:55:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:21 smithi107 ceph-mon[32659]: pgmap v2562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3856.829199 secs 2024-07-26T13:55:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:21 smithi132 ceph-mon[37580]: pgmap v2562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:23 smithi112 ceph-mon[37501]: pgmap v2563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:23 smithi107 ceph-mon[32659]: pgmap v2563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:23 smithi132 ceph-mon[37580]: pgmap v2563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:24 smithi112 ceph-mon[37501]: pgmap v2564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:55:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:24 smithi112 ceph-mon[37501]: pgmap v2565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:55:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:24 smithi107 ceph-mon[32659]: pgmap v2564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:55:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:24 smithi107 ceph-mon[32659]: pgmap v2565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:55:25.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:24 smithi132 ceph-mon[37580]: pgmap v2564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:55:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:24 smithi132 ceph-mon[37580]: pgmap v2565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:55:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:55:25.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:25.306+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:25.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:25.306+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:25.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:25.306+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3861.829316 secs 2024-07-26T13:55:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:26 smithi112 ceph-mon[37501]: pgmap v2566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:55:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3861.829316 secs 2024-07-26T13:55:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:26 smithi107 ceph-mon[32659]: pgmap v2566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:55:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3861.829316 secs 2024-07-26T13:55:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:26 smithi132 ceph-mon[37580]: pgmap v2566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:55:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:27.449+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:27.449+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:27.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:27.449+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:28 smithi112 ceph-mon[37501]: pgmap v2567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:55:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:28 smithi107 ceph-mon[32659]: pgmap v2567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:55:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:28 smithi132 ceph-mon[37580]: pgmap v2567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:55:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:30 smithi112 ceph-mon[37501]: pgmap v2568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:55:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:30 smithi107 ceph-mon[32659]: pgmap v2568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:55:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:30 smithi132 ceph-mon[37580]: pgmap v2568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:55:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3866.829437 secs 2024-07-26T13:55:31.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3866.829437 secs 2024-07-26T13:55:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3866.829437 secs 2024-07-26T13:55:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:32 smithi112 ceph-mon[37501]: pgmap v2569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:32 smithi107 ceph-mon[32659]: pgmap v2569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:32 smithi132 ceph-mon[37580]: pgmap v2569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:34 smithi112 ceph-mon[37501]: pgmap v2570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:55:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:34 smithi107 ceph-mon[32659]: pgmap v2570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:55:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:34 smithi132 ceph-mon[37580]: pgmap v2570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:55:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3871.829555 secs 2024-07-26T13:55:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:36 smithi112 ceph-mon[37501]: pgmap v2571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:55:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3871.829555 secs 2024-07-26T13:55:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:36 smithi107 ceph-mon[32659]: pgmap v2571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:55:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3871.829555 secs 2024-07-26T13:55:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:36 smithi132 ceph-mon[37580]: pgmap v2571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:55:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:38 smithi107 ceph-mon[32659]: pgmap v2572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:55:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:38 smithi132 ceph-mon[37580]: pgmap v2572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:55:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:38 smithi112 ceph-mon[37501]: pgmap v2572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:55:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:40.322+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:40.322+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:40.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:40.322+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:40 smithi107 ceph-mon[32659]: pgmap v2573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:40 smithi132 ceph-mon[37580]: pgmap v2573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:40 smithi112 ceph-mon[37501]: pgmap v2573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3876.829637 secs 2024-07-26T13:55:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3876.829637 secs 2024-07-26T13:55:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3876.829637 secs 2024-07-26T13:55:42.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:42.452+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:42.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:42.452+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:42.453 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:42.452+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:42 smithi107 ceph-mon[32659]: pgmap v2574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:55:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:42 smithi132 ceph-mon[37580]: pgmap v2574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:55:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:42 smithi112 ceph-mon[37501]: pgmap v2574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:55:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:44 smithi107 ceph-mon[32659]: pgmap v2575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:55:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:44 smithi132 ceph-mon[37580]: pgmap v2575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:55:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:44 smithi112 ceph-mon[37501]: pgmap v2575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:55:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3881.829716 secs 2024-07-26T13:55:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3881.829716 secs 2024-07-26T13:55:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3881.829716 secs 2024-07-26T13:55:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:46 smithi107 ceph-mon[32659]: pgmap v2576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:46 smithi132 ceph-mon[37580]: pgmap v2576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:46 smithi112 ceph-mon[37501]: pgmap v2576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:47 smithi132 ceph-mon[37580]: pgmap v2577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:47 smithi112 ceph-mon[37501]: pgmap v2577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:47 smithi107 ceph-mon[32659]: pgmap v2577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:50 smithi112 ceph-mon[37501]: pgmap v2578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:50 smithi107 ceph-mon[32659]: pgmap v2578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:50 smithi132 ceph-mon[37580]: pgmap v2578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3886.829851 secs 2024-07-26T13:55:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3886.829851 secs 2024-07-26T13:55:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3886.829851 secs 2024-07-26T13:55:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:52 smithi112 ceph-mon[37501]: pgmap v2579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:52 smithi107 ceph-mon[32659]: pgmap v2579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:52 smithi132 ceph-mon[37580]: pgmap v2579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:54 smithi112 ceph-mon[37501]: pgmap v2580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:54 smithi107 ceph-mon[32659]: pgmap v2580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:54 smithi132 ceph-mon[37580]: pgmap v2580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:55.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:55.338+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:55.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:55.338+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:55.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:55:55.338+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3891.829966 secs 2024-07-26T13:55:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:56 smithi112 ceph-mon[37501]: pgmap v2581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3891.829966 secs 2024-07-26T13:55:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:56 smithi107 ceph-mon[32659]: pgmap v2581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3891.829966 secs 2024-07-26T13:55:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:56 smithi132 ceph-mon[37580]: pgmap v2581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:57.461+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:57.462+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:55:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:55:57.462+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:55:58 smithi107 ceph-mon[32659]: pgmap v2582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:55:58 smithi132 ceph-mon[37580]: pgmap v2582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:55:58 smithi112 ceph-mon[37501]: pgmap v2582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:00 smithi107 ceph-mon[32659]: pgmap v2583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:00 smithi132 ceph-mon[37580]: pgmap v2583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:00 smithi112 ceph-mon[37501]: pgmap v2583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3896.830100 secs 2024-07-26T13:56:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3896.830100 secs 2024-07-26T13:56:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3896.830100 secs 2024-07-26T13:56:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:02 smithi107 ceph-mon[32659]: pgmap v2584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:02 smithi132 ceph-mon[37580]: pgmap v2584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:02 smithi112 ceph-mon[37501]: pgmap v2584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:04 smithi107 ceph-mon[32659]: pgmap v2585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:04 smithi132 ceph-mon[37580]: pgmap v2585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:04 smithi112 ceph-mon[37501]: pgmap v2585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3901.830233 secs 2024-07-26T13:56:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3901.830233 secs 2024-07-26T13:56:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3901.830233 secs 2024-07-26T13:56:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:06 smithi107 ceph-mon[32659]: pgmap v2586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:07.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:06 smithi132 ceph-mon[37580]: pgmap v2586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:06 smithi112 ceph-mon[37501]: pgmap v2586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:08 smithi107 ceph-mon[32659]: pgmap v2587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:08 smithi132 ceph-mon[37580]: pgmap v2587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:08 smithi112 ceph-mon[37501]: pgmap v2587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:09 smithi107 ceph-mon[32659]: pgmap v2588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:09 smithi132 ceph-mon[37580]: pgmap v2588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:09 smithi112 ceph-mon[37501]: pgmap v2588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:10.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:10.352+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:10.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:10.352+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:10.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:10.352+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3906.830346 secs 2024-07-26T13:56:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3906.830346 secs 2024-07-26T13:56:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3906.830346 secs 2024-07-26T13:56:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:11 smithi107 ceph-mon[32659]: pgmap v2589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:11 smithi132 ceph-mon[37580]: pgmap v2589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:11 smithi112 ceph-mon[37501]: pgmap v2589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:12.476+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:12.476+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:12.478 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:12.476+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:14 smithi112 ceph-mon[37501]: pgmap v2590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:14 smithi107 ceph-mon[32659]: pgmap v2590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:14 smithi132 ceph-mon[37580]: pgmap v2590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3911.830483 secs 2024-07-26T13:56:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:16 smithi112 ceph-mon[37501]: pgmap v2591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3911.830483 secs 2024-07-26T13:56:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:16 smithi107 ceph-mon[32659]: pgmap v2591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3911.830483 secs 2024-07-26T13:56:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:16 smithi132 ceph-mon[37580]: pgmap v2591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:18 smithi107 ceph-mon[32659]: pgmap v2592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:18 smithi132 ceph-mon[37580]: pgmap v2592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:18 smithi112 ceph-mon[37501]: pgmap v2592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:21.002 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:20 smithi112 ceph-mon[37501]: pgmap v2593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:20 smithi107 ceph-mon[32659]: pgmap v2593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:20 smithi132 ceph-mon[37580]: pgmap v2593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3916.830560 secs 2024-07-26T13:56:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3916.830560 secs 2024-07-26T13:56:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3916.830560 secs 2024-07-26T13:56:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:22 smithi107 ceph-mon[32659]: pgmap v2594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:22 smithi132 ceph-mon[37580]: pgmap v2594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:22 smithi112 ceph-mon[37501]: pgmap v2594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:24 smithi107 ceph-mon[32659]: pgmap v2595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:24 smithi132 ceph-mon[37580]: pgmap v2595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:24 smithi112 ceph-mon[37501]: pgmap v2595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:25.368+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:25.368+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:25.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:25.368+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3921.830674 secs 2024-07-26T13:56:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3921.830674 secs 2024-07-26T13:56:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3921.830674 secs 2024-07-26T13:56:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:56:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:26 smithi107 ceph-mon[32659]: pgmap v2596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:56:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:26 smithi107 ceph-mon[32659]: pgmap v2597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:26 smithi132 ceph-mon[37580]: pgmap v2596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:56:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:26 smithi132 ceph-mon[37580]: pgmap v2597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:26 smithi112 ceph-mon[37501]: pgmap v2596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:56:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:26 smithi112 ceph-mon[37501]: pgmap v2597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:27.481 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:27.479+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:27.481 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:27.479+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:27.481 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:27.479+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:28 smithi107 ceph-mon[32659]: pgmap v2598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:56:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:28 smithi132 ceph-mon[37580]: pgmap v2598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:56:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:28 smithi112 ceph-mon[37501]: pgmap v2598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:56:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:30 smithi107 ceph-mon[32659]: pgmap v2599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3926.830795 secs 2024-07-26T13:56:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:30 smithi132 ceph-mon[37580]: pgmap v2599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3926.830795 secs 2024-07-26T13:56:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:30 smithi112 ceph-mon[37501]: pgmap v2599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3926.830795 secs 2024-07-26T13:56:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:31 smithi107 ceph-mon[32659]: pgmap v2600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:31 smithi132 ceph-mon[37580]: pgmap v2600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:31 smithi112 ceph-mon[37501]: pgmap v2600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:34 smithi112 ceph-mon[37501]: pgmap v2601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:34 smithi107 ceph-mon[32659]: pgmap v2601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:34 smithi132 ceph-mon[37580]: pgmap v2601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:56:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3931.830879 secs 2024-07-26T13:56:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:36 smithi112 ceph-mon[37501]: pgmap v2602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:56:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3931.830879 secs 2024-07-26T13:56:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:36 smithi107 ceph-mon[32659]: pgmap v2602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:56:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3931.830879 secs 2024-07-26T13:56:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:36 smithi132 ceph-mon[37580]: pgmap v2602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:56:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:38 smithi112 ceph-mon[37501]: pgmap v2603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:38 smithi107 ceph-mon[32659]: pgmap v2603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:38 smithi132 ceph-mon[37580]: pgmap v2603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:40.382+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:40.382+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:40.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:40.382+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:40 smithi107 ceph-mon[32659]: pgmap v2604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:56:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:40 smithi112 ceph-mon[37501]: pgmap v2604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:56:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:40 smithi132 ceph-mon[37580]: pgmap v2604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:56:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3936.831027 secs 2024-07-26T13:56:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3936.831027 secs 2024-07-26T13:56:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3936.831027 secs 2024-07-26T13:56:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:42.495+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:42.495+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:42.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:42.495+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:42 smithi112 ceph-mon[37501]: pgmap v2605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:42 smithi107 ceph-mon[32659]: pgmap v2605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:42 smithi132 ceph-mon[37580]: pgmap v2605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:44 smithi112 ceph-mon[37501]: pgmap v2606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:44 smithi107 ceph-mon[32659]: pgmap v2606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:44 smithi132 ceph-mon[37580]: pgmap v2606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3941.831154 secs 2024-07-26T13:56:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:46 smithi112 ceph-mon[37501]: pgmap v2607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3941.831154 secs 2024-07-26T13:56:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:46 smithi107 ceph-mon[32659]: pgmap v2607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3941.831154 secs 2024-07-26T13:56:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:46 smithi132 ceph-mon[37580]: pgmap v2607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:48 smithi112 ceph-mon[37501]: pgmap v2608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:48 smithi107 ceph-mon[32659]: pgmap v2608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:48 smithi132 ceph-mon[37580]: pgmap v2608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:50 smithi112 ceph-mon[37501]: pgmap v2609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:56:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:50 smithi107 ceph-mon[32659]: pgmap v2609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:56:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:50 smithi132 ceph-mon[37580]: pgmap v2609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:56:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3946.831277 secs 2024-07-26T13:56:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3946.831277 secs 2024-07-26T13:56:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3946.831277 secs 2024-07-26T13:56:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:52 smithi112 ceph-mon[37501]: pgmap v2610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:52 smithi107 ceph-mon[32659]: pgmap v2610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:52 smithi132 ceph-mon[37580]: pgmap v2610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:54 smithi112 ceph-mon[37501]: pgmap v2611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:54 smithi107 ceph-mon[32659]: pgmap v2611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:54 smithi132 ceph-mon[37580]: pgmap v2611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:55.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:55.398+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:55.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:55.398+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:55.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:56:55.398+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3951.831371 secs 2024-07-26T13:56:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:56 smithi112 ceph-mon[37501]: pgmap v2612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3951.831371 secs 2024-07-26T13:56:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:56 smithi107 ceph-mon[32659]: pgmap v2612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3951.831371 secs 2024-07-26T13:56:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:56 smithi132 ceph-mon[37580]: pgmap v2612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:57.511 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:57.511+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:57.511+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:56:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:56:57.511+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:56:58 smithi112 ceph-mon[37501]: pgmap v2613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:56:58 smithi107 ceph-mon[32659]: pgmap v2613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:56:58 smithi132 ceph-mon[37580]: pgmap v2613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:00 smithi112 ceph-mon[37501]: pgmap v2614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:00 smithi107 ceph-mon[32659]: pgmap v2614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:00 smithi132 ceph-mon[37580]: pgmap v2614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3956.831497 secs 2024-07-26T13:57:01.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3956.831497 secs 2024-07-26T13:57:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3956.831497 secs 2024-07-26T13:57:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:02 smithi112 ceph-mon[37501]: pgmap v2615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:02 smithi107 ceph-mon[32659]: pgmap v2615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:02 smithi132 ceph-mon[37580]: pgmap v2615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:04 smithi112 ceph-mon[37501]: pgmap v2616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:04 smithi107 ceph-mon[32659]: pgmap v2616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:04 smithi132 ceph-mon[37580]: pgmap v2616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3961.831618 secs 2024-07-26T13:57:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:06 smithi112 ceph-mon[37501]: pgmap v2617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3961.831618 secs 2024-07-26T13:57:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:06 smithi107 ceph-mon[32659]: pgmap v2617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3961.831618 secs 2024-07-26T13:57:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:06 smithi132 ceph-mon[37580]: pgmap v2617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:08 smithi112 ceph-mon[37501]: pgmap v2618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:08 smithi107 ceph-mon[32659]: pgmap v2618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:08 smithi132 ceph-mon[37580]: pgmap v2618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:10.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:10.412+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:10.412+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:10.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:10.412+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:10 smithi112 ceph-mon[37501]: pgmap v2619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:10 smithi107 ceph-mon[32659]: pgmap v2619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:10 smithi132 ceph-mon[37580]: pgmap v2619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3966.831735 secs 2024-07-26T13:57:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3966.831735 secs 2024-07-26T13:57:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3966.831735 secs 2024-07-26T13:57:12.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:12.305+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:57:12.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:12.306+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:57:12.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:12.345+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:57:12.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:12.346+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:57:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:12.525+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:12.525+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:12.527 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:12.525+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:12 smithi107 ceph-mon[32659]: pgmap v2620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:12 smithi132 ceph-mon[37580]: pgmap v2620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:12 smithi112 ceph-mon[37501]: pgmap v2620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:13.305+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:57:13.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:13.306+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T13:57:13.347 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:13.346+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:57:13.348 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:13.346+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T13:57:14.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:14.447+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:57:14.448 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:14.447+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:57:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:14.447+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T13:57:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:14.447+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T13:57:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:14.447+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T13:57:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:14.447+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:57:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:14.448+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T13:57:14.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:14.448+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T13:57:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:14 smithi107 ceph-mon[32659]: pgmap v2621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:14 smithi132 ceph-mon[37580]: pgmap v2621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:14 smithi112 ceph-mon[37501]: pgmap v2621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3971.831828 secs 2024-07-26T13:57:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:16 smithi112 ceph-mon[37501]: pgmap v2622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3971.831828 secs 2024-07-26T13:57:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:16 smithi107 ceph-mon[32659]: pgmap v2622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3971.831828 secs 2024-07-26T13:57:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:16 smithi132 ceph-mon[37580]: pgmap v2622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:18 smithi112 ceph-mon[37501]: pgmap v2623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:18 smithi107 ceph-mon[32659]: pgmap v2623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:18 smithi132 ceph-mon[37580]: pgmap v2623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:20 smithi112 ceph-mon[37501]: pgmap v2624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:20 smithi107 ceph-mon[32659]: pgmap v2624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:20 smithi132 ceph-mon[37580]: pgmap v2624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3976.831954 secs 2024-07-26T13:57:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3976.831954 secs 2024-07-26T13:57:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3976.831954 secs 2024-07-26T13:57:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:22 smithi112 ceph-mon[37501]: pgmap v2625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:22 smithi107 ceph-mon[32659]: pgmap v2625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:22 smithi132 ceph-mon[37580]: pgmap v2625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:24 smithi112 ceph-mon[37501]: pgmap v2626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:24 smithi107 ceph-mon[32659]: pgmap v2626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:24 smithi132 ceph-mon[37580]: pgmap v2626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:25.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:25.427+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:25.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:25.427+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:25.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:25.427+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:26.905 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3981.832093 secs 2024-07-26T13:57:26.905 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:26 smithi112 ceph-mon[37501]: pgmap v2627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:26.905 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:26.905 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.905 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.905 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.905 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.951 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3981.832093 secs 2024-07-26T13:57:26.951 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:26 smithi132 ceph-mon[37580]: pgmap v2627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:26.952 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:26.952 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.952 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.952 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.952 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.978 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3981.832093 secs 2024-07-26T13:57:26.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:26 smithi107 ceph-mon[32659]: pgmap v2627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:26.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:26.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:26.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:27.540+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:27.540+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:27.540+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:28 smithi132 ceph-mon[37580]: pgmap v2628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:28 smithi112 ceph-mon[37501]: pgmap v2628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:28 smithi107 ceph-mon[32659]: pgmap v2628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:30.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:30.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:30.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:30 smithi107 ceph-mon[32659]: pgmap v2629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:30 smithi112 ceph-mon[37501]: pgmap v2629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:30 smithi132 ceph-mon[37580]: pgmap v2629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:57:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3986.832230 secs 2024-07-26T13:57:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:31 smithi112 ceph-mon[37501]: pgmap v2630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:57:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:31 smithi112 ceph-mon[37501]: pgmap v2631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:57:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:31.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3986.832230 secs 2024-07-26T13:57:31.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:31.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:31.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:31 smithi132 ceph-mon[37580]: pgmap v2630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:57:31.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:31 smithi132 ceph-mon[37580]: pgmap v2631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:57:31.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3986.832230 secs 2024-07-26T13:57:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:31 smithi107 ceph-mon[32659]: pgmap v2630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:57:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:31 smithi107 ceph-mon[32659]: pgmap v2631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:57:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:57:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:33 smithi107 ceph-mon[32659]: pgmap v2632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T13:57:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:33 smithi132 ceph-mon[37580]: pgmap v2632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T13:57:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:33 smithi112 ceph-mon[37501]: pgmap v2632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T13:57:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:35 smithi107 ceph-mon[32659]: pgmap v2633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:35 smithi132 ceph-mon[37580]: pgmap v2633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:35 smithi112 ceph-mon[37501]: pgmap v2633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3991.832345 secs 2024-07-26T13:57:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3991.832345 secs 2024-07-26T13:57:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3991.832345 secs 2024-07-26T13:57:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:37 smithi107 ceph-mon[32659]: pgmap v2634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T13:57:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:37 smithi132 ceph-mon[37580]: pgmap v2634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T13:57:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:37 smithi112 ceph-mon[37501]: pgmap v2634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T13:57:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:39 smithi107 ceph-mon[32659]: pgmap v2635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:39 smithi132 ceph-mon[37580]: pgmap v2635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:39 smithi112 ceph-mon[37501]: pgmap v2635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:40.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:40.442+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:40.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:40.442+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:40.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:40.442+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 3996.832465 secs 2024-07-26T13:57:41.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:41 smithi107 ceph-mon[32659]: pgmap v2636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:57:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 3996.832465 secs 2024-07-26T13:57:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:41 smithi132 ceph-mon[37580]: pgmap v2636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:57:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 3996.832465 secs 2024-07-26T13:57:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:41 smithi112 ceph-mon[37501]: pgmap v2636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:57:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:42.557+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:42.557+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:42.557+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:43 smithi107 ceph-mon[32659]: pgmap v2637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:43 smithi132 ceph-mon[37580]: pgmap v2637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:43 smithi112 ceph-mon[37501]: pgmap v2637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:45 smithi107 ceph-mon[32659]: pgmap v2638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:45 smithi132 ceph-mon[37580]: pgmap v2638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:45 smithi112 ceph-mon[37501]: pgmap v2638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4001.832583 secs 2024-07-26T13:57:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4001.832583 secs 2024-07-26T13:57:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4001.832583 secs 2024-07-26T13:57:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:47 smithi132 ceph-mon[37580]: pgmap v2639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:47 smithi112 ceph-mon[37501]: pgmap v2639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:47 smithi107 ceph-mon[32659]: pgmap v2639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:49 smithi112 ceph-mon[37501]: pgmap v2640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:49 smithi107 ceph-mon[32659]: pgmap v2640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:49 smithi132 ceph-mon[37580]: pgmap v2640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4006.832704 secs 2024-07-26T13:57:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:51 smithi112 ceph-mon[37501]: pgmap v2641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:51.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4006.832704 secs 2024-07-26T13:57:51.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:51 smithi107 ceph-mon[32659]: pgmap v2641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4006.832704 secs 2024-07-26T13:57:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:51 smithi132 ceph-mon[37580]: pgmap v2641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:53 smithi112 ceph-mon[37501]: pgmap v2642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:53 smithi107 ceph-mon[32659]: pgmap v2642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:53 smithi132 ceph-mon[37580]: pgmap v2642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:55.454+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:55.454+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:55.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:57:55.454+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:55 smithi112 ceph-mon[37501]: pgmap v2643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:55 smithi107 ceph-mon[32659]: pgmap v2643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:55 smithi132 ceph-mon[37580]: pgmap v2643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4011.832782 secs 2024-07-26T13:57:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4011.832782 secs 2024-07-26T13:57:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4011.832782 secs 2024-07-26T13:57:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:57.562+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:57.562+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:57:57.562+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:57:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:57 smithi112 ceph-mon[37501]: pgmap v2644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:57 smithi107 ceph-mon[32659]: pgmap v2644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:57 smithi132 ceph-mon[37580]: pgmap v2644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:57:59 smithi112 ceph-mon[37501]: pgmap v2645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:57:59 smithi107 ceph-mon[32659]: pgmap v2645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:57:59 smithi132 ceph-mon[37580]: pgmap v2645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:01.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4016.832928 secs 2024-07-26T13:58:01.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:01 smithi132 ceph-mon[37580]: pgmap v2646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4016.832928 secs 2024-07-26T13:58:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:01 smithi112 ceph-mon[37501]: pgmap v2646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4016.832928 secs 2024-07-26T13:58:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:01 smithi107 ceph-mon[32659]: pgmap v2646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:03 smithi112 ceph-mon[37501]: pgmap v2647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:03 smithi107 ceph-mon[32659]: pgmap v2647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:03 smithi132 ceph-mon[37580]: pgmap v2647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:05 smithi112 ceph-mon[37501]: pgmap v2648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:05 smithi107 ceph-mon[32659]: pgmap v2648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:05 smithi132 ceph-mon[37580]: pgmap v2648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4021.833004 secs 2024-07-26T13:58:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4021.833004 secs 2024-07-26T13:58:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4021.833004 secs 2024-07-26T13:58:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:07 smithi112 ceph-mon[37501]: pgmap v2649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:07 smithi107 ceph-mon[32659]: pgmap v2649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:07 smithi132 ceph-mon[37580]: pgmap v2649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:09 smithi107 ceph-mon[32659]: pgmap v2650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:09 smithi132 ceph-mon[37580]: pgmap v2650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:09 smithi112 ceph-mon[37501]: pgmap v2650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:10.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:10.455+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:10.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:10.455+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:10.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:10.455+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4026.833108 secs 2024-07-26T13:58:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:11 smithi107 ceph-mon[32659]: pgmap v2651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4026.833108 secs 2024-07-26T13:58:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:11 smithi132 ceph-mon[37580]: pgmap v2651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4026.833108 secs 2024-07-26T13:58:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:11 smithi112 ceph-mon[37501]: pgmap v2651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:12.567 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:12.566+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:12.566+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:12.568 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:12.566+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:13 smithi107 ceph-mon[32659]: pgmap v2652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:13 smithi132 ceph-mon[37580]: pgmap v2652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:13 smithi112 ceph-mon[37501]: pgmap v2652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:15 smithi107 ceph-mon[32659]: pgmap v2653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:15 smithi132 ceph-mon[37580]: pgmap v2653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:15 smithi112 ceph-mon[37501]: pgmap v2653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4031.833224 secs 2024-07-26T13:58:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4031.833224 secs 2024-07-26T13:58:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4031.833224 secs 2024-07-26T13:58:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:17 smithi107 ceph-mon[32659]: pgmap v2654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:17 smithi132 ceph-mon[37580]: pgmap v2654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:17 smithi112 ceph-mon[37501]: pgmap v2654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:19 smithi107 ceph-mon[32659]: pgmap v2655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:19 smithi132 ceph-mon[37580]: pgmap v2655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:19 smithi112 ceph-mon[37501]: pgmap v2655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4036.833343 secs 2024-07-26T13:58:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:21 smithi132 ceph-mon[37580]: pgmap v2656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4036.833343 secs 2024-07-26T13:58:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:21 smithi112 ceph-mon[37501]: pgmap v2656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4036.833343 secs 2024-07-26T13:58:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:21 smithi107 ceph-mon[32659]: pgmap v2656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:23 smithi132 ceph-mon[37580]: pgmap v2657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:23 smithi112 ceph-mon[37501]: pgmap v2657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:23 smithi107 ceph-mon[32659]: pgmap v2657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:25.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:25.459+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:25.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:25.459+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:25.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:25.459+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:25 smithi107 ceph-mon[32659]: pgmap v2658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:25 smithi132 ceph-mon[37580]: pgmap v2658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:25 smithi112 ceph-mon[37501]: pgmap v2658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4041.833488 secs 2024-07-26T13:58:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4041.833488 secs 2024-07-26T13:58:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4041.833488 secs 2024-07-26T13:58:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:27.581+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:27.581+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:27.583 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:27.581+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:27 smithi112 ceph-mon[37501]: pgmap v2659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:27 smithi107 ceph-mon[32659]: pgmap v2659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:27 smithi132 ceph-mon[37580]: pgmap v2659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:29 smithi112 ceph-mon[37501]: pgmap v2660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:29 smithi107 ceph-mon[32659]: pgmap v2660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:29 smithi132 ceph-mon[37580]: pgmap v2660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4046.833614 secs 2024-07-26T13:58:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:31 smithi112 ceph-mon[37501]: pgmap v2661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:58:31.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4046.833614 secs 2024-07-26T13:58:31.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:31 smithi132 ceph-mon[37580]: pgmap v2661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:31.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:31.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:31.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:58:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4046.833614 secs 2024-07-26T13:58:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:31 smithi107 ceph-mon[32659]: pgmap v2661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:58:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:32 smithi112 ceph-mon[37501]: pgmap v2662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:58:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:32 smithi112 ceph-mon[37501]: pgmap v2663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:58:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:32 smithi107 ceph-mon[32659]: pgmap v2662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:58:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:32 smithi107 ceph-mon[32659]: pgmap v2663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:58:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:32 smithi132 ceph-mon[37580]: pgmap v2662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:58:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:32 smithi132 ceph-mon[37580]: pgmap v2663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:58:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:34 smithi112 ceph-mon[37501]: pgmap v2664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:34 smithi107 ceph-mon[32659]: pgmap v2664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:34 smithi132 ceph-mon[37580]: pgmap v2664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4051.833730 secs 2024-07-26T13:58:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:36 smithi112 ceph-mon[37501]: pgmap v2665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4051.833730 secs 2024-07-26T13:58:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:36 smithi107 ceph-mon[32659]: pgmap v2665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4051.833730 secs 2024-07-26T13:58:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:36 smithi132 ceph-mon[37580]: pgmap v2665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:38 smithi112 ceph-mon[37501]: pgmap v2666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:38 smithi107 ceph-mon[32659]: pgmap v2666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:38 smithi132 ceph-mon[37580]: pgmap v2666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:40.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:40.475+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:40.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:40.475+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:40.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:40.475+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:40 smithi107 ceph-mon[32659]: pgmap v2667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:40 smithi112 ceph-mon[37501]: pgmap v2667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:40 smithi132 ceph-mon[37580]: pgmap v2667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:58:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4056.833853 secs 2024-07-26T13:58:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4056.833853 secs 2024-07-26T13:58:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4056.833853 secs 2024-07-26T13:58:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:42.598+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:42.598+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:42.598+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:42 smithi112 ceph-mon[37501]: pgmap v2668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:58:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:42 smithi107 ceph-mon[32659]: pgmap v2668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:58:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:42 smithi132 ceph-mon[37580]: pgmap v2668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:58:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:44 smithi112 ceph-mon[37501]: pgmap v2669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:58:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:44 smithi107 ceph-mon[32659]: pgmap v2669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:58:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:44 smithi132 ceph-mon[37580]: pgmap v2669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:58:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4061.833970 secs 2024-07-26T13:58:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:46 smithi112 ceph-mon[37501]: pgmap v2670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4061.833970 secs 2024-07-26T13:58:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:46 smithi107 ceph-mon[32659]: pgmap v2670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4061.833970 secs 2024-07-26T13:58:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:46 smithi132 ceph-mon[37580]: pgmap v2670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:48 smithi112 ceph-mon[37501]: pgmap v2671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:48 smithi107 ceph-mon[32659]: pgmap v2671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:48 smithi132 ceph-mon[37580]: pgmap v2671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:50 smithi112 ceph-mon[37501]: pgmap v2672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:58:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:50 smithi107 ceph-mon[32659]: pgmap v2672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:58:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:50 smithi132 ceph-mon[37580]: pgmap v2672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:58:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4066.834108 secs 2024-07-26T13:58:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4066.834108 secs 2024-07-26T13:58:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4066.834108 secs 2024-07-26T13:58:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:52 smithi112 ceph-mon[37501]: pgmap v2673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:58:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:52 smithi107 ceph-mon[32659]: pgmap v2673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:58:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:52 smithi132 ceph-mon[37580]: pgmap v2673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:58:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:54 smithi112 ceph-mon[37501]: pgmap v2674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:54 smithi107 ceph-mon[32659]: pgmap v2674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:54 smithi132 ceph-mon[37580]: pgmap v2674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:55.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:55.489+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:55.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:55.489+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:55.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:58:55.489+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4071.834257 secs 2024-07-26T13:58:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:56 smithi112 ceph-mon[37501]: pgmap v2675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4071.834257 secs 2024-07-26T13:58:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:56 smithi107 ceph-mon[32659]: pgmap v2675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4071.834257 secs 2024-07-26T13:58:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:56 smithi132 ceph-mon[37580]: pgmap v2675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:57.613+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:57.614+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:58:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:58:57.614+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:58:58 smithi112 ceph-mon[37501]: pgmap v2676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:58:58 smithi107 ceph-mon[32659]: pgmap v2676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:58:58 smithi132 ceph-mon[37580]: pgmap v2676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:00 smithi107 ceph-mon[32659]: pgmap v2677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:00 smithi132 ceph-mon[37580]: pgmap v2677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:00 smithi112 ceph-mon[37501]: pgmap v2677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:01.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4076.834354 secs 2024-07-26T13:59:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4076.834354 secs 2024-07-26T13:59:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4076.834354 secs 2024-07-26T13:59:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:02 smithi107 ceph-mon[32659]: pgmap v2678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:02 smithi132 ceph-mon[37580]: pgmap v2678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:02 smithi112 ceph-mon[37501]: pgmap v2678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:04 smithi107 ceph-mon[32659]: pgmap v2679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:04 smithi132 ceph-mon[37580]: pgmap v2679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:04 smithi112 ceph-mon[37501]: pgmap v2679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4081.834490 secs 2024-07-26T13:59:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:06 smithi107 ceph-mon[32659]: pgmap v2680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4081.834490 secs 2024-07-26T13:59:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:06 smithi132 ceph-mon[37580]: pgmap v2680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4081.834490 secs 2024-07-26T13:59:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:06 smithi112 ceph-mon[37501]: pgmap v2680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:08 smithi107 ceph-mon[32659]: pgmap v2681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:08 smithi132 ceph-mon[37580]: pgmap v2681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:08 smithi112 ceph-mon[37501]: pgmap v2681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:10.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:10.504+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:10.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:10.504+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:10.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:10.504+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:10 smithi107 ceph-mon[32659]: pgmap v2682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:10 smithi132 ceph-mon[37580]: pgmap v2682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:10 smithi112 ceph-mon[37501]: pgmap v2682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4086.834606 secs 2024-07-26T13:59:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4086.834606 secs 2024-07-26T13:59:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4086.834606 secs 2024-07-26T13:59:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:12.628+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:12.628+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:12.630 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:12.628+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:12 smithi107 ceph-mon[32659]: pgmap v2683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:12 smithi132 ceph-mon[37580]: pgmap v2683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:12 smithi112 ceph-mon[37501]: pgmap v2683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:14 smithi107 ceph-mon[32659]: pgmap v2684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:14 smithi132 ceph-mon[37580]: pgmap v2684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:14 smithi112 ceph-mon[37501]: pgmap v2684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4091.834725 secs 2024-07-26T13:59:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4091.834725 secs 2024-07-26T13:59:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4091.834725 secs 2024-07-26T13:59:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:16 smithi107 ceph-mon[32659]: pgmap v2685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:16 smithi132 ceph-mon[37580]: pgmap v2685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:16 smithi112 ceph-mon[37501]: pgmap v2685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:18 smithi107 ceph-mon[32659]: pgmap v2686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:18 smithi132 ceph-mon[37580]: pgmap v2686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:18 smithi112 ceph-mon[37501]: pgmap v2686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:20 smithi107 ceph-mon[32659]: pgmap v2687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4096.834845 secs 2024-07-26T13:59:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:20 smithi132 ceph-mon[37580]: pgmap v2687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4096.834845 secs 2024-07-26T13:59:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:20 smithi112 ceph-mon[37501]: pgmap v2687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4096.834845 secs 2024-07-26T13:59:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:22 smithi107 ceph-mon[32659]: pgmap v2688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:22 smithi132 ceph-mon[37580]: pgmap v2688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:22 smithi112 ceph-mon[37501]: pgmap v2688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:23 smithi132 ceph-mon[37580]: pgmap v2689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:23 smithi112 ceph-mon[37501]: pgmap v2689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:23 smithi107 ceph-mon[32659]: pgmap v2689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:25.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:25.507+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:25.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:25.507+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:25.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:25.507+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4101.835002 secs 2024-07-26T13:59:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:26 smithi107 ceph-mon[32659]: pgmap v2690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4101.835002 secs 2024-07-26T13:59:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:26 smithi132 ceph-mon[37580]: pgmap v2690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4101.835002 secs 2024-07-26T13:59:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:26 smithi112 ceph-mon[37501]: pgmap v2690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:27.644+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:27.644+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:27.644+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:28 smithi107 ceph-mon[32659]: pgmap v2691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:28 smithi132 ceph-mon[37580]: pgmap v2691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:28 smithi112 ceph-mon[37501]: pgmap v2691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:30 smithi107 ceph-mon[32659]: pgmap v2692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:30 smithi132 ceph-mon[37580]: pgmap v2692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:30 smithi112 ceph-mon[37501]: pgmap v2692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T13:59:31.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4106.835100 secs 2024-07-26T13:59:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4106.835100 secs 2024-07-26T13:59:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4106.835100 secs 2024-07-26T13:59:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:32 smithi107 ceph-mon[32659]: pgmap v2693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:32 smithi132 ceph-mon[37580]: pgmap v2693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:32 smithi112 ceph-mon[37501]: pgmap v2693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:59:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:59:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:33 smithi107 ceph-mon[32659]: pgmap v2694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:59:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:33 smithi107 ceph-mon[32659]: pgmap v2695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:59:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:59:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:59:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:59:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:33 smithi132 ceph-mon[37580]: pgmap v2694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:59:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:33 smithi132 ceph-mon[37580]: pgmap v2695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:59:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:59:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:59:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:59:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:33 smithi112 ceph-mon[37501]: pgmap v2694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:59:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:33 smithi112 ceph-mon[37501]: pgmap v2695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:59:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T13:59:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:35 smithi112 ceph-mon[37501]: pgmap v2696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:35 smithi107 ceph-mon[32659]: pgmap v2696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:35 smithi132 ceph-mon[37580]: pgmap v2696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4111.835227 secs 2024-07-26T13:59:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4111.835227 secs 2024-07-26T13:59:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4111.835227 secs 2024-07-26T13:59:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:37 smithi112 ceph-mon[37501]: pgmap v2697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:37 smithi107 ceph-mon[32659]: pgmap v2697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:37 smithi132 ceph-mon[37580]: pgmap v2697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:39 smithi112 ceph-mon[37501]: pgmap v2698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:59:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:39 smithi107 ceph-mon[32659]: pgmap v2698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:59:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:39 smithi132 ceph-mon[37580]: pgmap v2698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:59:40.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:40.523+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:40.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:40.523+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:40.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:40.523+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4116.835355 secs 2024-07-26T13:59:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:41 smithi112 ceph-mon[37501]: pgmap v2699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4116.835355 secs 2024-07-26T13:59:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:41 smithi107 ceph-mon[32659]: pgmap v2699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4116.835355 secs 2024-07-26T13:59:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:41 smithi132 ceph-mon[37580]: pgmap v2699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:42.661 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:42.660+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:42.661+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:42.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:42.661+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:43 smithi112 ceph-mon[37501]: pgmap v2700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:59:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:43 smithi107 ceph-mon[32659]: pgmap v2700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:59:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:43 smithi132 ceph-mon[37580]: pgmap v2700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:59:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:45 smithi112 ceph-mon[37501]: pgmap v2701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:59:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:45 smithi107 ceph-mon[32659]: pgmap v2701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:59:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:45 smithi132 ceph-mon[37580]: pgmap v2701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:59:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4121.835479 secs 2024-07-26T13:59:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4121.835479 secs 2024-07-26T13:59:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4121.835479 secs 2024-07-26T13:59:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:47 smithi112 ceph-mon[37501]: pgmap v2702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:59:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:47 smithi107 ceph-mon[32659]: pgmap v2702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:59:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:47 smithi132 ceph-mon[37580]: pgmap v2702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:59:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:49 smithi112 ceph-mon[37501]: pgmap v2703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:49 smithi107 ceph-mon[32659]: pgmap v2703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:49 smithi132 ceph-mon[37580]: pgmap v2703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:52.029 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4126.835605 secs 2024-07-26T13:59:52.030 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:51 smithi112 ceph-mon[37501]: pgmap v2704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4126.835605 secs 2024-07-26T13:59:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:51 smithi107 ceph-mon[32659]: pgmap v2704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4126.835605 secs 2024-07-26T13:59:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:51 smithi132 ceph-mon[37580]: pgmap v2704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:53 smithi107 ceph-mon[32659]: pgmap v2705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:53 smithi132 ceph-mon[37580]: pgmap v2705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:53 smithi112 ceph-mon[37501]: pgmap v2705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:59:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:55.538+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:55.538+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:55.539 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T13:59:55.538+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:55 smithi107 ceph-mon[32659]: pgmap v2706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:59:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:55 smithi132 ceph-mon[37580]: pgmap v2706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:59:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:55 smithi112 ceph-mon[37501]: pgmap v2706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:59:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4131.835759 secs 2024-07-26T13:59:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4131.835759 secs 2024-07-26T13:59:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4131.835759 secs 2024-07-26T13:59:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:57.671+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:57.671+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T13:59:57.671+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T13:59:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:57 smithi107 ceph-mon[32659]: pgmap v2707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:59:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:57 smithi132 ceph-mon[37580]: pgmap v2707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:59:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:57 smithi112 ceph-mon[37501]: pgmap v2707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:59:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 13:59:59 smithi112 ceph-mon[37501]: pgmap v2708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 13:59:59 smithi107 ceph-mon[32659]: pgmap v2708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:00.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 13:59:59 smithi132 ceph-mon[37580]: pgmap v2708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:00:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:00:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:00:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4136.835908 secs 2024-07-26T14:00:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:01 smithi112 ceph-mon[37501]: pgmap v2709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:01.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4136.835908 secs 2024-07-26T14:00:01.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:01 smithi132 ceph-mon[37580]: pgmap v2709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4136.835908 secs 2024-07-26T14:00:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:01 smithi107 ceph-mon[32659]: pgmap v2709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:03 smithi112 ceph-mon[37501]: pgmap v2710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:03 smithi107 ceph-mon[32659]: pgmap v2710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:04.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:03 smithi132 ceph-mon[37580]: pgmap v2710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:05 smithi112 ceph-mon[37501]: pgmap v2711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:05 smithi107 ceph-mon[32659]: pgmap v2711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:05 smithi132 ceph-mon[37580]: pgmap v2711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4141.836030 secs 2024-07-26T14:00:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4141.836030 secs 2024-07-26T14:00:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4141.836030 secs 2024-07-26T14:00:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:07 smithi112 ceph-mon[37501]: pgmap v2712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:07 smithi107 ceph-mon[32659]: pgmap v2712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:07 smithi132 ceph-mon[37580]: pgmap v2712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:09 smithi112 ceph-mon[37501]: pgmap v2713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:10.017 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:09 smithi132 ceph-mon[37580]: pgmap v2713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:09 smithi107 ceph-mon[32659]: pgmap v2713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:10.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:10.553+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:10.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:10.553+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:10.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:10.553+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4146.836175 secs 2024-07-26T14:00:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:11 smithi112 ceph-mon[37501]: pgmap v2714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4146.836175 secs 2024-07-26T14:00:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:11 smithi107 ceph-mon[32659]: pgmap v2714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4146.836175 secs 2024-07-26T14:00:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:11 smithi132 ceph-mon[37580]: pgmap v2714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:12.686+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:12.686+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:12.688 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:12.686+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:13.338 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:13.336+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:00:13.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:13.337+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:00:13.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:13.375+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:00:13.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:13.375+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:00:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:13 smithi112 ceph-mon[37501]: pgmap v2715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:13 smithi107 ceph-mon[32659]: pgmap v2715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:13 smithi132 ceph-mon[37580]: pgmap v2715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:14.479 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:14.478+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:14.478+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:14.479+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:00:14.480 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:14.479+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:00:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:15 smithi107 ceph-mon[32659]: pgmap v2716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:15 smithi132 ceph-mon[37580]: pgmap v2716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:15 smithi112 ceph-mon[37501]: pgmap v2716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4151.836300 secs 2024-07-26T14:00:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4151.836300 secs 2024-07-26T14:00:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4151.836300 secs 2024-07-26T14:00:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:17 smithi107 ceph-mon[32659]: pgmap v2717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:17 smithi132 ceph-mon[37580]: pgmap v2717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:17 smithi112 ceph-mon[37501]: pgmap v2717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:19 smithi107 ceph-mon[32659]: pgmap v2718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:19 smithi132 ceph-mon[37580]: pgmap v2718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:19 smithi112 ceph-mon[37501]: pgmap v2718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:22.002 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4156.836426 secs 2024-07-26T14:00:22.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:21 smithi112 ceph-mon[37501]: pgmap v2719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4156.836426 secs 2024-07-26T14:00:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:21 smithi107 ceph-mon[32659]: pgmap v2719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4156.836426 secs 2024-07-26T14:00:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:21 smithi132 ceph-mon[37580]: pgmap v2719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:23 smithi107 ceph-mon[32659]: pgmap v2720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:23 smithi132 ceph-mon[37580]: pgmap v2720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:23 smithi112 ceph-mon[37501]: pgmap v2720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:25.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:25.567+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:25.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:25.567+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:25.607 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:25.567+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:25 smithi107 ceph-mon[32659]: pgmap v2721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:25 smithi132 ceph-mon[37580]: pgmap v2721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:25 smithi112 ceph-mon[37501]: pgmap v2721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4161.836491 secs 2024-07-26T14:00:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4161.836491 secs 2024-07-26T14:00:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4161.836491 secs 2024-07-26T14:00:27.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:27.689+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:27.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:27.689+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:27.691 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:27.689+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:27 smithi107 ceph-mon[32659]: pgmap v2722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:27 smithi132 ceph-mon[37580]: pgmap v2722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:27 smithi112 ceph-mon[37501]: pgmap v2722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:29 smithi107 ceph-mon[32659]: pgmap v2723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:29 smithi132 ceph-mon[37580]: pgmap v2723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:29 smithi112 ceph-mon[37501]: pgmap v2723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4166.836607 secs 2024-07-26T14:00:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4166.836607 secs 2024-07-26T14:00:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4166.836607 secs 2024-07-26T14:00:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:31 smithi107 ceph-mon[32659]: pgmap v2724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:31 smithi132 ceph-mon[37580]: pgmap v2724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:31 smithi112 ceph-mon[37501]: pgmap v2724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:33 smithi107 ceph-mon[32659]: pgmap v2725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:00:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:33 smithi132 ceph-mon[37580]: pgmap v2725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:00:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:33 smithi112 ceph-mon[37501]: pgmap v2725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:00:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:34 smithi107 ceph-mon[32659]: pgmap v2726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:34 smithi107 ceph-mon[32659]: pgmap v2727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:00:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:00:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:34 smithi132 ceph-mon[37580]: pgmap v2726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:34 smithi132 ceph-mon[37580]: pgmap v2727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:00:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:00:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:34 smithi112 ceph-mon[37501]: pgmap v2726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:34 smithi112 ceph-mon[37501]: pgmap v2727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:00:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:00:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4171.836729 secs 2024-07-26T14:00:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4171.836729 secs 2024-07-26T14:00:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4171.836729 secs 2024-07-26T14:00:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:36 smithi132 ceph-mon[37580]: pgmap v2728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:00:37.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:36 smithi107 ceph-mon[32659]: pgmap v2728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:00:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:36 smithi112 ceph-mon[37501]: pgmap v2728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:00:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:37 smithi112 ceph-mon[37501]: pgmap v2729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:00:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:37 smithi107 ceph-mon[32659]: pgmap v2729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:00:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:37 smithi132 ceph-mon[37580]: pgmap v2729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:00:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:39 smithi112 ceph-mon[37501]: pgmap v2730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:39 smithi107 ceph-mon[32659]: pgmap v2730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:39 smithi132 ceph-mon[37580]: pgmap v2730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:40.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:40.584+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:40.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:40.584+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:40.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:40.584+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4176.836850 secs 2024-07-26T14:00:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4176.836850 secs 2024-07-26T14:00:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4176.836850 secs 2024-07-26T14:00:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:41 smithi112 ceph-mon[37501]: pgmap v2731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:41 smithi107 ceph-mon[32659]: pgmap v2731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:41 smithi132 ceph-mon[37580]: pgmap v2731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:42.699+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:42.699+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:42.699+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:43 smithi112 ceph-mon[37501]: pgmap v2732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:43 smithi107 ceph-mon[32659]: pgmap v2732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:43 smithi132 ceph-mon[37580]: pgmap v2732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4181.836975 secs 2024-07-26T14:00:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:45 smithi112 ceph-mon[37501]: pgmap v2733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4181.836975 secs 2024-07-26T14:00:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:45 smithi107 ceph-mon[32659]: pgmap v2733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4181.836975 secs 2024-07-26T14:00:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:45 smithi132 ceph-mon[37580]: pgmap v2733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:47 smithi112 ceph-mon[37501]: pgmap v2734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:47 smithi107 ceph-mon[32659]: pgmap v2734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:47 smithi132 ceph-mon[37580]: pgmap v2734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:49 smithi112 ceph-mon[37501]: pgmap v2735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:49 smithi107 ceph-mon[32659]: pgmap v2735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:49 smithi132 ceph-mon[37580]: pgmap v2735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4186.837101 secs 2024-07-26T14:00:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4186.837101 secs 2024-07-26T14:00:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4186.837101 secs 2024-07-26T14:00:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:51 smithi112 ceph-mon[37501]: pgmap v2736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:51 smithi107 ceph-mon[32659]: pgmap v2736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:51 smithi132 ceph-mon[37580]: pgmap v2736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:00:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:54 smithi112 ceph-mon[37501]: pgmap v2737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:54 smithi107 ceph-mon[32659]: pgmap v2737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:54 smithi132 ceph-mon[37580]: pgmap v2737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:55.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:55.598+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:55.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:55.598+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:55.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:00:55.598+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4191.837213 secs 2024-07-26T14:00:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4191.837213 secs 2024-07-26T14:00:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4191.837213 secs 2024-07-26T14:00:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:56 smithi112 ceph-mon[37501]: pgmap v2738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:56 smithi107 ceph-mon[32659]: pgmap v2738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:56 smithi132 ceph-mon[37580]: pgmap v2738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:57.715+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:57.715+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:00:57.715+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:00:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:00:58 smithi112 ceph-mon[37501]: pgmap v2739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:00:58 smithi107 ceph-mon[32659]: pgmap v2739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:00:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:00:58 smithi132 ceph-mon[37580]: pgmap v2739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:00 smithi112 ceph-mon[37501]: pgmap v2740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4196.837340 secs 2024-07-26T14:01:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:00 smithi107 ceph-mon[32659]: pgmap v2740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4196.837340 secs 2024-07-26T14:01:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:00 smithi132 ceph-mon[37580]: pgmap v2740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4196.837340 secs 2024-07-26T14:01:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:02 smithi112 ceph-mon[37501]: pgmap v2741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:02 smithi107 ceph-mon[32659]: pgmap v2741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:02 smithi132 ceph-mon[37580]: pgmap v2741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:04 smithi112 ceph-mon[37501]: pgmap v2742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:04 smithi107 ceph-mon[32659]: pgmap v2742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:04 smithi132 ceph-mon[37580]: pgmap v2742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4201.837456 secs 2024-07-26T14:01:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4201.837456 secs 2024-07-26T14:01:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4201.837456 secs 2024-07-26T14:01:07.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:06 smithi112 ceph-mon[37501]: pgmap v2743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:06 smithi107 ceph-mon[32659]: pgmap v2743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:06 smithi132 ceph-mon[37580]: pgmap v2743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:08 smithi112 ceph-mon[37501]: pgmap v2744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:08 smithi107 ceph-mon[32659]: pgmap v2744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:08 smithi132 ceph-mon[37580]: pgmap v2744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:10.606+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:10.606+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:10.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:10.606+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:10 smithi112 ceph-mon[37501]: pgmap v2745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4206.837579 secs 2024-07-26T14:01:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:10 smithi107 ceph-mon[32659]: pgmap v2745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4206.837579 secs 2024-07-26T14:01:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:10 smithi132 ceph-mon[37580]: pgmap v2745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4206.837579 secs 2024-07-26T14:01:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:12.718+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:12.718+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:12.720 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:12.718+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:12 smithi112 ceph-mon[37501]: pgmap v2746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:12 smithi107 ceph-mon[32659]: pgmap v2746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:12 smithi132 ceph-mon[37580]: pgmap v2746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:15 smithi112 ceph-mon[37501]: pgmap v2747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:15 smithi107 ceph-mon[32659]: pgmap v2747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:15 smithi132 ceph-mon[37580]: pgmap v2747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4211.837657 secs 2024-07-26T14:01:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4211.837657 secs 2024-07-26T14:01:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4211.837657 secs 2024-07-26T14:01:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:17 smithi112 ceph-mon[37501]: pgmap v2748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:17 smithi107 ceph-mon[32659]: pgmap v2748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:17 smithi132 ceph-mon[37580]: pgmap v2748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:19 smithi112 ceph-mon[37501]: pgmap v2749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:19 smithi107 ceph-mon[32659]: pgmap v2749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:19 smithi132 ceph-mon[37580]: pgmap v2749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:21 smithi112 ceph-mon[37501]: pgmap v2750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4216.837723 secs 2024-07-26T14:01:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:21 smithi107 ceph-mon[32659]: pgmap v2750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4216.837723 secs 2024-07-26T14:01:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:21 smithi132 ceph-mon[37580]: pgmap v2750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4216.837723 secs 2024-07-26T14:01:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:23 smithi112 ceph-mon[37501]: pgmap v2751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:23 smithi107 ceph-mon[32659]: pgmap v2751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:23 smithi132 ceph-mon[37580]: pgmap v2751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:25.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:25.622+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:25.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:25.622+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:25.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:25.622+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:25 smithi112 ceph-mon[37501]: pgmap v2752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:25 smithi107 ceph-mon[32659]: pgmap v2752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:25 smithi132 ceph-mon[37580]: pgmap v2752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4221.837848 secs 2024-07-26T14:01:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4221.837848 secs 2024-07-26T14:01:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4221.837848 secs 2024-07-26T14:01:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:27.726+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:27.726+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:27.726+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:27 smithi112 ceph-mon[37501]: pgmap v2753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:27 smithi107 ceph-mon[32659]: pgmap v2753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:27 smithi132 ceph-mon[37580]: pgmap v2753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:29 smithi112 ceph-mon[37501]: pgmap v2754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:29 smithi107 ceph-mon[32659]: pgmap v2754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:29 smithi132 ceph-mon[37580]: pgmap v2754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:31 smithi112 ceph-mon[37501]: pgmap v2755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4226.837979 secs 2024-07-26T14:01:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:31 smithi107 ceph-mon[32659]: pgmap v2755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4226.837979 secs 2024-07-26T14:01:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:31 smithi132 ceph-mon[37580]: pgmap v2755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4226.837979 secs 2024-07-26T14:01:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:33 smithi112 ceph-mon[37501]: pgmap v2756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:33 smithi107 ceph-mon[32659]: pgmap v2756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:33 smithi132 ceph-mon[37580]: pgmap v2756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:35 smithi112 ceph-mon[37501]: pgmap v2757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:01:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:01:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:35 smithi107 ceph-mon[32659]: pgmap v2757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:01:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:01:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:35 smithi132 ceph-mon[37580]: pgmap v2757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:01:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:01:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:36 smithi112 ceph-mon[37501]: pgmap v2758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:01:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:36 smithi112 ceph-mon[37501]: pgmap v2759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4231.838095 secs 2024-07-26T14:01:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:36 smithi107 ceph-mon[32659]: pgmap v2758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:01:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:36 smithi107 ceph-mon[32659]: pgmap v2759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4231.838095 secs 2024-07-26T14:01:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:36 smithi132 ceph-mon[37580]: pgmap v2758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:01:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:36 smithi132 ceph-mon[37580]: pgmap v2759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:01:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4231.838095 secs 2024-07-26T14:01:39.034 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:38 smithi107 ceph-mon[32659]: pgmap v2760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:38 smithi132 ceph-mon[37580]: pgmap v2760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:38 smithi112 ceph-mon[37501]: pgmap v2760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:40.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:40.634+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:40.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:40.634+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:40.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:40.634+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:40 smithi107 ceph-mon[32659]: pgmap v2761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:01:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:40 smithi132 ceph-mon[37580]: pgmap v2761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:01:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:40 smithi112 ceph-mon[37501]: pgmap v2761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:01:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4236.838238 secs 2024-07-26T14:01:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4236.838238 secs 2024-07-26T14:01:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4236.838238 secs 2024-07-26T14:01:42.739 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:42.738+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:42.739 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:42.738+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:42.739 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:42.738+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:42 smithi107 ceph-mon[32659]: pgmap v2762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:01:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:42 smithi132 ceph-mon[37580]: pgmap v2762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:01:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:42 smithi112 ceph-mon[37501]: pgmap v2762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:01:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:44 smithi107 ceph-mon[32659]: pgmap v2763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:01:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:44 smithi132 ceph-mon[37580]: pgmap v2763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:01:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:44 smithi112 ceph-mon[37501]: pgmap v2763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:01:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:46 smithi107 ceph-mon[32659]: pgmap v2764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:01:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4241.838319 secs 2024-07-26T14:01:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:46 smithi132 ceph-mon[37580]: pgmap v2764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:01:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4241.838319 secs 2024-07-26T14:01:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:46 smithi112 ceph-mon[37501]: pgmap v2764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:01:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4241.838319 secs 2024-07-26T14:01:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:48 smithi107 ceph-mon[32659]: pgmap v2765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:48 smithi132 ceph-mon[37580]: pgmap v2765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:48 smithi112 ceph-mon[37501]: pgmap v2765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:50 smithi107 ceph-mon[32659]: pgmap v2766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:50 smithi132 ceph-mon[37580]: pgmap v2766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:50 smithi112 ceph-mon[37501]: pgmap v2766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4246.838458 secs 2024-07-26T14:01:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4246.838458 secs 2024-07-26T14:01:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4246.838458 secs 2024-07-26T14:01:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:52 smithi107 ceph-mon[32659]: pgmap v2767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:52 smithi132 ceph-mon[37580]: pgmap v2767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:53.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:52 smithi112 ceph-mon[37501]: pgmap v2767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:54 smithi107 ceph-mon[32659]: pgmap v2768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:54 smithi132 ceph-mon[37580]: pgmap v2768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:54 smithi112 ceph-mon[37501]: pgmap v2768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:01:55.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:55.650+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:55.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:55.650+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:55.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:01:55.650+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:56 smithi107 ceph-mon[32659]: pgmap v2769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4251.838532 secs 2024-07-26T14:01:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:56 smithi132 ceph-mon[37580]: pgmap v2769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4251.838532 secs 2024-07-26T14:01:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:56 smithi112 ceph-mon[37501]: pgmap v2769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4251.838532 secs 2024-07-26T14:01:57.754 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:57.754+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:57.754+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:57.755 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:01:57.754+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:01:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:01:58 smithi107 ceph-mon[32659]: pgmap v2770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:59.083 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:01:58 smithi132 ceph-mon[37580]: pgmap v2770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:01:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:01:58 smithi112 ceph-mon[37501]: pgmap v2770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:00 smithi107 ceph-mon[32659]: pgmap v2771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4256.838664 secs 2024-07-26T14:02:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:00 smithi132 ceph-mon[37580]: pgmap v2771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4256.838664 secs 2024-07-26T14:02:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:00 smithi112 ceph-mon[37501]: pgmap v2771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4256.838664 secs 2024-07-26T14:02:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:02 smithi107 ceph-mon[32659]: pgmap v2772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:02 smithi132 ceph-mon[37580]: pgmap v2772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:02 smithi112 ceph-mon[37501]: pgmap v2772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:04 smithi107 ceph-mon[32659]: pgmap v2773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:04 smithi132 ceph-mon[37580]: pgmap v2773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:04 smithi112 ceph-mon[37501]: pgmap v2773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4261.838836 secs 2024-07-26T14:02:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4261.838836 secs 2024-07-26T14:02:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4261.838836 secs 2024-07-26T14:02:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:06 smithi132 ceph-mon[37580]: pgmap v2774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:06 smithi112 ceph-mon[37501]: pgmap v2774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:06 smithi107 ceph-mon[32659]: pgmap v2774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:08 smithi112 ceph-mon[37501]: pgmap v2775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:09.242 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:08 smithi107 ceph-mon[32659]: pgmap v2775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:08 smithi132 ceph-mon[37580]: pgmap v2775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:10.651+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:10.651+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:10.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:10.651+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:10 smithi107 ceph-mon[32659]: pgmap v2776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4266.838963 secs 2024-07-26T14:02:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:10 smithi112 ceph-mon[37501]: pgmap v2776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4266.838963 secs 2024-07-26T14:02:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:10 smithi132 ceph-mon[37580]: pgmap v2776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4266.838963 secs 2024-07-26T14:02:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:12.761+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:12.761+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:12.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:12.761+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:12 smithi112 ceph-mon[37501]: pgmap v2777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:12 smithi107 ceph-mon[32659]: pgmap v2777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:12 smithi132 ceph-mon[37580]: pgmap v2777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:13 smithi112 ceph-mon[37501]: pgmap v2778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:13 smithi107 ceph-mon[32659]: pgmap v2778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:13 smithi132 ceph-mon[37580]: pgmap v2778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:16 smithi112 ceph-mon[37501]: pgmap v2779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4271.839107 secs 2024-07-26T14:02:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:16 smithi107 ceph-mon[32659]: pgmap v2779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4271.839107 secs 2024-07-26T14:02:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:16 smithi132 ceph-mon[37580]: pgmap v2779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4271.839107 secs 2024-07-26T14:02:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:18 smithi112 ceph-mon[37501]: pgmap v2780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:18 smithi107 ceph-mon[32659]: pgmap v2780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:18 smithi132 ceph-mon[37580]: pgmap v2780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:20 smithi112 ceph-mon[37501]: pgmap v2781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:20 smithi107 ceph-mon[32659]: pgmap v2781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:20 smithi132 ceph-mon[37580]: pgmap v2781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4276.839171 secs 2024-07-26T14:02:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4276.839171 secs 2024-07-26T14:02:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4276.839171 secs 2024-07-26T14:02:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:22 smithi112 ceph-mon[37501]: pgmap v2782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:22 smithi107 ceph-mon[32659]: pgmap v2782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:22 smithi132 ceph-mon[37580]: pgmap v2782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:24 smithi112 ceph-mon[37501]: pgmap v2783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:24 smithi107 ceph-mon[32659]: pgmap v2783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:24 smithi132 ceph-mon[37580]: pgmap v2783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:25.667+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:25.667+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:25.668 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:25.667+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:26 smithi112 ceph-mon[37501]: pgmap v2784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4281.839300 secs 2024-07-26T14:02:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:26 smithi107 ceph-mon[32659]: pgmap v2784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4281.839300 secs 2024-07-26T14:02:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:26 smithi132 ceph-mon[37580]: pgmap v2784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4281.839300 secs 2024-07-26T14:02:27.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:27.775+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:27.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:27.775+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:27.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:27.775+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:28 smithi112 ceph-mon[37501]: pgmap v2785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:28 smithi107 ceph-mon[32659]: pgmap v2785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:28 smithi132 ceph-mon[37580]: pgmap v2785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:30 smithi112 ceph-mon[37501]: pgmap v2786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:30 smithi107 ceph-mon[32659]: pgmap v2786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:30 smithi132 ceph-mon[37580]: pgmap v2786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4286.839407 secs 2024-07-26T14:02:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4286.839407 secs 2024-07-26T14:02:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4286.839407 secs 2024-07-26T14:02:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:32 smithi112 ceph-mon[37501]: pgmap v2787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:32.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:32 smithi132 ceph-mon[37580]: pgmap v2787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:32 smithi107 ceph-mon[32659]: pgmap v2787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:34 smithi112 ceph-mon[37501]: pgmap v2788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:34 smithi107 ceph-mon[32659]: pgmap v2788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:34 smithi132 ceph-mon[37580]: pgmap v2788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:36 smithi112 ceph-mon[37501]: pgmap v2789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4291.839537 secs 2024-07-26T14:02:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:36 smithi107 ceph-mon[32659]: pgmap v2789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4291.839537 secs 2024-07-26T14:02:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:36 smithi132 ceph-mon[37580]: pgmap v2789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4291.839537 secs 2024-07-26T14:02:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:37 smithi112 ceph-mon[37501]: pgmap v2790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:02:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:02:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:37 smithi107 ceph-mon[32659]: pgmap v2790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:02:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:02:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:37 smithi132 ceph-mon[37580]: pgmap v2790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:02:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:02:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:39 smithi107 ceph-mon[32659]: pgmap v2791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:02:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:39 smithi132 ceph-mon[37580]: pgmap v2791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:02:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:39 smithi112 ceph-mon[37501]: pgmap v2791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:02:40.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:40.669+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:40.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:40.669+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:40.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:40.669+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4296.839657 secs 2024-07-26T14:02:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:41 smithi132 ceph-mon[37580]: pgmap v2792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:02:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4296.839657 secs 2024-07-26T14:02:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:41 smithi112 ceph-mon[37501]: pgmap v2792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:02:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4296.839657 secs 2024-07-26T14:02:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:41 smithi107 ceph-mon[32659]: pgmap v2792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:02:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:42.792+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:42.792+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:42.792+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:43 smithi132 ceph-mon[37580]: pgmap v2793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:02:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:43 smithi112 ceph-mon[37501]: pgmap v2793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:02:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:43 smithi107 ceph-mon[32659]: pgmap v2793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:02:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:45 smithi132 ceph-mon[37580]: pgmap v2794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:02:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:45 smithi112 ceph-mon[37501]: pgmap v2794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:02:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:45 smithi107 ceph-mon[32659]: pgmap v2794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:02:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4301.839779 secs 2024-07-26T14:02:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4301.839779 secs 2024-07-26T14:02:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4301.839779 secs 2024-07-26T14:02:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:47 smithi112 ceph-mon[37501]: pgmap v2795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:02:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:47 smithi107 ceph-mon[32659]: pgmap v2795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:02:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:47 smithi132 ceph-mon[37580]: pgmap v2795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:02:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:49 smithi112 ceph-mon[37501]: pgmap v2796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:49 smithi107 ceph-mon[32659]: pgmap v2796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:49 smithi132 ceph-mon[37580]: pgmap v2796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4306.839902 secs 2024-07-26T14:02:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:51 smithi112 ceph-mon[37501]: pgmap v2797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4306.839902 secs 2024-07-26T14:02:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:51 smithi107 ceph-mon[32659]: pgmap v2797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4306.839902 secs 2024-07-26T14:02:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:51 smithi132 ceph-mon[37580]: pgmap v2797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:53 smithi112 ceph-mon[37501]: pgmap v2798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:53 smithi107 ceph-mon[32659]: pgmap v2798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:53 smithi132 ceph-mon[37580]: pgmap v2798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:55.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:55.685+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:55.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:55.685+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:55.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:02:55.685+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:55.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:55 smithi107 ceph-mon[32659]: pgmap v2799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:55 smithi112 ceph-mon[37501]: pgmap v2799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:55 smithi132 ceph-mon[37580]: pgmap v2799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4311.840055 secs 2024-07-26T14:02:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4311.840055 secs 2024-07-26T14:02:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4311.840055 secs 2024-07-26T14:02:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:57 smithi112 ceph-mon[37501]: pgmap v2800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:57 smithi107 ceph-mon[32659]: pgmap v2800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:57.807+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:57.807+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:02:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:02:57.807+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:57 smithi132 ceph-mon[37580]: pgmap v2800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:02:59 smithi112 ceph-mon[37501]: pgmap v2801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:02:59 smithi107 ceph-mon[32659]: pgmap v2801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:02:59 smithi132 ceph-mon[37580]: pgmap v2801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4316.840172 secs 2024-07-26T14:03:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:01 smithi112 ceph-mon[37501]: pgmap v2802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4316.840172 secs 2024-07-26T14:03:01.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:01 smithi107 ceph-mon[32659]: pgmap v2802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4316.840172 secs 2024-07-26T14:03:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:01 smithi132 ceph-mon[37580]: pgmap v2802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:03 smithi112 ceph-mon[37501]: pgmap v2803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:03 smithi107 ceph-mon[32659]: pgmap v2803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:03 smithi132 ceph-mon[37580]: pgmap v2803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:05 smithi112 ceph-mon[37501]: pgmap v2804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:05 smithi107 ceph-mon[32659]: pgmap v2804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:05 smithi132 ceph-mon[37580]: pgmap v2804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4321.840296 secs 2024-07-26T14:03:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4321.840296 secs 2024-07-26T14:03:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4321.840296 secs 2024-07-26T14:03:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:07 smithi112 ceph-mon[37501]: pgmap v2805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:07 smithi107 ceph-mon[32659]: pgmap v2805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:07 smithi132 ceph-mon[37580]: pgmap v2805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:09 smithi112 ceph-mon[37501]: pgmap v2806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:09 smithi107 ceph-mon[32659]: pgmap v2806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:09 smithi132 ceph-mon[37580]: pgmap v2806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:10.701+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:10.701+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:10.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:10.701+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4326.840424 secs 2024-07-26T14:03:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:11 smithi112 ceph-mon[37501]: pgmap v2807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4326.840424 secs 2024-07-26T14:03:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:11 smithi107 ceph-mon[32659]: pgmap v2807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4326.840424 secs 2024-07-26T14:03:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:11 smithi132 ceph-mon[37580]: pgmap v2807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:12.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:12.822+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:12.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:12.822+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:12.824 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:12.822+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:13.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:13.367+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:03:13.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:13.368+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:03:13.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:13.403+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:03:13.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:13.404+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:03:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:13 smithi112 ceph-mon[37501]: pgmap v2808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:13 smithi107 ceph-mon[32659]: pgmap v2808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:13 smithi132 ceph-mon[37580]: pgmap v2808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:14.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:14.512+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:03:14.513 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:14.512+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:03:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:14.512+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:03:14.514 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:14.513+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:03:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:15 smithi112 ceph-mon[37501]: pgmap v2809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:15 smithi107 ceph-mon[32659]: pgmap v2809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:15 smithi132 ceph-mon[37580]: pgmap v2809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4331.840545 secs 2024-07-26T14:03:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4331.840545 secs 2024-07-26T14:03:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4331.840545 secs 2024-07-26T14:03:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:17 smithi112 ceph-mon[37501]: pgmap v2810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:17 smithi107 ceph-mon[32659]: pgmap v2810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:17 smithi132 ceph-mon[37580]: pgmap v2810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:19 smithi112 ceph-mon[37501]: pgmap v2811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:19 smithi107 ceph-mon[32659]: pgmap v2811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:19 smithi132 ceph-mon[37580]: pgmap v2811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4336.840629 secs 2024-07-26T14:03:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:21 smithi112 ceph-mon[37501]: pgmap v2812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4336.840629 secs 2024-07-26T14:03:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:21 smithi107 ceph-mon[32659]: pgmap v2812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4336.840629 secs 2024-07-26T14:03:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:21 smithi132 ceph-mon[37580]: pgmap v2812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:23 smithi107 ceph-mon[32659]: pgmap v2813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:23 smithi132 ceph-mon[37580]: pgmap v2813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:23 smithi112 ceph-mon[37501]: pgmap v2813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:25.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:25.709+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:25.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:25.709+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:25.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:25.709+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:25.711 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:25 smithi107 ceph-mon[32659]: pgmap v2814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:25.752 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:25 smithi112 ceph-mon[37501]: pgmap v2814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:25 smithi132 ceph-mon[37580]: pgmap v2814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4341.840764 secs 2024-07-26T14:03:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4341.840764 secs 2024-07-26T14:03:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4341.840764 secs 2024-07-26T14:03:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:27 smithi107 ceph-mon[32659]: pgmap v2815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:27 smithi132 ceph-mon[37580]: pgmap v2815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:27.825 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:27.824+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:27.824+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:27.824+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:27 smithi112 ceph-mon[37501]: pgmap v2815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:29 smithi107 ceph-mon[32659]: pgmap v2816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:29 smithi132 ceph-mon[37580]: pgmap v2816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:29 smithi112 ceph-mon[37501]: pgmap v2816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4346.840911 secs 2024-07-26T14:03:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:31 smithi107 ceph-mon[32659]: pgmap v2817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4346.840911 secs 2024-07-26T14:03:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:31 smithi132 ceph-mon[37580]: pgmap v2817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4346.840911 secs 2024-07-26T14:03:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:31 smithi112 ceph-mon[37501]: pgmap v2817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:33 smithi107 ceph-mon[32659]: pgmap v2818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:33 smithi132 ceph-mon[37580]: pgmap v2818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:33 smithi112 ceph-mon[37501]: pgmap v2818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:35 smithi107 ceph-mon[32659]: pgmap v2819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:35 smithi132 ceph-mon[37580]: pgmap v2819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:35 smithi112 ceph-mon[37501]: pgmap v2819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4351.841037 secs 2024-07-26T14:03:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4351.841037 secs 2024-07-26T14:03:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4351.841037 secs 2024-07-26T14:03:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: pgmap v2820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: pgmap v2820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:03:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: pgmap v2820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:03:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:38 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:03:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:38 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:03:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:38 smithi107 ceph-mon[32659]: pgmap v2821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:03:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:38 smithi107 ceph-mon[32659]: pgmap v2822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:03:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:38 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:03:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:38 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:03:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:38 smithi132 ceph-mon[37580]: pgmap v2821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:03:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:38 smithi132 ceph-mon[37580]: pgmap v2822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:03:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:38 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:03:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:38 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:03:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:38 smithi112 ceph-mon[37501]: pgmap v2821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:03:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:38 smithi112 ceph-mon[37501]: pgmap v2822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:03:40.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:40.725+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:40.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:40.725+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:40.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:40.725+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:40 smithi107 ceph-mon[32659]: pgmap v2823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:03:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:40 smithi132 ceph-mon[37580]: pgmap v2823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:03:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:40 smithi112 ceph-mon[37501]: pgmap v2823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:03:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4356.841162 secs 2024-07-26T14:03:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4356.841162 secs 2024-07-26T14:03:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4356.841162 secs 2024-07-26T14:03:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:42 smithi107 ceph-mon[32659]: pgmap v2824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:03:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:42 smithi132 ceph-mon[37580]: pgmap v2824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:03:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:42.828+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:42.828+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:42.829 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:42.828+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:42 smithi112 ceph-mon[37501]: pgmap v2824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:03:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:44 smithi107 ceph-mon[32659]: pgmap v2825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:03:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:44 smithi132 ceph-mon[37580]: pgmap v2825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:03:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:44 smithi112 ceph-mon[37501]: pgmap v2825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:03:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4361.841286 secs 2024-07-26T14:03:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:46 smithi132 ceph-mon[37580]: pgmap v2826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:03:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4361.841286 secs 2024-07-26T14:03:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:46 smithi112 ceph-mon[37501]: pgmap v2826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:03:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4361.841286 secs 2024-07-26T14:03:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:46 smithi107 ceph-mon[32659]: pgmap v2826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:03:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:48 smithi132 ceph-mon[37580]: pgmap v2827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:03:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:48 smithi112 ceph-mon[37501]: pgmap v2827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:03:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:48 smithi107 ceph-mon[32659]: pgmap v2827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:03:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:50 smithi132 ceph-mon[37580]: pgmap v2828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:03:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:50 smithi112 ceph-mon[37501]: pgmap v2828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:03:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:50 smithi107 ceph-mon[32659]: pgmap v2828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:03:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4366.841409 secs 2024-07-26T14:03:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4366.841409 secs 2024-07-26T14:03:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4366.841409 secs 2024-07-26T14:03:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:52 smithi112 ceph-mon[37501]: pgmap v2829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:03:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:52 smithi107 ceph-mon[32659]: pgmap v2829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:03:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:52 smithi132 ceph-mon[37580]: pgmap v2829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:03:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:54 smithi112 ceph-mon[37501]: pgmap v2830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:54 smithi107 ceph-mon[32659]: pgmap v2830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:54 smithi132 ceph-mon[37580]: pgmap v2830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:55.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:55.741+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:55.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:55.741+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:55.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:03:55.741+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4371.841489 secs 2024-07-26T14:03:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:56 smithi112 ceph-mon[37501]: pgmap v2831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4371.841489 secs 2024-07-26T14:03:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:56 smithi107 ceph-mon[32659]: pgmap v2831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4371.841489 secs 2024-07-26T14:03:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:56 smithi132 ceph-mon[37580]: pgmap v2831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:57.843+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:57.843+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:03:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:03:57.843+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:03:58 smithi112 ceph-mon[37501]: pgmap v2832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:03:58 smithi107 ceph-mon[32659]: pgmap v2832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:03:58 smithi132 ceph-mon[37580]: pgmap v2832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:00 smithi112 ceph-mon[37501]: pgmap v2833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:00 smithi107 ceph-mon[32659]: pgmap v2833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:00 smithi132 ceph-mon[37580]: pgmap v2833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4376.841615 secs 2024-07-26T14:04:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4376.841615 secs 2024-07-26T14:04:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4376.841615 secs 2024-07-26T14:04:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:02 smithi112 ceph-mon[37501]: pgmap v2834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:02 smithi107 ceph-mon[32659]: pgmap v2834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:02 smithi132 ceph-mon[37580]: pgmap v2834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:04.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:04 smithi132 ceph-mon[37580]: pgmap v2835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:04 smithi112 ceph-mon[37501]: pgmap v2835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:04 smithi107 ceph-mon[32659]: pgmap v2835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4381.841754 secs 2024-07-26T14:04:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:06 smithi112 ceph-mon[37501]: pgmap v2836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4381.841754 secs 2024-07-26T14:04:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:06 smithi107 ceph-mon[32659]: pgmap v2836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4381.841754 secs 2024-07-26T14:04:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:06 smithi132 ceph-mon[37580]: pgmap v2836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:08 smithi112 ceph-mon[37501]: pgmap v2837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:08 smithi107 ceph-mon[32659]: pgmap v2837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:08 smithi132 ceph-mon[37580]: pgmap v2837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:10.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:10.755+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:10.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:10.755+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:10.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:10.755+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:10 smithi112 ceph-mon[37501]: pgmap v2838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:10 smithi107 ceph-mon[32659]: pgmap v2838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:10 smithi132 ceph-mon[37580]: pgmap v2838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:11.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4386.841887 secs 2024-07-26T14:04:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4386.841887 secs 2024-07-26T14:04:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4386.841887 secs 2024-07-26T14:04:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:12.848+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:12.848+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:12.848+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:12 smithi112 ceph-mon[37501]: pgmap v2839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:12 smithi107 ceph-mon[32659]: pgmap v2839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:12 smithi132 ceph-mon[37580]: pgmap v2839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:14 smithi112 ceph-mon[37501]: pgmap v2840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:14 smithi107 ceph-mon[32659]: pgmap v2840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:14 smithi132 ceph-mon[37580]: pgmap v2840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4391.841960 secs 2024-07-26T14:04:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:16 smithi112 ceph-mon[37501]: pgmap v2841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4391.841960 secs 2024-07-26T14:04:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:16 smithi107 ceph-mon[32659]: pgmap v2841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4391.841960 secs 2024-07-26T14:04:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:16 smithi132 ceph-mon[37580]: pgmap v2841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:18 smithi112 ceph-mon[37501]: pgmap v2842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:18 smithi107 ceph-mon[32659]: pgmap v2842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:18 smithi132 ceph-mon[37580]: pgmap v2842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:20 smithi112 ceph-mon[37501]: pgmap v2843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:20 smithi107 ceph-mon[32659]: pgmap v2843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:20 smithi132 ceph-mon[37580]: pgmap v2843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4396.842034 secs 2024-07-26T14:04:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4396.842034 secs 2024-07-26T14:04:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4396.842034 secs 2024-07-26T14:04:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:22 smithi112 ceph-mon[37501]: pgmap v2844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:22 smithi107 ceph-mon[32659]: pgmap v2844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:22 smithi132 ceph-mon[37580]: pgmap v2844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:24 smithi112 ceph-mon[37501]: pgmap v2845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:24 smithi107 ceph-mon[32659]: pgmap v2845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:24 smithi132 ceph-mon[37580]: pgmap v2845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:25.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:25.768+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:25.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:25.768+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:25.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:25.768+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4401.842176 secs 2024-07-26T14:04:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:26 smithi112 ceph-mon[37501]: pgmap v2846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4401.842176 secs 2024-07-26T14:04:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:26 smithi107 ceph-mon[32659]: pgmap v2846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4401.842176 secs 2024-07-26T14:04:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:26 smithi132 ceph-mon[37580]: pgmap v2846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:27.865 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:27.864+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:27.864+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:27.864+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:28 smithi112 ceph-mon[37501]: pgmap v2847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:28 smithi107 ceph-mon[32659]: pgmap v2847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:28 smithi132 ceph-mon[37580]: pgmap v2847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:30 smithi107 ceph-mon[32659]: pgmap v2848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:30 smithi132 ceph-mon[37580]: pgmap v2848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:30 smithi112 ceph-mon[37501]: pgmap v2848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4406.842297 secs 2024-07-26T14:04:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4406.842297 secs 2024-07-26T14:04:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4406.842297 secs 2024-07-26T14:04:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:32 smithi107 ceph-mon[32659]: pgmap v2849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:32 smithi132 ceph-mon[37580]: pgmap v2849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:32 smithi112 ceph-mon[37501]: pgmap v2849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:04:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:34 smithi107 ceph-mon[32659]: pgmap v2850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:34 smithi132 ceph-mon[37580]: pgmap v2850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:34 smithi112 ceph-mon[37501]: pgmap v2850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4411.842399 secs 2024-07-26T14:04:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:36 smithi107 ceph-mon[32659]: pgmap v2851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4411.842399 secs 2024-07-26T14:04:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:36 smithi132 ceph-mon[37580]: pgmap v2851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4411.842399 secs 2024-07-26T14:04:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:36 smithi112 ceph-mon[37501]: pgmap v2851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: pgmap v2852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: pgmap v2852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: pgmap v2852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:39.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:39.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:39.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:04:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:39 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:04:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:39 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:04:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:39 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:04:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:39 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:04:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:39 smithi107 ceph-mon[32659]: pgmap v2853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:39 smithi107 ceph-mon[32659]: pgmap v2854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:04:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:39 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:04:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:39 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:04:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:39 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:04:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:39 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:04:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:39 smithi132 ceph-mon[37580]: pgmap v2853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:39 smithi132 ceph-mon[37580]: pgmap v2854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:04:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:39 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:04:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:39 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:04:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:39 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:04:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:39 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:04:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:39 smithi112 ceph-mon[37501]: pgmap v2853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:04:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:39 smithi112 ceph-mon[37501]: pgmap v2854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:04:40.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:40.784+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:40.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:40.784+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:40.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:40.784+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4416.842465 secs 2024-07-26T14:04:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:41 smithi107 ceph-mon[32659]: pgmap v2855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:04:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4416.842465 secs 2024-07-26T14:04:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:41 smithi132 ceph-mon[37580]: pgmap v2855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:04:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4416.842465 secs 2024-07-26T14:04:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:41 smithi112 ceph-mon[37501]: pgmap v2855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:04:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:42.871+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:42.871+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:42.871+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:43 smithi107 ceph-mon[32659]: pgmap v2856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:04:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:43 smithi132 ceph-mon[37580]: pgmap v2856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:04:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:43 smithi112 ceph-mon[37501]: pgmap v2856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:04:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:45 smithi107 ceph-mon[32659]: pgmap v2857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:45 smithi132 ceph-mon[37580]: pgmap v2857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:45 smithi112 ceph-mon[37501]: pgmap v2857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4421.842617 secs 2024-07-26T14:04:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4421.842617 secs 2024-07-26T14:04:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4421.842617 secs 2024-07-26T14:04:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:47 smithi107 ceph-mon[32659]: pgmap v2858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:04:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:47 smithi132 ceph-mon[37580]: pgmap v2858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:04:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:47 smithi112 ceph-mon[37501]: pgmap v2858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:04:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:49 smithi107 ceph-mon[32659]: pgmap v2859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:04:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:49 smithi132 ceph-mon[37580]: pgmap v2859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:04:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:49 smithi112 ceph-mon[37501]: pgmap v2859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:04:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4426.842764 secs 2024-07-26T14:04:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:51 smithi132 ceph-mon[37580]: pgmap v2860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:04:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4426.842764 secs 2024-07-26T14:04:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:51 smithi112 ceph-mon[37501]: pgmap v2860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:04:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4426.842764 secs 2024-07-26T14:04:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:51 smithi107 ceph-mon[32659]: pgmap v2860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:04:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:53 smithi132 ceph-mon[37580]: pgmap v2861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:04:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:53 smithi112 ceph-mon[37501]: pgmap v2861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:04:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:53 smithi107 ceph-mon[32659]: pgmap v2861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:04:55.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:55.800+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:55.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:55.800+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:55.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:04:55.800+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:55 smithi132 ceph-mon[37580]: pgmap v2862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4431.842889 secs 2024-07-26T14:04:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:55 smithi112 ceph-mon[37501]: pgmap v2862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4431.842889 secs 2024-07-26T14:04:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:55 smithi107 ceph-mon[32659]: pgmap v2862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:04:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4431.842889 secs 2024-07-26T14:04:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:57.881+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:57.881 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:57.881+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:04:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:04:57.881+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:57 smithi132 ceph-mon[37580]: pgmap v2863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:57 smithi112 ceph-mon[37501]: pgmap v2863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:57 smithi107 ceph-mon[32659]: pgmap v2863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:04:59 smithi112 ceph-mon[37501]: pgmap v2864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:04:59 smithi107 ceph-mon[32659]: pgmap v2864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:04:59 smithi132 ceph-mon[37580]: pgmap v2864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4436.843020 secs 2024-07-26T14:05:01.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4436.843020 secs 2024-07-26T14:05:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4436.843020 secs 2024-07-26T14:05:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:01 smithi112 ceph-mon[37501]: pgmap v2865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:01 smithi107 ceph-mon[32659]: pgmap v2865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:01 smithi132 ceph-mon[37580]: pgmap v2865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:04.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:03 smithi132 ceph-mon[37580]: pgmap v2866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:04.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:03 smithi112 ceph-mon[37501]: pgmap v2866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:03 smithi107 ceph-mon[32659]: pgmap v2866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:05 smithi112 ceph-mon[37501]: pgmap v2867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:05:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4441.843153 secs 2024-07-26T14:05:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:05 smithi107 ceph-mon[32659]: pgmap v2867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:05:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4441.843153 secs 2024-07-26T14:05:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:05 smithi132 ceph-mon[37580]: pgmap v2867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:05:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4441.843153 secs 2024-07-26T14:05:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:07 smithi112 ceph-mon[37501]: pgmap v2868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:07 smithi107 ceph-mon[32659]: pgmap v2868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:07 smithi132 ceph-mon[37580]: pgmap v2868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:09 smithi112 ceph-mon[37501]: pgmap v2869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:09 smithi107 ceph-mon[32659]: pgmap v2869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:09 smithi132 ceph-mon[37580]: pgmap v2869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:10.816 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:10.815+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:10.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:10.816+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:10.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:10.816+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:11.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4446.843277 secs 2024-07-26T14:05:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4446.843277 secs 2024-07-26T14:05:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4446.843277 secs 2024-07-26T14:05:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:11 smithi112 ceph-mon[37501]: pgmap v2870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:11 smithi107 ceph-mon[32659]: pgmap v2870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:11 smithi132 ceph-mon[37580]: pgmap v2870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:12.896+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:12.896+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:12.896+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:12 smithi112 ceph-mon[37501]: pgmap v2871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:12 smithi107 ceph-mon[32659]: pgmap v2871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:12 smithi132 ceph-mon[37580]: pgmap v2871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:15 smithi107 ceph-mon[32659]: pgmap v2872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:15 smithi132 ceph-mon[37580]: pgmap v2872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:15 smithi112 ceph-mon[37501]: pgmap v2872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4451.843365 secs 2024-07-26T14:05:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4451.843365 secs 2024-07-26T14:05:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4451.843365 secs 2024-07-26T14:05:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:17 smithi107 ceph-mon[32659]: pgmap v2873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:17 smithi132 ceph-mon[37580]: pgmap v2873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:17 smithi112 ceph-mon[37501]: pgmap v2873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:19 smithi107 ceph-mon[32659]: pgmap v2874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:19 smithi132 ceph-mon[37580]: pgmap v2874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:19 smithi112 ceph-mon[37501]: pgmap v2874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4456.843494 secs 2024-07-26T14:05:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:21 smithi107 ceph-mon[32659]: pgmap v2875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4456.843494 secs 2024-07-26T14:05:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:21 smithi132 ceph-mon[37580]: pgmap v2875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4456.843494 secs 2024-07-26T14:05:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:21 smithi112 ceph-mon[37501]: pgmap v2875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:23 smithi107 ceph-mon[32659]: pgmap v2876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:23 smithi132 ceph-mon[37580]: pgmap v2876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:24.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:23 smithi112 ceph-mon[37501]: pgmap v2876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:25.830+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:25.830+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:25.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:25.830+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:26.002 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:25 smithi112 ceph-mon[37501]: pgmap v2877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:26.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:25 smithi107 ceph-mon[32659]: pgmap v2877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:25 smithi132 ceph-mon[37580]: pgmap v2877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4461.843595 secs 2024-07-26T14:05:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4461.843595 secs 2024-07-26T14:05:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4461.843595 secs 2024-07-26T14:05:27.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:27.390+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:05:27.392 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:27.390+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:05:27.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:27.426+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:05:27.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:27.426+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:05:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:27.911+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:27.911+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:27.911+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:27 smithi107 ceph-mon[32659]: pgmap v2878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:05:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:27 smithi132 ceph-mon[37580]: pgmap v2878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:05:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:27 smithi112 ceph-mon[37501]: pgmap v2878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:05:29.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:29.536+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:05:29.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:29.536+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:05:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:29.537+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:05:29.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:29.537+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:05:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:29 smithi107 ceph-mon[32659]: pgmap v2879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:29 smithi132 ceph-mon[37580]: pgmap v2879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:29 smithi112 ceph-mon[37501]: pgmap v2879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4466.843677 secs 2024-07-26T14:05:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:31 smithi107 ceph-mon[32659]: pgmap v2880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4466.843677 secs 2024-07-26T14:05:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:31 smithi132 ceph-mon[37580]: pgmap v2880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4466.843677 secs 2024-07-26T14:05:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:31 smithi112 ceph-mon[37501]: pgmap v2880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:33 smithi107 ceph-mon[32659]: pgmap v2881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:33 smithi132 ceph-mon[37580]: pgmap v2881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:33 smithi112 ceph-mon[37501]: pgmap v2881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:35 smithi107 ceph-mon[32659]: pgmap v2882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:35 smithi132 ceph-mon[37580]: pgmap v2882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:35 smithi112 ceph-mon[37501]: pgmap v2882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4471.843805 secs 2024-07-26T14:05:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4471.843805 secs 2024-07-26T14:05:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4471.843805 secs 2024-07-26T14:05:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:37 smithi107 ceph-mon[32659]: pgmap v2883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:37 smithi132 ceph-mon[37580]: pgmap v2883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:37 smithi112 ceph-mon[37501]: pgmap v2883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:39 smithi132 ceph-mon[37580]: pgmap v2884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:39 smithi112 ceph-mon[37501]: pgmap v2884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:39 smithi107 ceph-mon[32659]: pgmap v2884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:40.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:40.834+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:40.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:40.834+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:40.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:40.834+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:40 smithi132 ceph-mon[37580]: pgmap v2885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:05:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:40 smithi132 ceph-mon[37580]: pgmap v2886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:41.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:41.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:40 smithi107 ceph-mon[32659]: pgmap v2885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:05:41.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:40 smithi107 ceph-mon[32659]: pgmap v2886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:41.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:40 smithi112 ceph-mon[37501]: pgmap v2885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:05:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:40 smithi112 ceph-mon[37501]: pgmap v2886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:05:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4476.843920 secs 2024-07-26T14:05:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4476.843920 secs 2024-07-26T14:05:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4476.843920 secs 2024-07-26T14:05:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:42.914+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:42.914+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:42.914+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:42 smithi112 ceph-mon[37501]: pgmap v2887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:42 smithi107 ceph-mon[32659]: pgmap v2887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:42 smithi132 ceph-mon[37580]: pgmap v2887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:44 smithi112 ceph-mon[37501]: pgmap v2888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:05:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:44 smithi107 ceph-mon[32659]: pgmap v2888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:05:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:44 smithi132 ceph-mon[37580]: pgmap v2888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:05:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4481.844042 secs 2024-07-26T14:05:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4481.844042 secs 2024-07-26T14:05:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4481.844042 secs 2024-07-26T14:05:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:46 smithi112 ceph-mon[37501]: pgmap v2889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:05:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:46 smithi107 ceph-mon[32659]: pgmap v2889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:05:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:46 smithi132 ceph-mon[37580]: pgmap v2889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:05:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:48 smithi112 ceph-mon[37501]: pgmap v2890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:05:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:48 smithi107 ceph-mon[32659]: pgmap v2890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:05:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:48 smithi132 ceph-mon[37580]: pgmap v2890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:05:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:50 smithi112 ceph-mon[37501]: pgmap v2891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4486.844152 secs 2024-07-26T14:05:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:50 smithi107 ceph-mon[32659]: pgmap v2891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4486.844152 secs 2024-07-26T14:05:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:50 smithi132 ceph-mon[37580]: pgmap v2891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4486.844152 secs 2024-07-26T14:05:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:52 smithi112 ceph-mon[37501]: pgmap v2892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:52 smithi107 ceph-mon[32659]: pgmap v2892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:52 smithi132 ceph-mon[37580]: pgmap v2892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:05:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:54 smithi112 ceph-mon[37501]: pgmap v2893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:54 smithi107 ceph-mon[32659]: pgmap v2893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:54 smithi132 ceph-mon[37580]: pgmap v2893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:55.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:55.835+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:55.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:55.835+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:55.837 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:05:55.835+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4491.844286 secs 2024-07-26T14:05:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4491.844286 secs 2024-07-26T14:05:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4491.844286 secs 2024-07-26T14:05:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:56 smithi112 ceph-mon[37501]: pgmap v2894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:56 smithi107 ceph-mon[32659]: pgmap v2894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:56 smithi132 ceph-mon[37580]: pgmap v2894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:57.930+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:57.930+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:05:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:05:57.930+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:05:57 smithi112 ceph-mon[37501]: pgmap v2895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:05:57 smithi107 ceph-mon[32659]: pgmap v2895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:05:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:05:57 smithi132 ceph-mon[37580]: pgmap v2895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:00 smithi112 ceph-mon[37501]: pgmap v2896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4496.844356 secs 2024-07-26T14:06:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:00 smithi107 ceph-mon[32659]: pgmap v2896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4496.844356 secs 2024-07-26T14:06:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:00 smithi132 ceph-mon[37580]: pgmap v2896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4496.844356 secs 2024-07-26T14:06:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:02 smithi112 ceph-mon[37501]: pgmap v2897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:06:03.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:02 smithi107 ceph-mon[32659]: pgmap v2897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:06:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:02 smithi132 ceph-mon[37580]: pgmap v2897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:06:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:03 smithi112 ceph-mon[37501]: pgmap v2898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:06:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:03 smithi107 ceph-mon[32659]: pgmap v2898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:06:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:03 smithi132 ceph-mon[37580]: pgmap v2898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:06:06.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4501.844481 secs 2024-07-26T14:06:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4501.844481 secs 2024-07-26T14:06:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4501.844481 secs 2024-07-26T14:06:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:06 smithi112 ceph-mon[37501]: pgmap v2899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:06:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:06 smithi107 ceph-mon[32659]: pgmap v2899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:06:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:06 smithi132 ceph-mon[37580]: pgmap v2899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:06:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:07 smithi112 ceph-mon[37501]: pgmap v2900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:06:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:07 smithi107 ceph-mon[32659]: pgmap v2900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:06:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:07 smithi132 ceph-mon[37580]: pgmap v2900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:06:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:10.851+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:10.851+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:10.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:10.851+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:10 smithi112 ceph-mon[37501]: pgmap v2901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4506.844611 secs 2024-07-26T14:06:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:10 smithi107 ceph-mon[32659]: pgmap v2901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4506.844611 secs 2024-07-26T14:06:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:10 smithi132 ceph-mon[37580]: pgmap v2901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4506.844611 secs 2024-07-26T14:06:11.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:11.649+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0c1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000257) for more than 900000000 us, fault. 2024-07-26T14:06:11.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:11.649+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c1e10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f802d040 comp rx=0 tx=0).stop 2024-07-26T14:06:11.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:11.649+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0c1e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:06:11.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:11.649+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c1e10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:06:11.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:11.650+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bfc40 0x7ff0ec0c0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:06:11.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:11.651+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bfc40 0x7ff0ec0c0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:06:11.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:11.652+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bfc40 0x7ff0ec0c0050 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fd30 tx=0x7ff0f801f040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:12.941+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:12.941+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:12.941+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:12 smithi112 ceph-mon[37501]: pgmap v2902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:06:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:12 smithi107 ceph-mon[32659]: pgmap v2902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:06:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:12 smithi132 ceph-mon[37580]: pgmap v2902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:06:13.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:13.399+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:06:13.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:13.399+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:06:13.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:13.433+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:06:13.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:13.434+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:06:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:13.757+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T14:06:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:13.757+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:06:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:13.757+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c27f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efc480063a0 tx=0x7efc680e6000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:06:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:13.757+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c27f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efc480063a0 tx=0x7efc680e6000 comp rx=0 tx=0).stop 2024-07-26T14:06:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:13.757+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c27f0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T14:06:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:13.757+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c27f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:06:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:13.757+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c27f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:06:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:13.757+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2270 0x7efc5c0c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:06:13.759 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:13.758+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2270 0x7efc5c0c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:06:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:13.760+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2270 0x7efc5c0c2d30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7efc6809b040 tx=0x7efc680a9000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:06:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:13 smithi112 ceph-mon[37501]: pgmap v2903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:13 smithi107 ceph-mon[32659]: pgmap v2903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:13 smithi132 ceph-mon[37580]: pgmap v2903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:14.546 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:14.545+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:06:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:14.545+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:06:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:14.546+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:06:14.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:14.546+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:06:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4511.844741 secs 2024-07-26T14:06:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4511.844741 secs 2024-07-26T14:06:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4511.844741 secs 2024-07-26T14:06:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:16 smithi112 ceph-mon[37501]: pgmap v2904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:16 smithi107 ceph-mon[32659]: pgmap v2904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:16 smithi132 ceph-mon[37580]: pgmap v2904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:17 smithi112 ceph-mon[37501]: pgmap v2905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:17 smithi107 ceph-mon[32659]: pgmap v2905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:17 smithi132 ceph-mon[37580]: pgmap v2905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:20 smithi112 ceph-mon[37501]: pgmap v2906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4516.844864 secs 2024-07-26T14:06:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:20 smithi107 ceph-mon[32659]: pgmap v2906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4516.844864 secs 2024-07-26T14:06:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:20 smithi132 ceph-mon[37580]: pgmap v2906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4516.844864 secs 2024-07-26T14:06:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:21 smithi112 ceph-mon[37501]: pgmap v2907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:21 smithi107 ceph-mon[32659]: pgmap v2907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:21 smithi132 ceph-mon[37580]: pgmap v2907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:24 smithi107 ceph-mon[32659]: pgmap v2908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:24 smithi132 ceph-mon[37580]: pgmap v2908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:24 smithi112 ceph-mon[37501]: pgmap v2908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:25.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:25.856+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:25.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:25.856+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:25.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:25.856+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4521.845007 secs 2024-07-26T14:06:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4521.845007 secs 2024-07-26T14:06:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4521.845007 secs 2024-07-26T14:06:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:26 smithi112 ceph-mon[37501]: pgmap v2909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:26 smithi107 ceph-mon[32659]: pgmap v2909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:26 smithi132 ceph-mon[37580]: pgmap v2909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:27.956 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:27.955+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:27.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:27.955+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:27.957 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:27.955+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:28 smithi112 ceph-mon[37501]: pgmap v2910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:28 smithi107 ceph-mon[32659]: pgmap v2910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:28 smithi132 ceph-mon[37580]: pgmap v2910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:30 smithi112 ceph-mon[37501]: pgmap v2911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4526.845137 secs 2024-07-26T14:06:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:30 smithi107 ceph-mon[32659]: pgmap v2911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4526.845137 secs 2024-07-26T14:06:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:30 smithi132 ceph-mon[37580]: pgmap v2911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4526.845137 secs 2024-07-26T14:06:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:32 smithi112 ceph-mon[37501]: pgmap v2912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:32 smithi107 ceph-mon[32659]: pgmap v2912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:32 smithi132 ceph-mon[37580]: pgmap v2912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:33 smithi112 ceph-mon[37501]: pgmap v2913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:33 smithi107 ceph-mon[32659]: pgmap v2913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:33 smithi132 ceph-mon[37580]: pgmap v2913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:36.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4531.845265 secs 2024-07-26T14:06:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4531.845265 secs 2024-07-26T14:06:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4531.845265 secs 2024-07-26T14:06:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:36 smithi112 ceph-mon[37501]: pgmap v2914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:36 smithi107 ceph-mon[32659]: pgmap v2914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:36 smithi132 ceph-mon[37580]: pgmap v2914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:38 smithi112 ceph-mon[37501]: pgmap v2915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:38 smithi107 ceph-mon[32659]: pgmap v2915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:38 smithi132 ceph-mon[37580]: pgmap v2915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:40.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:40.871+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:40.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:40.871+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:40.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:40.871+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:40 smithi112 ceph-mon[37501]: pgmap v2916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4536.845379 secs 2024-07-26T14:06:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:40 smithi107 ceph-mon[32659]: pgmap v2916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4536.845379 secs 2024-07-26T14:06:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:40 smithi132 ceph-mon[37580]: pgmap v2916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:06:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4536.845379 secs 2024-07-26T14:06:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:42.193 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:42.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:42.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:06:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:42.958+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:42.958+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:42.958+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:42 smithi112 ceph-mon[37501]: pgmap v2917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:06:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:42 smithi112 ceph-mon[37501]: pgmap v2918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:06:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:42 smithi107 ceph-mon[32659]: pgmap v2917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:06:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:42 smithi107 ceph-mon[32659]: pgmap v2918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:06:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:42 smithi132 ceph-mon[37580]: pgmap v2917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:06:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:42 smithi132 ceph-mon[37580]: pgmap v2918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:06:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:44 smithi107 ceph-mon[32659]: pgmap v2919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:44 smithi132 ceph-mon[37580]: pgmap v2919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:44 smithi112 ceph-mon[37501]: pgmap v2919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4541.845513 secs 2024-07-26T14:06:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:46 smithi107 ceph-mon[32659]: pgmap v2920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4541.845513 secs 2024-07-26T14:06:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:46 smithi132 ceph-mon[37580]: pgmap v2920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4541.845513 secs 2024-07-26T14:06:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:46 smithi112 ceph-mon[37501]: pgmap v2920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:48 smithi107 ceph-mon[32659]: pgmap v2921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:48 smithi132 ceph-mon[37580]: pgmap v2921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:48 smithi112 ceph-mon[37501]: pgmap v2921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:50 smithi107 ceph-mon[32659]: pgmap v2922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:50 smithi132 ceph-mon[37580]: pgmap v2922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:50 smithi112 ceph-mon[37501]: pgmap v2922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:06:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4546.845593 secs 2024-07-26T14:06:52.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4546.845593 secs 2024-07-26T14:06:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4546.845593 secs 2024-07-26T14:06:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:52 smithi107 ceph-mon[32659]: pgmap v2923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:06:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:52 smithi132 ceph-mon[37580]: pgmap v2923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:06:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:52 smithi112 ceph-mon[37501]: pgmap v2923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:06:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:54 smithi107 ceph-mon[32659]: pgmap v2924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:54 smithi132 ceph-mon[37580]: pgmap v2924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:54 smithi112 ceph-mon[37501]: pgmap v2924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:55.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:55.875+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:55.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:55.875+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:55.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:06:55.875+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4551.845725 secs 2024-07-26T14:06:57.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:56 smithi107 ceph-mon[32659]: pgmap v2925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:57.070 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4551.845725 secs 2024-07-26T14:06:57.070 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:56 smithi132 ceph-mon[37580]: pgmap v2925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4551.845725 secs 2024-07-26T14:06:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:56 smithi112 ceph-mon[37501]: pgmap v2925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:57.966+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:57.966+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:06:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:06:57.966+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:06:58 smithi132 ceph-mon[37580]: pgmap v2926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:06:58 smithi112 ceph-mon[37501]: pgmap v2926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:06:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:06:58 smithi107 ceph-mon[32659]: pgmap v2926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:00 smithi132 ceph-mon[37580]: pgmap v2927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:00 smithi112 ceph-mon[37501]: pgmap v2927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:00 smithi107 ceph-mon[32659]: pgmap v2927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4556.845851 secs 2024-07-26T14:07:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4556.845851 secs 2024-07-26T14:07:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4556.845851 secs 2024-07-26T14:07:03.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:02 smithi112 ceph-mon[37501]: pgmap v2928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:02 smithi107 ceph-mon[32659]: pgmap v2928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:02 smithi132 ceph-mon[37580]: pgmap v2928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:05.178 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:04 smithi107 ceph-mon[32659]: pgmap v2929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:04 smithi112 ceph-mon[37501]: pgmap v2929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:04 smithi132 ceph-mon[37580]: pgmap v2929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4561.845978 secs 2024-07-26T14:07:06.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4561.845978 secs 2024-07-26T14:07:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4561.845978 secs 2024-07-26T14:07:07.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:06 smithi112 ceph-mon[37501]: pgmap v2930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:06 smithi107 ceph-mon[32659]: pgmap v2930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:06 smithi132 ceph-mon[37580]: pgmap v2930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:08 smithi112 ceph-mon[37501]: pgmap v2931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:08 smithi107 ceph-mon[32659]: pgmap v2931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:08 smithi132 ceph-mon[37580]: pgmap v2931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:10.889 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:10.888+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:10.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:10.888+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:10.890 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:10.888+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:10 smithi112 ceph-mon[37501]: pgmap v2932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4566.846145 secs 2024-07-26T14:07:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:10 smithi107 ceph-mon[32659]: pgmap v2932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4566.846145 secs 2024-07-26T14:07:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:10 smithi132 ceph-mon[37580]: pgmap v2932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4566.846145 secs 2024-07-26T14:07:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:12.982+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:12.982+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:12.982+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:12 smithi112 ceph-mon[37501]: pgmap v2933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:12 smithi107 ceph-mon[32659]: pgmap v2933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:12 smithi132 ceph-mon[37580]: pgmap v2933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:14 smithi112 ceph-mon[37501]: pgmap v2934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:14 smithi107 ceph-mon[32659]: pgmap v2934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:14 smithi132 ceph-mon[37580]: pgmap v2934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4571.846277 secs 2024-07-26T14:07:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4571.846277 secs 2024-07-26T14:07:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4571.846277 secs 2024-07-26T14:07:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:16 smithi112 ceph-mon[37501]: pgmap v2935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:16 smithi107 ceph-mon[32659]: pgmap v2935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:16 smithi132 ceph-mon[37580]: pgmap v2935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:18 smithi112 ceph-mon[37501]: pgmap v2936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:18 smithi107 ceph-mon[32659]: pgmap v2936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:18 smithi132 ceph-mon[37580]: pgmap v2936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:20 smithi112 ceph-mon[37501]: pgmap v2937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4576.846413 secs 2024-07-26T14:07:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:20 smithi107 ceph-mon[32659]: pgmap v2937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4576.846413 secs 2024-07-26T14:07:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:20 smithi132 ceph-mon[37580]: pgmap v2937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4576.846413 secs 2024-07-26T14:07:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:22 smithi112 ceph-mon[37501]: pgmap v2938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:22 smithi107 ceph-mon[32659]: pgmap v2938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:22 smithi132 ceph-mon[37580]: pgmap v2938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:24 smithi112 ceph-mon[37501]: pgmap v2939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:07:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:24 smithi107 ceph-mon[32659]: pgmap v2939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:07:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:24 smithi132 ceph-mon[37580]: pgmap v2939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:07:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:25.890+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:25.890+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:25.890+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4581.846516 secs 2024-07-26T14:07:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:25 smithi112 ceph-mon[37501]: pgmap v2940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:07:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4581.846516 secs 2024-07-26T14:07:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:25 smithi107 ceph-mon[32659]: pgmap v2940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:07:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4581.846516 secs 2024-07-26T14:07:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:25 smithi132 ceph-mon[37580]: pgmap v2940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:07:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:27.995+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:27.995+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:27.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:27.995+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:28 smithi107 ceph-mon[32659]: pgmap v2941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:07:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:28 smithi132 ceph-mon[37580]: pgmap v2941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:07:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:28 smithi112 ceph-mon[37501]: pgmap v2941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:07:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:30 smithi107 ceph-mon[32659]: pgmap v2942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:30 smithi132 ceph-mon[37580]: pgmap v2942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:30 smithi112 ceph-mon[37501]: pgmap v2942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4586.846605 secs 2024-07-26T14:07:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4586.846605 secs 2024-07-26T14:07:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4586.846605 secs 2024-07-26T14:07:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:32 smithi107 ceph-mon[32659]: pgmap v2943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:07:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:32 smithi132 ceph-mon[37580]: pgmap v2943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:07:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:32 smithi112 ceph-mon[37501]: pgmap v2943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:07:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:34 smithi107 ceph-mon[32659]: pgmap v2944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:07:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:34 smithi132 ceph-mon[37580]: pgmap v2944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:07:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:34 smithi112 ceph-mon[37501]: pgmap v2944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:07:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4591.846753 secs 2024-07-26T14:07:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:36 smithi107 ceph-mon[32659]: pgmap v2945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4591.846753 secs 2024-07-26T14:07:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:36 smithi132 ceph-mon[37580]: pgmap v2945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4591.846753 secs 2024-07-26T14:07:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:36 smithi112 ceph-mon[37501]: pgmap v2945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:38 smithi107 ceph-mon[32659]: pgmap v2946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:38 smithi132 ceph-mon[37580]: pgmap v2946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:38 smithi112 ceph-mon[37501]: pgmap v2946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:40.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:40.906+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:40.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:40.906+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:40.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:40.906+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:40 smithi132 ceph-mon[37580]: pgmap v2947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:40 smithi112 ceph-mon[37501]: pgmap v2947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:40 smithi107 ceph-mon[32659]: pgmap v2947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4596.846817 secs 2024-07-26T14:07:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4596.846817 secs 2024-07-26T14:07:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4596.846817 secs 2024-07-26T14:07:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:43.010+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:43.010+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:43.010+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:43.057 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:42 smithi132 ceph-mon[37580]: pgmap v2948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:43.057 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:43.057 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.058 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.058 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.058 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:42 smithi107 ceph-mon[32659]: pgmap v2948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:43.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:43.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.146 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:42 smithi112 ceph-mon[37501]: pgmap v2948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:07:43.146 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:43.146 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.146 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.147 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:43.147 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:44 smithi107 ceph-mon[32659]: pgmap v2949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:44 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:44 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:44 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:44 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:44 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:44 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:44 smithi132 ceph-mon[37580]: pgmap v2949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:44 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:44 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:44 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:44 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:44 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:44 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:44 smithi112 ceph-mon[37501]: pgmap v2949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:07:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:44 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:44 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:44 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:44 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:44 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:44 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:47.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4601.846975 secs 2024-07-26T14:07:47.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:46 smithi132 ceph-mon[37580]: pgmap v2950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:47.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:47.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:47.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:47.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:47.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4601.846975 secs 2024-07-26T14:07:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:46 smithi112 ceph-mon[37501]: pgmap v2950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4601.846975 secs 2024-07-26T14:07:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:46 smithi107 ceph-mon[32659]: pgmap v2950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:07:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:47 smithi132 ceph-mon[37580]: pgmap v2951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:07:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:47 smithi112 ceph-mon[37501]: pgmap v2951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:07:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:47 smithi107 ceph-mon[32659]: pgmap v2951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:07:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:49 smithi112 ceph-mon[37501]: pgmap v2952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:07:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:49 smithi107 ceph-mon[32659]: pgmap v2952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:07:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:49 smithi132 ceph-mon[37580]: pgmap v2952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:07:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4606.847103 secs 2024-07-26T14:07:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:51 smithi112 ceph-mon[37501]: pgmap v2953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:07:52.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4606.847103 secs 2024-07-26T14:07:52.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:51 smithi107 ceph-mon[32659]: pgmap v2953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:07:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4606.847103 secs 2024-07-26T14:07:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:51 smithi132 ceph-mon[37580]: pgmap v2953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:07:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:53 smithi112 ceph-mon[37501]: pgmap v2954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T14:07:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:53 smithi107 ceph-mon[32659]: pgmap v2954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T14:07:54.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:53 smithi132 ceph-mon[37580]: pgmap v2954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T14:07:55.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:55.922+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:55.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:55.922+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:55.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:07:55.922+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:55 smithi112 ceph-mon[37501]: pgmap v2955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:07:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4611.847226 secs 2024-07-26T14:07:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:55 smithi107 ceph-mon[32659]: pgmap v2955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:07:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4611.847226 secs 2024-07-26T14:07:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:55 smithi132 ceph-mon[37580]: pgmap v2955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:07:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4611.847226 secs 2024-07-26T14:07:58.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:58.022+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:58.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:58.022+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:58.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:07:58.022+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:07:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:57 smithi112 ceph-mon[37501]: pgmap v2956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:07:58.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:57 smithi107 ceph-mon[32659]: pgmap v2956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:07:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:57 smithi132 ceph-mon[37580]: pgmap v2956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:08:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:07:59 smithi112 ceph-mon[37501]: pgmap v2957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:07:59 smithi107 ceph-mon[32659]: pgmap v2957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:07:59 smithi132 ceph-mon[37580]: pgmap v2957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4616.847353 secs 2024-07-26T14:08:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4616.847353 secs 2024-07-26T14:08:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4616.847353 secs 2024-07-26T14:08:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:01 smithi112 ceph-mon[37501]: pgmap v2958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:01 smithi107 ceph-mon[32659]: pgmap v2958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:01 smithi132 ceph-mon[37580]: pgmap v2958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:03 smithi112 ceph-mon[37501]: pgmap v2959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:03 smithi107 ceph-mon[32659]: pgmap v2959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:03 smithi132 ceph-mon[37580]: pgmap v2959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:06.128 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:05 smithi132 ceph-mon[37580]: pgmap v2960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:06.128 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4621.847489 secs 2024-07-26T14:08:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:05 smithi112 ceph-mon[37501]: pgmap v2960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4621.847489 secs 2024-07-26T14:08:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:05 smithi107 ceph-mon[32659]: pgmap v2960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4621.847489 secs 2024-07-26T14:08:08.154 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:07 smithi107 ceph-mon[32659]: pgmap v2961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:07 smithi112 ceph-mon[37501]: pgmap v2961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:07 smithi132 ceph-mon[37580]: pgmap v2961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:09 smithi112 ceph-mon[37501]: pgmap v2962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:09 smithi107 ceph-mon[32659]: pgmap v2962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:09 smithi132 ceph-mon[37580]: pgmap v2962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:10.936+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:10.936+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:10.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:10.936+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4626.847619 secs 2024-07-26T14:08:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4626.847619 secs 2024-07-26T14:08:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4626.847619 secs 2024-07-26T14:08:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:11 smithi112 ceph-mon[37501]: pgmap v2963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:11 smithi107 ceph-mon[32659]: pgmap v2963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:11 smithi132 ceph-mon[37580]: pgmap v2963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:13.032+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:13.032+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:13.032+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:13.193 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:12 smithi107 ceph-mon[32659]: pgmap v2964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:12 smithi112 ceph-mon[37501]: pgmap v2964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:12 smithi132 ceph-mon[37580]: pgmap v2964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:15 smithi112 ceph-mon[37501]: pgmap v2965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4631.847742 secs 2024-07-26T14:08:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:15 smithi107 ceph-mon[32659]: pgmap v2965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4631.847742 secs 2024-07-26T14:08:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:15 smithi132 ceph-mon[37580]: pgmap v2965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4631.847742 secs 2024-07-26T14:08:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:17 smithi112 ceph-mon[37501]: pgmap v2966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:17 smithi107 ceph-mon[32659]: pgmap v2966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:17 smithi132 ceph-mon[37580]: pgmap v2966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:19 smithi112 ceph-mon[37501]: pgmap v2967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:19 smithi107 ceph-mon[32659]: pgmap v2967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:19 smithi132 ceph-mon[37580]: pgmap v2967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4636.847867 secs 2024-07-26T14:08:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4636.847867 secs 2024-07-26T14:08:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4636.847867 secs 2024-07-26T14:08:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:21 smithi112 ceph-mon[37501]: pgmap v2968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:21 smithi107 ceph-mon[32659]: pgmap v2968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:21 smithi132 ceph-mon[37580]: pgmap v2968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:23 smithi112 ceph-mon[37501]: pgmap v2969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:23 smithi107 ceph-mon[32659]: pgmap v2969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:23 smithi132 ceph-mon[37580]: pgmap v2969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:25.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:25.939+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:25.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:25.939+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:25.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:25.939+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:25 smithi112 ceph-mon[37501]: pgmap v2970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4641.847993 secs 2024-07-26T14:08:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:25 smithi107 ceph-mon[32659]: pgmap v2970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4641.847993 secs 2024-07-26T14:08:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:25 smithi132 ceph-mon[37580]: pgmap v2970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4641.847993 secs 2024-07-26T14:08:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:28.047+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:28.047+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:28.047+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:27 smithi112 ceph-mon[37501]: pgmap v2971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:27 smithi107 ceph-mon[32659]: pgmap v2971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:27 smithi132 ceph-mon[37580]: pgmap v2971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:29 smithi112 ceph-mon[37501]: pgmap v2972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:29 smithi107 ceph-mon[32659]: pgmap v2972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:29 smithi132 ceph-mon[37580]: pgmap v2972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4646.848095 secs 2024-07-26T14:08:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4646.848095 secs 2024-07-26T14:08:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4646.848095 secs 2024-07-26T14:08:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:31 smithi112 ceph-mon[37501]: pgmap v2973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:31 smithi107 ceph-mon[32659]: pgmap v2973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:31 smithi132 ceph-mon[37580]: pgmap v2973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:32 smithi112 ceph-mon[37501]: pgmap v2974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:32 smithi107 ceph-mon[32659]: pgmap v2974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:32 smithi132 ceph-mon[37580]: pgmap v2974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:35 smithi112 ceph-mon[37501]: pgmap v2975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4651.848226 secs 2024-07-26T14:08:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:35 smithi107 ceph-mon[32659]: pgmap v2975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4651.848226 secs 2024-07-26T14:08:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:35 smithi132 ceph-mon[37580]: pgmap v2975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4651.848226 secs 2024-07-26T14:08:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:37 smithi112 ceph-mon[37501]: pgmap v2976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:37 smithi107 ceph-mon[32659]: pgmap v2976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:37 smithi132 ceph-mon[37580]: pgmap v2976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:39 smithi112 ceph-mon[37501]: pgmap v2977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:39 smithi107 ceph-mon[32659]: pgmap v2977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:39 smithi132 ceph-mon[37580]: pgmap v2977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:40.955+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:40.955+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:40.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:40.955+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4656.848370 secs 2024-07-26T14:08:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4656.848370 secs 2024-07-26T14:08:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4656.848370 secs 2024-07-26T14:08:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:41 smithi112 ceph-mon[37501]: pgmap v2978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:41 smithi107 ceph-mon[32659]: pgmap v2978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:41 smithi132 ceph-mon[37580]: pgmap v2978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:43.062+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:43.062+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:43.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:43.062+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:43 smithi112 ceph-mon[37501]: pgmap v2979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:43 smithi107 ceph-mon[32659]: pgmap v2979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:43 smithi132 ceph-mon[37580]: pgmap v2979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:08:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:45 smithi112 ceph-mon[37501]: pgmap v2980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4661.848457 secs 2024-07-26T14:08:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:45 smithi107 ceph-mon[32659]: pgmap v2980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4661.848457 secs 2024-07-26T14:08:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:45 smithi132 ceph-mon[37580]: pgmap v2980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4661.848457 secs 2024-07-26T14:08:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:47 smithi112 ceph-mon[37501]: pgmap v2981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:08:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:47 smithi107 ceph-mon[32659]: pgmap v2981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:08:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:47 smithi132 ceph-mon[37580]: pgmap v2981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:08:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:08:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:48 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:08:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:48 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:08:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:48 smithi112 ceph-mon[37501]: pgmap v2982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:08:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:48 smithi112 ceph-mon[37501]: pgmap v2983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:08:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:48 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:08:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:48 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:08:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:48 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:08:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:48 smithi107 ceph-mon[32659]: pgmap v2982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:08:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:48 smithi107 ceph-mon[32659]: pgmap v2983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:08:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:48 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:08:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:48 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:08:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:48 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:08:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:48 smithi132 ceph-mon[37580]: pgmap v2982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:08:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:48 smithi132 ceph-mon[37580]: pgmap v2983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:08:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:48 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:08:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:51 smithi112 ceph-mon[37501]: pgmap v2984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:08:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4666.848559 secs 2024-07-26T14:08:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:51 smithi107 ceph-mon[32659]: pgmap v2984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:08:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4666.848559 secs 2024-07-26T14:08:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:51 smithi132 ceph-mon[37580]: pgmap v2984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:08:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4666.848559 secs 2024-07-26T14:08:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:53 smithi112 ceph-mon[37501]: pgmap v2985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:53 smithi107 ceph-mon[32659]: pgmap v2985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:53 smithi132 ceph-mon[37580]: pgmap v2985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:55 smithi112 ceph-mon[37501]: pgmap v2986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:08:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:55 smithi107 ceph-mon[32659]: pgmap v2986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:08:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:55 smithi132 ceph-mon[37580]: pgmap v2986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:08:55.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:55.962+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:55.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:55.962+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:55.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:08:55.962+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4671.848710 secs 2024-07-26T14:08:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4671.848710 secs 2024-07-26T14:08:56.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4671.848710 secs 2024-07-26T14:08:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:57 smithi112 ceph-mon[37501]: pgmap v2987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:08:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:57 smithi107 ceph-mon[32659]: pgmap v2987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:08:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:57 smithi132 ceph-mon[37580]: pgmap v2987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:08:58.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:58.068+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:58.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:58.068+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:58.069 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:08:58.068+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:08:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:08:59 smithi112 ceph-mon[37501]: pgmap v2988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:08:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:08:59 smithi107 ceph-mon[32659]: pgmap v2988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:08:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:08:59 smithi132 ceph-mon[37580]: pgmap v2988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:09:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:01 smithi112 ceph-mon[37501]: pgmap v2989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4676.848818 secs 2024-07-26T14:09:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:01 smithi107 ceph-mon[32659]: pgmap v2989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4676.848818 secs 2024-07-26T14:09:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:01 smithi132 ceph-mon[37580]: pgmap v2989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4676.848818 secs 2024-07-26T14:09:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:03 smithi112 ceph-mon[37501]: pgmap v2990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:03 smithi107 ceph-mon[32659]: pgmap v2990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:03 smithi132 ceph-mon[37580]: pgmap v2990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:05 smithi112 ceph-mon[37501]: pgmap v2991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:05 smithi107 ceph-mon[32659]: pgmap v2991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:05 smithi132 ceph-mon[37580]: pgmap v2991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4681.848950 secs 2024-07-26T14:09:06.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4681.848950 secs 2024-07-26T14:09:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4681.848950 secs 2024-07-26T14:09:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:07 smithi112 ceph-mon[37501]: pgmap v2992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:07 smithi107 ceph-mon[32659]: pgmap v2992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:07 smithi132 ceph-mon[37580]: pgmap v2992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:09 smithi112 ceph-mon[37501]: pgmap v2993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:09 smithi107 ceph-mon[32659]: pgmap v2993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:09 smithi132 ceph-mon[37580]: pgmap v2993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:10.967+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:10.968 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:10.967+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:10.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:10.967+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:11 smithi107 ceph-mon[32659]: pgmap v2994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4686.849051 secs 2024-07-26T14:09:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:11 smithi112 ceph-mon[37501]: pgmap v2994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4686.849051 secs 2024-07-26T14:09:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:11 smithi132 ceph-mon[37580]: pgmap v2994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4686.849051 secs 2024-07-26T14:09:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:13.084+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:13.084+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:13.085 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:13.084+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:13.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:13.429+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:09:13.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:13 smithi107 ceph-mon[32659]: pgmap v2995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:13.431 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:13.430+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:09:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:13 smithi112 ceph-mon[37501]: pgmap v2995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:13.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:13.462+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:09:13.464 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:13.463+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:09:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:13 smithi132 ceph-mon[37580]: pgmap v2995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:14.579+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:09:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:14.579+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:09:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:14.580+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:09:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:14.580+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:09:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:15 smithi112 ceph-mon[37501]: pgmap v2996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:15 smithi107 ceph-mon[32659]: pgmap v2996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:15 smithi132 ceph-mon[37580]: pgmap v2996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4691.849087 secs 2024-07-26T14:09:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4691.849087 secs 2024-07-26T14:09:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4691.849087 secs 2024-07-26T14:09:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:17 smithi112 ceph-mon[37501]: pgmap v2997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:17 smithi107 ceph-mon[32659]: pgmap v2997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:17 smithi132 ceph-mon[37580]: pgmap v2997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:19 smithi112 ceph-mon[37501]: pgmap v2998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:19 smithi107 ceph-mon[32659]: pgmap v2998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:19 smithi132 ceph-mon[37580]: pgmap v2998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4696.849224 secs 2024-07-26T14:09:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4696.849224 secs 2024-07-26T14:09:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4696.849224 secs 2024-07-26T14:09:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:21 smithi112 ceph-mon[37501]: pgmap v2999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:21 smithi107 ceph-mon[32659]: pgmap v2999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:21 smithi132 ceph-mon[37580]: pgmap v2999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:23 smithi112 ceph-mon[37501]: pgmap v3000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:23 smithi107 ceph-mon[32659]: pgmap v3000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:23 smithi132 ceph-mon[37580]: pgmap v3000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:25.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:25.973+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:25.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:25.973+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:25.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:25.973+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:25 smithi112 ceph-mon[37501]: pgmap v3001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4701.849314 secs 2024-07-26T14:09:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:25 smithi107 ceph-mon[32659]: pgmap v3001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4701.849314 secs 2024-07-26T14:09:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:25 smithi132 ceph-mon[37580]: pgmap v3001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4701.849314 secs 2024-07-26T14:09:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:28.098+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:28.098+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:28.098+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:27 smithi112 ceph-mon[37501]: pgmap v3002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:27 smithi107 ceph-mon[32659]: pgmap v3002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:27 smithi132 ceph-mon[37580]: pgmap v3002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:29 smithi112 ceph-mon[37501]: pgmap v3003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:29 smithi107 ceph-mon[32659]: pgmap v3003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:29 smithi132 ceph-mon[37580]: pgmap v3003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4706.849443 secs 2024-07-26T14:09:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4706.849443 secs 2024-07-26T14:09:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4706.849443 secs 2024-07-26T14:09:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:31 smithi112 ceph-mon[37501]: pgmap v3004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:31 smithi107 ceph-mon[32659]: pgmap v3004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:31 smithi132 ceph-mon[37580]: pgmap v3004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:33 smithi112 ceph-mon[37501]: pgmap v3005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:33 smithi107 ceph-mon[32659]: pgmap v3005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:33 smithi132 ceph-mon[37580]: pgmap v3005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:35 smithi112 ceph-mon[37501]: pgmap v3006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4711.849594 secs 2024-07-26T14:09:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:35 smithi107 ceph-mon[32659]: pgmap v3006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4711.849594 secs 2024-07-26T14:09:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:35 smithi132 ceph-mon[37580]: pgmap v3006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4711.849594 secs 2024-07-26T14:09:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:37 smithi112 ceph-mon[37501]: pgmap v3007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:37 smithi107 ceph-mon[32659]: pgmap v3007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:37 smithi132 ceph-mon[37580]: pgmap v3007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:39 smithi112 ceph-mon[37501]: pgmap v3008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:39 smithi107 ceph-mon[32659]: pgmap v3008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:39 smithi132 ceph-mon[37580]: pgmap v3008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:40.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:40.989+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:40.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:40.989+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:40.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:40.989+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4716.849689 secs 2024-07-26T14:09:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4716.849689 secs 2024-07-26T14:09:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4716.849689 secs 2024-07-26T14:09:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:41 smithi112 ceph-mon[37501]: pgmap v3009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:41 smithi107 ceph-mon[32659]: pgmap v3009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:41 smithi132 ceph-mon[37580]: pgmap v3009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:43.115 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:43.114+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:43.115 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:43.114+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:43.116 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:43.114+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:42 smithi112 ceph-mon[37501]: pgmap v3010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:42 smithi107 ceph-mon[32659]: pgmap v3010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:42 smithi132 ceph-mon[37580]: pgmap v3010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:09:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:45 smithi112 ceph-mon[37501]: pgmap v3011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:45 smithi107 ceph-mon[32659]: pgmap v3011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:45 smithi132 ceph-mon[37580]: pgmap v3011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:46.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4721.849820 secs 2024-07-26T14:09:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4721.849820 secs 2024-07-26T14:09:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4721.849820 secs 2024-07-26T14:09:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:47 smithi112 ceph-mon[37501]: pgmap v3012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:47 smithi107 ceph-mon[32659]: pgmap v3012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:47 smithi132 ceph-mon[37580]: pgmap v3012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:49 smithi112 ceph-mon[37501]: pgmap v3013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:09:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:49 smithi107 ceph-mon[32659]: pgmap v3013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:09:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:49 smithi132 ceph-mon[37580]: pgmap v3013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:09:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:09:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:09:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:09:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:50 smithi112 ceph-mon[37501]: pgmap v3014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:09:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:50 smithi112 ceph-mon[37501]: pgmap v3015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:09:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:09:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:09:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:09:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:50 smithi107 ceph-mon[32659]: pgmap v3014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:09:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:50 smithi107 ceph-mon[32659]: pgmap v3015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:09:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:09:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:09:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:09:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:50 smithi132 ceph-mon[37580]: pgmap v3014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:09:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:50 smithi132 ceph-mon[37580]: pgmap v3015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:09:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:09:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4726.849970 secs 2024-07-26T14:09:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4726.849970 secs 2024-07-26T14:09:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4726.849970 secs 2024-07-26T14:09:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:52 smithi112 ceph-mon[37501]: pgmap v3016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:09:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:52 smithi107 ceph-mon[32659]: pgmap v3016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:09:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:52 smithi132 ceph-mon[37580]: pgmap v3016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:09:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:54 smithi112 ceph-mon[37501]: pgmap v3017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:09:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:54 smithi107 ceph-mon[32659]: pgmap v3017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:09:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:54 smithi132 ceph-mon[37580]: pgmap v3017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:09:56.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:56.005+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:56.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:56.005+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:56.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:09:56.005+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4731.850129 secs 2024-07-26T14:09:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:56 smithi112 ceph-mon[37501]: pgmap v3018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:09:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4731.850129 secs 2024-07-26T14:09:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:56 smithi107 ceph-mon[32659]: pgmap v3018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:09:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4731.850129 secs 2024-07-26T14:09:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:56 smithi132 ceph-mon[37580]: pgmap v3018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:09:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:58.130+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:58.130+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:58.131 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:09:58.130+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:09:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:09:58 smithi112 ceph-mon[37501]: pgmap v3019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:09:58 smithi107 ceph-mon[32659]: pgmap v3019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:09:58 smithi132 ceph-mon[37580]: pgmap v3019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:10:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:00 smithi112 ceph-mon[37501]: pgmap v3020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:10:00.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:10:00.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:00 smithi107 ceph-mon[32659]: pgmap v3020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:10:00.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:10:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:00 smithi132 ceph-mon[37580]: pgmap v3020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:10:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:10:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4736.850257 secs 2024-07-26T14:10:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4736.850257 secs 2024-07-26T14:10:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4736.850257 secs 2024-07-26T14:10:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:02 smithi112 ceph-mon[37501]: pgmap v3021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:02 smithi107 ceph-mon[32659]: pgmap v3021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:02 smithi132 ceph-mon[37580]: pgmap v3021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:04 smithi112 ceph-mon[37501]: pgmap v3022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:04 smithi107 ceph-mon[32659]: pgmap v3022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:04 smithi132 ceph-mon[37580]: pgmap v3022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4741.850389 secs 2024-07-26T14:10:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:06 smithi107 ceph-mon[32659]: pgmap v3023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4741.850389 secs 2024-07-26T14:10:06.740 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:06 smithi112 ceph-mon[37501]: pgmap v3023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4741.850389 secs 2024-07-26T14:10:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:06 smithi132 ceph-mon[37580]: pgmap v3023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:08 smithi112 ceph-mon[37501]: pgmap v3024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:08 smithi107 ceph-mon[32659]: pgmap v3024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:08 smithi132 ceph-mon[37580]: pgmap v3024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:10 smithi112 ceph-mon[37501]: pgmap v3025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:10 smithi107 ceph-mon[32659]: pgmap v3025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:10 smithi132 ceph-mon[37580]: pgmap v3025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:11.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:11.021+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:11.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:11.021+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:11.023 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:11.021+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4746.850546 secs 2024-07-26T14:10:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4746.850546 secs 2024-07-26T14:10:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4746.850546 secs 2024-07-26T14:10:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:12 smithi112 ceph-mon[37501]: pgmap v3026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:12 smithi107 ceph-mon[32659]: pgmap v3026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:12 smithi132 ceph-mon[37580]: pgmap v3026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:13.146+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:13.146+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:13.146+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:14 smithi112 ceph-mon[37501]: pgmap v3027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:14 smithi107 ceph-mon[32659]: pgmap v3027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:14 smithi132 ceph-mon[37580]: pgmap v3027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4751.850681 secs 2024-07-26T14:10:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:16 smithi112 ceph-mon[37501]: pgmap v3028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4751.850681 secs 2024-07-26T14:10:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:16 smithi107 ceph-mon[32659]: pgmap v3028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4751.850681 secs 2024-07-26T14:10:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:16 smithi132 ceph-mon[37580]: pgmap v3028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:18 smithi112 ceph-mon[37501]: pgmap v3029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:18 smithi107 ceph-mon[32659]: pgmap v3029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:18.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:18 smithi132 ceph-mon[37580]: pgmap v3029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:20 smithi112 ceph-mon[37501]: pgmap v3030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:20 smithi107 ceph-mon[32659]: pgmap v3030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:20 smithi132 ceph-mon[37580]: pgmap v3030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4756.850821 secs 2024-07-26T14:10:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4756.850821 secs 2024-07-26T14:10:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4756.850821 secs 2024-07-26T14:10:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:22 smithi112 ceph-mon[37501]: pgmap v3031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:22 smithi107 ceph-mon[32659]: pgmap v3031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:22 smithi132 ceph-mon[37580]: pgmap v3031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:24 smithi112 ceph-mon[37501]: pgmap v3032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:24 smithi107 ceph-mon[32659]: pgmap v3032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:24.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:24 smithi132 ceph-mon[37580]: pgmap v3032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:26.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:26.036+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:26.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:26.036+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:26.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:26.036+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4761.850949 secs 2024-07-26T14:10:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:26 smithi112 ceph-mon[37501]: pgmap v3033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4761.850949 secs 2024-07-26T14:10:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:26 smithi107 ceph-mon[32659]: pgmap v3033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4761.850949 secs 2024-07-26T14:10:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:26 smithi132 ceph-mon[37580]: pgmap v3033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:28.153+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:28.153+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:28.153+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:28.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:28 smithi107 ceph-mon[32659]: pgmap v3034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:28 smithi112 ceph-mon[37501]: pgmap v3034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:28 smithi132 ceph-mon[37580]: pgmap v3034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:30 smithi107 ceph-mon[32659]: pgmap v3035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:30 smithi132 ceph-mon[37580]: pgmap v3035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:30 smithi112 ceph-mon[37501]: pgmap v3035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4766.851104 secs 2024-07-26T14:10:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4766.851104 secs 2024-07-26T14:10:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4766.851104 secs 2024-07-26T14:10:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:32 smithi107 ceph-mon[32659]: pgmap v3036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:32 smithi132 ceph-mon[37580]: pgmap v3036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:32 smithi112 ceph-mon[37501]: pgmap v3036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:34 smithi107 ceph-mon[32659]: pgmap v3037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:34 smithi132 ceph-mon[37580]: pgmap v3037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:34 smithi112 ceph-mon[37501]: pgmap v3037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4771.851236 secs 2024-07-26T14:10:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:36 smithi107 ceph-mon[32659]: pgmap v3038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4771.851236 secs 2024-07-26T14:10:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:36 smithi132 ceph-mon[37580]: pgmap v3038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4771.851236 secs 2024-07-26T14:10:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:36 smithi112 ceph-mon[37501]: pgmap v3038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:38 smithi107 ceph-mon[32659]: pgmap v3039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:38 smithi132 ceph-mon[37580]: pgmap v3039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:38 smithi112 ceph-mon[37501]: pgmap v3039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:40 smithi107 ceph-mon[32659]: pgmap v3040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:40 smithi132 ceph-mon[37580]: pgmap v3040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:40 smithi112 ceph-mon[37501]: pgmap v3040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:41.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:41.042+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:41.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:41.042+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:41.043 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:41.042+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4776.851310 secs 2024-07-26T14:10:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4776.851310 secs 2024-07-26T14:10:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4776.851310 secs 2024-07-26T14:10:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:42 smithi107 ceph-mon[32659]: pgmap v3041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:42 smithi132 ceph-mon[37580]: pgmap v3041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:42 smithi112 ceph-mon[37501]: pgmap v3041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:43.170 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:43.169+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:43.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:43.169+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:43.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:43.169+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:44 smithi107 ceph-mon[32659]: pgmap v3042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:44 smithi132 ceph-mon[37580]: pgmap v3042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:44 smithi112 ceph-mon[37501]: pgmap v3042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4781.851408 secs 2024-07-26T14:10:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:46 smithi107 ceph-mon[32659]: pgmap v3043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4781.851408 secs 2024-07-26T14:10:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:46 smithi132 ceph-mon[37580]: pgmap v3043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4781.851408 secs 2024-07-26T14:10:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:46 smithi112 ceph-mon[37501]: pgmap v3043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:48 smithi107 ceph-mon[32659]: pgmap v3044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:48 smithi132 ceph-mon[37580]: pgmap v3044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:48 smithi112 ceph-mon[37501]: pgmap v3044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:10:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:10:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:10:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:10:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:50 smithi107 ceph-mon[32659]: pgmap v3045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:10:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:10:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:10:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:50 smithi132 ceph-mon[37580]: pgmap v3045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:10:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:10:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:10:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:50 smithi112 ceph-mon[37501]: pgmap v3045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:10:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:10:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:10:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:10:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4786.851559 secs 2024-07-26T14:10:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:51 smithi132 ceph-mon[37580]: pgmap v3046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:10:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:51 smithi132 ceph-mon[37580]: pgmap v3047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4786.851559 secs 2024-07-26T14:10:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:51 smithi112 ceph-mon[37501]: pgmap v3046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:10:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:51 smithi112 ceph-mon[37501]: pgmap v3047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4786.851559 secs 2024-07-26T14:10:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:51 smithi107 ceph-mon[32659]: pgmap v3046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:10:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:51 smithi107 ceph-mon[32659]: pgmap v3047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:53 smithi132 ceph-mon[37580]: pgmap v3048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:10:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:53 smithi112 ceph-mon[37501]: pgmap v3048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:10:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:53 smithi107 ceph-mon[32659]: pgmap v3048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:10:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:55 smithi132 ceph-mon[37580]: pgmap v3049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:55 smithi112 ceph-mon[37501]: pgmap v3049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:55 smithi107 ceph-mon[32659]: pgmap v3049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:56.058+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:56.058+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:56.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:10:56.058+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4791.851705 secs 2024-07-26T14:10:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4791.851705 secs 2024-07-26T14:10:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4791.851705 secs 2024-07-26T14:10:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:57 smithi112 ceph-mon[37501]: pgmap v3050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:10:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:57 smithi107 ceph-mon[32659]: pgmap v3050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:10:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:57 smithi132 ceph-mon[37580]: pgmap v3050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:10:58.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:58.185+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:58.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:58.185+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:10:58.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:10:58.185+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:10:59 smithi112 ceph-mon[37501]: pgmap v3051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:11:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:10:59 smithi107 ceph-mon[32659]: pgmap v3051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:11:00.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:10:59 smithi132 ceph-mon[37580]: pgmap v3051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:11:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4796.851847 secs 2024-07-26T14:11:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:01 smithi112 ceph-mon[37501]: pgmap v3052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:11:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4796.851847 secs 2024-07-26T14:11:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:01 smithi107 ceph-mon[32659]: pgmap v3052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:11:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4796.851847 secs 2024-07-26T14:11:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:01 smithi132 ceph-mon[37580]: pgmap v3052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:11:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:03 smithi112 ceph-mon[37501]: pgmap v3053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:03 smithi107 ceph-mon[32659]: pgmap v3053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:04.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:03 smithi132 ceph-mon[37580]: pgmap v3053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:05 smithi112 ceph-mon[37501]: pgmap v3054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:05 smithi107 ceph-mon[32659]: pgmap v3054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:05 smithi132 ceph-mon[37580]: pgmap v3054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4801.851969 secs 2024-07-26T14:11:06.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4801.851969 secs 2024-07-26T14:11:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4801.851969 secs 2024-07-26T14:11:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:07 smithi112 ceph-mon[37501]: pgmap v3055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:07 smithi107 ceph-mon[32659]: pgmap v3055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:07 smithi132 ceph-mon[37580]: pgmap v3055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:09 smithi112 ceph-mon[37501]: pgmap v3056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:09 smithi107 ceph-mon[32659]: pgmap v3056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:09 smithi132 ceph-mon[37580]: pgmap v3056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:11.059+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:11.059+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:11.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:11.059+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4806.852090 secs 2024-07-26T14:11:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:11 smithi112 ceph-mon[37501]: pgmap v3057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4806.852090 secs 2024-07-26T14:11:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:11 smithi107 ceph-mon[32659]: pgmap v3057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4806.852090 secs 2024-07-26T14:11:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:11 smithi132 ceph-mon[37580]: pgmap v3057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:13.201+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:13.201+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:13.201+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:13 smithi112 ceph-mon[37501]: pgmap v3058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:14.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:13 smithi107 ceph-mon[32659]: pgmap v3058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:13 smithi132 ceph-mon[37580]: pgmap v3058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:15 smithi112 ceph-mon[37501]: pgmap v3059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:15 smithi107 ceph-mon[32659]: pgmap v3059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:15 smithi132 ceph-mon[37580]: pgmap v3059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4811.852217 secs 2024-07-26T14:11:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4811.852217 secs 2024-07-26T14:11:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4811.852217 secs 2024-07-26T14:11:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:17 smithi112 ceph-mon[37501]: pgmap v3060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:11:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:17 smithi107 ceph-mon[32659]: pgmap v3060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:11:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:17 smithi132 ceph-mon[37580]: pgmap v3060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:11:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:19 smithi112 ceph-mon[37501]: pgmap v3061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:19 smithi107 ceph-mon[32659]: pgmap v3061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:19 smithi132 ceph-mon[37580]: pgmap v3061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4816.852343 secs 2024-07-26T14:11:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:21 smithi112 ceph-mon[37501]: pgmap v3062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:11:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4816.852343 secs 2024-07-26T14:11:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:21 smithi107 ceph-mon[32659]: pgmap v3062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:11:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4816.852343 secs 2024-07-26T14:11:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:21 smithi132 ceph-mon[37580]: pgmap v3062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:11:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:23 smithi112 ceph-mon[37501]: pgmap v3063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:11:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:23 smithi107 ceph-mon[32659]: pgmap v3063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:11:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:23 smithi132 ceph-mon[37580]: pgmap v3063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:11:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:25 smithi112 ceph-mon[37501]: pgmap v3064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:25 smithi107 ceph-mon[32659]: pgmap v3064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:25 smithi132 ceph-mon[37580]: pgmap v3064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:26.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:26.074+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:26.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:26.074+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:26.076 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:26.074+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4821.852484 secs 2024-07-26T14:11:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4821.852484 secs 2024-07-26T14:11:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4821.852484 secs 2024-07-26T14:11:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:27 smithi112 ceph-mon[37501]: pgmap v3065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:11:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:27 smithi107 ceph-mon[32659]: pgmap v3065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:11:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:27 smithi132 ceph-mon[37580]: pgmap v3065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:11:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:28.213+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:28.213+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:28.213+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:29 smithi112 ceph-mon[37501]: pgmap v3066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:29 smithi107 ceph-mon[32659]: pgmap v3066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:29 smithi132 ceph-mon[37580]: pgmap v3066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4826.852569 secs 2024-07-26T14:11:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:31 smithi112 ceph-mon[37501]: pgmap v3067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4826.852569 secs 2024-07-26T14:11:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:31 smithi107 ceph-mon[32659]: pgmap v3067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4826.852569 secs 2024-07-26T14:11:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:31 smithi132 ceph-mon[37580]: pgmap v3067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:33 smithi112 ceph-mon[37501]: pgmap v3068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:33 smithi107 ceph-mon[32659]: pgmap v3068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:33 smithi132 ceph-mon[37580]: pgmap v3068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:35 smithi107 ceph-mon[32659]: pgmap v3069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:35 smithi132 ceph-mon[37580]: pgmap v3069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:35 smithi112 ceph-mon[37501]: pgmap v3069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4831.852688 secs 2024-07-26T14:11:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4831.852688 secs 2024-07-26T14:11:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4831.852688 secs 2024-07-26T14:11:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:37 smithi107 ceph-mon[32659]: pgmap v3070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:37 smithi132 ceph-mon[37580]: pgmap v3070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:37 smithi112 ceph-mon[37501]: pgmap v3070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:39 smithi107 ceph-mon[32659]: pgmap v3071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:39 smithi132 ceph-mon[37580]: pgmap v3071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:39 smithi112 ceph-mon[37501]: pgmap v3071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:41.082+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:41.082+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:41.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:41.082+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4836.852825 secs 2024-07-26T14:11:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:41 smithi107 ceph-mon[32659]: pgmap v3072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4836.852825 secs 2024-07-26T14:11:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:41 smithi132 ceph-mon[37580]: pgmap v3072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4836.852825 secs 2024-07-26T14:11:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:41 smithi112 ceph-mon[37501]: pgmap v3072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:43.229+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:43.229+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:43.229+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:43 smithi107 ceph-mon[32659]: pgmap v3073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:43 smithi132 ceph-mon[37580]: pgmap v3073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:43 smithi112 ceph-mon[37501]: pgmap v3073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:45 smithi107 ceph-mon[32659]: pgmap v3074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:45 smithi132 ceph-mon[37580]: pgmap v3074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:45 smithi112 ceph-mon[37501]: pgmap v3074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:11:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4841.852947 secs 2024-07-26T14:11:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4841.852947 secs 2024-07-26T14:11:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4841.852947 secs 2024-07-26T14:11:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:47 smithi107 ceph-mon[32659]: pgmap v3075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:47 smithi132 ceph-mon[37580]: pgmap v3075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:47 smithi112 ceph-mon[37501]: pgmap v3075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:49 smithi107 ceph-mon[32659]: pgmap v3076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:49 smithi132 ceph-mon[37580]: pgmap v3076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:49 smithi112 ceph-mon[37501]: pgmap v3076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:11:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:11:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:11:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4846.853104 secs 2024-07-26T14:11:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:51 smithi132 ceph-mon[37580]: pgmap v3077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:11:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:11:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:11:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4846.853104 secs 2024-07-26T14:11:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:51 smithi112 ceph-mon[37501]: pgmap v3077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:11:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:11:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:11:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4846.853104 secs 2024-07-26T14:11:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:51 smithi107 ceph-mon[32659]: pgmap v3077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:11:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:11:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:11:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:52 smithi132 ceph-mon[37580]: pgmap v3078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:11:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:52 smithi132 ceph-mon[37580]: pgmap v3079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:52 smithi112 ceph-mon[37501]: pgmap v3078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:11:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:52 smithi112 ceph-mon[37501]: pgmap v3079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:52 smithi107 ceph-mon[32659]: pgmap v3078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:11:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:52 smithi107 ceph-mon[32659]: pgmap v3079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:54 smithi132 ceph-mon[37580]: pgmap v3080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:11:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:54 smithi112 ceph-mon[37501]: pgmap v3080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:11:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:54 smithi107 ceph-mon[32659]: pgmap v3080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:11:56.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:56.099+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:56.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:56.099+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:56.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:11:56.099+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4851.853233 secs 2024-07-26T14:11:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:56 smithi132 ceph-mon[37580]: pgmap v3081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:11:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4851.853233 secs 2024-07-26T14:11:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:56 smithi112 ceph-mon[37501]: pgmap v3081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:11:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4851.853233 secs 2024-07-26T14:11:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:56 smithi107 ceph-mon[32659]: pgmap v3081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:11:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:58.233+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:58.233+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:58.234 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:11:58.233+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:11:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:11:58 smithi112 ceph-mon[37501]: pgmap v3082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:11:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:11:58 smithi107 ceph-mon[32659]: pgmap v3082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:11:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:11:58 smithi132 ceph-mon[37580]: pgmap v3082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:12:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:00 smithi112 ceph-mon[37501]: pgmap v3083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:00 smithi107 ceph-mon[32659]: pgmap v3083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:00 smithi132 ceph-mon[37580]: pgmap v3083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:02.138 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4856.853323 secs 2024-07-26T14:12:02.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4856.853323 secs 2024-07-26T14:12:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4856.853323 secs 2024-07-26T14:12:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:02 smithi112 ceph-mon[37501]: pgmap v3084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:12:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:02 smithi107 ceph-mon[32659]: pgmap v3084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:12:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:02 smithi132 ceph-mon[37580]: pgmap v3084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:12:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:04 smithi112 ceph-mon[37501]: pgmap v3085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:04 smithi107 ceph-mon[32659]: pgmap v3085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:04 smithi132 ceph-mon[37580]: pgmap v3085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:07.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4861.853455 secs 2024-07-26T14:12:07.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:06 smithi132 ceph-mon[37580]: pgmap v3086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4861.853455 secs 2024-07-26T14:12:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:06 smithi112 ceph-mon[37501]: pgmap v3086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4861.853455 secs 2024-07-26T14:12:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:06 smithi107 ceph-mon[32659]: pgmap v3086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:08 smithi112 ceph-mon[37501]: pgmap v3087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:08 smithi107 ceph-mon[32659]: pgmap v3087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:08 smithi132 ceph-mon[37580]: pgmap v3087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:11.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:11.103+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:11.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:11.103+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:11.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:11.103+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:11.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:10 smithi107 ceph-mon[32659]: pgmap v3088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:12:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:10 smithi112 ceph-mon[37501]: pgmap v3088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:12:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:10 smithi132 ceph-mon[37580]: pgmap v3088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:12:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4866.853576 secs 2024-07-26T14:12:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4866.853576 secs 2024-07-26T14:12:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4866.853576 secs 2024-07-26T14:12:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:12 smithi112 ceph-mon[37501]: pgmap v3089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:12:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:13.249+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:13.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:13.249+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:13.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:13.249+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:12 smithi107 ceph-mon[32659]: pgmap v3089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:12:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:12 smithi132 ceph-mon[37580]: pgmap v3089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:12:13.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:13.460+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:12:13.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:13.461+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:12:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:13.492+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:12:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:13.492+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:12:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:14.612+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:12:14.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:14.612+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:12:14.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:14.612+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:12:14.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:14.613+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:12:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:14 smithi112 ceph-mon[37501]: pgmap v3090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:12:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:14 smithi107 ceph-mon[32659]: pgmap v3090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:12:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:14 smithi132 ceph-mon[37580]: pgmap v3090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:12:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4871.853726 secs 2024-07-26T14:12:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4871.853726 secs 2024-07-26T14:12:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4871.853726 secs 2024-07-26T14:12:17.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:16 smithi107 ceph-mon[32659]: pgmap v3091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:12:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:16 smithi112 ceph-mon[37501]: pgmap v3091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:12:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:16 smithi132 ceph-mon[37580]: pgmap v3091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:12:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:18 smithi112 ceph-mon[37501]: pgmap v3092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:12:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:18 smithi107 ceph-mon[32659]: pgmap v3092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:12:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:18 smithi132 ceph-mon[37580]: pgmap v3092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:12:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:20 smithi112 ceph-mon[37501]: pgmap v3093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4876.853862 secs 2024-07-26T14:12:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:20 smithi107 ceph-mon[32659]: pgmap v3093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4876.853862 secs 2024-07-26T14:12:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:20 smithi132 ceph-mon[37580]: pgmap v3093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4876.853862 secs 2024-07-26T14:12:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:22 smithi112 ceph-mon[37501]: pgmap v3094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:22 smithi107 ceph-mon[32659]: pgmap v3094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:22 smithi132 ceph-mon[37580]: pgmap v3094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:24 smithi112 ceph-mon[37501]: pgmap v3095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:24 smithi107 ceph-mon[32659]: pgmap v3095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:24 smithi132 ceph-mon[37580]: pgmap v3095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:26.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:26.118+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:26.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:26.118+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:26.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:26.118+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4881.854013 secs 2024-07-26T14:12:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4881.854013 secs 2024-07-26T14:12:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4881.854013 secs 2024-07-26T14:12:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:26 smithi112 ceph-mon[37501]: pgmap v3096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:26 smithi107 ceph-mon[32659]: pgmap v3096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:26 smithi132 ceph-mon[37580]: pgmap v3096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:28.263+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:28.263+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:28.263+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:28 smithi112 ceph-mon[37501]: pgmap v3097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:28 smithi107 ceph-mon[32659]: pgmap v3097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:28 smithi132 ceph-mon[37580]: pgmap v3097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:30 smithi107 ceph-mon[32659]: pgmap v3098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4886.854178 secs 2024-07-26T14:12:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:30 smithi132 ceph-mon[37580]: pgmap v3098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4886.854178 secs 2024-07-26T14:12:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:30 smithi112 ceph-mon[37501]: pgmap v3098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4886.854178 secs 2024-07-26T14:12:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:32 smithi107 ceph-mon[32659]: pgmap v3099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:32 smithi132 ceph-mon[37580]: pgmap v3099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:32 smithi112 ceph-mon[37501]: pgmap v3099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:34 smithi107 ceph-mon[32659]: pgmap v3100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:34 smithi132 ceph-mon[37580]: pgmap v3100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:34 smithi112 ceph-mon[37501]: pgmap v3100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4891.854324 secs 2024-07-26T14:12:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:36 smithi107 ceph-mon[32659]: pgmap v3101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4891.854324 secs 2024-07-26T14:12:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:36 smithi132 ceph-mon[37580]: pgmap v3101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4891.854324 secs 2024-07-26T14:12:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:36 smithi112 ceph-mon[37501]: pgmap v3101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:38 smithi107 ceph-mon[32659]: pgmap v3102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:38 smithi132 ceph-mon[37580]: pgmap v3102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:38 smithi112 ceph-mon[37501]: pgmap v3102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:40 smithi107 ceph-mon[32659]: pgmap v3103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:40 smithi132 ceph-mon[37580]: pgmap v3103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:41.134+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:41.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:41.134+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:41.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:41.134+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:40 smithi112 ceph-mon[37501]: pgmap v3103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4896.854415 secs 2024-07-26T14:12:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4896.854415 secs 2024-07-26T14:12:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4896.854415 secs 2024-07-26T14:12:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:42 smithi107 ceph-mon[32659]: pgmap v3104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:12:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:42 smithi132 ceph-mon[37580]: pgmap v3104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:12:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:42 smithi112 ceph-mon[37501]: pgmap v3104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:12:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:43.273+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:43.273+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:43.273+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:44 smithi107 ceph-mon[32659]: pgmap v3105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:44 smithi132 ceph-mon[37580]: pgmap v3105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:44 smithi112 ceph-mon[37501]: pgmap v3105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4901.854495 secs 2024-07-26T14:12:47.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:46 smithi132 ceph-mon[37580]: pgmap v3106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:12:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4901.854495 secs 2024-07-26T14:12:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:46 smithi112 ceph-mon[37501]: pgmap v3106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:12:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4901.854495 secs 2024-07-26T14:12:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:46 smithi107 ceph-mon[32659]: pgmap v3106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:12:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:48 smithi132 ceph-mon[37580]: pgmap v3107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:48 smithi112 ceph-mon[37501]: pgmap v3107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:48 smithi107 ceph-mon[32659]: pgmap v3107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:50 smithi132 ceph-mon[37580]: pgmap v3108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:12:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:50 smithi112 ceph-mon[37501]: pgmap v3108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:12:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:50 smithi107 ceph-mon[32659]: pgmap v3108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:12:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4906.854620 secs 2024-07-26T14:12:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4906.854620 secs 2024-07-26T14:12:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4906.854620 secs 2024-07-26T14:12:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:52 smithi112 ceph-mon[37501]: pgmap v3109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:12:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:12:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:52 smithi107 ceph-mon[32659]: pgmap v3109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:12:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:12:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:52 smithi132 ceph-mon[37580]: pgmap v3109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:12:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:12:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:12:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:12:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:53 smithi112 ceph-mon[37501]: pgmap v3110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:12:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:53 smithi112 ceph-mon[37501]: pgmap v3111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:12:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:12:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:12:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:12:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:53 smithi107 ceph-mon[32659]: pgmap v3110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:12:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:53 smithi107 ceph-mon[32659]: pgmap v3111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:12:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:12:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:12:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:12:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:53 smithi132 ceph-mon[37580]: pgmap v3110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:12:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:53 smithi132 ceph-mon[37580]: pgmap v3111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:12:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:12:56.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:56.143+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:56.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:56.143+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:56.145 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:12:56.143+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:55 smithi112 ceph-mon[37501]: pgmap v3112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4911.854752 secs 2024-07-26T14:12:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:55 smithi107 ceph-mon[32659]: pgmap v3112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4911.854752 secs 2024-07-26T14:12:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:55 smithi132 ceph-mon[37580]: pgmap v3112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4911.854752 secs 2024-07-26T14:12:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:57 smithi112 ceph-mon[37501]: pgmap v3113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:12:58.290 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:58.290+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:58.290+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:12:58.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:12:58.290+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:57 smithi107 ceph-mon[32659]: pgmap v3113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:12:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:57 smithi132 ceph-mon[37580]: pgmap v3113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:13:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:12:59 smithi112 ceph-mon[37501]: pgmap v3114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:13:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:12:59 smithi107 ceph-mon[32659]: pgmap v3114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:13:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:12:59 smithi132 ceph-mon[37580]: pgmap v3114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:13:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4916.854879 secs 2024-07-26T14:13:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4916.854879 secs 2024-07-26T14:13:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4916.854879 secs 2024-07-26T14:13:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:01 smithi112 ceph-mon[37501]: pgmap v3115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:13:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:01 smithi107 ceph-mon[32659]: pgmap v3115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:13:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:01 smithi132 ceph-mon[37580]: pgmap v3115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:13:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:03 smithi112 ceph-mon[37501]: pgmap v3116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:13:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:03 smithi107 ceph-mon[32659]: pgmap v3116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:13:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:03 smithi132 ceph-mon[37580]: pgmap v3116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:13:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:05 smithi107 ceph-mon[32659]: pgmap v3117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4921.855012 secs 2024-07-26T14:13:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:05 smithi132 ceph-mon[37580]: pgmap v3117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4921.855012 secs 2024-07-26T14:13:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:05 smithi112 ceph-mon[37501]: pgmap v3117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4921.855012 secs 2024-07-26T14:13:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:07 smithi107 ceph-mon[32659]: pgmap v3118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:07 smithi132 ceph-mon[37580]: pgmap v3118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:07 smithi112 ceph-mon[37501]: pgmap v3118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:09 smithi107 ceph-mon[32659]: pgmap v3119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:09 smithi132 ceph-mon[37580]: pgmap v3119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:09 smithi112 ceph-mon[37501]: pgmap v3119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:11.160 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:11.159+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:11.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:11.159+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:11.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:11.159+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4926.855142 secs 2024-07-26T14:13:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:10 smithi107 ceph-mon[32659]: pgmap v3120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4926.855142 secs 2024-07-26T14:13:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:10 smithi132 ceph-mon[37580]: pgmap v3120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4926.855142 secs 2024-07-26T14:13:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:10 smithi112 ceph-mon[37501]: pgmap v3120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:13.305+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:13.305+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:13.306 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:13.305+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:13 smithi112 ceph-mon[37501]: pgmap v3121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:13 smithi107 ceph-mon[32659]: pgmap v3121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:13 smithi132 ceph-mon[37580]: pgmap v3121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:15 smithi112 ceph-mon[37501]: pgmap v3122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4931.855245 secs 2024-07-26T14:13:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:15 smithi107 ceph-mon[32659]: pgmap v3122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4931.855245 secs 2024-07-26T14:13:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:15 smithi132 ceph-mon[37580]: pgmap v3122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4931.855245 secs 2024-07-26T14:13:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:17 smithi112 ceph-mon[37501]: pgmap v3123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:17 smithi107 ceph-mon[32659]: pgmap v3123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:17 smithi132 ceph-mon[37580]: pgmap v3123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:19 smithi112 ceph-mon[37501]: pgmap v3124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:19 smithi107 ceph-mon[32659]: pgmap v3124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:19 smithi132 ceph-mon[37580]: pgmap v3124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4936.855382 secs 2024-07-26T14:13:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4936.855382 secs 2024-07-26T14:13:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4936.855382 secs 2024-07-26T14:13:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:21 smithi107 ceph-mon[32659]: pgmap v3125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:21 smithi132 ceph-mon[37580]: pgmap v3125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:21 smithi112 ceph-mon[37501]: pgmap v3125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:23 smithi107 ceph-mon[32659]: pgmap v3126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:23 smithi132 ceph-mon[37580]: pgmap v3126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:23 smithi112 ceph-mon[37501]: pgmap v3126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:24 smithi107 ceph-mon[32659]: pgmap v3127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:24 smithi132 ceph-mon[37580]: pgmap v3127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:24 smithi112 ceph-mon[37501]: pgmap v3127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:26.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:26.159+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:26.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:26.159+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:26.161 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:26.159+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4941.855510 secs 2024-07-26T14:13:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4941.855510 secs 2024-07-26T14:13:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4941.855510 secs 2024-07-26T14:13:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:26 smithi107 ceph-mon[32659]: pgmap v3128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:26 smithi132 ceph-mon[37580]: pgmap v3128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:26 smithi112 ceph-mon[37501]: pgmap v3128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:28.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:28.314+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:28.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:28.314+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:28.316 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:28.314+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:29 smithi112 ceph-mon[37501]: pgmap v3129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:29 smithi107 ceph-mon[32659]: pgmap v3129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:29 smithi132 ceph-mon[37580]: pgmap v3129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4946.855620 secs 2024-07-26T14:13:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4946.855620 secs 2024-07-26T14:13:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4946.855620 secs 2024-07-26T14:13:32.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:31 smithi107 ceph-mon[32659]: pgmap v3130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:31 smithi132 ceph-mon[37580]: pgmap v3130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:31 smithi112 ceph-mon[37501]: pgmap v3130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:33 smithi107 ceph-mon[32659]: pgmap v3131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:33 smithi132 ceph-mon[37580]: pgmap v3131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:33 smithi112 ceph-mon[37501]: pgmap v3131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:35 smithi107 ceph-mon[32659]: pgmap v3132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4951.855767 secs 2024-07-26T14:13:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:35 smithi132 ceph-mon[37580]: pgmap v3132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4951.855767 secs 2024-07-26T14:13:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:35 smithi112 ceph-mon[37501]: pgmap v3132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4951.855767 secs 2024-07-26T14:13:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:36 smithi107 ceph-mon[32659]: pgmap v3133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:36 smithi132 ceph-mon[37580]: pgmap v3133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:36 smithi112 ceph-mon[37501]: pgmap v3133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:39 smithi112 ceph-mon[37501]: pgmap v3134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:39 smithi107 ceph-mon[32659]: pgmap v3134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:39 smithi132 ceph-mon[37580]: pgmap v3134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:41.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:41.174+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:41.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:41.174+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:41.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:41.174+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4956.855911 secs 2024-07-26T14:13:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4956.855911 secs 2024-07-26T14:13:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4956.855911 secs 2024-07-26T14:13:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:41 smithi107 ceph-mon[32659]: pgmap v3135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:41 smithi132 ceph-mon[37580]: pgmap v3135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:41 smithi112 ceph-mon[37501]: pgmap v3135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:42.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:42.475+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:13:42.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:42.475+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:13:42.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:42.505+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:13:42.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:42.506+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:13:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:43.329+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:43.329+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:43.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:43.329+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:43 smithi107 ceph-mon[32659]: pgmap v3136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:43 smithi132 ceph-mon[37580]: pgmap v3136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:43 smithi112 ceph-mon[37501]: pgmap v3136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:44.628 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:44.628+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:13:44.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:44.628+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:13:44.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:44.628+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:13:44.629 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:44.628+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:13:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:45 smithi107 ceph-mon[32659]: pgmap v3137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4961.856018 secs 2024-07-26T14:13:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:45 smithi132 ceph-mon[37580]: pgmap v3137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4961.856018 secs 2024-07-26T14:13:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:45 smithi112 ceph-mon[37501]: pgmap v3137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4961.856018 secs 2024-07-26T14:13:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:46 smithi107 ceph-mon[32659]: pgmap v3138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:46 smithi132 ceph-mon[37580]: pgmap v3138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:46 smithi112 ceph-mon[37501]: pgmap v3138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:49 smithi107 ceph-mon[32659]: pgmap v3139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:49 smithi132 ceph-mon[37580]: pgmap v3139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:49 smithi112 ceph-mon[37501]: pgmap v3139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4966.856143 secs 2024-07-26T14:13:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4966.856143 secs 2024-07-26T14:13:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4966.856143 secs 2024-07-26T14:13:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:51 smithi107 ceph-mon[32659]: pgmap v3140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:51 smithi132 ceph-mon[37580]: pgmap v3140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:51 smithi112 ceph-mon[37501]: pgmap v3140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:53.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:53 smithi107 ceph-mon[32659]: pgmap v3141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:53 smithi132 ceph-mon[37580]: pgmap v3141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:53 smithi112 ceph-mon[37501]: pgmap v3141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:13:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:13:54.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:13:55.148 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:55 smithi107 ceph-mon[32659]: pgmap v3142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:55 smithi132 ceph-mon[37580]: pgmap v3142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:55.403 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:55 smithi112 ceph-mon[37501]: pgmap v3142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:56.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:56.181+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:56.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:56.181+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:56.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:13:56.181+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:56.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4971.856271 secs 2024-07-26T14:13:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4971.856271 secs 2024-07-26T14:13:56.436 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4971.856271 secs 2024-07-26T14:13:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:57 smithi112 ceph-mon[37501]: pgmap v3143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:13:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:13:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:57 smithi107 ceph-mon[32659]: pgmap v3143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:13:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:13:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:57 smithi132 ceph-mon[37580]: pgmap v3143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:13:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:13:58.345 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:58.345+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:58.345+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:13:58.346 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:13:58.345+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: pgmap v3144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:14:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:14:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:13:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:14:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: pgmap v3144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:14:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:14:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:14:00.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:14:00.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:13:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: pgmap v3144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:14:00.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:14:00.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:13:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:14:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:00 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:14:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:00 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:14:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:00 smithi112 ceph-mon[37501]: pgmap v3145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:14:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:00 smithi112 ceph-mon[37501]: pgmap v3146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T14:14:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4976.856401 secs 2024-07-26T14:14:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:00 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:14:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:00 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:14:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:00 smithi107 ceph-mon[32659]: pgmap v3145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:14:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:00 smithi107 ceph-mon[32659]: pgmap v3146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T14:14:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4976.856401 secs 2024-07-26T14:14:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:00 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:14:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:00 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:14:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:00 smithi132 ceph-mon[37580]: pgmap v3145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:14:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:00 smithi132 ceph-mon[37580]: pgmap v3146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T14:14:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4976.856401 secs 2024-07-26T14:14:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:02 smithi112 ceph-mon[37501]: pgmap v3147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:14:03.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:02 smithi107 ceph-mon[32659]: pgmap v3147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:14:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:02 smithi132 ceph-mon[37580]: pgmap v3147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:14:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:04 smithi112 ceph-mon[37501]: pgmap v3148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T14:14:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:04 smithi107 ceph-mon[32659]: pgmap v3148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T14:14:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:04 smithi132 ceph-mon[37580]: pgmap v3148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T14:14:06.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4981.856500 secs 2024-07-26T14:14:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4981.856500 secs 2024-07-26T14:14:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4981.856500 secs 2024-07-26T14:14:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:06 smithi112 ceph-mon[37501]: pgmap v3149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:07.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:06 smithi107 ceph-mon[32659]: pgmap v3149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:06 smithi132 ceph-mon[37580]: pgmap v3149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:08 smithi107 ceph-mon[32659]: pgmap v3150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:14:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:08 smithi112 ceph-mon[37501]: pgmap v3150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:14:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:08 smithi132 ceph-mon[37580]: pgmap v3150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:14:11.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:11.197+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:11.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:11.197+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:11.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:11.197+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:10 smithi107 ceph-mon[32659]: pgmap v3151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4986.856635 secs 2024-07-26T14:14:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:10 smithi132 ceph-mon[37580]: pgmap v3151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4986.856635 secs 2024-07-26T14:14:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:10 smithi112 ceph-mon[37501]: pgmap v3151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4986.856635 secs 2024-07-26T14:14:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:12 smithi107 ceph-mon[32659]: pgmap v3152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:12 smithi132 ceph-mon[37580]: pgmap v3152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:13.359+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:13.359+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:13.360 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:13.359+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:12 smithi112 ceph-mon[37501]: pgmap v3152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:14 smithi107 ceph-mon[32659]: pgmap v3153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:14 smithi132 ceph-mon[37580]: pgmap v3153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:14 smithi112 ceph-mon[37501]: pgmap v3153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4991.856770 secs 2024-07-26T14:14:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4991.856770 secs 2024-07-26T14:14:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4991.856770 secs 2024-07-26T14:14:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:16 smithi107 ceph-mon[32659]: pgmap v3154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:16 smithi132 ceph-mon[37580]: pgmap v3154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:16 smithi112 ceph-mon[37501]: pgmap v3154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:18.243 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:17 smithi107 ceph-mon[32659]: pgmap v3155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:17 smithi132 ceph-mon[37580]: pgmap v3155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:17 smithi112 ceph-mon[37501]: pgmap v3155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:20 smithi107 ceph-mon[32659]: pgmap v3156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:20 smithi132 ceph-mon[37580]: pgmap v3156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:20 smithi112 ceph-mon[37501]: pgmap v3156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 4996.856898 secs 2024-07-26T14:14:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 4996.856898 secs 2024-07-26T14:14:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 4996.856898 secs 2024-07-26T14:14:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:22 smithi107 ceph-mon[32659]: pgmap v3157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:14:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:22 smithi132 ceph-mon[37580]: pgmap v3157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:14:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:22 smithi112 ceph-mon[37501]: pgmap v3157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:14:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:24 smithi107 ceph-mon[32659]: pgmap v3158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:24 smithi132 ceph-mon[37580]: pgmap v3158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:24 smithi112 ceph-mon[37501]: pgmap v3158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:26.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:26.212+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:26.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:26.212+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:26.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:26.212+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5001.857025 secs 2024-07-26T14:14:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:26 smithi107 ceph-mon[32659]: pgmap v3159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5001.857025 secs 2024-07-26T14:14:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:26 smithi132 ceph-mon[37580]: pgmap v3159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5001.857025 secs 2024-07-26T14:14:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:26 smithi112 ceph-mon[37501]: pgmap v3159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:28.362+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:28.362+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:28.362+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:28 smithi132 ceph-mon[37580]: pgmap v3160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:28 smithi112 ceph-mon[37501]: pgmap v3160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:28 smithi107 ceph-mon[32659]: pgmap v3160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:30 smithi132 ceph-mon[37580]: pgmap v3161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:30 smithi112 ceph-mon[37501]: pgmap v3161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:30 smithi107 ceph-mon[32659]: pgmap v3161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5006.857155 secs 2024-07-26T14:14:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5006.857155 secs 2024-07-26T14:14:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5006.857155 secs 2024-07-26T14:14:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:32 smithi112 ceph-mon[37501]: pgmap v3162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:32 smithi107 ceph-mon[32659]: pgmap v3162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:32 smithi132 ceph-mon[37580]: pgmap v3162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:34 smithi112 ceph-mon[37501]: pgmap v3163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:34 smithi107 ceph-mon[32659]: pgmap v3163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:34 smithi132 ceph-mon[37580]: pgmap v3163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5011.857284 secs 2024-07-26T14:14:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:36 smithi112 ceph-mon[37501]: pgmap v3164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:37.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5011.857284 secs 2024-07-26T14:14:37.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:36 smithi107 ceph-mon[32659]: pgmap v3164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5011.857284 secs 2024-07-26T14:14:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:36 smithi132 ceph-mon[37580]: pgmap v3164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:38 smithi112 ceph-mon[37501]: pgmap v3165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:38 smithi107 ceph-mon[32659]: pgmap v3165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:38 smithi132 ceph-mon[37580]: pgmap v3165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:41.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:41.228+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:41.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:41.228+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:41.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:41.228+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:41.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:40 smithi107 ceph-mon[32659]: pgmap v3166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:41.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5016.857423 secs 2024-07-26T14:14:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:40 smithi132 ceph-mon[37580]: pgmap v3166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5016.857423 secs 2024-07-26T14:14:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:40 smithi112 ceph-mon[37501]: pgmap v3166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5016.857423 secs 2024-07-26T14:14:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:42 smithi107 ceph-mon[32659]: pgmap v3167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:42 smithi132 ceph-mon[37580]: pgmap v3167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:43.378+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:43.378+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:43.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:43.378+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:42 smithi112 ceph-mon[37501]: pgmap v3167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:44 smithi107 ceph-mon[32659]: pgmap v3168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:44 smithi132 ceph-mon[37580]: pgmap v3168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:44 smithi112 ceph-mon[37501]: pgmap v3168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5021.857554 secs 2024-07-26T14:14:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5021.857554 secs 2024-07-26T14:14:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5021.857554 secs 2024-07-26T14:14:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:46 smithi107 ceph-mon[32659]: pgmap v3169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:46 smithi132 ceph-mon[37580]: pgmap v3169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:46 smithi112 ceph-mon[37501]: pgmap v3169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:48 smithi107 ceph-mon[32659]: pgmap v3170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:48 smithi132 ceph-mon[37580]: pgmap v3170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:48 smithi112 ceph-mon[37501]: pgmap v3170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:50 smithi107 ceph-mon[32659]: pgmap v3171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:50 smithi132 ceph-mon[37580]: pgmap v3171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:50 smithi112 ceph-mon[37501]: pgmap v3171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5026.857680 secs 2024-07-26T14:14:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5026.857680 secs 2024-07-26T14:14:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5026.857680 secs 2024-07-26T14:14:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:52 smithi132 ceph-mon[37580]: pgmap v3172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:53.146 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:52 smithi107 ceph-mon[32659]: pgmap v3172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:52 smithi112 ceph-mon[37501]: pgmap v3172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:14:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:54 smithi132 ceph-mon[37580]: pgmap v3173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:54 smithi112 ceph-mon[37501]: pgmap v3173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:54 smithi107 ceph-mon[32659]: pgmap v3173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:56.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:56.238+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:56.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:56.238+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:56.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:14:56.239+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5031.857810 secs 2024-07-26T14:14:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:56 smithi132 ceph-mon[37580]: pgmap v3174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5031.857810 secs 2024-07-26T14:14:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:56 smithi112 ceph-mon[37501]: pgmap v3174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5031.857810 secs 2024-07-26T14:14:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:56 smithi107 ceph-mon[32659]: pgmap v3174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:58.392+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:58.392+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:14:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:14:58.392+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:14:58 smithi112 ceph-mon[37501]: pgmap v3175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:14:58 smithi107 ceph-mon[32659]: pgmap v3175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:14:59.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:14:58 smithi132 ceph-mon[37580]: pgmap v3175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:01.123 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:00 smithi132 ceph-mon[37580]: pgmap v3176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:01.124 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:15:01.124 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.124 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.124 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.124 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.124 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.124 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.124 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.124 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:00 smithi112 ceph-mon[37501]: pgmap v3176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:15:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:00 smithi107 ceph-mon[32659]: pgmap v3176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:15:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:02.178 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5036.857898 secs 2024-07-26T14:15:02.179 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:01 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:15:02.179 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:01 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:15:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5036.857898 secs 2024-07-26T14:15:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:01 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:15:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:01 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:15:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5036.857898 secs 2024-07-26T14:15:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:01 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:15:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:01 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:15:03.082 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:02 smithi132 ceph-mon[37580]: pgmap v3177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:02 smithi112 ceph-mon[37501]: pgmap v3177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:02 smithi107 ceph-mon[32659]: pgmap v3177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: pgmap v3178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:15:05.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:15:05.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:15:05.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:15:05.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: pgmap v3179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:15:05.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: pgmap v3180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:15:05.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:15:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: pgmap v3178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:15:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:15:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:15:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:15:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:15:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:15:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: pgmap v3179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:15:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: pgmap v3180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:15:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:15:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: pgmap v3178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:15:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:15:05.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:15:05.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: pgmap v3179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:15:05.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: pgmap v3180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:15:05.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:15:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5041.858019 secs 2024-07-26T14:15:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5041.858019 secs 2024-07-26T14:15:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5041.858019 secs 2024-07-26T14:15:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:06 smithi112 ceph-mon[37501]: pgmap v3181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:15:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:06 smithi107 ceph-mon[32659]: pgmap v3181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:15:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:06 smithi132 ceph-mon[37580]: pgmap v3181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:15:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:08 smithi112 ceph-mon[37501]: pgmap v3182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:15:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:08 smithi107 ceph-mon[32659]: pgmap v3182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:15:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:08 smithi132 ceph-mon[37580]: pgmap v3182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:15:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:11.254+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:11.254+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:11.254+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:11.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:10 smithi107 ceph-mon[32659]: pgmap v3183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T14:15:11.256 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5046.858148 secs 2024-07-26T14:15:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:10 smithi132 ceph-mon[37580]: pgmap v3183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T14:15:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5046.858148 secs 2024-07-26T14:15:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:10 smithi112 ceph-mon[37501]: pgmap v3183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T14:15:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5046.858148 secs 2024-07-26T14:15:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:12 smithi107 ceph-mon[32659]: pgmap v3184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:15:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:12 smithi132 ceph-mon[37580]: pgmap v3184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:15:13.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:13.407+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:13.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:13.407+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:13.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:13.407+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:12 smithi112 ceph-mon[37501]: pgmap v3184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:15:13.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:13.491+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:15:13.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:13.491+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:15:13.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:13.520+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:15:13.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:13.521+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:15:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:14.644+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:15:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:14.644+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:15:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:14.645+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:15:14.645 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:14.645+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:15:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:14 smithi107 ceph-mon[32659]: pgmap v3185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:15:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:14 smithi132 ceph-mon[37580]: pgmap v3185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:15:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:14 smithi112 ceph-mon[37501]: pgmap v3185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:15:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5051.858297 secs 2024-07-26T14:15:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5051.858297 secs 2024-07-26T14:15:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5051.858297 secs 2024-07-26T14:15:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:16 smithi107 ceph-mon[32659]: pgmap v3186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:16 smithi132 ceph-mon[37580]: pgmap v3186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:16 smithi112 ceph-mon[37501]: pgmap v3186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:18 smithi107 ceph-mon[32659]: pgmap v3187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:18 smithi132 ceph-mon[37580]: pgmap v3187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:18 smithi112 ceph-mon[37501]: pgmap v3187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:20 smithi107 ceph-mon[32659]: pgmap v3188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5056.858415 secs 2024-07-26T14:15:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:20 smithi132 ceph-mon[37580]: pgmap v3188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5056.858415 secs 2024-07-26T14:15:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:20 smithi112 ceph-mon[37501]: pgmap v3188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5056.858415 secs 2024-07-26T14:15:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:22 smithi107 ceph-mon[32659]: pgmap v3189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:22 smithi132 ceph-mon[37580]: pgmap v3189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:22 smithi112 ceph-mon[37501]: pgmap v3189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:24 smithi112 ceph-mon[37501]: pgmap v3190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:24 smithi107 ceph-mon[32659]: pgmap v3190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:24 smithi132 ceph-mon[37580]: pgmap v3190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5061.858570 secs 2024-07-26T14:15:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:26.269+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:26.269+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:26.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:26.269+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:26.271 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5061.858570 secs 2024-07-26T14:15:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5061.858570 secs 2024-07-26T14:15:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:26 smithi112 ceph-mon[37501]: pgmap v3191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:26 smithi107 ceph-mon[32659]: pgmap v3191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:26 smithi132 ceph-mon[37580]: pgmap v3191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:28.421+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:28.421+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:28.423 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:28.421+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:28 smithi112 ceph-mon[37501]: pgmap v3192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:28 smithi107 ceph-mon[32659]: pgmap v3192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:28 smithi132 ceph-mon[37580]: pgmap v3192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:30 smithi112 ceph-mon[37501]: pgmap v3193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5066.858672 secs 2024-07-26T14:15:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:30 smithi107 ceph-mon[32659]: pgmap v3193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5066.858672 secs 2024-07-26T14:15:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:30 smithi132 ceph-mon[37580]: pgmap v3193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5066.858672 secs 2024-07-26T14:15:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:32 smithi112 ceph-mon[37501]: pgmap v3194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:32 smithi107 ceph-mon[32659]: pgmap v3194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:32 smithi132 ceph-mon[37580]: pgmap v3194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:34 smithi107 ceph-mon[32659]: pgmap v3195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:34 smithi132 ceph-mon[37580]: pgmap v3195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:34 smithi112 ceph-mon[37501]: pgmap v3195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5071.858795 secs 2024-07-26T14:15:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5071.858795 secs 2024-07-26T14:15:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5071.858795 secs 2024-07-26T14:15:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:36 smithi107 ceph-mon[32659]: pgmap v3196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:36 smithi132 ceph-mon[37580]: pgmap v3196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:36 smithi112 ceph-mon[37501]: pgmap v3196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:39.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:38 smithi107 ceph-mon[32659]: pgmap v3197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:38 smithi132 ceph-mon[37580]: pgmap v3197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:38 smithi112 ceph-mon[37501]: pgmap v3197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:41.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:41.285+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:41.285+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:41.287 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:41.285+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:41.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:40 smithi107 ceph-mon[32659]: pgmap v3198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:41.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5076.858932 secs 2024-07-26T14:15:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:40 smithi132 ceph-mon[37580]: pgmap v3198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5076.858932 secs 2024-07-26T14:15:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:40 smithi112 ceph-mon[37501]: pgmap v3198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5076.858932 secs 2024-07-26T14:15:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:42 smithi107 ceph-mon[32659]: pgmap v3199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:42 smithi132 ceph-mon[37580]: pgmap v3199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:43.428+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:43.428+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:43.430 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:43.428+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:42 smithi112 ceph-mon[37501]: pgmap v3199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:44 smithi107 ceph-mon[32659]: pgmap v3200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:44 smithi132 ceph-mon[37580]: pgmap v3200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:44 smithi112 ceph-mon[37501]: pgmap v3200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5081.859107 secs 2024-07-26T14:15:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5081.859107 secs 2024-07-26T14:15:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5081.859107 secs 2024-07-26T14:15:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:47 smithi107 ceph-mon[32659]: pgmap v3201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:47 smithi132 ceph-mon[37580]: pgmap v3201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:47 smithi112 ceph-mon[37501]: pgmap v3201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:48 smithi107 ceph-mon[32659]: pgmap v3202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:48 smithi132 ceph-mon[37580]: pgmap v3202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:48 smithi112 ceph-mon[37501]: pgmap v3202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:50 smithi112 ceph-mon[37501]: pgmap v3203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5086.859219 secs 2024-07-26T14:15:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:50 smithi107 ceph-mon[32659]: pgmap v3203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5086.859219 secs 2024-07-26T14:15:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:50 smithi132 ceph-mon[37580]: pgmap v3203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5086.859219 secs 2024-07-26T14:15:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:52 smithi112 ceph-mon[37501]: pgmap v3204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:53.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:52 smithi107 ceph-mon[32659]: pgmap v3204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:52 smithi132 ceph-mon[37580]: pgmap v3204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:15:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:54 smithi112 ceph-mon[37501]: pgmap v3205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:54 smithi107 ceph-mon[32659]: pgmap v3205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:54 smithi132 ceph-mon[37580]: pgmap v3205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5091.859369 secs 2024-07-26T14:15:56.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:56.301+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:56.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:56.301+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:56.302 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:15:56.301+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5091.859369 secs 2024-07-26T14:15:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5091.859369 secs 2024-07-26T14:15:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:56 smithi107 ceph-mon[32659]: pgmap v3206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:56 smithi132 ceph-mon[37580]: pgmap v3206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:56 smithi112 ceph-mon[37501]: pgmap v3206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:58.445+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:58.445+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:15:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:15:58.445+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:15:58 smithi107 ceph-mon[32659]: pgmap v3207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:15:58 smithi132 ceph-mon[37580]: pgmap v3207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:15:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:15:58 smithi112 ceph-mon[37501]: pgmap v3207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:01.252 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:00 smithi112 ceph-mon[37501]: pgmap v3208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:01.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5096.859478 secs 2024-07-26T14:16:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:00 smithi107 ceph-mon[32659]: pgmap v3208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5096.859478 secs 2024-07-26T14:16:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:00 smithi132 ceph-mon[37580]: pgmap v3208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5096.859478 secs 2024-07-26T14:16:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:02 smithi107 ceph-mon[32659]: pgmap v3209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:02 smithi132 ceph-mon[37580]: pgmap v3209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:02 smithi112 ceph-mon[37501]: pgmap v3209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:04 smithi107 ceph-mon[32659]: pgmap v3210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:16:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:04 smithi132 ceph-mon[37580]: pgmap v3210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:16:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:04 smithi112 ceph-mon[37501]: pgmap v3210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:16:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:16:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:16:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:16:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5101.859611 secs 2024-07-26T14:16:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:16:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:16:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:16:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5101.859611 secs 2024-07-26T14:16:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:16:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:16:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:16:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5101.859611 secs 2024-07-26T14:16:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:06 smithi107 ceph-mon[32659]: pgmap v3211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:16:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:06 smithi107 ceph-mon[32659]: pgmap v3212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:16:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:06 smithi132 ceph-mon[37580]: pgmap v3211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:16:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:06 smithi132 ceph-mon[37580]: pgmap v3212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:16:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:06 smithi112 ceph-mon[37501]: pgmap v3211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:16:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:06 smithi112 ceph-mon[37501]: pgmap v3212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:16:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:09 smithi107 ceph-mon[32659]: pgmap v3213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:09 smithi132 ceph-mon[37580]: pgmap v3213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:09 smithi112 ceph-mon[37501]: pgmap v3213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:10 smithi107 ceph-mon[32659]: pgmap v3214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:16:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:10 smithi132 ceph-mon[37580]: pgmap v3214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:16:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:10 smithi112 ceph-mon[37501]: pgmap v3214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:16:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5106.859730 secs 2024-07-26T14:16:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5106.859730 secs 2024-07-26T14:16:11.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:11.316+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:11.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:11.317+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:11.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:11.317+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5106.859730 secs 2024-07-26T14:16:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:12 smithi107 ceph-mon[32659]: pgmap v3215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:16:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:12 smithi132 ceph-mon[37580]: pgmap v3215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:16:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:12 smithi112 ceph-mon[37501]: pgmap v3215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:16:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:13.460+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:13.460+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:13.461 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:13.460+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:14 smithi112 ceph-mon[37501]: pgmap v3216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:16:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:14 smithi107 ceph-mon[32659]: pgmap v3216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:16:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:14 smithi132 ceph-mon[37580]: pgmap v3216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:16:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:16 smithi112 ceph-mon[37501]: pgmap v3217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:16:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5111.859861 secs 2024-07-26T14:16:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:16 smithi107 ceph-mon[32659]: pgmap v3217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:16:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5111.859861 secs 2024-07-26T14:16:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:16 smithi132 ceph-mon[37580]: pgmap v3217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:16:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5111.859861 secs 2024-07-26T14:16:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:18 smithi112 ceph-mon[37501]: pgmap v3218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:18 smithi107 ceph-mon[32659]: pgmap v3218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:18 smithi132 ceph-mon[37580]: pgmap v3218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:20 smithi112 ceph-mon[37501]: pgmap v3219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:20 smithi107 ceph-mon[32659]: pgmap v3219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:20 smithi132 ceph-mon[37580]: pgmap v3219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5116.859992 secs 2024-07-26T14:16:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5116.859992 secs 2024-07-26T14:16:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5116.859992 secs 2024-07-26T14:16:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:22 smithi112 ceph-mon[37501]: pgmap v3220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:22 smithi107 ceph-mon[32659]: pgmap v3220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:22 smithi132 ceph-mon[37580]: pgmap v3220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:24 smithi112 ceph-mon[37501]: pgmap v3221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:24 smithi107 ceph-mon[32659]: pgmap v3221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:24 smithi132 ceph-mon[37580]: pgmap v3221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:26.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:26.331+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:26.333 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:26.331+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:26.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:26.331+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:26 smithi112 ceph-mon[37501]: pgmap v3222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5121.860108 secs 2024-07-26T14:16:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:26 smithi107 ceph-mon[32659]: pgmap v3222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5121.860108 secs 2024-07-26T14:16:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:26 smithi132 ceph-mon[37580]: pgmap v3222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5121.860108 secs 2024-07-26T14:16:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:28 smithi112 ceph-mon[37501]: pgmap v3223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:28.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:28.475+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:28.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:28.475+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:28.477 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:28.475+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:28 smithi107 ceph-mon[32659]: pgmap v3223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:28 smithi132 ceph-mon[37580]: pgmap v3223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:30 smithi107 ceph-mon[32659]: pgmap v3224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:30 smithi132 ceph-mon[37580]: pgmap v3224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:30 smithi112 ceph-mon[37501]: pgmap v3224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5126.860179 secs 2024-07-26T14:16:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5126.860179 secs 2024-07-26T14:16:31.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5126.860179 secs 2024-07-26T14:16:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:32 smithi107 ceph-mon[32659]: pgmap v3225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:32 smithi132 ceph-mon[37580]: pgmap v3225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:32 smithi112 ceph-mon[37501]: pgmap v3225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:34 smithi107 ceph-mon[32659]: pgmap v3226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:34 smithi132 ceph-mon[37580]: pgmap v3226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:34 smithi112 ceph-mon[37501]: pgmap v3226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:36 smithi107 ceph-mon[32659]: pgmap v3227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5131.860264 secs 2024-07-26T14:16:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:36 smithi132 ceph-mon[37580]: pgmap v3227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5131.860264 secs 2024-07-26T14:16:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:36 smithi112 ceph-mon[37501]: pgmap v3227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5131.860264 secs 2024-07-26T14:16:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:38 smithi107 ceph-mon[32659]: pgmap v3228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:38 smithi132 ceph-mon[37580]: pgmap v3228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:38 smithi112 ceph-mon[37501]: pgmap v3228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:40 smithi107 ceph-mon[32659]: pgmap v3229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:40 smithi132 ceph-mon[37580]: pgmap v3229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:40 smithi112 ceph-mon[37501]: pgmap v3229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:41.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:41.347+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:41.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:41.347+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:41.349 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:41.347+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:41.443 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5136.860376 secs 2024-07-26T14:16:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5136.860376 secs 2024-07-26T14:16:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5136.860376 secs 2024-07-26T14:16:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:42 smithi107 ceph-mon[32659]: pgmap v3230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:42 smithi132 ceph-mon[37580]: pgmap v3230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:42 smithi112 ceph-mon[37501]: pgmap v3230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:43.484+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:43.484+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:43.484+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:44.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:44 smithi107 ceph-mon[32659]: pgmap v3231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:44 smithi132 ceph-mon[37580]: pgmap v3231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:44 smithi112 ceph-mon[37501]: pgmap v3231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:46 smithi107 ceph-mon[32659]: pgmap v3232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5141.860497 secs 2024-07-26T14:16:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:46 smithi132 ceph-mon[37580]: pgmap v3232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5141.860497 secs 2024-07-26T14:16:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:46 smithi112 ceph-mon[37501]: pgmap v3232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5141.860497 secs 2024-07-26T14:16:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:48 smithi107 ceph-mon[32659]: pgmap v3233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:48 smithi132 ceph-mon[37580]: pgmap v3233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:48 smithi112 ceph-mon[37501]: pgmap v3233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:50 smithi107 ceph-mon[32659]: pgmap v3234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:50 smithi132 ceph-mon[37580]: pgmap v3234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:50 smithi112 ceph-mon[37501]: pgmap v3234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5146.860617 secs 2024-07-26T14:16:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5146.860617 secs 2024-07-26T14:16:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5146.860617 secs 2024-07-26T14:16:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:52 smithi107 ceph-mon[32659]: pgmap v3235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:52 smithi132 ceph-mon[37580]: pgmap v3235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:52 smithi112 ceph-mon[37501]: pgmap v3235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:54 smithi132 ceph-mon[37580]: pgmap v3236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:54 smithi112 ceph-mon[37501]: pgmap v3236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:54 smithi107 ceph-mon[32659]: pgmap v3236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:16:56.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:56.363+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:56.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:56.364+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:56.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:16:56.364+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:56 smithi132 ceph-mon[37580]: pgmap v3237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5151.860753 secs 2024-07-26T14:16:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:56 smithi112 ceph-mon[37501]: pgmap v3237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5151.860753 secs 2024-07-26T14:16:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:56 smithi107 ceph-mon[32659]: pgmap v3237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5151.860753 secs 2024-07-26T14:16:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:58.501+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:58.501+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:16:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:16:58.501+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:16:58 smithi132 ceph-mon[37580]: pgmap v3238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:16:58 smithi112 ceph-mon[37501]: pgmap v3238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:16:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:16:58 smithi107 ceph-mon[32659]: pgmap v3238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:00 smithi112 ceph-mon[37501]: pgmap v3239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:00 smithi107 ceph-mon[32659]: pgmap v3239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:00 smithi132 ceph-mon[37580]: pgmap v3239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:01.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5156.860893 secs 2024-07-26T14:17:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5156.860893 secs 2024-07-26T14:17:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5156.860893 secs 2024-07-26T14:17:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:02 smithi112 ceph-mon[37501]: pgmap v3240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:02 smithi107 ceph-mon[32659]: pgmap v3240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:02 smithi132 ceph-mon[37580]: pgmap v3240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:04 smithi112 ceph-mon[37501]: pgmap v3241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:04 smithi107 ceph-mon[32659]: pgmap v3241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:04 smithi132 ceph-mon[37580]: pgmap v3241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:17:05.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:17:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:17:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:06 smithi112 ceph-mon[37501]: pgmap v3242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5161.861017 secs 2024-07-26T14:17:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:17:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:17:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:06 smithi107 ceph-mon[32659]: pgmap v3242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5161.861017 secs 2024-07-26T14:17:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:17:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:17:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:06 smithi132 ceph-mon[37580]: pgmap v3242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5161.861017 secs 2024-07-26T14:17:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:17:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:17:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:17:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:17:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:07 smithi112 ceph-mon[37501]: pgmap v3243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:17:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:07 smithi112 ceph-mon[37501]: pgmap v3244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:17:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:17:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:17:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:17:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:07 smithi107 ceph-mon[32659]: pgmap v3243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:17:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:07 smithi107 ceph-mon[32659]: pgmap v3244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:17:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:17:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:17:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:17:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:07 smithi132 ceph-mon[37580]: pgmap v3243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:17:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:07 smithi132 ceph-mon[37580]: pgmap v3244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:17:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:17:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:09 smithi112 ceph-mon[37501]: pgmap v3245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:17:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:09 smithi107 ceph-mon[32659]: pgmap v3245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:17:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:09 smithi132 ceph-mon[37580]: pgmap v3245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:17:11.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5166.861133 secs 2024-07-26T14:17:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5166.861133 secs 2024-07-26T14:17:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:11.379+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:11.379+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:11.379+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5166.861133 secs 2024-07-26T14:17:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:11 smithi107 ceph-mon[32659]: pgmap v3246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:11 smithi132 ceph-mon[37580]: pgmap v3246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:11 smithi112 ceph-mon[37501]: pgmap v3246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:13.506+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:13.506+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:13.506+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:14.229 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:13 smithi107 ceph-mon[32659]: pgmap v3247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:17:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:13 smithi132 ceph-mon[37580]: pgmap v3247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:17:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:13 smithi112 ceph-mon[37501]: pgmap v3247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:17:15.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:15 smithi107 ceph-mon[32659]: pgmap v3248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:15 smithi132 ceph-mon[37580]: pgmap v3248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:15 smithi112 ceph-mon[37501]: pgmap v3248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5171.861238 secs 2024-07-26T14:17:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5171.861238 secs 2024-07-26T14:17:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5171.861238 secs 2024-07-26T14:17:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:17 smithi132 ceph-mon[37580]: pgmap v3249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:17:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:17 smithi112 ceph-mon[37501]: pgmap v3249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:17:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:17 smithi107 ceph-mon[32659]: pgmap v3249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:17:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:19 smithi107 ceph-mon[32659]: pgmap v3250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:19 smithi132 ceph-mon[37580]: pgmap v3250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:19 smithi112 ceph-mon[37501]: pgmap v3250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5176.861361 secs 2024-07-26T14:17:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5176.861361 secs 2024-07-26T14:17:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5176.861361 secs 2024-07-26T14:17:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:21 smithi107 ceph-mon[32659]: pgmap v3251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:21 smithi132 ceph-mon[37580]: pgmap v3251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:21 smithi112 ceph-mon[37501]: pgmap v3251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:23 smithi107 ceph-mon[32659]: pgmap v3252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:23 smithi132 ceph-mon[37580]: pgmap v3252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:23 smithi112 ceph-mon[37501]: pgmap v3252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:25 smithi107 ceph-mon[32659]: pgmap v3253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5181.861436 secs 2024-07-26T14:17:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:25 smithi132 ceph-mon[37580]: pgmap v3253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5181.861436 secs 2024-07-26T14:17:26.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:26.395+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:26.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:26.395+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:26.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:26.395+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:25 smithi112 ceph-mon[37501]: pgmap v3253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5181.861436 secs 2024-07-26T14:17:28.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:27 smithi107 ceph-mon[32659]: pgmap v3254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:27 smithi132 ceph-mon[37580]: pgmap v3254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:27 smithi112 ceph-mon[37501]: pgmap v3254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:28.521+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:28.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:28.521+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:28.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:28.521+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:29 smithi107 ceph-mon[32659]: pgmap v3255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:29 smithi132 ceph-mon[37580]: pgmap v3255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:29 smithi112 ceph-mon[37501]: pgmap v3255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5186.861552 secs 2024-07-26T14:17:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5186.861552 secs 2024-07-26T14:17:31.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5186.861552 secs 2024-07-26T14:17:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:32 smithi107 ceph-mon[32659]: pgmap v3256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:32 smithi132 ceph-mon[37580]: pgmap v3256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:32 smithi112 ceph-mon[37501]: pgmap v3256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:34 smithi107 ceph-mon[32659]: pgmap v3257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:34 smithi132 ceph-mon[37580]: pgmap v3257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:34 smithi112 ceph-mon[37501]: pgmap v3257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:36 smithi107 ceph-mon[32659]: pgmap v3258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5191.861674 secs 2024-07-26T14:17:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:36 smithi132 ceph-mon[37580]: pgmap v3258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5191.861674 secs 2024-07-26T14:17:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:36 smithi112 ceph-mon[37501]: pgmap v3258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5191.861674 secs 2024-07-26T14:17:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:37 smithi112 ceph-mon[37501]: pgmap v3259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:37 smithi107 ceph-mon[32659]: pgmap v3259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:37 smithi132 ceph-mon[37580]: pgmap v3259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:39 smithi107 ceph-mon[32659]: pgmap v3260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:39 smithi132 ceph-mon[37580]: pgmap v3260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:39 smithi112 ceph-mon[37501]: pgmap v3260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5196.861797 secs 2024-07-26T14:17:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5196.861797 secs 2024-07-26T14:17:41.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:41.409+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:41.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:41.409+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:41.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:41.409+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5196.861797 secs 2024-07-26T14:17:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:41 smithi107 ceph-mon[32659]: pgmap v3261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:41 smithi132 ceph-mon[37580]: pgmap v3261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:41 smithi112 ceph-mon[37501]: pgmap v3261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:43.537+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:43.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:43.537+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:43.537+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:44.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:43 smithi107 ceph-mon[32659]: pgmap v3262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:43 smithi132 ceph-mon[37580]: pgmap v3262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:43 smithi112 ceph-mon[37501]: pgmap v3262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:45 smithi107 ceph-mon[32659]: pgmap v3263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5201.861921 secs 2024-07-26T14:17:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:45 smithi132 ceph-mon[37580]: pgmap v3263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5201.861921 secs 2024-07-26T14:17:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:45 smithi112 ceph-mon[37501]: pgmap v3263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5201.861921 secs 2024-07-26T14:17:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:47 smithi107 ceph-mon[32659]: pgmap v3264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:47 smithi132 ceph-mon[37580]: pgmap v3264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:47 smithi112 ceph-mon[37501]: pgmap v3264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:50 smithi107 ceph-mon[32659]: pgmap v3265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:50 smithi132 ceph-mon[37580]: pgmap v3265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:50 smithi112 ceph-mon[37501]: pgmap v3265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5206.862014 secs 2024-07-26T14:17:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5206.862014 secs 2024-07-26T14:17:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5206.862014 secs 2024-07-26T14:17:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:52 smithi107 ceph-mon[32659]: pgmap v3266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:52 smithi132 ceph-mon[37580]: pgmap v3266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:52 smithi112 ceph-mon[37501]: pgmap v3266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:53 smithi112 ceph-mon[37501]: pgmap v3267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:53 smithi107 ceph-mon[32659]: pgmap v3267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:53 smithi132 ceph-mon[37580]: pgmap v3267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:56.246 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:55 smithi107 ceph-mon[32659]: pgmap v3268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:56.246 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5211.862153 secs 2024-07-26T14:17:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:55 smithi132 ceph-mon[37580]: pgmap v3268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5211.862153 secs 2024-07-26T14:17:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:56.426+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:56.426+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:56.427 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:17:56.426+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:55 smithi112 ceph-mon[37501]: pgmap v3268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5211.862153 secs 2024-07-26T14:17:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:57 smithi107 ceph-mon[32659]: pgmap v3269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:57 smithi132 ceph-mon[37580]: pgmap v3269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:57 smithi112 ceph-mon[37501]: pgmap v3269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:58.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:58.553+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:58.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:58.553+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:17:58.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:17:58.553+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:17:59 smithi107 ceph-mon[32659]: pgmap v3270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:17:59 smithi132 ceph-mon[37580]: pgmap v3270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:17:59 smithi112 ceph-mon[37501]: pgmap v3270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5216.862280 secs 2024-07-26T14:18:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5216.862280 secs 2024-07-26T14:18:01.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5216.862280 secs 2024-07-26T14:18:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:02 smithi107 ceph-mon[32659]: pgmap v3271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:02 smithi132 ceph-mon[37580]: pgmap v3271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:02 smithi112 ceph-mon[37501]: pgmap v3271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:04 smithi107 ceph-mon[32659]: pgmap v3272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:04 smithi132 ceph-mon[37580]: pgmap v3272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:04 smithi112 ceph-mon[37501]: pgmap v3272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:06 smithi107 ceph-mon[32659]: pgmap v3273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5221.862404 secs 2024-07-26T14:18:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:06 smithi132 ceph-mon[37580]: pgmap v3273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5221.862404 secs 2024-07-26T14:18:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:06 smithi112 ceph-mon[37501]: pgmap v3273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5221.862404 secs 2024-07-26T14:18:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:08 smithi107 ceph-mon[32659]: pgmap v3274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:18:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:08 smithi132 ceph-mon[37580]: pgmap v3274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:18:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.357 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:08 smithi112 ceph-mon[37501]: pgmap v3274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:08.357 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:18:08.357 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.357 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.357 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:08.357 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:09 smithi112 ceph-mon[37501]: pgmap v3275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:18:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:09 smithi107 ceph-mon[32659]: pgmap v3275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:18:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:09 smithi132 ceph-mon[37580]: pgmap v3275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:18:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:11.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:11.431+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:11.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:11.431+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:11.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:11.431+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5226.862524 secs 2024-07-26T14:18:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:11 smithi107 ceph-mon[32659]: pgmap v3276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:18:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5226.862524 secs 2024-07-26T14:18:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:11 smithi132 ceph-mon[37580]: pgmap v3276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:18:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5226.862524 secs 2024-07-26T14:18:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:11 smithi112 ceph-mon[37501]: pgmap v3276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:18:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:18:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:18:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:12 smithi107 ceph-mon[32659]: pgmap v3277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:18:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:12 smithi107 ceph-mon[32659]: pgmap v3278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:18:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:18:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:18:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:12 smithi132 ceph-mon[37580]: pgmap v3277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:18:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:12 smithi132 ceph-mon[37580]: pgmap v3278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:18:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:18:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:18:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:12 smithi112 ceph-mon[37501]: pgmap v3277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:18:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:12 smithi112 ceph-mon[37501]: pgmap v3278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:18:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:18:13.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:13.521+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:18:13.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:13.522+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:18:13.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:13.550+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:18:13.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:13.550+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:18:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:13.569+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:13.569+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:13.569 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:13.569+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:14.676+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:18:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:14.676+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:18:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:14.677+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:18:14.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:14.677+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:18:14.881 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:14 smithi107 ceph-mon[32659]: pgmap v3279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:14 smithi112 ceph-mon[37501]: pgmap v3279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:14 smithi132 ceph-mon[37580]: pgmap v3279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5231.862636 secs 2024-07-26T14:18:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:16 smithi112 ceph-mon[37501]: pgmap v3280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:18:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5231.862636 secs 2024-07-26T14:18:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:16 smithi107 ceph-mon[32659]: pgmap v3280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:18:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5231.862636 secs 2024-07-26T14:18:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:16 smithi132 ceph-mon[37580]: pgmap v3280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:18:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:18 smithi112 ceph-mon[37501]: pgmap v3281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:18:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:18 smithi107 ceph-mon[32659]: pgmap v3281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:18:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:18 smithi132 ceph-mon[37580]: pgmap v3281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:18:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:20 smithi112 ceph-mon[37501]: pgmap v3282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:18:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:20 smithi107 ceph-mon[32659]: pgmap v3282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:18:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:20 smithi132 ceph-mon[37580]: pgmap v3282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:18:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5236.862753 secs 2024-07-26T14:18:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5236.862753 secs 2024-07-26T14:18:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5236.862753 secs 2024-07-26T14:18:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:22 smithi112 ceph-mon[37501]: pgmap v3283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:18:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:22 smithi107 ceph-mon[32659]: pgmap v3283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:18:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:22 smithi132 ceph-mon[37580]: pgmap v3283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:18:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:24 smithi112 ceph-mon[37501]: pgmap v3284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:24 smithi107 ceph-mon[32659]: pgmap v3284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:24 smithi132 ceph-mon[37580]: pgmap v3284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:26.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:26.447+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:26.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:26.447+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:26.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:26.447+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5241.862841 secs 2024-07-26T14:18:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:26 smithi107 ceph-mon[32659]: pgmap v3285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5241.862841 secs 2024-07-26T14:18:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:26 smithi112 ceph-mon[37501]: pgmap v3285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5241.862841 secs 2024-07-26T14:18:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:26 smithi132 ceph-mon[37580]: pgmap v3285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:28.583+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:28.583+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:28.583+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:28 smithi112 ceph-mon[37501]: pgmap v3286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:28 smithi107 ceph-mon[32659]: pgmap v3286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:28 smithi132 ceph-mon[37580]: pgmap v3286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:30 smithi112 ceph-mon[37501]: pgmap v3287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:18:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:30 smithi107 ceph-mon[32659]: pgmap v3287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:18:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:30 smithi132 ceph-mon[37580]: pgmap v3287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:18:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5246.862965 secs 2024-07-26T14:18:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5246.862965 secs 2024-07-26T14:18:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5246.862965 secs 2024-07-26T14:18:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:32 smithi112 ceph-mon[37501]: pgmap v3288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:32 smithi107 ceph-mon[32659]: pgmap v3288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:32 smithi132 ceph-mon[37580]: pgmap v3288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:34 smithi112 ceph-mon[37501]: pgmap v3289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:34 smithi107 ceph-mon[32659]: pgmap v3289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:34 smithi132 ceph-mon[37580]: pgmap v3289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5251.863062 secs 2024-07-26T14:18:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:36 smithi107 ceph-mon[32659]: pgmap v3290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5251.863062 secs 2024-07-26T14:18:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:36 smithi132 ceph-mon[37580]: pgmap v3290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5251.863062 secs 2024-07-26T14:18:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:36 smithi112 ceph-mon[37501]: pgmap v3290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:38 smithi107 ceph-mon[32659]: pgmap v3291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:38 smithi132 ceph-mon[37580]: pgmap v3291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:38 smithi112 ceph-mon[37501]: pgmap v3291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:40 smithi107 ceph-mon[32659]: pgmap v3292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:18:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:40 smithi132 ceph-mon[37580]: pgmap v3292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:18:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:40 smithi112 ceph-mon[37501]: pgmap v3292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:18:41.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:41.457+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:41.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:41.457+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:41.460 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:41.457+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5256.863199 secs 2024-07-26T14:18:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5256.863199 secs 2024-07-26T14:18:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5256.863199 secs 2024-07-26T14:18:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:42 smithi107 ceph-mon[32659]: pgmap v3293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:42 smithi132 ceph-mon[37580]: pgmap v3293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:42 smithi112 ceph-mon[37501]: pgmap v3293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:18:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:43.588+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:43.588+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:43.588+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:44 smithi107 ceph-mon[32659]: pgmap v3294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:44 smithi132 ceph-mon[37580]: pgmap v3294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:44 smithi112 ceph-mon[37501]: pgmap v3294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5261.863264 secs 2024-07-26T14:18:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:46 smithi107 ceph-mon[32659]: pgmap v3295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5261.863264 secs 2024-07-26T14:18:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:46 smithi132 ceph-mon[37580]: pgmap v3295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5261.863264 secs 2024-07-26T14:18:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:46 smithi112 ceph-mon[37501]: pgmap v3295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:48 smithi107 ceph-mon[32659]: pgmap v3296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:48 smithi132 ceph-mon[37580]: pgmap v3296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:48 smithi112 ceph-mon[37501]: pgmap v3296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:18:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:50 smithi107 ceph-mon[32659]: pgmap v3297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:18:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:50 smithi132 ceph-mon[37580]: pgmap v3297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:18:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:50 smithi112 ceph-mon[37501]: pgmap v3297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:18:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5266.863397 secs 2024-07-26T14:18:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5266.863397 secs 2024-07-26T14:18:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5266.863397 secs 2024-07-26T14:18:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:52 smithi107 ceph-mon[32659]: pgmap v3298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:18:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:52 smithi132 ceph-mon[37580]: pgmap v3298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:18:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:52 smithi112 ceph-mon[37501]: pgmap v3298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:18:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:54 smithi107 ceph-mon[32659]: pgmap v3299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:18:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:54 smithi132 ceph-mon[37580]: pgmap v3299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:18:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:54 smithi112 ceph-mon[37501]: pgmap v3299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:18:56.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:56.466+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:56.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:56.466+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:56.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:18:56.466+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5271.863522 secs 2024-07-26T14:18:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:56 smithi107 ceph-mon[32659]: pgmap v3300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:18:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5271.863522 secs 2024-07-26T14:18:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:56 smithi132 ceph-mon[37580]: pgmap v3300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:18:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5271.863522 secs 2024-07-26T14:18:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:56 smithi112 ceph-mon[37501]: pgmap v3300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:18:58.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:58.605+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:58.605+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:18:58.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:18:58.605+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:18:58 smithi132 ceph-mon[37580]: pgmap v3301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:18:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:18:58 smithi112 ceph-mon[37501]: pgmap v3301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:18:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:18:58 smithi107 ceph-mon[32659]: pgmap v3301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:19:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:00 smithi132 ceph-mon[37580]: pgmap v3302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:19:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:00 smithi112 ceph-mon[37501]: pgmap v3302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:19:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:00 smithi107 ceph-mon[32659]: pgmap v3302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:19:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5276.863644 secs 2024-07-26T14:19:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5276.863644 secs 2024-07-26T14:19:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5276.863644 secs 2024-07-26T14:19:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:02 smithi112 ceph-mon[37501]: pgmap v3303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:19:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:02 smithi107 ceph-mon[32659]: pgmap v3303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:19:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:02 smithi132 ceph-mon[37580]: pgmap v3303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:19:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:04 smithi112 ceph-mon[37501]: pgmap v3304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:04 smithi107 ceph-mon[32659]: pgmap v3304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:04 smithi132 ceph-mon[37580]: pgmap v3304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5281.863779 secs 2024-07-26T14:19:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:06 smithi112 ceph-mon[37501]: pgmap v3305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5281.863779 secs 2024-07-26T14:19:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:06 smithi107 ceph-mon[32659]: pgmap v3305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5281.863779 secs 2024-07-26T14:19:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:06 smithi132 ceph-mon[37580]: pgmap v3305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:08 smithi112 ceph-mon[37501]: pgmap v3306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:08 smithi107 ceph-mon[32659]: pgmap v3306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:08 smithi132 ceph-mon[37580]: pgmap v3306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:11.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:10 smithi132 ceph-mon[37580]: pgmap v3307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:19:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:10 smithi112 ceph-mon[37501]: pgmap v3307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:19:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:10 smithi107 ceph-mon[32659]: pgmap v3307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:19:11.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:11.475+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:11.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:11.475+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:11.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:11.475+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5286.863913 secs 2024-07-26T14:19:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:19:12.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5286.863913 secs 2024-07-26T14:19:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:19:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5286.863913 secs 2024-07-26T14:19:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:19:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:12 smithi112 ceph-mon[37501]: pgmap v3308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:19:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:19:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:19:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:12 smithi107 ceph-mon[32659]: pgmap v3308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:19:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:19:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:19:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:12 smithi132 ceph-mon[37580]: pgmap v3308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:19:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:19:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:19:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:13.612+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:13.613+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:13.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:13.613+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:13 smithi112 ceph-mon[37501]: pgmap v3309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:19:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:13 smithi112 ceph-mon[37501]: pgmap v3310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:19:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:13 smithi107 ceph-mon[32659]: pgmap v3309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:19:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:13 smithi107 ceph-mon[32659]: pgmap v3310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:19:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:13 smithi132 ceph-mon[37580]: pgmap v3309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:19:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:13 smithi132 ceph-mon[37580]: pgmap v3310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:19:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:15 smithi112 ceph-mon[37501]: pgmap v3311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:19:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:15 smithi107 ceph-mon[32659]: pgmap v3311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:19:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:15 smithi132 ceph-mon[37580]: pgmap v3311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:19:17.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5291.864015 secs 2024-07-26T14:19:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5291.864015 secs 2024-07-26T14:19:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5291.864015 secs 2024-07-26T14:19:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:17 smithi112 ceph-mon[37501]: pgmap v3312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:17 smithi107 ceph-mon[32659]: pgmap v3312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:17 smithi132 ceph-mon[37580]: pgmap v3312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:19 smithi112 ceph-mon[37501]: pgmap v3313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:19:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:19 smithi107 ceph-mon[32659]: pgmap v3313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:19:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:19 smithi132 ceph-mon[37580]: pgmap v3313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:19:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5296.864160 secs 2024-07-26T14:19:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:21 smithi112 ceph-mon[37501]: pgmap v3314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:19:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5296.864160 secs 2024-07-26T14:19:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:21 smithi107 ceph-mon[32659]: pgmap v3314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:19:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5296.864160 secs 2024-07-26T14:19:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:21 smithi132 ceph-mon[37580]: pgmap v3314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:19:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:23 smithi112 ceph-mon[37501]: pgmap v3315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:19:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:23 smithi107 ceph-mon[32659]: pgmap v3315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:19:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:23 smithi132 ceph-mon[37580]: pgmap v3315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:19:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:25 smithi112 ceph-mon[37501]: pgmap v3316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:25 smithi107 ceph-mon[32659]: pgmap v3316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:25 smithi132 ceph-mon[37580]: pgmap v3316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:26.478+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:26.478+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:26.478+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5301.864244 secs 2024-07-26T14:19:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5301.864244 secs 2024-07-26T14:19:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5301.864244 secs 2024-07-26T14:19:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:27 smithi112 ceph-mon[37501]: pgmap v3317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:27 smithi107 ceph-mon[32659]: pgmap v3317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:27 smithi132 ceph-mon[37580]: pgmap v3317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:28.625+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:28.625+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:28.625+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:29 smithi112 ceph-mon[37501]: pgmap v3318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:29 smithi107 ceph-mon[32659]: pgmap v3318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:29 smithi132 ceph-mon[37580]: pgmap v3318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5306.864366 secs 2024-07-26T14:19:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5306.864366 secs 2024-07-26T14:19:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5306.864366 secs 2024-07-26T14:19:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:31 smithi107 ceph-mon[32659]: pgmap v3319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:31 smithi132 ceph-mon[37580]: pgmap v3319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:31 smithi112 ceph-mon[37501]: pgmap v3319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:34.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:33 smithi107 ceph-mon[32659]: pgmap v3320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:33 smithi132 ceph-mon[37580]: pgmap v3320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:33 smithi112 ceph-mon[37501]: pgmap v3320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:35 smithi107 ceph-mon[32659]: pgmap v3321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5311.864491 secs 2024-07-26T14:19:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:35 smithi132 ceph-mon[37580]: pgmap v3321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5311.864491 secs 2024-07-26T14:19:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:35 smithi112 ceph-mon[37501]: pgmap v3321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5311.864491 secs 2024-07-26T14:19:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:37 smithi107 ceph-mon[32659]: pgmap v3322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:37 smithi132 ceph-mon[37580]: pgmap v3322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:37 smithi112 ceph-mon[37501]: pgmap v3322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:40 smithi107 ceph-mon[32659]: pgmap v3323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:40 smithi132 ceph-mon[37580]: pgmap v3323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:40 smithi112 ceph-mon[37501]: pgmap v3323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5316.864622 secs 2024-07-26T14:19:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5316.864622 secs 2024-07-26T14:19:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5316.864622 secs 2024-07-26T14:19:41.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:41.480+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:41.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:41.480+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:41.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:41.480+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:42 smithi107 ceph-mon[32659]: pgmap v3324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:42 smithi132 ceph-mon[37580]: pgmap v3324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:42 smithi112 ceph-mon[37501]: pgmap v3324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:43.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:43.639+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:43.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:43.639+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:43.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:43.639+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:44 smithi107 ceph-mon[32659]: pgmap v3325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:44 smithi132 ceph-mon[37580]: pgmap v3325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:44 smithi112 ceph-mon[37501]: pgmap v3325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:46 smithi107 ceph-mon[32659]: pgmap v3326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5321.864771 secs 2024-07-26T14:19:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:46 smithi132 ceph-mon[37580]: pgmap v3326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5321.864771 secs 2024-07-26T14:19:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:46 smithi112 ceph-mon[37501]: pgmap v3326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5321.864771 secs 2024-07-26T14:19:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:48 smithi107 ceph-mon[32659]: pgmap v3327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:48 smithi132 ceph-mon[37580]: pgmap v3327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:48 smithi112 ceph-mon[37501]: pgmap v3327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:50 smithi107 ceph-mon[32659]: pgmap v3328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:50 smithi132 ceph-mon[37580]: pgmap v3328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:50 smithi112 ceph-mon[37501]: pgmap v3328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5326.864896 secs 2024-07-26T14:19:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5326.864896 secs 2024-07-26T14:19:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5326.864896 secs 2024-07-26T14:19:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:52 smithi132 ceph-mon[37580]: pgmap v3329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:52 smithi112 ceph-mon[37501]: pgmap v3329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:52 smithi107 ceph-mon[32659]: pgmap v3329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:53 smithi112 ceph-mon[37501]: pgmap v3330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:53 smithi107 ceph-mon[32659]: pgmap v3330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:53 smithi132 ceph-mon[37580]: pgmap v3330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:19:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:55 smithi112 ceph-mon[37501]: pgmap v3331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:56.246 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:55 smithi107 ceph-mon[32659]: pgmap v3331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:55 smithi132 ceph-mon[37580]: pgmap v3331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:56.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:56.489+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:56.489+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:56.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:19:56.489+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5331.865022 secs 2024-07-26T14:19:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5331.865022 secs 2024-07-26T14:19:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5331.865022 secs 2024-07-26T14:19:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:57 smithi112 ceph-mon[37501]: pgmap v3332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:57 smithi107 ceph-mon[32659]: pgmap v3332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:57 smithi132 ceph-mon[37580]: pgmap v3332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:19:58.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:58.649+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:58.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:58.649+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:19:58.650 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:19:58.649+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:19:59 smithi112 ceph-mon[37501]: pgmap v3333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:19:59 smithi107 ceph-mon[32659]: pgmap v3333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:19:59 smithi132 ceph-mon[37580]: pgmap v3333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:20:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:20:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:20:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5336.865148 secs 2024-07-26T14:20:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:01 smithi112 ceph-mon[37501]: pgmap v3334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5336.865148 secs 2024-07-26T14:20:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:01 smithi107 ceph-mon[32659]: pgmap v3334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5336.865148 secs 2024-07-26T14:20:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:01 smithi132 ceph-mon[37580]: pgmap v3334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:03 smithi112 ceph-mon[37501]: pgmap v3335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:03 smithi107 ceph-mon[32659]: pgmap v3335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:03 smithi132 ceph-mon[37580]: pgmap v3335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:06.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:05 smithi107 ceph-mon[32659]: pgmap v3336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:06.211 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5341.865262 secs 2024-07-26T14:20:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:05 smithi132 ceph-mon[37580]: pgmap v3336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5341.865262 secs 2024-07-26T14:20:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:05 smithi112 ceph-mon[37501]: pgmap v3336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5341.865262 secs 2024-07-26T14:20:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:07 smithi107 ceph-mon[32659]: pgmap v3337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:07 smithi132 ceph-mon[37580]: pgmap v3337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:07 smithi112 ceph-mon[37501]: pgmap v3337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:09 smithi107 ceph-mon[32659]: pgmap v3338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:09 smithi132 ceph-mon[37580]: pgmap v3338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:09 smithi112 ceph-mon[37501]: pgmap v3338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5346.865379 secs 2024-07-26T14:20:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5346.865379 secs 2024-07-26T14:20:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5346.865379 secs 2024-07-26T14:20:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:11.498+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:11.498+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:11.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:11.498+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:11 smithi107 ceph-mon[32659]: pgmap v3339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:11 smithi132 ceph-mon[37580]: pgmap v3339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:11 smithi112 ceph-mon[37501]: pgmap v3339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:13.664+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:13.664+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:13.664+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:13 smithi107 ceph-mon[32659]: pgmap v3340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:20:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:20:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:20:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:13 smithi132 ceph-mon[37580]: pgmap v3340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:20:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:20:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:20:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:13 smithi112 ceph-mon[37501]: pgmap v3340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:20:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:20:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:20:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:14 smithi107 ceph-mon[32659]: pgmap v3341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:20:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:14 smithi107 ceph-mon[32659]: pgmap v3342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:20:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:20:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:14 smithi132 ceph-mon[37580]: pgmap v3341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:20:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:14 smithi132 ceph-mon[37580]: pgmap v3342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:20:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:20:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:14 smithi112 ceph-mon[37501]: pgmap v3341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:20:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:14 smithi112 ceph-mon[37501]: pgmap v3342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:20:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:20:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5351.865506 secs 2024-07-26T14:20:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5351.865506 secs 2024-07-26T14:20:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5351.865506 secs 2024-07-26T14:20:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:17 smithi107 ceph-mon[32659]: pgmap v3343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:20:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:17 smithi132 ceph-mon[37580]: pgmap v3343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:20:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:17 smithi112 ceph-mon[37501]: pgmap v3343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:20:19.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:19 smithi107 ceph-mon[32659]: pgmap v3344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:20:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:19 smithi132 ceph-mon[37580]: pgmap v3344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:20:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:19 smithi112 ceph-mon[37501]: pgmap v3344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:20:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:20 smithi112 ceph-mon[37501]: pgmap v3345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:20:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:20 smithi107 ceph-mon[32659]: pgmap v3345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:20:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:20 smithi132 ceph-mon[37580]: pgmap v3345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:20:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5356.865629 secs 2024-07-26T14:20:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5356.865629 secs 2024-07-26T14:20:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5356.865629 secs 2024-07-26T14:20:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:22 smithi112 ceph-mon[37501]: pgmap v3346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:22 smithi107 ceph-mon[32659]: pgmap v3346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:22 smithi132 ceph-mon[37580]: pgmap v3346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:25 smithi107 ceph-mon[32659]: pgmap v3347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:20:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:25 smithi132 ceph-mon[37580]: pgmap v3347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:20:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:25 smithi112 ceph-mon[37501]: pgmap v3347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:20:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5361.865697 secs 2024-07-26T14:20:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5361.865697 secs 2024-07-26T14:20:26.508 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5361.865697 secs 2024-07-26T14:20:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:26.507+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:26.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:26.507+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:26.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:26.507+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:27 smithi132 ceph-mon[37580]: pgmap v3348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:27 smithi112 ceph-mon[37501]: pgmap v3348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:27 smithi107 ceph-mon[32659]: pgmap v3348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:28 smithi112 ceph-mon[37501]: pgmap v3349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:28 smithi107 ceph-mon[32659]: pgmap v3349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:28 smithi132 ceph-mon[37580]: pgmap v3349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:28.680+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:28.680+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:28.680+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:31 smithi107 ceph-mon[32659]: pgmap v3350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5366.865808 secs 2024-07-26T14:20:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:31 smithi132 ceph-mon[37580]: pgmap v3350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5366.865808 secs 2024-07-26T14:20:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:31 smithi112 ceph-mon[37501]: pgmap v3350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5366.865808 secs 2024-07-26T14:20:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:33 smithi132 ceph-mon[37580]: pgmap v3351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:33 smithi112 ceph-mon[37501]: pgmap v3351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:33 smithi107 ceph-mon[32659]: pgmap v3351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:35 smithi132 ceph-mon[37580]: pgmap v3352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:35 smithi112 ceph-mon[37501]: pgmap v3352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:35 smithi107 ceph-mon[32659]: pgmap v3352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5371.865928 secs 2024-07-26T14:20:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5371.865928 secs 2024-07-26T14:20:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5371.865928 secs 2024-07-26T14:20:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:37 smithi112 ceph-mon[37501]: pgmap v3353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:37 smithi107 ceph-mon[32659]: pgmap v3353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:37 smithi132 ceph-mon[37580]: pgmap v3353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:38 smithi112 ceph-mon[37501]: pgmap v3354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:38 smithi107 ceph-mon[32659]: pgmap v3354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:38 smithi132 ceph-mon[37580]: pgmap v3354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:41 smithi132 ceph-mon[37580]: pgmap v3355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5376.866090 secs 2024-07-26T14:20:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:41 smithi112 ceph-mon[37501]: pgmap v3355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5376.866090 secs 2024-07-26T14:20:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:41.508+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:41.508+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:41.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:41.508+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:41.510 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:41 smithi107 ceph-mon[32659]: pgmap v3355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:41.511 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5376.866090 secs 2024-07-26T14:20:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:43 smithi132 ceph-mon[37580]: pgmap v3356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:43 smithi112 ceph-mon[37501]: pgmap v3356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:43 smithi107 ceph-mon[32659]: pgmap v3356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:43.685+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:43.685+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:43.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:43.685+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:45 smithi132 ceph-mon[37580]: pgmap v3357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:45 smithi112 ceph-mon[37501]: pgmap v3357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:45 smithi107 ceph-mon[32659]: pgmap v3357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5381.866209 secs 2024-07-26T14:20:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:46 smithi112 ceph-mon[37501]: pgmap v3358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5381.866209 secs 2024-07-26T14:20:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:46 smithi107 ceph-mon[32659]: pgmap v3358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5381.866209 secs 2024-07-26T14:20:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:46 smithi132 ceph-mon[37580]: pgmap v3358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:49 smithi132 ceph-mon[37580]: pgmap v3359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:49 smithi112 ceph-mon[37501]: pgmap v3359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:49 smithi107 ceph-mon[32659]: pgmap v3359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:51 smithi107 ceph-mon[32659]: pgmap v3360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5386.866273 secs 2024-07-26T14:20:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:51 smithi132 ceph-mon[37580]: pgmap v3360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5386.866273 secs 2024-07-26T14:20:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:51 smithi112 ceph-mon[37501]: pgmap v3360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5386.866273 secs 2024-07-26T14:20:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:52 smithi112 ceph-mon[37501]: pgmap v3361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:52 smithi107 ceph-mon[32659]: pgmap v3361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:52 smithi132 ceph-mon[37580]: pgmap v3361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:55 smithi132 ceph-mon[37580]: pgmap v3362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:55 smithi112 ceph-mon[37501]: pgmap v3362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:55 smithi107 ceph-mon[32659]: pgmap v3362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:56.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5391.866360 secs 2024-07-26T14:20:56.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:56 smithi107 ceph-mon[32659]: pgmap v3363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5391.866360 secs 2024-07-26T14:20:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:56 smithi112 ceph-mon[37501]: pgmap v3363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:56.525 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:56.523+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:56.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:56.524+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:56.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:20:56.524+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5391.866360 secs 2024-07-26T14:20:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:56 smithi132 ceph-mon[37580]: pgmap v3363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:58.702 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:58.701+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:58.701+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:20:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:20:58.701+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:20:59 smithi132 ceph-mon[37580]: pgmap v3364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:20:59 smithi112 ceph-mon[37501]: pgmap v3364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:20:59 smithi107 ceph-mon[32659]: pgmap v3364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:00 smithi112 ceph-mon[37501]: pgmap v3365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:00 smithi107 ceph-mon[32659]: pgmap v3365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:00 smithi132 ceph-mon[37580]: pgmap v3365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5396.866486 secs 2024-07-26T14:21:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5396.866486 secs 2024-07-26T14:21:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5396.866486 secs 2024-07-26T14:21:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:02 smithi112 ceph-mon[37501]: pgmap v3366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:02 smithi107 ceph-mon[32659]: pgmap v3366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:02 smithi132 ceph-mon[37580]: pgmap v3366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:05 smithi112 ceph-mon[37501]: pgmap v3367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:05 smithi107 ceph-mon[32659]: pgmap v3367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:05 smithi132 ceph-mon[37580]: pgmap v3367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5401.866605 secs 2024-07-26T14:21:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:06 smithi112 ceph-mon[37501]: pgmap v3368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5401.866605 secs 2024-07-26T14:21:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:06 smithi107 ceph-mon[32659]: pgmap v3368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5401.866605 secs 2024-07-26T14:21:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:06 smithi132 ceph-mon[37580]: pgmap v3368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:08 smithi112 ceph-mon[37501]: pgmap v3369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:08 smithi107 ceph-mon[32659]: pgmap v3369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:08 smithi132 ceph-mon[37580]: pgmap v3369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:10 smithi112 ceph-mon[37501]: pgmap v3370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:10 smithi107 ceph-mon[32659]: pgmap v3370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:10 smithi132 ceph-mon[37580]: pgmap v3370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5406.866732 secs 2024-07-26T14:21:11.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5406.866732 secs 2024-07-26T14:21:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:11.540+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:11.540+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:11.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:11.540+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5406.866732 secs 2024-07-26T14:21:11.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:11.652+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bfc40 msgr2=0x7ff0ec0c0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000276) for more than 900000000 us, fault. 2024-07-26T14:21:11.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:11.652+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bfc40 0x7ff0ec0c0050 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fd30 tx=0x7ff0f801f040 comp rx=0 tx=0).stop 2024-07-26T14:21:11.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:11.652+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bfc40 msgr2=0x7ff0ec0c0050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:21:11.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:11.652+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0bfc40 0x7ff0ec0c0050 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:21:11.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:11.652+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:21:11.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:11.653+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:21:11.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:11.654+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c0590 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f8051040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:21:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:12 smithi112 ceph-mon[37501]: pgmap v3371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:12 smithi107 ceph-mon[32659]: pgmap v3371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:12 smithi132 ceph-mon[37580]: pgmap v3371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:13.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:13.552+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:21:13.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:13.552+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:21:13.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:13.578+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:21:13.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:13.579+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:21:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.708+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.708+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:13.709 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.708+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.760+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2270 msgr2=0x7efc5c0c2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T14:21:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.760+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2270 msgr2=0x7efc5c0c2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:21:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.760+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2270 0x7efc5c0c2d30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7efc6809b040 tx=0x7efc680a9000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:21:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.760+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2270 0x7efc5c0c2d30 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7efc6809b040 tx=0x7efc680a9000 comp rx=0 tx=0).stop 2024-07-26T14:21:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.760+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2270 msgr2=0x7efc5c0c2d30 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T14:21:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.760+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2270 msgr2=0x7efc5c0c2d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:21:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.760+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2270 0x7efc5c0c2d30 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:21:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.761+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:21:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.761+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:21:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:13.763+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c3270 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7efc48005a20 tx=0x7efc6809a000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:21:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:14 smithi112 ceph-mon[37501]: pgmap v3372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:14 smithi107 ceph-mon[32659]: pgmap v3372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:14 smithi132 ceph-mon[37580]: pgmap v3372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:14.709 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:14.708+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:21:14.709 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:14.708+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:21:14.709 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:14.709+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:21:14.709 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:14.709+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:21:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:21:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:21:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:21:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:21:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:21:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5411.866792 secs 2024-07-26T14:21:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:16 smithi112 ceph-mon[37501]: pgmap v3373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:21:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:16 smithi112 ceph-mon[37501]: pgmap v3374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:21:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:21:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:21:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:21:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5411.866792 secs 2024-07-26T14:21:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:16 smithi107 ceph-mon[32659]: pgmap v3373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:21:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:16 smithi107 ceph-mon[32659]: pgmap v3374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:21:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:21:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:21:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:21:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5411.866792 secs 2024-07-26T14:21:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:16 smithi132 ceph-mon[37580]: pgmap v3373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:21:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:16 smithi132 ceph-mon[37580]: pgmap v3374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:21:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:21:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:18 smithi107 ceph-mon[32659]: pgmap v3375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:18.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:18 smithi132 ceph-mon[37580]: pgmap v3375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:18 smithi112 ceph-mon[37501]: pgmap v3375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:20 smithi107 ceph-mon[32659]: pgmap v3376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:21:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:20 smithi132 ceph-mon[37580]: pgmap v3376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:21:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:20 smithi112 ceph-mon[37501]: pgmap v3376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:21:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5416.866925 secs 2024-07-26T14:21:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5416.866925 secs 2024-07-26T14:21:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5416.866925 secs 2024-07-26T14:21:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:22 smithi107 ceph-mon[32659]: pgmap v3377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:21:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:22 smithi132 ceph-mon[37580]: pgmap v3377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:21:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:22 smithi112 ceph-mon[37501]: pgmap v3377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:21:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:24 smithi107 ceph-mon[32659]: pgmap v3378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:21:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:24 smithi132 ceph-mon[37580]: pgmap v3378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:21:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:24 smithi112 ceph-mon[37501]: pgmap v3378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:21:26.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:26.552+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:26.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:26.552+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:26.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:26.552+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:26.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5421.867088 secs 2024-07-26T14:21:26.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:26 smithi107 ceph-mon[32659]: pgmap v3379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:21:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5421.867088 secs 2024-07-26T14:21:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:26 smithi132 ceph-mon[37580]: pgmap v3379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:21:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5421.867088 secs 2024-07-26T14:21:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:26 smithi112 ceph-mon[37501]: pgmap v3379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:21:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:28 smithi132 ceph-mon[37580]: pgmap v3380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:28 smithi112 ceph-mon[37501]: pgmap v3380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:28.712+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:28.712+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:28.713 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:28.712+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:28 smithi107 ceph-mon[32659]: pgmap v3380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:30 smithi132 ceph-mon[37580]: pgmap v3381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:30 smithi112 ceph-mon[37501]: pgmap v3381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:30 smithi107 ceph-mon[32659]: pgmap v3381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:31.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5426.867158 secs 2024-07-26T14:21:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5426.867158 secs 2024-07-26T14:21:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5426.867158 secs 2024-07-26T14:21:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:32 smithi112 ceph-mon[37501]: pgmap v3382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:32 smithi107 ceph-mon[32659]: pgmap v3382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:32 smithi132 ceph-mon[37580]: pgmap v3382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:34 smithi107 ceph-mon[32659]: pgmap v3383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:34 smithi112 ceph-mon[37501]: pgmap v3383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:34 smithi132 ceph-mon[37580]: pgmap v3383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5431.867229 secs 2024-07-26T14:21:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:36 smithi112 ceph-mon[37501]: pgmap v3384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5431.867229 secs 2024-07-26T14:21:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:36 smithi107 ceph-mon[32659]: pgmap v3384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5431.867229 secs 2024-07-26T14:21:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:36 smithi132 ceph-mon[37580]: pgmap v3384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:38.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:38 smithi107 ceph-mon[32659]: pgmap v3385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:38 smithi112 ceph-mon[37501]: pgmap v3385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:38 smithi132 ceph-mon[37580]: pgmap v3385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:40 smithi112 ceph-mon[37501]: pgmap v3386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:40 smithi107 ceph-mon[32659]: pgmap v3386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:40 smithi132 ceph-mon[37580]: pgmap v3386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:41.567+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:41.567+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:41.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:41.567+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5436.867347 secs 2024-07-26T14:21:41.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5436.867347 secs 2024-07-26T14:21:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5436.867347 secs 2024-07-26T14:21:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:42 smithi112 ceph-mon[37501]: pgmap v3387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:42 smithi107 ceph-mon[32659]: pgmap v3387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:42 smithi132 ceph-mon[37580]: pgmap v3387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:43.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:43.727+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:43.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:43.727+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:43.728 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:43.727+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:44 smithi112 ceph-mon[37501]: pgmap v3388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:44 smithi107 ceph-mon[32659]: pgmap v3388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:44 smithi132 ceph-mon[37580]: pgmap v3388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5441.867439 secs 2024-07-26T14:21:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:46 smithi112 ceph-mon[37501]: pgmap v3389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5441.867439 secs 2024-07-26T14:21:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:46 smithi107 ceph-mon[32659]: pgmap v3389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5441.867439 secs 2024-07-26T14:21:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:46 smithi132 ceph-mon[37580]: pgmap v3389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:48 smithi112 ceph-mon[37501]: pgmap v3390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:48 smithi107 ceph-mon[32659]: pgmap v3390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:48 smithi132 ceph-mon[37580]: pgmap v3390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:50 smithi112 ceph-mon[37501]: pgmap v3391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:50 smithi107 ceph-mon[32659]: pgmap v3391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:50 smithi132 ceph-mon[37580]: pgmap v3391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:21:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5446.867540 secs 2024-07-26T14:21:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5446.867540 secs 2024-07-26T14:21:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5446.867540 secs 2024-07-26T14:21:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:52 smithi112 ceph-mon[37501]: pgmap v3392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:52 smithi107 ceph-mon[32659]: pgmap v3392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:52 smithi132 ceph-mon[37580]: pgmap v3392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:54 smithi112 ceph-mon[37501]: pgmap v3393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:54 smithi107 ceph-mon[32659]: pgmap v3393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:54 smithi132 ceph-mon[37580]: pgmap v3393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:56.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:56.571+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:56.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:56.571+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:56.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:56.571+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5451.867686 secs 2024-07-26T14:21:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:56 smithi112 ceph-mon[37501]: pgmap v3394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5451.867686 secs 2024-07-26T14:21:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:56 smithi107 ceph-mon[32659]: pgmap v3394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5451.867686 secs 2024-07-26T14:21:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:56 smithi132 ceph-mon[37580]: pgmap v3394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:57.560 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:57.559+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:21:57.561 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:57.559+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:21:57.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:57.585+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:21:57.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:21:57.585+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:21:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:21:58 smithi112 ceph-mon[37501]: pgmap v3395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:58.744 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:58.743+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:58.743+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:58.743+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:21:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:21:58 smithi107 ceph-mon[32659]: pgmap v3395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:21:58 smithi132 ceph-mon[37580]: pgmap v3395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:21:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:59.715+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:21:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:59.715+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:21:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:59.716+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:21:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:21:59.716+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:22:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:00 smithi112 ceph-mon[37501]: pgmap v3396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:00 smithi107 ceph-mon[32659]: pgmap v3396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:00 smithi132 ceph-mon[37580]: pgmap v3396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5456.867756 secs 2024-07-26T14:22:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5456.867756 secs 2024-07-26T14:22:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5456.867756 secs 2024-07-26T14:22:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:02 smithi112 ceph-mon[37501]: pgmap v3397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:02 smithi107 ceph-mon[32659]: pgmap v3397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:02 smithi132 ceph-mon[37580]: pgmap v3397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:04 smithi112 ceph-mon[37501]: pgmap v3398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:04 smithi107 ceph-mon[32659]: pgmap v3398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:04 smithi132 ceph-mon[37580]: pgmap v3398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5461.867882 secs 2024-07-26T14:22:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:06 smithi112 ceph-mon[37501]: pgmap v3399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5461.867882 secs 2024-07-26T14:22:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:06 smithi107 ceph-mon[32659]: pgmap v3399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5461.867882 secs 2024-07-26T14:22:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:06 smithi132 ceph-mon[37580]: pgmap v3399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:08 smithi112 ceph-mon[37501]: pgmap v3400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:22:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:08 smithi107 ceph-mon[32659]: pgmap v3400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:22:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:08 smithi132 ceph-mon[37580]: pgmap v3400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:22:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:10 smithi112 ceph-mon[37501]: pgmap v3401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:10 smithi107 ceph-mon[32659]: pgmap v3401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:10 smithi132 ceph-mon[37580]: pgmap v3401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:11.574+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:11.576 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:11.575+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:11.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:11.575+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:11.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5466.868032 secs 2024-07-26T14:22:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5466.868032 secs 2024-07-26T14:22:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5466.868032 secs 2024-07-26T14:22:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:12 smithi107 ceph-mon[32659]: pgmap v3402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:12 smithi132 ceph-mon[37580]: pgmap v3402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:12 smithi112 ceph-mon[37501]: pgmap v3402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:13.759+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:13.759+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:13.759+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:14 smithi107 ceph-mon[32659]: pgmap v3403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:14 smithi132 ceph-mon[37580]: pgmap v3403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:14 smithi112 ceph-mon[37501]: pgmap v3403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:22:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:22:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:22:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5471.868196 secs 2024-07-26T14:22:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:16 smithi107 ceph-mon[32659]: pgmap v3404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:22:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:22:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:22:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5471.868196 secs 2024-07-26T14:22:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:16 smithi132 ceph-mon[37580]: pgmap v3404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:22:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:22:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:22:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5471.868196 secs 2024-07-26T14:22:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:16 smithi112 ceph-mon[37501]: pgmap v3404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:22:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:22:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:22:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:17 smithi107 ceph-mon[32659]: pgmap v3405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:22:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:17 smithi107 ceph-mon[32659]: pgmap v3406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:17 smithi132 ceph-mon[37580]: pgmap v3405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:22:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:17 smithi132 ceph-mon[37580]: pgmap v3406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:17 smithi112 ceph-mon[37501]: pgmap v3405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:22:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:17 smithi112 ceph-mon[37501]: pgmap v3406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:19 smithi107 ceph-mon[32659]: pgmap v3407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:19 smithi132 ceph-mon[37580]: pgmap v3407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:19 smithi112 ceph-mon[37501]: pgmap v3407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5476.868258 secs 2024-07-26T14:22:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:21 smithi107 ceph-mon[32659]: pgmap v3408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5476.868258 secs 2024-07-26T14:22:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:21 smithi132 ceph-mon[37580]: pgmap v3408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5476.868258 secs 2024-07-26T14:22:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:21 smithi112 ceph-mon[37501]: pgmap v3408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:23 smithi107 ceph-mon[32659]: pgmap v3409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:22:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:23 smithi132 ceph-mon[37580]: pgmap v3409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:22:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:23 smithi112 ceph-mon[37501]: pgmap v3409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:22:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:25 smithi107 ceph-mon[32659]: pgmap v3410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:25 smithi132 ceph-mon[37580]: pgmap v3410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:25 smithi112 ceph-mon[37501]: pgmap v3410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:22:26.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:26.576+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:26.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:26.576+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:26.577 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:26.576+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5481.868385 secs 2024-07-26T14:22:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5481.868385 secs 2024-07-26T14:22:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5481.868385 secs 2024-07-26T14:22:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:27 smithi107 ceph-mon[32659]: pgmap v3411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:22:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:27 smithi132 ceph-mon[37580]: pgmap v3411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:22:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:27 smithi112 ceph-mon[37501]: pgmap v3411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:22:28.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:28.764+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:28.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:28.764+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:28.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:28.764+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:29 smithi132 ceph-mon[37580]: pgmap v3412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:29 smithi112 ceph-mon[37501]: pgmap v3412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:29 smithi107 ceph-mon[32659]: pgmap v3412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5486.868478 secs 2024-07-26T14:22:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:31 smithi132 ceph-mon[37580]: pgmap v3413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5486.868478 secs 2024-07-26T14:22:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:31 smithi112 ceph-mon[37501]: pgmap v3413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5486.868478 secs 2024-07-26T14:22:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:31 smithi107 ceph-mon[32659]: pgmap v3413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:33 smithi132 ceph-mon[37580]: pgmap v3414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:33 smithi112 ceph-mon[37501]: pgmap v3414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:33 smithi107 ceph-mon[32659]: pgmap v3414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:35 smithi112 ceph-mon[37501]: pgmap v3415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:22:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:35 smithi107 ceph-mon[32659]: pgmap v3415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:22:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:35 smithi132 ceph-mon[37580]: pgmap v3415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:22:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5491.868613 secs 2024-07-26T14:22:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5491.868613 secs 2024-07-26T14:22:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5491.868613 secs 2024-07-26T14:22:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:37 smithi112 ceph-mon[37501]: pgmap v3416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:22:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:37 smithi107 ceph-mon[32659]: pgmap v3416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:22:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:37 smithi132 ceph-mon[37580]: pgmap v3416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:22:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:39 smithi112 ceph-mon[37501]: pgmap v3417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:39 smithi107 ceph-mon[32659]: pgmap v3417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:39 smithi132 ceph-mon[37580]: pgmap v3417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:41.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:41.590+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:41.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:41.591+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:41.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:41.591+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:41.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5496.868747 secs 2024-07-26T14:22:41.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:41 smithi132 ceph-mon[37580]: pgmap v3418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5496.868747 secs 2024-07-26T14:22:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:41 smithi112 ceph-mon[37501]: pgmap v3418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5496.868747 secs 2024-07-26T14:22:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:41 smithi107 ceph-mon[32659]: pgmap v3418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:43.770+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:43.770+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:43.771 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:43.770+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:43 smithi112 ceph-mon[37501]: pgmap v3419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:43 smithi107 ceph-mon[32659]: pgmap v3419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:43 smithi132 ceph-mon[37580]: pgmap v3419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:22:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:45 smithi112 ceph-mon[37501]: pgmap v3420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:22:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:45 smithi107 ceph-mon[32659]: pgmap v3420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:22:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:45 smithi132 ceph-mon[37580]: pgmap v3420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:22:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5501.868868 secs 2024-07-26T14:22:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5501.868868 secs 2024-07-26T14:22:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5501.868868 secs 2024-07-26T14:22:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:47 smithi112 ceph-mon[37501]: pgmap v3421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:22:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:47 smithi107 ceph-mon[32659]: pgmap v3421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:22:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:47 smithi132 ceph-mon[37580]: pgmap v3421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:22:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:49 smithi112 ceph-mon[37501]: pgmap v3422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:49 smithi107 ceph-mon[32659]: pgmap v3422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:49 smithi132 ceph-mon[37580]: pgmap v3422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5506.869031 secs 2024-07-26T14:22:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:51 smithi112 ceph-mon[37501]: pgmap v3423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:22:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5506.869031 secs 2024-07-26T14:22:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:51 smithi107 ceph-mon[32659]: pgmap v3423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:22:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5506.869031 secs 2024-07-26T14:22:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:51 smithi132 ceph-mon[37580]: pgmap v3423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:22:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:53 smithi112 ceph-mon[37501]: pgmap v3424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:53 smithi107 ceph-mon[32659]: pgmap v3424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:53 smithi132 ceph-mon[37580]: pgmap v3424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:55 smithi107 ceph-mon[32659]: pgmap v3425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:22:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:55 smithi132 ceph-mon[37580]: pgmap v3425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:22:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:55 smithi112 ceph-mon[37501]: pgmap v3425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:22:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5511.869116 secs 2024-07-26T14:22:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5511.869116 secs 2024-07-26T14:22:56.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:56.597+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:56.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:56.597+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:56.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:22:56.597+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5511.869116 secs 2024-07-26T14:22:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:57 smithi107 ceph-mon[32659]: pgmap v3426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:22:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:57 smithi132 ceph-mon[37580]: pgmap v3426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:22:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:57 smithi112 ceph-mon[37501]: pgmap v3426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:22:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:58.776+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:58.776+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:22:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:22:58.776+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:22:59 smithi107 ceph-mon[32659]: pgmap v3427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:22:59 smithi132 ceph-mon[37580]: pgmap v3427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:22:59 smithi112 ceph-mon[37501]: pgmap v3427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5516.869235 secs 2024-07-26T14:23:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:01 smithi107 ceph-mon[32659]: pgmap v3428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5516.869235 secs 2024-07-26T14:23:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:01 smithi132 ceph-mon[37580]: pgmap v3428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5516.869235 secs 2024-07-26T14:23:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:01 smithi112 ceph-mon[37501]: pgmap v3428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:03 smithi132 ceph-mon[37580]: pgmap v3429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:03 smithi112 ceph-mon[37501]: pgmap v3429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:03 smithi107 ceph-mon[32659]: pgmap v3429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:05 smithi132 ceph-mon[37580]: pgmap v3430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:05 smithi112 ceph-mon[37501]: pgmap v3430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:06.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:05 smithi107 ceph-mon[32659]: pgmap v3430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5521.869362 secs 2024-07-26T14:23:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5521.869362 secs 2024-07-26T14:23:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5521.869362 secs 2024-07-26T14:23:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:07 smithi112 ceph-mon[37501]: pgmap v3431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:07 smithi107 ceph-mon[32659]: pgmap v3431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:07 smithi132 ceph-mon[37580]: pgmap v3431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:09 smithi112 ceph-mon[37501]: pgmap v3432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:09 smithi107 ceph-mon[32659]: pgmap v3432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:09 smithi132 ceph-mon[37580]: pgmap v3432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:11.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:11.613+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:11.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:11.613+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:11.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:11.613+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:11.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5526.869487 secs 2024-07-26T14:23:11.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:11 smithi132 ceph-mon[37580]: pgmap v3433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5526.869487 secs 2024-07-26T14:23:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:11 smithi112 ceph-mon[37501]: pgmap v3433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5526.869487 secs 2024-07-26T14:23:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:11 smithi107 ceph-mon[32659]: pgmap v3433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:13.792+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:13.792+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:13.792+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:13 smithi112 ceph-mon[37501]: pgmap v3434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:14.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:13 smithi107 ceph-mon[32659]: pgmap v3434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:13 smithi132 ceph-mon[37580]: pgmap v3434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:15 smithi107 ceph-mon[32659]: pgmap v3435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:15 smithi132 ceph-mon[37580]: pgmap v3435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:15 smithi112 ceph-mon[37501]: pgmap v3435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5531.869554 secs 2024-07-26T14:23:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5531.869554 secs 2024-07-26T14:23:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5531.869554 secs 2024-07-26T14:23:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:17 smithi107 ceph-mon[32659]: pgmap v3436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:23:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:17 smithi132 ceph-mon[37580]: pgmap v3436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:23:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:17 smithi112 ceph-mon[37501]: pgmap v3436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:23:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:23:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:23:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:18 smithi107 ceph-mon[32659]: pgmap v3437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:23:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:18 smithi107 ceph-mon[32659]: pgmap v3438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:23:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:23:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:23:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:23:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:18 smithi132 ceph-mon[37580]: pgmap v3437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:23:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:18 smithi132 ceph-mon[37580]: pgmap v3438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:23:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:23:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:23:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:23:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:18 smithi112 ceph-mon[37501]: pgmap v3437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:23:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:18 smithi112 ceph-mon[37501]: pgmap v3438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:23:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:23:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:20 smithi112 ceph-mon[37501]: pgmap v3439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:20 smithi107 ceph-mon[32659]: pgmap v3439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:20 smithi132 ceph-mon[37580]: pgmap v3439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5536.869686 secs 2024-07-26T14:23:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5536.869686 secs 2024-07-26T14:23:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5536.869686 secs 2024-07-26T14:23:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:22 smithi112 ceph-mon[37501]: pgmap v3440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:22 smithi107 ceph-mon[32659]: pgmap v3440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:22 smithi132 ceph-mon[37580]: pgmap v3440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:24 smithi112 ceph-mon[37501]: pgmap v3441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:23:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:24 smithi107 ceph-mon[32659]: pgmap v3441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:23:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:24 smithi132 ceph-mon[37580]: pgmap v3441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:23:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:26.629+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:26.629+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:26.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:26.629+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5541.869831 secs 2024-07-26T14:23:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:26 smithi107 ceph-mon[32659]: pgmap v3442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:23:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5541.869831 secs 2024-07-26T14:23:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:26 smithi132 ceph-mon[37580]: pgmap v3442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:23:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5541.869831 secs 2024-07-26T14:23:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:26 smithi112 ceph-mon[37501]: pgmap v3442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:23:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:28.807+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:28.808+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:28.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:28.808+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:28 smithi107 ceph-mon[32659]: pgmap v3443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:23:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:28 smithi132 ceph-mon[37580]: pgmap v3443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:23:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:28 smithi112 ceph-mon[37501]: pgmap v3443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:23:31.002 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:30 smithi112 ceph-mon[37501]: pgmap v3444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:30 smithi107 ceph-mon[32659]: pgmap v3444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:30 smithi132 ceph-mon[37580]: pgmap v3444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5546.869978 secs 2024-07-26T14:23:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5546.869978 secs 2024-07-26T14:23:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5546.869978 secs 2024-07-26T14:23:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:32 smithi107 ceph-mon[32659]: pgmap v3445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:32 smithi132 ceph-mon[37580]: pgmap v3445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:32 smithi112 ceph-mon[37501]: pgmap v3445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:34 smithi107 ceph-mon[32659]: pgmap v3446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:34 smithi132 ceph-mon[37580]: pgmap v3446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:34 smithi112 ceph-mon[37501]: pgmap v3446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5551.870109 secs 2024-07-26T14:23:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:36 smithi107 ceph-mon[32659]: pgmap v3447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5551.870109 secs 2024-07-26T14:23:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:36 smithi132 ceph-mon[37580]: pgmap v3447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5551.870109 secs 2024-07-26T14:23:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:36 smithi112 ceph-mon[37501]: pgmap v3447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:38 smithi107 ceph-mon[32659]: pgmap v3448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:38 smithi132 ceph-mon[37580]: pgmap v3448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:38 smithi112 ceph-mon[37501]: pgmap v3448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:40 smithi107 ceph-mon[32659]: pgmap v3449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:40 smithi132 ceph-mon[37580]: pgmap v3449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:40 smithi112 ceph-mon[37501]: pgmap v3449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:41.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:41.639+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:41.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:41.639+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:41.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:41.639+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:42.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5556.870160 secs 2024-07-26T14:23:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5556.870160 secs 2024-07-26T14:23:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5556.870160 secs 2024-07-26T14:23:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:42 smithi107 ceph-mon[32659]: pgmap v3450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:42 smithi132 ceph-mon[37580]: pgmap v3450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:42 smithi112 ceph-mon[37501]: pgmap v3450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:43.816+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:43.817 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:43.816+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:43.816+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:44 smithi107 ceph-mon[32659]: pgmap v3451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:44 smithi132 ceph-mon[37580]: pgmap v3451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:44 smithi112 ceph-mon[37501]: pgmap v3451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5561.870285 secs 2024-07-26T14:23:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:46 smithi107 ceph-mon[32659]: pgmap v3452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5561.870285 secs 2024-07-26T14:23:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:46 smithi132 ceph-mon[37580]: pgmap v3452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5561.870285 secs 2024-07-26T14:23:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:46 smithi112 ceph-mon[37501]: pgmap v3452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:48 smithi107 ceph-mon[32659]: pgmap v3453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:48 smithi132 ceph-mon[37580]: pgmap v3453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:48 smithi112 ceph-mon[37501]: pgmap v3453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:50 smithi107 ceph-mon[32659]: pgmap v3454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:50 smithi132 ceph-mon[37580]: pgmap v3454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:51.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:50 smithi112 ceph-mon[37501]: pgmap v3454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:23:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5566.870399 secs 2024-07-26T14:23:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5566.870399 secs 2024-07-26T14:23:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5566.870399 secs 2024-07-26T14:23:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:52 smithi132 ceph-mon[37580]: pgmap v3455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:52 smithi112 ceph-mon[37501]: pgmap v3455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:52 smithi107 ceph-mon[32659]: pgmap v3455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:54 smithi132 ceph-mon[37580]: pgmap v3456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:54 smithi112 ceph-mon[37501]: pgmap v3456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:54 smithi107 ceph-mon[32659]: pgmap v3456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:56.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:56.652+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:56.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:56.652+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:56.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:23:56.652+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5571.870528 secs 2024-07-26T14:23:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:56 smithi107 ceph-mon[32659]: pgmap v3457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5571.870528 secs 2024-07-26T14:23:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:56 smithi112 ceph-mon[37501]: pgmap v3457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5571.870528 secs 2024-07-26T14:23:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:56 smithi132 ceph-mon[37580]: pgmap v3457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:58.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:58.831+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:58.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:58.831+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:23:58.833 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:23:58.831+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:23:58 smithi112 ceph-mon[37501]: pgmap v3458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:23:58 smithi107 ceph-mon[32659]: pgmap v3458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:23:58 smithi132 ceph-mon[37580]: pgmap v3458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:00 smithi112 ceph-mon[37501]: pgmap v3459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:24:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:00 smithi107 ceph-mon[32659]: pgmap v3459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:24:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:00 smithi132 ceph-mon[37580]: pgmap v3459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:24:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5576.870652 secs 2024-07-26T14:24:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5576.870652 secs 2024-07-26T14:24:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5576.870652 secs 2024-07-26T14:24:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:02 smithi112 ceph-mon[37501]: pgmap v3460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:02 smithi107 ceph-mon[32659]: pgmap v3460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:02 smithi132 ceph-mon[37580]: pgmap v3460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:04 smithi112 ceph-mon[37501]: pgmap v3461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:24:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:04 smithi107 ceph-mon[32659]: pgmap v3461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:24:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:04 smithi132 ceph-mon[37580]: pgmap v3461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:24:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5581.870799 secs 2024-07-26T14:24:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:06 smithi112 ceph-mon[37501]: pgmap v3462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:24:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5581.870799 secs 2024-07-26T14:24:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:06 smithi107 ceph-mon[32659]: pgmap v3462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:24:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5581.870799 secs 2024-07-26T14:24:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:06 smithi132 ceph-mon[37580]: pgmap v3462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:24:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:08 smithi112 ceph-mon[37501]: pgmap v3463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:24:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:08 smithi107 ceph-mon[32659]: pgmap v3463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:24:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:08 smithi132 ceph-mon[37580]: pgmap v3463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:24:11.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:10 smithi132 ceph-mon[37580]: pgmap v3464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:24:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:10 smithi112 ceph-mon[37501]: pgmap v3464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:24:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:10 smithi107 ceph-mon[32659]: pgmap v3464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:24:11.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:11.669+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:11.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:11.669+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:11.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:11.669+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5586.870925 secs 2024-07-26T14:24:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5586.870925 secs 2024-07-26T14:24:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5586.870925 secs 2024-07-26T14:24:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:12 smithi112 ceph-mon[37501]: pgmap v3465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:24:13.275 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:12 smithi107 ceph-mon[32659]: pgmap v3465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:24:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:12 smithi132 ceph-mon[37580]: pgmap v3465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:24:13.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:13.582+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:24:13.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:13.583+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:24:13.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:13.607+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:24:13.608 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:13.608+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:24:13.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:13.848+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:13.848 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:13.848+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:13.848+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:14.742+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:24:14.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:14.742+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:24:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:14.742+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:24:14.743 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:14.743+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:24:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:14 smithi112 ceph-mon[37501]: pgmap v3466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:24:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:14 smithi107 ceph-mon[32659]: pgmap v3466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:24:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:14 smithi132 ceph-mon[37580]: pgmap v3466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:24:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5591.871055 secs 2024-07-26T14:24:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:16 smithi112 ceph-mon[37501]: pgmap v3467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5591.871055 secs 2024-07-26T14:24:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:16 smithi107 ceph-mon[32659]: pgmap v3467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5591.871055 secs 2024-07-26T14:24:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:16 smithi132 ceph-mon[37580]: pgmap v3467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:18.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:24:18.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:24:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:24:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: pgmap v3468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:24:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:24:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:24:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: pgmap v3468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:24:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:24:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:24:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: pgmap v3468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:24:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:19 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:24:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:19 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:24:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:19 smithi112 ceph-mon[37501]: pgmap v3469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:24:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:19 smithi112 ceph-mon[37501]: pgmap v3470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:24:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:19 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:24:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:19 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:24:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:19 smithi107 ceph-mon[32659]: pgmap v3469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:24:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:19 smithi107 ceph-mon[32659]: pgmap v3470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:24:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:19 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:24:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:19 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:24:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:19 smithi132 ceph-mon[37580]: pgmap v3469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:24:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:19 smithi132 ceph-mon[37580]: pgmap v3470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:24:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5596.871176 secs 2024-07-26T14:24:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:21 smithi112 ceph-mon[37501]: pgmap v3471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:24:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5596.871176 secs 2024-07-26T14:24:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:21 smithi107 ceph-mon[32659]: pgmap v3471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:24:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5596.871176 secs 2024-07-26T14:24:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:21 smithi132 ceph-mon[37580]: pgmap v3471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:24:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:23 smithi112 ceph-mon[37501]: pgmap v3472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:24:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:23 smithi107 ceph-mon[32659]: pgmap v3472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:24:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:23 smithi132 ceph-mon[37580]: pgmap v3472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:24:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:25 smithi112 ceph-mon[37501]: pgmap v3473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:24:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:25 smithi107 ceph-mon[32659]: pgmap v3473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:24:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:25 smithi132 ceph-mon[37580]: pgmap v3473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:24:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:26.684+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:26.684+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:26.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:26.684+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:27.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5601.871263 secs 2024-07-26T14:24:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5601.871263 secs 2024-07-26T14:24:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5601.871263 secs 2024-07-26T14:24:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:27 smithi112 ceph-mon[37501]: pgmap v3474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:27 smithi107 ceph-mon[32659]: pgmap v3474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:27 smithi132 ceph-mon[37580]: pgmap v3474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:28.851+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:28.851+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:28.851+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:29 smithi112 ceph-mon[37501]: pgmap v3475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:24:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:29 smithi107 ceph-mon[32659]: pgmap v3475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:24:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:29 smithi132 ceph-mon[37580]: pgmap v3475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:24:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5606.871384 secs 2024-07-26T14:24:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:31 smithi112 ceph-mon[37501]: pgmap v3476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5606.871384 secs 2024-07-26T14:24:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:31 smithi107 ceph-mon[32659]: pgmap v3476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5606.871384 secs 2024-07-26T14:24:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:31 smithi132 ceph-mon[37580]: pgmap v3476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:33 smithi107 ceph-mon[32659]: pgmap v3477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:33 smithi132 ceph-mon[37580]: pgmap v3477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:33 smithi112 ceph-mon[37501]: pgmap v3477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:35 smithi107 ceph-mon[32659]: pgmap v3478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:35 smithi132 ceph-mon[37580]: pgmap v3478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:35 smithi112 ceph-mon[37501]: pgmap v3478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5611.871504 secs 2024-07-26T14:24:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5611.871504 secs 2024-07-26T14:24:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5611.871504 secs 2024-07-26T14:24:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:37 smithi107 ceph-mon[32659]: pgmap v3479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:37 smithi132 ceph-mon[37580]: pgmap v3479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:37 smithi112 ceph-mon[37501]: pgmap v3479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:39 smithi107 ceph-mon[32659]: pgmap v3480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:39 smithi132 ceph-mon[37580]: pgmap v3480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:39 smithi112 ceph-mon[37501]: pgmap v3480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5616.871634 secs 2024-07-26T14:24:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5616.871634 secs 2024-07-26T14:24:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5616.871634 secs 2024-07-26T14:24:41.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:41.699+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:41.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:41.699+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:41.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:41.699+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:41 smithi107 ceph-mon[32659]: pgmap v3481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:41 smithi132 ceph-mon[37580]: pgmap v3481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:41 smithi112 ceph-mon[37501]: pgmap v3481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:43.859 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:43.857+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:43.859 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:43.857+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:43.859 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:43.857+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:43 smithi107 ceph-mon[32659]: pgmap v3482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:43 smithi132 ceph-mon[37580]: pgmap v3482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:43 smithi112 ceph-mon[37501]: pgmap v3482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:46 smithi107 ceph-mon[32659]: pgmap v3483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5621.871760 secs 2024-07-26T14:24:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:46 smithi132 ceph-mon[37580]: pgmap v3483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5621.871760 secs 2024-07-26T14:24:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:46 smithi112 ceph-mon[37501]: pgmap v3483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5621.871760 secs 2024-07-26T14:24:48.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:48 smithi107 ceph-mon[32659]: pgmap v3484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:48 smithi132 ceph-mon[37580]: pgmap v3484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:48 smithi112 ceph-mon[37501]: pgmap v3484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:50 smithi107 ceph-mon[32659]: pgmap v3485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:50 smithi132 ceph-mon[37580]: pgmap v3485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:50 smithi112 ceph-mon[37501]: pgmap v3485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5626.871836 secs 2024-07-26T14:24:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5626.871836 secs 2024-07-26T14:24:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5626.871836 secs 2024-07-26T14:24:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:52 smithi107 ceph-mon[32659]: pgmap v3486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:52 smithi132 ceph-mon[37580]: pgmap v3486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:52 smithi112 ceph-mon[37501]: pgmap v3486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:54 smithi107 ceph-mon[32659]: pgmap v3487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:54.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:54 smithi132 ceph-mon[37580]: pgmap v3487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:54 smithi112 ceph-mon[37501]: pgmap v3487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:56 smithi107 ceph-mon[32659]: pgmap v3488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5631.871964 secs 2024-07-26T14:24:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:56 smithi132 ceph-mon[37580]: pgmap v3488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5631.871964 secs 2024-07-26T14:24:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:56 smithi112 ceph-mon[37501]: pgmap v3488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5631.871964 secs 2024-07-26T14:24:56.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:56.714+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:56.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:56.714+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:56.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:24:56.714+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:24:58 smithi132 ceph-mon[37580]: pgmap v3489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:24:58 smithi112 ceph-mon[37501]: pgmap v3489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:24:58 smithi107 ceph-mon[32659]: pgmap v3489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:58.864+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:58.864+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:24:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:24:58.864+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:00 smithi107 ceph-mon[32659]: pgmap v3490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:00 smithi132 ceph-mon[37580]: pgmap v3490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:00 smithi112 ceph-mon[37501]: pgmap v3490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5636.872109 secs 2024-07-26T14:25:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5636.872109 secs 2024-07-26T14:25:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5636.872109 secs 2024-07-26T14:25:02.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:02 smithi112 ceph-mon[37501]: pgmap v3491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:02 smithi107 ceph-mon[32659]: pgmap v3491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:02 smithi132 ceph-mon[37580]: pgmap v3491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:04 smithi112 ceph-mon[37501]: pgmap v3492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:04 smithi107 ceph-mon[32659]: pgmap v3492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:04 smithi132 ceph-mon[37580]: pgmap v3492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:06 smithi112 ceph-mon[37501]: pgmap v3493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5641.872234 secs 2024-07-26T14:25:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:06 smithi107 ceph-mon[32659]: pgmap v3493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5641.872234 secs 2024-07-26T14:25:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:06 smithi132 ceph-mon[37580]: pgmap v3493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5641.872234 secs 2024-07-26T14:25:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:08 smithi112 ceph-mon[37501]: pgmap v3494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:08 smithi107 ceph-mon[32659]: pgmap v3494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:08 smithi132 ceph-mon[37580]: pgmap v3494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:10 smithi112 ceph-mon[37501]: pgmap v3495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:10 smithi107 ceph-mon[32659]: pgmap v3495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:10 smithi132 ceph-mon[37580]: pgmap v3495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5646.872382 secs 2024-07-26T14:25:11.460 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5646.872382 secs 2024-07-26T14:25:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5646.872382 secs 2024-07-26T14:25:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:11.720+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:11.720+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:11.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:11.720+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:12 smithi112 ceph-mon[37501]: pgmap v3496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:12 smithi107 ceph-mon[32659]: pgmap v3496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:12 smithi132 ceph-mon[37580]: pgmap v3496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:13.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:13.873+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:13.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:13.873+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:13.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:13.873+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:14 smithi112 ceph-mon[37501]: pgmap v3497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:14 smithi107 ceph-mon[32659]: pgmap v3497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:14 smithi132 ceph-mon[37580]: pgmap v3497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:15 smithi107 ceph-mon[32659]: pgmap v3498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:15 smithi132 ceph-mon[37580]: pgmap v3498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:15 smithi112 ceph-mon[37501]: pgmap v3498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5651.872504 secs 2024-07-26T14:25:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5651.872504 secs 2024-07-26T14:25:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5651.872504 secs 2024-07-26T14:25:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:17 smithi107 ceph-mon[32659]: pgmap v3499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:17 smithi132 ceph-mon[37580]: pgmap v3499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:17 smithi112 ceph-mon[37501]: pgmap v3499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: pgmap v3500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:25:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: pgmap v3500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: pgmap v3500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:20.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:20 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:25:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:20 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:25:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:20 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:25:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:20 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:25:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:25:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:25:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:20 smithi112 ceph-mon[37501]: pgmap v3501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:25:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:20 smithi112 ceph-mon[37501]: pgmap v3502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:25:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:25:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:20 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:25:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:20 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:25:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:20 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:25:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:20 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:25:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:25:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:25:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:20 smithi107 ceph-mon[32659]: pgmap v3501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:25:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:20 smithi107 ceph-mon[32659]: pgmap v3502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:25:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:25:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:20 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:25:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:20 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:25:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:20 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:25:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:20 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:25:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:25:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:25:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:20 smithi132 ceph-mon[37580]: pgmap v3501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:25:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:20 smithi132 ceph-mon[37580]: pgmap v3502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:25:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:25:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5656.872595 secs 2024-07-26T14:25:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5656.872595 secs 2024-07-26T14:25:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5656.872595 secs 2024-07-26T14:25:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:22 smithi112 ceph-mon[37501]: pgmap v3503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:22 smithi107 ceph-mon[32659]: pgmap v3503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:22 smithi132 ceph-mon[37580]: pgmap v3503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:25 smithi132 ceph-mon[37580]: pgmap v3504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:25:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:25 smithi112 ceph-mon[37501]: pgmap v3504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:25:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:25 smithi107 ceph-mon[32659]: pgmap v3504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:25:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5661.872725 secs 2024-07-26T14:25:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5661.872725 secs 2024-07-26T14:25:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5661.872725 secs 2024-07-26T14:25:26.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:26.730+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:26.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:26.730+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:26.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:26.730+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:27 smithi112 ceph-mon[37501]: pgmap v3505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:25:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:27 smithi107 ceph-mon[32659]: pgmap v3505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:25:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:27 smithi132 ceph-mon[37580]: pgmap v3505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:25:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:28.889+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:28.889+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:28.889+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:29 smithi112 ceph-mon[37501]: pgmap v3506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:25:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:29 smithi107 ceph-mon[32659]: pgmap v3506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:25:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:29 smithi132 ceph-mon[37580]: pgmap v3506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:25:31.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:31 smithi107 ceph-mon[32659]: pgmap v3507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:25:31.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5666.872824 secs 2024-07-26T14:25:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:31 smithi112 ceph-mon[37501]: pgmap v3507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:25:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5666.872824 secs 2024-07-26T14:25:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:31 smithi132 ceph-mon[37580]: pgmap v3507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:25:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5666.872824 secs 2024-07-26T14:25:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:33 smithi112 ceph-mon[37501]: pgmap v3508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:33 smithi107 ceph-mon[32659]: pgmap v3508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:33 smithi132 ceph-mon[37580]: pgmap v3508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:35 smithi112 ceph-mon[37501]: pgmap v3509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:35 smithi107 ceph-mon[32659]: pgmap v3509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:35 smithi132 ceph-mon[37580]: pgmap v3509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5671.872988 secs 2024-07-26T14:25:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5671.872988 secs 2024-07-26T14:25:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5671.872988 secs 2024-07-26T14:25:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:37 smithi112 ceph-mon[37501]: pgmap v3510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:37 smithi107 ceph-mon[32659]: pgmap v3510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:37 smithi132 ceph-mon[37580]: pgmap v3510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:39 smithi112 ceph-mon[37501]: pgmap v3511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:39 smithi107 ceph-mon[32659]: pgmap v3511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:39 smithi132 ceph-mon[37580]: pgmap v3511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:41 smithi112 ceph-mon[37501]: pgmap v3512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5676.873132 secs 2024-07-26T14:25:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:41 smithi107 ceph-mon[32659]: pgmap v3512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5676.873132 secs 2024-07-26T14:25:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:41 smithi132 ceph-mon[37580]: pgmap v3512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5676.873132 secs 2024-07-26T14:25:41.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:41.745+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:41.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:41.745+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:41.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:41.745+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:43 smithi112 ceph-mon[37501]: pgmap v3513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:43 smithi107 ceph-mon[32659]: pgmap v3513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:43 smithi132 ceph-mon[37580]: pgmap v3513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:43.889+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:43.889+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:43.889+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:44 smithi112 ceph-mon[37501]: pgmap v3514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:44 smithi107 ceph-mon[32659]: pgmap v3514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:44 smithi132 ceph-mon[37580]: pgmap v3514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5681.873255 secs 2024-07-26T14:25:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5681.873255 secs 2024-07-26T14:25:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5681.873255 secs 2024-07-26T14:25:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:47 smithi107 ceph-mon[32659]: pgmap v3515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:47 smithi112 ceph-mon[37501]: pgmap v3515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:47 smithi132 ceph-mon[37580]: pgmap v3515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:49 smithi112 ceph-mon[37501]: pgmap v3516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:49 smithi107 ceph-mon[32659]: pgmap v3516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:49 smithi132 ceph-mon[37580]: pgmap v3516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:51 smithi112 ceph-mon[37501]: pgmap v3517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5686.873378 secs 2024-07-26T14:25:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:51 smithi107 ceph-mon[32659]: pgmap v3517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5686.873378 secs 2024-07-26T14:25:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:51 smithi132 ceph-mon[37580]: pgmap v3517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5686.873378 secs 2024-07-26T14:25:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:53 smithi112 ceph-mon[37501]: pgmap v3518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:53 smithi107 ceph-mon[32659]: pgmap v3518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:53.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:53 smithi132 ceph-mon[37580]: pgmap v3518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:55 smithi112 ceph-mon[37501]: pgmap v3519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:55 smithi107 ceph-mon[32659]: pgmap v3519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:55 smithi132 ceph-mon[37580]: pgmap v3519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5691.873506 secs 2024-07-26T14:25:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5691.873506 secs 2024-07-26T14:25:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5691.873506 secs 2024-07-26T14:25:56.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:56.753+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:56.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:56.753+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:56.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:25:56.753+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:57 smithi112 ceph-mon[37501]: pgmap v3520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:57.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:57 smithi107 ceph-mon[32659]: pgmap v3520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:57 smithi132 ceph-mon[37580]: pgmap v3520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:58.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:58.905+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:58.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:58.905+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:25:58.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:25:58.905+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:25:59 smithi112 ceph-mon[37501]: pgmap v3521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:25:59 smithi107 ceph-mon[32659]: pgmap v3521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:25:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:25:59 smithi132 ceph-mon[37580]: pgmap v3521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:01 smithi112 ceph-mon[37501]: pgmap v3522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5696.873637 secs 2024-07-26T14:26:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:01 smithi107 ceph-mon[32659]: pgmap v3522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5696.873637 secs 2024-07-26T14:26:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:01 smithi132 ceph-mon[37580]: pgmap v3522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5696.873637 secs 2024-07-26T14:26:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:02 smithi112 ceph-mon[37501]: pgmap v3523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:02 smithi107 ceph-mon[32659]: pgmap v3523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:02 smithi132 ceph-mon[37580]: pgmap v3523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:05 smithi112 ceph-mon[37501]: pgmap v3524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:05 smithi107 ceph-mon[32659]: pgmap v3524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:05 smithi132 ceph-mon[37580]: pgmap v3524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5701.873776 secs 2024-07-26T14:26:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5701.873776 secs 2024-07-26T14:26:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5701.873776 secs 2024-07-26T14:26:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:07 smithi112 ceph-mon[37501]: pgmap v3525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:07 smithi107 ceph-mon[32659]: pgmap v3525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:07 smithi132 ceph-mon[37580]: pgmap v3525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:09 smithi112 ceph-mon[37501]: pgmap v3526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:09 smithi107 ceph-mon[32659]: pgmap v3526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:09 smithi132 ceph-mon[37580]: pgmap v3526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:11.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:11 smithi132 ceph-mon[37580]: pgmap v3527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:11.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5706.873924 secs 2024-07-26T14:26:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:11 smithi112 ceph-mon[37501]: pgmap v3527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5706.873924 secs 2024-07-26T14:26:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:11 smithi107 ceph-mon[32659]: pgmap v3527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5706.873924 secs 2024-07-26T14:26:11.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:11.770+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:11.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:11.770+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:11.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:11.770+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:13 smithi112 ceph-mon[37501]: pgmap v3528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:13 smithi107 ceph-mon[32659]: pgmap v3528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:13 smithi132 ceph-mon[37580]: pgmap v3528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:13.917+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:13.917+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:13.918 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:13.917+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:15 smithi112 ceph-mon[37501]: pgmap v3529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:15 smithi107 ceph-mon[32659]: pgmap v3529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:15 smithi132 ceph-mon[37580]: pgmap v3529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5711.874101 secs 2024-07-26T14:26:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5711.874101 secs 2024-07-26T14:26:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5711.874101 secs 2024-07-26T14:26:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:17 smithi112 ceph-mon[37501]: pgmap v3530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:17.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:17 smithi107 ceph-mon[32659]: pgmap v3530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:17 smithi132 ceph-mon[37580]: pgmap v3530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:18 smithi112 ceph-mon[37501]: pgmap v3531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:18 smithi107 ceph-mon[32659]: pgmap v3531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:18 smithi132 ceph-mon[37580]: pgmap v3531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:21 smithi112 ceph-mon[37501]: pgmap v3532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:26:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5716.874168 secs 2024-07-26T14:26:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:21 smithi107 ceph-mon[32659]: pgmap v3532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:26:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5716.874168 secs 2024-07-26T14:26:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:21 smithi132 ceph-mon[37580]: pgmap v3532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:26:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5716.874168 secs 2024-07-26T14:26:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:26:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:26:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:26:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:26:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:26:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:26:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:26:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:26:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:26:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:23 smithi112 ceph-mon[37501]: pgmap v3533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:26:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:23 smithi112 ceph-mon[37501]: pgmap v3534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:26:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:23 smithi107 ceph-mon[32659]: pgmap v3533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:26:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:23 smithi107 ceph-mon[32659]: pgmap v3534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:26:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:23 smithi132 ceph-mon[37580]: pgmap v3533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:26:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:23 smithi132 ceph-mon[37580]: pgmap v3534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:26:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:25 smithi112 ceph-mon[37501]: pgmap v3535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:26:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:25 smithi107 ceph-mon[32659]: pgmap v3535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:26:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:25 smithi132 ceph-mon[37580]: pgmap v3535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:26:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5721.874293 secs 2024-07-26T14:26:26.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5721.874293 secs 2024-07-26T14:26:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5721.874293 secs 2024-07-26T14:26:26.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:26.785+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:26.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:26.785+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:26.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:26.785+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:27 smithi112 ceph-mon[37501]: pgmap v3536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:26:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:27 smithi107 ceph-mon[32659]: pgmap v3536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:26:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:27 smithi132 ceph-mon[37580]: pgmap v3536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:26:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:28.920+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:28.920+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:28.920+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:29 smithi112 ceph-mon[37501]: pgmap v3537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:26:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:29 smithi107 ceph-mon[32659]: pgmap v3537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:26:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:29 smithi132 ceph-mon[37580]: pgmap v3537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:26:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:31 smithi112 ceph-mon[37501]: pgmap v3538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:26:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5726.874434 secs 2024-07-26T14:26:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:31 smithi107 ceph-mon[32659]: pgmap v3538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:26:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5726.874434 secs 2024-07-26T14:26:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:31 smithi132 ceph-mon[37580]: pgmap v3538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:26:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5726.874434 secs 2024-07-26T14:26:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:32 smithi112 ceph-mon[37501]: pgmap v3539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:26:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:32 smithi107 ceph-mon[32659]: pgmap v3539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:26:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:32 smithi132 ceph-mon[37580]: pgmap v3539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:26:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:34 smithi107 ceph-mon[32659]: pgmap v3540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:26:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:34 smithi132 ceph-mon[37580]: pgmap v3540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:26:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:34 smithi112 ceph-mon[37501]: pgmap v3540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:26:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:36 smithi107 ceph-mon[32659]: pgmap v3541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5731.874502 secs 2024-07-26T14:26:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:36 smithi132 ceph-mon[37580]: pgmap v3541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5731.874502 secs 2024-07-26T14:26:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:36 smithi112 ceph-mon[37501]: pgmap v3541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5731.874502 secs 2024-07-26T14:26:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:38 smithi107 ceph-mon[32659]: pgmap v3542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:38 smithi132 ceph-mon[37580]: pgmap v3542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:38 smithi112 ceph-mon[37501]: pgmap v3542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:40 smithi107 ceph-mon[32659]: pgmap v3543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:40 smithi132 ceph-mon[37580]: pgmap v3543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:40 smithi112 ceph-mon[37501]: pgmap v3543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5736.874570 secs 2024-07-26T14:26:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5736.874570 secs 2024-07-26T14:26:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5736.874570 secs 2024-07-26T14:26:41.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:41.792+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:41.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:41.792+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:41.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:41.792+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:42 smithi107 ceph-mon[32659]: pgmap v3544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:42 smithi132 ceph-mon[37580]: pgmap v3544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:42 smithi112 ceph-mon[37501]: pgmap v3544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:43.922+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:43.922+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:43.922+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:44 smithi132 ceph-mon[37580]: pgmap v3545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:44 smithi112 ceph-mon[37501]: pgmap v3545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:44 smithi107 ceph-mon[32659]: pgmap v3545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5741.874691 secs 2024-07-26T14:26:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:46 smithi132 ceph-mon[37580]: pgmap v3546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5741.874691 secs 2024-07-26T14:26:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:46 smithi112 ceph-mon[37501]: pgmap v3546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5741.874691 secs 2024-07-26T14:26:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:46 smithi107 ceph-mon[32659]: pgmap v3546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:48 smithi132 ceph-mon[37580]: pgmap v3547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:48 smithi112 ceph-mon[37501]: pgmap v3547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:48 smithi107 ceph-mon[32659]: pgmap v3547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:50 smithi112 ceph-mon[37501]: pgmap v3548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:50 smithi107 ceph-mon[32659]: pgmap v3548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:50.815 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:50 smithi132 ceph-mon[37580]: pgmap v3548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5746.874786 secs 2024-07-26T14:26:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5746.874786 secs 2024-07-26T14:26:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5746.874786 secs 2024-07-26T14:26:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:52 smithi112 ceph-mon[37501]: pgmap v3549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:52 smithi107 ceph-mon[32659]: pgmap v3549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:52 smithi132 ceph-mon[37580]: pgmap v3549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:54 smithi112 ceph-mon[37501]: pgmap v3550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:54.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:54 smithi107 ceph-mon[32659]: pgmap v3550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:54 smithi132 ceph-mon[37580]: pgmap v3550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:26:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5751.874931 secs 2024-07-26T14:26:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:56 smithi112 ceph-mon[37501]: pgmap v3551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5751.874931 secs 2024-07-26T14:26:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:56 smithi107 ceph-mon[32659]: pgmap v3551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5751.874931 secs 2024-07-26T14:26:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:56 smithi132 ceph-mon[37580]: pgmap v3551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:56.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:56.808+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:56.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:56.808+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:56.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:26:56.808+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:26:58 smithi112 ceph-mon[37501]: pgmap v3552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:26:58 smithi107 ceph-mon[32659]: pgmap v3552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:26:58 smithi132 ceph-mon[37580]: pgmap v3552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:26:58.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:58.924+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:58.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:58.924+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:26:58.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:26:58.924+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:00 smithi112 ceph-mon[37501]: pgmap v3553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:00 smithi107 ceph-mon[32659]: pgmap v3553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:00 smithi132 ceph-mon[37580]: pgmap v3553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5756.875061 secs 2024-07-26T14:27:01.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5756.875061 secs 2024-07-26T14:27:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5756.875061 secs 2024-07-26T14:27:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:02 smithi112 ceph-mon[37501]: pgmap v3554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:02 smithi107 ceph-mon[32659]: pgmap v3554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:02 smithi132 ceph-mon[37580]: pgmap v3554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:04 smithi107 ceph-mon[32659]: pgmap v3555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:04 smithi112 ceph-mon[37501]: pgmap v3555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:04 smithi132 ceph-mon[37580]: pgmap v3555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:06 smithi112 ceph-mon[37501]: 1 slow requests, 1 included below; oldest blocked for > 5761.875184 secs 2024-07-26T14:27:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:06 smithi112 ceph-mon[37501]: slow request 5761.875183 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T14:27:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:06 smithi112 ceph-mon[37501]: pgmap v3556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:06 smithi107 ceph-mon[32659]: 1 slow requests, 1 included below; oldest blocked for > 5761.875184 secs 2024-07-26T14:27:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:06 smithi107 ceph-mon[32659]: slow request 5761.875183 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T14:27:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:06 smithi107 ceph-mon[32659]: pgmap v3556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:06 smithi132 ceph-mon[37580]: 1 slow requests, 1 included below; oldest blocked for > 5761.875184 secs 2024-07-26T14:27:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:06 smithi132 ceph-mon[37580]: slow request 5761.875183 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T14:27:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:06 smithi132 ceph-mon[37580]: pgmap v3556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:08 smithi112 ceph-mon[37501]: pgmap v3557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:08 smithi107 ceph-mon[32659]: pgmap v3557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:08 smithi132 ceph-mon[37580]: pgmap v3557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:10 smithi112 ceph-mon[37501]: pgmap v3558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:10 smithi107 ceph-mon[32659]: pgmap v3558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:10 smithi132 ceph-mon[37580]: pgmap v3558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5766.875306 secs 2024-07-26T14:27:11.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5766.875306 secs 2024-07-26T14:27:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5766.875306 secs 2024-07-26T14:27:11.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:11.824+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:11.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:11.824+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:11.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:11.824+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:12 smithi112 ceph-mon[37501]: pgmap v3559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:12 smithi107 ceph-mon[32659]: pgmap v3559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:12 smithi132 ceph-mon[37580]: pgmap v3559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:13.612+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:27:13.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:13.613+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:27:13.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:13.636+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:27:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:13.637+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:27:13.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:13.941+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:13.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:13.941+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:13.942 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:13.941+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:14 smithi112 ceph-mon[37501]: pgmap v3560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:14.774+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:27:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:14.774+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:27:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:14.775+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:27:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:14.775+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:27:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:14 smithi107 ceph-mon[32659]: pgmap v3560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:14 smithi132 ceph-mon[37580]: pgmap v3560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5771.875430 secs 2024-07-26T14:27:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:16 smithi112 ceph-mon[37501]: pgmap v3561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5771.875430 secs 2024-07-26T14:27:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:16 smithi107 ceph-mon[32659]: pgmap v3561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5771.875430 secs 2024-07-26T14:27:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:16 smithi132 ceph-mon[37580]: pgmap v3561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:18 smithi112 ceph-mon[37501]: pgmap v3562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:18 smithi107 ceph-mon[32659]: pgmap v3562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:18.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:18 smithi132 ceph-mon[37580]: pgmap v3562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:20 smithi112 ceph-mon[37501]: pgmap v3563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:20 smithi107 ceph-mon[32659]: pgmap v3563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:20 smithi132 ceph-mon[37580]: pgmap v3563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5776.875560 secs 2024-07-26T14:27:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:27:21.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5776.875560 secs 2024-07-26T14:27:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:27:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5776.875560 secs 2024-07-26T14:27:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:27:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:22 smithi107 ceph-mon[32659]: pgmap v3564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:27:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:27:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:22 smithi132 ceph-mon[37580]: pgmap v3564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:27:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:27:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:22 smithi112 ceph-mon[37501]: pgmap v3564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:27:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:27:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:27:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:27:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:23 smithi107 ceph-mon[32659]: pgmap v3565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:27:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:23 smithi107 ceph-mon[32659]: pgmap v3566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:27:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:27:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:27:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:23 smithi132 ceph-mon[37580]: pgmap v3565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:27:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:23 smithi132 ceph-mon[37580]: pgmap v3566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:27:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:27:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:27:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:23 smithi112 ceph-mon[37501]: pgmap v3565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:27:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:23 smithi112 ceph-mon[37501]: pgmap v3566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:27:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:26 smithi112 ceph-mon[37501]: pgmap v3567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5781.875686 secs 2024-07-26T14:27:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:26 smithi107 ceph-mon[32659]: pgmap v3567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5781.875686 secs 2024-07-26T14:27:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:26 smithi132 ceph-mon[37580]: pgmap v3567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5781.875686 secs 2024-07-26T14:27:26.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:26.826+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:26.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:26.826+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:26.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:26.826+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:28 smithi112 ceph-mon[37501]: pgmap v3568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:28 smithi107 ceph-mon[32659]: pgmap v3568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:28 smithi132 ceph-mon[37580]: pgmap v3568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:28.946+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:28.946+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:28.946+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:30 smithi112 ceph-mon[37501]: pgmap v3569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:27:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:30 smithi107 ceph-mon[32659]: pgmap v3569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:27:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:30 smithi132 ceph-mon[37580]: pgmap v3569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:27:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5786.875833 secs 2024-07-26T14:27:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5786.875833 secs 2024-07-26T14:27:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5786.875833 secs 2024-07-26T14:27:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:32 smithi112 ceph-mon[37501]: pgmap v3570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:32 smithi107 ceph-mon[32659]: pgmap v3570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:32 smithi132 ceph-mon[37580]: pgmap v3570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:27:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:34 smithi112 ceph-mon[37501]: pgmap v3571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:27:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:34 smithi107 ceph-mon[32659]: pgmap v3571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:27:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:34 smithi132 ceph-mon[37580]: pgmap v3571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:27:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:36 smithi107 ceph-mon[32659]: pgmap v3572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5791.875899 secs 2024-07-26T14:27:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:36 smithi112 ceph-mon[37501]: pgmap v3572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5791.875899 secs 2024-07-26T14:27:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:36 smithi132 ceph-mon[37580]: pgmap v3572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5791.875899 secs 2024-07-26T14:27:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:38 smithi112 ceph-mon[37501]: pgmap v3573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:38 smithi107 ceph-mon[32659]: pgmap v3573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:38 smithi132 ceph-mon[37580]: pgmap v3573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:40 smithi112 ceph-mon[37501]: pgmap v3574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:40 smithi107 ceph-mon[32659]: pgmap v3574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:40 smithi132 ceph-mon[37580]: pgmap v3574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5796.876036 secs 2024-07-26T14:27:41.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5796.876036 secs 2024-07-26T14:27:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5796.876036 secs 2024-07-26T14:27:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:41.841+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:41.841+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:41.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:41.841+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:42 smithi112 ceph-mon[37501]: pgmap v3575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:42 smithi107 ceph-mon[32659]: pgmap v3575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:42 smithi132 ceph-mon[37580]: pgmap v3575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:43 smithi112 ceph-mon[37501]: pgmap v3576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:43 smithi107 ceph-mon[32659]: pgmap v3576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:43 smithi132 ceph-mon[37580]: pgmap v3576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:43.962 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:43.961+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:43.961+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:43.963 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:43.961+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:46 smithi112 ceph-mon[37501]: pgmap v3577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5801.876163 secs 2024-07-26T14:27:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:46 smithi107 ceph-mon[32659]: pgmap v3577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5801.876163 secs 2024-07-26T14:27:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:46 smithi132 ceph-mon[37580]: pgmap v3577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5801.876163 secs 2024-07-26T14:27:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:48 smithi112 ceph-mon[37501]: pgmap v3578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:48 smithi107 ceph-mon[32659]: pgmap v3578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:48 smithi132 ceph-mon[37580]: pgmap v3578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:50 smithi112 ceph-mon[37501]: pgmap v3579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:50 smithi107 ceph-mon[32659]: pgmap v3579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:50 smithi132 ceph-mon[37580]: pgmap v3579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5806.876290 secs 2024-07-26T14:27:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5806.876290 secs 2024-07-26T14:27:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5806.876290 secs 2024-07-26T14:27:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:52 smithi112 ceph-mon[37501]: pgmap v3580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:52 smithi107 ceph-mon[32659]: pgmap v3580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:52 smithi132 ceph-mon[37580]: pgmap v3580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:54 smithi112 ceph-mon[37501]: pgmap v3581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:54 smithi107 ceph-mon[32659]: pgmap v3581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:54 smithi132 ceph-mon[37580]: pgmap v3581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:56 smithi112 ceph-mon[37501]: pgmap v3582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5811.876426 secs 2024-07-26T14:27:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:56 smithi107 ceph-mon[32659]: pgmap v3582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5811.876426 secs 2024-07-26T14:27:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:56 smithi132 ceph-mon[37580]: pgmap v3582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5811.876426 secs 2024-07-26T14:27:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:56.857+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:56.857+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:56.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:27:56.857+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:58 smithi112 ceph-mon[37501]: pgmap v3583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:58 smithi107 ceph-mon[32659]: pgmap v3583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:58 smithi132 ceph-mon[37580]: pgmap v3583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:27:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:58.970+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:58.970+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:27:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:27:58.970+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:27:59 smithi112 ceph-mon[37501]: pgmap v3584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:27:59 smithi107 ceph-mon[32659]: pgmap v3584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:27:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:27:59 smithi132 ceph-mon[37580]: pgmap v3584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5816.876550 secs 2024-07-26T14:28:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5816.876550 secs 2024-07-26T14:28:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5816.876550 secs 2024-07-26T14:28:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:02 smithi112 ceph-mon[37501]: pgmap v3585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:02 smithi107 ceph-mon[32659]: pgmap v3585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:02 smithi132 ceph-mon[37580]: pgmap v3585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:04 smithi112 ceph-mon[37501]: pgmap v3586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:04 smithi107 ceph-mon[32659]: pgmap v3586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:04 smithi132 ceph-mon[37580]: pgmap v3586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:06.344 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:06 smithi107 ceph-mon[32659]: pgmap v3587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:06.344 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5821.876697 secs 2024-07-26T14:28:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:06 smithi112 ceph-mon[37501]: pgmap v3587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5821.876697 secs 2024-07-26T14:28:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:06 smithi132 ceph-mon[37580]: pgmap v3587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5821.876697 secs 2024-07-26T14:28:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:08 smithi112 ceph-mon[37501]: pgmap v3588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:08 smithi107 ceph-mon[32659]: pgmap v3588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:08 smithi132 ceph-mon[37580]: pgmap v3588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:10 smithi112 ceph-mon[37501]: pgmap v3589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:10 smithi107 ceph-mon[32659]: pgmap v3589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:10 smithi132 ceph-mon[37580]: pgmap v3589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5826.876825 secs 2024-07-26T14:28:11.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5826.876825 secs 2024-07-26T14:28:11.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5826.876825 secs 2024-07-26T14:28:11.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:11.858+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:11.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:11.858+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:11.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:11.858+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:12 smithi107 ceph-mon[32659]: pgmap v3590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:12 smithi112 ceph-mon[37501]: pgmap v3590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:12 smithi132 ceph-mon[37580]: pgmap v3590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:13.980+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:13.980+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:13.981 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:13.980+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:14 smithi112 ceph-mon[37501]: pgmap v3591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:14 smithi107 ceph-mon[32659]: pgmap v3591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:14 smithi132 ceph-mon[37580]: pgmap v3591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:16 smithi112 ceph-mon[37501]: pgmap v3592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5831.876973 secs 2024-07-26T14:28:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:16 smithi107 ceph-mon[32659]: pgmap v3592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5831.876973 secs 2024-07-26T14:28:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:16 smithi132 ceph-mon[37580]: pgmap v3592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5831.876973 secs 2024-07-26T14:28:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:18 smithi112 ceph-mon[37501]: pgmap v3593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:18 smithi107 ceph-mon[32659]: pgmap v3593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:18 smithi132 ceph-mon[37580]: pgmap v3593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:20 smithi112 ceph-mon[37501]: pgmap v3594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:20 smithi107 ceph-mon[32659]: pgmap v3594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:20 smithi132 ceph-mon[37580]: pgmap v3594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5836.877061 secs 2024-07-26T14:28:21.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5836.877061 secs 2024-07-26T14:28:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5836.877061 secs 2024-07-26T14:28:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:22 smithi112 ceph-mon[37501]: pgmap v3595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:22.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:22 smithi107 ceph-mon[32659]: pgmap v3595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:22 smithi132 ceph-mon[37580]: pgmap v3595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:24 smithi112 ceph-mon[37501]: pgmap v3596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:28:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.483 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:24 smithi132 ceph-mon[37580]: pgmap v3596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:24.483 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:28:24.483 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.483 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.483 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.483 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:24 smithi107 ceph-mon[32659]: pgmap v3596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:24.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:28:24.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:24.532 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:26 smithi112 ceph-mon[37501]: pgmap v3597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5841.877106 secs 2024-07-26T14:28:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:26 smithi107 ceph-mon[32659]: pgmap v3597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5841.877106 secs 2024-07-26T14:28:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:26 smithi132 ceph-mon[37580]: pgmap v3597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5841.877106 secs 2024-07-26T14:28:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:26.873+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:26.873+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:26.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:26.873+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:27 smithi112 ceph-mon[37501]: pgmap v3598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:27 smithi107 ceph-mon[32659]: pgmap v3598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:27 smithi132 ceph-mon[37580]: pgmap v3598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:28.996 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:28.996+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:28.996 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:28.996+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:28.996 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:28.996+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:28:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:28:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:28 smithi107 ceph-mon[32659]: pgmap v3599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:28:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:28 smithi107 ceph-mon[32659]: pgmap v3600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:28:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:28:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:28:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:28 smithi132 ceph-mon[37580]: pgmap v3599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:28:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:28 smithi132 ceph-mon[37580]: pgmap v3600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:28:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:28:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:28:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:28 smithi112 ceph-mon[37501]: pgmap v3599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:28:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:28 smithi112 ceph-mon[37501]: pgmap v3600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:28:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:28:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:30 smithi112 ceph-mon[37501]: pgmap v3601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:28:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:30 smithi107 ceph-mon[32659]: pgmap v3601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:28:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:30 smithi132 ceph-mon[37580]: pgmap v3601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:28:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5846.877229 secs 2024-07-26T14:28:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5846.877229 secs 2024-07-26T14:28:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5846.877229 secs 2024-07-26T14:28:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:32 smithi112 ceph-mon[37501]: pgmap v3602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T14:28:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:32 smithi107 ceph-mon[32659]: pgmap v3602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T14:28:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:32 smithi132 ceph-mon[37580]: pgmap v3602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T14:28:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:34 smithi112 ceph-mon[37501]: pgmap v3603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:34 smithi107 ceph-mon[32659]: pgmap v3603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:34 smithi132 ceph-mon[37580]: pgmap v3603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5851.877353 secs 2024-07-26T14:28:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:36 smithi112 ceph-mon[37501]: pgmap v3604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:28:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5851.877353 secs 2024-07-26T14:28:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:36 smithi107 ceph-mon[32659]: pgmap v3604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:28:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5851.877353 secs 2024-07-26T14:28:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:36 smithi132 ceph-mon[37580]: pgmap v3604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:28:39.198 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:38 smithi112 ceph-mon[37501]: pgmap v3605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:28:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:38 smithi107 ceph-mon[32659]: pgmap v3605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:28:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:38 smithi132 ceph-mon[37580]: pgmap v3605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:28:41.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:40 smithi132 ceph-mon[37580]: pgmap v3606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:40 smithi112 ceph-mon[37501]: pgmap v3606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:41.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:40 smithi107 ceph-mon[32659]: pgmap v3606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:41.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:41.880+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:41.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:41.880+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:41.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:41.880+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:42.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5856.877471 secs 2024-07-26T14:28:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5856.877471 secs 2024-07-26T14:28:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5856.877471 secs 2024-07-26T14:28:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:42 smithi112 ceph-mon[37501]: pgmap v3607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:42 smithi107 ceph-mon[32659]: pgmap v3607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:42 smithi132 ceph-mon[37580]: pgmap v3607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:44.011+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:44.011+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:44.011+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:44 smithi112 ceph-mon[37501]: pgmap v3608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:44 smithi107 ceph-mon[32659]: pgmap v3608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:44 smithi132 ceph-mon[37580]: pgmap v3608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5861.877596 secs 2024-07-26T14:28:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:46 smithi112 ceph-mon[37501]: pgmap v3609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5861.877596 secs 2024-07-26T14:28:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:46 smithi107 ceph-mon[32659]: pgmap v3609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5861.877596 secs 2024-07-26T14:28:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:46 smithi132 ceph-mon[37580]: pgmap v3609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:48 smithi112 ceph-mon[37501]: pgmap v3610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:48 smithi107 ceph-mon[32659]: pgmap v3610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:48 smithi132 ceph-mon[37580]: pgmap v3610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:50 smithi112 ceph-mon[37501]: pgmap v3611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:50 smithi107 ceph-mon[32659]: pgmap v3611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:50 smithi132 ceph-mon[37580]: pgmap v3611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5866.877660 secs 2024-07-26T14:28:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5866.877660 secs 2024-07-26T14:28:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5866.877660 secs 2024-07-26T14:28:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:52 smithi112 ceph-mon[37501]: pgmap v3612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:52 smithi107 ceph-mon[32659]: pgmap v3612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:52 smithi132 ceph-mon[37580]: pgmap v3612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:28:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:54 smithi112 ceph-mon[37501]: pgmap v3613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:54 smithi107 ceph-mon[32659]: pgmap v3613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:54 smithi132 ceph-mon[37580]: pgmap v3613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:56.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:56.895+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:56.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:56.895+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:56.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:28:56.895+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5871.877740 secs 2024-07-26T14:28:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:56 smithi112 ceph-mon[37501]: pgmap v3614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5871.877740 secs 2024-07-26T14:28:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:56 smithi107 ceph-mon[32659]: pgmap v3614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5871.877740 secs 2024-07-26T14:28:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:56 smithi132 ceph-mon[37580]: pgmap v3614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:59.018+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:59.018+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:28:59.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:28:59.018+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:28:58 smithi112 ceph-mon[37501]: pgmap v3615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:28:58 smithi107 ceph-mon[32659]: pgmap v3615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:28:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:28:58 smithi132 ceph-mon[37580]: pgmap v3615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:00 smithi112 ceph-mon[37501]: pgmap v3616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:00 smithi107 ceph-mon[32659]: pgmap v3616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:00 smithi132 ceph-mon[37580]: pgmap v3616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5876.877881 secs 2024-07-26T14:29:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5876.877881 secs 2024-07-26T14:29:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5876.877881 secs 2024-07-26T14:29:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:02 smithi112 ceph-mon[37501]: pgmap v3617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:02 smithi107 ceph-mon[32659]: pgmap v3617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:02 smithi132 ceph-mon[37580]: pgmap v3617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:04 smithi112 ceph-mon[37501]: pgmap v3618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:04 smithi107 ceph-mon[32659]: pgmap v3618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:04 smithi132 ceph-mon[37580]: pgmap v3618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5881.878010 secs 2024-07-26T14:29:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:06 smithi112 ceph-mon[37501]: pgmap v3619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5881.878010 secs 2024-07-26T14:29:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:06 smithi107 ceph-mon[32659]: pgmap v3619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5881.878010 secs 2024-07-26T14:29:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:06 smithi132 ceph-mon[37580]: pgmap v3619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:08 smithi107 ceph-mon[32659]: pgmap v3620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:08 smithi132 ceph-mon[37580]: pgmap v3620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:08 smithi112 ceph-mon[37501]: pgmap v3620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:10 smithi107 ceph-mon[32659]: pgmap v3621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:10 smithi132 ceph-mon[37580]: pgmap v3621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:10 smithi112 ceph-mon[37501]: pgmap v3621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:11.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:11.911+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:11.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:11.911+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:11.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:11.911+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5886.878149 secs 2024-07-26T14:29:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5886.878149 secs 2024-07-26T14:29:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5886.878149 secs 2024-07-26T14:29:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:12 smithi107 ceph-mon[32659]: pgmap v3622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:12 smithi132 ceph-mon[37580]: pgmap v3622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:12 smithi112 ceph-mon[37501]: pgmap v3622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:14.022+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:14.022+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:14.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:14.022+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:14 smithi107 ceph-mon[32659]: pgmap v3623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:14 smithi132 ceph-mon[37580]: pgmap v3623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:14 smithi112 ceph-mon[37501]: pgmap v3623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5891.878276 secs 2024-07-26T14:29:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:16 smithi107 ceph-mon[32659]: pgmap v3624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5891.878276 secs 2024-07-26T14:29:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:16 smithi132 ceph-mon[37580]: pgmap v3624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5891.878276 secs 2024-07-26T14:29:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:16 smithi112 ceph-mon[37501]: pgmap v3624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:18 smithi107 ceph-mon[32659]: pgmap v3625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:18 smithi132 ceph-mon[37580]: pgmap v3625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:18 smithi112 ceph-mon[37501]: pgmap v3625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:20 smithi107 ceph-mon[32659]: pgmap v3626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:20 smithi132 ceph-mon[37580]: pgmap v3626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:20 smithi112 ceph-mon[37501]: pgmap v3626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5896.878365 secs 2024-07-26T14:29:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5896.878365 secs 2024-07-26T14:29:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5896.878365 secs 2024-07-26T14:29:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:23 smithi107 ceph-mon[32659]: pgmap v3627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:23 smithi132 ceph-mon[37580]: pgmap v3627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:23 smithi112 ceph-mon[37501]: pgmap v3627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:25 smithi107 ceph-mon[32659]: pgmap v3628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:25 smithi132 ceph-mon[37580]: pgmap v3628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:25 smithi112 ceph-mon[37501]: pgmap v3628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5901.878486 secs 2024-07-26T14:29:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5901.878486 secs 2024-07-26T14:29:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5901.878486 secs 2024-07-26T14:29:26.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:26.927+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:26.927+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:26.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:26.927+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:27 smithi107 ceph-mon[32659]: pgmap v3629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:27 smithi112 ceph-mon[37501]: pgmap v3629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:27 smithi132 ceph-mon[37580]: pgmap v3629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:29:28.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:29:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:29:29.037 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:29.037+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:29.037+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:29.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:29.037+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:29 smithi112 ceph-mon[37501]: pgmap v3630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:29:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:29:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:29:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:29 smithi107 ceph-mon[32659]: pgmap v3630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:29:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:29:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:29:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:29 smithi132 ceph-mon[37580]: pgmap v3630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:29:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:29:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:29:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:30 smithi112 ceph-mon[37501]: pgmap v3631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:29:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:30 smithi112 ceph-mon[37501]: pgmap v3632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:29:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:30 smithi107 ceph-mon[32659]: pgmap v3631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:29:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:30 smithi107 ceph-mon[32659]: pgmap v3632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:29:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:30 smithi132 ceph-mon[37580]: pgmap v3631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:29:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:30 smithi132 ceph-mon[37580]: pgmap v3632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:29:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5906.878616 secs 2024-07-26T14:29:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5906.878616 secs 2024-07-26T14:29:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5906.878616 secs 2024-07-26T14:29:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:32 smithi112 ceph-mon[37501]: pgmap v3633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:29:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:32 smithi107 ceph-mon[32659]: pgmap v3633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:29:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:32 smithi132 ceph-mon[37580]: pgmap v3633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:29:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:34 smithi112 ceph-mon[37501]: pgmap v3634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:29:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:34 smithi107 ceph-mon[32659]: pgmap v3634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:29:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:34 smithi132 ceph-mon[37580]: pgmap v3634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:29:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:36 smithi112 ceph-mon[37501]: pgmap v3635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:29:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5911.878741 secs 2024-07-26T14:29:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:36 smithi107 ceph-mon[32659]: pgmap v3635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:29:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5911.878741 secs 2024-07-26T14:29:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:36 smithi132 ceph-mon[37580]: pgmap v3635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:29:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5911.878741 secs 2024-07-26T14:29:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:38 smithi112 ceph-mon[37501]: pgmap v3636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:38 smithi107 ceph-mon[32659]: pgmap v3636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:38 smithi132 ceph-mon[37580]: pgmap v3636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:40 smithi107 ceph-mon[32659]: pgmap v3637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:29:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:40 smithi112 ceph-mon[37501]: pgmap v3637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:29:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:40 smithi132 ceph-mon[37580]: pgmap v3637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:29:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5916.878866 secs 2024-07-26T14:29:41.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5916.878866 secs 2024-07-26T14:29:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5916.878866 secs 2024-07-26T14:29:41.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:41.927+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:41.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:41.927+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:41.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:41.927+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:42 smithi107 ceph-mon[32659]: pgmap v3638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:42 smithi112 ceph-mon[37501]: pgmap v3638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:42 smithi132 ceph-mon[37580]: pgmap v3638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:44.045+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:44.045+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:44.045+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:44 smithi112 ceph-mon[37501]: pgmap v3639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:44 smithi107 ceph-mon[32659]: pgmap v3639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:44 smithi132 ceph-mon[37580]: pgmap v3639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:46 smithi112 ceph-mon[37501]: pgmap v3640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5921.878964 secs 2024-07-26T14:29:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:46 smithi107 ceph-mon[32659]: pgmap v3640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5921.878964 secs 2024-07-26T14:29:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:46 smithi132 ceph-mon[37580]: pgmap v3640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5921.878964 secs 2024-07-26T14:29:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:48 smithi112 ceph-mon[37501]: pgmap v3641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:48 smithi107 ceph-mon[32659]: pgmap v3641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:48 smithi132 ceph-mon[37580]: pgmap v3641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:49 smithi112 ceph-mon[37501]: pgmap v3642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:49 smithi107 ceph-mon[32659]: pgmap v3642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:49 smithi132 ceph-mon[37580]: pgmap v3642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5926.879107 secs 2024-07-26T14:29:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:52 smithi107 ceph-mon[32659]: pgmap v3643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5926.879107 secs 2024-07-26T14:29:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:52 smithi132 ceph-mon[37580]: pgmap v3643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5926.879107 secs 2024-07-26T14:29:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:52 smithi112 ceph-mon[37501]: pgmap v3643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:54 smithi107 ceph-mon[32659]: pgmap v3644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:54.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:54 smithi132 ceph-mon[37580]: pgmap v3644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:54 smithi112 ceph-mon[37501]: pgmap v3644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:29:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:56 smithi107 ceph-mon[32659]: pgmap v3645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:56 smithi132 ceph-mon[37580]: pgmap v3645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:56 smithi112 ceph-mon[37501]: pgmap v3645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:56.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:56.937+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:56.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:56.937+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:56.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:29:56.937+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:57.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5931.879230 secs 2024-07-26T14:29:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5931.879230 secs 2024-07-26T14:29:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5931.879230 secs 2024-07-26T14:29:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:29:58 smithi107 ceph-mon[32659]: pgmap v3646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:29:58 smithi132 ceph-mon[37580]: pgmap v3646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:29:58 smithi112 ceph-mon[37501]: pgmap v3646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:29:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:59.061+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:59.061+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:29:59.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:29:59.061+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:00 smithi107 ceph-mon[32659]: pgmap v3647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:30:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:00 smithi132 ceph-mon[37580]: pgmap v3647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:30:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:00 smithi112 ceph-mon[37501]: pgmap v3647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:30:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5936.879317 secs 2024-07-26T14:30:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5936.879317 secs 2024-07-26T14:30:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5936.879317 secs 2024-07-26T14:30:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:02 smithi132 ceph-mon[37580]: pgmap v3648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:30:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:02 smithi112 ceph-mon[37501]: pgmap v3648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:30:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:02 smithi107 ceph-mon[32659]: pgmap v3648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:30:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:04 smithi132 ceph-mon[37580]: pgmap v3649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:04 smithi112 ceph-mon[37501]: pgmap v3649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:04 smithi107 ceph-mon[32659]: pgmap v3649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:06 smithi112 ceph-mon[37501]: pgmap v3650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5941.879386 secs 2024-07-26T14:30:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:06 smithi107 ceph-mon[32659]: pgmap v3650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5941.879386 secs 2024-07-26T14:30:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:06 smithi132 ceph-mon[37580]: pgmap v3650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5941.879386 secs 2024-07-26T14:30:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:08 smithi112 ceph-mon[37501]: pgmap v3651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:08 smithi107 ceph-mon[32659]: pgmap v3651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:08 smithi132 ceph-mon[37580]: pgmap v3651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:10 smithi132 ceph-mon[37580]: pgmap v3652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:10 smithi112 ceph-mon[37501]: pgmap v3652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:10 smithi107 ceph-mon[32659]: pgmap v3652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:11.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5946.879513 secs 2024-07-26T14:30:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5946.879513 secs 2024-07-26T14:30:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5946.879513 secs 2024-07-26T14:30:11.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:11.953+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:11.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:11.953+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:11.955 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:11.953+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:12 smithi107 ceph-mon[32659]: pgmap v3653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:12 smithi112 ceph-mon[37501]: pgmap v3653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:12 smithi132 ceph-mon[37580]: pgmap v3653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:12.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:12.642+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:30:12.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:12.643+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:30:12.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:12.665+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:30:12.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:12.666+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:30:13.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:13.643+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:30:13.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:13.644+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:30:13.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:13.665+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:30:13.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:13.666+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:30:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.076+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.076+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.076+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:14 smithi112 ceph-mon[37501]: pgmap v3654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:14 smithi107 ceph-mon[32659]: pgmap v3654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:14 smithi132 ceph-mon[37580]: pgmap v3654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.807+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:30:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.807+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:30:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.808+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7efc48003690 con 0x7efc78103340 2024-07-26T14:30:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.808+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7efc44006660 con 0x55aecd6b64d0 2024-07-26T14:30:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.808+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:30:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.808+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T14:30:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.808+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:30:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:14.809+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T14:30:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:16 smithi112 ceph-mon[37501]: pgmap v3655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5951.879642 secs 2024-07-26T14:30:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:16 smithi107 ceph-mon[32659]: pgmap v3655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5951.879642 secs 2024-07-26T14:30:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:16 smithi132 ceph-mon[37580]: pgmap v3655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5951.879642 secs 2024-07-26T14:30:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:18 smithi112 ceph-mon[37501]: pgmap v3656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:18 smithi107 ceph-mon[32659]: pgmap v3656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:18 smithi132 ceph-mon[37580]: pgmap v3656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:20 smithi112 ceph-mon[37501]: pgmap v3657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:20 smithi107 ceph-mon[32659]: pgmap v3657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:20 smithi132 ceph-mon[37580]: pgmap v3657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5956.879764 secs 2024-07-26T14:30:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5956.879764 secs 2024-07-26T14:30:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5956.879764 secs 2024-07-26T14:30:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:22 smithi112 ceph-mon[37501]: pgmap v3658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:22 smithi107 ceph-mon[32659]: pgmap v3658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:22 smithi132 ceph-mon[37580]: pgmap v3658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:24 smithi112 ceph-mon[37501]: pgmap v3659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:24 smithi107 ceph-mon[32659]: pgmap v3659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:24 smithi132 ceph-mon[37580]: pgmap v3659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:26 smithi112 ceph-mon[37501]: pgmap v3660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5961.879899 secs 2024-07-26T14:30:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:26 smithi107 ceph-mon[32659]: pgmap v3660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5961.879899 secs 2024-07-26T14:30:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:26 smithi132 ceph-mon[37580]: pgmap v3660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5961.879899 secs 2024-07-26T14:30:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:26.960+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:26.960+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:26.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:26.960+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:28 smithi112 ceph-mon[37501]: pgmap v3661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:28 smithi107 ceph-mon[32659]: pgmap v3661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:28 smithi132 ceph-mon[37580]: pgmap v3661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:29.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:29.092+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:29.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:29.092+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:29.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:29.092+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:30 smithi112 ceph-mon[37501]: pgmap v3662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:30:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:30 smithi107 ceph-mon[32659]: pgmap v3662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:30:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:30 smithi132 ceph-mon[37580]: pgmap v3662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:30:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:30:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:30:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:31 smithi112 ceph-mon[37501]: pgmap v3663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:30:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:30:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5966.879994 secs 2024-07-26T14:30:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:30:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:30:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:31 smithi107 ceph-mon[32659]: pgmap v3663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:30:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:30:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5966.879994 secs 2024-07-26T14:30:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:30:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:30:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:31 smithi132 ceph-mon[37580]: pgmap v3663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:30:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:30:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5966.879994 secs 2024-07-26T14:30:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:32 smithi107 ceph-mon[32659]: pgmap v3664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:30:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:32 smithi132 ceph-mon[37580]: pgmap v3664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:30:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:32 smithi112 ceph-mon[37501]: pgmap v3664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:30:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:34 smithi107 ceph-mon[32659]: pgmap v3665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:30:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:34 smithi132 ceph-mon[37580]: pgmap v3665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:30:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:34 smithi112 ceph-mon[37501]: pgmap v3665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:30:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5971.880094 secs 2024-07-26T14:30:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5971.880094 secs 2024-07-26T14:30:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5971.880094 secs 2024-07-26T14:30:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:37 smithi112 ceph-mon[37501]: pgmap v3666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:30:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:37 smithi107 ceph-mon[32659]: pgmap v3666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:30:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:37 smithi132 ceph-mon[37580]: pgmap v3666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:30:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:38 smithi107 ceph-mon[32659]: pgmap v3667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:38 smithi132 ceph-mon[37580]: pgmap v3667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:38 smithi112 ceph-mon[37501]: pgmap v3667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:30:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:40 smithi107 ceph-mon[32659]: pgmap v3668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:30:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:40 smithi132 ceph-mon[37580]: pgmap v3668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:30:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:40 smithi112 ceph-mon[37501]: pgmap v3668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:30:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5976.880162 secs 2024-07-26T14:30:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5976.880162 secs 2024-07-26T14:30:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5976.880162 secs 2024-07-26T14:30:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:41.963+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:41.963+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:41.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:41.963+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:42 smithi107 ceph-mon[32659]: pgmap v3669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:42 smithi132 ceph-mon[37580]: pgmap v3669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:42 smithi112 ceph-mon[37501]: pgmap v3669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:44.107+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:44.108 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:44.107+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:44.109 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:44.107+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:44 smithi107 ceph-mon[32659]: pgmap v3670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:44 smithi132 ceph-mon[37580]: pgmap v3670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:44 smithi112 ceph-mon[37501]: pgmap v3670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5981.880282 secs 2024-07-26T14:30:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5981.880282 secs 2024-07-26T14:30:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5981.880282 secs 2024-07-26T14:30:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:47 smithi112 ceph-mon[37501]: pgmap v3671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:47 smithi107 ceph-mon[32659]: pgmap v3671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:47 smithi132 ceph-mon[37580]: pgmap v3671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:48 smithi107 ceph-mon[32659]: pgmap v3672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:48 smithi132 ceph-mon[37580]: pgmap v3672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:48 smithi112 ceph-mon[37501]: pgmap v3672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:50 smithi107 ceph-mon[32659]: pgmap v3673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:50 smithi132 ceph-mon[37580]: pgmap v3673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:50 smithi112 ceph-mon[37501]: pgmap v3673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5986.880369 secs 2024-07-26T14:30:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5986.880369 secs 2024-07-26T14:30:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5986.880369 secs 2024-07-26T14:30:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:52 smithi107 ceph-mon[32659]: pgmap v3674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:52 smithi132 ceph-mon[37580]: pgmap v3674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:52 smithi112 ceph-mon[37501]: pgmap v3674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:30:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:54 smithi107 ceph-mon[32659]: pgmap v3675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:54 smithi132 ceph-mon[37580]: pgmap v3675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:54 smithi112 ceph-mon[37501]: pgmap v3675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5991.880528 secs 2024-07-26T14:30:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5991.880528 secs 2024-07-26T14:30:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5991.880528 secs 2024-07-26T14:30:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:56.967+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:56.967+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:30:56.967+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:57 smithi107 ceph-mon[32659]: pgmap v3676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:57 smithi112 ceph-mon[37501]: pgmap v3676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:57 smithi132 ceph-mon[37580]: pgmap v3676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:59.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:59.121+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:59.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:59.121+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:30:59.123 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:30:59.121+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:30:59 smithi107 ceph-mon[32659]: pgmap v3677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:30:59 smithi132 ceph-mon[37580]: pgmap v3677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:30:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:30:59 smithi112 ceph-mon[37501]: pgmap v3677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:01 smithi107 ceph-mon[32659]: pgmap v3678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 5996.880630 secs 2024-07-26T14:31:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:01 smithi132 ceph-mon[37580]: pgmap v3678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 5996.880630 secs 2024-07-26T14:31:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:01 smithi112 ceph-mon[37501]: pgmap v3678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 5996.880630 secs 2024-07-26T14:31:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:03 smithi107 ceph-mon[32659]: pgmap v3679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:03 smithi132 ceph-mon[37580]: pgmap v3679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:03 smithi112 ceph-mon[37501]: pgmap v3679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:05 smithi107 ceph-mon[32659]: pgmap v3680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:05 smithi132 ceph-mon[37580]: pgmap v3680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:05 smithi112 ceph-mon[37501]: pgmap v3680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6001.880708 secs 2024-07-26T14:31:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6001.880708 secs 2024-07-26T14:31:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6001.880708 secs 2024-07-26T14:31:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:07 smithi107 ceph-mon[32659]: pgmap v3681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:07 smithi132 ceph-mon[37580]: pgmap v3681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:07 smithi112 ceph-mon[37501]: pgmap v3681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:09 smithi107 ceph-mon[32659]: pgmap v3682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:09 smithi132 ceph-mon[37580]: pgmap v3682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:09 smithi112 ceph-mon[37501]: pgmap v3682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:11 smithi112 ceph-mon[37501]: pgmap v3683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6006.880787 secs 2024-07-26T14:31:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:11 smithi107 ceph-mon[32659]: pgmap v3683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6006.880787 secs 2024-07-26T14:31:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:11 smithi132 ceph-mon[37580]: pgmap v3683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6006.880787 secs 2024-07-26T14:31:11.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:11.972+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:11.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:11.972+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:11.973 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:11.972+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:13 smithi112 ceph-mon[37501]: pgmap v3684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:13 smithi107 ceph-mon[32659]: pgmap v3684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:13 smithi132 ceph-mon[37580]: pgmap v3684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:14.133+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:14.133+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:14.133+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:15 smithi112 ceph-mon[37501]: pgmap v3685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:15 smithi107 ceph-mon[32659]: pgmap v3685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:15 smithi132 ceph-mon[37580]: pgmap v3685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6011.880850 secs 2024-07-26T14:31:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6011.880850 secs 2024-07-26T14:31:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6011.880850 secs 2024-07-26T14:31:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:17 smithi112 ceph-mon[37501]: pgmap v3686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:17 smithi107 ceph-mon[32659]: pgmap v3686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:17 smithi132 ceph-mon[37580]: pgmap v3686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:19 smithi107 ceph-mon[32659]: pgmap v3687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:19 smithi132 ceph-mon[37580]: pgmap v3687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:19 smithi112 ceph-mon[37501]: pgmap v3687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:21 smithi107 ceph-mon[32659]: pgmap v3688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6016.880969 secs 2024-07-26T14:31:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:21 smithi132 ceph-mon[37580]: pgmap v3688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6016.880969 secs 2024-07-26T14:31:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:21 smithi112 ceph-mon[37501]: pgmap v3688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6016.880969 secs 2024-07-26T14:31:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:23 smithi107 ceph-mon[32659]: pgmap v3689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:23 smithi132 ceph-mon[37580]: pgmap v3689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:23 smithi112 ceph-mon[37501]: pgmap v3689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:25 smithi107 ceph-mon[32659]: pgmap v3690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:25 smithi132 ceph-mon[37580]: pgmap v3690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:25 smithi112 ceph-mon[37501]: pgmap v3690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6021.881113 secs 2024-07-26T14:31:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6021.881113 secs 2024-07-26T14:31:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6021.881113 secs 2024-07-26T14:31:26.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:26.974+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:26.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:26.974+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:26.975 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:26.974+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:27 smithi107 ceph-mon[32659]: pgmap v3691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:27 smithi132 ceph-mon[37580]: pgmap v3691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:27 smithi112 ceph-mon[37501]: pgmap v3691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:29.149 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:29.149+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:29.149+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:29.150 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:29.149+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:29 smithi107 ceph-mon[32659]: pgmap v3692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:29 smithi132 ceph-mon[37580]: pgmap v3692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:29 smithi112 ceph-mon[37501]: pgmap v3692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:30.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:31:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:31:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:31:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:31 smithi107 ceph-mon[32659]: pgmap v3693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6026.881198 secs 2024-07-26T14:31:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:31:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:31:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:31:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:31 smithi132 ceph-mon[37580]: pgmap v3693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6026.881198 secs 2024-07-26T14:31:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:31:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:31:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:31:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:31 smithi112 ceph-mon[37501]: pgmap v3693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6026.881198 secs 2024-07-26T14:31:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:31:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:31:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:31:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:32 smithi107 ceph-mon[32659]: pgmap v3694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:32 smithi132 ceph-mon[37580]: pgmap v3694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:32 smithi112 ceph-mon[37501]: pgmap v3694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:34 smithi107 ceph-mon[32659]: pgmap v3695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:31:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:34 smithi132 ceph-mon[37580]: pgmap v3695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:31:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:34 smithi112 ceph-mon[37501]: pgmap v3695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:31:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6031.881315 secs 2024-07-26T14:31:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:36 smithi107 ceph-mon[32659]: pgmap v3696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6031.881315 secs 2024-07-26T14:31:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:36 smithi132 ceph-mon[37580]: pgmap v3696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6031.881315 secs 2024-07-26T14:31:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:36 smithi112 ceph-mon[37501]: pgmap v3696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:38 smithi107 ceph-mon[32659]: pgmap v3697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:31:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:38 smithi132 ceph-mon[37580]: pgmap v3697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:31:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:38 smithi112 ceph-mon[37501]: pgmap v3697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:31:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:40 smithi107 ceph-mon[32659]: pgmap v3698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:31:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:40 smithi132 ceph-mon[37580]: pgmap v3698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:31:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:40 smithi112 ceph-mon[37501]: pgmap v3698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:31:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6036.881440 secs 2024-07-26T14:31:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6036.881440 secs 2024-07-26T14:31:41.992 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6036.881440 secs 2024-07-26T14:31:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:41.988+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:41.988+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:41.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:41.988+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:42 smithi132 ceph-mon[37580]: pgmap v3699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:42 smithi112 ceph-mon[37501]: pgmap v3699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:42 smithi107 ceph-mon[32659]: pgmap v3699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:44.158+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:44.158+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:44.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:44.158+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:44 smithi132 ceph-mon[37580]: pgmap v3700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:44.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:44 smithi107 ceph-mon[32659]: pgmap v3700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:44 smithi112 ceph-mon[37501]: pgmap v3700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6041.881586 secs 2024-07-26T14:31:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:46 smithi132 ceph-mon[37580]: pgmap v3701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6041.881586 secs 2024-07-26T14:31:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:46 smithi112 ceph-mon[37501]: pgmap v3701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6041.881586 secs 2024-07-26T14:31:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:46 smithi107 ceph-mon[32659]: pgmap v3701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:48 smithi112 ceph-mon[37501]: pgmap v3702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:48 smithi107 ceph-mon[32659]: pgmap v3702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:48 smithi132 ceph-mon[37580]: pgmap v3702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:50 smithi112 ceph-mon[37501]: pgmap v3703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:50 smithi107 ceph-mon[32659]: pgmap v3703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:50 smithi132 ceph-mon[37580]: pgmap v3703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6046.881712 secs 2024-07-26T14:31:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6046.881712 secs 2024-07-26T14:31:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6046.881712 secs 2024-07-26T14:31:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:52 smithi112 ceph-mon[37501]: pgmap v3704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:52 smithi107 ceph-mon[32659]: pgmap v3704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:52 smithi132 ceph-mon[37580]: pgmap v3704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:54 smithi112 ceph-mon[37501]: pgmap v3705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:54 smithi107 ceph-mon[32659]: pgmap v3705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:54 smithi132 ceph-mon[37580]: pgmap v3705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6051.881833 secs 2024-07-26T14:31:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:56 smithi112 ceph-mon[37501]: pgmap v3706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:57.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:57.000+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:57.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:57.000+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:57.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:31:57.000+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:57.003 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6051.881833 secs 2024-07-26T14:31:57.003 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:56 smithi107 ceph-mon[32659]: pgmap v3706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6051.881833 secs 2024-07-26T14:31:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:56 smithi132 ceph-mon[37580]: pgmap v3706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:31:58 smithi112 ceph-mon[37501]: pgmap v3707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:31:58 smithi107 ceph-mon[32659]: pgmap v3707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:31:58 smithi132 ceph-mon[37580]: pgmap v3707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:59.172+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:59.172+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:31:59.173 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:31:59.172+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:00 smithi112 ceph-mon[37501]: pgmap v3708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:32:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:00 smithi107 ceph-mon[32659]: pgmap v3708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:32:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:00 smithi132 ceph-mon[37580]: pgmap v3708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:32:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6056.881932 secs 2024-07-26T14:32:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6056.881932 secs 2024-07-26T14:32:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6056.881932 secs 2024-07-26T14:32:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:02 smithi112 ceph-mon[37501]: pgmap v3709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:02 smithi107 ceph-mon[32659]: pgmap v3709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:02 smithi132 ceph-mon[37580]: pgmap v3709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:04 smithi112 ceph-mon[37501]: pgmap v3710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:04 smithi107 ceph-mon[32659]: pgmap v3710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:04 smithi132 ceph-mon[37580]: pgmap v3710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6061.882053 secs 2024-07-26T14:32:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:06 smithi112 ceph-mon[37501]: pgmap v3711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6061.882053 secs 2024-07-26T14:32:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:06 smithi107 ceph-mon[32659]: pgmap v3711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6061.882053 secs 2024-07-26T14:32:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:06 smithi132 ceph-mon[37580]: pgmap v3711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:08 smithi112 ceph-mon[37501]: pgmap v3712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:08 smithi107 ceph-mon[32659]: pgmap v3712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:08 smithi132 ceph-mon[37580]: pgmap v3712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:10 smithi112 ceph-mon[37501]: pgmap v3713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:32:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:10 smithi107 ceph-mon[32659]: pgmap v3713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:32:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:10 smithi132 ceph-mon[37580]: pgmap v3713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:32:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6066.882119 secs 2024-07-26T14:32:11.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6066.882119 secs 2024-07-26T14:32:12.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:12.002+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:12.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:12.002+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:12.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:12.002+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:12.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6066.882119 secs 2024-07-26T14:32:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:12 smithi112 ceph-mon[37501]: pgmap v3714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:32:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:12 smithi107 ceph-mon[32659]: pgmap v3714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:32:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:12 smithi132 ceph-mon[37580]: pgmap v3714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:32:14.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:14.174+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:14.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:14.174+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:14.175 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:14.174+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:14 smithi112 ceph-mon[37501]: pgmap v3715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:14 smithi107 ceph-mon[32659]: pgmap v3715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:14 smithi132 ceph-mon[37580]: pgmap v3715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6071.882247 secs 2024-07-26T14:32:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:16 smithi112 ceph-mon[37501]: pgmap v3716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6071.882247 secs 2024-07-26T14:32:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:16 smithi107 ceph-mon[32659]: pgmap v3716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6071.882247 secs 2024-07-26T14:32:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:16 smithi132 ceph-mon[37580]: pgmap v3716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:18 smithi112 ceph-mon[37501]: pgmap v3717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:18 smithi107 ceph-mon[32659]: pgmap v3717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:18 smithi132 ceph-mon[37580]: pgmap v3717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:20 smithi112 ceph-mon[37501]: pgmap v3718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:32:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:20 smithi107 ceph-mon[32659]: pgmap v3718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:32:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:20 smithi132 ceph-mon[37580]: pgmap v3718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:32:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6076.882365 secs 2024-07-26T14:32:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6076.882365 secs 2024-07-26T14:32:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6076.882365 secs 2024-07-26T14:32:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:22 smithi112 ceph-mon[37501]: pgmap v3719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:32:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:22 smithi107 ceph-mon[32659]: pgmap v3719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:32:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:22 smithi132 ceph-mon[37580]: pgmap v3719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:32:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:24 smithi112 ceph-mon[37501]: pgmap v3720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:24 smithi107 ceph-mon[32659]: pgmap v3720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:24 smithi132 ceph-mon[37580]: pgmap v3720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:27.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:27.018+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:27.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:27.018+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:27.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:27.018+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:27.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6081.882490 secs 2024-07-26T14:32:27.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:26 smithi107 ceph-mon[32659]: pgmap v3721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6081.882490 secs 2024-07-26T14:32:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:26 smithi132 ceph-mon[37580]: pgmap v3721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6081.882490 secs 2024-07-26T14:32:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:26 smithi112 ceph-mon[37501]: pgmap v3721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:28 smithi107 ceph-mon[32659]: pgmap v3722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:28 smithi132 ceph-mon[37580]: pgmap v3722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:29.186+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:29.186+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:29.187 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:29.186+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:28 smithi112 ceph-mon[37501]: pgmap v3722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:30 smithi107 ceph-mon[32659]: pgmap v3723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:32:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:30 smithi132 ceph-mon[37580]: pgmap v3723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:32:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:30 smithi112 ceph-mon[37501]: pgmap v3723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:32:32.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6086.882609 secs 2024-07-26T14:32:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:32:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6086.882609 secs 2024-07-26T14:32:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:32:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6086.882609 secs 2024-07-26T14:32:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:32:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:32 smithi107 ceph-mon[32659]: pgmap v3724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:32:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:32:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:32:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:32:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:32 smithi132 ceph-mon[37580]: pgmap v3724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:32:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:32:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:32:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:32:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:32 smithi112 ceph-mon[37501]: pgmap v3724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:32:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:32:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:32:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:32:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:33 smithi107 ceph-mon[32659]: pgmap v3725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:32:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:33 smithi107 ceph-mon[32659]: pgmap v3726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:32:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:33 smithi132 ceph-mon[37580]: pgmap v3725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:32:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:33 smithi132 ceph-mon[37580]: pgmap v3726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:32:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:33 smithi112 ceph-mon[37501]: pgmap v3725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:32:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:33 smithi112 ceph-mon[37501]: pgmap v3726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:32:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:35 smithi107 ceph-mon[32659]: pgmap v3727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:32:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:35 smithi132 ceph-mon[37580]: pgmap v3727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:32:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:35 smithi112 ceph-mon[37501]: pgmap v3727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:32:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6091.882736 secs 2024-07-26T14:32:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6091.882736 secs 2024-07-26T14:32:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6091.882736 secs 2024-07-26T14:32:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:37 smithi107 ceph-mon[32659]: pgmap v3728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:37 smithi132 ceph-mon[37580]: pgmap v3728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:37 smithi112 ceph-mon[37501]: pgmap v3728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:39 smithi107 ceph-mon[32659]: pgmap v3729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:32:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:39 smithi132 ceph-mon[37580]: pgmap v3729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:32:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:39 smithi112 ceph-mon[37501]: pgmap v3729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:32:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:42.032+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:42.032+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:42.035 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:42.032+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:42.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6096.882857 secs 2024-07-26T14:32:42.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:41 smithi107 ceph-mon[32659]: pgmap v3730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:32:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6096.882857 secs 2024-07-26T14:32:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:41 smithi132 ceph-mon[37580]: pgmap v3730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:32:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6096.882857 secs 2024-07-26T14:32:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:41 smithi112 ceph-mon[37501]: pgmap v3730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:32:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:43 smithi107 ceph-mon[32659]: pgmap v3731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:32:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:43 smithi132 ceph-mon[37580]: pgmap v3731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:32:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:43 smithi112 ceph-mon[37501]: pgmap v3731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:32:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:44.200+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:44.200+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:44.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:44.200+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:45 smithi107 ceph-mon[32659]: pgmap v3732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:45 smithi132 ceph-mon[37580]: pgmap v3732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:46.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:45 smithi112 ceph-mon[37501]: pgmap v3732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6101.882994 secs 2024-07-26T14:32:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6101.882994 secs 2024-07-26T14:32:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6101.882994 secs 2024-07-26T14:32:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:47 smithi107 ceph-mon[32659]: pgmap v3733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:47 smithi132 ceph-mon[37580]: pgmap v3733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:47 smithi112 ceph-mon[37501]: pgmap v3733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:49 smithi132 ceph-mon[37580]: pgmap v3734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:49 smithi112 ceph-mon[37501]: pgmap v3734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:50.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:49 smithi107 ceph-mon[32659]: pgmap v3734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6106.883136 secs 2024-07-26T14:32:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:51 smithi132 ceph-mon[37580]: pgmap v3735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6106.883136 secs 2024-07-26T14:32:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:51 smithi112 ceph-mon[37501]: pgmap v3735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6106.883136 secs 2024-07-26T14:32:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:51 smithi107 ceph-mon[32659]: pgmap v3735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:53 smithi132 ceph-mon[37580]: pgmap v3736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:53 smithi112 ceph-mon[37501]: pgmap v3736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:53 smithi107 ceph-mon[32659]: pgmap v3736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:55 smithi112 ceph-mon[37501]: pgmap v3737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:55 smithi107 ceph-mon[32659]: pgmap v3737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:55 smithi132 ceph-mon[37580]: pgmap v3737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:57.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:57.048+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:57.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:57.048+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:57.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:32:57.048+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:57.193 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6111.883254 secs 2024-07-26T14:32:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6111.883254 secs 2024-07-26T14:32:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6111.883254 secs 2024-07-26T14:32:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:57 smithi112 ceph-mon[37501]: pgmap v3738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:32:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:57 smithi107 ceph-mon[32659]: pgmap v3738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:32:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:57 smithi132 ceph-mon[37580]: pgmap v3738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:32:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:59.216+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:32:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:59.216+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:59.218 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:32:59.216+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:32:59 smithi112 ceph-mon[37501]: pgmap v3739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:00.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:32:59 smithi107 ceph-mon[32659]: pgmap v3739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:32:59 smithi132 ceph-mon[37580]: pgmap v3739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:02.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6116.883324 secs 2024-07-26T14:33:02.170 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:01 smithi107 ceph-mon[32659]: pgmap v3740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6116.883324 secs 2024-07-26T14:33:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:01 smithi112 ceph-mon[37501]: pgmap v3740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6116.883324 secs 2024-07-26T14:33:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:01 smithi132 ceph-mon[37580]: pgmap v3740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:03 smithi112 ceph-mon[37501]: pgmap v3741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:03 smithi107 ceph-mon[32659]: pgmap v3741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:03 smithi132 ceph-mon[37580]: pgmap v3741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:05 smithi112 ceph-mon[37501]: pgmap v3742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:05 smithi107 ceph-mon[32659]: pgmap v3742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:05 smithi132 ceph-mon[37580]: pgmap v3742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6121.883409 secs 2024-07-26T14:33:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6121.883409 secs 2024-07-26T14:33:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6121.883409 secs 2024-07-26T14:33:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:07 smithi112 ceph-mon[37501]: pgmap v3743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:07 smithi107 ceph-mon[32659]: pgmap v3743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:07 smithi132 ceph-mon[37580]: pgmap v3743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:09 smithi112 ceph-mon[37501]: pgmap v3744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:09 smithi107 ceph-mon[32659]: pgmap v3744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:09 smithi132 ceph-mon[37580]: pgmap v3744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:12.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:12.065+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:12.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:12.065+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:12.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:12.065+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6126.883487 secs 2024-07-26T14:33:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:11 smithi112 ceph-mon[37501]: pgmap v3745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6126.883487 secs 2024-07-26T14:33:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:11 smithi107 ceph-mon[32659]: pgmap v3745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6126.883487 secs 2024-07-26T14:33:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:11 smithi132 ceph-mon[37580]: pgmap v3745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:13.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:13.673+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:33:13.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:13.674+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:33:13.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:13.695+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:33:13.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:13.696+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:33:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:13 smithi107 ceph-mon[32659]: pgmap v3746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:13 smithi112 ceph-mon[37501]: pgmap v3746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:14.233+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:14.233+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:14.233 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:14.233+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:13 smithi132 ceph-mon[37580]: pgmap v3746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:14.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:14.838+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:33:14.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:14.838+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T14:33:14.839 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:14.839+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:33:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:14.839+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T14:33:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:15 smithi107 ceph-mon[32659]: pgmap v3747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:15 smithi132 ceph-mon[37580]: pgmap v3747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:15 smithi112 ceph-mon[37501]: pgmap v3747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6131.883552 secs 2024-07-26T14:33:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6131.883552 secs 2024-07-26T14:33:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6131.883552 secs 2024-07-26T14:33:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:17 smithi107 ceph-mon[32659]: pgmap v3748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:33:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:17 smithi132 ceph-mon[37580]: pgmap v3748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:33:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:17 smithi112 ceph-mon[37501]: pgmap v3748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:33:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:19 smithi107 ceph-mon[32659]: pgmap v3749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:19 smithi132 ceph-mon[37580]: pgmap v3749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:19 smithi112 ceph-mon[37501]: pgmap v3749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6136.883674 secs 2024-07-26T14:33:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:21 smithi107 ceph-mon[32659]: pgmap v3750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:33:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6136.883674 secs 2024-07-26T14:33:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:21 smithi132 ceph-mon[37580]: pgmap v3750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:33:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6136.883674 secs 2024-07-26T14:33:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:21 smithi112 ceph-mon[37501]: pgmap v3750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:33:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:23 smithi107 ceph-mon[32659]: pgmap v3751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:23 smithi132 ceph-mon[37580]: pgmap v3751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:23 smithi112 ceph-mon[37501]: pgmap v3751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:25 smithi107 ceph-mon[32659]: pgmap v3752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:33:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:25 smithi132 ceph-mon[37580]: pgmap v3752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:33:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:25 smithi112 ceph-mon[37501]: pgmap v3752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:33:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6141.883738 secs 2024-07-26T14:33:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6141.883738 secs 2024-07-26T14:33:27.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:27.080+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:27.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:27.080+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:27.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:27.080+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6141.883738 secs 2024-07-26T14:33:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:27 smithi107 ceph-mon[32659]: pgmap v3753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:33:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:27 smithi132 ceph-mon[37580]: pgmap v3753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:33:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:27 smithi112 ceph-mon[37501]: pgmap v3753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:33:29.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:29.248+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:29.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:29.248+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:29.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:29.248+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:29 smithi107 ceph-mon[32659]: pgmap v3754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:29 smithi132 ceph-mon[37580]: pgmap v3754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:29 smithi112 ceph-mon[37501]: pgmap v3754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6146.883831 secs 2024-07-26T14:33:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:31 smithi107 ceph-mon[32659]: pgmap v3755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6146.883831 secs 2024-07-26T14:33:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:31 smithi132 ceph-mon[37580]: pgmap v3755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6146.883831 secs 2024-07-26T14:33:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:31 smithi112 ceph-mon[37501]: pgmap v3755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:33 smithi107 ceph-mon[32659]: pgmap v3756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:33:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:33 smithi132 ceph-mon[37580]: pgmap v3756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:33:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:33 smithi112 ceph-mon[37501]: pgmap v3756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:33:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:33:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:33:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:34 smithi107 ceph-mon[32659]: pgmap v3757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:33:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:34 smithi107 ceph-mon[32659]: pgmap v3758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:33:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:33:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:33:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:33:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:34 smithi132 ceph-mon[37580]: pgmap v3757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:33:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:34 smithi132 ceph-mon[37580]: pgmap v3758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:33:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:33:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:33:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:33:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:34 smithi112 ceph-mon[37501]: pgmap v3757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:33:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:34 smithi112 ceph-mon[37501]: pgmap v3758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:33:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:33:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6151.883969 secs 2024-07-26T14:33:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:36 smithi112 ceph-mon[37501]: pgmap v3759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:33:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6151.883969 secs 2024-07-26T14:33:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:36 smithi107 ceph-mon[32659]: pgmap v3759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:33:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6151.883969 secs 2024-07-26T14:33:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:36 smithi132 ceph-mon[37580]: pgmap v3759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:33:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:38 smithi112 ceph-mon[37501]: pgmap v3760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:33:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:38 smithi107 ceph-mon[32659]: pgmap v3760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:33:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:38 smithi132 ceph-mon[37580]: pgmap v3760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:33:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:40 smithi112 ceph-mon[37501]: pgmap v3761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:40 smithi107 ceph-mon[32659]: pgmap v3761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:40 smithi132 ceph-mon[37580]: pgmap v3761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:42.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:42.095+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:42.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:42.095+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:42.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:42.095+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6156.884033 secs 2024-07-26T14:33:42.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6156.884033 secs 2024-07-26T14:33:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6156.884033 secs 2024-07-26T14:33:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:42 smithi112 ceph-mon[37501]: pgmap v3762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:42 smithi107 ceph-mon[32659]: pgmap v3762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:42 smithi132 ceph-mon[37580]: pgmap v3762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:44.263+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:44.263+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:44.263+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:44 smithi112 ceph-mon[37501]: pgmap v3763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:33:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:44 smithi107 ceph-mon[32659]: pgmap v3763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:33:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:44 smithi132 ceph-mon[37580]: pgmap v3763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:33:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6161.884145 secs 2024-07-26T14:33:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:46 smithi112 ceph-mon[37501]: pgmap v3764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6161.884145 secs 2024-07-26T14:33:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:46 smithi107 ceph-mon[32659]: pgmap v3764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6161.884145 secs 2024-07-26T14:33:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:46 smithi132 ceph-mon[37580]: pgmap v3764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:48 smithi112 ceph-mon[37501]: pgmap v3765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:48 smithi107 ceph-mon[32659]: pgmap v3765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:48 smithi132 ceph-mon[37580]: pgmap v3765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:50 smithi112 ceph-mon[37501]: pgmap v3766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:50 smithi107 ceph-mon[32659]: pgmap v3766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:50 smithi132 ceph-mon[37580]: pgmap v3766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6166.884264 secs 2024-07-26T14:33:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6166.884264 secs 2024-07-26T14:33:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6166.884264 secs 2024-07-26T14:33:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:52 smithi112 ceph-mon[37501]: pgmap v3767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:52 smithi107 ceph-mon[32659]: pgmap v3767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:52 smithi132 ceph-mon[37580]: pgmap v3767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:54 smithi112 ceph-mon[37501]: pgmap v3768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:54 smithi107 ceph-mon[32659]: pgmap v3768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:54 smithi132 ceph-mon[37580]: pgmap v3768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:57.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:57.111+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:57.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:57.111+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:57.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:33:57.111+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6171.884360 secs 2024-07-26T14:33:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:56 smithi107 ceph-mon[32659]: pgmap v3769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6171.884360 secs 2024-07-26T14:33:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:56 smithi132 ceph-mon[37580]: pgmap v3769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6171.884360 secs 2024-07-26T14:33:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:56 smithi112 ceph-mon[37501]: pgmap v3769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:59.265+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:59.265+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:33:59.267 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:33:59.265+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:33:58 smithi107 ceph-mon[32659]: pgmap v3770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:33:58 smithi132 ceph-mon[37580]: pgmap v3770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:33:58 smithi112 ceph-mon[37501]: pgmap v3770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:00 smithi107 ceph-mon[32659]: pgmap v3771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:00 smithi132 ceph-mon[37580]: pgmap v3771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:00 smithi112 ceph-mon[37501]: pgmap v3771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6176.884445 secs 2024-07-26T14:34:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6176.884445 secs 2024-07-26T14:34:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6176.884445 secs 2024-07-26T14:34:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:02 smithi107 ceph-mon[32659]: pgmap v3772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:02 smithi132 ceph-mon[37580]: pgmap v3772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:02 smithi112 ceph-mon[37501]: pgmap v3772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:04 smithi107 ceph-mon[32659]: pgmap v3773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:34:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:04 smithi132 ceph-mon[37580]: pgmap v3773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:34:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:04 smithi112 ceph-mon[37501]: pgmap v3773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:34:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6181.884568 secs 2024-07-26T14:34:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:06 smithi107 ceph-mon[32659]: pgmap v3774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:34:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6181.884568 secs 2024-07-26T14:34:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:06 smithi132 ceph-mon[37580]: pgmap v3774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:34:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6181.884568 secs 2024-07-26T14:34:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:06 smithi112 ceph-mon[37501]: pgmap v3774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:34:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:08 smithi107 ceph-mon[32659]: pgmap v3775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:34:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:08 smithi132 ceph-mon[37580]: pgmap v3775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:34:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:08 smithi112 ceph-mon[37501]: pgmap v3775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:34:11.252 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:10 smithi112 ceph-mon[37501]: pgmap v3776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:10 smithi107 ceph-mon[32659]: pgmap v3776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:10 smithi132 ceph-mon[37580]: pgmap v3776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:12.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:12.127+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:12.128 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:12.127+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:12.129 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:12.127+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6186.884688 secs 2024-07-26T14:34:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6186.884688 secs 2024-07-26T14:34:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6186.884688 secs 2024-07-26T14:34:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:13 smithi107 ceph-mon[32659]: pgmap v3777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:13 smithi132 ceph-mon[37580]: pgmap v3777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:13 smithi112 ceph-mon[37501]: pgmap v3777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:14.282+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:14.282+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:14.282+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:15 smithi107 ceph-mon[32659]: pgmap v3778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:15 smithi132 ceph-mon[37580]: pgmap v3778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:15 smithi112 ceph-mon[37501]: pgmap v3778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6191.884808 secs 2024-07-26T14:34:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:17 smithi107 ceph-mon[32659]: pgmap v3779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6191.884808 secs 2024-07-26T14:34:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:17 smithi132 ceph-mon[37580]: pgmap v3779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6191.884808 secs 2024-07-26T14:34:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:17 smithi112 ceph-mon[37501]: pgmap v3779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:19 smithi107 ceph-mon[32659]: pgmap v3780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:19 smithi132 ceph-mon[37580]: pgmap v3780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:19 smithi112 ceph-mon[37501]: pgmap v3780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:21 smithi107 ceph-mon[32659]: pgmap v3781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:21 smithi132 ceph-mon[37580]: pgmap v3781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:21 smithi112 ceph-mon[37501]: pgmap v3781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6196.884927 secs 2024-07-26T14:34:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6196.884927 secs 2024-07-26T14:34:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6196.884927 secs 2024-07-26T14:34:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:23 smithi132 ceph-mon[37580]: pgmap v3782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:23 smithi112 ceph-mon[37501]: pgmap v3782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:23 smithi107 ceph-mon[32659]: pgmap v3782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:25 smithi132 ceph-mon[37580]: pgmap v3783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:25 smithi112 ceph-mon[37501]: pgmap v3783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:25 smithi107 ceph-mon[32659]: pgmap v3783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:27.143+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:27.143+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:27.144 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:27.143+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:27.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6201.885050 secs 2024-07-26T14:34:27.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:27 smithi107 ceph-mon[32659]: pgmap v3784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6201.885050 secs 2024-07-26T14:34:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:27 smithi112 ceph-mon[37501]: pgmap v3784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6201.885050 secs 2024-07-26T14:34:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:27 smithi132 ceph-mon[37580]: pgmap v3784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:29.297+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:29.297+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:29.297+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:29 smithi112 ceph-mon[37501]: pgmap v3785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:29 smithi107 ceph-mon[32659]: pgmap v3785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:29 smithi132 ceph-mon[37580]: pgmap v3785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:31 smithi112 ceph-mon[37501]: pgmap v3786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6206.885193 secs 2024-07-26T14:34:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:31 smithi107 ceph-mon[32659]: pgmap v3786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6206.885193 secs 2024-07-26T14:34:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:31 smithi132 ceph-mon[37580]: pgmap v3786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6206.885193 secs 2024-07-26T14:34:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:33 smithi112 ceph-mon[37501]: pgmap v3787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:33.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:33 smithi107 ceph-mon[32659]: pgmap v3787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:33 smithi132 ceph-mon[37580]: pgmap v3787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:34:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:34:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:34:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: pgmap v3788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:34:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:34:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:34:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: pgmap v3788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:34:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:34:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:34:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: pgmap v3788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:34:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:34:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:36 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:34:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:36 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:34:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:36 smithi112 ceph-mon[37501]: pgmap v3789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:34:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:36 smithi112 ceph-mon[37501]: pgmap v3790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6211.885262 secs 2024-07-26T14:34:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:36 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:34:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:36 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:34:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:36 smithi107 ceph-mon[32659]: pgmap v3789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:34:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:36 smithi107 ceph-mon[32659]: pgmap v3790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6211.885262 secs 2024-07-26T14:34:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:36 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:34:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:36 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:34:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:36 smithi132 ceph-mon[37580]: pgmap v3789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:34:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:36 smithi132 ceph-mon[37580]: pgmap v3790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6211.885262 secs 2024-07-26T14:34:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:38 smithi112 ceph-mon[37501]: pgmap v3791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:38 smithi107 ceph-mon[32659]: pgmap v3791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:38 smithi132 ceph-mon[37580]: pgmap v3791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:40 smithi112 ceph-mon[37501]: pgmap v3792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:34:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:40 smithi107 ceph-mon[32659]: pgmap v3792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:34:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:40 smithi132 ceph-mon[37580]: pgmap v3792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:34:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6216.885366 secs 2024-07-26T14:34:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6216.885366 secs 2024-07-26T14:34:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6216.885366 secs 2024-07-26T14:34:42.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:42.157+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:42.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:42.157+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:42.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:42.157+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:42 smithi112 ceph-mon[37501]: pgmap v3793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:34:42.460 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:42 smithi132 ceph-mon[37580]: pgmap v3793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:34:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:42 smithi107 ceph-mon[32659]: pgmap v3793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:34:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:44.311+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:44.311+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:44.311+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:44 smithi112 ceph-mon[37501]: pgmap v3794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:34:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:44 smithi107 ceph-mon[32659]: pgmap v3794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:34:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:44 smithi132 ceph-mon[37580]: pgmap v3794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:34:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:46 smithi112 ceph-mon[37501]: pgmap v3795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:34:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6221.885496 secs 2024-07-26T14:34:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:46 smithi107 ceph-mon[32659]: pgmap v3795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:34:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6221.885496 secs 2024-07-26T14:34:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:46 smithi132 ceph-mon[37580]: pgmap v3795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:34:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6221.885496 secs 2024-07-26T14:34:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:48 smithi112 ceph-mon[37501]: pgmap v3796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:48 smithi107 ceph-mon[32659]: pgmap v3796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:48 smithi132 ceph-mon[37580]: pgmap v3796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:50 smithi112 ceph-mon[37501]: pgmap v3797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:50 smithi107 ceph-mon[32659]: pgmap v3797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:50.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:50 smithi132 ceph-mon[37580]: pgmap v3797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6226.885623 secs 2024-07-26T14:34:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6226.885623 secs 2024-07-26T14:34:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6226.885623 secs 2024-07-26T14:34:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:52 smithi112 ceph-mon[37501]: pgmap v3798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:52 smithi107 ceph-mon[32659]: pgmap v3798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:52 smithi132 ceph-mon[37580]: pgmap v3798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:54 smithi112 ceph-mon[37501]: pgmap v3799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:54 smithi107 ceph-mon[32659]: pgmap v3799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:54 smithi132 ceph-mon[37580]: pgmap v3799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:34:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:56 smithi112 ceph-mon[37501]: pgmap v3800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6231.885726 secs 2024-07-26T14:34:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:56 smithi107 ceph-mon[32659]: pgmap v3800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6231.885726 secs 2024-07-26T14:34:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:56 smithi132 ceph-mon[37580]: pgmap v3800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6231.885726 secs 2024-07-26T14:34:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:57.172+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:57.172+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:57.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:34:57.172+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:34:58 smithi112 ceph-mon[37501]: pgmap v3801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:34:58 smithi107 ceph-mon[32659]: pgmap v3801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:34:58 smithi132 ceph-mon[37580]: pgmap v3801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:34:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:59.323+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:34:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:59.323+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:59.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:34:59.323+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:00 smithi112 ceph-mon[37501]: pgmap v3802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:00 smithi107 ceph-mon[32659]: pgmap v3802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:00 smithi132 ceph-mon[37580]: pgmap v3802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6236.885871 secs 2024-07-26T14:35:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6236.885871 secs 2024-07-26T14:35:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6236.885871 secs 2024-07-26T14:35:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:02 smithi112 ceph-mon[37501]: pgmap v3803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:02 smithi107 ceph-mon[32659]: pgmap v3803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:02 smithi132 ceph-mon[37580]: pgmap v3803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:04 smithi107 ceph-mon[32659]: pgmap v3804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:04 smithi132 ceph-mon[37580]: pgmap v3804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:04 smithi112 ceph-mon[37501]: pgmap v3804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:06 smithi107 ceph-mon[32659]: pgmap v3805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6241.885995 secs 2024-07-26T14:35:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:06 smithi132 ceph-mon[37580]: pgmap v3805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6241.885995 secs 2024-07-26T14:35:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:06 smithi112 ceph-mon[37501]: pgmap v3805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6241.885995 secs 2024-07-26T14:35:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:08 smithi107 ceph-mon[32659]: pgmap v3806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:08 smithi132 ceph-mon[37580]: pgmap v3806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:08 smithi112 ceph-mon[37501]: pgmap v3806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:10 smithi107 ceph-mon[32659]: pgmap v3807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:10 smithi132 ceph-mon[37580]: pgmap v3807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:10 smithi112 ceph-mon[37501]: pgmap v3807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6246.886062 secs 2024-07-26T14:35:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6246.886062 secs 2024-07-26T14:35:11.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6246.886062 secs 2024-07-26T14:35:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:12.185+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:12.185+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:12.185+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:12 smithi107 ceph-mon[32659]: pgmap v3808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:12 smithi132 ceph-mon[37580]: pgmap v3808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:12 smithi112 ceph-mon[37501]: pgmap v3808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:14.339+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:14.339+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:14.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:14.339+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:14 smithi107 ceph-mon[32659]: pgmap v3809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:14 smithi132 ceph-mon[37580]: pgmap v3809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:14 smithi112 ceph-mon[37501]: pgmap v3809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:15 smithi107 ceph-mon[32659]: pgmap v3810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:15 smithi132 ceph-mon[37580]: pgmap v3810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:15 smithi112 ceph-mon[37501]: pgmap v3810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6251.886178 secs 2024-07-26T14:35:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6251.886178 secs 2024-07-26T14:35:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6251.886178 secs 2024-07-26T14:35:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:17 smithi107 ceph-mon[32659]: pgmap v3811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:17 smithi132 ceph-mon[37580]: pgmap v3811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:17 smithi112 ceph-mon[37501]: pgmap v3811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:20 smithi112 ceph-mon[37501]: pgmap v3812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:20 smithi107 ceph-mon[32659]: pgmap v3812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:20 smithi132 ceph-mon[37580]: pgmap v3812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6256.886326 secs 2024-07-26T14:35:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6256.886326 secs 2024-07-26T14:35:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6256.886326 secs 2024-07-26T14:35:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:22 smithi107 ceph-mon[32659]: pgmap v3813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:22 smithi112 ceph-mon[37501]: pgmap v3813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:22 smithi132 ceph-mon[37580]: pgmap v3813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:24 smithi112 ceph-mon[37501]: pgmap v3814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:24 smithi107 ceph-mon[32659]: pgmap v3814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:24 smithi132 ceph-mon[37580]: pgmap v3814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:26 smithi112 ceph-mon[37501]: pgmap v3815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6261.886442 secs 2024-07-26T14:35:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:26 smithi107 ceph-mon[32659]: pgmap v3815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6261.886442 secs 2024-07-26T14:35:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:26 smithi132 ceph-mon[37580]: pgmap v3815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6261.886442 secs 2024-07-26T14:35:27.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:27.199+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:27.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:27.199+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:27.201 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:27.199+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:28 smithi112 ceph-mon[37501]: pgmap v3816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:28 smithi107 ceph-mon[32659]: pgmap v3816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:28 smithi132 ceph-mon[37580]: pgmap v3816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:29.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:29.355+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:29.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:29.355+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:29.356 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:29.355+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:30 smithi112 ceph-mon[37501]: pgmap v3817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:30 smithi107 ceph-mon[32659]: pgmap v3817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:30 smithi132 ceph-mon[37580]: pgmap v3817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6266.886562 secs 2024-07-26T14:35:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6266.886562 secs 2024-07-26T14:35:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6266.886562 secs 2024-07-26T14:35:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:32 smithi112 ceph-mon[37501]: pgmap v3818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:32 smithi107 ceph-mon[32659]: pgmap v3818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:32 smithi132 ceph-mon[37580]: pgmap v3818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:34 smithi112 ceph-mon[37501]: pgmap v3819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:34 smithi107 ceph-mon[32659]: pgmap v3819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:34 smithi132 ceph-mon[37580]: pgmap v3819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: pgmap v3820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6271.886685 secs 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: pgmap v3820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6271.886685 secs 2024-07-26T14:35:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:35:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: pgmap v3820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6271.886685 secs 2024-07-26T14:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:37 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:35:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:37 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:35:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:37 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:35:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:37 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:35:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:35:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:35:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:35:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:37 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:35:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:37 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:35:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:37 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:35:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:37 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:35:37.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:35:37.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:35:37.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:35:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:37 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:35:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:37 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:35:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:37 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:35:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:37 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:35:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:35:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:35:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:35:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:38 smithi112 ceph-mon[37501]: pgmap v3821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:35:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:38 smithi112 ceph-mon[37501]: pgmap v3822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:35:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:38 smithi107 ceph-mon[32659]: pgmap v3821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:35:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:38 smithi107 ceph-mon[32659]: pgmap v3822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:35:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:38 smithi132 ceph-mon[37580]: pgmap v3821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:35:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:38 smithi132 ceph-mon[37580]: pgmap v3822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:35:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:40 smithi112 ceph-mon[37501]: pgmap v3823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:35:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:40 smithi107 ceph-mon[32659]: pgmap v3823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:35:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:40 smithi132 ceph-mon[37580]: pgmap v3823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:35:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6276.886801 secs 2024-07-26T14:35:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6276.886801 secs 2024-07-26T14:35:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6276.886801 secs 2024-07-26T14:35:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:42.214+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:42.214+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:42.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:42.214+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:42.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:42 smithi132 ceph-mon[37580]: pgmap v3824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:35:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:42 smithi107 ceph-mon[32659]: pgmap v3824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:35:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:42 smithi112 ceph-mon[37501]: pgmap v3824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:35:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:44.370+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:44.370+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:44.372 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:44.370+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:44 smithi107 ceph-mon[32659]: pgmap v3825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:44 smithi132 ceph-mon[37580]: pgmap v3825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:44 smithi112 ceph-mon[37501]: pgmap v3825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:46 smithi107 ceph-mon[32659]: pgmap v3826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:35:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6281.886865 secs 2024-07-26T14:35:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:46 smithi132 ceph-mon[37580]: pgmap v3826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:35:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6281.886865 secs 2024-07-26T14:35:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:46 smithi112 ceph-mon[37501]: pgmap v3826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:35:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6281.886865 secs 2024-07-26T14:35:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:48 smithi107 ceph-mon[32659]: pgmap v3827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:35:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:48 smithi132 ceph-mon[37580]: pgmap v3827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:35:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:48 smithi112 ceph-mon[37501]: pgmap v3827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:35:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:50 smithi107 ceph-mon[32659]: pgmap v3828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:50 smithi132 ceph-mon[37580]: pgmap v3828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:50 smithi112 ceph-mon[37501]: pgmap v3828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:35:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6286.886988 secs 2024-07-26T14:35:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6286.886988 secs 2024-07-26T14:35:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6286.886988 secs 2024-07-26T14:35:52.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:52 smithi107 ceph-mon[32659]: pgmap v3829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:52 smithi132 ceph-mon[37580]: pgmap v3829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:52 smithi112 ceph-mon[37501]: pgmap v3829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:53 smithi107 ceph-mon[32659]: pgmap v3830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:53 smithi132 ceph-mon[37580]: pgmap v3830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:53 smithi112 ceph-mon[37501]: pgmap v3830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:35:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:55 smithi112 ceph-mon[37501]: pgmap v3831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:35:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:55 smithi107 ceph-mon[32659]: pgmap v3831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:35:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:55 smithi132 ceph-mon[37580]: pgmap v3831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:35:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6291.887107 secs 2024-07-26T14:35:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6291.887107 secs 2024-07-26T14:35:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6291.887107 secs 2024-07-26T14:35:57.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:57.229+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:57.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:57.229+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:57.231 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:35:57.229+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:57 smithi107 ceph-mon[32659]: pgmap v3832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:35:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:57 smithi112 ceph-mon[37501]: pgmap v3832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:35:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:57 smithi132 ceph-mon[37580]: pgmap v3832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:35:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:59.386+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:59.386+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:35:59.386+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:35:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:35:59 smithi112 ceph-mon[37501]: pgmap v3833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:35:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:35:59 smithi107 ceph-mon[32659]: pgmap v3833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:35:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:35:59 smithi132 ceph-mon[37580]: pgmap v3833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:36:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6296.887220 secs 2024-07-26T14:36:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:01 smithi112 ceph-mon[37501]: pgmap v3834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6296.887220 secs 2024-07-26T14:36:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:01 smithi107 ceph-mon[32659]: pgmap v3834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6296.887220 secs 2024-07-26T14:36:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:01 smithi132 ceph-mon[37580]: pgmap v3834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:03 smithi112 ceph-mon[37501]: pgmap v3835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:03 smithi107 ceph-mon[32659]: pgmap v3835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:03 smithi132 ceph-mon[37580]: pgmap v3835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:05 smithi112 ceph-mon[37501]: pgmap v3836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:36:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:05 smithi107 ceph-mon[32659]: pgmap v3836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:36:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:05 smithi132 ceph-mon[37580]: pgmap v3836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:36:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6301.887289 secs 2024-07-26T14:36:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6301.887289 secs 2024-07-26T14:36:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6301.887289 secs 2024-07-26T14:36:07.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:07 smithi112 ceph-mon[37501]: pgmap v3837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:07 smithi107 ceph-mon[32659]: pgmap v3837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:07 smithi132 ceph-mon[37580]: pgmap v3837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:09 smithi112 ceph-mon[37501]: pgmap v3838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:09 smithi107 ceph-mon[32659]: pgmap v3838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:09 smithi132 ceph-mon[37580]: pgmap v3838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:11.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:11.654+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0c0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T14:36:11.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:11.654+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0c0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:36:11.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:11.654+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c0590 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f8051040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:36:11.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:11.654+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c0590 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f8051040 comp rx=0 tx=0).stop 2024-07-26T14:36:11.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:11.654+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0c0590 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T14:36:11.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:11.654+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 msgr2=0x7ff0ec0c0590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:36:11.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:11.654+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0b3d40 0x7ff0ec0c0590 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:36:11.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:11.654+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ad0 0x7ff0ec0c0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:36:11.657 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6306.887408 secs 2024-07-26T14:36:11.657 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:11 smithi107 ceph-mon[32659]: pgmap v3839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:11.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:11.655+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ad0 0x7ff0ec0c0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:36:11.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:11.657+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ad0 0x7ff0ec0c0ee0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fa00 tx=0x7ff0f801f040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:36:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6306.887408 secs 2024-07-26T14:36:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:11 smithi112 ceph-mon[37501]: pgmap v3839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6306.887408 secs 2024-07-26T14:36:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:11 smithi132 ceph-mon[37580]: pgmap v3839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:12.231+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:12.231+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:12.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:12.231+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:13 smithi112 ceph-mon[37501]: pgmap v3840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:13.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:13.704+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:36:13.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:13 smithi107 ceph-mon[32659]: pgmap v3840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:13.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:13.704+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:36:13.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:13.724+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:36:13.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:13.725+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:36:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:13.763+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000214) for more than 900000000 us, fault. 2024-07-26T14:36:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:13.763+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c3270 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7efc48005a20 tx=0x7efc6809a000 comp rx=0 tx=0).stop 2024-07-26T14:36:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:13.764+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c3270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:36:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:13.764+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c3270 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:36:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:13.764+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2b10 0x7efc5c0c37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:36:13.765 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:13.764+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2b10 0x7efc5c0c37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:36:13.766 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:13.766+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2b10 0x7efc5c0c37b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7efc680aacd0 tx=0x7efc680e6070 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:36:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:13 smithi132 ceph-mon[37580]: pgmap v3840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:14.390+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:14.390+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:14.391 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:14.390+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:14.871+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:36:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:14.871+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc48006620 con 0x7efc78103340 2024-07-26T14:36:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:14.871+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:36:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:14.872+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc48006620 con 0x7efc78103340 2024-07-26T14:36:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:15 smithi112 ceph-mon[37501]: pgmap v3841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:15 smithi107 ceph-mon[32659]: pgmap v3841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:15 smithi132 ceph-mon[37580]: pgmap v3841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6311.887533 secs 2024-07-26T14:36:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6311.887533 secs 2024-07-26T14:36:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6311.887533 secs 2024-07-26T14:36:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:17 smithi112 ceph-mon[37501]: pgmap v3842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:17 smithi107 ceph-mon[32659]: pgmap v3842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:17 smithi132 ceph-mon[37580]: pgmap v3842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:19 smithi112 ceph-mon[37501]: pgmap v3843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:19 smithi107 ceph-mon[32659]: pgmap v3843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:19 smithi132 ceph-mon[37580]: pgmap v3843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6316.887669 secs 2024-07-26T14:36:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:21 smithi112 ceph-mon[37501]: pgmap v3844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6316.887669 secs 2024-07-26T14:36:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:21 smithi107 ceph-mon[32659]: pgmap v3844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6316.887669 secs 2024-07-26T14:36:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:21 smithi132 ceph-mon[37580]: pgmap v3844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:23 smithi112 ceph-mon[37501]: pgmap v3845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:23 smithi107 ceph-mon[32659]: pgmap v3845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:23 smithi132 ceph-mon[37580]: pgmap v3845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:25 smithi112 ceph-mon[37501]: pgmap v3846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:25 smithi107 ceph-mon[32659]: pgmap v3846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:25 smithi132 ceph-mon[37580]: pgmap v3846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6321.887787 secs 2024-07-26T14:36:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6321.887787 secs 2024-07-26T14:36:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6321.887787 secs 2024-07-26T14:36:27.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:27.247+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:27.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:27.247+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:27.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:27.247+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:27 smithi107 ceph-mon[32659]: pgmap v3847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:27 smithi132 ceph-mon[37580]: pgmap v3847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:27 smithi112 ceph-mon[37501]: pgmap v3847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:29.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:29.406+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:29.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:29.406+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:29.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:29.406+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:29 smithi107 ceph-mon[32659]: pgmap v3848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:29 smithi132 ceph-mon[37580]: pgmap v3848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:29 smithi112 ceph-mon[37501]: pgmap v3848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6326.887907 secs 2024-07-26T14:36:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:31 smithi107 ceph-mon[32659]: pgmap v3849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6326.887907 secs 2024-07-26T14:36:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:31 smithi132 ceph-mon[37580]: pgmap v3849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6326.887907 secs 2024-07-26T14:36:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:31 smithi112 ceph-mon[37501]: pgmap v3849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:33 smithi107 ceph-mon[32659]: pgmap v3850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:33 smithi132 ceph-mon[37580]: pgmap v3850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:33 smithi112 ceph-mon[37501]: pgmap v3850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:35 smithi107 ceph-mon[32659]: pgmap v3851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:35 smithi132 ceph-mon[37580]: pgmap v3851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:35 smithi112 ceph-mon[37501]: pgmap v3851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:36.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6331.888024 secs 2024-07-26T14:36:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:36:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6331.888024 secs 2024-07-26T14:36:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:36:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6331.888024 secs 2024-07-26T14:36:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:36:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:37 smithi107 ceph-mon[32659]: pgmap v3852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:36:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:36:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:36:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:37 smithi132 ceph-mon[37580]: pgmap v3852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:36:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:36:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:36:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:37 smithi112 ceph-mon[37501]: pgmap v3852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:36:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:36:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:36:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:38 smithi107 ceph-mon[32659]: pgmap v3853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:36:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:38 smithi107 ceph-mon[32659]: pgmap v3854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:38 smithi132 ceph-mon[37580]: pgmap v3853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:36:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:38 smithi132 ceph-mon[37580]: pgmap v3854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:38 smithi112 ceph-mon[37501]: pgmap v3853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:36:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:38 smithi112 ceph-mon[37501]: pgmap v3854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:40 smithi107 ceph-mon[32659]: pgmap v3855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:40 smithi132 ceph-mon[37580]: pgmap v3855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:40 smithi112 ceph-mon[37501]: pgmap v3855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6336.888144 secs 2024-07-26T14:36:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6336.888144 secs 2024-07-26T14:36:41.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6336.888144 secs 2024-07-26T14:36:42.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:42.262+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:42.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:42.262+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:42.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:42.262+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:42 smithi107 ceph-mon[32659]: pgmap v3856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:36:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:42 smithi132 ceph-mon[37580]: pgmap v3856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:36:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:42 smithi112 ceph-mon[37501]: pgmap v3856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:36:44.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:44.406+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:44.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:44.406+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:44.408 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:44.406+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:44 smithi107 ceph-mon[32659]: pgmap v3857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:44 smithi132 ceph-mon[37580]: pgmap v3857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:44 smithi112 ceph-mon[37501]: pgmap v3857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6341.888268 secs 2024-07-26T14:36:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:46 smithi132 ceph-mon[37580]: pgmap v3858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6341.888268 secs 2024-07-26T14:36:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:46 smithi112 ceph-mon[37501]: pgmap v3858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6341.888268 secs 2024-07-26T14:36:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:46 smithi107 ceph-mon[32659]: pgmap v3858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:48 smithi132 ceph-mon[37580]: pgmap v3859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:36:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:48 smithi112 ceph-mon[37501]: pgmap v3859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:36:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:48 smithi107 ceph-mon[32659]: pgmap v3859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:36:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:50 smithi132 ceph-mon[37580]: pgmap v3860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:36:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:50 smithi112 ceph-mon[37501]: pgmap v3860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:36:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:50 smithi107 ceph-mon[32659]: pgmap v3860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:36:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6346.888386 secs 2024-07-26T14:36:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6346.888386 secs 2024-07-26T14:36:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6346.888386 secs 2024-07-26T14:36:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:52 smithi112 ceph-mon[37501]: pgmap v3861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:52 smithi107 ceph-mon[32659]: pgmap v3861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:52 smithi132 ceph-mon[37580]: pgmap v3861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:54 smithi112 ceph-mon[37501]: pgmap v3862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:54 smithi107 ceph-mon[32659]: pgmap v3862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:54 smithi132 ceph-mon[37580]: pgmap v3862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6351.888504 secs 2024-07-26T14:36:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:56 smithi112 ceph-mon[37501]: pgmap v3863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6351.888504 secs 2024-07-26T14:36:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:56 smithi107 ceph-mon[32659]: pgmap v3863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6351.888504 secs 2024-07-26T14:36:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:56 smithi132 ceph-mon[37580]: pgmap v3863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:57.264+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:57.264+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:57.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:36:57.264+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:36:58 smithi112 ceph-mon[37501]: pgmap v3864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:36:58 smithi107 ceph-mon[32659]: pgmap v3864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:36:58 smithi132 ceph-mon[37580]: pgmap v3864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:36:59.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:59.419+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:59.420 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:59.419+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:36:59.421 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:36:59.419+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:00 smithi112 ceph-mon[37501]: pgmap v3865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:00 smithi107 ceph-mon[32659]: pgmap v3865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:00 smithi132 ceph-mon[37580]: pgmap v3865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6356.888621 secs 2024-07-26T14:37:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6356.888621 secs 2024-07-26T14:37:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6356.888621 secs 2024-07-26T14:37:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:02 smithi112 ceph-mon[37501]: pgmap v3866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:03.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:02 smithi107 ceph-mon[32659]: pgmap v3866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:02 smithi132 ceph-mon[37580]: pgmap v3866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:04 smithi112 ceph-mon[37501]: pgmap v3867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:04 smithi107 ceph-mon[32659]: pgmap v3867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:04 smithi132 ceph-mon[37580]: pgmap v3867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6361.888757 secs 2024-07-26T14:37:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:06 smithi112 ceph-mon[37501]: pgmap v3868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6361.888757 secs 2024-07-26T14:37:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:06 smithi107 ceph-mon[32659]: pgmap v3868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6361.888757 secs 2024-07-26T14:37:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:06 smithi132 ceph-mon[37580]: pgmap v3868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:08 smithi112 ceph-mon[37501]: pgmap v3869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:08 smithi107 ceph-mon[32659]: pgmap v3869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:08 smithi132 ceph-mon[37580]: pgmap v3869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:10 smithi112 ceph-mon[37501]: pgmap v3870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:11.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:10 smithi107 ceph-mon[32659]: pgmap v3870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:10 smithi132 ceph-mon[37580]: pgmap v3870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6366.888870 secs 2024-07-26T14:37:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6366.888870 secs 2024-07-26T14:37:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6366.888870 secs 2024-07-26T14:37:12.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:12.281+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:12.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:12.281+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:12.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:12.281+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:12 smithi112 ceph-mon[37501]: pgmap v3871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:12 smithi107 ceph-mon[32659]: pgmap v3871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:12 smithi132 ceph-mon[37580]: pgmap v3871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:14.435+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:14.435+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:14.435+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:14 smithi112 ceph-mon[37501]: pgmap v3872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:15.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:14 smithi107 ceph-mon[32659]: pgmap v3872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:14 smithi132 ceph-mon[37580]: pgmap v3872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6371.888990 secs 2024-07-26T14:37:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:16 smithi112 ceph-mon[37501]: pgmap v3873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6371.888990 secs 2024-07-26T14:37:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:16 smithi107 ceph-mon[32659]: pgmap v3873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:17.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6371.888990 secs 2024-07-26T14:37:17.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:16 smithi132 ceph-mon[37580]: pgmap v3873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:18 smithi112 ceph-mon[37501]: pgmap v3874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:18 smithi107 ceph-mon[32659]: pgmap v3874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:18 smithi132 ceph-mon[37580]: pgmap v3874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:20 smithi112 ceph-mon[37501]: pgmap v3875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:20 smithi107 ceph-mon[32659]: pgmap v3875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:20 smithi132 ceph-mon[37580]: pgmap v3875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6376.889130 secs 2024-07-26T14:37:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6376.889130 secs 2024-07-26T14:37:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6376.889130 secs 2024-07-26T14:37:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:22 smithi112 ceph-mon[37501]: pgmap v3876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:23.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:22 smithi107 ceph-mon[32659]: pgmap v3876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:22 smithi132 ceph-mon[37580]: pgmap v3876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:24 smithi112 ceph-mon[37501]: pgmap v3877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:24 smithi107 ceph-mon[32659]: pgmap v3877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:24 smithi132 ceph-mon[37580]: pgmap v3877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6381.889226 secs 2024-07-26T14:37:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:26 smithi112 ceph-mon[37501]: pgmap v3878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6381.889226 secs 2024-07-26T14:37:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:26 smithi107 ceph-mon[32659]: pgmap v3878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6381.889226 secs 2024-07-26T14:37:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:26 smithi132 ceph-mon[37580]: pgmap v3878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:27.291+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:27.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:27.291+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:27.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:27.292+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:28 smithi112 ceph-mon[37501]: pgmap v3879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:28 smithi107 ceph-mon[32659]: pgmap v3879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:28 smithi132 ceph-mon[37580]: pgmap v3879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:29.449+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:29.449+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:29.450 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:29.449+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:30 smithi112 ceph-mon[37501]: pgmap v3880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:30 smithi107 ceph-mon[32659]: pgmap v3880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:30 smithi132 ceph-mon[37580]: pgmap v3880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:37:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6386.889345 secs 2024-07-26T14:37:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6386.889345 secs 2024-07-26T14:37:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6386.889345 secs 2024-07-26T14:37:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:32 smithi107 ceph-mon[32659]: pgmap v3881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:32 smithi132 ceph-mon[37580]: pgmap v3881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:32 smithi112 ceph-mon[37501]: pgmap v3881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:37:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:34 smithi107 ceph-mon[32659]: pgmap v3882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:34 smithi132 ceph-mon[37580]: pgmap v3882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:34 smithi112 ceph-mon[37501]: pgmap v3882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6391.889471 secs 2024-07-26T14:37:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:36 smithi107 ceph-mon[32659]: pgmap v3883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6391.889471 secs 2024-07-26T14:37:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:36 smithi132 ceph-mon[37580]: pgmap v3883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6391.889471 secs 2024-07-26T14:37:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:36 smithi112 ceph-mon[37501]: pgmap v3883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:37:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:37:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:37:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:39 smithi107 ceph-mon[32659]: pgmap v3884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:37:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:37:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:37:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:37:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:39 smithi132 ceph-mon[37580]: pgmap v3884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:37:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:37:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:37:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:37:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:39 smithi112 ceph-mon[37501]: pgmap v3884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:37:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:37:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:37:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:37:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:40 smithi107 ceph-mon[32659]: pgmap v3885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:37:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:37:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:40 smithi132 ceph-mon[37580]: pgmap v3885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:37:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:37:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:40 smithi112 ceph-mon[37501]: pgmap v3885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:37:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:37:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6396.889608 secs 2024-07-26T14:37:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6396.889608 secs 2024-07-26T14:37:41.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6396.889608 secs 2024-07-26T14:37:42.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:42.298+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:42.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:42.298+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:42.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:42.298+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:42 smithi107 ceph-mon[32659]: pgmap v3886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:37:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:42 smithi132 ceph-mon[37580]: pgmap v3886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:37:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:42 smithi112 ceph-mon[37501]: pgmap v3886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:37:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:44.465+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:44.465+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:44.465+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:44 smithi107 ceph-mon[32659]: pgmap v3887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:37:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:44 smithi132 ceph-mon[37580]: pgmap v3887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:37:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:44 smithi112 ceph-mon[37501]: pgmap v3887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:37:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:45 smithi112 ceph-mon[37501]: pgmap v3888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:37:45.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:45 smithi107 ceph-mon[32659]: pgmap v3888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:37:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:45 smithi132 ceph-mon[37580]: pgmap v3888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:37:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6401.889734 secs 2024-07-26T14:37:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6401.889734 secs 2024-07-26T14:37:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6401.889734 secs 2024-07-26T14:37:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:47 smithi112 ceph-mon[37501]: pgmap v3889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:37:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:47 smithi107 ceph-mon[32659]: pgmap v3889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:37:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:47 smithi132 ceph-mon[37580]: pgmap v3889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:37:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:49 smithi112 ceph-mon[37501]: pgmap v3890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:37:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:49 smithi107 ceph-mon[32659]: pgmap v3890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:37:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:49 smithi132 ceph-mon[37580]: pgmap v3890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:37:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6406.889873 secs 2024-07-26T14:37:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6406.889873 secs 2024-07-26T14:37:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6406.889873 secs 2024-07-26T14:37:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:52 smithi112 ceph-mon[37501]: pgmap v3891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:52 smithi107 ceph-mon[32659]: pgmap v3891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:52 smithi132 ceph-mon[37580]: pgmap v3891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:53 smithi112 ceph-mon[37501]: pgmap v3892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:53 smithi107 ceph-mon[32659]: pgmap v3892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:53 smithi132 ceph-mon[37580]: pgmap v3892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:55 smithi112 ceph-mon[37501]: pgmap v3893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:55 smithi107 ceph-mon[32659]: pgmap v3893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:55 smithi132 ceph-mon[37580]: pgmap v3893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6411.890016 secs 2024-07-26T14:37:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6411.890016 secs 2024-07-26T14:37:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6411.890016 secs 2024-07-26T14:37:57.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:57.313+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:57.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:57.313+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:57.315 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:37:57.313+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:57.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:57 smithi112 ceph-mon[37501]: pgmap v3894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:57 smithi107 ceph-mon[32659]: pgmap v3894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:57 smithi132 ceph-mon[37580]: pgmap v3894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:59.480+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:59.480+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:37:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:37:59.480+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:37:59 smithi112 ceph-mon[37501]: pgmap v3895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:37:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:37:59 smithi107 ceph-mon[32659]: pgmap v3895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:37:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:37:59 smithi132 ceph-mon[37580]: pgmap v3895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6416.890114 secs 2024-07-26T14:38:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6416.890114 secs 2024-07-26T14:38:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6416.890114 secs 2024-07-26T14:38:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:02 smithi112 ceph-mon[37501]: pgmap v3896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:02 smithi107 ceph-mon[32659]: pgmap v3896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:02 smithi132 ceph-mon[37580]: pgmap v3896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:03 smithi112 ceph-mon[37501]: pgmap v3897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:03 smithi107 ceph-mon[32659]: pgmap v3897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:03 smithi132 ceph-mon[37580]: pgmap v3897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:05 smithi112 ceph-mon[37501]: pgmap v3898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:05.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:05 smithi107 ceph-mon[32659]: pgmap v3898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:05 smithi132 ceph-mon[37580]: pgmap v3898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6421.890192 secs 2024-07-26T14:38:06.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6421.890192 secs 2024-07-26T14:38:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6421.890192 secs 2024-07-26T14:38:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:07 smithi112 ceph-mon[37501]: pgmap v3899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:07 smithi107 ceph-mon[32659]: pgmap v3899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:07 smithi132 ceph-mon[37580]: pgmap v3899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:09 smithi112 ceph-mon[37501]: pgmap v3900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:09 smithi107 ceph-mon[32659]: pgmap v3900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:09 smithi132 ceph-mon[37580]: pgmap v3900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6426.890289 secs 2024-07-26T14:38:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6426.890289 secs 2024-07-26T14:38:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6426.890289 secs 2024-07-26T14:38:12.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:12.329+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:12.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:12.329+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:12.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:12.329+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:12 smithi112 ceph-mon[37501]: pgmap v3901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:12 smithi107 ceph-mon[32659]: pgmap v3901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:12 smithi132 ceph-mon[37580]: pgmap v3901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:13 smithi112 ceph-mon[37501]: pgmap v3902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:13 smithi107 ceph-mon[32659]: pgmap v3902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:13 smithi132 ceph-mon[37580]: pgmap v3902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:14.496+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:14.496+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:14.496 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:14.496+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:15.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:15 smithi132 ceph-mon[37580]: pgmap v3903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:15 smithi112 ceph-mon[37501]: pgmap v3903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:15 smithi107 ceph-mon[32659]: pgmap v3903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6431.890409 secs 2024-07-26T14:38:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6431.890409 secs 2024-07-26T14:38:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6431.890409 secs 2024-07-26T14:38:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:17 smithi112 ceph-mon[37501]: pgmap v3904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:17 smithi107 ceph-mon[32659]: pgmap v3904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:17 smithi132 ceph-mon[37580]: pgmap v3904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:19 smithi112 ceph-mon[37501]: pgmap v3905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:19 smithi107 ceph-mon[32659]: pgmap v3905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:19 smithi132 ceph-mon[37580]: pgmap v3905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6436.890543 secs 2024-07-26T14:38:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6436.890543 secs 2024-07-26T14:38:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6436.890543 secs 2024-07-26T14:38:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:22 smithi112 ceph-mon[37501]: pgmap v3906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:22 smithi107 ceph-mon[32659]: pgmap v3906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:22 smithi132 ceph-mon[37580]: pgmap v3906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:23 smithi112 ceph-mon[37501]: pgmap v3907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:23 smithi107 ceph-mon[32659]: pgmap v3907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:23 smithi132 ceph-mon[37580]: pgmap v3907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:25 smithi112 ceph-mon[37501]: pgmap v3908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:25 smithi107 ceph-mon[32659]: pgmap v3908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:25 smithi132 ceph-mon[37580]: pgmap v3908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6441.890666 secs 2024-07-26T14:38:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6441.890666 secs 2024-07-26T14:38:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6441.890666 secs 2024-07-26T14:38:27.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:27.344+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:27.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:27.344+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:27.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:27.344+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:27.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:27 smithi107 ceph-mon[32659]: pgmap v3909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:27 smithi112 ceph-mon[37501]: pgmap v3909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:27.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:27.727+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:38:27.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:27.727+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:38:27.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:27.745+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:38:27.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:27.746+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:38:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:27 smithi132 ceph-mon[37580]: pgmap v3909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:29.507+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:29.507+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:29.507+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:29 smithi112 ceph-mon[37501]: pgmap v3910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:29 smithi107 ceph-mon[32659]: pgmap v3910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:29 smithi132 ceph-mon[37580]: pgmap v3910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:29.896+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc480065c0 con 0x7efc78103340 2024-07-26T14:38:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:29.896+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:38:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:29.897+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:38:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:29.897+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc480065c0 con 0x7efc78103340 2024-07-26T14:38:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6446.890744 secs 2024-07-26T14:38:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6446.890744 secs 2024-07-26T14:38:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6446.890744 secs 2024-07-26T14:38:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:32 smithi112 ceph-mon[37501]: pgmap v3911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:32 smithi107 ceph-mon[32659]: pgmap v3911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:32 smithi132 ceph-mon[37580]: pgmap v3911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:33 smithi112 ceph-mon[37501]: pgmap v3912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:33 smithi107 ceph-mon[32659]: pgmap v3912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:33 smithi132 ceph-mon[37580]: pgmap v3912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:35 smithi112 ceph-mon[37501]: pgmap v3913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:35 smithi107 ceph-mon[32659]: pgmap v3913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:35 smithi132 ceph-mon[37580]: pgmap v3913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6451.890875 secs 2024-07-26T14:38:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6451.890875 secs 2024-07-26T14:38:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6451.890875 secs 2024-07-26T14:38:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:37 smithi112 ceph-mon[37501]: pgmap v3914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:37 smithi107 ceph-mon[32659]: pgmap v3914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:37 smithi132 ceph-mon[37580]: pgmap v3914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:39 smithi112 ceph-mon[37501]: pgmap v3915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:39 smithi107 ceph-mon[32659]: pgmap v3915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:39.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:39 smithi132 ceph-mon[37580]: pgmap v3915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:38:40.638 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:38:40.638 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.638 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.638 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.638 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.654 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:38:40.655 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.655 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.655 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.655 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:38:40.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:40.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6456.890945 secs 2024-07-26T14:38:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:41 smithi112 ceph-mon[37501]: pgmap v3916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6456.890945 secs 2024-07-26T14:38:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:41 smithi107 ceph-mon[32659]: pgmap v3916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6456.890945 secs 2024-07-26T14:38:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:41 smithi132 ceph-mon[37580]: pgmap v3916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:42.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:42.359+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:42.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:42.359+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:42.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:42.359+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:42.918 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:42.919 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:42.919 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:42.919 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:42.919 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:42.919 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:43 smithi107 ceph-mon[32659]: pgmap v3917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:43 smithi132 ceph-mon[37580]: pgmap v3917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:43 smithi112 ceph-mon[37501]: pgmap v3917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:44.522+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:44.522+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:44.522+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:38:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:38:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:38:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:38:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:38:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:38:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:38:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:46 smithi107 ceph-mon[32659]: pgmap v3918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:38:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6461.891087 secs 2024-07-26T14:38:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:46 smithi132 ceph-mon[37580]: pgmap v3918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:38:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6461.891087 secs 2024-07-26T14:38:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:46 smithi112 ceph-mon[37501]: pgmap v3918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:38:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6461.891087 secs 2024-07-26T14:38:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:47 smithi132 ceph-mon[37580]: pgmap v3919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:38:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:47 smithi112 ceph-mon[37501]: pgmap v3919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:38:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:47 smithi107 ceph-mon[32659]: pgmap v3919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:38:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:49 smithi112 ceph-mon[37501]: pgmap v3920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:38:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:49 smithi107 ceph-mon[32659]: pgmap v3920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:38:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:49 smithi132 ceph-mon[37580]: pgmap v3920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:38:51.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6466.891136 secs 2024-07-26T14:38:51.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:51 smithi112 ceph-mon[37501]: pgmap v3921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:38:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6466.891136 secs 2024-07-26T14:38:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:51 smithi107 ceph-mon[32659]: pgmap v3921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:38:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6466.891136 secs 2024-07-26T14:38:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:51 smithi132 ceph-mon[37580]: pgmap v3921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:38:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:53 smithi112 ceph-mon[37501]: pgmap v3922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:38:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:53 smithi107 ceph-mon[32659]: pgmap v3922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:38:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:53 smithi132 ceph-mon[37580]: pgmap v3922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:38:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:55 smithi112 ceph-mon[37501]: pgmap v3923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:38:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:55 smithi107 ceph-mon[32659]: pgmap v3923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:38:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:55 smithi132 ceph-mon[37580]: pgmap v3923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:38:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6471.891203 secs 2024-07-26T14:38:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6471.891203 secs 2024-07-26T14:38:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6471.891203 secs 2024-07-26T14:38:57.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:57.369+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:57.369+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:57.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:38:57.369+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:57.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:57 smithi107 ceph-mon[32659]: pgmap v3924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:57 smithi112 ceph-mon[37501]: pgmap v3924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:57 smithi132 ceph-mon[37580]: pgmap v3924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:38:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:59.537+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:59.537+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:38:59.538 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:38:59.537+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:38:59 smithi112 ceph-mon[37501]: pgmap v3925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:38:59 smithi107 ceph-mon[32659]: pgmap v3925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:38:59 smithi132 ceph-mon[37580]: pgmap v3925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6476.891267 secs 2024-07-26T14:39:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:01 smithi112 ceph-mon[37501]: pgmap v3926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6476.891267 secs 2024-07-26T14:39:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:01 smithi107 ceph-mon[32659]: pgmap v3926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6476.891267 secs 2024-07-26T14:39:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:01 smithi132 ceph-mon[37580]: pgmap v3926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:03 smithi112 ceph-mon[37501]: pgmap v3927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:03 smithi107 ceph-mon[32659]: pgmap v3927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:03 smithi132 ceph-mon[37580]: pgmap v3927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:05 smithi112 ceph-mon[37501]: pgmap v3928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:05 smithi107 ceph-mon[32659]: pgmap v3928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:05 smithi132 ceph-mon[37580]: pgmap v3928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6481.891351 secs 2024-07-26T14:39:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6481.891351 secs 2024-07-26T14:39:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6481.891351 secs 2024-07-26T14:39:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:07 smithi112 ceph-mon[37501]: pgmap v3929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:07 smithi107 ceph-mon[32659]: pgmap v3929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:07 smithi132 ceph-mon[37580]: pgmap v3929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:09 smithi112 ceph-mon[37501]: pgmap v3930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:09.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:09 smithi107 ceph-mon[32659]: pgmap v3930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:09 smithi132 ceph-mon[37580]: pgmap v3930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6486.891416 secs 2024-07-26T14:39:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:11 smithi112 ceph-mon[37501]: pgmap v3931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6486.891416 secs 2024-07-26T14:39:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:11 smithi107 ceph-mon[32659]: pgmap v3931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6486.891416 secs 2024-07-26T14:39:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:11 smithi132 ceph-mon[37580]: pgmap v3931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:12.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:12.381+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:12.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:12.381+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:12.383 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:12.381+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:13 smithi112 ceph-mon[37501]: pgmap v3932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:13.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:13.734+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:39:13.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:13 smithi107 ceph-mon[32659]: pgmap v3932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:13.736 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:13.734+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:39:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:13.752+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:39:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:13.752+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:39:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:13 smithi132 ceph-mon[37580]: pgmap v3932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:14.553+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:14.553+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:14.554 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:14.553+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:14.904+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:39:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:14.904+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065c0 con 0x7efc78103340 2024-07-26T14:39:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:14.904+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065c0 con 0x7efc78103340 2024-07-26T14:39:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:14.904+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:39:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:15 smithi112 ceph-mon[37501]: pgmap v3933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:15 smithi107 ceph-mon[32659]: pgmap v3933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:15 smithi132 ceph-mon[37580]: pgmap v3933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6491.891494 secs 2024-07-26T14:39:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6491.891494 secs 2024-07-26T14:39:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6491.891494 secs 2024-07-26T14:39:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:17 smithi107 ceph-mon[32659]: pgmap v3934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:17 smithi132 ceph-mon[37580]: pgmap v3934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:17 smithi112 ceph-mon[37501]: pgmap v3934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:19 smithi107 ceph-mon[32659]: pgmap v3935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:19 smithi132 ceph-mon[37580]: pgmap v3935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:19 smithi112 ceph-mon[37501]: pgmap v3935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6496.891576 secs 2024-07-26T14:39:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:21 smithi107 ceph-mon[32659]: pgmap v3936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6496.891576 secs 2024-07-26T14:39:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:21 smithi132 ceph-mon[37580]: pgmap v3936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6496.891576 secs 2024-07-26T14:39:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:21 smithi112 ceph-mon[37501]: pgmap v3936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:23 smithi107 ceph-mon[32659]: pgmap v3937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:23 smithi132 ceph-mon[37580]: pgmap v3937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:23 smithi112 ceph-mon[37501]: pgmap v3937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:25 smithi112 ceph-mon[37501]: pgmap v3938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:25 smithi107 ceph-mon[32659]: pgmap v3938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:25 smithi132 ceph-mon[37580]: pgmap v3938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6501.891693 secs 2024-07-26T14:39:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6501.891693 secs 2024-07-26T14:39:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6501.891693 secs 2024-07-26T14:39:27.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:27.385+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:27.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:27.385+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:27.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:27.385+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:27 smithi112 ceph-mon[37501]: pgmap v3939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:27 smithi107 ceph-mon[32659]: pgmap v3939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:27 smithi132 ceph-mon[37580]: pgmap v3939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:29.560 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:29.560+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:29.560+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:29.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:29.560+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:29 smithi112 ceph-mon[37501]: pgmap v3940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:29 smithi107 ceph-mon[32659]: pgmap v3940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:29 smithi132 ceph-mon[37580]: pgmap v3940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6506.891812 secs 2024-07-26T14:39:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:31 smithi112 ceph-mon[37501]: pgmap v3941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6506.891812 secs 2024-07-26T14:39:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:31 smithi107 ceph-mon[32659]: pgmap v3941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6506.891812 secs 2024-07-26T14:39:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:31 smithi132 ceph-mon[37580]: pgmap v3941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:33 smithi112 ceph-mon[37501]: pgmap v3942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:33 smithi107 ceph-mon[32659]: pgmap v3942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:33 smithi132 ceph-mon[37580]: pgmap v3942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:35 smithi112 ceph-mon[37501]: pgmap v3943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:36.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:35 smithi107 ceph-mon[32659]: pgmap v3943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:35 smithi132 ceph-mon[37580]: pgmap v3943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6511.891959 secs 2024-07-26T14:39:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6511.891959 secs 2024-07-26T14:39:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6511.891959 secs 2024-07-26T14:39:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:37 smithi112 ceph-mon[37501]: pgmap v3944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:37 smithi107 ceph-mon[32659]: pgmap v3944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:37 smithi132 ceph-mon[37580]: pgmap v3944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:39 smithi107 ceph-mon[32659]: pgmap v3945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:39 smithi132 ceph-mon[37580]: pgmap v3945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:39 smithi112 ceph-mon[37501]: pgmap v3945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:42.002 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6516.892102 secs 2024-07-26T14:39:42.002 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:41 smithi112 ceph-mon[37501]: pgmap v3946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6516.892102 secs 2024-07-26T14:39:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:41 smithi107 ceph-mon[32659]: pgmap v3946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6516.892102 secs 2024-07-26T14:39:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:41 smithi132 ceph-mon[37580]: pgmap v3946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:42.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:42.396+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:42.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:42.396+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:42.397 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:42.396+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:43 smithi107 ceph-mon[32659]: pgmap v3947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:43 smithi132 ceph-mon[37580]: pgmap v3947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:43 smithi112 ceph-mon[37501]: pgmap v3947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:44.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:44.575+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:44.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:44.575+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:44.576 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:44.575+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:39:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:39:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:39:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:45 smithi107 ceph-mon[32659]: pgmap v3948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:46.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:39:46.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:39:46.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:39:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:45 smithi132 ceph-mon[37580]: pgmap v3948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:39:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:39:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:39:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:45 smithi112 ceph-mon[37501]: pgmap v3948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:39:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:39:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:39:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:39:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6521.892212 secs 2024-07-26T14:39:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:46 smithi107 ceph-mon[32659]: pgmap v3949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:39:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:46 smithi107 ceph-mon[32659]: pgmap v3950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:39:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6521.892212 secs 2024-07-26T14:39:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:46 smithi132 ceph-mon[37580]: pgmap v3949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:39:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:46 smithi132 ceph-mon[37580]: pgmap v3950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:39:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6521.892212 secs 2024-07-26T14:39:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:46 smithi112 ceph-mon[37501]: pgmap v3949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:39:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:46 smithi112 ceph-mon[37501]: pgmap v3950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:39:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:48 smithi107 ceph-mon[32659]: pgmap v3951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:39:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:48 smithi132 ceph-mon[37580]: pgmap v3951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:39:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:48 smithi112 ceph-mon[37501]: pgmap v3951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:39:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:50 smithi107 ceph-mon[32659]: pgmap v3952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:39:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:50 smithi132 ceph-mon[37580]: pgmap v3952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:39:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:50 smithi112 ceph-mon[37501]: pgmap v3952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:39:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6526.892326 secs 2024-07-26T14:39:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6526.892326 secs 2024-07-26T14:39:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6526.892326 secs 2024-07-26T14:39:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:52 smithi107 ceph-mon[32659]: pgmap v3953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:39:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:52 smithi132 ceph-mon[37580]: pgmap v3953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:39:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:52 smithi112 ceph-mon[37501]: pgmap v3953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:39:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:54 smithi107 ceph-mon[32659]: pgmap v3954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:39:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:54 smithi132 ceph-mon[37580]: pgmap v3954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:39:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:54 smithi112 ceph-mon[37501]: pgmap v3954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:39:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6531.892394 secs 2024-07-26T14:39:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:56 smithi107 ceph-mon[32659]: pgmap v3955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:39:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6531.892394 secs 2024-07-26T14:39:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:56 smithi132 ceph-mon[37580]: pgmap v3955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:39:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6531.892394 secs 2024-07-26T14:39:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:56 smithi112 ceph-mon[37501]: pgmap v3955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:39:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:57.401+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:57.401+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:57.403 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:39:57.401+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:39:58 smithi107 ceph-mon[32659]: pgmap v3956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:39:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:39:58 smithi132 ceph-mon[37580]: pgmap v3956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:39:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:39:58 smithi112 ceph-mon[37501]: pgmap v3956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:39:59.586 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:59.585+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:59.586 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:59.585+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:39:59.586 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:39:59.585+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:00 smithi132 ceph-mon[37580]: pgmap v3957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:40:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:00 smithi112 ceph-mon[37501]: pgmap v3957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:40:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:00 smithi107 ceph-mon[32659]: pgmap v3957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:40:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6536.892487 secs 2024-07-26T14:40:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6536.892487 secs 2024-07-26T14:40:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6536.892487 secs 2024-07-26T14:40:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:02 smithi112 ceph-mon[37501]: pgmap v3958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:40:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:02 smithi107 ceph-mon[32659]: pgmap v3958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:40:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:02 smithi132 ceph-mon[37580]: pgmap v3958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:40:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:04 smithi112 ceph-mon[37501]: pgmap v3959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:40:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:04 smithi107 ceph-mon[32659]: pgmap v3959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:40:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:04 smithi132 ceph-mon[37580]: pgmap v3959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:40:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6541.892604 secs 2024-07-26T14:40:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:06 smithi112 ceph-mon[37501]: pgmap v3960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6541.892604 secs 2024-07-26T14:40:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:06 smithi107 ceph-mon[32659]: pgmap v3960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6541.892604 secs 2024-07-26T14:40:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:06 smithi132 ceph-mon[37580]: pgmap v3960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:08 smithi112 ceph-mon[37501]: pgmap v3961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:08 smithi107 ceph-mon[32659]: pgmap v3961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:08 smithi132 ceph-mon[37580]: pgmap v3961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:10 smithi112 ceph-mon[37501]: pgmap v3962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:11.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:10 smithi107 ceph-mon[32659]: pgmap v3962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:10 smithi132 ceph-mon[37580]: pgmap v3962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:12.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6546.892743 secs 2024-07-26T14:40:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6546.892743 secs 2024-07-26T14:40:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6546.892743 secs 2024-07-26T14:40:12.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:12.416+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:12.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:12.416+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:12.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:12.416+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:12 smithi112 ceph-mon[37501]: pgmap v3963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:12 smithi107 ceph-mon[32659]: pgmap v3963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:12 smithi132 ceph-mon[37580]: pgmap v3963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:14.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:14.591+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:14.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:14.591+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:14.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:14.591+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:15.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:14 smithi132 ceph-mon[37580]: pgmap v3964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:14 smithi112 ceph-mon[37501]: pgmap v3964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:14 smithi107 ceph-mon[32659]: pgmap v3964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6551.892877 secs 2024-07-26T14:40:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:16 smithi112 ceph-mon[37501]: pgmap v3965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6551.892877 secs 2024-07-26T14:40:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:16 smithi107 ceph-mon[32659]: pgmap v3965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6551.892877 secs 2024-07-26T14:40:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:16 smithi132 ceph-mon[37580]: pgmap v3965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:18 smithi112 ceph-mon[37501]: pgmap v3966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:18 smithi107 ceph-mon[32659]: pgmap v3966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:18 smithi132 ceph-mon[37580]: pgmap v3966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:20 smithi112 ceph-mon[37501]: pgmap v3967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:20 smithi107 ceph-mon[32659]: pgmap v3967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:20 smithi132 ceph-mon[37580]: pgmap v3967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6556.892994 secs 2024-07-26T14:40:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6556.892994 secs 2024-07-26T14:40:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6556.892994 secs 2024-07-26T14:40:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:22 smithi112 ceph-mon[37501]: pgmap v3968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:22 smithi107 ceph-mon[32659]: pgmap v3968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:22 smithi132 ceph-mon[37580]: pgmap v3968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:24 smithi112 ceph-mon[37501]: pgmap v3969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:24 smithi107 ceph-mon[32659]: pgmap v3969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:24 smithi132 ceph-mon[37580]: pgmap v3969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6561.893143 secs 2024-07-26T14:40:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:26 smithi112 ceph-mon[37501]: pgmap v3970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6561.893143 secs 2024-07-26T14:40:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:26 smithi107 ceph-mon[32659]: pgmap v3970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6561.893143 secs 2024-07-26T14:40:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:26 smithi132 ceph-mon[37580]: pgmap v3970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:27.432+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:27.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:27.432+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:27.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:27.432+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:28 smithi107 ceph-mon[32659]: pgmap v3971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:28 smithi112 ceph-mon[37501]: pgmap v3971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:28 smithi132 ceph-mon[37580]: pgmap v3971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:29.607+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:29.607+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:29.607+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:31.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:30 smithi107 ceph-mon[32659]: pgmap v3972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:30 smithi112 ceph-mon[37501]: pgmap v3972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:30 smithi132 ceph-mon[37580]: pgmap v3972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6566.893259 secs 2024-07-26T14:40:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6566.893259 secs 2024-07-26T14:40:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6566.893259 secs 2024-07-26T14:40:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:32 smithi112 ceph-mon[37501]: pgmap v3973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:32 smithi107 ceph-mon[32659]: pgmap v3973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:32 smithi132 ceph-mon[37580]: pgmap v3973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:34 smithi112 ceph-mon[37501]: pgmap v3974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:34 smithi107 ceph-mon[32659]: pgmap v3974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:34 smithi132 ceph-mon[37580]: pgmap v3974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6571.893344 secs 2024-07-26T14:40:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:36 smithi112 ceph-mon[37501]: pgmap v3975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6571.893344 secs 2024-07-26T14:40:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:36 smithi107 ceph-mon[32659]: pgmap v3975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6571.893344 secs 2024-07-26T14:40:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:36 smithi132 ceph-mon[37580]: pgmap v3975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:38 smithi112 ceph-mon[37501]: pgmap v3976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:38 smithi107 ceph-mon[32659]: pgmap v3976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:38 smithi132 ceph-mon[37580]: pgmap v3976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:40:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:40 smithi112 ceph-mon[37501]: pgmap v3977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:40 smithi107 ceph-mon[32659]: pgmap v3977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:40 smithi132 ceph-mon[37580]: pgmap v3977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:40:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6576.893467 secs 2024-07-26T14:40:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6576.893467 secs 2024-07-26T14:40:42.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:42.448+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:42.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:42.448+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:42.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:42.448+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6576.893467 secs 2024-07-26T14:40:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:42 smithi107 ceph-mon[32659]: pgmap v3978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:42 smithi132 ceph-mon[37580]: pgmap v3978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:42 smithi112 ceph-mon[37501]: pgmap v3978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:44.622+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:44.622+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:44.623 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:44.622+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:44 smithi107 ceph-mon[32659]: pgmap v3979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:44 smithi132 ceph-mon[37580]: pgmap v3979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:44 smithi112 ceph-mon[37501]: pgmap v3979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:40:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:40:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:40:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6581.893585 secs 2024-07-26T14:40:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:46 smithi107 ceph-mon[32659]: pgmap v3980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:40:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:40:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:40:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6581.893585 secs 2024-07-26T14:40:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:46 smithi132 ceph-mon[37580]: pgmap v3980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:40:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:40:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:40:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6581.893585 secs 2024-07-26T14:40:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:46 smithi112 ceph-mon[37501]: pgmap v3980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:40:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:40:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:40:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:47 smithi107 ceph-mon[32659]: pgmap v3981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:40:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:47 smithi132 ceph-mon[37580]: pgmap v3981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:40:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:47 smithi112 ceph-mon[37501]: pgmap v3981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:40:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:49 smithi107 ceph-mon[32659]: pgmap v3982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:40:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:49 smithi132 ceph-mon[37580]: pgmap v3982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:40:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:49 smithi112 ceph-mon[37501]: pgmap v3982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:40:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6586.893701 secs 2024-07-26T14:40:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:51 smithi107 ceph-mon[32659]: pgmap v3983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:40:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6586.893701 secs 2024-07-26T14:40:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:51 smithi132 ceph-mon[37580]: pgmap v3983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:40:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6586.893701 secs 2024-07-26T14:40:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:51 smithi112 ceph-mon[37501]: pgmap v3983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:40:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:54 smithi107 ceph-mon[32659]: pgmap v3984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:40:54.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:54 smithi132 ceph-mon[37580]: pgmap v3984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:40:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:54 smithi112 ceph-mon[37501]: pgmap v3984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:40:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:56 smithi107 ceph-mon[32659]: pgmap v3985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:40:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:56 smithi132 ceph-mon[37580]: pgmap v3985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:40:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:56 smithi112 ceph-mon[37501]: pgmap v3985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:40:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6591.893850 secs 2024-07-26T14:40:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6591.893850 secs 2024-07-26T14:40:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6591.893850 secs 2024-07-26T14:40:57.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:57.456+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:57.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:57.456+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:57.458 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:40:57.456+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:40:58 smithi107 ceph-mon[32659]: pgmap v3986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:40:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:40:58 smithi132 ceph-mon[37580]: pgmap v3986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:40:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:40:58 smithi112 ceph-mon[37501]: pgmap v3986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:40:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:59.637+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:59.637+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:40:59.639 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:40:59.637+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:00 smithi107 ceph-mon[32659]: pgmap v3987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:00 smithi132 ceph-mon[37580]: pgmap v3987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:00 smithi112 ceph-mon[37501]: pgmap v3987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6596.893945 secs 2024-07-26T14:41:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:02 smithi107 ceph-mon[32659]: pgmap v3988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6596.893945 secs 2024-07-26T14:41:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:02 smithi132 ceph-mon[37580]: pgmap v3988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6596.893945 secs 2024-07-26T14:41:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:02 smithi112 ceph-mon[37501]: pgmap v3988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:04 smithi107 ceph-mon[32659]: pgmap v3989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:04 smithi132 ceph-mon[37580]: pgmap v3989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:04 smithi112 ceph-mon[37501]: pgmap v3989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:06 smithi132 ceph-mon[37580]: pgmap v3990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:41:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:06 smithi112 ceph-mon[37501]: pgmap v3990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:41:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:06 smithi107 ceph-mon[32659]: pgmap v3990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:41:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6601.894097 secs 2024-07-26T14:41:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6601.894097 secs 2024-07-26T14:41:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6601.894097 secs 2024-07-26T14:41:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:08 smithi112 ceph-mon[37501]: pgmap v3991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:41:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:08 smithi107 ceph-mon[32659]: pgmap v3991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:41:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:08 smithi132 ceph-mon[37580]: pgmap v3991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:41:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:10 smithi112 ceph-mon[37501]: pgmap v3992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:10 smithi107 ceph-mon[32659]: pgmap v3992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:10 smithi132 ceph-mon[37580]: pgmap v3992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6606.894141 secs 2024-07-26T14:41:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:12 smithi112 ceph-mon[37501]: pgmap v3993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:12.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:12.457+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:12.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:12.457+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:12.459 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:12.457+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:12.460 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6606.894141 secs 2024-07-26T14:41:12.460 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:12 smithi107 ceph-mon[32659]: pgmap v3993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6606.894141 secs 2024-07-26T14:41:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:12 smithi132 ceph-mon[37580]: pgmap v3993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:14 smithi112 ceph-mon[37501]: pgmap v3994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:14 smithi107 ceph-mon[32659]: pgmap v3994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:14 smithi132 ceph-mon[37580]: pgmap v3994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:14.653+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:14.653+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:14.654 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:14.653+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:16.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:16 smithi132 ceph-mon[37580]: pgmap v3995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:41:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:16 smithi112 ceph-mon[37501]: pgmap v3995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:41:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:16 smithi107 ceph-mon[32659]: pgmap v3995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:41:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6611.894264 secs 2024-07-26T14:41:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6611.894264 secs 2024-07-26T14:41:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6611.894264 secs 2024-07-26T14:41:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:18 smithi112 ceph-mon[37501]: pgmap v3996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:41:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:18 smithi107 ceph-mon[32659]: pgmap v3996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:41:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:18 smithi132 ceph-mon[37580]: pgmap v3996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:41:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:20 smithi112 ceph-mon[37501]: pgmap v3997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:41:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:20 smithi107 ceph-mon[32659]: pgmap v3997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:41:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:20 smithi132 ceph-mon[37580]: pgmap v3997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:41:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6616.894389 secs 2024-07-26T14:41:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:22 smithi112 ceph-mon[37501]: pgmap v3998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:41:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6616.894389 secs 2024-07-26T14:41:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:22 smithi107 ceph-mon[32659]: pgmap v3998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:41:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6616.894389 secs 2024-07-26T14:41:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:22 smithi132 ceph-mon[37580]: pgmap v3998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:41:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:24 smithi112 ceph-mon[37501]: pgmap v3999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:41:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:24 smithi107 ceph-mon[32659]: pgmap v3999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:41:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:24 smithi132 ceph-mon[37580]: pgmap v3999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:41:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:26 smithi112 ceph-mon[37501]: pgmap v4000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:41:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:26 smithi107 ceph-mon[32659]: pgmap v4000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:41:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:26 smithi132 ceph-mon[37580]: pgmap v4000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:41:27.443 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6621.894511 secs 2024-07-26T14:41:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6621.894511 secs 2024-07-26T14:41:27.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:27.461+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:27.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:27.461+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:27.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:27.461+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6621.894511 secs 2024-07-26T14:41:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:28 smithi112 ceph-mon[37501]: pgmap v4001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:41:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:28 smithi107 ceph-mon[32659]: pgmap v4001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:41:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:28 smithi132 ceph-mon[37580]: pgmap v4001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:41:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:29.669+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:29.669+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:29.669+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:30 smithi112 ceph-mon[37501]: pgmap v4002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:30 smithi107 ceph-mon[32659]: pgmap v4002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:30 smithi132 ceph-mon[37580]: pgmap v4002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6626.894608 secs 2024-07-26T14:41:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6626.894608 secs 2024-07-26T14:41:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6626.894608 secs 2024-07-26T14:41:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:32 smithi112 ceph-mon[37501]: pgmap v4003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:32 smithi107 ceph-mon[32659]: pgmap v4003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:32 smithi132 ceph-mon[37580]: pgmap v4003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:34 smithi112 ceph-mon[37501]: pgmap v4004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:34 smithi107 ceph-mon[32659]: pgmap v4004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:34 smithi132 ceph-mon[37580]: pgmap v4004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:36 smithi112 ceph-mon[37501]: pgmap v4005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:41:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6631.894723 secs 2024-07-26T14:41:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:36 smithi107 ceph-mon[32659]: pgmap v4005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:41:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6631.894723 secs 2024-07-26T14:41:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:36 smithi132 ceph-mon[37580]: pgmap v4005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:41:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6631.894723 secs 2024-07-26T14:41:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:38 smithi112 ceph-mon[37501]: pgmap v4006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:41:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:38 smithi107 ceph-mon[32659]: pgmap v4006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:41:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:38 smithi132 ceph-mon[37580]: pgmap v4006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:41:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:40 smithi112 ceph-mon[37501]: pgmap v4007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:40 smithi107 ceph-mon[32659]: pgmap v4007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:40 smithi132 ceph-mon[37580]: pgmap v4007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:41:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6636.894836 secs 2024-07-26T14:41:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6636.894836 secs 2024-07-26T14:41:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6636.894836 secs 2024-07-26T14:41:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:42 smithi112 ceph-mon[37501]: pgmap v4008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:42.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:42.469+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:42.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:42.469+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:42.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:42.469+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:42.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:42 smithi107 ceph-mon[32659]: pgmap v4008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:42 smithi132 ceph-mon[37580]: pgmap v4008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:44 smithi112 ceph-mon[37501]: pgmap v4009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:41:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:44 smithi107 ceph-mon[32659]: pgmap v4009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:41:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:44 smithi132 ceph-mon[37580]: pgmap v4009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:41:44.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:44.685+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:44.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:44.685+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:44.686 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:44.685+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:46 smithi107 ceph-mon[32659]: pgmap v4010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:41:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6641.894956 secs 2024-07-26T14:41:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:46 smithi132 ceph-mon[37580]: pgmap v4010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:41:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6641.894956 secs 2024-07-26T14:41:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:46 smithi112 ceph-mon[37501]: pgmap v4010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:41:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6641.894956 secs 2024-07-26T14:41:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:41:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:41:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:41:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:48 smithi107 ceph-mon[32659]: pgmap v4011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:41:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:48 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:41:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:48 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:41:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:48 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:41:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:48 smithi132 ceph-mon[37580]: pgmap v4011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:41:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:48 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:41:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:48 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:41:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:48 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:41:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:48 smithi112 ceph-mon[37501]: pgmap v4011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:41:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:48 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:41:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:48 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:41:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:48 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:41:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:49 smithi107 ceph-mon[32659]: pgmap v4012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:41:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:49 smithi107 ceph-mon[32659]: pgmap v4013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:41:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:49 smithi132 ceph-mon[37580]: pgmap v4012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:41:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:49 smithi132 ceph-mon[37580]: pgmap v4013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:41:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:49 smithi112 ceph-mon[37501]: pgmap v4012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:41:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:49 smithi112 ceph-mon[37501]: pgmap v4013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:41:51.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:51 smithi107 ceph-mon[32659]: pgmap v4014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:41:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6646.895094 secs 2024-07-26T14:41:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:51 smithi132 ceph-mon[37580]: pgmap v4014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:41:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6646.895094 secs 2024-07-26T14:41:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:51 smithi112 ceph-mon[37501]: pgmap v4014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:41:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6646.895094 secs 2024-07-26T14:41:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:53 smithi107 ceph-mon[32659]: pgmap v4015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:53 smithi132 ceph-mon[37580]: pgmap v4015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:53 smithi112 ceph-mon[37501]: pgmap v4015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:55 smithi107 ceph-mon[32659]: pgmap v4016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:41:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:55 smithi132 ceph-mon[37580]: pgmap v4016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:41:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:55 smithi112 ceph-mon[37501]: pgmap v4016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:41:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6651.895212 secs 2024-07-26T14:41:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6651.895212 secs 2024-07-26T14:41:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6651.895212 secs 2024-07-26T14:41:57.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:57.474+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:57.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:57.474+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:57.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:41:57.474+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:57 smithi107 ceph-mon[32659]: pgmap v4017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:41:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:57 smithi132 ceph-mon[37580]: pgmap v4017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:41:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:57 smithi112 ceph-mon[37501]: pgmap v4017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:41:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:41:59 smithi107 ceph-mon[32659]: pgmap v4018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:41:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:41:59 smithi132 ceph-mon[37580]: pgmap v4018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:41:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:41:59 smithi112 ceph-mon[37501]: pgmap v4018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:41:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:59.700+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:59.700+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:41:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:41:59.700+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:01 smithi107 ceph-mon[32659]: pgmap v4019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6656.895314 secs 2024-07-26T14:42:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:01 smithi132 ceph-mon[37580]: pgmap v4019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6656.895314 secs 2024-07-26T14:42:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:01 smithi112 ceph-mon[37501]: pgmap v4019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6656.895314 secs 2024-07-26T14:42:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:03 smithi107 ceph-mon[32659]: pgmap v4020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:03 smithi132 ceph-mon[37580]: pgmap v4020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:03 smithi112 ceph-mon[37501]: pgmap v4020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:05 smithi107 ceph-mon[32659]: pgmap v4021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:05 smithi132 ceph-mon[37580]: pgmap v4021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:05 smithi112 ceph-mon[37501]: pgmap v4021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6661.895436 secs 2024-07-26T14:42:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6661.895436 secs 2024-07-26T14:42:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6661.895436 secs 2024-07-26T14:42:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:07 smithi132 ceph-mon[37580]: pgmap v4022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:07 smithi112 ceph-mon[37501]: pgmap v4022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:07.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:07 smithi107 ceph-mon[32659]: pgmap v4022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:09 smithi132 ceph-mon[37580]: pgmap v4023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:09 smithi112 ceph-mon[37501]: pgmap v4023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:09 smithi107 ceph-mon[32659]: pgmap v4023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:11 smithi112 ceph-mon[37501]: pgmap v4024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6666.895555 secs 2024-07-26T14:42:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:11 smithi107 ceph-mon[32659]: pgmap v4024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6666.895555 secs 2024-07-26T14:42:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:11 smithi132 ceph-mon[37580]: pgmap v4024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6666.895555 secs 2024-07-26T14:42:12.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:12.490+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:12.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:12.490+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:12.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:12.490+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:13 smithi112 ceph-mon[37501]: pgmap v4025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:13.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:13.765+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:42:13.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:13 smithi107 ceph-mon[32659]: pgmap v4025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:13.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:13.765+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:42:13.782 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:13.781+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:42:13.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:13.782+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:42:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:13 smithi132 ceph-mon[37580]: pgmap v4025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:14.713+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:14.713+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:14.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:14.713+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:14.937 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:14.937+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:42:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:14.937+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:42:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:14.937+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:42:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:14.937+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:42:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:15 smithi112 ceph-mon[37501]: pgmap v4026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:15 smithi107 ceph-mon[32659]: pgmap v4026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:15 smithi132 ceph-mon[37580]: pgmap v4026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6671.895671 secs 2024-07-26T14:42:16.710 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6671.895671 secs 2024-07-26T14:42:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6671.895671 secs 2024-07-26T14:42:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:17 smithi112 ceph-mon[37501]: pgmap v4027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:17 smithi107 ceph-mon[32659]: pgmap v4027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:17 smithi132 ceph-mon[37580]: pgmap v4027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:18 smithi112 ceph-mon[37501]: pgmap v4028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:18 smithi107 ceph-mon[32659]: pgmap v4028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:18.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:18 smithi132 ceph-mon[37580]: pgmap v4028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:20 smithi112 ceph-mon[37501]: pgmap v4029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:21.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:20 smithi107 ceph-mon[32659]: pgmap v4029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:20 smithi132 ceph-mon[37580]: pgmap v4029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6676.895794 secs 2024-07-26T14:42:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6676.895794 secs 2024-07-26T14:42:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6676.895794 secs 2024-07-26T14:42:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:22 smithi107 ceph-mon[32659]: pgmap v4030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:22 smithi132 ceph-mon[37580]: pgmap v4030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:22 smithi112 ceph-mon[37501]: pgmap v4030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:24 smithi107 ceph-mon[32659]: pgmap v4031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:24 smithi132 ceph-mon[37580]: pgmap v4031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:24 smithi112 ceph-mon[37501]: pgmap v4031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6681.895912 secs 2024-07-26T14:42:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:26 smithi107 ceph-mon[32659]: pgmap v4032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6681.895912 secs 2024-07-26T14:42:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:26 smithi132 ceph-mon[37580]: pgmap v4032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6681.895912 secs 2024-07-26T14:42:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:26 smithi112 ceph-mon[37501]: pgmap v4032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:27.506+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:27.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:27.506+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:27.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:27.506+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:28 smithi107 ceph-mon[32659]: pgmap v4033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:28 smithi132 ceph-mon[37580]: pgmap v4033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:28 smithi112 ceph-mon[37501]: pgmap v4033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:29.725+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:29.725+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:29.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:29.725+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:30 smithi107 ceph-mon[32659]: pgmap v4034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:30 smithi132 ceph-mon[37580]: pgmap v4034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:30 smithi112 ceph-mon[37501]: pgmap v4034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6686.896040 secs 2024-07-26T14:42:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6686.896040 secs 2024-07-26T14:42:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6686.896040 secs 2024-07-26T14:42:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:32 smithi107 ceph-mon[32659]: pgmap v4035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:32 smithi132 ceph-mon[37580]: pgmap v4035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:32 smithi112 ceph-mon[37501]: pgmap v4035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:34 smithi107 ceph-mon[32659]: pgmap v4036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:35.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:34 smithi132 ceph-mon[37580]: pgmap v4036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:34 smithi112 ceph-mon[37501]: pgmap v4036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6691.896164 secs 2024-07-26T14:42:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:36 smithi107 ceph-mon[32659]: pgmap v4037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6691.896164 secs 2024-07-26T14:42:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:37 smithi132 ceph-mon[37580]: pgmap v4037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6691.896164 secs 2024-07-26T14:42:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:36 smithi112 ceph-mon[37501]: pgmap v4037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:39 smithi107 ceph-mon[32659]: pgmap v4038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:39 smithi132 ceph-mon[37580]: pgmap v4038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:39 smithi112 ceph-mon[37501]: pgmap v4038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:41 smithi107 ceph-mon[32659]: pgmap v4039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:41 smithi132 ceph-mon[37580]: pgmap v4039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:41 smithi112 ceph-mon[37501]: pgmap v4039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6696.896232 secs 2024-07-26T14:42:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6696.896232 secs 2024-07-26T14:42:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6696.896232 secs 2024-07-26T14:42:42.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:42.522+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:42.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:42.522+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:42.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:42.522+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:43 smithi107 ceph-mon[32659]: pgmap v4040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:43 smithi132 ceph-mon[37580]: pgmap v4040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:43.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:43 smithi112 ceph-mon[37501]: pgmap v4040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:44.741 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:44.741+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:44.741+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:44.742 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:44.741+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:45 smithi107 ceph-mon[32659]: pgmap v4041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:45 smithi132 ceph-mon[37580]: pgmap v4041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:45 smithi112 ceph-mon[37501]: pgmap v4041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6701.896350 secs 2024-07-26T14:42:47.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:47 smithi107 ceph-mon[32659]: pgmap v4042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6701.896350 secs 2024-07-26T14:42:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:47 smithi132 ceph-mon[37580]: pgmap v4042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6701.896350 secs 2024-07-26T14:42:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:47 smithi112 ceph-mon[37501]: pgmap v4042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:49 smithi132 ceph-mon[37580]: pgmap v4043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:42:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:49 smithi112 ceph-mon[37501]: pgmap v4043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:42:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:49 smithi107 ceph-mon[32659]: pgmap v4043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:42:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:42:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:42:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:50 smithi132 ceph-mon[37580]: pgmap v4044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:42:50.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:50 smithi132 ceph-mon[37580]: pgmap v4045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:50.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:42:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:42:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:42:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:50 smithi112 ceph-mon[37501]: pgmap v4044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:42:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:50 smithi112 ceph-mon[37501]: pgmap v4045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:42:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:42:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:42:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:50 smithi107 ceph-mon[32659]: pgmap v4044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:42:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:50 smithi107 ceph-mon[32659]: pgmap v4045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:42:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6706.896472 secs 2024-07-26T14:42:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:52 smithi112 ceph-mon[37501]: pgmap v4046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:42:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6706.896472 secs 2024-07-26T14:42:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:52 smithi107 ceph-mon[32659]: pgmap v4046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:42:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6706.896472 secs 2024-07-26T14:42:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:52 smithi132 ceph-mon[37580]: pgmap v4046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:42:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:54 smithi112 ceph-mon[37501]: pgmap v4047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:54 smithi107 ceph-mon[32659]: pgmap v4047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:54 smithi132 ceph-mon[37580]: pgmap v4047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:56 smithi112 ceph-mon[37501]: pgmap v4048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6711.896612 secs 2024-07-26T14:42:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:56 smithi107 ceph-mon[32659]: pgmap v4048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6711.896612 secs 2024-07-26T14:42:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:56 smithi132 ceph-mon[37580]: pgmap v4048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6711.896612 secs 2024-07-26T14:42:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:57.530+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:57.530+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:57.532 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:42:57.530+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:42:58 smithi112 ceph-mon[37501]: pgmap v4049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:42:58 smithi107 ceph-mon[32659]: pgmap v4049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:42:58 smithi132 ceph-mon[37580]: pgmap v4049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:59.755+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:59.755+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:42:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:42:59.755+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:00 smithi112 ceph-mon[37501]: pgmap v4050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:43:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:00 smithi107 ceph-mon[32659]: pgmap v4050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:43:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:00 smithi132 ceph-mon[37580]: pgmap v4050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:43:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6716.896754 secs 2024-07-26T14:43:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6716.896754 secs 2024-07-26T14:43:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6716.896754 secs 2024-07-26T14:43:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:02 smithi112 ceph-mon[37501]: pgmap v4051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:02 smithi107 ceph-mon[32659]: pgmap v4051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:02 smithi132 ceph-mon[37580]: pgmap v4051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:04 smithi112 ceph-mon[37501]: pgmap v4052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:04 smithi107 ceph-mon[32659]: pgmap v4052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:04 smithi132 ceph-mon[37580]: pgmap v4052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:06 smithi112 ceph-mon[37501]: pgmap v4053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6721.896888 secs 2024-07-26T14:43:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:06 smithi107 ceph-mon[32659]: pgmap v4053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6721.896888 secs 2024-07-26T14:43:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:06 smithi132 ceph-mon[37580]: pgmap v4053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6721.896888 secs 2024-07-26T14:43:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:08 smithi112 ceph-mon[37501]: pgmap v4054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:08 smithi107 ceph-mon[32659]: pgmap v4054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:08 smithi132 ceph-mon[37580]: pgmap v4054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:10 smithi112 ceph-mon[37501]: pgmap v4055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:10 smithi107 ceph-mon[32659]: pgmap v4055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:10 smithi132 ceph-mon[37580]: pgmap v4055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6726.897008 secs 2024-07-26T14:43:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6726.897008 secs 2024-07-26T14:43:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6726.897008 secs 2024-07-26T14:43:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:12 smithi107 ceph-mon[32659]: pgmap v4056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:12.546+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:12.546+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:12.548 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:12.546+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:12 smithi132 ceph-mon[37580]: pgmap v4056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:12 smithi112 ceph-mon[37501]: pgmap v4056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:14.501 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:14 smithi112 ceph-mon[37501]: pgmap v4057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:14.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:14 smithi107 ceph-mon[32659]: pgmap v4057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:14 smithi132 ceph-mon[37580]: pgmap v4057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:14.771+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:14.771+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:14.771+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:16 smithi107 ceph-mon[32659]: pgmap v4058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6731.897166 secs 2024-07-26T14:43:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:16 smithi132 ceph-mon[37580]: pgmap v4058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6731.897166 secs 2024-07-26T14:43:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:16 smithi112 ceph-mon[37501]: pgmap v4058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6731.897166 secs 2024-07-26T14:43:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:18 smithi107 ceph-mon[32659]: pgmap v4059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:18 smithi132 ceph-mon[37580]: pgmap v4059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:18 smithi112 ceph-mon[37501]: pgmap v4059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:20 smithi107 ceph-mon[32659]: pgmap v4060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:20 smithi132 ceph-mon[37580]: pgmap v4060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:20 smithi112 ceph-mon[37501]: pgmap v4060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6736.897231 secs 2024-07-26T14:43:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6736.897231 secs 2024-07-26T14:43:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6736.897231 secs 2024-07-26T14:43:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:22 smithi107 ceph-mon[32659]: pgmap v4061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:22 smithi132 ceph-mon[37580]: pgmap v4061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:22 smithi112 ceph-mon[37501]: pgmap v4061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:24 smithi107 ceph-mon[32659]: pgmap v4062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:24 smithi132 ceph-mon[37580]: pgmap v4062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:24 smithi112 ceph-mon[37501]: pgmap v4062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:26 smithi107 ceph-mon[32659]: pgmap v4063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6741.897348 secs 2024-07-26T14:43:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:26 smithi132 ceph-mon[37580]: pgmap v4063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6741.897348 secs 2024-07-26T14:43:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:26 smithi112 ceph-mon[37501]: pgmap v4063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6741.897348 secs 2024-07-26T14:43:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:27.562+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:27.563 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:27.562+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:27.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:27.562+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:28 smithi107 ceph-mon[32659]: pgmap v4064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:28 smithi132 ceph-mon[37580]: pgmap v4064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:28 smithi112 ceph-mon[37501]: pgmap v4064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:29.788 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:29.787+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:29.788 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:29.787+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:29.789 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:29.787+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:30 smithi107 ceph-mon[32659]: pgmap v4065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:30 smithi132 ceph-mon[37580]: pgmap v4065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:30 smithi112 ceph-mon[37501]: pgmap v4065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6746.897444 secs 2024-07-26T14:43:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6746.897444 secs 2024-07-26T14:43:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6746.897444 secs 2024-07-26T14:43:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:32 smithi107 ceph-mon[32659]: pgmap v4066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:32 smithi132 ceph-mon[37580]: pgmap v4066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:32 smithi112 ceph-mon[37501]: pgmap v4066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:34 smithi107 ceph-mon[32659]: pgmap v4067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:34 smithi132 ceph-mon[37580]: pgmap v4067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:34 smithi112 ceph-mon[37501]: pgmap v4067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:36 smithi132 ceph-mon[37580]: pgmap v4068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6751.897563 secs 2024-07-26T14:43:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:36 smithi112 ceph-mon[37501]: pgmap v4068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6751.897563 secs 2024-07-26T14:43:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:36 smithi107 ceph-mon[32659]: pgmap v4068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6751.897563 secs 2024-07-26T14:43:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:38 smithi132 ceph-mon[37580]: pgmap v4069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:38 smithi112 ceph-mon[37501]: pgmap v4069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:38 smithi107 ceph-mon[32659]: pgmap v4069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:40 smithi112 ceph-mon[37501]: pgmap v4070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:40 smithi107 ceph-mon[32659]: pgmap v4070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:40 smithi132 ceph-mon[37580]: pgmap v4070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6756.897681 secs 2024-07-26T14:43:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6756.897681 secs 2024-07-26T14:43:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6756.897681 secs 2024-07-26T14:43:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:42.578+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:42.578+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:42.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:42.578+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:42 smithi112 ceph-mon[37501]: pgmap v4071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:42.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:42 smithi107 ceph-mon[32659]: pgmap v4071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:42 smithi132 ceph-mon[37580]: pgmap v4071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:44 smithi112 ceph-mon[37501]: pgmap v4072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:44 smithi107 ceph-mon[32659]: pgmap v4072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:44.802+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:44.802+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:44.802+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:44 smithi132 ceph-mon[37580]: pgmap v4072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:46 smithi112 ceph-mon[37501]: pgmap v4073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6761.897825 secs 2024-07-26T14:43:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:46 smithi107 ceph-mon[32659]: pgmap v4073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6761.897825 secs 2024-07-26T14:43:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:46 smithi132 ceph-mon[37580]: pgmap v4073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6761.897825 secs 2024-07-26T14:43:48.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:48 smithi132 ceph-mon[37580]: pgmap v4074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:48 smithi112 ceph-mon[37501]: pgmap v4074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:48 smithi107 ceph-mon[32659]: pgmap v4074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:43:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:43:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:43:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:50 smithi112 ceph-mon[37501]: pgmap v4075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:43:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:43:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:50 smithi112 ceph-mon[37501]: pgmap v4076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:43:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:50 smithi112 ceph-mon[37501]: pgmap v4077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:43:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:50 smithi107 ceph-mon[32659]: pgmap v4075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:43:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:43:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:50 smithi107 ceph-mon[32659]: pgmap v4076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:43:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:50 smithi107 ceph-mon[32659]: pgmap v4077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:43:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:50 smithi132 ceph-mon[37580]: pgmap v4075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:43:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:43:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:50 smithi132 ceph-mon[37580]: pgmap v4076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:43:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:50 smithi132 ceph-mon[37580]: pgmap v4077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:43:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6766.897948 secs 2024-07-26T14:43:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6766.897948 secs 2024-07-26T14:43:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6766.897948 secs 2024-07-26T14:43:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:52 smithi112 ceph-mon[37501]: pgmap v4078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:43:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:52 smithi107 ceph-mon[32659]: pgmap v4078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:43:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:52 smithi132 ceph-mon[37580]: pgmap v4078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:43:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:55 smithi132 ceph-mon[37580]: pgmap v4079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:55 smithi112 ceph-mon[37501]: pgmap v4079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:55 smithi107 ceph-mon[32659]: pgmap v4079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6771.898091 secs 2024-07-26T14:43:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6771.898091 secs 2024-07-26T14:43:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6771.898091 secs 2024-07-26T14:43:57.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:57.585+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:57.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:57.585+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:57.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:43:57.585+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:57.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:57 smithi107 ceph-mon[32659]: pgmap v4080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:43:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:57 smithi112 ceph-mon[37501]: pgmap v4080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:43:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:57 smithi132 ceph-mon[37580]: pgmap v4080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:43:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:43:59 smithi112 ceph-mon[37501]: pgmap v4081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:43:59 smithi107 ceph-mon[32659]: pgmap v4081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:43:59 smithi132 ceph-mon[37580]: pgmap v4081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:59.817+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:59.817+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:43:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:43:59.817+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6776.898220 secs 2024-07-26T14:44:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:01 smithi112 ceph-mon[37501]: pgmap v4082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:44:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6776.898220 secs 2024-07-26T14:44:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:01 smithi107 ceph-mon[32659]: pgmap v4082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:44:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6776.898220 secs 2024-07-26T14:44:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:01 smithi132 ceph-mon[37580]: pgmap v4082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:44:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:03 smithi112 ceph-mon[37501]: pgmap v4083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:03 smithi107 ceph-mon[32659]: pgmap v4083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:03 smithi132 ceph-mon[37580]: pgmap v4083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:05 smithi112 ceph-mon[37501]: pgmap v4084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:05 smithi107 ceph-mon[32659]: pgmap v4084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:05 smithi132 ceph-mon[37580]: pgmap v4084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6781.898347 secs 2024-07-26T14:44:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6781.898347 secs 2024-07-26T14:44:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6781.898347 secs 2024-07-26T14:44:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:07 smithi112 ceph-mon[37501]: pgmap v4085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:07 smithi107 ceph-mon[32659]: pgmap v4085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:07 smithi132 ceph-mon[37580]: pgmap v4085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:08 smithi112 ceph-mon[37501]: pgmap v4086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:08 smithi107 ceph-mon[32659]: pgmap v4086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:08 smithi132 ceph-mon[37580]: pgmap v4086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6786.898474 secs 2024-07-26T14:44:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:11 smithi112 ceph-mon[37501]: pgmap v4087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6786.898474 secs 2024-07-26T14:44:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:11 smithi107 ceph-mon[32659]: pgmap v4087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6786.898474 secs 2024-07-26T14:44:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:11 smithi132 ceph-mon[37580]: pgmap v4087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:12.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:12.601+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:12.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:12.601+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:12.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:12.601+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:13 smithi112 ceph-mon[37501]: pgmap v4088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:13 smithi107 ceph-mon[32659]: pgmap v4088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:13 smithi132 ceph-mon[37580]: pgmap v4088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:14.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:14.833+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:14.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:14.833+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:14.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:14.833+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:15 smithi112 ceph-mon[37501]: pgmap v4089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:15 smithi107 ceph-mon[32659]: pgmap v4089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:15 smithi132 ceph-mon[37580]: pgmap v4089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6791.898581 secs 2024-07-26T14:44:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6791.898581 secs 2024-07-26T14:44:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6791.898581 secs 2024-07-26T14:44:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:17 smithi112 ceph-mon[37501]: pgmap v4090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:17 smithi107 ceph-mon[32659]: pgmap v4090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:17 smithi132 ceph-mon[37580]: pgmap v4090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:19.611 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:19 smithi132 ceph-mon[37580]: pgmap v4091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:19 smithi112 ceph-mon[37501]: pgmap v4091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:19 smithi107 ceph-mon[32659]: pgmap v4091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6796.898746 secs 2024-07-26T14:44:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:21 smithi112 ceph-mon[37501]: pgmap v4092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6796.898746 secs 2024-07-26T14:44:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:21 smithi107 ceph-mon[32659]: pgmap v4092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6796.898746 secs 2024-07-26T14:44:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:21 smithi132 ceph-mon[37580]: pgmap v4092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:23 smithi112 ceph-mon[37501]: pgmap v4093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:23 smithi107 ceph-mon[32659]: pgmap v4093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:23 smithi132 ceph-mon[37580]: pgmap v4093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:24 smithi112 ceph-mon[37501]: pgmap v4094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:44:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:24 smithi107 ceph-mon[32659]: pgmap v4094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:44:24.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:24 smithi132 ceph-mon[37580]: pgmap v4094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:44:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6801.898887 secs 2024-07-26T14:44:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6801.898887 secs 2024-07-26T14:44:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6801.898887 secs 2024-07-26T14:44:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:27 smithi107 ceph-mon[32659]: pgmap v4095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:27 smithi132 ceph-mon[37580]: pgmap v4095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:27.618 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:27.617+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:27.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:27.617+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:27.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:27.617+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:27 smithi112 ceph-mon[37501]: pgmap v4095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:29 smithi112 ceph-mon[37501]: pgmap v4096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:29 smithi107 ceph-mon[32659]: pgmap v4096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:29 smithi132 ceph-mon[37580]: pgmap v4096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:29.842+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:29.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:29.842+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:29.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:29.842+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6806.899012 secs 2024-07-26T14:44:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:31 smithi112 ceph-mon[37501]: pgmap v4097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6806.899012 secs 2024-07-26T14:44:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:31 smithi107 ceph-mon[32659]: pgmap v4097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6806.899012 secs 2024-07-26T14:44:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:31 smithi132 ceph-mon[37580]: pgmap v4097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:33 smithi112 ceph-mon[37501]: pgmap v4098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:33 smithi107 ceph-mon[32659]: pgmap v4098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:33 smithi132 ceph-mon[37580]: pgmap v4098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:35 smithi112 ceph-mon[37501]: pgmap v4099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:44:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:35 smithi107 ceph-mon[32659]: pgmap v4099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:44:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:35 smithi132 ceph-mon[37580]: pgmap v4099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:44:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6811.899137 secs 2024-07-26T14:44:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6811.899137 secs 2024-07-26T14:44:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6811.899137 secs 2024-07-26T14:44:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:37 smithi112 ceph-mon[37501]: pgmap v4100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:37 smithi107 ceph-mon[32659]: pgmap v4100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:37 smithi132 ceph-mon[37580]: pgmap v4100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:38 smithi112 ceph-mon[37501]: pgmap v4101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:38 smithi107 ceph-mon[32659]: pgmap v4101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:38 smithi132 ceph-mon[37580]: pgmap v4101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6816.899260 secs 2024-07-26T14:44:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:41 smithi112 ceph-mon[37501]: pgmap v4102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6816.899260 secs 2024-07-26T14:44:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:41 smithi107 ceph-mon[32659]: pgmap v4102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6816.899260 secs 2024-07-26T14:44:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:41 smithi132 ceph-mon[37580]: pgmap v4102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:42.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:42.633+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:42.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:42.633+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:42.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:42.633+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:43 smithi112 ceph-mon[37501]: pgmap v4103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:43 smithi107 ceph-mon[32659]: pgmap v4103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:43 smithi132 ceph-mon[37580]: pgmap v4103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:44.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:44.858+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:44.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:44.858+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:44.859 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:44.858+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:45 smithi112 ceph-mon[37501]: pgmap v4104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:44:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:45 smithi107 ceph-mon[32659]: pgmap v4104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:44:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:45 smithi132 ceph-mon[37580]: pgmap v4104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:44:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6821.899383 secs 2024-07-26T14:44:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6821.899383 secs 2024-07-26T14:44:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6821.899383 secs 2024-07-26T14:44:47.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:47 smithi112 ceph-mon[37501]: pgmap v4105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:47 smithi107 ceph-mon[32659]: pgmap v4105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:47 smithi132 ceph-mon[37580]: pgmap v4105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:49 smithi112 ceph-mon[37501]: pgmap v4106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:49.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:49 smithi107 ceph-mon[32659]: pgmap v4106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:49 smithi132 ceph-mon[37580]: pgmap v4106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:50 smithi112 ceph-mon[37501]: pgmap v4107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:50 smithi107 ceph-mon[32659]: pgmap v4107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:50 smithi132 ceph-mon[37580]: pgmap v4107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:51.591 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6826.899504 secs 2024-07-26T14:44:51.591 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:44:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6826.899504 secs 2024-07-26T14:44:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:44:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6826.899504 secs 2024-07-26T14:44:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:44:52.426 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:52 smithi107 ceph-mon[32659]: pgmap v4108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:52.639 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:52 smithi112 ceph-mon[37501]: pgmap v4108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:52 smithi132 ceph-mon[37580]: pgmap v4108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:55 smithi112 ceph-mon[37501]: pgmap v4109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:44:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:55 smithi107 ceph-mon[32659]: pgmap v4109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:44:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:55 smithi132 ceph-mon[37580]: pgmap v4109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:44:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6831.899650 secs 2024-07-26T14:44:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6831.899650 secs 2024-07-26T14:44:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6831.899650 secs 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: pgmap v4110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:44:57.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:44:57.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: pgmap v4110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:44:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:44:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:57.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:57.633+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:57.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:57.633+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:57.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:44:57.633+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: pgmap v4110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:44:57.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:44:57.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:44:57.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:44:58.422 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:58 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:44:58.422 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:58 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:44:58.422 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:58 smithi107 ceph-mon[32659]: pgmap v4111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:44:58.422 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:44:58 smithi107 ceph-mon[32659]: pgmap v4112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T14:44:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:58 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:44:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:58 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:44:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:58 smithi132 ceph-mon[37580]: pgmap v4111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:44:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:44:58 smithi132 ceph-mon[37580]: pgmap v4112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T14:44:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:58 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:44:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:58 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:44:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:58 smithi112 ceph-mon[37501]: pgmap v4111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:44:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:44:58 smithi112 ceph-mon[37501]: pgmap v4112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T14:44:59.861 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:59.859+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:44:59.862 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:59.859+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:59.862 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:44:59.859+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:00 smithi107 ceph-mon[32659]: pgmap v4113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:45:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:00 smithi132 ceph-mon[37580]: pgmap v4113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:45:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:00 smithi112 ceph-mon[37501]: pgmap v4113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:45:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6836.899718 secs 2024-07-26T14:45:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6836.899718 secs 2024-07-26T14:45:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6836.899718 secs 2024-07-26T14:45:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:02 smithi107 ceph-mon[32659]: pgmap v4114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T14:45:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:02 smithi132 ceph-mon[37580]: pgmap v4114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T14:45:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:02 smithi112 ceph-mon[37501]: pgmap v4114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T14:45:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:04 smithi112 ceph-mon[37501]: pgmap v4115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:45:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:04 smithi107 ceph-mon[32659]: pgmap v4115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:45:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:04 smithi132 ceph-mon[37580]: pgmap v4115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:45:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:06 smithi107 ceph-mon[32659]: pgmap v4116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T14:45:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6841.899842 secs 2024-07-26T14:45:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:06 smithi132 ceph-mon[37580]: pgmap v4116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T14:45:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6841.899842 secs 2024-07-26T14:45:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:06 smithi112 ceph-mon[37501]: pgmap v4116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T14:45:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6841.899842 secs 2024-07-26T14:45:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:08 smithi107 ceph-mon[32659]: pgmap v4117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:45:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:08 smithi132 ceph-mon[37580]: pgmap v4117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:45:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:08 smithi112 ceph-mon[37501]: pgmap v4117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:45:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:10 smithi107 ceph-mon[32659]: pgmap v4118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:10 smithi132 ceph-mon[37580]: pgmap v4118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:10 smithi112 ceph-mon[37501]: pgmap v4118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6846.899966 secs 2024-07-26T14:45:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6846.899966 secs 2024-07-26T14:45:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6846.899966 secs 2024-07-26T14:45:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:12 smithi107 ceph-mon[32659]: pgmap v4119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:12 smithi132 ceph-mon[37580]: pgmap v4119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:12.648+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:12.648+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:12.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:12.648+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:12 smithi112 ceph-mon[37501]: pgmap v4119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:13.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:13.793+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:45:13.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:13.794+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:45:13.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:13.811+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:45:13.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:13.812+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:45:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:14 smithi107 ceph-mon[32659]: pgmap v4120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:14 smithi132 ceph-mon[37580]: pgmap v4120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:14 smithi112 ceph-mon[37501]: pgmap v4120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:14.869+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:14.870+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:14.870+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:14.970 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:14.969+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:45:14.970 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:14.969+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:45:14.970 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:14.969+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:45:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:14.969+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:45:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:15 smithi112 ceph-mon[37501]: pgmap v4121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:15 smithi107 ceph-mon[32659]: pgmap v4121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:15 smithi132 ceph-mon[37580]: pgmap v4121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6851.900109 secs 2024-07-26T14:45:17.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6851.900109 secs 2024-07-26T14:45:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6851.900109 secs 2024-07-26T14:45:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:17 smithi112 ceph-mon[37501]: pgmap v4122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:17 smithi107 ceph-mon[32659]: pgmap v4122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:17 smithi132 ceph-mon[37580]: pgmap v4122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:19 smithi112 ceph-mon[37501]: pgmap v4123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:19 smithi107 ceph-mon[32659]: pgmap v4123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:19 smithi132 ceph-mon[37580]: pgmap v4123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6856.900236 secs 2024-07-26T14:45:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:21 smithi112 ceph-mon[37501]: pgmap v4124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6856.900236 secs 2024-07-26T14:45:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:21 smithi107 ceph-mon[32659]: pgmap v4124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6856.900236 secs 2024-07-26T14:45:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:21 smithi132 ceph-mon[37580]: pgmap v4124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:23 smithi112 ceph-mon[37501]: pgmap v4125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:23 smithi107 ceph-mon[32659]: pgmap v4125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:23 smithi132 ceph-mon[37580]: pgmap v4125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:25 smithi112 ceph-mon[37501]: pgmap v4126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:25 smithi107 ceph-mon[32659]: pgmap v4126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:25 smithi132 ceph-mon[37580]: pgmap v4126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6861.900303 secs 2024-07-26T14:45:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6861.900303 secs 2024-07-26T14:45:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6861.900303 secs 2024-07-26T14:45:27.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:27.665+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:27.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:27.665+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:27.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:27.665+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:27 smithi112 ceph-mon[37501]: pgmap v4127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:27 smithi107 ceph-mon[32659]: pgmap v4127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:27 smithi132 ceph-mon[37580]: pgmap v4127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:29.886+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:29.886+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:29.886+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:29 smithi112 ceph-mon[37501]: pgmap v4128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:29 smithi107 ceph-mon[32659]: pgmap v4128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:29 smithi132 ceph-mon[37580]: pgmap v4128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6866.900424 secs 2024-07-26T14:45:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:31 smithi112 ceph-mon[37501]: pgmap v4129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6866.900424 secs 2024-07-26T14:45:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:31 smithi107 ceph-mon[32659]: pgmap v4129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6866.900424 secs 2024-07-26T14:45:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:31 smithi132 ceph-mon[37580]: pgmap v4129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:33 smithi107 ceph-mon[32659]: pgmap v4130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:33 smithi132 ceph-mon[37580]: pgmap v4130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:33 smithi112 ceph-mon[37501]: pgmap v4130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:35 smithi107 ceph-mon[32659]: pgmap v4131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:35 smithi132 ceph-mon[37580]: pgmap v4131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:35 smithi112 ceph-mon[37501]: pgmap v4131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6871.900560 secs 2024-07-26T14:45:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6871.900560 secs 2024-07-26T14:45:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6871.900560 secs 2024-07-26T14:45:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:37 smithi107 ceph-mon[32659]: pgmap v4132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:45:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:37 smithi132 ceph-mon[37580]: pgmap v4132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:45:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:37 smithi112 ceph-mon[37501]: pgmap v4132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:45:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:39 smithi107 ceph-mon[32659]: pgmap v4133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:45:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:39 smithi132 ceph-mon[37580]: pgmap v4133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:45:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:39 smithi112 ceph-mon[37501]: pgmap v4133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:45:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6876.900705 secs 2024-07-26T14:45:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:41 smithi107 ceph-mon[32659]: pgmap v4134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:45:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6876.900705 secs 2024-07-26T14:45:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:41 smithi132 ceph-mon[37580]: pgmap v4134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:45:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6876.900705 secs 2024-07-26T14:45:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:41 smithi112 ceph-mon[37501]: pgmap v4134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:45:42.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:42.680+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:42.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:42.680+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:42.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:42.680+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:43 smithi107 ceph-mon[32659]: pgmap v4135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:45:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:43 smithi132 ceph-mon[37580]: pgmap v4135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:45:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:43 smithi112 ceph-mon[37501]: pgmap v4135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:45:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:44.896+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:44.896+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:44.897 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:44.896+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:46.252 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:45 smithi112 ceph-mon[37501]: pgmap v4136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:45 smithi107 ceph-mon[32659]: pgmap v4136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:45 smithi132 ceph-mon[37580]: pgmap v4136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6881.900856 secs 2024-07-26T14:45:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6881.900856 secs 2024-07-26T14:45:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6881.900856 secs 2024-07-26T14:45:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:48 smithi107 ceph-mon[32659]: pgmap v4137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:48 smithi132 ceph-mon[37580]: pgmap v4137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:48 smithi112 ceph-mon[37501]: pgmap v4137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:50 smithi107 ceph-mon[32659]: pgmap v4138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:50 smithi132 ceph-mon[37580]: pgmap v4138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:50 smithi112 ceph-mon[37501]: pgmap v4138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6886.901000 secs 2024-07-26T14:45:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:52 smithi107 ceph-mon[32659]: pgmap v4139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6886.901000 secs 2024-07-26T14:45:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:52 smithi132 ceph-mon[37580]: pgmap v4139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6886.901000 secs 2024-07-26T14:45:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:52 smithi112 ceph-mon[37501]: pgmap v4139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:54 smithi107 ceph-mon[32659]: pgmap v4140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:54.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:54 smithi132 ceph-mon[37580]: pgmap v4140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:54 smithi112 ceph-mon[37501]: pgmap v4140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:45:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:56 smithi107 ceph-mon[32659]: pgmap v4141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:56 smithi132 ceph-mon[37580]: pgmap v4141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:56 smithi112 ceph-mon[37501]: pgmap v4141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6891.901131 secs 2024-07-26T14:45:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6891.901131 secs 2024-07-26T14:45:57.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6891.901131 secs 2024-07-26T14:45:57.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:57.694+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:57.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:57.694+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:57.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:45:57.694+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:58.224 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:58 smithi132 ceph-mon[37580]: pgmap v4142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:58.225 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:45:58.225 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.225 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.225 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.225 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.225 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.225 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.225 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.225 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:58 smithi112 ceph-mon[37501]: pgmap v4142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:45:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:58 smithi107 ceph-mon[32659]: pgmap v4142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:45:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:58.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:59 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:45:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:45:59 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:45:59.297 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:59 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:45:59.297 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:45:59 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:45:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:59 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:45:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:45:59 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:45:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:59.905+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:59.905+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:45:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:45:59.905+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:00.393 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:00 smithi132 ceph-mon[37580]: pgmap v4143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:00 smithi112 ceph-mon[37501]: pgmap v4143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:00 smithi107 ceph-mon[32659]: pgmap v4143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6896.901197 secs 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: pgmap v4144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:46:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:46:02.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:46:02.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:46:02.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: pgmap v4145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:46:02.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: pgmap v4146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:46:02.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:46:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6896.901197 secs 2024-07-26T14:46:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: pgmap v4144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:46:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:46:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:46:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: pgmap v4145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:46:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: pgmap v4146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:46:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:46:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6896.901197 secs 2024-07-26T14:46:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: pgmap v4144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:46:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:46:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:46:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:46:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:46:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:46:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: pgmap v4145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:46:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: pgmap v4146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:46:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:46:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:03 smithi107 ceph-mon[32659]: pgmap v4147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:46:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:03 smithi132 ceph-mon[37580]: pgmap v4147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:46:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:03 smithi112 ceph-mon[37501]: pgmap v4147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:46:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:06 smithi107 ceph-mon[32659]: pgmap v4148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T14:46:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:06 smithi132 ceph-mon[37580]: pgmap v4148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T14:46:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:06 smithi112 ceph-mon[37501]: pgmap v4148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T14:46:07.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6901.901284 secs 2024-07-26T14:46:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6901.901284 secs 2024-07-26T14:46:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6901.901284 secs 2024-07-26T14:46:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:08 smithi107 ceph-mon[32659]: pgmap v4149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T14:46:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:08 smithi132 ceph-mon[37580]: pgmap v4149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T14:46:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:08 smithi112 ceph-mon[37501]: pgmap v4149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T14:46:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:10 smithi107 ceph-mon[32659]: pgmap v4150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:46:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:10 smithi132 ceph-mon[37580]: pgmap v4150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:46:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:10 smithi112 ceph-mon[37501]: pgmap v4150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:46:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6906.901351 secs 2024-07-26T14:46:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:12 smithi107 ceph-mon[32659]: pgmap v4151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:46:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6906.901351 secs 2024-07-26T14:46:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:12 smithi132 ceph-mon[37580]: pgmap v4151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:46:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6906.901351 secs 2024-07-26T14:46:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:12 smithi112 ceph-mon[37501]: pgmap v4151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:46:12.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:12.707+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:12.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:12.707+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:12.709 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:12.707+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:14 smithi107 ceph-mon[32659]: pgmap v4152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:14 smithi132 ceph-mon[37580]: pgmap v4152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:14 smithi112 ceph-mon[37501]: pgmap v4152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:14.907+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:14.907+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:14.907+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:16 smithi107 ceph-mon[32659]: pgmap v4153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:16 smithi132 ceph-mon[37580]: pgmap v4153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:16 smithi112 ceph-mon[37501]: pgmap v4153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6911.901473 secs 2024-07-26T14:46:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6911.901473 secs 2024-07-26T14:46:17.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6911.901473 secs 2024-07-26T14:46:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:18 smithi132 ceph-mon[37580]: pgmap v4154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:18.440 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:18 smithi112 ceph-mon[37501]: pgmap v4154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:18 smithi107 ceph-mon[32659]: pgmap v4154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:20 smithi112 ceph-mon[37501]: pgmap v4155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:20 smithi107 ceph-mon[32659]: pgmap v4155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:20.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:20 smithi132 ceph-mon[37580]: pgmap v4155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6916.901551 secs 2024-07-26T14:46:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:22 smithi112 ceph-mon[37501]: pgmap v4156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6916.901551 secs 2024-07-26T14:46:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:22 smithi107 ceph-mon[32659]: pgmap v4156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6916.901551 secs 2024-07-26T14:46:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:22 smithi132 ceph-mon[37580]: pgmap v4156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:24 smithi112 ceph-mon[37501]: pgmap v4157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:24 smithi107 ceph-mon[32659]: pgmap v4157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:24 smithi132 ceph-mon[37580]: pgmap v4157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:26 smithi112 ceph-mon[37501]: pgmap v4158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:26 smithi107 ceph-mon[32659]: pgmap v4158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:26 smithi132 ceph-mon[37580]: pgmap v4158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6921.901651 secs 2024-07-26T14:46:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6921.901651 secs 2024-07-26T14:46:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6921.901651 secs 2024-07-26T14:46:27.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:27.724+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:27.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:27.724+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:27.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:27.724+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:28 smithi112 ceph-mon[37501]: pgmap v4159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:28 smithi107 ceph-mon[32659]: pgmap v4159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:28 smithi132 ceph-mon[37580]: pgmap v4159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:29.924+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:29.924+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:29.924+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:30 smithi112 ceph-mon[37501]: pgmap v4160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:30 smithi107 ceph-mon[32659]: pgmap v4160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:30 smithi132 ceph-mon[37580]: pgmap v4160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:32 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6926.901716 secs 2024-07-26T14:46:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:32 smithi112 ceph-mon[37501]: pgmap v4161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:32 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6926.901716 secs 2024-07-26T14:46:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:32 smithi107 ceph-mon[32659]: pgmap v4161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:32 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6926.901716 secs 2024-07-26T14:46:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:32 smithi132 ceph-mon[37580]: pgmap v4161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:34 smithi112 ceph-mon[37501]: pgmap v4162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:34 smithi107 ceph-mon[32659]: pgmap v4162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:34 smithi132 ceph-mon[37580]: pgmap v4162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:36 smithi112 ceph-mon[37501]: pgmap v4163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:36 smithi107 ceph-mon[32659]: pgmap v4163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:36 smithi132 ceph-mon[37580]: pgmap v4163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6931.901817 secs 2024-07-26T14:46:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6931.901817 secs 2024-07-26T14:46:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6931.901817 secs 2024-07-26T14:46:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:38 smithi112 ceph-mon[37501]: pgmap v4164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:38 smithi107 ceph-mon[32659]: pgmap v4164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:38 smithi132 ceph-mon[37580]: pgmap v4164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:40 smithi112 ceph-mon[37501]: pgmap v4165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:40 smithi107 ceph-mon[32659]: pgmap v4165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:40 smithi132 ceph-mon[37580]: pgmap v4165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6936.901950 secs 2024-07-26T14:46:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:42 smithi112 ceph-mon[37501]: pgmap v4166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6936.901950 secs 2024-07-26T14:46:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:42 smithi107 ceph-mon[32659]: pgmap v4166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6936.901950 secs 2024-07-26T14:46:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:42 smithi132 ceph-mon[37580]: pgmap v4166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:42.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:42.737+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:42.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:42.737+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:42.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:42.737+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:42.809+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:46:42.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:42.810+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:46:42.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:42.826+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:46:42.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:42.827+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:46:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:44 smithi112 ceph-mon[37501]: pgmap v4167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:44 smithi107 ceph-mon[32659]: pgmap v4167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:44 smithi132 ceph-mon[37580]: pgmap v4167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:44.939+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:44.939+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:44.940 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:44.939+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:44.985+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:46:44.986 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:44.985+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:46:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:44.986+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:46:44.987 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:44.986+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:46:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:46 smithi107 ceph-mon[32659]: pgmap v4168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:46 smithi112 ceph-mon[37501]: pgmap v4168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:46.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:46 smithi132 ceph-mon[37580]: pgmap v4168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6941.902091 secs 2024-07-26T14:46:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6941.902091 secs 2024-07-26T14:46:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6941.902091 secs 2024-07-26T14:46:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:48 smithi112 ceph-mon[37501]: pgmap v4169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:48 smithi107 ceph-mon[32659]: pgmap v4169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:48 smithi132 ceph-mon[37580]: pgmap v4169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:50 smithi112 ceph-mon[37501]: pgmap v4170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:50 smithi107 ceph-mon[32659]: pgmap v4170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:50 smithi132 ceph-mon[37580]: pgmap v4170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6946.902211 secs 2024-07-26T14:46:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6946.902211 secs 2024-07-26T14:46:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6946.902211 secs 2024-07-26T14:46:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:52 smithi107 ceph-mon[32659]: pgmap v4171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:52 smithi132 ceph-mon[37580]: pgmap v4171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:52 smithi112 ceph-mon[37501]: pgmap v4171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:54 smithi107 ceph-mon[32659]: pgmap v4172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:46:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:54 smithi132 ceph-mon[37580]: pgmap v4172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:46:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:54 smithi112 ceph-mon[37501]: pgmap v4172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:46:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:56 smithi107 ceph-mon[32659]: pgmap v4173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6951.902298 secs 2024-07-26T14:46:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:56 smithi132 ceph-mon[37580]: pgmap v4173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6951.902298 secs 2024-07-26T14:46:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:56 smithi112 ceph-mon[37501]: pgmap v4173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6951.902298 secs 2024-07-26T14:46:57.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:57.741+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:57.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:57.741+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:57.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:46:57.741+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:46:58 smithi107 ceph-mon[32659]: pgmap v4174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:46:58 smithi132 ceph-mon[37580]: pgmap v4174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:46:58 smithi112 ceph-mon[37501]: pgmap v4174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:59.951+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:59.951+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:46:59.953 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:46:59.951+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:00 smithi107 ceph-mon[32659]: pgmap v4175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:47:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:00 smithi132 ceph-mon[37580]: pgmap v4175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:47:00.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:00 smithi112 ceph-mon[37501]: pgmap v4175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:47:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6956.902397 secs 2024-07-26T14:47:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:47:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6956.902397 secs 2024-07-26T14:47:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:47:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6956.902397 secs 2024-07-26T14:47:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:47:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:02 smithi107 ceph-mon[32659]: pgmap v4176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:47:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:47:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:47:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:47:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:02 smithi132 ceph-mon[37580]: pgmap v4176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:47:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:47:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:47:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:47:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:02 smithi112 ceph-mon[37501]: pgmap v4176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:47:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:47:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:47:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:47:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:03 smithi107 ceph-mon[32659]: pgmap v4177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:47:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:03 smithi107 ceph-mon[32659]: pgmap v4178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:03 smithi132 ceph-mon[37580]: pgmap v4177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:47:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:03 smithi132 ceph-mon[37580]: pgmap v4178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:03 smithi112 ceph-mon[37501]: pgmap v4177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:47:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:03 smithi112 ceph-mon[37501]: pgmap v4178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:05 smithi107 ceph-mon[32659]: pgmap v4179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:05 smithi132 ceph-mon[37580]: pgmap v4179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:05 smithi112 ceph-mon[37501]: pgmap v4179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6961.902515 secs 2024-07-26T14:47:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6961.902515 secs 2024-07-26T14:47:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6961.902515 secs 2024-07-26T14:47:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:07 smithi107 ceph-mon[32659]: pgmap v4180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:47:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:07 smithi132 ceph-mon[37580]: pgmap v4180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:47:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:07 smithi112 ceph-mon[37501]: pgmap v4180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:47:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:09 smithi132 ceph-mon[37580]: pgmap v4181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:09 smithi112 ceph-mon[37501]: pgmap v4181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:09 smithi107 ceph-mon[32659]: pgmap v4181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:11 smithi112 ceph-mon[37501]: pgmap v4182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6966.902637 secs 2024-07-26T14:47:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:11 smithi107 ceph-mon[32659]: pgmap v4182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6966.902637 secs 2024-07-26T14:47:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:11 smithi132 ceph-mon[37580]: pgmap v4182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6966.902637 secs 2024-07-26T14:47:12.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:12.746+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:12.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:12.746+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:12.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:12.746+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:13 smithi112 ceph-mon[37501]: pgmap v4183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:47:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:13 smithi107 ceph-mon[32659]: pgmap v4183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:47:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:13 smithi132 ceph-mon[37580]: pgmap v4183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:47:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:47:14.959+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:47:14.959+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:47:14.959+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:15 smithi112 ceph-mon[37501]: pgmap v4184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:15 smithi107 ceph-mon[32659]: pgmap v4184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:15 smithi132 ceph-mon[37580]: pgmap v4184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6971.902774 secs 2024-07-26T14:47:16.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6971.902774 secs 2024-07-26T14:47:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6971.902774 secs 2024-07-26T14:47:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:17 smithi112 ceph-mon[37501]: pgmap v4185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:17 smithi107 ceph-mon[32659]: pgmap v4185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:17 smithi132 ceph-mon[37580]: pgmap v4185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:19 smithi112 ceph-mon[37501]: pgmap v4186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:19 smithi107 ceph-mon[32659]: pgmap v4186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:19 smithi132 ceph-mon[37580]: pgmap v4186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:21.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:21 smithi132 ceph-mon[37580]: pgmap v4187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:21.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6976.902871 secs 2024-07-26T14:47:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:21 smithi112 ceph-mon[37501]: pgmap v4187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6976.902871 secs 2024-07-26T14:47:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:21 smithi107 ceph-mon[32659]: pgmap v4187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6976.902871 secs 2024-07-26T14:47:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:23 smithi112 ceph-mon[37501]: pgmap v4188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:23 smithi107 ceph-mon[32659]: pgmap v4188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:23.808 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:23 smithi132 ceph-mon[37580]: pgmap v4188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:25 smithi112 ceph-mon[37501]: pgmap v4189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:25 smithi107 ceph-mon[32659]: pgmap v4189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:25 smithi132 ceph-mon[37580]: pgmap v4189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6981.902999 secs 2024-07-26T14:47:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6981.902999 secs 2024-07-26T14:47:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6981.902999 secs 2024-07-26T14:47:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:27 smithi112 ceph-mon[37501]: pgmap v4190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:27.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:27.762+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:27.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:27.762+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:27.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:27.762+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:27.764 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:27 smithi107 ceph-mon[32659]: pgmap v4190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:27 smithi132 ceph-mon[37580]: pgmap v4190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:29 smithi112 ceph-mon[37501]: pgmap v4191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:29 smithi107 ceph-mon[32659]: pgmap v4191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:29 smithi132 ceph-mon[37580]: pgmap v4191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:47:29.975+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:47:29.975+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:29.976 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:47:29.975+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:31 smithi112 ceph-mon[37501]: pgmap v4192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6986.903123 secs 2024-07-26T14:47:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:31 smithi107 ceph-mon[32659]: pgmap v4192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6986.903123 secs 2024-07-26T14:47:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:31 smithi132 ceph-mon[37580]: pgmap v4192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6986.903123 secs 2024-07-26T14:47:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:33 smithi112 ceph-mon[37501]: pgmap v4193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:33 smithi107 ceph-mon[32659]: pgmap v4193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:33 smithi132 ceph-mon[37580]: pgmap v4193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:34 smithi107 ceph-mon[32659]: pgmap v4194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:34 smithi132 ceph-mon[37580]: pgmap v4194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:34 smithi112 ceph-mon[37501]: pgmap v4194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6991.903192 secs 2024-07-26T14:47:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6991.903192 secs 2024-07-26T14:47:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6991.903192 secs 2024-07-26T14:47:37.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:37 smithi107 ceph-mon[32659]: pgmap v4195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:37 smithi132 ceph-mon[37580]: pgmap v4195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:37 smithi112 ceph-mon[37501]: pgmap v4195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:39 smithi107 ceph-mon[32659]: pgmap v4196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:39 smithi132 ceph-mon[37580]: pgmap v4196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:39 smithi112 ceph-mon[37501]: pgmap v4196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:41 smithi107 ceph-mon[32659]: pgmap v4197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 6996.903311 secs 2024-07-26T14:47:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:41 smithi132 ceph-mon[37580]: pgmap v4197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 6996.903311 secs 2024-07-26T14:47:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:41 smithi112 ceph-mon[37501]: pgmap v4197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 6996.903311 secs 2024-07-26T14:47:42.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:42.772+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:42.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:42.772+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:42.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:42.772+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:43 smithi107 ceph-mon[32659]: pgmap v4198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:43 smithi132 ceph-mon[37580]: pgmap v4198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:43 smithi112 ceph-mon[37501]: pgmap v4198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:47:44.991+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:47:44.991+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:47:44.991+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:45 smithi107 ceph-mon[32659]: pgmap v4199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:45 smithi132 ceph-mon[37580]: pgmap v4199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:45 smithi112 ceph-mon[37501]: pgmap v4199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7001.903466 secs 2024-07-26T14:47:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7001.903466 secs 2024-07-26T14:47:46.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7001.903466 secs 2024-07-26T14:47:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:47 smithi107 ceph-mon[32659]: pgmap v4200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:47 smithi132 ceph-mon[37580]: pgmap v4200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:47 smithi112 ceph-mon[37501]: pgmap v4200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:49 smithi107 ceph-mon[32659]: pgmap v4201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:49 smithi132 ceph-mon[37580]: pgmap v4201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:49 smithi112 ceph-mon[37501]: pgmap v4201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:51 smithi132 ceph-mon[37580]: pgmap v4202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7006.903599 secs 2024-07-26T14:47:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:51 smithi112 ceph-mon[37501]: pgmap v4202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7006.903599 secs 2024-07-26T14:47:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:51 smithi107 ceph-mon[32659]: pgmap v4202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7006.903599 secs 2024-07-26T14:47:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:53 smithi132 ceph-mon[37580]: pgmap v4203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:53 smithi112 ceph-mon[37501]: pgmap v4203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:53 smithi107 ceph-mon[32659]: pgmap v4203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:55 smithi132 ceph-mon[37580]: pgmap v4204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:55 smithi112 ceph-mon[37501]: pgmap v4204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:55 smithi107 ceph-mon[32659]: pgmap v4204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7011.903722 secs 2024-07-26T14:47:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7011.903722 secs 2024-07-26T14:47:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7011.903722 secs 2024-07-26T14:47:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:57 smithi112 ceph-mon[37501]: pgmap v4205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:57.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:57.781+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:57.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:57.781+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:57.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:47:57.781+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:47:57.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:57 smithi107 ceph-mon[32659]: pgmap v4205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:57 smithi132 ceph-mon[37580]: pgmap v4205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:47:59 smithi112 ceph-mon[37501]: pgmap v4206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:47:59 smithi107 ceph-mon[32659]: pgmap v4206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:47:59 smithi132 ceph-mon[37580]: pgmap v4206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:00.007 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:00.006+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:00.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:00.006+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:00.008 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:00.006+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:01.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:01 smithi107 ceph-mon[32659]: pgmap v4207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:01.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7016.903839 secs 2024-07-26T14:48:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:01 smithi112 ceph-mon[37501]: pgmap v4207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7016.903839 secs 2024-07-26T14:48:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:01 smithi132 ceph-mon[37580]: pgmap v4207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7016.903839 secs 2024-07-26T14:48:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:03 smithi112 ceph-mon[37501]: pgmap v4208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:48:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:48:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:48:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:03 smithi107 ceph-mon[32659]: pgmap v4208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:48:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:48:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:48:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:03 smithi132 ceph-mon[37580]: pgmap v4208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:48:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:48:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:48:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:48:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:48:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:48:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:48:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:48:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:48:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:48:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:48:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:48:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:05 smithi112 ceph-mon[37501]: pgmap v4209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:48:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:05 smithi112 ceph-mon[37501]: pgmap v4210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:48:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:05 smithi107 ceph-mon[32659]: pgmap v4209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:48:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:05 smithi107 ceph-mon[32659]: pgmap v4210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:48:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:05 smithi132 ceph-mon[37580]: pgmap v4209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:48:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:05 smithi132 ceph-mon[37580]: pgmap v4210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:48:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7021.903962 secs 2024-07-26T14:48:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7021.903962 secs 2024-07-26T14:48:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7021.903962 secs 2024-07-26T14:48:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:07 smithi112 ceph-mon[37501]: pgmap v4211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:48:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:07 smithi107 ceph-mon[32659]: pgmap v4211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:48:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:07 smithi132 ceph-mon[37580]: pgmap v4211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:48:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:09 smithi112 ceph-mon[37501]: pgmap v4212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:48:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:09 smithi107 ceph-mon[32659]: pgmap v4212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:48:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:09 smithi132 ceph-mon[37580]: pgmap v4212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:48:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:11 smithi112 ceph-mon[37501]: pgmap v4213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:48:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7026.904057 secs 2024-07-26T14:48:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:11 smithi107 ceph-mon[32659]: pgmap v4213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:48:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7026.904057 secs 2024-07-26T14:48:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:11 smithi132 ceph-mon[37580]: pgmap v4213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:48:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7026.904057 secs 2024-07-26T14:48:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:12.793+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:12.793+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:12.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:12.793+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:13 smithi112 ceph-mon[37501]: pgmap v4214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:48:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:13 smithi107 ceph-mon[32659]: pgmap v4214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:48:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:13 smithi132 ceph-mon[37580]: pgmap v4214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:48:13.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:13.824+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:48:13.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:13.824+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:48:13.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:13.841+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:48:13.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:13.842+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:48:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:15.001+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:48:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:15.001+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:48:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:15.002+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:48:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:15.002+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:48:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:15.021+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:15.021+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:15.021+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:15 smithi112 ceph-mon[37501]: pgmap v4215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:48:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:15 smithi107 ceph-mon[32659]: pgmap v4215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:48:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:15 smithi132 ceph-mon[37580]: pgmap v4215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:48:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7031.904089 secs 2024-07-26T14:48:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7031.904089 secs 2024-07-26T14:48:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7031.904089 secs 2024-07-26T14:48:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:17 smithi112 ceph-mon[37501]: pgmap v4216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:17 smithi107 ceph-mon[32659]: pgmap v4216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:17 smithi132 ceph-mon[37580]: pgmap v4216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:19 smithi107 ceph-mon[32659]: pgmap v4217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:19 smithi112 ceph-mon[37501]: pgmap v4217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:19 smithi132 ceph-mon[37580]: pgmap v4217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:20 smithi112 ceph-mon[37501]: pgmap v4218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:20.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:20 smithi107 ceph-mon[32659]: pgmap v4218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:20 smithi132 ceph-mon[37580]: pgmap v4218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7036.904212 secs 2024-07-26T14:48:21.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7036.904212 secs 2024-07-26T14:48:21.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7036.904212 secs 2024-07-26T14:48:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:22 smithi112 ceph-mon[37501]: pgmap v4219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:22 smithi107 ceph-mon[32659]: pgmap v4219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:22.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:22 smithi132 ceph-mon[37580]: pgmap v4219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:25 smithi132 ceph-mon[37580]: pgmap v4220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:25 smithi112 ceph-mon[37501]: pgmap v4220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:25 smithi107 ceph-mon[32659]: pgmap v4220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7041.904281 secs 2024-07-26T14:48:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:27 smithi132 ceph-mon[37580]: pgmap v4221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7041.904281 secs 2024-07-26T14:48:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:27 smithi112 ceph-mon[37501]: pgmap v4221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7041.904281 secs 2024-07-26T14:48:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:27 smithi107 ceph-mon[32659]: pgmap v4221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:27.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:27.810+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:27.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:27.810+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:27.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:27.810+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:29 smithi112 ceph-mon[37501]: pgmap v4222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:29 smithi107 ceph-mon[32659]: pgmap v4222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:29 smithi132 ceph-mon[37580]: pgmap v4222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:30.038 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:30.038+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:30.038+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:30.039 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:30.038+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:31.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:31 smithi107 ceph-mon[32659]: pgmap v4223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:31 smithi112 ceph-mon[37501]: pgmap v4223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:31 smithi132 ceph-mon[37580]: pgmap v4223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:32 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7046.904380 secs 2024-07-26T14:48:32.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:32 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7046.904380 secs 2024-07-26T14:48:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:32 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7046.904380 secs 2024-07-26T14:48:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:33 smithi112 ceph-mon[37501]: pgmap v4224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:33 smithi107 ceph-mon[32659]: pgmap v4224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:33 smithi132 ceph-mon[37580]: pgmap v4224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:35 smithi107 ceph-mon[32659]: pgmap v4225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:35 smithi112 ceph-mon[37501]: pgmap v4225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:35 smithi132 ceph-mon[37580]: pgmap v4225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7051.904535 secs 2024-07-26T14:48:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:37 smithi112 ceph-mon[37501]: pgmap v4226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7051.904535 secs 2024-07-26T14:48:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:37 smithi107 ceph-mon[32659]: pgmap v4226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7051.904535 secs 2024-07-26T14:48:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:37 smithi132 ceph-mon[37580]: pgmap v4226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:39 smithi112 ceph-mon[37501]: pgmap v4227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:39 smithi107 ceph-mon[32659]: pgmap v4227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:39 smithi132 ceph-mon[37580]: pgmap v4227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:41 smithi112 ceph-mon[37501]: pgmap v4228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:41 smithi107 ceph-mon[32659]: pgmap v4228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:41 smithi132 ceph-mon[37580]: pgmap v4228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7056.904604 secs 2024-07-26T14:48:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7056.904604 secs 2024-07-26T14:48:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7056.904604 secs 2024-07-26T14:48:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:42.816+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:42.817+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:42.817+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:43 smithi112 ceph-mon[37501]: pgmap v4229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:43 smithi107 ceph-mon[32659]: pgmap v4229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:43 smithi132 ceph-mon[37580]: pgmap v4229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:45.045+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:45.045+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:48:45.045+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:45 smithi112 ceph-mon[37501]: pgmap v4230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:45 smithi107 ceph-mon[32659]: pgmap v4230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:45 smithi132 ceph-mon[37580]: pgmap v4230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7061.904734 secs 2024-07-26T14:48:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:47 smithi112 ceph-mon[37501]: pgmap v4231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7061.904734 secs 2024-07-26T14:48:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:47 smithi107 ceph-mon[32659]: pgmap v4231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7061.904734 secs 2024-07-26T14:48:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:47 smithi132 ceph-mon[37580]: pgmap v4231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:49 smithi112 ceph-mon[37501]: pgmap v4232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:49 smithi107 ceph-mon[32659]: pgmap v4232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:49 smithi132 ceph-mon[37580]: pgmap v4232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:51 smithi112 ceph-mon[37501]: pgmap v4233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:51 smithi107 ceph-mon[32659]: pgmap v4233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:51 smithi132 ceph-mon[37580]: pgmap v4233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7066.904796 secs 2024-07-26T14:48:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7066.904796 secs 2024-07-26T14:48:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7066.904796 secs 2024-07-26T14:48:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:53 smithi112 ceph-mon[37501]: pgmap v4234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:53 smithi107 ceph-mon[32659]: pgmap v4234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:53 smithi132 ceph-mon[37580]: pgmap v4234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:48:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:55 smithi112 ceph-mon[37501]: pgmap v4235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:55 smithi107 ceph-mon[32659]: pgmap v4235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:55 smithi132 ceph-mon[37580]: pgmap v4235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7071.904879 secs 2024-07-26T14:48:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:57 smithi112 ceph-mon[37501]: pgmap v4236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7071.904879 secs 2024-07-26T14:48:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:57 smithi107 ceph-mon[32659]: pgmap v4236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7071.904879 secs 2024-07-26T14:48:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:57 smithi132 ceph-mon[37580]: pgmap v4236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:57.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:57.831+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:57.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:57.832+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:48:57.834 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:48:57.832+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:48:59 smithi112 ceph-mon[37501]: pgmap v4237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:48:59 smithi107 ceph-mon[32659]: pgmap v4237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:48:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:48:59 smithi132 ceph-mon[37580]: pgmap v4237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:00.058+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:00.058+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:00.059 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:00.058+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:01 smithi112 ceph-mon[37501]: pgmap v4238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:01 smithi107 ceph-mon[32659]: pgmap v4238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:01 smithi132 ceph-mon[37580]: pgmap v4238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7076.905006 secs 2024-07-26T14:49:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7076.905006 secs 2024-07-26T14:49:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7076.905006 secs 2024-07-26T14:49:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:03 smithi112 ceph-mon[37501]: pgmap v4239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:03 smithi107 ceph-mon[32659]: pgmap v4239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:03 smithi132 ceph-mon[37580]: pgmap v4239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:05.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:05 smithi107 ceph-mon[32659]: pgmap v4240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:05.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:49:05.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:05 smithi112 ceph-mon[37501]: pgmap v4240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:49:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:05 smithi132 ceph-mon[37580]: pgmap v4240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:49:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7081.905119 secs 2024-07-26T14:49:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:07 smithi107 ceph-mon[32659]: pgmap v4241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7081.905119 secs 2024-07-26T14:49:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:07 smithi132 ceph-mon[37580]: pgmap v4241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7081.905119 secs 2024-07-26T14:49:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:07 smithi112 ceph-mon[37501]: pgmap v4241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:08 smithi107 ceph-mon[32659]: pgmap v4242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:08 smithi132 ceph-mon[37580]: pgmap v4242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:08 smithi112 ceph-mon[37501]: pgmap v4242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:49:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:49:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:09 smithi107 ceph-mon[32659]: pgmap v4243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:49:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:09 smithi107 ceph-mon[32659]: pgmap v4244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T14:49:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:49:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:49:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:09 smithi132 ceph-mon[37580]: pgmap v4243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:49:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:09 smithi132 ceph-mon[37580]: pgmap v4244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T14:49:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:49:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:49:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:09 smithi112 ceph-mon[37501]: pgmap v4243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:49:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:09 smithi112 ceph-mon[37501]: pgmap v4244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T14:49:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:49:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7086.905259 secs 2024-07-26T14:49:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:11 smithi112 ceph-mon[37501]: pgmap v4245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:49:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7086.905259 secs 2024-07-26T14:49:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:11 smithi107 ceph-mon[32659]: pgmap v4245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:49:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7086.905259 secs 2024-07-26T14:49:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:11 smithi132 ceph-mon[37580]: pgmap v4245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:49:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:12.834+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:12.834+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:12.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:12.834+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:13 smithi112 ceph-mon[37501]: pgmap v4246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:49:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:13 smithi107 ceph-mon[32659]: pgmap v4246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:49:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:13 smithi132 ceph-mon[37580]: pgmap v4246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:49:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:15.072+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:15.074 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:15.073+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:15.075 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:15.073+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:15 smithi112 ceph-mon[37501]: pgmap v4247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:49:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:15 smithi107 ceph-mon[32659]: pgmap v4247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:49:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:15 smithi132 ceph-mon[37580]: pgmap v4247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:49:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7091.905382 secs 2024-07-26T14:49:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7091.905382 secs 2024-07-26T14:49:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7091.905382 secs 2024-07-26T14:49:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:17 smithi112 ceph-mon[37501]: pgmap v4248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:49:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:17 smithi107 ceph-mon[32659]: pgmap v4248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:49:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:17 smithi132 ceph-mon[37580]: pgmap v4248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:49:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:19 smithi112 ceph-mon[37501]: pgmap v4249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:49:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:19 smithi107 ceph-mon[32659]: pgmap v4249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:49:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:19 smithi132 ceph-mon[37580]: pgmap v4249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:49:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7096.905449 secs 2024-07-26T14:49:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:21 smithi112 ceph-mon[37501]: pgmap v4250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7096.905449 secs 2024-07-26T14:49:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:21 smithi107 ceph-mon[32659]: pgmap v4250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7096.905449 secs 2024-07-26T14:49:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:21 smithi132 ceph-mon[37580]: pgmap v4250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:23 smithi112 ceph-mon[37501]: pgmap v4251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:23 smithi107 ceph-mon[32659]: pgmap v4251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:23 smithi132 ceph-mon[37580]: pgmap v4251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:25 smithi107 ceph-mon[32659]: pgmap v4252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:25 smithi132 ceph-mon[37580]: pgmap v4252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:25 smithi112 ceph-mon[37501]: pgmap v4252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7101.905517 secs 2024-07-26T14:49:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7101.905517 secs 2024-07-26T14:49:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7101.905517 secs 2024-07-26T14:49:27.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:27.851+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:27.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:27.851+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:27.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:27.851+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:27 smithi107 ceph-mon[32659]: pgmap v4253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:27 smithi132 ceph-mon[37580]: pgmap v4253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:27 smithi112 ceph-mon[37501]: pgmap v4253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:30.078+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:30.079+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:30.080 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:30.079+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:29 smithi107 ceph-mon[32659]: pgmap v4254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:29 smithi132 ceph-mon[37580]: pgmap v4254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:29 smithi112 ceph-mon[37501]: pgmap v4254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7106.905641 secs 2024-07-26T14:49:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:31 smithi107 ceph-mon[32659]: pgmap v4255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7106.905641 secs 2024-07-26T14:49:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:31 smithi132 ceph-mon[37580]: pgmap v4255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7106.905641 secs 2024-07-26T14:49:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:31 smithi112 ceph-mon[37501]: pgmap v4255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:33 smithi107 ceph-mon[32659]: pgmap v4256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:33 smithi132 ceph-mon[37580]: pgmap v4256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:33 smithi112 ceph-mon[37501]: pgmap v4256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:35 smithi107 ceph-mon[32659]: pgmap v4257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:35 smithi132 ceph-mon[37580]: pgmap v4257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:35 smithi112 ceph-mon[37501]: pgmap v4257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7111.905764 secs 2024-07-26T14:49:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7111.905764 secs 2024-07-26T14:49:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7111.905764 secs 2024-07-26T14:49:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:38 smithi107 ceph-mon[32659]: pgmap v4258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:38 smithi132 ceph-mon[37580]: pgmap v4258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:38 smithi112 ceph-mon[37501]: pgmap v4258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:40 smithi107 ceph-mon[32659]: pgmap v4259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:40 smithi132 ceph-mon[37580]: pgmap v4259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:40 smithi112 ceph-mon[37501]: pgmap v4259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7116.905878 secs 2024-07-26T14:49:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:42 smithi107 ceph-mon[32659]: pgmap v4260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7116.905878 secs 2024-07-26T14:49:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:42 smithi132 ceph-mon[37580]: pgmap v4260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7116.905878 secs 2024-07-26T14:49:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:42 smithi112 ceph-mon[37501]: pgmap v4260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:42.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:42.858+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:42.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:42.858+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:42.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:42.858+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:44 smithi107 ceph-mon[32659]: pgmap v4261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:44 smithi132 ceph-mon[37580]: pgmap v4261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:44 smithi112 ceph-mon[37501]: pgmap v4261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:45.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:45.091+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:45.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:45.091+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:45.092 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:49:45.091+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:46 smithi107 ceph-mon[32659]: pgmap v4262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:46 smithi132 ceph-mon[37580]: pgmap v4262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:46 smithi112 ceph-mon[37501]: pgmap v4262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7121.906025 secs 2024-07-26T14:49:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7121.906025 secs 2024-07-26T14:49:47.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7121.906025 secs 2024-07-26T14:49:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:48 smithi132 ceph-mon[37580]: pgmap v4263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:48 smithi112 ceph-mon[37501]: pgmap v4263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:48 smithi107 ceph-mon[32659]: pgmap v4263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:50 smithi132 ceph-mon[37580]: pgmap v4264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:50 smithi112 ceph-mon[37501]: pgmap v4264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:50 smithi107 ceph-mon[32659]: pgmap v4264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7126.906148 secs 2024-07-26T14:49:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:52 smithi107 ceph-mon[32659]: pgmap v4265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7126.906148 secs 2024-07-26T14:49:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:52 smithi132 ceph-mon[37580]: pgmap v4265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7126.906148 secs 2024-07-26T14:49:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:52 smithi112 ceph-mon[37501]: pgmap v4265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:54.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:54 smithi132 ceph-mon[37580]: pgmap v4266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:54 smithi112 ceph-mon[37501]: pgmap v4266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:54 smithi107 ceph-mon[32659]: pgmap v4266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:56 smithi112 ceph-mon[37501]: pgmap v4267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:56 smithi107 ceph-mon[32659]: pgmap v4267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:56 smithi132 ceph-mon[37580]: pgmap v4267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7131.906266 secs 2024-07-26T14:49:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7131.906266 secs 2024-07-26T14:49:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7131.906266 secs 2024-07-26T14:49:57.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:57.874+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:57.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:57.874+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:49:57.875 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:49:57.874+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:49:58 smithi107 ceph-mon[32659]: pgmap v4268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:49:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:49:58 smithi112 ceph-mon[37501]: pgmap v4268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:49:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:49:58 smithi132 ceph-mon[37580]: pgmap v4268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:50:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:00.106+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:00.107 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:00.106+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:00.108 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:00.106+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:00 smithi112 ceph-mon[37501]: pgmap v4269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:50:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:00 smithi107 ceph-mon[32659]: pgmap v4269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:50:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:00 smithi132 ceph-mon[37580]: pgmap v4269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T14:50:02.344 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7136.906395 secs 2024-07-26T14:50:02.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:02 smithi107 ceph-mon[32659]: pgmap v4270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7136.906395 secs 2024-07-26T14:50:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:02 smithi112 ceph-mon[37501]: pgmap v4270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7136.906395 secs 2024-07-26T14:50:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:02 smithi132 ceph-mon[37580]: pgmap v4270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:04 smithi112 ceph-mon[37501]: pgmap v4271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:04 smithi107 ceph-mon[32659]: pgmap v4271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:04 smithi132 ceph-mon[37580]: pgmap v4271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:06 smithi112 ceph-mon[37501]: pgmap v4272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:06 smithi107 ceph-mon[32659]: pgmap v4272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:06 smithi132 ceph-mon[37580]: pgmap v4272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7141.906523 secs 2024-07-26T14:50:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7141.906523 secs 2024-07-26T14:50:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7141.906523 secs 2024-07-26T14:50:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:08 smithi112 ceph-mon[37501]: pgmap v4273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:08 smithi107 ceph-mon[32659]: pgmap v4273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:08 smithi132 ceph-mon[37580]: pgmap v4273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:50:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:50:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:50:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:10 smithi132 ceph-mon[37580]: pgmap v4274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:50:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:50:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:50:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:10 smithi112 ceph-mon[37501]: pgmap v4274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:50:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:50:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:50:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:10 smithi107 ceph-mon[32659]: pgmap v4274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:50:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:50:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:50:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:11 smithi112 ceph-mon[37501]: pgmap v4275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:50:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:11 smithi112 ceph-mon[37501]: pgmap v4276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:11 smithi107 ceph-mon[32659]: pgmap v4275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:50:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:11 smithi107 ceph-mon[32659]: pgmap v4276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:11 smithi132 ceph-mon[37580]: pgmap v4275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:50:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:11 smithi132 ceph-mon[37580]: pgmap v4276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7146.906644 secs 2024-07-26T14:50:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7146.906644 secs 2024-07-26T14:50:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7146.906644 secs 2024-07-26T14:50:12.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:12.888+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:12.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:12.888+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:12.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:12.888+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:13 smithi112 ceph-mon[37501]: pgmap v4277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:13 smithi107 ceph-mon[32659]: pgmap v4277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:13 smithi132 ceph-mon[37580]: pgmap v4277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:15.119+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:15.119+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:15.121 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:15.119+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:15 smithi112 ceph-mon[37501]: pgmap v4278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:50:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:15 smithi107 ceph-mon[32659]: pgmap v4278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:50:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:15 smithi132 ceph-mon[37580]: pgmap v4278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:50:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7151.906775 secs 2024-07-26T14:50:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:17 smithi112 ceph-mon[37501]: pgmap v4279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:50:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7151.906775 secs 2024-07-26T14:50:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:17 smithi107 ceph-mon[32659]: pgmap v4279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:50:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7151.906775 secs 2024-07-26T14:50:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:17 smithi132 ceph-mon[37580]: pgmap v4279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:50:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:19 smithi112 ceph-mon[37501]: pgmap v4280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:50:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:19 smithi107 ceph-mon[32659]: pgmap v4280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:50:19.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:19 smithi132 ceph-mon[37580]: pgmap v4280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:50:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:21 smithi112 ceph-mon[37501]: pgmap v4281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:50:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:21 smithi107 ceph-mon[32659]: pgmap v4281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:50:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:21 smithi132 ceph-mon[37580]: pgmap v4281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:50:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7156.906917 secs 2024-07-26T14:50:22.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7156.906917 secs 2024-07-26T14:50:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7156.906917 secs 2024-07-26T14:50:23.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:23 smithi107 ceph-mon[32659]: pgmap v4282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:23 smithi112 ceph-mon[37501]: pgmap v4282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:23 smithi132 ceph-mon[37580]: pgmap v4282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:25 smithi112 ceph-mon[37501]: pgmap v4283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:25 smithi107 ceph-mon[32659]: pgmap v4283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:25 smithi132 ceph-mon[37580]: pgmap v4283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7161.907050 secs 2024-07-26T14:50:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:27 smithi107 ceph-mon[32659]: pgmap v4284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7161.907050 secs 2024-07-26T14:50:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:27 smithi132 ceph-mon[37580]: pgmap v4284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7161.907050 secs 2024-07-26T14:50:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:27 smithi112 ceph-mon[37501]: pgmap v4284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:27.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:27.905+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:27.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:27.905+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:27.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:27.905+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:29 smithi112 ceph-mon[37501]: pgmap v4285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:29 smithi107 ceph-mon[32659]: pgmap v4285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:29 smithi132 ceph-mon[37580]: pgmap v4285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:30.136+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:30.136+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:30.136+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:31 smithi107 ceph-mon[32659]: pgmap v4286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7166.907121 secs 2024-07-26T14:50:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:31 smithi132 ceph-mon[37580]: pgmap v4286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7166.907121 secs 2024-07-26T14:50:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:31 smithi112 ceph-mon[37501]: pgmap v4286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7166.907121 secs 2024-07-26T14:50:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:33 smithi107 ceph-mon[32659]: pgmap v4287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:33 smithi132 ceph-mon[37580]: pgmap v4287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:33 smithi112 ceph-mon[37501]: pgmap v4287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:35 smithi107 ceph-mon[32659]: pgmap v4288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:35 smithi132 ceph-mon[37580]: pgmap v4288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:35 smithi112 ceph-mon[37501]: pgmap v4288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7171.907187 secs 2024-07-26T14:50:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7171.907187 secs 2024-07-26T14:50:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7171.907187 secs 2024-07-26T14:50:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:37 smithi107 ceph-mon[32659]: pgmap v4289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:37 smithi132 ceph-mon[37580]: pgmap v4289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:37 smithi112 ceph-mon[37501]: pgmap v4289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:39 smithi107 ceph-mon[32659]: pgmap v4290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:39 smithi132 ceph-mon[37580]: pgmap v4290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:39 smithi112 ceph-mon[37501]: pgmap v4290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:41 smithi107 ceph-mon[32659]: pgmap v4291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7176.907316 secs 2024-07-26T14:50:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:41 smithi132 ceph-mon[37580]: pgmap v4291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7176.907316 secs 2024-07-26T14:50:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:41 smithi112 ceph-mon[37501]: pgmap v4291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7176.907316 secs 2024-07-26T14:50:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:42.907+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:42.907+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:42.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:42.907+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:43 smithi107 ceph-mon[32659]: pgmap v4292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:43 smithi132 ceph-mon[37580]: pgmap v4292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:43 smithi112 ceph-mon[37501]: pgmap v4292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:45.151+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:45.151+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:50:45.151+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:45 smithi107 ceph-mon[32659]: pgmap v4293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:45 smithi132 ceph-mon[37580]: pgmap v4293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:45 smithi112 ceph-mon[37501]: pgmap v4293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7181.907438 secs 2024-07-26T14:50:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7181.907438 secs 2024-07-26T14:50:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7181.907438 secs 2024-07-26T14:50:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:47 smithi107 ceph-mon[32659]: pgmap v4294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:47 smithi132 ceph-mon[37580]: pgmap v4294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:47.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:47 smithi112 ceph-mon[37501]: pgmap v4294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:49 smithi107 ceph-mon[32659]: pgmap v4295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:49 smithi132 ceph-mon[37580]: pgmap v4295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:49 smithi112 ceph-mon[37501]: pgmap v4295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:51 smithi132 ceph-mon[37580]: pgmap v4296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7186.907577 secs 2024-07-26T14:50:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:51 smithi112 ceph-mon[37501]: pgmap v4296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7186.907577 secs 2024-07-26T14:50:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:51 smithi107 ceph-mon[32659]: pgmap v4296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7186.907577 secs 2024-07-26T14:50:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:53 smithi132 ceph-mon[37580]: pgmap v4297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:53 smithi112 ceph-mon[37501]: pgmap v4297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:53 smithi107 ceph-mon[32659]: pgmap v4297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:55 smithi112 ceph-mon[37501]: pgmap v4298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:55 smithi107 ceph-mon[32659]: pgmap v4298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:55 smithi132 ceph-mon[37580]: pgmap v4298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7191.907699 secs 2024-07-26T14:50:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7191.907699 secs 2024-07-26T14:50:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7191.907699 secs 2024-07-26T14:50:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:57 smithi112 ceph-mon[37501]: pgmap v4299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:57 smithi107 ceph-mon[32659]: pgmap v4299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:57 smithi132 ceph-mon[37580]: pgmap v4299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:57.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:57.923+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:57.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:57.923+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:50:57.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:50:57.923+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:50:59 smithi112 ceph-mon[37501]: pgmap v4300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:50:59 smithi107 ceph-mon[32659]: pgmap v4300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:50:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:50:59 smithi132 ceph-mon[37580]: pgmap v4300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:51:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:00.166+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:00.166+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:00.166+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:01 smithi112 ceph-mon[37501]: pgmap v4301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7196.907819 secs 2024-07-26T14:51:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:01 smithi107 ceph-mon[32659]: pgmap v4301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7196.907819 secs 2024-07-26T14:51:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:01 smithi132 ceph-mon[37580]: pgmap v4301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7196.907819 secs 2024-07-26T14:51:03.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:03 smithi107 ceph-mon[32659]: pgmap v4302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:03 smithi112 ceph-mon[37501]: pgmap v4302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:03 smithi132 ceph-mon[37580]: pgmap v4302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:05 smithi112 ceph-mon[37501]: pgmap v4303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:51:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:05 smithi107 ceph-mon[32659]: pgmap v4303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:51:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:05 smithi132 ceph-mon[37580]: pgmap v4303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:51:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7201.907937 secs 2024-07-26T14:51:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7201.907937 secs 2024-07-26T14:51:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7201.907937 secs 2024-07-26T14:51:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:07 smithi112 ceph-mon[37501]: pgmap v4304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:07 smithi107 ceph-mon[32659]: pgmap v4304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:07 smithi132 ceph-mon[37580]: pgmap v4304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:09 smithi112 ceph-mon[37501]: pgmap v4305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:51:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:09 smithi107 ceph-mon[32659]: pgmap v4305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:51:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:09 smithi132 ceph-mon[37580]: pgmap v4305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:51:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:51:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:51:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:51:11.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:11.657+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ad0 msgr2=0x7ff0ec0c0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T14:51:11.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:11.657+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ad0 msgr2=0x7ff0ec0c0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:51:11.659 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:11.657+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ad0 0x7ff0ec0c0ee0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fa00 tx=0x7ff0f801f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:51:11.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:11.657+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ad0 0x7ff0ec0c0ee0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fa00 tx=0x7ff0f801f040 comp rx=0 tx=0).stop 2024-07-26T14:51:11.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:11.657+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ad0 msgr2=0x7ff0ec0c0ee0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T14:51:11.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:11.657+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ad0 msgr2=0x7ff0ec0c0ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:51:11.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:11.657+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ad0 0x7ff0ec0c0ee0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:51:11.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:11.657+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0260 0x7ff0ec0c1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:51:11.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:11 smithi107 ceph-mon[32659]: pgmap v4306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:11.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7206.908009 secs 2024-07-26T14:51:11.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:51:11.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:51:11.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:51:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:11.658+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0260 0x7ff0ec0c1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:51:11.661 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:11.660+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0260 0x7ff0ec0c1420 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f801f040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:51:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:11 smithi112 ceph-mon[37501]: pgmap v4306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7206.908009 secs 2024-07-26T14:51:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:51:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:51:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:51:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:11 smithi132 ceph-mon[37580]: pgmap v4306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7206.908009 secs 2024-07-26T14:51:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:51:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:51:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:51:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:12 smithi112 ceph-mon[37501]: pgmap v4307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:51:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:12 smithi112 ceph-mon[37501]: pgmap v4308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:51:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:12 smithi107 ceph-mon[32659]: pgmap v4307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:51:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:12 smithi107 ceph-mon[32659]: pgmap v4308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:51:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:12 smithi132 ceph-mon[37580]: pgmap v4307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:51:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:12 smithi132 ceph-mon[37580]: pgmap v4308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:51:12.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:12.924+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:12.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:12.924+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:12.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:12.924+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:13.766+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2b10 msgr2=0x7efc5c0c37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000259) for more than 900000000 us, fault. 2024-07-26T14:51:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:13.766+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2b10 0x7efc5c0c37b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7efc680aacd0 tx=0x7efc680e6070 comp rx=0 tx=0).stop 2024-07-26T14:51:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:13.766+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2b10 msgr2=0x7efc5c0c37b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:51:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:13.766+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2b10 0x7efc5c0c37b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:51:13.767 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:13.766+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:51:13.768 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:13.766+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:51:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:13.767+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7efc480063a0 tx=0x7efc680da000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:51:13.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:13.855+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:51:13.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:13.855+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:51:13.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:13.871+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:51:13.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:13.872+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:51:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:14 smithi112 ceph-mon[37501]: pgmap v4309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:14 smithi107 ceph-mon[32659]: pgmap v4309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:14 smithi132 ceph-mon[37580]: pgmap v4309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:15.034+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:51:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:15.034+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:51:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:15.034+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:51:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:15.034+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:51:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:15.182+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:15.182+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:15.183 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:15.182+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:16 smithi112 ceph-mon[37501]: pgmap v4310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7211.908129 secs 2024-07-26T14:51:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:16 smithi107 ceph-mon[32659]: pgmap v4310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7211.908129 secs 2024-07-26T14:51:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:16 smithi132 ceph-mon[37580]: pgmap v4310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7211.908129 secs 2024-07-26T14:51:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:18 smithi112 ceph-mon[37501]: pgmap v4311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:18 smithi107 ceph-mon[32659]: pgmap v4311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:18.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:18 smithi132 ceph-mon[37580]: pgmap v4311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:20 smithi112 ceph-mon[37501]: pgmap v4312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:20 smithi107 ceph-mon[32659]: pgmap v4312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:20 smithi132 ceph-mon[37580]: pgmap v4312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7216.908270 secs 2024-07-26T14:51:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7216.908270 secs 2024-07-26T14:51:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7216.908270 secs 2024-07-26T14:51:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:22 smithi112 ceph-mon[37501]: pgmap v4313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:51:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:22 smithi107 ceph-mon[32659]: pgmap v4313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:51:22.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:22 smithi132 ceph-mon[37580]: pgmap v4313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:51:24.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:24 smithi112 ceph-mon[37501]: pgmap v4314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:51:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:24 smithi107 ceph-mon[32659]: pgmap v4314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:51:24.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:24 smithi132 ceph-mon[37580]: pgmap v4314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:51:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:26 smithi112 ceph-mon[37501]: pgmap v4315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:51:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7221.908383 secs 2024-07-26T14:51:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:26 smithi107 ceph-mon[32659]: pgmap v4315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:51:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7221.908383 secs 2024-07-26T14:51:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:26 smithi132 ceph-mon[37580]: pgmap v4315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:51:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7221.908383 secs 2024-07-26T14:51:27.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:27.940+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:27.940+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:27.942 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:27.941+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:28 smithi112 ceph-mon[37501]: pgmap v4316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:51:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:28 smithi107 ceph-mon[32659]: pgmap v4316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:51:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:28 smithi132 ceph-mon[37580]: pgmap v4316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:51:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:30.185+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:30.185+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:30.186 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:30.185+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:30 smithi112 ceph-mon[37501]: pgmap v4317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:51:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:30 smithi107 ceph-mon[32659]: pgmap v4317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:51:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:30 smithi132 ceph-mon[37580]: pgmap v4317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:51:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7226.908460 secs 2024-07-26T14:51:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:31 smithi107 ceph-mon[32659]: pgmap v4318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:51:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7226.908460 secs 2024-07-26T14:51:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:31 smithi132 ceph-mon[37580]: pgmap v4318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:51:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7226.908460 secs 2024-07-26T14:51:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:31 smithi112 ceph-mon[37501]: pgmap v4318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T14:51:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:34 smithi107 ceph-mon[32659]: pgmap v4319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:51:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:34 smithi132 ceph-mon[37580]: pgmap v4319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:51:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:34 smithi112 ceph-mon[37501]: pgmap v4319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:51:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:36 smithi107 ceph-mon[32659]: pgmap v4320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:51:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7231.908607 secs 2024-07-26T14:51:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:36 smithi132 ceph-mon[37580]: pgmap v4320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:51:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7231.908607 secs 2024-07-26T14:51:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:36 smithi112 ceph-mon[37501]: pgmap v4320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:51:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7231.908607 secs 2024-07-26T14:51:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:38 smithi107 ceph-mon[32659]: pgmap v4321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:38 smithi132 ceph-mon[37580]: pgmap v4321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:38 smithi112 ceph-mon[37501]: pgmap v4321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:40 smithi107 ceph-mon[32659]: pgmap v4322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:51:40.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:40 smithi132 ceph-mon[37580]: pgmap v4322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:51:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:40 smithi112 ceph-mon[37501]: pgmap v4322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:51:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7236.908695 secs 2024-07-26T14:51:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7236.908695 secs 2024-07-26T14:51:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7236.908695 secs 2024-07-26T14:51:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:42 smithi132 ceph-mon[37580]: pgmap v4323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:42 smithi112 ceph-mon[37501]: pgmap v4323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:42 smithi107 ceph-mon[32659]: pgmap v4323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:42.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:42.942+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:42.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:42.942+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:42.943 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:42.942+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:44 smithi132 ceph-mon[37580]: pgmap v4324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:44 smithi112 ceph-mon[37501]: pgmap v4324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:44 smithi107 ceph-mon[32659]: pgmap v4324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:45.201+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:45.201+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:51:45.201+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:46 smithi132 ceph-mon[37580]: pgmap v4325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7241.908796 secs 2024-07-26T14:51:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:46 smithi112 ceph-mon[37501]: pgmap v4325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7241.908796 secs 2024-07-26T14:51:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:46 smithi107 ceph-mon[32659]: pgmap v4325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7241.908796 secs 2024-07-26T14:51:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:48 smithi112 ceph-mon[37501]: pgmap v4326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:48 smithi107 ceph-mon[32659]: pgmap v4326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:48 smithi132 ceph-mon[37580]: pgmap v4326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:50 smithi112 ceph-mon[37501]: pgmap v4327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:51:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:50 smithi107 ceph-mon[32659]: pgmap v4327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:51:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:50 smithi132 ceph-mon[37580]: pgmap v4327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:51:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7246.908920 secs 2024-07-26T14:51:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7246.908920 secs 2024-07-26T14:51:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7246.908920 secs 2024-07-26T14:51:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:52 smithi112 ceph-mon[37501]: pgmap v4328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:52.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:52 smithi107 ceph-mon[32659]: pgmap v4328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:52 smithi132 ceph-mon[37580]: pgmap v4328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:54.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:54 smithi132 ceph-mon[37580]: pgmap v4329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:54 smithi112 ceph-mon[37501]: pgmap v4329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:54 smithi107 ceph-mon[32659]: pgmap v4329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:56 smithi112 ceph-mon[37501]: pgmap v4330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7251.909040 secs 2024-07-26T14:51:56.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:56 smithi107 ceph-mon[32659]: pgmap v4330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:56.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7251.909040 secs 2024-07-26T14:51:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:56 smithi132 ceph-mon[37580]: pgmap v4330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7251.909040 secs 2024-07-26T14:51:57.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:57.947+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:57.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:57.947+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:51:57.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:51:57.947+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:51:58 smithi112 ceph-mon[37501]: pgmap v4331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:51:58 smithi107 ceph-mon[32659]: pgmap v4331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:51:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:51:58 smithi132 ceph-mon[37580]: pgmap v4331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:00.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:00.212+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:00.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:00.212+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:00.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:00.212+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:00 smithi112 ceph-mon[37501]: pgmap v4332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:00 smithi107 ceph-mon[32659]: pgmap v4332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:00 smithi132 ceph-mon[37580]: pgmap v4332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7256.909158 secs 2024-07-26T14:52:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7256.909158 secs 2024-07-26T14:52:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7256.909158 secs 2024-07-26T14:52:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:02 smithi112 ceph-mon[37501]: pgmap v4333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:02 smithi107 ceph-mon[32659]: pgmap v4333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:02 smithi132 ceph-mon[37580]: pgmap v4333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:04 smithi112 ceph-mon[37501]: pgmap v4334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:04 smithi107 ceph-mon[32659]: pgmap v4334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:04 smithi132 ceph-mon[37580]: pgmap v4334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:06 smithi112 ceph-mon[37501]: pgmap v4335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7261.909282 secs 2024-07-26T14:52:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:06 smithi107 ceph-mon[32659]: pgmap v4335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7261.909282 secs 2024-07-26T14:52:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:06 smithi132 ceph-mon[37580]: pgmap v4335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7261.909282 secs 2024-07-26T14:52:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:08 smithi112 ceph-mon[37501]: pgmap v4336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:08 smithi107 ceph-mon[32659]: pgmap v4336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:08 smithi132 ceph-mon[37580]: pgmap v4336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:10 smithi112 ceph-mon[37501]: pgmap v4337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:10 smithi107 ceph-mon[32659]: pgmap v4337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:10 smithi132 ceph-mon[37580]: pgmap v4337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7266.909396 secs 2024-07-26T14:52:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7266.909396 secs 2024-07-26T14:52:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7266.909396 secs 2024-07-26T14:52:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:12 smithi112 ceph-mon[37501]: pgmap v4338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:52:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:52:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:52:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:12 smithi107 ceph-mon[32659]: pgmap v4338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:52:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:52:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:52:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:12 smithi132 ceph-mon[37580]: pgmap v4338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:52:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:52:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:52:12.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:12.957+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:12.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:12.957+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:12.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:12.957+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:13 smithi112 ceph-mon[37501]: pgmap v4339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:52:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:13 smithi112 ceph-mon[37501]: pgmap v4340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:52:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:52:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:13 smithi107 ceph-mon[32659]: pgmap v4339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:52:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:13 smithi107 ceph-mon[32659]: pgmap v4340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:52:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:52:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:13 smithi132 ceph-mon[37580]: pgmap v4339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:52:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:13 smithi132 ceph-mon[37580]: pgmap v4340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:52:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:52:15.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:15.226+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:15.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:15.226+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:15.228 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:15.226+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:15 smithi112 ceph-mon[37501]: pgmap v4341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:15 smithi107 ceph-mon[32659]: pgmap v4341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:15 smithi132 ceph-mon[37580]: pgmap v4341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7271.909511 secs 2024-07-26T14:52:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7271.909511 secs 2024-07-26T14:52:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7271.909511 secs 2024-07-26T14:52:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:17 smithi112 ceph-mon[37501]: pgmap v4342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:17 smithi107 ceph-mon[32659]: pgmap v4342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:17 smithi132 ceph-mon[37580]: pgmap v4342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:19.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:19 smithi112 ceph-mon[37501]: pgmap v4343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:19 smithi107 ceph-mon[32659]: pgmap v4343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:19 smithi132 ceph-mon[37580]: pgmap v4343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7276.909633 secs 2024-07-26T14:52:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:21 smithi112 ceph-mon[37501]: pgmap v4344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7276.909633 secs 2024-07-26T14:52:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:21 smithi107 ceph-mon[32659]: pgmap v4344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7276.909633 secs 2024-07-26T14:52:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:21 smithi132 ceph-mon[37580]: pgmap v4344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:23 smithi107 ceph-mon[32659]: pgmap v4345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:52:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:23 smithi132 ceph-mon[37580]: pgmap v4345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:52:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:23 smithi112 ceph-mon[37501]: pgmap v4345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:52:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:25 smithi107 ceph-mon[32659]: pgmap v4346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:25 smithi132 ceph-mon[37580]: pgmap v4346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:25 smithi112 ceph-mon[37501]: pgmap v4346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7281.909756 secs 2024-07-26T14:52:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:26 smithi107 ceph-mon[32659]: pgmap v4347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7281.909756 secs 2024-07-26T14:52:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:26 smithi132 ceph-mon[37580]: pgmap v4347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7281.909756 secs 2024-07-26T14:52:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:26 smithi112 ceph-mon[37501]: pgmap v4347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:27.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:27.973+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:27.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:27.973+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:27.974 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:27.973+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:28 smithi107 ceph-mon[32659]: pgmap v4348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:28 smithi132 ceph-mon[37580]: pgmap v4348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:28 smithi112 ceph-mon[37501]: pgmap v4348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:30.231+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:30.231+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:30.232 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:30.231+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:30 smithi107 ceph-mon[32659]: pgmap v4349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:30 smithi132 ceph-mon[37580]: pgmap v4349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:30 smithi112 ceph-mon[37501]: pgmap v4349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7286.909850 secs 2024-07-26T14:52:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7286.909850 secs 2024-07-26T14:52:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7286.909850 secs 2024-07-26T14:52:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:32 smithi107 ceph-mon[32659]: pgmap v4350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:32 smithi132 ceph-mon[37580]: pgmap v4350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:32 smithi112 ceph-mon[37501]: pgmap v4350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:34 smithi107 ceph-mon[32659]: pgmap v4351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:34 smithi132 ceph-mon[37580]: pgmap v4351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:34 smithi112 ceph-mon[37501]: pgmap v4351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7291.909975 secs 2024-07-26T14:52:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7291.909975 secs 2024-07-26T14:52:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7291.909975 secs 2024-07-26T14:52:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:37 smithi107 ceph-mon[32659]: pgmap v4352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:37 smithi132 ceph-mon[37580]: pgmap v4352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:37 smithi112 ceph-mon[37501]: pgmap v4352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:38 smithi107 ceph-mon[32659]: pgmap v4353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:38 smithi132 ceph-mon[37580]: pgmap v4353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:38 smithi112 ceph-mon[37501]: pgmap v4353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:40 smithi107 ceph-mon[32659]: pgmap v4354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:40 smithi132 ceph-mon[37580]: pgmap v4354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:40 smithi112 ceph-mon[37501]: pgmap v4354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7296.910089 secs 2024-07-26T14:52:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7296.910089 secs 2024-07-26T14:52:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7296.910089 secs 2024-07-26T14:52:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:42 smithi107 ceph-mon[32659]: pgmap v4355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:42 smithi132 ceph-mon[37580]: pgmap v4355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:42 smithi112 ceph-mon[37501]: pgmap v4355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:42.982+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:42.982+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:42.983 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:42.982+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:44 smithi107 ceph-mon[32659]: pgmap v4356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:44 smithi132 ceph-mon[37580]: pgmap v4356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:44 smithi112 ceph-mon[37501]: pgmap v4356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:45.239 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:45.238+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:45.239+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:45.240 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:52:45.239+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7301.910195 secs 2024-07-26T14:52:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7301.910195 secs 2024-07-26T14:52:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7301.910195 secs 2024-07-26T14:52:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:47 smithi107 ceph-mon[32659]: pgmap v4357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:47 smithi132 ceph-mon[37580]: pgmap v4357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:47 smithi112 ceph-mon[37501]: pgmap v4357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:48 smithi107 ceph-mon[32659]: pgmap v4358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:48 smithi132 ceph-mon[37580]: pgmap v4358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:48 smithi112 ceph-mon[37501]: pgmap v4358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:50 smithi107 ceph-mon[32659]: pgmap v4359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:50 smithi132 ceph-mon[37580]: pgmap v4359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:50 smithi112 ceph-mon[37501]: pgmap v4359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7306.910347 secs 2024-07-26T14:52:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7306.910347 secs 2024-07-26T14:52:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7306.910347 secs 2024-07-26T14:52:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:52 smithi107 ceph-mon[32659]: pgmap v4360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:52 smithi132 ceph-mon[37580]: pgmap v4360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:52 smithi112 ceph-mon[37501]: pgmap v4360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:54 smithi107 ceph-mon[32659]: pgmap v4361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:54 smithi132 ceph-mon[37580]: pgmap v4361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:54 smithi112 ceph-mon[37501]: pgmap v4361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:52:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7311.910473 secs 2024-07-26T14:52:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7311.910473 secs 2024-07-26T14:52:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7311.910473 secs 2024-07-26T14:52:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:57 smithi132 ceph-mon[37580]: pgmap v4362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:57 smithi112 ceph-mon[37501]: pgmap v4362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:57 smithi107 ceph-mon[32659]: pgmap v4362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:57.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:57.997+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:57.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:57.997+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:52:57.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:52:57.997+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:52:58 smithi107 ceph-mon[32659]: pgmap v4363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:52:58 smithi132 ceph-mon[37580]: pgmap v4363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:52:58 smithi112 ceph-mon[37501]: pgmap v4363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:00.248+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:00.248+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:00.249 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:00.248+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:00 smithi107 ceph-mon[32659]: pgmap v4364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:00 smithi132 ceph-mon[37580]: pgmap v4364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:00 smithi112 ceph-mon[37501]: pgmap v4364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7316.910594 secs 2024-07-26T14:53:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7316.910594 secs 2024-07-26T14:53:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7316.910594 secs 2024-07-26T14:53:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:02 smithi107 ceph-mon[32659]: pgmap v4365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:02 smithi132 ceph-mon[37580]: pgmap v4365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:02 smithi112 ceph-mon[37501]: pgmap v4365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:04 smithi107 ceph-mon[32659]: pgmap v4366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:04 smithi132 ceph-mon[37580]: pgmap v4366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:04 smithi112 ceph-mon[37501]: pgmap v4366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7321.910720 secs 2024-07-26T14:53:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7321.910720 secs 2024-07-26T14:53:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7321.910720 secs 2024-07-26T14:53:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:07 smithi132 ceph-mon[37580]: pgmap v4367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:07 smithi112 ceph-mon[37501]: pgmap v4367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:07 smithi107 ceph-mon[32659]: pgmap v4367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:08 smithi107 ceph-mon[32659]: pgmap v4368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:08 smithi132 ceph-mon[37580]: pgmap v4368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:08 smithi112 ceph-mon[37501]: pgmap v4368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:10 smithi107 ceph-mon[32659]: pgmap v4369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:10 smithi132 ceph-mon[37580]: pgmap v4369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:10 smithi112 ceph-mon[37501]: pgmap v4369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7326.910857 secs 2024-07-26T14:53:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7326.910857 secs 2024-07-26T14:53:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7326.910857 secs 2024-07-26T14:53:12.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:12 smithi107 ceph-mon[32659]: pgmap v4370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:12 smithi132 ceph-mon[37580]: pgmap v4370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:12 smithi112 ceph-mon[37501]: pgmap v4370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:13.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:13.012+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:13.012+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:13.012+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:13.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:53:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:53:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:53:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:53:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:53:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:14 smithi132 ceph-mon[37580]: pgmap v4371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:53:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:14 smithi132 ceph-mon[37580]: pgmap v4372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:53:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:53:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:53:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:53:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:14 smithi112 ceph-mon[37501]: pgmap v4371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:53:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:14 smithi112 ceph-mon[37501]: pgmap v4372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:53:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:53:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:53:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:53:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:14 smithi107 ceph-mon[32659]: pgmap v4371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:53:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:14 smithi107 ceph-mon[32659]: pgmap v4372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:53:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:53:15.257 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:15.255+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:15.257 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:15.255+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:15.257 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:15.255+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7331.910984 secs 2024-07-26T14:53:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:17 smithi107 ceph-mon[32659]: pgmap v4373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:53:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7331.910984 secs 2024-07-26T14:53:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:17 smithi132 ceph-mon[37580]: pgmap v4373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:53:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7331.910984 secs 2024-07-26T14:53:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:17 smithi112 ceph-mon[37501]: pgmap v4373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:53:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:19 smithi107 ceph-mon[32659]: pgmap v4374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:53:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:19 smithi132 ceph-mon[37580]: pgmap v4374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:53:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:19 smithi112 ceph-mon[37501]: pgmap v4374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:53:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:21 smithi107 ceph-mon[32659]: pgmap v4375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:53:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:21 smithi132 ceph-mon[37580]: pgmap v4375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:53:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:21 smithi112 ceph-mon[37501]: pgmap v4375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:53:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7336.911051 secs 2024-07-26T14:53:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7336.911051 secs 2024-07-26T14:53:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7336.911051 secs 2024-07-26T14:53:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:23 smithi132 ceph-mon[37580]: pgmap v4376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:23 smithi112 ceph-mon[37501]: pgmap v4376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:23 smithi107 ceph-mon[32659]: pgmap v4376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:25 smithi132 ceph-mon[37580]: pgmap v4377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:53:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:25 smithi112 ceph-mon[37501]: pgmap v4377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:53:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:25 smithi107 ceph-mon[32659]: pgmap v4377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:53:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7341.911138 secs 2024-07-26T14:53:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:27 smithi132 ceph-mon[37580]: pgmap v4378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7341.911138 secs 2024-07-26T14:53:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:27 smithi112 ceph-mon[37501]: pgmap v4378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7341.911138 secs 2024-07-26T14:53:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:27 smithi107 ceph-mon[32659]: pgmap v4378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:28.028+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:28.028+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:28.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:28.028+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:29 smithi112 ceph-mon[37501]: pgmap v4379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:29 smithi107 ceph-mon[32659]: pgmap v4379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:29 smithi132 ceph-mon[37580]: pgmap v4379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:30.272 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:30.272+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:30.273 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:30.272+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:30.273 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:30.272+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:31 smithi107 ceph-mon[32659]: pgmap v4380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:31 smithi112 ceph-mon[37501]: pgmap v4380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:31 smithi132 ceph-mon[37580]: pgmap v4380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:32 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7346.911266 secs 2024-07-26T14:53:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:32 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7346.911266 secs 2024-07-26T14:53:32.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:32 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7346.911266 secs 2024-07-26T14:53:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:33 smithi112 ceph-mon[37501]: pgmap v4381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:33 smithi107 ceph-mon[32659]: pgmap v4381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:33 smithi132 ceph-mon[37580]: pgmap v4381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:35.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:35 smithi107 ceph-mon[32659]: pgmap v4382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:35 smithi112 ceph-mon[37501]: pgmap v4382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:35 smithi132 ceph-mon[37580]: pgmap v4382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7351.911420 secs 2024-07-26T14:53:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:37 smithi112 ceph-mon[37501]: pgmap v4383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7351.911420 secs 2024-07-26T14:53:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:37 smithi107 ceph-mon[32659]: pgmap v4383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7351.911420 secs 2024-07-26T14:53:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:37 smithi132 ceph-mon[37580]: pgmap v4383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:39 smithi112 ceph-mon[37501]: pgmap v4384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:39 smithi107 ceph-mon[32659]: pgmap v4384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:39 smithi132 ceph-mon[37580]: pgmap v4384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:41 smithi112 ceph-mon[37501]: pgmap v4385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:41 smithi107 ceph-mon[32659]: pgmap v4385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:41 smithi132 ceph-mon[37580]: pgmap v4385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7356.911525 secs 2024-07-26T14:53:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7356.911525 secs 2024-07-26T14:53:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7356.911525 secs 2024-07-26T14:53:43.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:43.043+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:43.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:43.043+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:43.045 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:43.043+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:43 smithi112 ceph-mon[37501]: pgmap v4386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:43 smithi107 ceph-mon[32659]: pgmap v4386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:43 smithi132 ceph-mon[37580]: pgmap v4386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:45.274+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:45.274+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:53:45.274+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:45 smithi112 ceph-mon[37501]: pgmap v4387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:45 smithi107 ceph-mon[32659]: pgmap v4387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:45 smithi132 ceph-mon[37580]: pgmap v4387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7361.911638 secs 2024-07-26T14:53:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:47 smithi112 ceph-mon[37501]: pgmap v4388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7361.911638 secs 2024-07-26T14:53:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:47 smithi107 ceph-mon[32659]: pgmap v4388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7361.911638 secs 2024-07-26T14:53:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:47 smithi132 ceph-mon[37580]: pgmap v4388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:49 smithi112 ceph-mon[37501]: pgmap v4389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:49 smithi107 ceph-mon[32659]: pgmap v4389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:49 smithi132 ceph-mon[37580]: pgmap v4389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:51 smithi112 ceph-mon[37501]: pgmap v4390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:51.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:51 smithi107 ceph-mon[32659]: pgmap v4390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:51 smithi132 ceph-mon[37580]: pgmap v4390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:53:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7366.911779 secs 2024-07-26T14:53:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7366.911779 secs 2024-07-26T14:53:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7366.911779 secs 2024-07-26T14:53:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:53 smithi112 ceph-mon[37501]: pgmap v4391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:53 smithi107 ceph-mon[32659]: pgmap v4391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:53 smithi132 ceph-mon[37580]: pgmap v4391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:55 smithi112 ceph-mon[37501]: pgmap v4392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:55 smithi107 ceph-mon[32659]: pgmap v4392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:55 smithi132 ceph-mon[37580]: pgmap v4392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7371.911880 secs 2024-07-26T14:53:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:57 smithi112 ceph-mon[37501]: pgmap v4393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7371.911880 secs 2024-07-26T14:53:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:57 smithi107 ceph-mon[32659]: pgmap v4393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7371.911880 secs 2024-07-26T14:53:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:57 smithi132 ceph-mon[37580]: pgmap v4393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:58.050+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:58.050+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:53:58.050+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:53:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:53:59 smithi112 ceph-mon[37501]: pgmap v4394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:53:59 smithi107 ceph-mon[32659]: pgmap v4394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:53:59 smithi132 ceph-mon[37580]: pgmap v4394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:00.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:00.284+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:00.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:00.284+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:00.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:00.284+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:01 smithi112 ceph-mon[37501]: pgmap v4395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:54:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:01 smithi107 ceph-mon[32659]: pgmap v4395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:54:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:01 smithi132 ceph-mon[37580]: pgmap v4395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:54:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7376.911948 secs 2024-07-26T14:54:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7376.911948 secs 2024-07-26T14:54:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7376.911948 secs 2024-07-26T14:54:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:03 smithi112 ceph-mon[37501]: pgmap v4396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:03 smithi107 ceph-mon[32659]: pgmap v4396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:03 smithi132 ceph-mon[37580]: pgmap v4396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:05 smithi107 ceph-mon[32659]: pgmap v4397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:05 smithi132 ceph-mon[37580]: pgmap v4397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:05 smithi112 ceph-mon[37501]: pgmap v4397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7381.912116 secs 2024-07-26T14:54:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:07 smithi107 ceph-mon[32659]: pgmap v4398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7381.912116 secs 2024-07-26T14:54:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:07 smithi132 ceph-mon[37580]: pgmap v4398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7381.912116 secs 2024-07-26T14:54:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:07 smithi112 ceph-mon[37501]: pgmap v4398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:09 smithi107 ceph-mon[32659]: pgmap v4399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:09 smithi132 ceph-mon[37580]: pgmap v4399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:09 smithi112 ceph-mon[37501]: pgmap v4399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:11 smithi107 ceph-mon[32659]: pgmap v4400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:54:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:11 smithi132 ceph-mon[37580]: pgmap v4400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:54:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:11 smithi112 ceph-mon[37501]: pgmap v4400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:54:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7386.912258 secs 2024-07-26T14:54:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7386.912258 secs 2024-07-26T14:54:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7386.912258 secs 2024-07-26T14:54:13.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:13.065+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:13.065+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:13.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:13.065+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:13 smithi107 ceph-mon[32659]: pgmap v4401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:13 smithi132 ceph-mon[37580]: pgmap v4401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:13 smithi112 ceph-mon[37501]: pgmap v4401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:13.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:13.884+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:54:13.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:13.885+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:54:13.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:13.901+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:54:13.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:13.901+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:54:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:54:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:54:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:54:15.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:15.063+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:54:15.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:15.063+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:54:15.064 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:15.063+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:54:15.065 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:15.063+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:54:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:15.298+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:15.298+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:15.298+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:15 smithi107 ceph-mon[32659]: pgmap v4402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:54:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:54:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:54:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:15 smithi132 ceph-mon[37580]: pgmap v4402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:54:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:54:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:54:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:15 smithi112 ceph-mon[37501]: pgmap v4402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:54:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:54:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:54:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:16 smithi107 ceph-mon[32659]: pgmap v4403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:54:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:16 smithi107 ceph-mon[32659]: pgmap v4404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:54:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7391.912324 secs 2024-07-26T14:54:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:16 smithi132 ceph-mon[37580]: pgmap v4403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:54:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:16 smithi132 ceph-mon[37580]: pgmap v4404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:54:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7391.912324 secs 2024-07-26T14:54:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:16 smithi112 ceph-mon[37501]: pgmap v4403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:54:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:16 smithi112 ceph-mon[37501]: pgmap v4404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:54:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7391.912324 secs 2024-07-26T14:54:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:18 smithi107 ceph-mon[32659]: pgmap v4405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:18 smithi132 ceph-mon[37580]: pgmap v4405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:18 smithi112 ceph-mon[37501]: pgmap v4405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:20 smithi107 ceph-mon[32659]: pgmap v4406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:54:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:20 smithi132 ceph-mon[37580]: pgmap v4406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:54:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:20 smithi112 ceph-mon[37501]: pgmap v4406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:54:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7396.912451 secs 2024-07-26T14:54:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7396.912451 secs 2024-07-26T14:54:21.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7396.912451 secs 2024-07-26T14:54:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:22 smithi107 ceph-mon[32659]: pgmap v4407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:54:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:22 smithi132 ceph-mon[37580]: pgmap v4407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:54:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:22 smithi112 ceph-mon[37501]: pgmap v4407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:54:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:24 smithi107 ceph-mon[32659]: pgmap v4408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:24 smithi132 ceph-mon[37580]: pgmap v4408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:24 smithi112 ceph-mon[37501]: pgmap v4408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:26 smithi107 ceph-mon[32659]: pgmap v4409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:54:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7401.912568 secs 2024-07-26T14:54:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:26 smithi132 ceph-mon[37580]: pgmap v4409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:54:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7401.912568 secs 2024-07-26T14:54:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:26 smithi112 ceph-mon[37501]: pgmap v4409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:54:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7401.912568 secs 2024-07-26T14:54:28.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:28.081+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:28.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:28.081+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:28.083 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:28.081+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:28 smithi107 ceph-mon[32659]: pgmap v4410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:28 smithi132 ceph-mon[37580]: pgmap v4410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:28 smithi112 ceph-mon[37501]: pgmap v4410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:30.313+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:30.313+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:30.315 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:30.313+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:30.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:30 smithi132 ceph-mon[37580]: pgmap v4411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:54:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:30 smithi112 ceph-mon[37501]: pgmap v4411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:54:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:30 smithi107 ceph-mon[32659]: pgmap v4411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:54:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7406.912723 secs 2024-07-26T14:54:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7406.912723 secs 2024-07-26T14:54:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7406.912723 secs 2024-07-26T14:54:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:32 smithi112 ceph-mon[37501]: pgmap v4412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:54:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:32 smithi107 ceph-mon[32659]: pgmap v4412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:54:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:32 smithi132 ceph-mon[37580]: pgmap v4412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:54:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:34 smithi112 ceph-mon[37501]: pgmap v4413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:54:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:34 smithi107 ceph-mon[32659]: pgmap v4413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:54:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:34 smithi132 ceph-mon[37580]: pgmap v4413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:54:36.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:36 smithi107 ceph-mon[32659]: pgmap v4414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:54:36.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7411.912854 secs 2024-07-26T14:54:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:36 smithi112 ceph-mon[37501]: pgmap v4414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:54:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7411.912854 secs 2024-07-26T14:54:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:36 smithi132 ceph-mon[37580]: pgmap v4414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:54:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7411.912854 secs 2024-07-26T14:54:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:38 smithi112 ceph-mon[37501]: pgmap v4415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:38 smithi107 ceph-mon[32659]: pgmap v4415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:38 smithi132 ceph-mon[37580]: pgmap v4415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:40 smithi112 ceph-mon[37501]: pgmap v4416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:40 smithi107 ceph-mon[32659]: pgmap v4416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:40 smithi132 ceph-mon[37580]: pgmap v4416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7416.912921 secs 2024-07-26T14:54:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7416.912921 secs 2024-07-26T14:54:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7416.912921 secs 2024-07-26T14:54:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:42 smithi112 ceph-mon[37501]: pgmap v4417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:42 smithi107 ceph-mon[32659]: pgmap v4417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:42 smithi132 ceph-mon[37580]: pgmap v4417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:43.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:43.096+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:43.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:43.096+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:43.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:43.096+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:44 smithi112 ceph-mon[37501]: pgmap v4418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:44 smithi107 ceph-mon[32659]: pgmap v4418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:44 smithi132 ceph-mon[37580]: pgmap v4418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:45.316+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:45.316+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:54:45.316+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:46 smithi112 ceph-mon[37501]: pgmap v4419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7421.913033 secs 2024-07-26T14:54:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:46 smithi107 ceph-mon[32659]: pgmap v4419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7421.913033 secs 2024-07-26T14:54:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:46 smithi132 ceph-mon[37580]: pgmap v4419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7421.913033 secs 2024-07-26T14:54:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:48 smithi107 ceph-mon[32659]: pgmap v4420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:48 smithi112 ceph-mon[37501]: pgmap v4420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:48 smithi132 ceph-mon[37580]: pgmap v4420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:50 smithi112 ceph-mon[37501]: pgmap v4421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:50 smithi107 ceph-mon[32659]: pgmap v4421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:50 smithi132 ceph-mon[37580]: pgmap v4421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7426.913152 secs 2024-07-26T14:54:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7426.913152 secs 2024-07-26T14:54:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7426.913152 secs 2024-07-26T14:54:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:52 smithi112 ceph-mon[37501]: pgmap v4422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:52 smithi107 ceph-mon[32659]: pgmap v4422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:52 smithi132 ceph-mon[37580]: pgmap v4422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:54 smithi112 ceph-mon[37501]: pgmap v4423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:54 smithi107 ceph-mon[32659]: pgmap v4423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:54 smithi132 ceph-mon[37580]: pgmap v4423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:54:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:56 smithi112 ceph-mon[37501]: pgmap v4424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7431.913237 secs 2024-07-26T14:54:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:56 smithi107 ceph-mon[32659]: pgmap v4424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7431.913237 secs 2024-07-26T14:54:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:56 smithi132 ceph-mon[37580]: pgmap v4424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7431.913237 secs 2024-07-26T14:54:57.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:57.893+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:54:57.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:57.894+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:54:57.909 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:57.909+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:54:57.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:57.909+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:54:58.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:58.112+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:58.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:58.112+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:54:58.113 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:54:58.112+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:54:58 smithi112 ceph-mon[37501]: pgmap v4425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:54:58 smithi107 ceph-mon[32659]: pgmap v4425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:54:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:54:58 smithi132 ceph-mon[37580]: pgmap v4425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:00.071+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:55:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:00.071+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:55:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:00.072+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:55:00.073 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:00.072+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:00.332+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:00.332+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:00.332+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:00 smithi112 ceph-mon[37501]: pgmap v4426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:00 smithi107 ceph-mon[32659]: pgmap v4426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:00 smithi132 ceph-mon[37580]: pgmap v4426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7436.913354 secs 2024-07-26T14:55:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7436.913354 secs 2024-07-26T14:55:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7436.913354 secs 2024-07-26T14:55:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:02 smithi112 ceph-mon[37501]: pgmap v4427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:02 smithi107 ceph-mon[32659]: pgmap v4427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:02 smithi132 ceph-mon[37580]: pgmap v4427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:04 smithi112 ceph-mon[37501]: pgmap v4428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:04 smithi107 ceph-mon[32659]: pgmap v4428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:04 smithi132 ceph-mon[37580]: pgmap v4428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:06 smithi112 ceph-mon[37501]: pgmap v4429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7441.913511 secs 2024-07-26T14:55:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:06 smithi107 ceph-mon[32659]: pgmap v4429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7441.913511 secs 2024-07-26T14:55:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:06 smithi132 ceph-mon[37580]: pgmap v4429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7441.913511 secs 2024-07-26T14:55:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:08 smithi112 ceph-mon[37501]: pgmap v4430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:08 smithi107 ceph-mon[32659]: pgmap v4430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:08 smithi132 ceph-mon[37580]: pgmap v4430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:10 smithi112 ceph-mon[37501]: pgmap v4431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:10 smithi107 ceph-mon[32659]: pgmap v4431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:10 smithi132 ceph-mon[37580]: pgmap v4431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7446.913630 secs 2024-07-26T14:55:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7446.913630 secs 2024-07-26T14:55:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7446.913630 secs 2024-07-26T14:55:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:12 smithi107 ceph-mon[32659]: pgmap v4432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:12 smithi132 ceph-mon[37580]: pgmap v4432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:12 smithi112 ceph-mon[37501]: pgmap v4432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:13.120+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:13.120+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:13.122 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:13.120+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:14 smithi107 ceph-mon[32659]: pgmap v4433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:14 smithi132 ceph-mon[37580]: pgmap v4433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:14 smithi112 ceph-mon[37501]: pgmap v4433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:15.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:15.346+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:15.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:15.346+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:15.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:15.346+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:15.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:55:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:55:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:55:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: pgmap v4434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7451.913695 secs 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:55:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:55:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: pgmap v4434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7451.913695 secs 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:55:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:55:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: pgmap v4434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7451.913695 secs 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:55:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:55:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:17 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:55:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:17 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:55:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:17 smithi107 ceph-mon[32659]: pgmap v4435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:55:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:17 smithi107 ceph-mon[32659]: pgmap v4436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:55:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:17 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:55:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:17 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:55:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:17 smithi132 ceph-mon[37580]: pgmap v4435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:55:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:17 smithi132 ceph-mon[37580]: pgmap v4436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:55:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:17 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T14:55:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:17 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T14:55:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:17 smithi112 ceph-mon[37501]: pgmap v4435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:55:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:17 smithi112 ceph-mon[37501]: pgmap v4436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:55:18.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:18 smithi132 ceph-mon[37580]: pgmap v4437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:55:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:18 smithi112 ceph-mon[37501]: pgmap v4437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:55:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:18 smithi107 ceph-mon[32659]: pgmap v4437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:55:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:21 smithi132 ceph-mon[37580]: pgmap v4438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:55:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:21 smithi112 ceph-mon[37501]: pgmap v4438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:55:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:21 smithi107 ceph-mon[32659]: pgmap v4438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:55:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7456.913761 secs 2024-07-26T14:55:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7456.913761 secs 2024-07-26T14:55:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7456.913761 secs 2024-07-26T14:55:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:23 smithi112 ceph-mon[37501]: pgmap v4439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:23 smithi107 ceph-mon[32659]: pgmap v4439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:23 smithi132 ceph-mon[37580]: pgmap v4439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:25 smithi112 ceph-mon[37501]: pgmap v4440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:55:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:25 smithi107 ceph-mon[32659]: pgmap v4440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:55:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:25 smithi132 ceph-mon[37580]: pgmap v4440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:55:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7461.913885 secs 2024-07-26T14:55:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7461.913885 secs 2024-07-26T14:55:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7461.913885 secs 2024-07-26T14:55:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:27 smithi112 ceph-mon[37501]: pgmap v4441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:55:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:27 smithi107 ceph-mon[32659]: pgmap v4441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:55:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:27 smithi132 ceph-mon[37580]: pgmap v4441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:55:28.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:28.136+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:28.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:28.136+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:28.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:28.136+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:29.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:29 smithi132 ceph-mon[37580]: pgmap v4442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:29 smithi112 ceph-mon[37501]: pgmap v4442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:29 smithi107 ceph-mon[32659]: pgmap v4442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:30.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:30.362+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:30.362+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:30.364 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:30.362+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:31 smithi112 ceph-mon[37501]: pgmap v4443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7466.914047 secs 2024-07-26T14:55:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:31 smithi107 ceph-mon[32659]: pgmap v4443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7466.914047 secs 2024-07-26T14:55:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:31 smithi132 ceph-mon[37580]: pgmap v4443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7466.914047 secs 2024-07-26T14:55:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:33 smithi112 ceph-mon[37501]: pgmap v4444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:33 smithi107 ceph-mon[32659]: pgmap v4444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:33 smithi132 ceph-mon[37580]: pgmap v4444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:35 smithi107 ceph-mon[32659]: pgmap v4445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:35 smithi112 ceph-mon[37501]: pgmap v4445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:35 smithi132 ceph-mon[37580]: pgmap v4445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7471.914167 secs 2024-07-26T14:55:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7471.914167 secs 2024-07-26T14:55:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7471.914167 secs 2024-07-26T14:55:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:37 smithi112 ceph-mon[37501]: pgmap v4446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:37 smithi107 ceph-mon[32659]: pgmap v4446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:37 smithi132 ceph-mon[37580]: pgmap v4446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:39 smithi112 ceph-mon[37501]: pgmap v4447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:39 smithi107 ceph-mon[32659]: pgmap v4447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:39 smithi132 ceph-mon[37580]: pgmap v4447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:41 smithi112 ceph-mon[37501]: pgmap v4448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7476.914285 secs 2024-07-26T14:55:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:41 smithi107 ceph-mon[32659]: pgmap v4448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7476.914285 secs 2024-07-26T14:55:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:41 smithi132 ceph-mon[37580]: pgmap v4448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7476.914285 secs 2024-07-26T14:55:43.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:43.149+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:43.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:43.149+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:43.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:43.149+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:43 smithi112 ceph-mon[37501]: pgmap v4449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:43 smithi107 ceph-mon[32659]: pgmap v4449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:43 smithi132 ceph-mon[37580]: pgmap v4449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:45.378+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:45.378+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:55:45.378+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:45 smithi112 ceph-mon[37501]: pgmap v4450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:45 smithi107 ceph-mon[32659]: pgmap v4450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:45 smithi132 ceph-mon[37580]: pgmap v4450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7481.914410 secs 2024-07-26T14:55:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7481.914410 secs 2024-07-26T14:55:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7481.914410 secs 2024-07-26T14:55:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:47 smithi112 ceph-mon[37501]: pgmap v4451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:47 smithi107 ceph-mon[32659]: pgmap v4451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:47 smithi132 ceph-mon[37580]: pgmap v4451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:49 smithi112 ceph-mon[37501]: pgmap v4452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:49 smithi107 ceph-mon[32659]: pgmap v4452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:49 smithi132 ceph-mon[37580]: pgmap v4452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:51 smithi112 ceph-mon[37501]: pgmap v4453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7486.914526 secs 2024-07-26T14:55:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:51 smithi107 ceph-mon[32659]: pgmap v4453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7486.914526 secs 2024-07-26T14:55:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:51 smithi132 ceph-mon[37580]: pgmap v4453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7486.914526 secs 2024-07-26T14:55:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:53 smithi112 ceph-mon[37501]: pgmap v4454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:53 smithi107 ceph-mon[32659]: pgmap v4454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:53 smithi132 ceph-mon[37580]: pgmap v4454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:55:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:55 smithi112 ceph-mon[37501]: pgmap v4455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:55 smithi107 ceph-mon[32659]: pgmap v4455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:55 smithi132 ceph-mon[37580]: pgmap v4455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7491.914666 secs 2024-07-26T14:55:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7491.914666 secs 2024-07-26T14:55:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7491.914666 secs 2024-07-26T14:55:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:57 smithi112 ceph-mon[37501]: pgmap v4456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:57 smithi107 ceph-mon[32659]: pgmap v4456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:57 smithi132 ceph-mon[37580]: pgmap v4456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:58.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:58.158+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:58.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:58.158+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:55:58.159 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:55:58.158+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:55:59 smithi107 ceph-mon[32659]: pgmap v4457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:55:59 smithi132 ceph-mon[37580]: pgmap v4457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:55:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:55:59 smithi112 ceph-mon[37501]: pgmap v4457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:00.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:00.394+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:00.394+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:00.395 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:00.394+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:01 smithi107 ceph-mon[32659]: pgmap v4458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7496.914766 secs 2024-07-26T14:56:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:01 smithi132 ceph-mon[37580]: pgmap v4458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7496.914766 secs 2024-07-26T14:56:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:01 smithi112 ceph-mon[37501]: pgmap v4458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7496.914766 secs 2024-07-26T14:56:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:03 smithi107 ceph-mon[32659]: pgmap v4459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:03 smithi132 ceph-mon[37580]: pgmap v4459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:03 smithi112 ceph-mon[37501]: pgmap v4459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:05 smithi107 ceph-mon[32659]: pgmap v4460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:05 smithi132 ceph-mon[37580]: pgmap v4460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:05 smithi112 ceph-mon[37501]: pgmap v4460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:06.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7501.914886 secs 2024-07-26T14:56:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7501.914886 secs 2024-07-26T14:56:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7501.914886 secs 2024-07-26T14:56:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:07 smithi107 ceph-mon[32659]: pgmap v4461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:07 smithi132 ceph-mon[37580]: pgmap v4461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:07.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:07 smithi112 ceph-mon[37501]: pgmap v4461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:09 smithi107 ceph-mon[32659]: pgmap v4462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:09 smithi132 ceph-mon[37580]: pgmap v4462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:09 smithi112 ceph-mon[37501]: pgmap v4462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:11 smithi107 ceph-mon[32659]: pgmap v4463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7506.914977 secs 2024-07-26T14:56:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:11 smithi132 ceph-mon[37580]: pgmap v4463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7506.914977 secs 2024-07-26T14:56:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:11 smithi112 ceph-mon[37501]: pgmap v4463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7506.914977 secs 2024-07-26T14:56:13.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:13.161+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:13.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:13.161+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:13.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:13.161+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:13 smithi107 ceph-mon[32659]: pgmap v4464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:13 smithi132 ceph-mon[37580]: pgmap v4464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:13 smithi112 ceph-mon[37501]: pgmap v4464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:15.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:15.405+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:15.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:15.405+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:15.407 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:15.405+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:15 smithi107 ceph-mon[32659]: pgmap v4465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:15 smithi132 ceph-mon[37580]: pgmap v4465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:15 smithi112 ceph-mon[37501]: pgmap v4465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:16.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7511.915071 secs 2024-07-26T14:56:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:56:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7511.915071 secs 2024-07-26T14:56:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:56:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7511.915071 secs 2024-07-26T14:56:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: pgmap v4466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:56:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:56:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:56:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: pgmap v4466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:56:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:56:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:56:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: pgmap v4466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:17.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:56:17.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:56:17.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:56:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:18 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:56:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:18 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:56:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:18 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:56:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:18 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:56:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:18 smithi107 ceph-mon[32659]: pgmap v4467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:56:18.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:18 smithi107 ceph-mon[32659]: pgmap v4468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:18 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:56:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:18 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:56:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:18 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:56:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:18 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:56:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:18 smithi132 ceph-mon[37580]: pgmap v4467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:56:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:18 smithi132 ceph-mon[37580]: pgmap v4468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:18 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T14:56:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:18 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T14:56:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:18 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T14:56:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:18 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T14:56:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:18 smithi112 ceph-mon[37501]: pgmap v4467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:56:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:18 smithi112 ceph-mon[37501]: pgmap v4468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:20 smithi107 ceph-mon[32659]: pgmap v4469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7516.915150 secs 2024-07-26T14:56:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:20 smithi132 ceph-mon[37580]: pgmap v4469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:20.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7516.915150 secs 2024-07-26T14:56:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:20 smithi112 ceph-mon[37501]: pgmap v4469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7516.915150 secs 2024-07-26T14:56:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:22 smithi107 ceph-mon[32659]: pgmap v4470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:22 smithi132 ceph-mon[37580]: pgmap v4470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:22 smithi112 ceph-mon[37501]: pgmap v4470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:24 smithi132 ceph-mon[37580]: pgmap v4471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:24 smithi112 ceph-mon[37501]: pgmap v4471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:24.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:24 smithi107 ceph-mon[32659]: pgmap v4471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7521.915231 secs 2024-07-26T14:56:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7521.915231 secs 2024-07-26T14:56:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7521.915231 secs 2024-07-26T14:56:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:26 smithi132 ceph-mon[37580]: pgmap v4472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:26 smithi112 ceph-mon[37501]: pgmap v4472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:26 smithi107 ceph-mon[32659]: pgmap v4472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:28.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:28.175+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:28.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:28.175+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:28.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:28.175+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:28 smithi107 ceph-mon[32659]: pgmap v4473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:56:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:28 smithi112 ceph-mon[37501]: pgmap v4473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:56:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:28 smithi132 ceph-mon[37580]: pgmap v4473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:56:30.413 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:30.411+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:30.413 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:30.411+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:30.413 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:30.411+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:30.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:30 smithi132 ceph-mon[37580]: pgmap v4474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:56:30.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7526.915345 secs 2024-07-26T14:56:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:30 smithi112 ceph-mon[37501]: pgmap v4474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:56:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7526.915345 secs 2024-07-26T14:56:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:30 smithi107 ceph-mon[32659]: pgmap v4474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:56:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7526.915345 secs 2024-07-26T14:56:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:32 smithi112 ceph-mon[37501]: pgmap v4475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:32 smithi107 ceph-mon[32659]: pgmap v4475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:32 smithi132 ceph-mon[37580]: pgmap v4475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:34 smithi112 ceph-mon[37501]: pgmap v4476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:34 smithi107 ceph-mon[32659]: pgmap v4476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:34 smithi132 ceph-mon[37580]: pgmap v4476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7531.915470 secs 2024-07-26T14:56:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7531.915470 secs 2024-07-26T14:56:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7531.915470 secs 2024-07-26T14:56:36.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:36 smithi107 ceph-mon[32659]: pgmap v4477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:36 smithi112 ceph-mon[37501]: pgmap v4477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:36 smithi132 ceph-mon[37580]: pgmap v4477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:38 smithi112 ceph-mon[37501]: pgmap v4478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:38 smithi107 ceph-mon[32659]: pgmap v4478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:38 smithi132 ceph-mon[37580]: pgmap v4478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:40 smithi112 ceph-mon[37501]: pgmap v4479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7536.915590 secs 2024-07-26T14:56:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:40 smithi107 ceph-mon[32659]: pgmap v4479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7536.915590 secs 2024-07-26T14:56:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:40 smithi132 ceph-mon[37580]: pgmap v4479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7536.915590 secs 2024-07-26T14:56:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:42 smithi112 ceph-mon[37501]: pgmap v4480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:42 smithi107 ceph-mon[32659]: pgmap v4480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:42 smithi132 ceph-mon[37580]: pgmap v4480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:43.184 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:43.183+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:43.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:43.184+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:43.185 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:43.184+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:44 smithi112 ceph-mon[37501]: pgmap v4481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:44 smithi107 ceph-mon[32659]: pgmap v4481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:44 smithi132 ceph-mon[37580]: pgmap v4481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:45.428+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:45.428+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:56:45.428+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7541.915711 secs 2024-07-26T14:56:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7541.915711 secs 2024-07-26T14:56:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7541.915711 secs 2024-07-26T14:56:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:46 smithi112 ceph-mon[37501]: pgmap v4482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:46 smithi107 ceph-mon[32659]: pgmap v4482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:46 smithi132 ceph-mon[37580]: pgmap v4482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:48 smithi112 ceph-mon[37501]: pgmap v4483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:48 smithi107 ceph-mon[32659]: pgmap v4483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:48 smithi132 ceph-mon[37580]: pgmap v4483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:50 smithi112 ceph-mon[37501]: pgmap v4484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7546.915861 secs 2024-07-26T14:56:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:50 smithi107 ceph-mon[32659]: pgmap v4484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7546.915861 secs 2024-07-26T14:56:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:50 smithi132 ceph-mon[37580]: pgmap v4484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:56:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7546.915861 secs 2024-07-26T14:56:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:52 smithi107 ceph-mon[32659]: pgmap v4485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:52 smithi112 ceph-mon[37501]: pgmap v4485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:52 smithi132 ceph-mon[37580]: pgmap v4485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:54 smithi112 ceph-mon[37501]: pgmap v4486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:54 smithi107 ceph-mon[32659]: pgmap v4486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:54 smithi132 ceph-mon[37580]: pgmap v4486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7551.916012 secs 2024-07-26T14:56:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7551.916012 secs 2024-07-26T14:56:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7551.916012 secs 2024-07-26T14:56:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:56 smithi112 ceph-mon[37501]: pgmap v4487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:56 smithi107 ceph-mon[32659]: pgmap v4487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:56 smithi132 ceph-mon[37580]: pgmap v4487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:58.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:58.199+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:58.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:58.199+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:58.200 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:56:58.199+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:56:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:56:58 smithi107 ceph-mon[32659]: pgmap v4488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:56:58 smithi112 ceph-mon[37501]: pgmap v4488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:56:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:56:58 smithi132 ceph-mon[37580]: pgmap v4488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:00.440+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:00.440+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:00.440+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:00 smithi112 ceph-mon[37501]: pgmap v4489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7556.916128 secs 2024-07-26T14:57:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:00 smithi107 ceph-mon[32659]: pgmap v4489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7556.916128 secs 2024-07-26T14:57:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:00 smithi132 ceph-mon[37580]: pgmap v4489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7556.916128 secs 2024-07-26T14:57:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:02 smithi112 ceph-mon[37501]: pgmap v4490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:02 smithi107 ceph-mon[32659]: pgmap v4490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:02 smithi132 ceph-mon[37580]: pgmap v4490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:04 smithi112 ceph-mon[37501]: pgmap v4491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:04 smithi107 ceph-mon[32659]: pgmap v4491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:04 smithi132 ceph-mon[37580]: pgmap v4491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7561.916210 secs 2024-07-26T14:57:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7561.916210 secs 2024-07-26T14:57:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7561.916210 secs 2024-07-26T14:57:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:06 smithi107 ceph-mon[32659]: pgmap v4492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:06 smithi132 ceph-mon[37580]: pgmap v4492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:06 smithi112 ceph-mon[37501]: pgmap v4492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:08 smithi107 ceph-mon[32659]: pgmap v4493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:08 smithi132 ceph-mon[37580]: pgmap v4493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:08 smithi112 ceph-mon[37501]: pgmap v4493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:10 smithi107 ceph-mon[32659]: pgmap v4494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7566.916317 secs 2024-07-26T14:57:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:10 smithi132 ceph-mon[37580]: pgmap v4494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7566.916317 secs 2024-07-26T14:57:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:10 smithi112 ceph-mon[37501]: pgmap v4494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7566.916317 secs 2024-07-26T14:57:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:12 smithi107 ceph-mon[32659]: pgmap v4495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:12 smithi132 ceph-mon[37580]: pgmap v4495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:12 smithi112 ceph-mon[37501]: pgmap v4495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:13.214+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:13.214+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:13.216 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:13.214+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:13.917 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:13.916+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:57:13.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:13.916+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T14:57:13.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:13.930+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:57:13.933 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:13.931+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T14:57:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:14 smithi107 ceph-mon[32659]: pgmap v4496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:14 smithi132 ceph-mon[37580]: pgmap v4496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:14 smithi112 ceph-mon[37501]: pgmap v4496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:15.095+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:57:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:15.095+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:57:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:15.096+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T14:57:15.098 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:15.096+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T14:57:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:15.455+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:15.456 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:15.455+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:15.457 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:15.455+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:16 smithi112 ceph-mon[37501]: pgmap v4497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7571.916443 secs 2024-07-26T14:57:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:16 smithi107 ceph-mon[32659]: pgmap v4497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7571.916443 secs 2024-07-26T14:57:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:16 smithi132 ceph-mon[37580]: pgmap v4497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7571.916443 secs 2024-07-26T14:57:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:18 smithi107 ceph-mon[32659]: pgmap v4498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:57:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:18 smithi132 ceph-mon[37580]: pgmap v4498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:57:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:18 smithi112 ceph-mon[37501]: pgmap v4498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:57:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:57:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:57:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:19 smithi107 ceph-mon[32659]: pgmap v4499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:57:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:19 smithi107 ceph-mon[32659]: pgmap v4500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:57:19.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:57:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:57:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:19 smithi132 ceph-mon[37580]: pgmap v4499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:57:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:19 smithi132 ceph-mon[37580]: pgmap v4500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:57:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:57:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:57:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:19 smithi112 ceph-mon[37501]: pgmap v4499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:57:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:19 smithi112 ceph-mon[37501]: pgmap v4500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:57:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7576.916542 secs 2024-07-26T14:57:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7576.916542 secs 2024-07-26T14:57:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7576.916542 secs 2024-07-26T14:57:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:21 smithi112 ceph-mon[37501]: pgmap v4501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:21 smithi107 ceph-mon[32659]: pgmap v4501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:21 smithi132 ceph-mon[37580]: pgmap v4501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:23 smithi112 ceph-mon[37501]: pgmap v4502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:57:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:23 smithi107 ceph-mon[32659]: pgmap v4502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:57:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:23 smithi132 ceph-mon[37580]: pgmap v4502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:57:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:25 smithi112 ceph-mon[37501]: pgmap v4503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7581.916685 secs 2024-07-26T14:57:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:25 smithi107 ceph-mon[32659]: pgmap v4503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7581.916685 secs 2024-07-26T14:57:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:25 smithi132 ceph-mon[37580]: pgmap v4503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7581.916685 secs 2024-07-26T14:57:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:27 smithi112 ceph-mon[37501]: pgmap v4504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:57:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:27 smithi107 ceph-mon[32659]: pgmap v4504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:57:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:27 smithi132 ceph-mon[37580]: pgmap v4504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:57:28.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:28.222+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:28.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:28.222+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:28.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:28.222+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:29 smithi107 ceph-mon[32659]: pgmap v4505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:57:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:29 smithi132 ceph-mon[37580]: pgmap v4505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:57:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:29 smithi112 ceph-mon[37501]: pgmap v4505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:57:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:30.470+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:30.470+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:30.472 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:30.470+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7586.916825 secs 2024-07-26T14:57:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7586.916825 secs 2024-07-26T14:57:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7586.916825 secs 2024-07-26T14:57:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:31 smithi107 ceph-mon[32659]: pgmap v4506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:31 smithi132 ceph-mon[37580]: pgmap v4506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:31 smithi112 ceph-mon[37501]: pgmap v4506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:33 smithi107 ceph-mon[32659]: pgmap v4507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:33 smithi132 ceph-mon[37580]: pgmap v4507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:33 smithi112 ceph-mon[37501]: pgmap v4507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:35 smithi107 ceph-mon[32659]: pgmap v4508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:57:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7591.916938 secs 2024-07-26T14:57:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:35 smithi132 ceph-mon[37580]: pgmap v4508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:57:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7591.916938 secs 2024-07-26T14:57:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:35 smithi112 ceph-mon[37501]: pgmap v4508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:57:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7591.916938 secs 2024-07-26T14:57:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:37 smithi107 ceph-mon[32659]: pgmap v4509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:37 smithi132 ceph-mon[37580]: pgmap v4509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:37 smithi112 ceph-mon[37501]: pgmap v4509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:39 smithi107 ceph-mon[32659]: pgmap v4510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:39.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:39 smithi132 ceph-mon[37580]: pgmap v4510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:39 smithi112 ceph-mon[37501]: pgmap v4510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7596.917085 secs 2024-07-26T14:57:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7596.917085 secs 2024-07-26T14:57:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7596.917085 secs 2024-07-26T14:57:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:41 smithi107 ceph-mon[32659]: pgmap v4511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:41 smithi132 ceph-mon[37580]: pgmap v4511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:41 smithi112 ceph-mon[37501]: pgmap v4511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:43.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:43.238+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:43.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:43.238+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:43.240 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:43.238+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:43 smithi107 ceph-mon[32659]: pgmap v4512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:43 smithi132 ceph-mon[37580]: pgmap v4512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:43 smithi112 ceph-mon[37501]: pgmap v4512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:45.482+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:45.482+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:57:45.482+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:45 smithi107 ceph-mon[32659]: pgmap v4513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:57:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7601.917197 secs 2024-07-26T14:57:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:45 smithi132 ceph-mon[37580]: pgmap v4513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:57:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7601.917197 secs 2024-07-26T14:57:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:45 smithi112 ceph-mon[37501]: pgmap v4513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:57:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7601.917197 secs 2024-07-26T14:57:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:47 smithi107 ceph-mon[32659]: pgmap v4514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:47 smithi132 ceph-mon[37580]: pgmap v4514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:47 smithi112 ceph-mon[37501]: pgmap v4514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:49 smithi107 ceph-mon[32659]: pgmap v4515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:49 smithi132 ceph-mon[37580]: pgmap v4515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:49 smithi112 ceph-mon[37501]: pgmap v4515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7606.917318 secs 2024-07-26T14:57:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:50 smithi107 ceph-mon[32659]: pgmap v4516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7606.917318 secs 2024-07-26T14:57:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:50 smithi132 ceph-mon[37580]: pgmap v4516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7606.917318 secs 2024-07-26T14:57:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:50 smithi112 ceph-mon[37501]: pgmap v4516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:53 smithi107 ceph-mon[32659]: pgmap v4517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:53 smithi132 ceph-mon[37580]: pgmap v4517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:53 smithi112 ceph-mon[37501]: pgmap v4517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:55 smithi107 ceph-mon[32659]: pgmap v4518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:57:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7611.917436 secs 2024-07-26T14:57:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:55 smithi132 ceph-mon[37580]: pgmap v4518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:57:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7611.917436 secs 2024-07-26T14:57:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:55 smithi112 ceph-mon[37501]: pgmap v4518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:57:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7611.917436 secs 2024-07-26T14:57:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:57 smithi107 ceph-mon[32659]: pgmap v4519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:57 smithi132 ceph-mon[37580]: pgmap v4519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:57 smithi112 ceph-mon[37501]: pgmap v4519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:58.247+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:58.247+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:58.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:57:58.247+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:57:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:57:59 smithi107 ceph-mon[32659]: pgmap v4520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:57:59 smithi132 ceph-mon[37580]: pgmap v4520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:57:59 smithi112 ceph-mon[37501]: pgmap v4520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:00.498+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:00.498+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:00.498+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7616.917515 secs 2024-07-26T14:58:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7616.917515 secs 2024-07-26T14:58:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7616.917515 secs 2024-07-26T14:58:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:01 smithi107 ceph-mon[32659]: pgmap v4521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:01 smithi132 ceph-mon[37580]: pgmap v4521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:01 smithi112 ceph-mon[37501]: pgmap v4521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:03 smithi107 ceph-mon[32659]: pgmap v4522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:03 smithi132 ceph-mon[37580]: pgmap v4522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:03 smithi112 ceph-mon[37501]: pgmap v4522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:05 smithi107 ceph-mon[32659]: pgmap v4523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:58:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7621.917574 secs 2024-07-26T14:58:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:05 smithi132 ceph-mon[37580]: pgmap v4523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:58:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7621.917574 secs 2024-07-26T14:58:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:05 smithi112 ceph-mon[37501]: pgmap v4523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:58:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7621.917574 secs 2024-07-26T14:58:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:07 smithi107 ceph-mon[32659]: pgmap v4524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:58:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:07 smithi132 ceph-mon[37580]: pgmap v4524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:58:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:07 smithi112 ceph-mon[37501]: pgmap v4524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:58:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:09 smithi107 ceph-mon[32659]: pgmap v4525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:09 smithi132 ceph-mon[37580]: pgmap v4525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:09 smithi112 ceph-mon[37501]: pgmap v4525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7626.917694 secs 2024-07-26T14:58:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7626.917694 secs 2024-07-26T14:58:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7626.917694 secs 2024-07-26T14:58:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:11 smithi107 ceph-mon[32659]: pgmap v4526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:11 smithi132 ceph-mon[37580]: pgmap v4526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:11 smithi112 ceph-mon[37501]: pgmap v4526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:13.261+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:13.261+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:13.263 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:13.261+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:13 smithi107 ceph-mon[32659]: pgmap v4527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:13 smithi132 ceph-mon[37580]: pgmap v4527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:13 smithi112 ceph-mon[37501]: pgmap v4527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:14 smithi132 ceph-mon[37580]: pgmap v4528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:58:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:14 smithi112 ceph-mon[37501]: pgmap v4528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:58:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:14 smithi107 ceph-mon[32659]: pgmap v4528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T14:58:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:15.505+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:15.505+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:15.505+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7631.917822 secs 2024-07-26T14:58:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7631.917822 secs 2024-07-26T14:58:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7631.917822 secs 2024-07-26T14:58:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:16 smithi132 ceph-mon[37580]: pgmap v4529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:58:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:16 smithi112 ceph-mon[37501]: pgmap v4529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:58:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:16 smithi107 ceph-mon[32659]: pgmap v4529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:58:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:19 smithi107 ceph-mon[32659]: pgmap v4530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:58:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:58:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:58:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:19 smithi132 ceph-mon[37580]: pgmap v4530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:58:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:58:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:58:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:19 smithi112 ceph-mon[37501]: pgmap v4530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:58:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:58:19.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:58:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:58:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:58:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:58:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7636.917947 secs 2024-07-26T14:58:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:58:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:58:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:58:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7636.917947 secs 2024-07-26T14:58:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:58:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:58:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:58:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7636.917947 secs 2024-07-26T14:58:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:21 smithi107 ceph-mon[32659]: pgmap v4531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:58:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:21 smithi107 ceph-mon[32659]: pgmap v4532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:21 smithi132 ceph-mon[37580]: pgmap v4531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:58:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:21 smithi132 ceph-mon[37580]: pgmap v4532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:21 smithi112 ceph-mon[37501]: pgmap v4531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:58:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:21 smithi112 ceph-mon[37501]: pgmap v4532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:23.752 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:23 smithi112 ceph-mon[37501]: pgmap v4533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T14:58:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:23 smithi107 ceph-mon[32659]: pgmap v4533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T14:58:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:23 smithi132 ceph-mon[37580]: pgmap v4533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T14:58:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:25 smithi107 ceph-mon[32659]: pgmap v4534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:58:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7641.918062 secs 2024-07-26T14:58:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:25 smithi132 ceph-mon[37580]: pgmap v4534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:58:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7641.918062 secs 2024-07-26T14:58:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:25 smithi112 ceph-mon[37501]: pgmap v4534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:58:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7641.918062 secs 2024-07-26T14:58:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:27 smithi107 ceph-mon[32659]: pgmap v4535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:27 smithi132 ceph-mon[37580]: pgmap v4535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:27 smithi112 ceph-mon[37501]: pgmap v4535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:28.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:28.277+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:28.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:28.277+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:28.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:28.277+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:29 smithi107 ceph-mon[32659]: pgmap v4536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:29 smithi132 ceph-mon[37580]: pgmap v4536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:29 smithi112 ceph-mon[37501]: pgmap v4536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:30.521+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:30.521+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:30.523 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:30.521+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7646.918138 secs 2024-07-26T14:58:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7646.918138 secs 2024-07-26T14:58:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7646.918138 secs 2024-07-26T14:58:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:31 smithi107 ceph-mon[32659]: pgmap v4537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:58:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:31 smithi132 ceph-mon[37580]: pgmap v4537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:58:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:31 smithi112 ceph-mon[37501]: pgmap v4537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:58:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:33 smithi107 ceph-mon[32659]: pgmap v4538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:33 smithi132 ceph-mon[37580]: pgmap v4538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:33 smithi112 ceph-mon[37501]: pgmap v4538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:35 smithi107 ceph-mon[32659]: pgmap v4539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7651.918246 secs 2024-07-26T14:58:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:35 smithi132 ceph-mon[37580]: pgmap v4539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7651.918246 secs 2024-07-26T14:58:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:35 smithi112 ceph-mon[37501]: pgmap v4539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7651.918246 secs 2024-07-26T14:58:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:36 smithi112 ceph-mon[37501]: pgmap v4540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:36 smithi107 ceph-mon[32659]: pgmap v4540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:36 smithi132 ceph-mon[37580]: pgmap v4540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:39 smithi132 ceph-mon[37580]: pgmap v4541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:39 smithi112 ceph-mon[37501]: pgmap v4541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:39.745 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:39 smithi107 ceph-mon[32659]: pgmap v4541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7656.918366 secs 2024-07-26T14:58:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7656.918366 secs 2024-07-26T14:58:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7656.918366 secs 2024-07-26T14:58:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:41 smithi107 ceph-mon[32659]: pgmap v4542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:41 smithi112 ceph-mon[37501]: pgmap v4542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:41 smithi132 ceph-mon[37580]: pgmap v4542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:43.282+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:43.282+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:43.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:43.282+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:43 smithi107 ceph-mon[32659]: pgmap v4543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:43 smithi112 ceph-mon[37501]: pgmap v4543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:43 smithi132 ceph-mon[37580]: pgmap v4543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:45.534+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:45.534+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:58:45.534+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:45 smithi112 ceph-mon[37501]: pgmap v4544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7661.918484 secs 2024-07-26T14:58:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:45 smithi107 ceph-mon[32659]: pgmap v4544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7661.918484 secs 2024-07-26T14:58:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:45 smithi132 ceph-mon[37580]: pgmap v4544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7661.918484 secs 2024-07-26T14:58:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:47 smithi112 ceph-mon[37501]: pgmap v4545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:47 smithi107 ceph-mon[32659]: pgmap v4545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:47 smithi132 ceph-mon[37580]: pgmap v4545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:49 smithi112 ceph-mon[37501]: pgmap v4546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:49 smithi107 ceph-mon[32659]: pgmap v4546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:49 smithi132 ceph-mon[37580]: pgmap v4546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7666.918564 secs 2024-07-26T14:58:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7666.918564 secs 2024-07-26T14:58:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7666.918564 secs 2024-07-26T14:58:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:51 smithi112 ceph-mon[37501]: pgmap v4547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:51 smithi107 ceph-mon[32659]: pgmap v4547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:51 smithi132 ceph-mon[37580]: pgmap v4547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:53 smithi112 ceph-mon[37501]: pgmap v4548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:53 smithi107 ceph-mon[32659]: pgmap v4548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:53 smithi132 ceph-mon[37580]: pgmap v4548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:55 smithi112 ceph-mon[37501]: pgmap v4549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7671.918662 secs 2024-07-26T14:58:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:55 smithi107 ceph-mon[32659]: pgmap v4549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7671.918662 secs 2024-07-26T14:58:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:55 smithi132 ceph-mon[37580]: pgmap v4549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7671.918662 secs 2024-07-26T14:58:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:57 smithi112 ceph-mon[37501]: pgmap v4550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:57 smithi107 ceph-mon[32659]: pgmap v4550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:57 smithi132 ceph-mon[37580]: pgmap v4550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:58.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:58.284+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:58.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:58.284+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:58:58.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:58:58.284+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:58:59 smithi112 ceph-mon[37501]: pgmap v4551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:58:59 smithi107 ceph-mon[32659]: pgmap v4551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:58:59 smithi132 ceph-mon[37580]: pgmap v4551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:00.544+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:00.545+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:00.545 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:00.545+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7676.918787 secs 2024-07-26T14:59:00.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7676.918787 secs 2024-07-26T14:59:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7676.918787 secs 2024-07-26T14:59:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:01 smithi112 ceph-mon[37501]: pgmap v4552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:01 smithi107 ceph-mon[32659]: pgmap v4552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:01 smithi132 ceph-mon[37580]: pgmap v4552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:03 smithi112 ceph-mon[37501]: pgmap v4553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:03.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:03 smithi107 ceph-mon[32659]: pgmap v4553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:03 smithi132 ceph-mon[37580]: pgmap v4553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:05 smithi112 ceph-mon[37501]: pgmap v4554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7681.918908 secs 2024-07-26T14:59:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:05 smithi107 ceph-mon[32659]: pgmap v4554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7681.918908 secs 2024-07-26T14:59:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:05 smithi132 ceph-mon[37580]: pgmap v4554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7681.918908 secs 2024-07-26T14:59:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:07 smithi112 ceph-mon[37501]: pgmap v4555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:07 smithi107 ceph-mon[32659]: pgmap v4555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:07 smithi132 ceph-mon[37580]: pgmap v4555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:09 smithi112 ceph-mon[37501]: pgmap v4556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:09 smithi107 ceph-mon[32659]: pgmap v4556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:09 smithi132 ceph-mon[37580]: pgmap v4556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7686.918999 secs 2024-07-26T14:59:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7686.918999 secs 2024-07-26T14:59:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7686.918999 secs 2024-07-26T14:59:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:11 smithi112 ceph-mon[37501]: pgmap v4557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:11 smithi107 ceph-mon[32659]: pgmap v4557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:11 smithi132 ceph-mon[37580]: pgmap v4557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:13.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:13.289+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:13.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:13.289+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:13.291 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:13.289+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:13 smithi112 ceph-mon[37501]: pgmap v4558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:13 smithi107 ceph-mon[32659]: pgmap v4558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:13 smithi132 ceph-mon[37580]: pgmap v4558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:15.559+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:15.559+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:15.559+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:15 smithi112 ceph-mon[37501]: pgmap v4559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7691.919119 secs 2024-07-26T14:59:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:15 smithi107 ceph-mon[32659]: pgmap v4559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7691.919119 secs 2024-07-26T14:59:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:15 smithi132 ceph-mon[37580]: pgmap v4559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7691.919119 secs 2024-07-26T14:59:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:17 smithi112 ceph-mon[37501]: pgmap v4560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:17 smithi107 ceph-mon[32659]: pgmap v4560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:17 smithi132 ceph-mon[37580]: pgmap v4560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:19 smithi112 ceph-mon[37501]: pgmap v4561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:19 smithi107 ceph-mon[32659]: pgmap v4561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:19 smithi132 ceph-mon[37580]: pgmap v4561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7696.919221 secs 2024-07-26T14:59:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7696.919221 secs 2024-07-26T14:59:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7696.919221 secs 2024-07-26T14:59:21.758 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:21 smithi132 ceph-mon[37580]: pgmap v4562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:21.759 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:59:21.759 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.759 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.759 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.759 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:21 smithi107 ceph-mon[32659]: pgmap v4562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:59:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:21 smithi112 ceph-mon[37501]: pgmap v4562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:59:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:23 smithi112 ceph-mon[37501]: pgmap v4563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:23 smithi107 ceph-mon[32659]: pgmap v4563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:23 smithi132 ceph-mon[37580]: pgmap v4563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:24 smithi107 ceph-mon[32659]: pgmap v4564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:24 smithi132 ceph-mon[37580]: pgmap v4564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:24 smithi112 ceph-mon[37501]: pgmap v4564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:59:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:59:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:25 smithi132 ceph-mon[37580]: pgmap v4565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:59:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:25 smithi132 ceph-mon[37580]: pgmap v4566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T14:59:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7701.919344 secs 2024-07-26T14:59:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:59:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:59:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:25 smithi112 ceph-mon[37501]: pgmap v4565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:59:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:25 smithi112 ceph-mon[37501]: pgmap v4566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T14:59:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7701.919344 secs 2024-07-26T14:59:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:59:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:59:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:25 smithi107 ceph-mon[32659]: pgmap v4565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:59:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:25 smithi107 ceph-mon[32659]: pgmap v4566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T14:59:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T14:59:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7701.919344 secs 2024-07-26T14:59:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:27 smithi112 ceph-mon[37501]: pgmap v4567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T14:59:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:27 smithi107 ceph-mon[32659]: pgmap v4567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T14:59:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:28.305+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:28.305+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:28.305+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:27 smithi132 ceph-mon[37580]: pgmap v4567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T14:59:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:29 smithi112 ceph-mon[37501]: pgmap v4568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:59:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:29 smithi107 ceph-mon[32659]: pgmap v4568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:59:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:29 smithi132 ceph-mon[37580]: pgmap v4568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:59:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:30.572+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:30.572+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:30.572+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7706.919459 secs 2024-07-26T14:59:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7706.919459 secs 2024-07-26T14:59:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7706.919459 secs 2024-07-26T14:59:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:31 smithi112 ceph-mon[37501]: pgmap v4569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:31 smithi107 ceph-mon[32659]: pgmap v4569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:31 smithi132 ceph-mon[37580]: pgmap v4569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:33 smithi112 ceph-mon[37501]: pgmap v4570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:33 smithi107 ceph-mon[32659]: pgmap v4570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:33 smithi132 ceph-mon[37580]: pgmap v4570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:35 smithi112 ceph-mon[37501]: pgmap v4571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:59:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7711.919582 secs 2024-07-26T14:59:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:35 smithi107 ceph-mon[32659]: pgmap v4571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:59:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7711.919582 secs 2024-07-26T14:59:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:35 smithi132 ceph-mon[37580]: pgmap v4571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:59:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7711.919582 secs 2024-07-26T14:59:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:37 smithi112 ceph-mon[37501]: pgmap v4572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:37 smithi107 ceph-mon[32659]: pgmap v4572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:37 smithi132 ceph-mon[37580]: pgmap v4572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:39 smithi112 ceph-mon[37501]: pgmap v4573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:39 smithi107 ceph-mon[32659]: pgmap v4573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:39 smithi132 ceph-mon[37580]: pgmap v4573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7716.919715 secs 2024-07-26T14:59:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7716.919715 secs 2024-07-26T14:59:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7716.919715 secs 2024-07-26T14:59:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:41 smithi107 ceph-mon[32659]: pgmap v4574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:41 smithi112 ceph-mon[37501]: pgmap v4574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:41 smithi132 ceph-mon[37580]: pgmap v4574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:43.319+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:43.319+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:43.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:43.319+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:43 smithi112 ceph-mon[37501]: pgmap v4575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:43 smithi107 ceph-mon[32659]: pgmap v4575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:43 smithi132 ceph-mon[37580]: pgmap v4575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:45.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:45.589+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:45.590 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:45.589+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T14:59:45.589+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:45 smithi112 ceph-mon[37501]: pgmap v4576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7721.919811 secs 2024-07-26T14:59:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:45 smithi107 ceph-mon[32659]: pgmap v4576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7721.919811 secs 2024-07-26T14:59:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:45 smithi132 ceph-mon[37580]: pgmap v4576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7721.919811 secs 2024-07-26T14:59:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:47 smithi112 ceph-mon[37501]: pgmap v4577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:47 smithi107 ceph-mon[32659]: pgmap v4577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:47 smithi132 ceph-mon[37580]: pgmap v4577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:49 smithi112 ceph-mon[37501]: pgmap v4578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:49 smithi107 ceph-mon[32659]: pgmap v4578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:49 smithi132 ceph-mon[37580]: pgmap v4578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7726.919940 secs 2024-07-26T14:59:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7726.919940 secs 2024-07-26T14:59:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7726.919940 secs 2024-07-26T14:59:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:51 smithi107 ceph-mon[32659]: pgmap v4579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:51 smithi132 ceph-mon[37580]: pgmap v4579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:51 smithi112 ceph-mon[37501]: pgmap v4579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:54.252 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:53 smithi112 ceph-mon[37501]: pgmap v4580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:53 smithi107 ceph-mon[32659]: pgmap v4580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:54.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:53 smithi132 ceph-mon[37580]: pgmap v4580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:55 smithi107 ceph-mon[32659]: pgmap v4581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7731.920016 secs 2024-07-26T14:59:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:55 smithi132 ceph-mon[37580]: pgmap v4581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7731.920016 secs 2024-07-26T14:59:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:55 smithi112 ceph-mon[37501]: pgmap v4581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7731.920016 secs 2024-07-26T14:59:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:57 smithi107 ceph-mon[32659]: pgmap v4582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:58.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:57 smithi132 ceph-mon[37580]: pgmap v4582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:58.335+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:58.335+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T14:59:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T14:59:58.335+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:57 smithi112 ceph-mon[37501]: pgmap v4582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 14:59:59 smithi107 ceph-mon[32659]: pgmap v4583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:00.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 14:59:59 smithi132 ceph-mon[37580]: pgmap v4583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 14:59:59 smithi112 ceph-mon[37501]: pgmap v4583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:00.605+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:00.605+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:00.605+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:00:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7736.920138 secs 2024-07-26T15:00:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:00:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7736.920138 secs 2024-07-26T15:00:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:00:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7736.920138 secs 2024-07-26T15:00:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:01 smithi107 ceph-mon[32659]: pgmap v4584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:01 smithi132 ceph-mon[37580]: pgmap v4584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:01 smithi112 ceph-mon[37501]: pgmap v4584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:03 smithi107 ceph-mon[32659]: pgmap v4585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:04.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:03 smithi132 ceph-mon[37580]: pgmap v4585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:03 smithi112 ceph-mon[37501]: pgmap v4585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:06 smithi107 ceph-mon[32659]: pgmap v4586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7741.920260 secs 2024-07-26T15:00:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:06 smithi132 ceph-mon[37580]: pgmap v4586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7741.920260 secs 2024-07-26T15:00:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:06 smithi112 ceph-mon[37501]: pgmap v4586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7741.920260 secs 2024-07-26T15:00:08.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:08 smithi107 ceph-mon[32659]: pgmap v4587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:08.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:08 smithi132 ceph-mon[37580]: pgmap v4587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:08 smithi112 ceph-mon[37501]: pgmap v4587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:10 smithi107 ceph-mon[32659]: pgmap v4588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:10 smithi132 ceph-mon[37580]: pgmap v4588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:10 smithi112 ceph-mon[37501]: pgmap v4588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7746.920379 secs 2024-07-26T15:00:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7746.920379 secs 2024-07-26T15:00:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7746.920379 secs 2024-07-26T15:00:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:12 smithi107 ceph-mon[32659]: pgmap v4589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:12 smithi132 ceph-mon[37580]: pgmap v4589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:12 smithi112 ceph-mon[37501]: pgmap v4589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:13.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:13.349+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:13.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:13.349+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:13.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:13.349+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:13.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:13.947+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:00:13.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:13.947+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:00:13.961 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:13.959+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:00:13.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:13.960+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:00:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:14 smithi107 ceph-mon[32659]: pgmap v4590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:14 smithi132 ceph-mon[37580]: pgmap v4590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:14 smithi112 ceph-mon[37501]: pgmap v4590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:15.611+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:15.611+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:15.613 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:15.611+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:16.127+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:00:16.129 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:16.128+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:16.131+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:16.132+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:00:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:16 smithi107 ceph-mon[32659]: pgmap v4591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7751.920501 secs 2024-07-26T15:00:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:16 smithi132 ceph-mon[37580]: pgmap v4591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7751.920501 secs 2024-07-26T15:00:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:16 smithi112 ceph-mon[37501]: pgmap v4591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7751.920501 secs 2024-07-26T15:00:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:18 smithi132 ceph-mon[37580]: pgmap v4592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:18 smithi112 ceph-mon[37501]: pgmap v4592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:18 smithi107 ceph-mon[32659]: pgmap v4592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:20 smithi132 ceph-mon[37580]: pgmap v4593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:20 smithi112 ceph-mon[37501]: pgmap v4593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:20 smithi107 ceph-mon[32659]: pgmap v4593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7756.920655 secs 2024-07-26T15:00:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7756.920655 secs 2024-07-26T15:00:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7756.920655 secs 2024-07-26T15:00:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:22 smithi112 ceph-mon[37501]: pgmap v4594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:22 smithi107 ceph-mon[32659]: pgmap v4594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:22 smithi132 ceph-mon[37580]: pgmap v4594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:24 smithi112 ceph-mon[37501]: pgmap v4595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:24 smithi107 ceph-mon[32659]: pgmap v4595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:24 smithi132 ceph-mon[37580]: pgmap v4595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:00:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:00:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:00:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:26 smithi112 ceph-mon[37501]: pgmap v4596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7761.920726 secs 2024-07-26T15:00:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:00:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:00:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:00:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:26 smithi107 ceph-mon[32659]: pgmap v4596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7761.920726 secs 2024-07-26T15:00:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:00:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:00:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:00:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:26 smithi132 ceph-mon[37580]: pgmap v4596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7761.920726 secs 2024-07-26T15:00:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:00:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:00:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:00:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:27 smithi112 ceph-mon[37501]: pgmap v4597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:00:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:27 smithi112 ceph-mon[37501]: pgmap v4598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:00:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:27 smithi107 ceph-mon[32659]: pgmap v4597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:00:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:27 smithi107 ceph-mon[32659]: pgmap v4598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:00:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:27 smithi132 ceph-mon[37580]: pgmap v4597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:00:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:27 smithi132 ceph-mon[37580]: pgmap v4598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:00:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:28.351+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:28.351+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:28.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:28.351+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:29 smithi112 ceph-mon[37501]: pgmap v4599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:00:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:29 smithi107 ceph-mon[32659]: pgmap v4599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:00:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:29 smithi132 ceph-mon[37580]: pgmap v4599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:00:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:30.613+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:30.613+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:30.615 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:30.613+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:31.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:31 smithi132 ceph-mon[37580]: pgmap v4600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:00:31.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7766.920849 secs 2024-07-26T15:00:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:31 smithi112 ceph-mon[37501]: pgmap v4600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:00:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7766.920849 secs 2024-07-26T15:00:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:31 smithi107 ceph-mon[32659]: pgmap v4600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:00:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7766.920849 secs 2024-07-26T15:00:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:33 smithi112 ceph-mon[37501]: pgmap v4601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:33 smithi107 ceph-mon[32659]: pgmap v4601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:33 smithi132 ceph-mon[37580]: pgmap v4601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:35 smithi112 ceph-mon[37501]: pgmap v4602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:00:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:35 smithi107 ceph-mon[32659]: pgmap v4602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:00:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:35 smithi132 ceph-mon[37580]: pgmap v4602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:00:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7771.920987 secs 2024-07-26T15:00:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7771.920987 secs 2024-07-26T15:00:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7771.920987 secs 2024-07-26T15:00:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:37 smithi112 ceph-mon[37501]: pgmap v4603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:00:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:37 smithi107 ceph-mon[32659]: pgmap v4603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:00:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:37 smithi132 ceph-mon[37580]: pgmap v4603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:00:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:39 smithi112 ceph-mon[37501]: pgmap v4604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:39 smithi107 ceph-mon[32659]: pgmap v4604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:39 smithi132 ceph-mon[37580]: pgmap v4604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:41 smithi112 ceph-mon[37501]: pgmap v4605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7776.921091 secs 2024-07-26T15:00:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:41 smithi107 ceph-mon[32659]: pgmap v4605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7776.921091 secs 2024-07-26T15:00:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:41 smithi132 ceph-mon[37580]: pgmap v4605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7776.921091 secs 2024-07-26T15:00:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:43.356+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:43.357+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:43.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:43.357+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:43 smithi112 ceph-mon[37501]: pgmap v4606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:43 smithi107 ceph-mon[32659]: pgmap v4606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:43 smithi132 ceph-mon[37580]: pgmap v4606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:45 smithi112 ceph-mon[37501]: pgmap v4607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:45 smithi107 ceph-mon[32659]: pgmap v4607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:45 smithi132 ceph-mon[37580]: pgmap v4607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:45.616+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:45.616+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:00:45.616+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7781.921217 secs 2024-07-26T15:00:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7781.921217 secs 2024-07-26T15:00:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7781.921217 secs 2024-07-26T15:00:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:47 smithi112 ceph-mon[37501]: pgmap v4608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:47 smithi107 ceph-mon[32659]: pgmap v4608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:47 smithi132 ceph-mon[37580]: pgmap v4608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:49 smithi112 ceph-mon[37501]: pgmap v4609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:49 smithi107 ceph-mon[32659]: pgmap v4609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:49 smithi132 ceph-mon[37580]: pgmap v4609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:51 smithi107 ceph-mon[32659]: pgmap v4610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7786.921317 secs 2024-07-26T15:00:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:51 smithi132 ceph-mon[37580]: pgmap v4610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7786.921317 secs 2024-07-26T15:00:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:51 smithi112 ceph-mon[37501]: pgmap v4610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7786.921317 secs 2024-07-26T15:00:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:53 smithi107 ceph-mon[32659]: pgmap v4611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:53 smithi132 ceph-mon[37580]: pgmap v4611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:53 smithi112 ceph-mon[37501]: pgmap v4611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:55.502 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:55 smithi112 ceph-mon[37501]: pgmap v4612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:55.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:55 smithi107 ceph-mon[32659]: pgmap v4612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:55 smithi132 ceph-mon[37580]: pgmap v4612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7791.921416 secs 2024-07-26T15:00:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7791.921416 secs 2024-07-26T15:00:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7791.921416 secs 2024-07-26T15:00:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:57 smithi107 ceph-mon[32659]: pgmap v4613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:57 smithi132 ceph-mon[37580]: pgmap v4613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:57 smithi112 ceph-mon[37501]: pgmap v4613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:58.372+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:58.372+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:58.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:00:58.372+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:00:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:00:59 smithi107 ceph-mon[32659]: pgmap v4614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:00:59 smithi132 ceph-mon[37580]: pgmap v4614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:00:59 smithi112 ceph-mon[37501]: pgmap v4614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:00.632 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:00.632+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:00.633 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:00.632+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:00.633 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:00.632+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:01 smithi132 ceph-mon[37580]: pgmap v4615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7796.921541 secs 2024-07-26T15:01:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:01 smithi107 ceph-mon[32659]: pgmap v4615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7796.921541 secs 2024-07-26T15:01:01.651 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:01 smithi112 ceph-mon[37501]: pgmap v4615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:01.651 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7796.921541 secs 2024-07-26T15:01:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:03 smithi107 ceph-mon[32659]: pgmap v4616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:03 smithi132 ceph-mon[37580]: pgmap v4616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:03 smithi112 ceph-mon[37501]: pgmap v4616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:05 smithi107 ceph-mon[32659]: pgmap v4617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:05 smithi132 ceph-mon[37580]: pgmap v4617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:05 smithi112 ceph-mon[37501]: pgmap v4617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7801.921633 secs 2024-07-26T15:01:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7801.921633 secs 2024-07-26T15:01:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7801.921633 secs 2024-07-26T15:01:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:07 smithi132 ceph-mon[37580]: pgmap v4618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:07 smithi112 ceph-mon[37501]: pgmap v4618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:07 smithi107 ceph-mon[32659]: pgmap v4618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:09 smithi132 ceph-mon[37580]: pgmap v4619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:09 smithi112 ceph-mon[37501]: pgmap v4619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:09 smithi107 ceph-mon[32659]: pgmap v4619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7806.921752 secs 2024-07-26T15:01:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7806.921752 secs 2024-07-26T15:01:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7806.921752 secs 2024-07-26T15:01:11.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:11 smithi107 ceph-mon[32659]: pgmap v4620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:11 smithi112 ceph-mon[37501]: pgmap v4620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:11 smithi132 ceph-mon[37580]: pgmap v4620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:13.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:13.387+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:13.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:13.387+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:13.389 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:13.387+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:13 smithi112 ceph-mon[37501]: pgmap v4621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:13 smithi107 ceph-mon[32659]: pgmap v4621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:13 smithi132 ceph-mon[37580]: pgmap v4621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:15.642+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:15.642+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:15.642+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:15 smithi112 ceph-mon[37501]: pgmap v4622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7811.921820 secs 2024-07-26T15:01:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:15 smithi107 ceph-mon[32659]: pgmap v4622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7811.921820 secs 2024-07-26T15:01:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:15 smithi132 ceph-mon[37580]: pgmap v4622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7811.921820 secs 2024-07-26T15:01:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:17 smithi112 ceph-mon[37501]: pgmap v4623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:17 smithi107 ceph-mon[32659]: pgmap v4623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:17 smithi132 ceph-mon[37580]: pgmap v4623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:19 smithi112 ceph-mon[37501]: pgmap v4624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:19 smithi107 ceph-mon[32659]: pgmap v4624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:19 smithi132 ceph-mon[37580]: pgmap v4624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7816.921944 secs 2024-07-26T15:01:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7816.921944 secs 2024-07-26T15:01:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7816.921944 secs 2024-07-26T15:01:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:21 smithi112 ceph-mon[37501]: pgmap v4625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:21 smithi107 ceph-mon[32659]: pgmap v4625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:21 smithi132 ceph-mon[37580]: pgmap v4625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:23 smithi112 ceph-mon[37501]: pgmap v4626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:23 smithi107 ceph-mon[32659]: pgmap v4626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:23 smithi132 ceph-mon[37580]: pgmap v4626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:25 smithi112 ceph-mon[37501]: pgmap v4627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7821.922039 secs 2024-07-26T15:01:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:25 smithi107 ceph-mon[32659]: pgmap v4627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7821.922039 secs 2024-07-26T15:01:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:25 smithi132 ceph-mon[37580]: pgmap v4627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7821.922039 secs 2024-07-26T15:01:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:01:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:01:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:01:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:27 smithi112 ceph-mon[37501]: pgmap v4628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:01:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:01:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:01:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:27 smithi107 ceph-mon[32659]: pgmap v4628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:01:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:01:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:01:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:27 smithi132 ceph-mon[37580]: pgmap v4628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:01:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:01:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:01:28.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:28.403+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:28.404 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:28.403+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:28.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:28.403+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:28 smithi112 ceph-mon[37501]: pgmap v4629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:01:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:28 smithi112 ceph-mon[37501]: pgmap v4630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:28 smithi107 ceph-mon[32659]: pgmap v4629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:01:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:28 smithi107 ceph-mon[32659]: pgmap v4630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:28 smithi132 ceph-mon[37580]: pgmap v4629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:01:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:28 smithi132 ceph-mon[37580]: pgmap v4630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:30.659 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:30.658+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:30.658+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:30.660 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:30.658+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:30 smithi112 ceph-mon[37501]: pgmap v4631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7826.922159 secs 2024-07-26T15:01:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:30 smithi107 ceph-mon[32659]: pgmap v4631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7826.922159 secs 2024-07-26T15:01:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:30 smithi132 ceph-mon[37580]: pgmap v4631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7826.922159 secs 2024-07-26T15:01:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:32 smithi112 ceph-mon[37501]: pgmap v4632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:01:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:32 smithi107 ceph-mon[32659]: pgmap v4632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:01:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:32 smithi132 ceph-mon[37580]: pgmap v4632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:01:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:34 smithi112 ceph-mon[37501]: pgmap v4633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:34 smithi107 ceph-mon[32659]: pgmap v4633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:34 smithi132 ceph-mon[37580]: pgmap v4633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7831.922255 secs 2024-07-26T15:01:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7831.922255 secs 2024-07-26T15:01:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7831.922255 secs 2024-07-26T15:01:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:36 smithi112 ceph-mon[37501]: pgmap v4634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:36 smithi107 ceph-mon[32659]: pgmap v4634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:36 smithi132 ceph-mon[37580]: pgmap v4634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:38 smithi112 ceph-mon[37501]: pgmap v4635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:01:38.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:38 smithi107 ceph-mon[32659]: pgmap v4635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:01:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:38 smithi132 ceph-mon[37580]: pgmap v4635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:01:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:40 smithi112 ceph-mon[37501]: pgmap v4636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7836.922372 secs 2024-07-26T15:01:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:40 smithi107 ceph-mon[32659]: pgmap v4636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7836.922372 secs 2024-07-26T15:01:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:40 smithi132 ceph-mon[37580]: pgmap v4636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:01:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7836.922372 secs 2024-07-26T15:01:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:42 smithi112 ceph-mon[37501]: pgmap v4637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:42 smithi107 ceph-mon[32659]: pgmap v4637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:42 smithi132 ceph-mon[37580]: pgmap v4637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:43.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:43.419+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:43.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:43.419+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:43.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:43.419+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:44 smithi112 ceph-mon[37501]: pgmap v4638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:01:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:44 smithi107 ceph-mon[32659]: pgmap v4638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:01:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:44 smithi132 ceph-mon[37580]: pgmap v4638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:01:45.673 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:45.672+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:45.673 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:45.672+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:45.673 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:01:45.672+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7841.922493 secs 2024-07-26T15:01:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7841.922493 secs 2024-07-26T15:01:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7841.922493 secs 2024-07-26T15:01:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:46 smithi107 ceph-mon[32659]: pgmap v4639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:01:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:46 smithi132 ceph-mon[37580]: pgmap v4639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:01:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:46 smithi112 ceph-mon[37501]: pgmap v4639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:01:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:48 smithi107 ceph-mon[32659]: pgmap v4640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:01:48.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:48 smithi132 ceph-mon[37580]: pgmap v4640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:01:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:48 smithi112 ceph-mon[37501]: pgmap v4640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:01:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:50 smithi107 ceph-mon[32659]: pgmap v4641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7846.922592 secs 2024-07-26T15:01:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:50 smithi132 ceph-mon[37580]: pgmap v4641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7846.922592 secs 2024-07-26T15:01:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:50 smithi112 ceph-mon[37501]: pgmap v4641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:01:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7846.922592 secs 2024-07-26T15:01:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:52 smithi107 ceph-mon[32659]: pgmap v4642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:01:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:52 smithi132 ceph-mon[37580]: pgmap v4642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:01:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:52 smithi112 ceph-mon[37501]: pgmap v4642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:01:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:54 smithi107 ceph-mon[32659]: pgmap v4643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:01:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:54 smithi132 ceph-mon[37580]: pgmap v4643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:01:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:54 smithi112 ceph-mon[37501]: pgmap v4643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:01:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7851.922683 secs 2024-07-26T15:01:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7851.922683 secs 2024-07-26T15:01:55.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7851.922683 secs 2024-07-26T15:01:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:56 smithi107 ceph-mon[32659]: pgmap v4644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:56 smithi132 ceph-mon[37580]: pgmap v4644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:56 smithi112 ceph-mon[37501]: pgmap v4644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:58.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:58.429+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:58.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:58.430+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:01:58.434 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:01:58.430+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:01:58 smithi107 ceph-mon[32659]: pgmap v4645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:01:58 smithi132 ceph-mon[37580]: pgmap v4645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:01:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:01:58 smithi112 ceph-mon[37501]: pgmap v4645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:02:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:00.688+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:00.688+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:00.688+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:00 smithi107 ceph-mon[32659]: pgmap v4646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:02:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7856.922821 secs 2024-07-26T15:02:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:00 smithi132 ceph-mon[37580]: pgmap v4646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:02:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7856.922821 secs 2024-07-26T15:02:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:00 smithi112 ceph-mon[37501]: pgmap v4646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:02:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7856.922821 secs 2024-07-26T15:02:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:02 smithi107 ceph-mon[32659]: pgmap v4647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:02:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:02 smithi132 ceph-mon[37580]: pgmap v4647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:02:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:02 smithi112 ceph-mon[37501]: pgmap v4647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:02:04.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:04 smithi107 ceph-mon[32659]: pgmap v4648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:02:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:04 smithi132 ceph-mon[37580]: pgmap v4648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:02:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:04 smithi112 ceph-mon[37501]: pgmap v4648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:02:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7861.922940 secs 2024-07-26T15:02:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7861.922940 secs 2024-07-26T15:02:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7861.922940 secs 2024-07-26T15:02:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:06 smithi107 ceph-mon[32659]: pgmap v4649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:02:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:06 smithi132 ceph-mon[37580]: pgmap v4649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:02:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:06 smithi112 ceph-mon[37501]: pgmap v4649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:02:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:08 smithi107 ceph-mon[32659]: pgmap v4650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:02:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:08 smithi132 ceph-mon[37580]: pgmap v4650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:02:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:08 smithi112 ceph-mon[37501]: pgmap v4650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:02:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:10 smithi132 ceph-mon[37580]: pgmap v4651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:02:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7866.923086 secs 2024-07-26T15:02:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:10 smithi112 ceph-mon[37501]: pgmap v4651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:02:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7866.923086 secs 2024-07-26T15:02:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:10 smithi107 ceph-mon[32659]: pgmap v4651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:02:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7866.923086 secs 2024-07-26T15:02:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:12 smithi132 ceph-mon[37580]: pgmap v4652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:12.844 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:12 smithi107 ceph-mon[32659]: pgmap v4652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:12 smithi112 ceph-mon[37501]: pgmap v4652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:13.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:13.444+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:13.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:13.444+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:13.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:13.444+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:14 smithi132 ceph-mon[37580]: pgmap v4653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:02:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:14 smithi112 ceph-mon[37501]: pgmap v4653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:02:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:14 smithi107 ceph-mon[32659]: pgmap v4653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:02:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:15.699+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:15.700 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:15.699+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:15.701 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:15.699+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7871.923176 secs 2024-07-26T15:02:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7871.923176 secs 2024-07-26T15:02:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7871.923176 secs 2024-07-26T15:02:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:16 smithi112 ceph-mon[37501]: pgmap v4654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:02:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:16 smithi107 ceph-mon[32659]: pgmap v4654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:02:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:16 smithi132 ceph-mon[37580]: pgmap v4654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:02:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:18 smithi112 ceph-mon[37501]: pgmap v4655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:02:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:18 smithi107 ceph-mon[32659]: pgmap v4655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:02:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:18 smithi132 ceph-mon[37580]: pgmap v4655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:02:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:20 smithi112 ceph-mon[37501]: pgmap v4656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:02:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7876.923294 secs 2024-07-26T15:02:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:20 smithi107 ceph-mon[32659]: pgmap v4656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:02:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7876.923294 secs 2024-07-26T15:02:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:20 smithi132 ceph-mon[37580]: pgmap v4656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:02:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7876.923294 secs 2024-07-26T15:02:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:21 smithi112 ceph-mon[37501]: pgmap v4657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:02:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:21 smithi107 ceph-mon[32659]: pgmap v4657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:02:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:21 smithi132 ceph-mon[37580]: pgmap v4657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:02:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:24 smithi112 ceph-mon[37501]: pgmap v4658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:02:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:24 smithi107 ceph-mon[32659]: pgmap v4658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:02:24.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:24 smithi132 ceph-mon[37580]: pgmap v4658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:02:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7881.923399 secs 2024-07-26T15:02:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7881.923399 secs 2024-07-26T15:02:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7881.923399 secs 2024-07-26T15:02:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:26 smithi112 ceph-mon[37501]: pgmap v4659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:26 smithi107 ceph-mon[32659]: pgmap v4659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:26 smithi132 ceph-mon[37580]: pgmap v4659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:28.459+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:28.459+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:28.459+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:28 smithi112 ceph-mon[37501]: pgmap v4660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:02:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:02:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:28 smithi107 ceph-mon[32659]: pgmap v4660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:02:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:02:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:28 smithi132 ceph-mon[37580]: pgmap v4660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:02:28.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:02:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:02:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:02:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:29 smithi112 ceph-mon[37501]: pgmap v4661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:02:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:29 smithi112 ceph-mon[37501]: pgmap v4662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:02:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:02:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:02:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:29 smithi107 ceph-mon[32659]: pgmap v4661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:02:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:29 smithi107 ceph-mon[32659]: pgmap v4662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:02:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:02:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:02:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:29 smithi132 ceph-mon[37580]: pgmap v4661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:02:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:29 smithi132 ceph-mon[37580]: pgmap v4662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:02:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7886.923537 secs 2024-07-26T15:02:30.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:30.713+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:30.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:30.713+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:30.718 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:30.713+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7886.923537 secs 2024-07-26T15:02:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7886.923537 secs 2024-07-26T15:02:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:31 smithi112 ceph-mon[37501]: pgmap v4663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:31 smithi107 ceph-mon[32659]: pgmap v4663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:31 smithi132 ceph-mon[37580]: pgmap v4663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:33 smithi107 ceph-mon[32659]: pgmap v4664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:02:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:33 smithi132 ceph-mon[37580]: pgmap v4664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:02:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:33 smithi112 ceph-mon[37501]: pgmap v4664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:02:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:35 smithi107 ceph-mon[32659]: pgmap v4665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7891.923672 secs 2024-07-26T15:02:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:35 smithi132 ceph-mon[37580]: pgmap v4665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7891.923672 secs 2024-07-26T15:02:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:35 smithi112 ceph-mon[37501]: pgmap v4665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7891.923672 secs 2024-07-26T15:02:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:37 smithi107 ceph-mon[32659]: pgmap v4666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:02:37.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:37 smithi132 ceph-mon[37580]: pgmap v4666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:02:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:37 smithi112 ceph-mon[37501]: pgmap v4666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:02:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:39 smithi107 ceph-mon[32659]: pgmap v4667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:02:39.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:39 smithi132 ceph-mon[37580]: pgmap v4667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:02:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:39 smithi112 ceph-mon[37501]: pgmap v4667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:02:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7896.923793 secs 2024-07-26T15:02:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7896.923793 secs 2024-07-26T15:02:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7896.923793 secs 2024-07-26T15:02:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:41 smithi132 ceph-mon[37580]: pgmap v4668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:41 smithi112 ceph-mon[37501]: pgmap v4668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:41 smithi107 ceph-mon[32659]: pgmap v4668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:43.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:43.476+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:43.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:43.476+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:43.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:43.476+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:43 smithi107 ceph-mon[32659]: pgmap v4669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:02:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:43 smithi132 ceph-mon[37580]: pgmap v4669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:02:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:43 smithi112 ceph-mon[37501]: pgmap v4669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:02:45.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:45.729+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:45.730 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:45.730+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:45.731 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:02:45.730+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:45 smithi132 ceph-mon[37580]: pgmap v4670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:02:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7901.923859 secs 2024-07-26T15:02:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:45 smithi112 ceph-mon[37501]: pgmap v4670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:02:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7901.923859 secs 2024-07-26T15:02:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:45 smithi107 ceph-mon[32659]: pgmap v4670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:02:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7901.923859 secs 2024-07-26T15:02:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:47 smithi112 ceph-mon[37501]: pgmap v4671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:47 smithi107 ceph-mon[32659]: pgmap v4671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:47 smithi132 ceph-mon[37580]: pgmap v4671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:49 smithi112 ceph-mon[37501]: pgmap v4672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:49 smithi107 ceph-mon[32659]: pgmap v4672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:50.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:49 smithi132 ceph-mon[37580]: pgmap v4672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:50.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7906.923976 secs 2024-07-26T15:02:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7906.923976 secs 2024-07-26T15:02:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7906.923976 secs 2024-07-26T15:02:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:51 smithi112 ceph-mon[37501]: pgmap v4673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:51 smithi107 ceph-mon[32659]: pgmap v4673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:51 smithi132 ceph-mon[37580]: pgmap v4673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:53 smithi112 ceph-mon[37501]: pgmap v4674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:02:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:53 smithi107 ceph-mon[32659]: pgmap v4674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:02:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:53 smithi132 ceph-mon[37580]: pgmap v4674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:02:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:54 smithi112 ceph-mon[37501]: pgmap v4675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:02:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:54 smithi107 ceph-mon[32659]: pgmap v4675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:02:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:54 smithi132 ceph-mon[37580]: pgmap v4675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:02:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7911.924120 secs 2024-07-26T15:02:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7911.924120 secs 2024-07-26T15:02:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7911.924120 secs 2024-07-26T15:02:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:56 smithi112 ceph-mon[37501]: pgmap v4676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:56 smithi107 ceph-mon[32659]: pgmap v4676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:56 smithi132 ceph-mon[37580]: pgmap v4676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:58.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:58.491+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:58.491+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:58.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:02:58.491+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:02:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:02:59 smithi107 ceph-mon[32659]: pgmap v4677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:59.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:02:59 smithi132 ceph-mon[37580]: pgmap v4677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:02:59 smithi112 ceph-mon[37501]: pgmap v4677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:00.734 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:00.733+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:00.734 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:00.733+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:00.734 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:00.733+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7916.924246 secs 2024-07-26T15:03:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7916.924246 secs 2024-07-26T15:03:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7916.924246 secs 2024-07-26T15:03:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:01 smithi132 ceph-mon[37580]: pgmap v4678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:01 smithi112 ceph-mon[37501]: pgmap v4678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:01 smithi107 ceph-mon[32659]: pgmap v4678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:03 smithi132 ceph-mon[37580]: pgmap v4679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:03 smithi112 ceph-mon[37501]: pgmap v4679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:03 smithi107 ceph-mon[32659]: pgmap v4679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:05 smithi132 ceph-mon[37580]: pgmap v4680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:03:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7921.924363 secs 2024-07-26T15:03:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:05 smithi112 ceph-mon[37501]: pgmap v4680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:03:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7921.924363 secs 2024-07-26T15:03:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:05 smithi107 ceph-mon[32659]: pgmap v4680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:03:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7921.924363 secs 2024-07-26T15:03:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:07 smithi112 ceph-mon[37501]: pgmap v4681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:07 smithi107 ceph-mon[32659]: pgmap v4681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:07 smithi132 ceph-mon[37580]: pgmap v4681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:09 smithi112 ceph-mon[37501]: pgmap v4682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:09 smithi107 ceph-mon[32659]: pgmap v4682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:09 smithi132 ceph-mon[37580]: pgmap v4682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7926.924445 secs 2024-07-26T15:03:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7926.924445 secs 2024-07-26T15:03:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7926.924445 secs 2024-07-26T15:03:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:11 smithi112 ceph-mon[37501]: pgmap v4683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:11 smithi107 ceph-mon[32659]: pgmap v4683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:11 smithi132 ceph-mon[37580]: pgmap v4683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:12.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:12.977+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:03:12.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:12.977+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:03:12.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:12.989+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:03:12.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:12.989+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:03:13.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:13.506+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:13.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:13.506+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:13.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:13.506+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:13 smithi112 ceph-mon[37501]: pgmap v4684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:13.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:13.977+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:03:13.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:13 smithi107 ceph-mon[32659]: pgmap v4684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:13.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:13.978+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:03:13.990 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:13.989+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:03:13.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:13.989+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:03:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:13 smithi132 ceph-mon[37580]: pgmap v4684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:15.159+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:03:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:15.160+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:03:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:15.160+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:03:15.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:15.160+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:03:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:15.748+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:15.748+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:15.748+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:15 smithi112 ceph-mon[37501]: pgmap v4685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:03:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7931.924565 secs 2024-07-26T15:03:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:15 smithi107 ceph-mon[32659]: pgmap v4685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:03:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7931.924565 secs 2024-07-26T15:03:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:15 smithi132 ceph-mon[37580]: pgmap v4685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:03:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7931.924565 secs 2024-07-26T15:03:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:16.160+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:03:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:16.160+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:03:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:16.160+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:03:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:16.161+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:03:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:16 smithi112 ceph-mon[37501]: pgmap v4686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:16 smithi107 ceph-mon[32659]: pgmap v4686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:16 smithi132 ceph-mon[37580]: pgmap v4686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:19 smithi132 ceph-mon[37580]: pgmap v4687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:19 smithi112 ceph-mon[37501]: pgmap v4687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:19 smithi107 ceph-mon[32659]: pgmap v4687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7936.924692 secs 2024-07-26T15:03:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7936.924692 secs 2024-07-26T15:03:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7936.924692 secs 2024-07-26T15:03:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:21 smithi132 ceph-mon[37580]: pgmap v4688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:21 smithi112 ceph-mon[37501]: pgmap v4688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:21 smithi107 ceph-mon[32659]: pgmap v4688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:23 smithi112 ceph-mon[37501]: pgmap v4689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:23 smithi107 ceph-mon[32659]: pgmap v4689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:23 smithi132 ceph-mon[37580]: pgmap v4689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:25 smithi112 ceph-mon[37501]: pgmap v4690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:03:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7941.924790 secs 2024-07-26T15:03:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:25 smithi107 ceph-mon[32659]: pgmap v4690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:03:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7941.924790 secs 2024-07-26T15:03:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:25 smithi132 ceph-mon[37580]: pgmap v4690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:03:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7941.924790 secs 2024-07-26T15:03:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:27 smithi112 ceph-mon[37501]: pgmap v4691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:27 smithi107 ceph-mon[32659]: pgmap v4691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:27 smithi132 ceph-mon[37580]: pgmap v4691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:28.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:28.519+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:28.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:28.519+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:28.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:28.519+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:29 smithi112 ceph-mon[37501]: pgmap v4692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:03:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:29 smithi107 ceph-mon[32659]: pgmap v4692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:03:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:29 smithi132 ceph-mon[37580]: pgmap v4692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:03:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:30.758+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:30.758+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:30.758+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:03:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:03:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:03:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7946.924920 secs 2024-07-26T15:03:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:03:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:03:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:03:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7946.924920 secs 2024-07-26T15:03:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:03:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:03:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:03:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7946.924920 secs 2024-07-26T15:03:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:31 smithi112 ceph-mon[37501]: pgmap v4693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:03:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:31 smithi112 ceph-mon[37501]: pgmap v4694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:31 smithi107 ceph-mon[32659]: pgmap v4693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:03:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:31 smithi107 ceph-mon[32659]: pgmap v4694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:31 smithi132 ceph-mon[37580]: pgmap v4693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:03:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:31 smithi132 ceph-mon[37580]: pgmap v4694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:33.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:33 smithi132 ceph-mon[37580]: pgmap v4695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:33 smithi112 ceph-mon[37501]: pgmap v4695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:33 smithi107 ceph-mon[32659]: pgmap v4695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:34 smithi112 ceph-mon[37501]: pgmap v4696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:03:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:34 smithi107 ceph-mon[32659]: pgmap v4696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:03:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:34 smithi132 ceph-mon[37580]: pgmap v4696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:03:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7951.925001 secs 2024-07-26T15:03:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7951.925001 secs 2024-07-26T15:03:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7951.925001 secs 2024-07-26T15:03:36.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:36 smithi112 ceph-mon[37501]: pgmap v4697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:36 smithi107 ceph-mon[32659]: pgmap v4697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:36 smithi132 ceph-mon[37580]: pgmap v4697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:38 smithi107 ceph-mon[32659]: pgmap v4698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:03:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:38 smithi132 ceph-mon[37580]: pgmap v4698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:03:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:38 smithi112 ceph-mon[37501]: pgmap v4698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:03:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:40 smithi107 ceph-mon[32659]: pgmap v4699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:03:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7956.925119 secs 2024-07-26T15:03:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:40 smithi132 ceph-mon[37580]: pgmap v4699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:03:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7956.925119 secs 2024-07-26T15:03:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:40 smithi112 ceph-mon[37501]: pgmap v4699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:03:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7956.925119 secs 2024-07-26T15:03:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:42 smithi107 ceph-mon[32659]: pgmap v4700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:42 smithi132 ceph-mon[37580]: pgmap v4700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:42 smithi112 ceph-mon[37501]: pgmap v4700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:43.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:43.536+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:43.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:43.536+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:43.537 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:43.536+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:44 smithi107 ceph-mon[32659]: pgmap v4701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:44 smithi132 ceph-mon[37580]: pgmap v4701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:44 smithi112 ceph-mon[37501]: pgmap v4701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:45.775+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:45.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:45.775+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:45.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:03:45.775+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7961.925238 secs 2024-07-26T15:03:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7961.925238 secs 2024-07-26T15:03:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7961.925238 secs 2024-07-26T15:03:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:46 smithi107 ceph-mon[32659]: pgmap v4702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:46 smithi132 ceph-mon[37580]: pgmap v4702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:46 smithi112 ceph-mon[37501]: pgmap v4702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:48 smithi107 ceph-mon[32659]: pgmap v4703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:48 smithi132 ceph-mon[37580]: pgmap v4703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:48 smithi112 ceph-mon[37501]: pgmap v4703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:50 smithi107 ceph-mon[32659]: pgmap v4704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7966.925305 secs 2024-07-26T15:03:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:50 smithi132 ceph-mon[37580]: pgmap v4704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7966.925305 secs 2024-07-26T15:03:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:50 smithi112 ceph-mon[37501]: pgmap v4704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7966.925305 secs 2024-07-26T15:03:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:52 smithi107 ceph-mon[32659]: pgmap v4705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:52 smithi132 ceph-mon[37580]: pgmap v4705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:52 smithi112 ceph-mon[37501]: pgmap v4705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:54 smithi107 ceph-mon[32659]: pgmap v4706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:54 smithi132 ceph-mon[37580]: pgmap v4706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:54 smithi112 ceph-mon[37501]: pgmap v4706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7971.925423 secs 2024-07-26T15:03:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7971.925423 secs 2024-07-26T15:03:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7971.925423 secs 2024-07-26T15:03:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:56 smithi107 ceph-mon[32659]: pgmap v4707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:56 smithi132 ceph-mon[37580]: pgmap v4707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:56 smithi112 ceph-mon[37501]: pgmap v4707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:58.551+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:58.551+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:03:58.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:03:58.551+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:03:58 smithi107 ceph-mon[32659]: pgmap v4708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:03:58 smithi132 ceph-mon[37580]: pgmap v4708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:03:58 smithi112 ceph-mon[37501]: pgmap v4708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:00.790+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:00.790+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:00.791 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:00.790+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:00 smithi132 ceph-mon[37580]: pgmap v4709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7976.925505 secs 2024-07-26T15:04:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:00 smithi112 ceph-mon[37501]: pgmap v4709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7976.925505 secs 2024-07-26T15:04:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:00 smithi107 ceph-mon[32659]: pgmap v4709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7976.925505 secs 2024-07-26T15:04:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:02 smithi132 ceph-mon[37580]: pgmap v4710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:02 smithi112 ceph-mon[37501]: pgmap v4710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:02 smithi107 ceph-mon[32659]: pgmap v4710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:04 smithi112 ceph-mon[37501]: pgmap v4711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:04 smithi107 ceph-mon[32659]: pgmap v4711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:04 smithi132 ceph-mon[37580]: pgmap v4711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7981.925595 secs 2024-07-26T15:04:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7981.925595 secs 2024-07-26T15:04:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7981.925595 secs 2024-07-26T15:04:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:06 smithi112 ceph-mon[37501]: pgmap v4712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:06 smithi107 ceph-mon[32659]: pgmap v4712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:06 smithi132 ceph-mon[37580]: pgmap v4712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:08 smithi112 ceph-mon[37501]: pgmap v4713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:08 smithi107 ceph-mon[32659]: pgmap v4713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:08 smithi132 ceph-mon[37580]: pgmap v4713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:10 smithi112 ceph-mon[37501]: pgmap v4714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7986.925694 secs 2024-07-26T15:04:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:10 smithi107 ceph-mon[32659]: pgmap v4714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7986.925694 secs 2024-07-26T15:04:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:10 smithi132 ceph-mon[37580]: pgmap v4714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7986.925694 secs 2024-07-26T15:04:13.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:12 smithi107 ceph-mon[32659]: pgmap v4715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:12 smithi112 ceph-mon[37501]: pgmap v4715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:12 smithi132 ceph-mon[37580]: pgmap v4715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:13.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:13.565+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:13.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:13.565+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:13.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:13.565+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:14 smithi112 ceph-mon[37501]: pgmap v4716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:14 smithi107 ceph-mon[32659]: pgmap v4716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:14 smithi132 ceph-mon[37580]: pgmap v4716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:15.805+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:15.805+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:15.805+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7991.925817 secs 2024-07-26T15:04:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7991.925817 secs 2024-07-26T15:04:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7991.925817 secs 2024-07-26T15:04:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:17 smithi112 ceph-mon[37501]: pgmap v4717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:17 smithi107 ceph-mon[32659]: pgmap v4717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:17 smithi132 ceph-mon[37580]: pgmap v4717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:19 smithi112 ceph-mon[37501]: pgmap v4718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:19 smithi107 ceph-mon[32659]: pgmap v4718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:19 smithi132 ceph-mon[37580]: pgmap v4718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 7996.925935 secs 2024-07-26T15:04:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 7996.925935 secs 2024-07-26T15:04:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 7996.925935 secs 2024-07-26T15:04:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:21 smithi112 ceph-mon[37501]: pgmap v4719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:21 smithi107 ceph-mon[32659]: pgmap v4719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:21 smithi132 ceph-mon[37580]: pgmap v4719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:23 smithi112 ceph-mon[37501]: pgmap v4720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:23 smithi107 ceph-mon[32659]: pgmap v4720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:23 smithi132 ceph-mon[37580]: pgmap v4720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:25 smithi112 ceph-mon[37501]: pgmap v4721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8001.926000 secs 2024-07-26T15:04:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:25 smithi107 ceph-mon[32659]: pgmap v4721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8001.926000 secs 2024-07-26T15:04:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:25 smithi132 ceph-mon[37580]: pgmap v4721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8001.926000 secs 2024-07-26T15:04:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:27 smithi112 ceph-mon[37501]: pgmap v4722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:27 smithi107 ceph-mon[32659]: pgmap v4722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:27 smithi132 ceph-mon[37580]: pgmap v4722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:28.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:28.580+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:28.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:28.580+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:28.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:28.580+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:29 smithi112 ceph-mon[37501]: pgmap v4723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:29 smithi107 ceph-mon[32659]: pgmap v4723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:29 smithi132 ceph-mon[37580]: pgmap v4723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:04:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8006.926121 secs 2024-07-26T15:04:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:04:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8006.926121 secs 2024-07-26T15:04:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:04:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8006.926121 secs 2024-07-26T15:04:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:30.820+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:30.820+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:30.820+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:31 smithi112 ceph-mon[37501]: pgmap v4724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:04:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:04:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:04:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:31 smithi107 ceph-mon[32659]: pgmap v4724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:04:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:04:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:04:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:31 smithi132 ceph-mon[37580]: pgmap v4724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:04:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:04:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:04:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:32 smithi112 ceph-mon[37501]: pgmap v4725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:04:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:32 smithi107 ceph-mon[32659]: pgmap v4725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:04:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:32 smithi132 ceph-mon[37580]: pgmap v4725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:04:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:34 smithi112 ceph-mon[37501]: pgmap v4726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:04:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:34 smithi107 ceph-mon[32659]: pgmap v4726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:04:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:34 smithi132 ceph-mon[37580]: pgmap v4726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:04:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8011.926212 secs 2024-07-26T15:04:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8011.926212 secs 2024-07-26T15:04:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8011.926212 secs 2024-07-26T15:04:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:36 smithi112 ceph-mon[37501]: pgmap v4727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:04:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:36 smithi107 ceph-mon[32659]: pgmap v4727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:04:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:36 smithi132 ceph-mon[37580]: pgmap v4727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:04:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:38 smithi112 ceph-mon[37501]: pgmap v4728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:04:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:38 smithi107 ceph-mon[32659]: pgmap v4728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:04:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:38 smithi132 ceph-mon[37580]: pgmap v4728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:04:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:40 smithi112 ceph-mon[37501]: pgmap v4729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:04:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8016.926325 secs 2024-07-26T15:04:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:40 smithi107 ceph-mon[32659]: pgmap v4729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:04:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8016.926325 secs 2024-07-26T15:04:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:40 smithi132 ceph-mon[37580]: pgmap v4729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:04:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8016.926325 secs 2024-07-26T15:04:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:42 smithi112 ceph-mon[37501]: pgmap v4730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:04:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:42 smithi107 ceph-mon[32659]: pgmap v4730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:04:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:42 smithi132 ceph-mon[37580]: pgmap v4730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:04:43.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:43.597+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:43.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:43.597+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:43.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:43.597+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:44 smithi107 ceph-mon[32659]: pgmap v4731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:44 smithi132 ceph-mon[37580]: pgmap v4731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:44 smithi112 ceph-mon[37501]: pgmap v4731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8021.926391 secs 2024-07-26T15:04:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8021.926391 secs 2024-07-26T15:04:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:45.837+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:45.837+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:04:45.837+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8021.926391 secs 2024-07-26T15:04:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:46 smithi107 ceph-mon[32659]: pgmap v4732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:46 smithi132 ceph-mon[37580]: pgmap v4732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:46 smithi112 ceph-mon[37501]: pgmap v4732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:48 smithi107 ceph-mon[32659]: pgmap v4733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:48 smithi132 ceph-mon[37580]: pgmap v4733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:48 smithi112 ceph-mon[37501]: pgmap v4733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:50 smithi107 ceph-mon[32659]: pgmap v4734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8026.926492 secs 2024-07-26T15:04:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:50 smithi132 ceph-mon[37580]: pgmap v4734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8026.926492 secs 2024-07-26T15:04:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:50 smithi112 ceph-mon[37501]: pgmap v4734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8026.926492 secs 2024-07-26T15:04:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:52 smithi107 ceph-mon[32659]: pgmap v4735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:52 smithi132 ceph-mon[37580]: pgmap v4735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:52 smithi112 ceph-mon[37501]: pgmap v4735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:54 smithi107 ceph-mon[32659]: pgmap v4736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:54 smithi132 ceph-mon[37580]: pgmap v4736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:54 smithi112 ceph-mon[37501]: pgmap v4736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8031.926614 secs 2024-07-26T15:04:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8031.926614 secs 2024-07-26T15:04:55.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8031.926614 secs 2024-07-26T15:04:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:56 smithi107 ceph-mon[32659]: pgmap v4737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:56 smithi132 ceph-mon[37580]: pgmap v4737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:56 smithi112 ceph-mon[37501]: pgmap v4737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:58.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:58.609+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:58.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:58.609+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:58.610 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:04:58.609+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:04:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:04:58 smithi107 ceph-mon[32659]: pgmap v4738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:04:58 smithi132 ceph-mon[37580]: pgmap v4738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:04:58 smithi112 ceph-mon[37501]: pgmap v4738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:00 smithi107 ceph-mon[32659]: pgmap v4739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8036.926679 secs 2024-07-26T15:05:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:00 smithi132 ceph-mon[37580]: pgmap v4739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8036.926679 secs 2024-07-26T15:05:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:00.852+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:00.853 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:00.852+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:00.854 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:00.852+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:00 smithi112 ceph-mon[37501]: pgmap v4739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8036.926679 secs 2024-07-26T15:05:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:02 smithi107 ceph-mon[32659]: pgmap v4740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:02 smithi132 ceph-mon[37580]: pgmap v4740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:02 smithi112 ceph-mon[37501]: pgmap v4740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:04 smithi107 ceph-mon[32659]: pgmap v4741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:04 smithi132 ceph-mon[37580]: pgmap v4741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:04 smithi112 ceph-mon[37501]: pgmap v4741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8041.926799 secs 2024-07-26T15:05:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8041.926799 secs 2024-07-26T15:05:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8041.926799 secs 2024-07-26T15:05:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:06 smithi132 ceph-mon[37580]: pgmap v4742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:06 smithi112 ceph-mon[37501]: pgmap v4742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:06 smithi107 ceph-mon[32659]: pgmap v4742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:08 smithi132 ceph-mon[37580]: pgmap v4743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:08.826 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:08 smithi107 ceph-mon[32659]: pgmap v4743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:08 smithi112 ceph-mon[37501]: pgmap v4743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:10 smithi112 ceph-mon[37501]: pgmap v4744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:11.060 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8046.926920 secs 2024-07-26T15:05:11.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:10 smithi132 ceph-mon[37580]: pgmap v4744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:11.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8046.926920 secs 2024-07-26T15:05:11.061 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:10 smithi107 ceph-mon[32659]: pgmap v4744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:11.061 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8046.926920 secs 2024-07-26T15:05:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:12 smithi112 ceph-mon[37501]: pgmap v4745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:12 smithi107 ceph-mon[32659]: pgmap v4745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:12 smithi132 ceph-mon[37580]: pgmap v4745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:13.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:13.625+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:13.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:13.625+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:13.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:13.625+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:14 smithi112 ceph-mon[37501]: pgmap v4746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:14 smithi107 ceph-mon[32659]: pgmap v4746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:14 smithi132 ceph-mon[37580]: pgmap v4746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:15.857 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:15.856+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:15.856+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:15.856+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8051.927038 secs 2024-07-26T15:05:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8051.927038 secs 2024-07-26T15:05:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8051.927038 secs 2024-07-26T15:05:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:16 smithi112 ceph-mon[37501]: pgmap v4747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:16 smithi107 ceph-mon[32659]: pgmap v4747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:16 smithi132 ceph-mon[37580]: pgmap v4747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:18 smithi112 ceph-mon[37501]: pgmap v4748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:18 smithi107 ceph-mon[32659]: pgmap v4748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:18 smithi132 ceph-mon[37580]: pgmap v4748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:20 smithi112 ceph-mon[37501]: pgmap v4749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8056.927156 secs 2024-07-26T15:05:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:20 smithi107 ceph-mon[32659]: pgmap v4749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8056.927156 secs 2024-07-26T15:05:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:20 smithi132 ceph-mon[37580]: pgmap v4749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8056.927156 secs 2024-07-26T15:05:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:22 smithi112 ceph-mon[37501]: pgmap v4750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:22 smithi107 ceph-mon[32659]: pgmap v4750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:22 smithi132 ceph-mon[37580]: pgmap v4750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:24 smithi112 ceph-mon[37501]: pgmap v4751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:24 smithi107 ceph-mon[32659]: pgmap v4751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:24 smithi132 ceph-mon[37580]: pgmap v4751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8061.927221 secs 2024-07-26T15:05:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8061.927221 secs 2024-07-26T15:05:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8061.927221 secs 2024-07-26T15:05:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:26 smithi112 ceph-mon[37501]: pgmap v4752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:26 smithi107 ceph-mon[32659]: pgmap v4752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:26 smithi132 ceph-mon[37580]: pgmap v4752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:27 smithi112 ceph-mon[37501]: pgmap v4753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:27 smithi107 ceph-mon[32659]: pgmap v4753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:27 smithi132 ceph-mon[37580]: pgmap v4753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:28.640+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:28.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:28.640+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:28.640+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:29 smithi107 ceph-mon[32659]: pgmap v4754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:29 smithi132 ceph-mon[37580]: pgmap v4754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:29 smithi112 ceph-mon[37501]: pgmap v4754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:30.872+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:30.872+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:30.872+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:31.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8066.927363 secs 2024-07-26T15:05:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8066.927363 secs 2024-07-26T15:05:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8066.927363 secs 2024-07-26T15:05:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:31 smithi107 ceph-mon[32659]: pgmap v4755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:05:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:31 smithi132 ceph-mon[37580]: pgmap v4755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:05:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:31 smithi112 ceph-mon[37501]: pgmap v4755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:05:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:05:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:05:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:05:33.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:05:33.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: pgmap v4756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:05:33.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: pgmap v4757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:33.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:05:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: pgmap v4756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: pgmap v4757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:33.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:05:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: pgmap v4756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:05:33.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: pgmap v4757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:33.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:05:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:35 smithi112 ceph-mon[37501]: pgmap v4758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:05:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:35 smithi107 ceph-mon[32659]: pgmap v4758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:05:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:35 smithi132 ceph-mon[37580]: pgmap v4758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:05:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8071.927432 secs 2024-07-26T15:05:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8071.927432 secs 2024-07-26T15:05:36.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8071.927432 secs 2024-07-26T15:05:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:37 smithi112 ceph-mon[37501]: pgmap v4759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:05:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:37 smithi107 ceph-mon[32659]: pgmap v4759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:05:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:37 smithi132 ceph-mon[37580]: pgmap v4759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:05:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:39 smithi112 ceph-mon[37501]: pgmap v4760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:05:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:39 smithi107 ceph-mon[32659]: pgmap v4760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:05:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:39 smithi132 ceph-mon[37580]: pgmap v4760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:05:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:41 smithi112 ceph-mon[37501]: pgmap v4761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:05:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8076.927565 secs 2024-07-26T15:05:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:41 smithi107 ceph-mon[32659]: pgmap v4761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:05:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8076.927565 secs 2024-07-26T15:05:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:41 smithi132 ceph-mon[37580]: pgmap v4761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:05:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8076.927565 secs 2024-07-26T15:05:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:43 smithi112 ceph-mon[37501]: pgmap v4762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:05:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:43 smithi107 ceph-mon[32659]: pgmap v4762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:05:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:43 smithi132 ceph-mon[37580]: pgmap v4762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:05:43.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:43.656+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:43.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:43.656+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:43.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:43.656+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:45 smithi112 ceph-mon[37501]: pgmap v4763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:45 smithi107 ceph-mon[32659]: pgmap v4763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:45 smithi132 ceph-mon[37580]: pgmap v4763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:45.888+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:45.888+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:45.889 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:05:45.888+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8081.927690 secs 2024-07-26T15:05:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8081.927690 secs 2024-07-26T15:05:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8081.927690 secs 2024-07-26T15:05:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:47 smithi107 ceph-mon[32659]: pgmap v4764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:47 smithi132 ceph-mon[37580]: pgmap v4764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:47 smithi112 ceph-mon[37501]: pgmap v4764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:49 smithi107 ceph-mon[32659]: pgmap v4765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:49 smithi132 ceph-mon[37580]: pgmap v4765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:49 smithi112 ceph-mon[37501]: pgmap v4765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:51 smithi107 ceph-mon[32659]: pgmap v4766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8086.927774 secs 2024-07-26T15:05:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:51 smithi132 ceph-mon[37580]: pgmap v4766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8086.927774 secs 2024-07-26T15:05:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:51 smithi112 ceph-mon[37501]: pgmap v4766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8086.927774 secs 2024-07-26T15:05:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:53 smithi107 ceph-mon[32659]: pgmap v4767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:53 smithi132 ceph-mon[37580]: pgmap v4767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:53 smithi112 ceph-mon[37501]: pgmap v4767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:55.502 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:55 smithi112 ceph-mon[37501]: pgmap v4768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:55 smithi107 ceph-mon[32659]: pgmap v4768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:55.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:55 smithi132 ceph-mon[37580]: pgmap v4768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8091.927914 secs 2024-07-26T15:05:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8091.927914 secs 2024-07-26T15:05:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8091.927914 secs 2024-07-26T15:05:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:57 smithi107 ceph-mon[32659]: pgmap v4769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:57.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:57 smithi132 ceph-mon[37580]: pgmap v4769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:57 smithi112 ceph-mon[37501]: pgmap v4769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:58.669+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:58.669+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:58.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:05:58.669+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:05:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:05:59 smithi107 ceph-mon[32659]: pgmap v4770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:59.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:05:59 smithi132 ceph-mon[37580]: pgmap v4770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:05:59 smithi112 ceph-mon[37501]: pgmap v4770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:00.890+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:00.890+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:00.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:00.890+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:01 smithi107 ceph-mon[32659]: pgmap v4771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8096.927981 secs 2024-07-26T15:06:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:01 smithi132 ceph-mon[37580]: pgmap v4771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8096.927981 secs 2024-07-26T15:06:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:01 smithi112 ceph-mon[37501]: pgmap v4771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8096.927981 secs 2024-07-26T15:06:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:03 smithi107 ceph-mon[32659]: pgmap v4772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:03 smithi132 ceph-mon[37580]: pgmap v4772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:03 smithi112 ceph-mon[37501]: pgmap v4772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:05 smithi107 ceph-mon[32659]: pgmap v4773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:05 smithi132 ceph-mon[37580]: pgmap v4773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:05 smithi112 ceph-mon[37501]: pgmap v4773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8101.928058 secs 2024-07-26T15:06:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8101.928058 secs 2024-07-26T15:06:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8101.928058 secs 2024-07-26T15:06:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:07 smithi107 ceph-mon[32659]: pgmap v4774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:07 smithi132 ceph-mon[37580]: pgmap v4774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:07 smithi112 ceph-mon[37501]: pgmap v4774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:09 smithi107 ceph-mon[32659]: pgmap v4775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:09 smithi132 ceph-mon[37580]: pgmap v4775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:09 smithi112 ceph-mon[37501]: pgmap v4775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:11 smithi107 ceph-mon[32659]: pgmap v4776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8106.928175 secs 2024-07-26T15:06:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:11 smithi132 ceph-mon[37580]: pgmap v4776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8106.928175 secs 2024-07-26T15:06:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:11.662+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0260 msgr2=0x7ff0ec0c1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T15:06:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:11.662+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0260 msgr2=0x7ff0ec0c1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:06:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:11.662+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0260 0x7ff0ec0c1420 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f801f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:06:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:11.662+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0260 0x7ff0ec0c1420 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f801f040 comp rx=0 tx=0).stop 2024-07-26T15:06:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:11.662+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0260 msgr2=0x7ff0ec0c1420 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T15:06:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:11.662+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0260 msgr2=0x7ff0ec0c1420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:06:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:11.662+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0260 0x7ff0ec0c1420 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:06:11.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:11.662+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0670 0x7ff0ec0c0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:06:11.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:11.662+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0670 0x7ff0ec0c0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:06:11.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:11.665+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0670 0x7ff0ec0c0ee0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fa00 tx=0x7ff0f8067040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:06:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:11 smithi112 ceph-mon[37501]: pgmap v4776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8106.928175 secs 2024-07-26T15:06:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:13 smithi132 ceph-mon[37580]: pgmap v4777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:13.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:13.671+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:13.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:13.671+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:13.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:13.671+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:13.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:13 smithi107 ceph-mon[32659]: pgmap v4777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:13 smithi112 ceph-mon[37501]: pgmap v4777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:13.768+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000262) for more than 900000000 us, fault. 2024-07-26T15:06:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:13.768+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7efc480063a0 tx=0x7efc680da000 comp rx=0 tx=0).stop 2024-07-26T15:06:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:13.768+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c2f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:06:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:13.768+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:06:13.769 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:13.768+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:06:13.770 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:13.769+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:06:13.771 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:13.771+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7efc680aacd0 tx=0x7efc680e6040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:06:14.009 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:14.008+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:06:14.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:14.009+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:06:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:14.019+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:06:14.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:14.020+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:06:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:15 smithi132 ceph-mon[37580]: pgmap v4778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:15 smithi112 ceph-mon[37501]: pgmap v4778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:15 smithi107 ceph-mon[32659]: pgmap v4778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:15.904+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:15.904+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:15.906 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:15.904+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:16.193+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:06:16.194 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:16.193+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:06:16.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:16.193+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:06:16.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:16.193+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:06:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8111.928274 secs 2024-07-26T15:06:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8111.928274 secs 2024-07-26T15:06:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8111.928274 secs 2024-07-26T15:06:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:17 smithi112 ceph-mon[37501]: pgmap v4779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:17 smithi107 ceph-mon[32659]: pgmap v4779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:17 smithi132 ceph-mon[37580]: pgmap v4779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:19 smithi112 ceph-mon[37501]: pgmap v4780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:19 smithi107 ceph-mon[32659]: pgmap v4780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:19.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:19 smithi132 ceph-mon[37580]: pgmap v4780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:21 smithi112 ceph-mon[37501]: pgmap v4781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8116.928392 secs 2024-07-26T15:06:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:21 smithi107 ceph-mon[32659]: pgmap v4781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8116.928392 secs 2024-07-26T15:06:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:21 smithi132 ceph-mon[37580]: pgmap v4781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8116.928392 secs 2024-07-26T15:06:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:23 smithi112 ceph-mon[37501]: pgmap v4782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:23 smithi107 ceph-mon[32659]: pgmap v4782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:23 smithi132 ceph-mon[37580]: pgmap v4782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:25 smithi112 ceph-mon[37501]: pgmap v4783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:25 smithi107 ceph-mon[32659]: pgmap v4783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:25 smithi132 ceph-mon[37580]: pgmap v4783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8121.928513 secs 2024-07-26T15:06:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8121.928513 secs 2024-07-26T15:06:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8121.928513 secs 2024-07-26T15:06:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:27 smithi107 ceph-mon[32659]: pgmap v4784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:27 smithi112 ceph-mon[37501]: pgmap v4784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:27 smithi132 ceph-mon[37580]: pgmap v4784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:28.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:28.679+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:28.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:28.679+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:28.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:28.679+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:29 smithi112 ceph-mon[37501]: pgmap v4785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:29 smithi107 ceph-mon[32659]: pgmap v4785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:29 smithi132 ceph-mon[37580]: pgmap v4785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8126.928653 secs 2024-07-26T15:06:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8126.928653 secs 2024-07-26T15:06:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8126.928653 secs 2024-07-26T15:06:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:30.908+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:30.908+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:30.910 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:30.908+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:31 smithi112 ceph-mon[37501]: pgmap v4786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:31 smithi107 ceph-mon[32659]: pgmap v4786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:31 smithi132 ceph-mon[37580]: pgmap v4786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:06:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:06:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:06:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: pgmap v4787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:06:33.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:06:33.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:06:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: pgmap v4787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:06:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:06:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:06:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: pgmap v4787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:06:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:06:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:06:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:34 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:06:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:34 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:06:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:34 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:06:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:34 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:06:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:34 smithi112 ceph-mon[37501]: pgmap v4788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:06:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:34 smithi112 ceph-mon[37501]: pgmap v4789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:34 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:06:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:34 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:06:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:34 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:06:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:34 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:06:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:34 smithi107 ceph-mon[32659]: pgmap v4788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:06:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:34 smithi107 ceph-mon[32659]: pgmap v4789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:34 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:06:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:34 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:06:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:34 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:06:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:34 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:06:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:34 smithi132 ceph-mon[37580]: pgmap v4788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:06:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:34 smithi132 ceph-mon[37580]: pgmap v4789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8131.928790 secs 2024-07-26T15:06:35.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8131.928790 secs 2024-07-26T15:06:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8131.928790 secs 2024-07-26T15:06:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:36 smithi112 ceph-mon[37501]: pgmap v4790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:36 smithi107 ceph-mon[32659]: pgmap v4790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:36 smithi132 ceph-mon[37580]: pgmap v4790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:38 smithi112 ceph-mon[37501]: pgmap v4791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:38 smithi107 ceph-mon[32659]: pgmap v4791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:38 smithi132 ceph-mon[37580]: pgmap v4791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:40 smithi112 ceph-mon[37501]: pgmap v4792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8136.928906 secs 2024-07-26T15:06:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:40 smithi107 ceph-mon[32659]: pgmap v4792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8136.928906 secs 2024-07-26T15:06:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:40 smithi132 ceph-mon[37580]: pgmap v4792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8136.928906 secs 2024-07-26T15:06:42.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:42 smithi112 ceph-mon[37501]: pgmap v4793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:06:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:42 smithi107 ceph-mon[32659]: pgmap v4793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:06:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:42 smithi132 ceph-mon[37580]: pgmap v4793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:06:43.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:43.695+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:43.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:43.695+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:43.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:43.695+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:44 smithi112 ceph-mon[37501]: pgmap v4794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:06:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:44 smithi107 ceph-mon[32659]: pgmap v4794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:06:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:44 smithi132 ceph-mon[37580]: pgmap v4794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:06:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8141.929033 secs 2024-07-26T15:06:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8141.929033 secs 2024-07-26T15:06:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8141.929033 secs 2024-07-26T15:06:45.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:45.925+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:45.925 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:45.925+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:45.926 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:06:45.925+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:46 smithi112 ceph-mon[37501]: pgmap v4795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:46 smithi107 ceph-mon[32659]: pgmap v4795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:46 smithi132 ceph-mon[37580]: pgmap v4795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:48 smithi112 ceph-mon[37501]: pgmap v4796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:48 smithi107 ceph-mon[32659]: pgmap v4796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:48 smithi132 ceph-mon[37580]: pgmap v4796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:50 smithi107 ceph-mon[32659]: pgmap v4797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8146.929153 secs 2024-07-26T15:06:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:50 smithi132 ceph-mon[37580]: pgmap v4797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8146.929153 secs 2024-07-26T15:06:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:50 smithi112 ceph-mon[37501]: pgmap v4797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:06:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8146.929153 secs 2024-07-26T15:06:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:52 smithi107 ceph-mon[32659]: pgmap v4798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:52 smithi132 ceph-mon[37580]: pgmap v4798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:52 smithi112 ceph-mon[37501]: pgmap v4798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:54 smithi107 ceph-mon[32659]: pgmap v4799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:54 smithi132 ceph-mon[37580]: pgmap v4799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:54 smithi112 ceph-mon[37501]: pgmap v4799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8151.929263 secs 2024-07-26T15:06:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8151.929263 secs 2024-07-26T15:06:55.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8151.929263 secs 2024-07-26T15:06:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:56 smithi107 ceph-mon[32659]: pgmap v4800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:56 smithi132 ceph-mon[37580]: pgmap v4800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:56 smithi112 ceph-mon[37501]: pgmap v4800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:58.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:58.711+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:58.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:58.711+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:58.712 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:06:58.711+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:06:58.712 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:06:58 smithi107 ceph-mon[32659]: pgmap v4801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:06:58 smithi132 ceph-mon[37580]: pgmap v4801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:06:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:06:58 smithi112 ceph-mon[37501]: pgmap v4801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:07:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:00 smithi107 ceph-mon[32659]: pgmap v4802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8156.929382 secs 2024-07-26T15:07:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:00 smithi132 ceph-mon[37580]: pgmap v4802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8156.929382 secs 2024-07-26T15:07:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:00.940+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:00.940+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:00.940+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:00 smithi112 ceph-mon[37501]: pgmap v4802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8156.929382 secs 2024-07-26T15:07:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:02 smithi107 ceph-mon[32659]: pgmap v4803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:02 smithi132 ceph-mon[37580]: pgmap v4803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:02 smithi112 ceph-mon[37501]: pgmap v4803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:04 smithi107 ceph-mon[32659]: pgmap v4804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:04 smithi132 ceph-mon[37580]: pgmap v4804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:04 smithi112 ceph-mon[37501]: pgmap v4804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8161.929501 secs 2024-07-26T15:07:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8161.929501 secs 2024-07-26T15:07:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8161.929501 secs 2024-07-26T15:07:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:06 smithi107 ceph-mon[32659]: pgmap v4805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:06 smithi132 ceph-mon[37580]: pgmap v4805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:06 smithi112 ceph-mon[37501]: pgmap v4805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:08 smithi107 ceph-mon[32659]: pgmap v4806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:07:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:08 smithi132 ceph-mon[37580]: pgmap v4806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:07:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:08 smithi112 ceph-mon[37501]: pgmap v4806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:07:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:10 smithi107 ceph-mon[32659]: pgmap v4807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8166.929619 secs 2024-07-26T15:07:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:10 smithi132 ceph-mon[37580]: pgmap v4807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8166.929619 secs 2024-07-26T15:07:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:10 smithi112 ceph-mon[37501]: pgmap v4807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8166.929619 secs 2024-07-26T15:07:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:12 smithi107 ceph-mon[32659]: pgmap v4808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:12 smithi132 ceph-mon[37580]: pgmap v4808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:12 smithi112 ceph-mon[37501]: pgmap v4808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:13.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:13.726+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:13.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:13.726+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:13.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:13.726+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:14 smithi132 ceph-mon[37580]: pgmap v4809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:14 smithi112 ceph-mon[37501]: pgmap v4809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:14 smithi107 ceph-mon[32659]: pgmap v4809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8171.929747 secs 2024-07-26T15:07:15.943 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:15.943+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:15.943+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:15.943+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8171.929747 secs 2024-07-26T15:07:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8171.929747 secs 2024-07-26T15:07:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:16 smithi112 ceph-mon[37501]: pgmap v4810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:16 smithi107 ceph-mon[32659]: pgmap v4810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:16 smithi132 ceph-mon[37580]: pgmap v4810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:18 smithi112 ceph-mon[37501]: pgmap v4811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:07:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:18 smithi107 ceph-mon[32659]: pgmap v4811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:07:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:18 smithi132 ceph-mon[37580]: pgmap v4811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:07:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:20 smithi112 ceph-mon[37501]: pgmap v4812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8176.929869 secs 2024-07-26T15:07:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:20 smithi107 ceph-mon[32659]: pgmap v4812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8176.929869 secs 2024-07-26T15:07:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:20 smithi132 ceph-mon[37580]: pgmap v4812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8176.929869 secs 2024-07-26T15:07:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:22 smithi112 ceph-mon[37501]: pgmap v4813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:22 smithi107 ceph-mon[32659]: pgmap v4813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:22 smithi132 ceph-mon[37580]: pgmap v4813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:24 smithi112 ceph-mon[37501]: pgmap v4814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:24 smithi107 ceph-mon[32659]: pgmap v4814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:24 smithi132 ceph-mon[37580]: pgmap v4814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:25.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8181.929990 secs 2024-07-26T15:07:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8181.929990 secs 2024-07-26T15:07:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8181.929990 secs 2024-07-26T15:07:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:26 smithi112 ceph-mon[37501]: pgmap v4815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:26 smithi107 ceph-mon[32659]: pgmap v4815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:26 smithi132 ceph-mon[37580]: pgmap v4815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:28.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:28.741+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:28.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:28.741+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:28.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:28.741+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:28 smithi112 ceph-mon[37501]: pgmap v4816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:07:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:28 smithi107 ceph-mon[32659]: pgmap v4816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:07:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:28 smithi132 ceph-mon[37580]: pgmap v4816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:07:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:30 smithi112 ceph-mon[37501]: pgmap v4817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8186.930103 secs 2024-07-26T15:07:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:30.958+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:30.958+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:30.959 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:30.958+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:30 smithi107 ceph-mon[32659]: pgmap v4817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8186.930103 secs 2024-07-26T15:07:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:30 smithi132 ceph-mon[37580]: pgmap v4817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8186.930103 secs 2024-07-26T15:07:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:32 smithi112 ceph-mon[37501]: pgmap v4818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:32 smithi107 ceph-mon[32659]: pgmap v4818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:32 smithi132 ceph-mon[37580]: pgmap v4818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:07:34.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:07:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:07:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:34 smithi112 ceph-mon[37501]: pgmap v4819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:07:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:07:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:07:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:34 smithi107 ceph-mon[32659]: pgmap v4819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:07:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:07:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:07:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:34 smithi132 ceph-mon[37580]: pgmap v4819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:07:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:07:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:07:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:35 smithi112 ceph-mon[37501]: pgmap v4820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:07:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:35 smithi112 ceph-mon[37501]: pgmap v4821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8191.930221 secs 2024-07-26T15:07:35.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:35 smithi132 ceph-mon[37580]: pgmap v4820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:07:35.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:35 smithi132 ceph-mon[37580]: pgmap v4821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:35.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8191.930221 secs 2024-07-26T15:07:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:35 smithi107 ceph-mon[32659]: pgmap v4820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:07:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:35 smithi107 ceph-mon[32659]: pgmap v4821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8191.930221 secs 2024-07-26T15:07:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:37 smithi112 ceph-mon[37501]: pgmap v4822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:07:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:37 smithi107 ceph-mon[32659]: pgmap v4822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:07:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:37 smithi132 ceph-mon[37580]: pgmap v4822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:07:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:39 smithi112 ceph-mon[37501]: pgmap v4823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:39 smithi107 ceph-mon[32659]: pgmap v4823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:39 smithi132 ceph-mon[37580]: pgmap v4823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8196.930346 secs 2024-07-26T15:07:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8196.930346 secs 2024-07-26T15:07:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8196.930346 secs 2024-07-26T15:07:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:41 smithi112 ceph-mon[37501]: pgmap v4824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:41 smithi107 ceph-mon[32659]: pgmap v4824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:41 smithi132 ceph-mon[37580]: pgmap v4824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:42 smithi112 ceph-mon[37501]: pgmap v4825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:07:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:42 smithi107 ceph-mon[32659]: pgmap v4825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:07:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:42 smithi132 ceph-mon[37580]: pgmap v4825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:07:43.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:43.756+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:43.757+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:43.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:43.757+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:45 smithi132 ceph-mon[37580]: pgmap v4826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:07:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8201.930446 secs 2024-07-26T15:07:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:45 smithi112 ceph-mon[37501]: pgmap v4826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:07:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8201.930446 secs 2024-07-26T15:07:45.974 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:45.974+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:45.974+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:45.975 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:07:45.974+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:45 smithi107 ceph-mon[32659]: pgmap v4826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:07:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8201.930446 secs 2024-07-26T15:07:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:47 smithi132 ceph-mon[37580]: pgmap v4827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:47 smithi112 ceph-mon[37501]: pgmap v4827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:47 smithi107 ceph-mon[32659]: pgmap v4827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:49 smithi132 ceph-mon[37580]: pgmap v4828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:49 smithi112 ceph-mon[37501]: pgmap v4828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:49 smithi107 ceph-mon[32659]: pgmap v4828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8206.930601 secs 2024-07-26T15:07:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8206.930601 secs 2024-07-26T15:07:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8206.930601 secs 2024-07-26T15:07:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:51 smithi112 ceph-mon[37501]: pgmap v4829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:51 smithi107 ceph-mon[32659]: pgmap v4829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:51 smithi132 ceph-mon[37580]: pgmap v4829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:53 smithi112 ceph-mon[37501]: pgmap v4830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:53 smithi107 ceph-mon[32659]: pgmap v4830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:53 smithi132 ceph-mon[37580]: pgmap v4830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:07:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:55 smithi112 ceph-mon[37501]: pgmap v4831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:07:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8211.930722 secs 2024-07-26T15:07:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:55 smithi107 ceph-mon[32659]: pgmap v4831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:07:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8211.930722 secs 2024-07-26T15:07:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:55 smithi132 ceph-mon[37580]: pgmap v4831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:07:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8211.930722 secs 2024-07-26T15:07:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:57 smithi112 ceph-mon[37501]: pgmap v4832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:57 smithi107 ceph-mon[32659]: pgmap v4832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:57 smithi132 ceph-mon[37580]: pgmap v4832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:58.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:58.769+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:58.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:58.769+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:58.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:07:58.769+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:07:59.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:07:59 smithi107 ceph-mon[32659]: pgmap v4833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:07:59 smithi112 ceph-mon[37501]: pgmap v4833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:00.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:07:59 smithi132 ceph-mon[37580]: pgmap v4833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8216.930821 secs 2024-07-26T15:08:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:00.983+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:00.984+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:00.984+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8216.930821 secs 2024-07-26T15:08:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8216.930821 secs 2024-07-26T15:08:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:01 smithi112 ceph-mon[37501]: pgmap v4834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:01 smithi107 ceph-mon[32659]: pgmap v4834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:01 smithi132 ceph-mon[37580]: pgmap v4834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:03 smithi112 ceph-mon[37501]: pgmap v4835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:03 smithi107 ceph-mon[32659]: pgmap v4835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:04.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:03 smithi132 ceph-mon[37580]: pgmap v4835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:05 smithi112 ceph-mon[37501]: pgmap v4836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8221.930972 secs 2024-07-26T15:08:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:05 smithi107 ceph-mon[32659]: pgmap v4836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8221.930972 secs 2024-07-26T15:08:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:05 smithi132 ceph-mon[37580]: pgmap v4836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8221.930972 secs 2024-07-26T15:08:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:07 smithi112 ceph-mon[37501]: pgmap v4837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:07 smithi107 ceph-mon[32659]: pgmap v4837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:07 smithi132 ceph-mon[37580]: pgmap v4837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:09 smithi112 ceph-mon[37501]: pgmap v4838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:08:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:09 smithi107 ceph-mon[32659]: pgmap v4838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:08:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:09 smithi132 ceph-mon[37580]: pgmap v4838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:08:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8226.931113 secs 2024-07-26T15:08:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8226.931113 secs 2024-07-26T15:08:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8226.931113 secs 2024-07-26T15:08:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:11 smithi112 ceph-mon[37501]: pgmap v4839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:08:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:11 smithi107 ceph-mon[32659]: pgmap v4839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:08:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:11 smithi132 ceph-mon[37580]: pgmap v4839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:08:13.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:13.782+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:13.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:13.782+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:13.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:13.782+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:13 smithi112 ceph-mon[37501]: pgmap v4840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:08:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:13 smithi107 ceph-mon[32659]: pgmap v4840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:08:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:13 smithi132 ceph-mon[37580]: pgmap v4840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:08:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:15 smithi112 ceph-mon[37501]: pgmap v4841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8231.931229 secs 2024-07-26T15:08:16.000 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:15.999+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:16.000 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:15.999+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:16.000 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:15.999+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:15 smithi107 ceph-mon[32659]: pgmap v4841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8231.931229 secs 2024-07-26T15:08:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:15 smithi132 ceph-mon[37580]: pgmap v4841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8231.931229 secs 2024-07-26T15:08:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:17 smithi112 ceph-mon[37501]: pgmap v4842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:08:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:17 smithi107 ceph-mon[32659]: pgmap v4842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:08:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:17 smithi132 ceph-mon[37580]: pgmap v4842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:08:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:18 smithi107 ceph-mon[32659]: pgmap v4843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:08:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:18 smithi132 ceph-mon[37580]: pgmap v4843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:08:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:18 smithi112 ceph-mon[37501]: pgmap v4843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:08:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8236.931362 secs 2024-07-26T15:08:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8236.931362 secs 2024-07-26T15:08:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8236.931362 secs 2024-07-26T15:08:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:21 smithi112 ceph-mon[37501]: pgmap v4844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:21 smithi107 ceph-mon[32659]: pgmap v4844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:21 smithi132 ceph-mon[37580]: pgmap v4844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:23 smithi112 ceph-mon[37501]: pgmap v4845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:23 smithi107 ceph-mon[32659]: pgmap v4845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:23 smithi132 ceph-mon[37580]: pgmap v4845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:25 smithi112 ceph-mon[37501]: pgmap v4846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8241.931481 secs 2024-07-26T15:08:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:25 smithi107 ceph-mon[32659]: pgmap v4846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8241.931481 secs 2024-07-26T15:08:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:25 smithi132 ceph-mon[37580]: pgmap v4846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8241.931481 secs 2024-07-26T15:08:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:27 smithi112 ceph-mon[37501]: pgmap v4847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:27 smithi107 ceph-mon[32659]: pgmap v4847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:27 smithi132 ceph-mon[37580]: pgmap v4847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:28.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:28.797+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:28.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:28.797+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:28.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:28.797+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:29 smithi112 ceph-mon[37501]: pgmap v4848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:29 smithi107 ceph-mon[32659]: pgmap v4848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:29 smithi132 ceph-mon[37580]: pgmap v4848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8246.931618 secs 2024-07-26T15:08:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:31.009+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:31.009+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:31.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:31.009+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8246.931618 secs 2024-07-26T15:08:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8246.931618 secs 2024-07-26T15:08:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:31 smithi112 ceph-mon[37501]: pgmap v4849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:31 smithi107 ceph-mon[32659]: pgmap v4849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:31 smithi132 ceph-mon[37580]: pgmap v4849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:33 smithi112 ceph-mon[37501]: pgmap v4850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:33 smithi107 ceph-mon[32659]: pgmap v4850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:33 smithi132 ceph-mon[37580]: pgmap v4850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:35 smithi112 ceph-mon[37501]: pgmap v4851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:08:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8251.931773 secs 2024-07-26T15:08:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:08:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:08:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:35 smithi107 ceph-mon[32659]: pgmap v4851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:08:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8251.931773 secs 2024-07-26T15:08:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:08:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:08:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:35 smithi132 ceph-mon[37580]: pgmap v4851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:08:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:08:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8251.931773 secs 2024-07-26T15:08:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:08:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:08:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:08:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:08:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:08:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:08:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:08:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:08:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:08:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:08:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:08:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:37 smithi112 ceph-mon[37501]: pgmap v4852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T15:08:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:37 smithi112 ceph-mon[37501]: pgmap v4853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:37 smithi107 ceph-mon[32659]: pgmap v4852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T15:08:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:37 smithi107 ceph-mon[32659]: pgmap v4853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:37 smithi132 ceph-mon[37580]: pgmap v4852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T15:08:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:37 smithi132 ceph-mon[37580]: pgmap v4853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:39 smithi112 ceph-mon[37501]: pgmap v4854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:39 smithi107 ceph-mon[32659]: pgmap v4854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:39 smithi132 ceph-mon[37580]: pgmap v4854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8256.931841 secs 2024-07-26T15:08:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8256.931841 secs 2024-07-26T15:08:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8256.931841 secs 2024-07-26T15:08:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:41 smithi112 ceph-mon[37501]: pgmap v4855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:41 smithi107 ceph-mon[32659]: pgmap v4855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:41 smithi132 ceph-mon[37580]: pgmap v4855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:43.813+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:43.813+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:43.815 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:43.813+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:43 smithi112 ceph-mon[37501]: pgmap v4856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T15:08:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:43 smithi107 ceph-mon[32659]: pgmap v4856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T15:08:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:43 smithi132 ceph-mon[37580]: pgmap v4856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T15:08:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:44 smithi112 ceph-mon[37501]: pgmap v4857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:44 smithi107 ceph-mon[32659]: pgmap v4857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:44 smithi132 ceph-mon[37580]: pgmap v4857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8261.931961 secs 2024-07-26T15:08:46.026 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:46.026+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:46.026+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:46.027 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:08:46.026+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8261.931961 secs 2024-07-26T15:08:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8261.931961 secs 2024-07-26T15:08:46.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:46 smithi107 ceph-mon[32659]: pgmap v4858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:08:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:46 smithi132 ceph-mon[37580]: pgmap v4858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:08:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:46 smithi112 ceph-mon[37501]: pgmap v4858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:08:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:49 smithi112 ceph-mon[37501]: pgmap v4859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:49 smithi107 ceph-mon[32659]: pgmap v4859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:49 smithi132 ceph-mon[37580]: pgmap v4859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8266.932045 secs 2024-07-26T15:08:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:51 smithi112 ceph-mon[37501]: pgmap v4860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8266.932045 secs 2024-07-26T15:08:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:51 smithi107 ceph-mon[32659]: pgmap v4860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8266.932045 secs 2024-07-26T15:08:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:51 smithi132 ceph-mon[37580]: pgmap v4860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:53 smithi112 ceph-mon[37501]: pgmap v4861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:08:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:53 smithi107 ceph-mon[32659]: pgmap v4861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:08:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:53 smithi132 ceph-mon[37580]: pgmap v4861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:08:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:55 smithi112 ceph-mon[37501]: pgmap v4862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:08:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:55 smithi107 ceph-mon[32659]: pgmap v4862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:08:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:55 smithi132 ceph-mon[37580]: pgmap v4862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:08:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8271.932142 secs 2024-07-26T15:08:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8271.932142 secs 2024-07-26T15:08:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8271.932142 secs 2024-07-26T15:08:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:57 smithi112 ceph-mon[37501]: pgmap v4863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:57 smithi107 ceph-mon[32659]: pgmap v4863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:57 smithi132 ceph-mon[37580]: pgmap v4863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:58.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:58.829+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:58.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:58.829+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:58.831 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:08:58.829+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:08:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:08:59 smithi112 ceph-mon[37501]: pgmap v4864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:08:59 smithi107 ceph-mon[32659]: pgmap v4864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:08:59 smithi132 ceph-mon[37580]: pgmap v4864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8276.932265 secs 2024-07-26T15:09:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8276.932265 secs 2024-07-26T15:09:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8276.932265 secs 2024-07-26T15:09:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:01.041+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:01.041+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:01.041+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:01 smithi112 ceph-mon[37501]: pgmap v4865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:01 smithi107 ceph-mon[32659]: pgmap v4865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:01 smithi132 ceph-mon[37580]: pgmap v4865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:03 smithi112 ceph-mon[37501]: pgmap v4866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:03 smithi107 ceph-mon[32659]: pgmap v4866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:03 smithi132 ceph-mon[37580]: pgmap v4866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:05 smithi112 ceph-mon[37501]: pgmap v4867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8281.932351 secs 2024-07-26T15:09:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:05 smithi107 ceph-mon[32659]: pgmap v4867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8281.932351 secs 2024-07-26T15:09:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:05 smithi132 ceph-mon[37580]: pgmap v4867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8281.932351 secs 2024-07-26T15:09:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:07 smithi112 ceph-mon[37501]: pgmap v4868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:07 smithi107 ceph-mon[32659]: pgmap v4868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:07 smithi132 ceph-mon[37580]: pgmap v4868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:09 smithi112 ceph-mon[37501]: pgmap v4869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:09 smithi107 ceph-mon[32659]: pgmap v4869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:09 smithi132 ceph-mon[37580]: pgmap v4869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8286.932433 secs 2024-07-26T15:09:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8286.932433 secs 2024-07-26T15:09:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8286.932433 secs 2024-07-26T15:09:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:11 smithi112 ceph-mon[37501]: pgmap v4870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:11 smithi107 ceph-mon[32659]: pgmap v4870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:11 smithi132 ceph-mon[37580]: pgmap v4870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:13 smithi112 ceph-mon[37501]: pgmap v4871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:13 smithi107 ceph-mon[32659]: pgmap v4871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:13 smithi132 ceph-mon[37580]: pgmap v4871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:13.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:13.834+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:13.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:13.834+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:13.835 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:13.834+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:14.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:14.039+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:09:14.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:14.040+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:09:14.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:14.048+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:09:14.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:14.049+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:09:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:15 smithi112 ceph-mon[37501]: pgmap v4872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8291.932515 secs 2024-07-26T15:09:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:15 smithi107 ceph-mon[32659]: pgmap v4872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8291.932515 secs 2024-07-26T15:09:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:15 smithi132 ceph-mon[37580]: pgmap v4872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8291.932515 secs 2024-07-26T15:09:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:16.054+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:16.054+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:16.054+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:16.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:16.226+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:09:16.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:16.226+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:09:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:16.226+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:09:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:16.226+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:09:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:17 smithi107 ceph-mon[32659]: pgmap v4873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:17.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:17 smithi132 ceph-mon[37580]: pgmap v4873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:17 smithi112 ceph-mon[37501]: pgmap v4873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:19 smithi107 ceph-mon[32659]: pgmap v4874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:19 smithi132 ceph-mon[37580]: pgmap v4874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:19 smithi112 ceph-mon[37501]: pgmap v4874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8296.932580 secs 2024-07-26T15:09:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8296.932580 secs 2024-07-26T15:09:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8296.932580 secs 2024-07-26T15:09:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:21 smithi107 ceph-mon[32659]: pgmap v4875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:21 smithi132 ceph-mon[37580]: pgmap v4875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:21 smithi112 ceph-mon[37501]: pgmap v4875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:23 smithi107 ceph-mon[32659]: pgmap v4876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:23 smithi132 ceph-mon[37580]: pgmap v4876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:23 smithi112 ceph-mon[37501]: pgmap v4876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:25 smithi107 ceph-mon[32659]: pgmap v4877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8301.932697 secs 2024-07-26T15:09:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:25 smithi132 ceph-mon[37580]: pgmap v4877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8301.932697 secs 2024-07-26T15:09:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:25 smithi112 ceph-mon[37501]: pgmap v4877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8301.932697 secs 2024-07-26T15:09:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:27 smithi107 ceph-mon[32659]: pgmap v4878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:27.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:27 smithi132 ceph-mon[37580]: pgmap v4878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:27 smithi112 ceph-mon[37501]: pgmap v4878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:28.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:28.849+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:28.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:28.849+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:28.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:28.849+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:29 smithi107 ceph-mon[32659]: pgmap v4879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:29 smithi132 ceph-mon[37580]: pgmap v4879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:29 smithi112 ceph-mon[37501]: pgmap v4879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8306.932819 secs 2024-07-26T15:09:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8306.932819 secs 2024-07-26T15:09:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8306.932819 secs 2024-07-26T15:09:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:31.069+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:31.069+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:31.069+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:31 smithi107 ceph-mon[32659]: pgmap v4880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:31 smithi132 ceph-mon[37580]: pgmap v4880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:31 smithi112 ceph-mon[37501]: pgmap v4880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:33 smithi132 ceph-mon[37580]: pgmap v4881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:33 smithi112 ceph-mon[37501]: pgmap v4881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:33 smithi107 ceph-mon[32659]: pgmap v4881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:35 smithi132 ceph-mon[37580]: pgmap v4882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8311.932939 secs 2024-07-26T15:09:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:35 smithi112 ceph-mon[37501]: pgmap v4882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8311.932939 secs 2024-07-26T15:09:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:35 smithi107 ceph-mon[32659]: pgmap v4882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:09:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8311.932939 secs 2024-07-26T15:09:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:09:37.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:09:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:09:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:37 smithi112 ceph-mon[37501]: pgmap v4883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:37 smithi107 ceph-mon[32659]: pgmap v4883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:37 smithi132 ceph-mon[37580]: pgmap v4883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:39 smithi112 ceph-mon[37501]: pgmap v4884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:39 smithi107 ceph-mon[32659]: pgmap v4884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:39 smithi132 ceph-mon[37580]: pgmap v4884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8316.933055 secs 2024-07-26T15:09:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8316.933055 secs 2024-07-26T15:09:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8316.933055 secs 2024-07-26T15:09:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:41 smithi107 ceph-mon[32659]: pgmap v4885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:09:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:09:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:41 smithi132 ceph-mon[37580]: pgmap v4885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:09:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:09:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:41 smithi112 ceph-mon[37501]: pgmap v4885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:09:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:09:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:09:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:42 smithi107 ceph-mon[32659]: pgmap v4886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:09:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:42 smithi107 ceph-mon[32659]: pgmap v4887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:09:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:42 smithi132 ceph-mon[37580]: pgmap v4886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:09:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:42 smithi132 ceph-mon[37580]: pgmap v4887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:09:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:42 smithi112 ceph-mon[37501]: pgmap v4886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:09:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:42 smithi112 ceph-mon[37501]: pgmap v4887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:09:43.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:43.865+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:43.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:43.865+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:43.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:43.865+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:44 smithi107 ceph-mon[32659]: pgmap v4888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:09:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:44 smithi132 ceph-mon[37580]: pgmap v4888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:09:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:44 smithi112 ceph-mon[37501]: pgmap v4888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:09:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8321.933108 secs 2024-07-26T15:09:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8321.933108 secs 2024-07-26T15:09:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8321.933108 secs 2024-07-26T15:09:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:46.085+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:46.085+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:46.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:09:46.085+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:46 smithi107 ceph-mon[32659]: pgmap v4889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:09:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:46 smithi132 ceph-mon[37580]: pgmap v4889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:09:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:46 smithi112 ceph-mon[37501]: pgmap v4889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:09:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:48 smithi107 ceph-mon[32659]: pgmap v4890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:09:48.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:48 smithi132 ceph-mon[37580]: pgmap v4890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:09:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:48 smithi112 ceph-mon[37501]: pgmap v4890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:09:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:50 smithi107 ceph-mon[32659]: pgmap v4891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:09:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8326.933188 secs 2024-07-26T15:09:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:50 smithi132 ceph-mon[37580]: pgmap v4891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:09:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8326.933188 secs 2024-07-26T15:09:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:50 smithi112 ceph-mon[37501]: pgmap v4891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:09:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8326.933188 secs 2024-07-26T15:09:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:52 smithi107 ceph-mon[32659]: pgmap v4892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:09:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:52 smithi132 ceph-mon[37580]: pgmap v4892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:09:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:52 smithi112 ceph-mon[37501]: pgmap v4892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:09:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:54 smithi107 ceph-mon[32659]: pgmap v4893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:09:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:54 smithi132 ceph-mon[37580]: pgmap v4893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:09:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:54 smithi112 ceph-mon[37501]: pgmap v4893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:09:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8331.933306 secs 2024-07-26T15:09:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8331.933306 secs 2024-07-26T15:09:55.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8331.933306 secs 2024-07-26T15:09:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:56 smithi107 ceph-mon[32659]: pgmap v4894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:09:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:56 smithi132 ceph-mon[37580]: pgmap v4894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:09:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:56 smithi112 ceph-mon[37501]: pgmap v4894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:09:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:09:58 smithi132 ceph-mon[37580]: pgmap v4895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:09:58.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:58.881+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:58.881+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:58.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:09:58.881+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:09:58.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:09:58 smithi107 ceph-mon[32659]: pgmap v4895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:09:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:09:58 smithi112 ceph-mon[37501]: pgmap v4895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:10:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:00 smithi132 ceph-mon[37580]: pgmap v4896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:10:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8336.933376 secs 2024-07-26T15:10:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:00 smithi112 ceph-mon[37501]: pgmap v4896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:10:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8336.933376 secs 2024-07-26T15:10:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:00 smithi107 ceph-mon[32659]: pgmap v4896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:10:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8336.933376 secs 2024-07-26T15:10:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:01.101+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:01.101+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:01.101+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:02 smithi132 ceph-mon[37580]: pgmap v4897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:10:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:02 smithi112 ceph-mon[37501]: pgmap v4897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:10:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:02 smithi107 ceph-mon[32659]: pgmap v4897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:10:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:04 smithi112 ceph-mon[37501]: pgmap v4898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:10:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:04 smithi107 ceph-mon[32659]: pgmap v4898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:10:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:04 smithi132 ceph-mon[37580]: pgmap v4898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:10:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8341.933476 secs 2024-07-26T15:10:05.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8341.933476 secs 2024-07-26T15:10:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8341.933476 secs 2024-07-26T15:10:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:06 smithi112 ceph-mon[37501]: pgmap v4899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:06.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:06 smithi132 ceph-mon[37580]: pgmap v4899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:06 smithi107 ceph-mon[32659]: pgmap v4899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:08 smithi112 ceph-mon[37501]: pgmap v4900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:08 smithi107 ceph-mon[32659]: pgmap v4900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:08 smithi132 ceph-mon[37580]: pgmap v4900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:10 smithi112 ceph-mon[37501]: pgmap v4901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8346.933599 secs 2024-07-26T15:10:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:10 smithi107 ceph-mon[32659]: pgmap v4901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8346.933599 secs 2024-07-26T15:10:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:10 smithi132 ceph-mon[37580]: pgmap v4901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8346.933599 secs 2024-07-26T15:10:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:12 smithi112 ceph-mon[37501]: pgmap v4902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:12 smithi107 ceph-mon[32659]: pgmap v4902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:12 smithi132 ceph-mon[37580]: pgmap v4902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:13.891 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:13.890+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:13.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:13.890+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:13.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:13.890+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:14 smithi112 ceph-mon[37501]: pgmap v4903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:14 smithi107 ceph-mon[32659]: pgmap v4903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:14 smithi132 ceph-mon[37580]: pgmap v4903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8351.933739 secs 2024-07-26T15:10:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8351.933739 secs 2024-07-26T15:10:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8351.933739 secs 2024-07-26T15:10:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:16.116+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:16.116+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:16.117 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:16.116+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:16 smithi112 ceph-mon[37501]: pgmap v4904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:16 smithi107 ceph-mon[32659]: pgmap v4904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:16 smithi132 ceph-mon[37580]: pgmap v4904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:18 smithi112 ceph-mon[37501]: pgmap v4905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:18 smithi107 ceph-mon[32659]: pgmap v4905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:18 smithi132 ceph-mon[37580]: pgmap v4905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:20 smithi112 ceph-mon[37501]: pgmap v4906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8356.933879 secs 2024-07-26T15:10:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:20 smithi107 ceph-mon[32659]: pgmap v4906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8356.933879 secs 2024-07-26T15:10:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:20 smithi132 ceph-mon[37580]: pgmap v4906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8356.933879 secs 2024-07-26T15:10:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:22 smithi112 ceph-mon[37501]: pgmap v4907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:22 smithi107 ceph-mon[32659]: pgmap v4907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:22 smithi132 ceph-mon[37580]: pgmap v4907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:24 smithi112 ceph-mon[37501]: pgmap v4908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:24 smithi107 ceph-mon[32659]: pgmap v4908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:24 smithi132 ceph-mon[37580]: pgmap v4908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8361.933992 secs 2024-07-26T15:10:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8361.933992 secs 2024-07-26T15:10:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8361.933992 secs 2024-07-26T15:10:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:26 smithi112 ceph-mon[37501]: pgmap v4909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:26 smithi107 ceph-mon[32659]: pgmap v4909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:26 smithi132 ceph-mon[37580]: pgmap v4909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:28.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:28.905+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:28.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:28.905+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:28.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:28.905+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:28.908 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:28 smithi107 ceph-mon[32659]: pgmap v4910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:28 smithi112 ceph-mon[37501]: pgmap v4910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:28 smithi132 ceph-mon[37580]: pgmap v4910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:30 smithi112 ceph-mon[37501]: pgmap v4911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8366.934104 secs 2024-07-26T15:10:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:30 smithi107 ceph-mon[32659]: pgmap v4911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8366.934104 secs 2024-07-26T15:10:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:30 smithi132 ceph-mon[37580]: pgmap v4911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8366.934104 secs 2024-07-26T15:10:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:31.131+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:31.131+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:31.131+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:32 smithi112 ceph-mon[37501]: pgmap v4912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:32 smithi107 ceph-mon[32659]: pgmap v4912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:32 smithi132 ceph-mon[37580]: pgmap v4912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:34 smithi112 ceph-mon[37501]: pgmap v4913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:34 smithi107 ceph-mon[32659]: pgmap v4913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:34 smithi132 ceph-mon[37580]: pgmap v4913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8371.934219 secs 2024-07-26T15:10:35.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8371.934219 secs 2024-07-26T15:10:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8371.934219 secs 2024-07-26T15:10:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:36 smithi112 ceph-mon[37501]: pgmap v4914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:37.018 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:36 smithi107 ceph-mon[32659]: pgmap v4914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:36 smithi132 ceph-mon[37580]: pgmap v4914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:37 smithi107 ceph-mon[32659]: pgmap v4915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:37 smithi132 ceph-mon[37580]: pgmap v4915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:37 smithi112 ceph-mon[37501]: pgmap v4915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:10:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:40 smithi132 ceph-mon[37580]: pgmap v4916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8376.934291 secs 2024-07-26T15:10:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:40 smithi112 ceph-mon[37501]: pgmap v4916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8376.934291 secs 2024-07-26T15:10:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:40 smithi107 ceph-mon[32659]: pgmap v4916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:10:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8376.934291 secs 2024-07-26T15:10:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:42 smithi132 ceph-mon[37580]: pgmap v4917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:10:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:42 smithi112 ceph-mon[37501]: pgmap v4917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:10:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:42 smithi107 ceph-mon[32659]: pgmap v4917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:10:43.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:10:43.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:10:43.924 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:43 smithi107 ceph-mon[32659]: pgmap v4918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:10:43.924 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:43 smithi107 ceph-mon[32659]: pgmap v4919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:10:43.924 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:10:43.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:43.921+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:43.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:43.921+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:43.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:43.921+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:10:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:10:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:43 smithi112 ceph-mon[37501]: pgmap v4918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:10:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:43 smithi112 ceph-mon[37501]: pgmap v4919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:10:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:10:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:10:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:10:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:43 smithi132 ceph-mon[37580]: pgmap v4918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:10:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:43 smithi132 ceph-mon[37580]: pgmap v4919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:10:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:10:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:44 smithi107 ceph-mon[32659]: pgmap v4920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:10:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:44 smithi132 ceph-mon[37580]: pgmap v4920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:10:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:44 smithi112 ceph-mon[37501]: pgmap v4920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:10:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8381.934392 secs 2024-07-26T15:10:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8381.934392 secs 2024-07-26T15:10:46.138 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:46.134+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:46.134+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:46.139 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:10:46.134+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8381.934392 secs 2024-07-26T15:10:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:46 smithi107 ceph-mon[32659]: pgmap v4921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:46 smithi132 ceph-mon[37580]: pgmap v4921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:46 smithi112 ceph-mon[37501]: pgmap v4921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:48 smithi107 ceph-mon[32659]: pgmap v4922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:10:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:48 smithi132 ceph-mon[37580]: pgmap v4922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:10:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:48 smithi112 ceph-mon[37501]: pgmap v4922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:10:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8386.934514 secs 2024-07-26T15:10:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8386.934514 secs 2024-07-26T15:10:50.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8386.934514 secs 2024-07-26T15:10:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:51 smithi112 ceph-mon[37501]: pgmap v4923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:10:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:51 smithi107 ceph-mon[32659]: pgmap v4923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:10:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:51 smithi132 ceph-mon[37580]: pgmap v4923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:10:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:53 smithi107 ceph-mon[32659]: pgmap v4924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:10:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:53 smithi132 ceph-mon[37580]: pgmap v4924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:10:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:53 smithi112 ceph-mon[37501]: pgmap v4924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:10:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:55 smithi107 ceph-mon[32659]: pgmap v4925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:10:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8391.934637 secs 2024-07-26T15:10:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:55 smithi132 ceph-mon[37580]: pgmap v4925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:10:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8391.934637 secs 2024-07-26T15:10:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:55 smithi112 ceph-mon[37501]: pgmap v4925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:10:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8391.934637 secs 2024-07-26T15:10:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:57 smithi107 ceph-mon[32659]: pgmap v4926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:10:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:57 smithi132 ceph-mon[37580]: pgmap v4926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:10:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:57 smithi112 ceph-mon[37501]: pgmap v4926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:10:58.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:58.930+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:10:58.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:58.930+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:58.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:10:58.930+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:10:59 smithi107 ceph-mon[32659]: pgmap v4927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:00.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:10:59 smithi132 ceph-mon[37580]: pgmap v4927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:10:59 smithi112 ceph-mon[37501]: pgmap v4927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8396.934718 secs 2024-07-26T15:11:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8396.934718 secs 2024-07-26T15:11:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:01.143+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:01.143+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:01.143+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8396.934718 secs 2024-07-26T15:11:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:01 smithi107 ceph-mon[32659]: pgmap v4928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:01 smithi132 ceph-mon[37580]: pgmap v4928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:01 smithi112 ceph-mon[37501]: pgmap v4928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:03 smithi107 ceph-mon[32659]: pgmap v4929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:04.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:03 smithi132 ceph-mon[37580]: pgmap v4929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:03 smithi112 ceph-mon[37501]: pgmap v4929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:05 smithi132 ceph-mon[37580]: pgmap v4930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8401.934856 secs 2024-07-26T15:11:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:05 smithi112 ceph-mon[37501]: pgmap v4930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8401.934856 secs 2024-07-26T15:11:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:05 smithi107 ceph-mon[32659]: pgmap v4930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8401.934856 secs 2024-07-26T15:11:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:07 smithi132 ceph-mon[37580]: pgmap v4931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:07 smithi112 ceph-mon[37501]: pgmap v4931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:07 smithi107 ceph-mon[32659]: pgmap v4931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:09 smithi132 ceph-mon[37580]: pgmap v4932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:09 smithi112 ceph-mon[37501]: pgmap v4932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:09 smithi107 ceph-mon[32659]: pgmap v4932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8406.934974 secs 2024-07-26T15:11:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8406.934974 secs 2024-07-26T15:11:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8406.934974 secs 2024-07-26T15:11:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:11 smithi112 ceph-mon[37501]: pgmap v4933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:11 smithi107 ceph-mon[32659]: pgmap v4933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:11 smithi132 ceph-mon[37580]: pgmap v4933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:13.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:13.946+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:13.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:13.946+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:13.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:13.946+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:13 smithi112 ceph-mon[37501]: pgmap v4934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:13 smithi107 ceph-mon[32659]: pgmap v4934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:14.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:13 smithi132 ceph-mon[37580]: pgmap v4934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:16.159+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:16.159+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:16.159+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:15 smithi112 ceph-mon[37501]: pgmap v4935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8411.935098 secs 2024-07-26T15:11:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:15 smithi107 ceph-mon[32659]: pgmap v4935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8411.935098 secs 2024-07-26T15:11:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:15 smithi132 ceph-mon[37580]: pgmap v4935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8411.935098 secs 2024-07-26T15:11:18.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:17 smithi107 ceph-mon[32659]: pgmap v4936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:17 smithi112 ceph-mon[37501]: pgmap v4936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:17 smithi132 ceph-mon[37580]: pgmap v4936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:19 smithi112 ceph-mon[37501]: pgmap v4937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:19 smithi107 ceph-mon[32659]: pgmap v4937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:19 smithi132 ceph-mon[37580]: pgmap v4937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8416.935217 secs 2024-07-26T15:11:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8416.935217 secs 2024-07-26T15:11:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8416.935217 secs 2024-07-26T15:11:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:21 smithi112 ceph-mon[37501]: pgmap v4938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:21 smithi107 ceph-mon[32659]: pgmap v4938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:21 smithi132 ceph-mon[37580]: pgmap v4938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:23 smithi112 ceph-mon[37501]: pgmap v4939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:23 smithi107 ceph-mon[32659]: pgmap v4939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:24.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:23 smithi132 ceph-mon[37580]: pgmap v4939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:25 smithi112 ceph-mon[37501]: pgmap v4940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8421.935298 secs 2024-07-26T15:11:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:25 smithi107 ceph-mon[32659]: pgmap v4940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8421.935298 secs 2024-07-26T15:11:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:25 smithi132 ceph-mon[37580]: pgmap v4940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8421.935298 secs 2024-07-26T15:11:28.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:28.062+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:11:28.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:28.062+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:11:28.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:28.069+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:11:28.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:28.070+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:11:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:27 smithi112 ceph-mon[37501]: pgmap v4941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:27 smithi107 ceph-mon[32659]: pgmap v4941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:27 smithi132 ceph-mon[37580]: pgmap v4941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:28.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:28.954+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:28.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:28.954+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:28.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:28.954+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:29 smithi112 ceph-mon[37501]: pgmap v4942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:30.249+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:11:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:30.249+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:11:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:30.249+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:11:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:30.249+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:11:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:29 smithi107 ceph-mon[32659]: pgmap v4942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:30.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:29 smithi132 ceph-mon[37580]: pgmap v4942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:31.165+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:31.165+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:31.167 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:31.165+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8426.935419 secs 2024-07-26T15:11:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8426.935419 secs 2024-07-26T15:11:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8426.935419 secs 2024-07-26T15:11:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:31 smithi112 ceph-mon[37501]: pgmap v4943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:31 smithi107 ceph-mon[32659]: pgmap v4943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:31 smithi132 ceph-mon[37580]: pgmap v4943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:33 smithi112 ceph-mon[37501]: pgmap v4944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:33 smithi107 ceph-mon[32659]: pgmap v4944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:34.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:33 smithi132 ceph-mon[37580]: pgmap v4944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:35 smithi112 ceph-mon[37501]: pgmap v4945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8431.935540 secs 2024-07-26T15:11:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:35 smithi107 ceph-mon[32659]: pgmap v4945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8431.935540 secs 2024-07-26T15:11:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:35 smithi132 ceph-mon[37580]: pgmap v4945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8431.935540 secs 2024-07-26T15:11:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:37 smithi112 ceph-mon[37501]: pgmap v4946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:37 smithi107 ceph-mon[32659]: pgmap v4946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:38.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:37 smithi132 ceph-mon[37580]: pgmap v4946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:11:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:39 smithi112 ceph-mon[37501]: pgmap v4947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:39 smithi107 ceph-mon[32659]: pgmap v4947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:40.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:39 smithi132 ceph-mon[37580]: pgmap v4947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8436.935662 secs 2024-07-26T15:11:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8436.935662 secs 2024-07-26T15:11:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8436.935662 secs 2024-07-26T15:11:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:41 smithi112 ceph-mon[37501]: pgmap v4948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:41 smithi107 ceph-mon[32659]: pgmap v4948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:42.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:41 smithi132 ceph-mon[37580]: pgmap v4948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:43.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:11:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:11:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:11:43.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:43.969+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:43.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:43.969+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:43.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:43.969+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:43 smithi112 ceph-mon[37501]: pgmap v4949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:11:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:11:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:11:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:43 smithi107 ceph-mon[32659]: pgmap v4949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:11:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:11:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:11:44.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:43 smithi132 ceph-mon[37580]: pgmap v4949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:11:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:11:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:11:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:44 smithi107 ceph-mon[32659]: pgmap v4950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:11:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:44 smithi107 ceph-mon[32659]: pgmap v4951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:11:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:44 smithi132 ceph-mon[37580]: pgmap v4950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:11:45.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:44 smithi132 ceph-mon[37580]: pgmap v4951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:11:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:44 smithi112 ceph-mon[37501]: pgmap v4950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:11:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:44 smithi112 ceph-mon[37501]: pgmap v4951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:11:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:46.181+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:46.181+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:46.183 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:11:46.181+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8441.935785 secs 2024-07-26T15:11:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8441.935785 secs 2024-07-26T15:11:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8441.935785 secs 2024-07-26T15:11:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:46 smithi107 ceph-mon[32659]: pgmap v4952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:11:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:46 smithi132 ceph-mon[37580]: pgmap v4952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:11:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:46 smithi112 ceph-mon[37501]: pgmap v4952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:11:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:48 smithi107 ceph-mon[32659]: pgmap v4953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:11:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:48 smithi132 ceph-mon[37580]: pgmap v4953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:11:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:48 smithi112 ceph-mon[37501]: pgmap v4953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:11:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:50 smithi107 ceph-mon[32659]: pgmap v4954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:11:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8446.935915 secs 2024-07-26T15:11:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:50 smithi132 ceph-mon[37580]: pgmap v4954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:11:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8446.935915 secs 2024-07-26T15:11:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:50 smithi112 ceph-mon[37501]: pgmap v4954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:11:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8446.935915 secs 2024-07-26T15:11:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:52 smithi107 ceph-mon[32659]: pgmap v4955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:52 smithi132 ceph-mon[37580]: pgmap v4955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:52 smithi112 ceph-mon[37501]: pgmap v4955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:55 smithi107 ceph-mon[32659]: pgmap v4956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:11:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:55 smithi132 ceph-mon[37580]: pgmap v4956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:11:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:55 smithi112 ceph-mon[37501]: pgmap v4956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:11:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8451.936035 secs 2024-07-26T15:11:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8451.936035 secs 2024-07-26T15:11:56.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8451.936035 secs 2024-07-26T15:11:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:57 smithi107 ceph-mon[32659]: pgmap v4957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:57 smithi132 ceph-mon[37580]: pgmap v4957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:57 smithi112 ceph-mon[37501]: pgmap v4957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:58.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:58.983+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:58.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:58.983+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:11:58.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:11:58.983+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:11:59 smithi107 ceph-mon[32659]: pgmap v4958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:11:59 smithi132 ceph-mon[37580]: pgmap v4958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:11:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:11:59 smithi112 ceph-mon[37501]: pgmap v4958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:01.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:01.197+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:01.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:01.197+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:01.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:01.197+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:01 smithi107 ceph-mon[32659]: pgmap v4959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8456.936156 secs 2024-07-26T15:12:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:01 smithi132 ceph-mon[37580]: pgmap v4959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8456.936156 secs 2024-07-26T15:12:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:01 smithi112 ceph-mon[37501]: pgmap v4959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8456.936156 secs 2024-07-26T15:12:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:03 smithi132 ceph-mon[37580]: pgmap v4960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:03 smithi112 ceph-mon[37501]: pgmap v4960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:03 smithi107 ceph-mon[32659]: pgmap v4960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:05 smithi132 ceph-mon[37580]: pgmap v4961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:05 smithi112 ceph-mon[37501]: pgmap v4961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:05 smithi107 ceph-mon[32659]: pgmap v4961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8461.936244 secs 2024-07-26T15:12:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8461.936244 secs 2024-07-26T15:12:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8461.936244 secs 2024-07-26T15:12:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:07 smithi112 ceph-mon[37501]: pgmap v4962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:07 smithi107 ceph-mon[32659]: pgmap v4962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:07 smithi132 ceph-mon[37580]: pgmap v4962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:09.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:09 smithi107 ceph-mon[32659]: pgmap v4963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:09 smithi112 ceph-mon[37501]: pgmap v4963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:09.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:09 smithi132 ceph-mon[37580]: pgmap v4963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:11 smithi112 ceph-mon[37501]: pgmap v4964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8466.936331 secs 2024-07-26T15:12:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:11 smithi107 ceph-mon[32659]: pgmap v4964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8466.936331 secs 2024-07-26T15:12:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:11 smithi132 ceph-mon[37580]: pgmap v4964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8466.936331 secs 2024-07-26T15:12:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:13 smithi112 ceph-mon[37501]: pgmap v4965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:13 smithi107 ceph-mon[32659]: pgmap v4965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:13 smithi132 ceph-mon[37580]: pgmap v4965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:13.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:13.995+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:13.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:13.995+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:13.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:13.995+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:14.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:14.070+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:12:14.072 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:14.071+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:12:14.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:14.077+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:12:14.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:14.078+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:12:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:15 smithi112 ceph-mon[37501]: pgmap v4966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:15 smithi107 ceph-mon[32659]: pgmap v4966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:15.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:15 smithi132 ceph-mon[37580]: pgmap v4966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:16.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:16.210+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:16.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:16.210+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:16.211 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:16.210+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:16.258+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:12:16.258 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:16.258+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:12:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:16.258+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:12:16.259 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:16.259+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:12:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8471.936415 secs 2024-07-26T15:12:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8471.936415 secs 2024-07-26T15:12:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8471.936415 secs 2024-07-26T15:12:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:17 smithi112 ceph-mon[37501]: pgmap v4967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:17 smithi107 ceph-mon[32659]: pgmap v4967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:17 smithi132 ceph-mon[37580]: pgmap v4967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:19 smithi112 ceph-mon[37501]: pgmap v4968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:19 smithi107 ceph-mon[32659]: pgmap v4968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:19.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:19 smithi132 ceph-mon[37580]: pgmap v4968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:21 smithi112 ceph-mon[37501]: pgmap v4969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8476.936523 secs 2024-07-26T15:12:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:21 smithi107 ceph-mon[32659]: pgmap v4969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8476.936523 secs 2024-07-26T15:12:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:21 smithi132 ceph-mon[37580]: pgmap v4969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:21.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8476.936523 secs 2024-07-26T15:12:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:23 smithi112 ceph-mon[37501]: pgmap v4970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:23 smithi107 ceph-mon[32659]: pgmap v4970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:23 smithi132 ceph-mon[37580]: pgmap v4970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:25 smithi112 ceph-mon[37501]: pgmap v4971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:25 smithi107 ceph-mon[32659]: pgmap v4971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:25 smithi132 ceph-mon[37580]: pgmap v4971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8481.936661 secs 2024-07-26T15:12:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8481.936661 secs 2024-07-26T15:12:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8481.936661 secs 2024-07-26T15:12:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:27 smithi112 ceph-mon[37501]: pgmap v4972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:27 smithi107 ceph-mon[32659]: pgmap v4972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:27.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:27 smithi132 ceph-mon[37580]: pgmap v4972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:29.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:29.009+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:29.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:29.009+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:29.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:29.009+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:29 smithi107 ceph-mon[32659]: pgmap v4973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:29 smithi112 ceph-mon[37501]: pgmap v4973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:29.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:29 smithi132 ceph-mon[37580]: pgmap v4973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:31.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:31.225+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:31.225+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:31.227 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:31.225+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:31 smithi112 ceph-mon[37501]: pgmap v4974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8486.936795 secs 2024-07-26T15:12:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:31 smithi107 ceph-mon[32659]: pgmap v4974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8486.936795 secs 2024-07-26T15:12:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:31 smithi132 ceph-mon[37580]: pgmap v4974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8486.936795 secs 2024-07-26T15:12:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:33 smithi112 ceph-mon[37501]: pgmap v4975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:33 smithi107 ceph-mon[32659]: pgmap v4975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:33.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:33 smithi132 ceph-mon[37580]: pgmap v4975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:35 smithi112 ceph-mon[37501]: pgmap v4976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:35 smithi107 ceph-mon[32659]: pgmap v4976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:35.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:35 smithi132 ceph-mon[37580]: pgmap v4976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8491.936915 secs 2024-07-26T15:12:36.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8491.936915 secs 2024-07-26T15:12:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8491.936915 secs 2024-07-26T15:12:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:37 smithi112 ceph-mon[37501]: pgmap v4977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:37 smithi107 ceph-mon[32659]: pgmap v4977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:37 smithi132 ceph-mon[37580]: pgmap v4977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:39 smithi112 ceph-mon[37501]: pgmap v4978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:39 smithi107 ceph-mon[32659]: pgmap v4978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:39 smithi132 ceph-mon[37580]: pgmap v4978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:41 smithi112 ceph-mon[37501]: pgmap v4979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8496.937037 secs 2024-07-26T15:12:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:41 smithi107 ceph-mon[32659]: pgmap v4979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8496.937037 secs 2024-07-26T15:12:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:41 smithi132 ceph-mon[37580]: pgmap v4979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8496.937037 secs 2024-07-26T15:12:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:43 smithi112 ceph-mon[37501]: pgmap v4980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:43 smithi107 ceph-mon[32659]: pgmap v4980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:43 smithi132 ceph-mon[37580]: pgmap v4980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:44.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:44.025+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:44.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:44.025+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:44.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:44.025+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:44.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:12:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:12:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:12:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:45 smithi107 ceph-mon[32659]: pgmap v4981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:12:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:12:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:12:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:45 smithi132 ceph-mon[37580]: pgmap v4981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:12:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:12:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:12:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:45 smithi112 ceph-mon[37501]: pgmap v4981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:12:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:12:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:12:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:12:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:46.241+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:46.241+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:46.242 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:12:46.241+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:46 smithi107 ceph-mon[32659]: pgmap v4982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:12:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:46 smithi107 ceph-mon[32659]: pgmap v4983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8501.937091 secs 2024-07-26T15:12:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:46 smithi132 ceph-mon[37580]: pgmap v4982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:12:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:46 smithi132 ceph-mon[37580]: pgmap v4983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8501.937091 secs 2024-07-26T15:12:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:46 smithi112 ceph-mon[37501]: pgmap v4982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:12:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:46 smithi112 ceph-mon[37501]: pgmap v4983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8501.937091 secs 2024-07-26T15:12:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:48 smithi107 ceph-mon[32659]: pgmap v4984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:48.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:48 smithi132 ceph-mon[37580]: pgmap v4984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:48 smithi112 ceph-mon[37501]: pgmap v4984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:50 smithi107 ceph-mon[32659]: pgmap v4985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:50 smithi132 ceph-mon[37580]: pgmap v4985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:50 smithi112 ceph-mon[37501]: pgmap v4985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8506.937218 secs 2024-07-26T15:12:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8506.937218 secs 2024-07-26T15:12:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8506.937218 secs 2024-07-26T15:12:52.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:52 smithi107 ceph-mon[32659]: pgmap v4986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:12:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:52 smithi132 ceph-mon[37580]: pgmap v4986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:12:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:52 smithi112 ceph-mon[37501]: pgmap v4986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:12:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:54 smithi107 ceph-mon[32659]: pgmap v4987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:12:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:54 smithi132 ceph-mon[37580]: pgmap v4987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:12:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:54 smithi112 ceph-mon[37501]: pgmap v4987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:12:56.502 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:56 smithi112 ceph-mon[37501]: pgmap v4988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:12:56.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8511.937288 secs 2024-07-26T15:12:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:56 smithi107 ceph-mon[32659]: pgmap v4988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:12:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8511.937288 secs 2024-07-26T15:12:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:56 smithi132 ceph-mon[37580]: pgmap v4988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:12:56.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8511.937288 secs 2024-07-26T15:12:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:12:58 smithi107 ceph-mon[32659]: pgmap v4989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:58.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:12:58 smithi132 ceph-mon[37580]: pgmap v4989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:12:58 smithi112 ceph-mon[37501]: pgmap v4989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:12:59.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:59.029+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:12:59.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:59.029+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:59.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:12:59.029+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:00 smithi107 ceph-mon[32659]: pgmap v4990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:00.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:00 smithi132 ceph-mon[37580]: pgmap v4990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:00 smithi112 ceph-mon[37501]: pgmap v4990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:01.258 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:01.257+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:01.258 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:01.257+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:01.258 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:01.257+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8516.937371 secs 2024-07-26T15:13:01.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8516.937371 secs 2024-07-26T15:13:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8516.937371 secs 2024-07-26T15:13:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:02 smithi107 ceph-mon[32659]: pgmap v4991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:02.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:02 smithi132 ceph-mon[37580]: pgmap v4991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:02 smithi112 ceph-mon[37501]: pgmap v4991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:04 smithi107 ceph-mon[32659]: pgmap v4992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:04.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:04 smithi132 ceph-mon[37580]: pgmap v4992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:04 smithi112 ceph-mon[37501]: pgmap v4992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:06 smithi107 ceph-mon[32659]: pgmap v4993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8521.937492 secs 2024-07-26T15:13:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:06 smithi132 ceph-mon[37580]: pgmap v4993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8521.937492 secs 2024-07-26T15:13:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:06 smithi112 ceph-mon[37501]: pgmap v4993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8521.937492 secs 2024-07-26T15:13:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:08 smithi107 ceph-mon[32659]: pgmap v4994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:08.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:08 smithi132 ceph-mon[37580]: pgmap v4994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:08 smithi112 ceph-mon[37501]: pgmap v4994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:10.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:10 smithi132 ceph-mon[37580]: pgmap v4995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:10 smithi112 ceph-mon[37501]: pgmap v4995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:10 smithi107 ceph-mon[32659]: pgmap v4995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:11.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8526.937610 secs 2024-07-26T15:13:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8526.937610 secs 2024-07-26T15:13:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8526.937610 secs 2024-07-26T15:13:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:12 smithi107 ceph-mon[32659]: pgmap v4996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:12 smithi112 ceph-mon[37501]: pgmap v4996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:12 smithi132 ceph-mon[37580]: pgmap v4996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:14.033+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:14.033+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:14.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:14.033+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:14 smithi112 ceph-mon[37501]: pgmap v4997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:14 smithi107 ceph-mon[32659]: pgmap v4997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:14.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:14 smithi132 ceph-mon[37580]: pgmap v4997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:16.269+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:16.269+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:16.270 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:16.269+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:16 smithi112 ceph-mon[37501]: pgmap v4998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8531.937730 secs 2024-07-26T15:13:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:16 smithi107 ceph-mon[32659]: pgmap v4998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8531.937730 secs 2024-07-26T15:13:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:16 smithi132 ceph-mon[37580]: pgmap v4998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8531.937730 secs 2024-07-26T15:13:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:18 smithi112 ceph-mon[37501]: pgmap v4999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:18 smithi107 ceph-mon[32659]: pgmap v4999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:18.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:18 smithi132 ceph-mon[37580]: pgmap v4999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:20.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:20 smithi107 ceph-mon[32659]: pgmap v5000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:20 smithi112 ceph-mon[37501]: pgmap v5000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:20.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:20 smithi132 ceph-mon[37580]: pgmap v5000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8536.937853 secs 2024-07-26T15:13:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8536.937853 secs 2024-07-26T15:13:21.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8536.937853 secs 2024-07-26T15:13:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:22 smithi112 ceph-mon[37501]: pgmap v5001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:22 smithi107 ceph-mon[32659]: pgmap v5001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:22.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:22 smithi132 ceph-mon[37580]: pgmap v5001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:24 smithi112 ceph-mon[37501]: pgmap v5002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:24 smithi107 ceph-mon[32659]: pgmap v5002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:24 smithi132 ceph-mon[37580]: pgmap v5002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:26 smithi112 ceph-mon[37501]: pgmap v5003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8541.937927 secs 2024-07-26T15:13:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:26 smithi107 ceph-mon[32659]: pgmap v5003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8541.937927 secs 2024-07-26T15:13:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:26 smithi132 ceph-mon[37580]: pgmap v5003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:26.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8541.937927 secs 2024-07-26T15:13:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:28 smithi112 ceph-mon[37501]: pgmap v5004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:28 smithi107 ceph-mon[32659]: pgmap v5004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:28 smithi132 ceph-mon[37580]: pgmap v5004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:29.047+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:29.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:29.047+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:29.050 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:29.047+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:30 smithi112 ceph-mon[37501]: pgmap v5005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:30 smithi107 ceph-mon[32659]: pgmap v5005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:30.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:30 smithi132 ceph-mon[37580]: pgmap v5005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:31.285 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:31.284+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:31.284+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:31.286 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:31.284+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8546.938046 secs 2024-07-26T15:13:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8546.938046 secs 2024-07-26T15:13:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8546.938046 secs 2024-07-26T15:13:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:32 smithi112 ceph-mon[37501]: pgmap v5006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:32 smithi107 ceph-mon[32659]: pgmap v5006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:32.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:32 smithi132 ceph-mon[37580]: pgmap v5006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:34 smithi112 ceph-mon[37501]: pgmap v5007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:34 smithi107 ceph-mon[32659]: pgmap v5007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:34.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:34 smithi132 ceph-mon[37580]: pgmap v5007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:36 smithi112 ceph-mon[37501]: pgmap v5008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8551.938161 secs 2024-07-26T15:13:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:36 smithi107 ceph-mon[32659]: pgmap v5008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8551.938161 secs 2024-07-26T15:13:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:36 smithi132 ceph-mon[37580]: pgmap v5008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8551.938161 secs 2024-07-26T15:13:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:38 smithi112 ceph-mon[37501]: pgmap v5009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:38 smithi107 ceph-mon[32659]: pgmap v5009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:38 smithi132 ceph-mon[37580]: pgmap v5009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:40 smithi112 ceph-mon[37501]: pgmap v5010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:40 smithi107 ceph-mon[32659]: pgmap v5010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:40 smithi132 ceph-mon[37580]: pgmap v5010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8556.938238 secs 2024-07-26T15:13:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8556.938238 secs 2024-07-26T15:13:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8556.938238 secs 2024-07-26T15:13:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:42 smithi112 ceph-mon[37501]: pgmap v5011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:42 smithi107 ceph-mon[32659]: pgmap v5011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:42.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:42 smithi132 ceph-mon[37580]: pgmap v5011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:44.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:44.063+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:44.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:44.063+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:44.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:44.063+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:44 smithi112 ceph-mon[37501]: pgmap v5012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:44 smithi107 ceph-mon[32659]: pgmap v5012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:44 smithi132 ceph-mon[37580]: pgmap v5012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8561.938305 secs 2024-07-26T15:13:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:13:45.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8561.938305 secs 2024-07-26T15:13:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:13:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8561.938305 secs 2024-07-26T15:13:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:13:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:46.291+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:46.291+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:13:46.291+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:46 smithi112 ceph-mon[37501]: pgmap v5013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:13:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:13:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:13:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:46 smithi107 ceph-mon[32659]: pgmap v5013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:13:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:13:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:13:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:46 smithi132 ceph-mon[37580]: pgmap v5013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:13:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:13:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:13:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:47 smithi112 ceph-mon[37501]: pgmap v5014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:47 smithi112 ceph-mon[37501]: pgmap v5015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:47 smithi107 ceph-mon[32659]: pgmap v5014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:47 smithi107 ceph-mon[32659]: pgmap v5015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:47 smithi132 ceph-mon[37580]: pgmap v5014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:47 smithi132 ceph-mon[37580]: pgmap v5015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:49 smithi112 ceph-mon[37501]: pgmap v5016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:49 smithi107 ceph-mon[32659]: pgmap v5016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:49 smithi132 ceph-mon[37580]: pgmap v5016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8566.938381 secs 2024-07-26T15:13:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8566.938381 secs 2024-07-26T15:13:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8566.938381 secs 2024-07-26T15:13:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:51 smithi107 ceph-mon[32659]: pgmap v5017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:51 smithi132 ceph-mon[37580]: pgmap v5017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:51 smithi112 ceph-mon[37501]: pgmap v5017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:53 smithi107 ceph-mon[32659]: pgmap v5018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:53 smithi132 ceph-mon[37580]: pgmap v5018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:53 smithi112 ceph-mon[37501]: pgmap v5018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:55 smithi107 ceph-mon[32659]: pgmap v5019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8571.938520 secs 2024-07-26T15:13:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:55 smithi132 ceph-mon[37580]: pgmap v5019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8571.938520 secs 2024-07-26T15:13:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:55 smithi112 ceph-mon[37501]: pgmap v5019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:13:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8571.938520 secs 2024-07-26T15:13:57.752 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:57 smithi112 ceph-mon[37501]: pgmap v5020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:57 smithi107 ceph-mon[32659]: pgmap v5020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:57 smithi132 ceph-mon[37580]: pgmap v5020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:59.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:59.079+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:59.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:59.080+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:13:59.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:13:59.080+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:13:59 smithi107 ceph-mon[32659]: pgmap v5021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:13:59 smithi132 ceph-mon[37580]: pgmap v5021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:13:59 smithi112 ceph-mon[37501]: pgmap v5021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8576.938663 secs 2024-07-26T15:14:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8576.938663 secs 2024-07-26T15:14:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8576.938663 secs 2024-07-26T15:14:01.301 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:01.301+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:01.301+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:01.302 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:01.301+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:01 smithi107 ceph-mon[32659]: pgmap v5022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:01 smithi132 ceph-mon[37580]: pgmap v5022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:01 smithi112 ceph-mon[37501]: pgmap v5022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:03 smithi107 ceph-mon[32659]: pgmap v5023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:03.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:03 smithi132 ceph-mon[37580]: pgmap v5023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:03 smithi112 ceph-mon[37501]: pgmap v5023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:05 smithi107 ceph-mon[32659]: pgmap v5024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8581.938808 secs 2024-07-26T15:14:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:05 smithi132 ceph-mon[37580]: pgmap v5024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8581.938808 secs 2024-07-26T15:14:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:05 smithi112 ceph-mon[37501]: pgmap v5024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8581.938808 secs 2024-07-26T15:14:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:07 smithi107 ceph-mon[32659]: pgmap v5025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:14:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:07 smithi132 ceph-mon[37580]: pgmap v5025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:14:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:07 smithi112 ceph-mon[37501]: pgmap v5025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:14:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:09 smithi107 ceph-mon[32659]: pgmap v5026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:14:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:09 smithi132 ceph-mon[37580]: pgmap v5026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:14:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:09 smithi112 ceph-mon[37501]: pgmap v5026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:14:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8586.938951 secs 2024-07-26T15:14:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8586.938951 secs 2024-07-26T15:14:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8586.938951 secs 2024-07-26T15:14:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:11 smithi107 ceph-mon[32659]: pgmap v5027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:14:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:11 smithi132 ceph-mon[37580]: pgmap v5027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:14:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:11 smithi112 ceph-mon[37501]: pgmap v5027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:14:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:13 smithi107 ceph-mon[32659]: pgmap v5028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:14:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:13 smithi132 ceph-mon[37580]: pgmap v5028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:14:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:13 smithi112 ceph-mon[37501]: pgmap v5028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:14:14.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:14.086+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:14.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:14.086+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:14.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:14.086+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:15 smithi132 ceph-mon[37580]: pgmap v5029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8591.939091 secs 2024-07-26T15:14:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:15 smithi112 ceph-mon[37501]: pgmap v5029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8591.939091 secs 2024-07-26T15:14:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:15 smithi107 ceph-mon[32659]: pgmap v5029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8591.939091 secs 2024-07-26T15:14:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:16.316+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:16.317 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:16.316+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:16.318 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:16.316+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:17 smithi112 ceph-mon[37501]: pgmap v5030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:17 smithi107 ceph-mon[32659]: pgmap v5030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:17 smithi132 ceph-mon[37580]: pgmap v5030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:19 smithi112 ceph-mon[37501]: pgmap v5031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:19 smithi107 ceph-mon[32659]: pgmap v5031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:19 smithi132 ceph-mon[37580]: pgmap v5031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:20.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8596.939203 secs 2024-07-26T15:14:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8596.939203 secs 2024-07-26T15:14:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8596.939203 secs 2024-07-26T15:14:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:21 smithi112 ceph-mon[37501]: pgmap v5032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:21 smithi107 ceph-mon[32659]: pgmap v5032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:21 smithi132 ceph-mon[37580]: pgmap v5032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:23 smithi112 ceph-mon[37501]: pgmap v5033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:23 smithi107 ceph-mon[32659]: pgmap v5033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:23 smithi132 ceph-mon[37580]: pgmap v5033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:25 smithi112 ceph-mon[37501]: pgmap v5034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8601.939324 secs 2024-07-26T15:14:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:25 smithi107 ceph-mon[32659]: pgmap v5034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8601.939324 secs 2024-07-26T15:14:26.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:25 smithi132 ceph-mon[37580]: pgmap v5034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:26.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8601.939324 secs 2024-07-26T15:14:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:27 smithi112 ceph-mon[37501]: pgmap v5035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:27 smithi107 ceph-mon[32659]: pgmap v5035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:27 smithi132 ceph-mon[37580]: pgmap v5035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:29.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:29.101+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:29.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:29.101+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:29.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:29.101+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:29 smithi112 ceph-mon[37501]: pgmap v5036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:29.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:29 smithi107 ceph-mon[32659]: pgmap v5036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:29 smithi132 ceph-mon[37580]: pgmap v5036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8606.939408 secs 2024-07-26T15:14:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8606.939408 secs 2024-07-26T15:14:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8606.939408 secs 2024-07-26T15:14:31.330 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:31.326+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:31.330 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:31.326+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:31.331 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:31.326+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:31 smithi112 ceph-mon[37501]: pgmap v5037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:31 smithi107 ceph-mon[32659]: pgmap v5037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:31 smithi132 ceph-mon[37580]: pgmap v5037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:33 smithi112 ceph-mon[37501]: pgmap v5038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:33 smithi107 ceph-mon[32659]: pgmap v5038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:33 smithi132 ceph-mon[37580]: pgmap v5038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:35 smithi112 ceph-mon[37501]: pgmap v5039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8611.939539 secs 2024-07-26T15:14:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:35 smithi107 ceph-mon[32659]: pgmap v5039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8611.939539 secs 2024-07-26T15:14:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:35 smithi132 ceph-mon[37580]: pgmap v5039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8611.939539 secs 2024-07-26T15:14:37.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:37 smithi132 ceph-mon[37580]: pgmap v5040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:37 smithi112 ceph-mon[37501]: pgmap v5040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:37 smithi107 ceph-mon[32659]: pgmap v5040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:39 smithi112 ceph-mon[37501]: pgmap v5041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:39 smithi107 ceph-mon[32659]: pgmap v5041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:39 smithi132 ceph-mon[37580]: pgmap v5041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8616.939661 secs 2024-07-26T15:14:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8616.939661 secs 2024-07-26T15:14:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8616.939661 secs 2024-07-26T15:14:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:41 smithi107 ceph-mon[32659]: pgmap v5042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:41 smithi132 ceph-mon[37580]: pgmap v5042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:41 smithi112 ceph-mon[37501]: pgmap v5042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:43 smithi107 ceph-mon[32659]: pgmap v5043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:44.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:43 smithi132 ceph-mon[37580]: pgmap v5043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:44.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:44.116+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:44.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:44.116+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:44.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:44.116+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:43 smithi112 ceph-mon[37501]: pgmap v5043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:45 smithi107 ceph-mon[32659]: pgmap v5044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8621.939757 secs 2024-07-26T15:14:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:45 smithi132 ceph-mon[37580]: pgmap v5044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8621.939757 secs 2024-07-26T15:14:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:45 smithi112 ceph-mon[37501]: pgmap v5044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:14:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8621.939757 secs 2024-07-26T15:14:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:46.330+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:46.330+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:46.332 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:14:46.330+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:47.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:14:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:14:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:14:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:47 smithi107 ceph-mon[32659]: pgmap v5045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:14:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:14:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:14:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:14:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:47 smithi132 ceph-mon[37580]: pgmap v5045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:14:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:14:48.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:14:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:14:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:47 smithi112 ceph-mon[37501]: pgmap v5045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:14:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:14:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:14:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:14:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:48 smithi107 ceph-mon[32659]: pgmap v5046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:14:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:48 smithi132 ceph-mon[37580]: pgmap v5046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:14:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:48 smithi112 ceph-mon[37501]: pgmap v5046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:14:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:49 smithi107 ceph-mon[32659]: pgmap v5047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:14:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:49 smithi132 ceph-mon[37580]: pgmap v5047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:14:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:49 smithi112 ceph-mon[37501]: pgmap v5047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:14:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8626.939912 secs 2024-07-26T15:14:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8626.939912 secs 2024-07-26T15:14:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8626.939912 secs 2024-07-26T15:14:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:51 smithi107 ceph-mon[32659]: pgmap v5048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:14:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:51 smithi132 ceph-mon[37580]: pgmap v5048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:14:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:51 smithi112 ceph-mon[37501]: pgmap v5048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:14:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:54 smithi112 ceph-mon[37501]: pgmap v5049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:14:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:54 smithi107 ceph-mon[32659]: pgmap v5049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:14:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:54 smithi132 ceph-mon[37580]: pgmap v5049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:14:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8631.940033 secs 2024-07-26T15:14:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8631.940033 secs 2024-07-26T15:14:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8631.940033 secs 2024-07-26T15:14:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:56 smithi112 ceph-mon[37501]: pgmap v5050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:14:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:56 smithi107 ceph-mon[32659]: pgmap v5050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:14:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:56 smithi132 ceph-mon[37580]: pgmap v5050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:14:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:14:58 smithi112 ceph-mon[37501]: pgmap v5051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:14:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:14:58 smithi107 ceph-mon[32659]: pgmap v5051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:14:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:14:58 smithi132 ceph-mon[37580]: pgmap v5051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:14:59.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:59.130+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:14:59.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:59.130+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:59.131 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:14:59.130+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:00 smithi112 ceph-mon[37501]: pgmap v5052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8636.940154 secs 2024-07-26T15:15:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:00 smithi107 ceph-mon[32659]: pgmap v5052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8636.940154 secs 2024-07-26T15:15:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:00 smithi132 ceph-mon[37580]: pgmap v5052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8636.940154 secs 2024-07-26T15:15:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:01.347+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:01.347+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:01.347+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:02 smithi112 ceph-mon[37501]: pgmap v5053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:02 smithi107 ceph-mon[32659]: pgmap v5053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:03.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:02 smithi132 ceph-mon[37580]: pgmap v5053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:04 smithi112 ceph-mon[37501]: pgmap v5054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:04 smithi107 ceph-mon[32659]: pgmap v5054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:04 smithi132 ceph-mon[37580]: pgmap v5054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8641.940282 secs 2024-07-26T15:15:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8641.940282 secs 2024-07-26T15:15:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8641.940282 secs 2024-07-26T15:15:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:06 smithi112 ceph-mon[37501]: pgmap v5055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:06 smithi107 ceph-mon[32659]: pgmap v5055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:06 smithi132 ceph-mon[37580]: pgmap v5055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:08 smithi112 ceph-mon[37501]: pgmap v5056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:08 smithi107 ceph-mon[32659]: pgmap v5056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:08 smithi132 ceph-mon[37580]: pgmap v5056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:10 smithi112 ceph-mon[37501]: pgmap v5057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8646.940378 secs 2024-07-26T15:15:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:10 smithi107 ceph-mon[32659]: pgmap v5057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8646.940378 secs 2024-07-26T15:15:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:10 smithi132 ceph-mon[37580]: pgmap v5057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8646.940378 secs 2024-07-26T15:15:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:12 smithi112 ceph-mon[37501]: pgmap v5058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:12 smithi107 ceph-mon[32659]: pgmap v5058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:12 smithi132 ceph-mon[37580]: pgmap v5058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:14.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:14.101+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:15:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:14.102+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:15:14.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:14.106+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:15:14.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:14.107+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:15:14.146 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:14.146+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:14.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:14.146+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:14.147 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:14.146+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:14 smithi112 ceph-mon[37501]: pgmap v5059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:14 smithi107 ceph-mon[32659]: pgmap v5059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:14 smithi132 ceph-mon[37580]: pgmap v5059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8651.940448 secs 2024-07-26T15:15:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8651.940448 secs 2024-07-26T15:15:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8651.940448 secs 2024-07-26T15:15:16.290 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:16.289+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:15:16.290 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:16.289+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:15:16.290 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:16.290+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:15:16.291 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:16.290+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:15:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:16.362+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:16.362+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:16.362+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:16.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:16 smithi112 ceph-mon[37501]: pgmap v5060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:16 smithi107 ceph-mon[32659]: pgmap v5060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:16 smithi132 ceph-mon[37580]: pgmap v5060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:18 smithi107 ceph-mon[32659]: pgmap v5061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:18 smithi132 ceph-mon[37580]: pgmap v5061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:19.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:18 smithi112 ceph-mon[37501]: pgmap v5061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:20 smithi107 ceph-mon[32659]: pgmap v5062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8656.940568 secs 2024-07-26T15:15:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:20 smithi132 ceph-mon[37580]: pgmap v5062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8656.940568 secs 2024-07-26T15:15:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:20 smithi112 ceph-mon[37501]: pgmap v5062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8656.940568 secs 2024-07-26T15:15:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:22 smithi107 ceph-mon[32659]: pgmap v5063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:22 smithi132 ceph-mon[37580]: pgmap v5063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:22 smithi112 ceph-mon[37501]: pgmap v5063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:24 smithi107 ceph-mon[32659]: pgmap v5064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:24 smithi132 ceph-mon[37580]: pgmap v5064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:24 smithi112 ceph-mon[37501]: pgmap v5064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8661.940707 secs 2024-07-26T15:15:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8661.940707 secs 2024-07-26T15:15:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8661.940707 secs 2024-07-26T15:15:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:26 smithi107 ceph-mon[32659]: pgmap v5065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:26 smithi132 ceph-mon[37580]: pgmap v5065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:26 smithi112 ceph-mon[37501]: pgmap v5065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:29.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:28 smithi112 ceph-mon[37501]: pgmap v5066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:28 smithi107 ceph-mon[32659]: pgmap v5066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:28 smithi132 ceph-mon[37580]: pgmap v5066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:29.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:29.150+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:29.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:29.150+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:29.152 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:29.150+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:30 smithi107 ceph-mon[32659]: pgmap v5067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8666.940799 secs 2024-07-26T15:15:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:30 smithi132 ceph-mon[37580]: pgmap v5067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8666.940799 secs 2024-07-26T15:15:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:30 smithi112 ceph-mon[37501]: pgmap v5067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8666.940799 secs 2024-07-26T15:15:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:31.377+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:31.377+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:31.379 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:31.377+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:31 smithi107 ceph-mon[32659]: pgmap v5068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:31 smithi132 ceph-mon[37580]: pgmap v5068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:31 smithi112 ceph-mon[37501]: pgmap v5068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:34 smithi112 ceph-mon[37501]: pgmap v5069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:34 smithi107 ceph-mon[32659]: pgmap v5069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:34 smithi132 ceph-mon[37580]: pgmap v5069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8671.940951 secs 2024-07-26T15:15:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8671.940951 secs 2024-07-26T15:15:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8671.940951 secs 2024-07-26T15:15:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:36 smithi112 ceph-mon[37501]: pgmap v5070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:36 smithi107 ceph-mon[32659]: pgmap v5070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:36 smithi132 ceph-mon[37580]: pgmap v5070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:38 smithi112 ceph-mon[37501]: pgmap v5071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:38 smithi107 ceph-mon[32659]: pgmap v5071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:39.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:38 smithi132 ceph-mon[37580]: pgmap v5071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:40 smithi112 ceph-mon[37501]: pgmap v5072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8676.941078 secs 2024-07-26T15:15:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:40 smithi107 ceph-mon[32659]: pgmap v5072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8676.941078 secs 2024-07-26T15:15:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:40 smithi132 ceph-mon[37580]: pgmap v5072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8676.941078 secs 2024-07-26T15:15:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:42 smithi112 ceph-mon[37501]: pgmap v5073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:42 smithi107 ceph-mon[32659]: pgmap v5073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:42 smithi132 ceph-mon[37580]: pgmap v5073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:44.153+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:44.153+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:44.153+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:44 smithi112 ceph-mon[37501]: pgmap v5074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:44 smithi107 ceph-mon[32659]: pgmap v5074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:44 smithi132 ceph-mon[37580]: pgmap v5074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8681.941194 secs 2024-07-26T15:15:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8681.941194 secs 2024-07-26T15:15:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8681.941194 secs 2024-07-26T15:15:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:46.393+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:46.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:46.393+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:46.395 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:15:46.393+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:46 smithi112 ceph-mon[37501]: pgmap v5075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:46 smithi107 ceph-mon[32659]: pgmap v5075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:46 smithi132 ceph-mon[37580]: pgmap v5075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:48.911 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:48 smithi112 ceph-mon[37501]: pgmap v5076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:48.912 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:48 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:15:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:48 smithi107 ceph-mon[32659]: pgmap v5076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:48 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:15:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:48 smithi132 ceph-mon[37580]: pgmap v5076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:48 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:15:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:50 smithi112 ceph-mon[37501]: pgmap v5077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8686.941321 secs 2024-07-26T15:15:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:50 smithi107 ceph-mon[32659]: pgmap v5077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8686.941321 secs 2024-07-26T15:15:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:50 smithi132 ceph-mon[37580]: pgmap v5077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:15:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8686.941321 secs 2024-07-26T15:15:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:52 smithi112 ceph-mon[37501]: pgmap v5078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:52 smithi107 ceph-mon[32659]: pgmap v5078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:52 smithi132 ceph-mon[37580]: pgmap v5078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:15:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: pgmap v5079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:15:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: pgmap v5079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:15:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:15:55.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: pgmap v5079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:15:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:55 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:15:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:55 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:15:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:55 smithi112 ceph-mon[37501]: pgmap v5080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:15:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:55 smithi112 ceph-mon[37501]: pgmap v5081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:15:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8691.941445 secs 2024-07-26T15:15:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:55 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:15:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:55 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:15:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:55 smithi107 ceph-mon[32659]: pgmap v5080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:15:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:55 smithi107 ceph-mon[32659]: pgmap v5081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:15:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8691.941445 secs 2024-07-26T15:15:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:55 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:15:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:55 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:15:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:55 smithi132 ceph-mon[37580]: pgmap v5080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:15:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:55 smithi132 ceph-mon[37580]: pgmap v5081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:15:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8691.941445 secs 2024-07-26T15:15:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:57 smithi112 ceph-mon[37501]: pgmap v5082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T15:15:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:57 smithi107 ceph-mon[32659]: pgmap v5082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T15:15:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:57 smithi132 ceph-mon[37580]: pgmap v5082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T15:15:59.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:59.170+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:59.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:59.170+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:15:59.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:15:59.170+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:15:59 smithi112 ceph-mon[37501]: pgmap v5083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:16:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:15:59 smithi107 ceph-mon[32659]: pgmap v5083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:16:00.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:15:59 smithi132 ceph-mon[37580]: pgmap v5083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:16:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8696.941509 secs 2024-07-26T15:16:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8696.941509 secs 2024-07-26T15:16:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8696.941509 secs 2024-07-26T15:16:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:01.403+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:01.403+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:01.403+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:01 smithi112 ceph-mon[37501]: pgmap v5084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:16:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:01 smithi107 ceph-mon[32659]: pgmap v5084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:16:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:01 smithi132 ceph-mon[37580]: pgmap v5084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:16:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:03 smithi112 ceph-mon[37501]: pgmap v5085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T15:16:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:03 smithi107 ceph-mon[32659]: pgmap v5085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T15:16:04.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:03 smithi132 ceph-mon[37580]: pgmap v5085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T15:16:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:05 smithi112 ceph-mon[37501]: pgmap v5086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:16:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8701.941608 secs 2024-07-26T15:16:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:05 smithi107 ceph-mon[32659]: pgmap v5086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:16:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8701.941608 secs 2024-07-26T15:16:06.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:05 smithi132 ceph-mon[37580]: pgmap v5086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:16:06.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8701.941608 secs 2024-07-26T15:16:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:07 smithi112 ceph-mon[37501]: pgmap v5087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:07 smithi107 ceph-mon[32659]: pgmap v5087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:07 smithi132 ceph-mon[37580]: pgmap v5087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:09 smithi112 ceph-mon[37501]: pgmap v5088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:09 smithi107 ceph-mon[32659]: pgmap v5088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:10.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:09 smithi132 ceph-mon[37580]: pgmap v5088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8706.941747 secs 2024-07-26T15:16:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8706.941747 secs 2024-07-26T15:16:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8706.941747 secs 2024-07-26T15:16:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:11 smithi112 ceph-mon[37501]: pgmap v5089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:11 smithi107 ceph-mon[32659]: pgmap v5089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:11 smithi132 ceph-mon[37580]: pgmap v5089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:13 smithi112 ceph-mon[37501]: pgmap v5090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:13 smithi107 ceph-mon[32659]: pgmap v5090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:13 smithi132 ceph-mon[37580]: pgmap v5090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:14.186 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:14.185+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:14.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:14.185+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:14.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:14.185+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:15 smithi112 ceph-mon[37501]: pgmap v5091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8711.941895 secs 2024-07-26T15:16:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:15 smithi107 ceph-mon[32659]: pgmap v5091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8711.941895 secs 2024-07-26T15:16:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:15 smithi132 ceph-mon[37580]: pgmap v5091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8711.941895 secs 2024-07-26T15:16:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:16.410+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:16.410+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:16.410+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:17 smithi112 ceph-mon[37501]: pgmap v5092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:17 smithi107 ceph-mon[32659]: pgmap v5092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:18.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:17 smithi132 ceph-mon[37580]: pgmap v5092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:19 smithi112 ceph-mon[37501]: pgmap v5093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:19.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:19 smithi107 ceph-mon[32659]: pgmap v5093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:19 smithi132 ceph-mon[37580]: pgmap v5093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8716.942023 secs 2024-07-26T15:16:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8716.942023 secs 2024-07-26T15:16:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8716.942023 secs 2024-07-26T15:16:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:21 smithi107 ceph-mon[32659]: pgmap v5094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:21 smithi132 ceph-mon[37580]: pgmap v5094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:21 smithi112 ceph-mon[37501]: pgmap v5094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:23 smithi107 ceph-mon[32659]: pgmap v5095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:23 smithi132 ceph-mon[37580]: pgmap v5095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:23 smithi112 ceph-mon[37501]: pgmap v5095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:25 smithi107 ceph-mon[32659]: pgmap v5096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8721.942158 secs 2024-07-26T15:16:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:25 smithi132 ceph-mon[37580]: pgmap v5096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8721.942158 secs 2024-07-26T15:16:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:25 smithi112 ceph-mon[37501]: pgmap v5096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8721.942158 secs 2024-07-26T15:16:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:27 smithi107 ceph-mon[32659]: pgmap v5097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:27 smithi132 ceph-mon[37580]: pgmap v5097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:27 smithi112 ceph-mon[37501]: pgmap v5097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:29.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:29.200+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:29.200+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:29.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:29.200+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:30.002 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:29 smithi112 ceph-mon[37501]: pgmap v5098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:29 smithi107 ceph-mon[32659]: pgmap v5098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:29 smithi132 ceph-mon[37580]: pgmap v5098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8726.942251 secs 2024-07-26T15:16:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8726.942251 secs 2024-07-26T15:16:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8726.942251 secs 2024-07-26T15:16:31.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:31.425+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:31.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:31.425+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:31.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:31.425+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:31 smithi107 ceph-mon[32659]: pgmap v5099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:31 smithi132 ceph-mon[37580]: pgmap v5099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:31 smithi112 ceph-mon[37501]: pgmap v5099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:33 smithi107 ceph-mon[32659]: pgmap v5100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:33 smithi132 ceph-mon[37580]: pgmap v5100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:33 smithi112 ceph-mon[37501]: pgmap v5100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:35 smithi107 ceph-mon[32659]: pgmap v5101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8731.942391 secs 2024-07-26T15:16:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:35 smithi132 ceph-mon[37580]: pgmap v5101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8731.942391 secs 2024-07-26T15:16:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:35 smithi112 ceph-mon[37501]: pgmap v5101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8731.942391 secs 2024-07-26T15:16:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:37 smithi107 ceph-mon[32659]: pgmap v5102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:38.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:37 smithi132 ceph-mon[37580]: pgmap v5102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:37 smithi112 ceph-mon[37501]: pgmap v5102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:39 smithi107 ceph-mon[32659]: pgmap v5103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:40.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:39 smithi132 ceph-mon[37580]: pgmap v5103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:39 smithi112 ceph-mon[37501]: pgmap v5103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8736.942510 secs 2024-07-26T15:16:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8736.942510 secs 2024-07-26T15:16:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8736.942510 secs 2024-07-26T15:16:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:41 smithi107 ceph-mon[32659]: pgmap v5104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:42.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:41 smithi132 ceph-mon[37580]: pgmap v5104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:41 smithi112 ceph-mon[37501]: pgmap v5104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:42 smithi132 ceph-mon[37580]: pgmap v5105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:42 smithi112 ceph-mon[37501]: pgmap v5105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:42 smithi107 ceph-mon[32659]: pgmap v5105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:44.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:44.201+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:44.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:44.201+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:44.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:44.201+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:45 smithi112 ceph-mon[37501]: pgmap v5106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:45 smithi107 ceph-mon[32659]: pgmap v5106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:45 smithi132 ceph-mon[37580]: pgmap v5106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:46.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:46.441+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:46.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:46.441+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:46.442 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:16:46.441+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8741.942617 secs 2024-07-26T15:16:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8741.942617 secs 2024-07-26T15:16:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8741.942617 secs 2024-07-26T15:16:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:47 smithi112 ceph-mon[37501]: pgmap v5107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:47 smithi107 ceph-mon[32659]: pgmap v5107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:47.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:47 smithi132 ceph-mon[37580]: pgmap v5107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:49 smithi112 ceph-mon[37501]: pgmap v5108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:49 smithi107 ceph-mon[32659]: pgmap v5108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:49.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:49 smithi132 ceph-mon[37580]: pgmap v5108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:51 smithi112 ceph-mon[37501]: pgmap v5109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8746.942740 secs 2024-07-26T15:16:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:51 smithi107 ceph-mon[32659]: pgmap v5109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8746.942740 secs 2024-07-26T15:16:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:51 smithi132 ceph-mon[37580]: pgmap v5109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:51.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8746.942740 secs 2024-07-26T15:16:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:53 smithi112 ceph-mon[37501]: pgmap v5110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:53 smithi107 ceph-mon[32659]: pgmap v5110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:53.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:53 smithi132 ceph-mon[37580]: pgmap v5110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:16:55.510 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:55 smithi132 ceph-mon[37580]: pgmap v5111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:55.510 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:16:55.510 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.510 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.510 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.510 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.511 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.511 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.511 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.511 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:55 smithi107 ceph-mon[32659]: pgmap v5111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:16:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:55 smithi112 ceph-mon[37501]: pgmap v5111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:16:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:56.495 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:56 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:16:56.495 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:56 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:16:56.495 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8751.942865 secs 2024-07-26T15:16:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:56 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:16:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:56 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:16:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8751.942865 secs 2024-07-26T15:16:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:56 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:16:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:56 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:16:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8751.942865 secs 2024-07-26T15:16:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:57 smithi107 ceph-mon[32659]: pgmap v5112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:57 smithi132 ceph-mon[37580]: pgmap v5112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:57 smithi112 ceph-mon[37501]: pgmap v5112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:59.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:59.204+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:59.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:59.204+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:59.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:16:59.204+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: pgmap v5113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:16:59.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:16:59.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:16:59.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:16:59.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: pgmap v5114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:16:59.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: pgmap v5115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:16:59.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:16:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:16:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: pgmap v5113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:16:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:16:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:16:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:16:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:16:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:16:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: pgmap v5114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:16:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: pgmap v5115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:16:59.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:16:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: pgmap v5113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:16:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:16:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:16:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:16:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: pgmap v5114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:16:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: pgmap v5115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:16:59.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:16:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:17:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:01.457+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:01.458 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:01.457+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:01.459 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:01.457+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:01 smithi107 ceph-mon[32659]: pgmap v5116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:17:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8756.943002 secs 2024-07-26T15:17:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:01 smithi132 ceph-mon[37580]: pgmap v5116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:17:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8756.943002 secs 2024-07-26T15:17:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:01 smithi112 ceph-mon[37501]: pgmap v5116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:17:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8756.943002 secs 2024-07-26T15:17:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:03 smithi107 ceph-mon[32659]: pgmap v5117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:17:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:03 smithi132 ceph-mon[37580]: pgmap v5117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:17:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:03 smithi112 ceph-mon[37501]: pgmap v5117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:17:05.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:05 smithi132 ceph-mon[37580]: pgmap v5118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:17:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:05 smithi112 ceph-mon[37501]: pgmap v5118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:17:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:05 smithi107 ceph-mon[32659]: pgmap v5118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:17:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8761.943129 secs 2024-07-26T15:17:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8761.943129 secs 2024-07-26T15:17:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8761.943129 secs 2024-07-26T15:17:07.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:07 smithi132 ceph-mon[37580]: pgmap v5119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:17:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:07 smithi112 ceph-mon[37501]: pgmap v5119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:17:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:07 smithi107 ceph-mon[32659]: pgmap v5119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:17:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:09 smithi112 ceph-mon[37501]: pgmap v5120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:17:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:09 smithi107 ceph-mon[32659]: pgmap v5120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:17:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:09 smithi132 ceph-mon[37580]: pgmap v5120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:17:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:11 smithi112 ceph-mon[37501]: pgmap v5121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8766.943161 secs 2024-07-26T15:17:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:11 smithi107 ceph-mon[32659]: pgmap v5121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8766.943161 secs 2024-07-26T15:17:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:11 smithi132 ceph-mon[37580]: pgmap v5121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8766.943161 secs 2024-07-26T15:17:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:13 smithi112 ceph-mon[37501]: pgmap v5122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:13 smithi107 ceph-mon[32659]: pgmap v5122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:13 smithi132 ceph-mon[37580]: pgmap v5122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:14.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:14.213+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:14.214 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:14.213+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:14.215 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:14.213+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:15 smithi112 ceph-mon[37501]: pgmap v5123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:15 smithi107 ceph-mon[32659]: pgmap v5123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:15 smithi132 ceph-mon[37580]: pgmap v5123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:16.466+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:16.466+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:16.466+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8771.943283 secs 2024-07-26T15:17:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8771.943283 secs 2024-07-26T15:17:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8771.943283 secs 2024-07-26T15:17:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:17 smithi107 ceph-mon[32659]: pgmap v5124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:17 smithi132 ceph-mon[37580]: pgmap v5124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:17 smithi112 ceph-mon[37501]: pgmap v5124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:19 smithi107 ceph-mon[32659]: pgmap v5125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:20.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:19 smithi132 ceph-mon[37580]: pgmap v5125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:19 smithi112 ceph-mon[37501]: pgmap v5125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8776.943402 secs 2024-07-26T15:17:21.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8776.943402 secs 2024-07-26T15:17:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8776.943402 secs 2024-07-26T15:17:22.009 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:21 smithi107 ceph-mon[32659]: pgmap v5126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:21 smithi132 ceph-mon[37580]: pgmap v5126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:21 smithi112 ceph-mon[37501]: pgmap v5126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:23 smithi107 ceph-mon[32659]: pgmap v5127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:23 smithi132 ceph-mon[37580]: pgmap v5127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:23 smithi112 ceph-mon[37501]: pgmap v5127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:25 smithi107 ceph-mon[32659]: pgmap v5128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8781.943468 secs 2024-07-26T15:17:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:25 smithi132 ceph-mon[37580]: pgmap v5128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8781.943468 secs 2024-07-26T15:17:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:25 smithi112 ceph-mon[37501]: pgmap v5128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8781.943468 secs 2024-07-26T15:17:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:27 smithi107 ceph-mon[32659]: pgmap v5129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:28.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:27 smithi132 ceph-mon[37580]: pgmap v5129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:27 smithi112 ceph-mon[37501]: pgmap v5129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:29.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:29.228+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:29.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:29.228+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:29.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:29.228+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:29 smithi107 ceph-mon[32659]: pgmap v5130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:29 smithi132 ceph-mon[37580]: pgmap v5130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:29 smithi112 ceph-mon[37501]: pgmap v5130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8786.943567 secs 2024-07-26T15:17:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8786.943567 secs 2024-07-26T15:17:31.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8786.943567 secs 2024-07-26T15:17:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:31.480+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:31.480+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:31.482 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:31.480+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:31 smithi107 ceph-mon[32659]: pgmap v5131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:31 smithi132 ceph-mon[37580]: pgmap v5131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:31 smithi112 ceph-mon[37501]: pgmap v5131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:33 smithi107 ceph-mon[32659]: pgmap v5132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:33 smithi132 ceph-mon[37580]: pgmap v5132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:33 smithi112 ceph-mon[37501]: pgmap v5132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:34 smithi112 ceph-mon[37501]: pgmap v5133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:34 smithi107 ceph-mon[32659]: pgmap v5133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:34 smithi132 ceph-mon[37580]: pgmap v5133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8791.943674 secs 2024-07-26T15:17:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8791.943674 secs 2024-07-26T15:17:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8791.943674 secs 2024-07-26T15:17:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:36 smithi112 ceph-mon[37501]: pgmap v5134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:36 smithi107 ceph-mon[32659]: pgmap v5134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:36 smithi132 ceph-mon[37580]: pgmap v5134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:39 smithi132 ceph-mon[37580]: pgmap v5135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:39 smithi112 ceph-mon[37501]: pgmap v5135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:39 smithi107 ceph-mon[32659]: pgmap v5135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:41 smithi112 ceph-mon[37501]: pgmap v5136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8796.943754 secs 2024-07-26T15:17:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:41 smithi107 ceph-mon[32659]: pgmap v5136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8796.943754 secs 2024-07-26T15:17:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:41 smithi132 ceph-mon[37580]: pgmap v5136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8796.943754 secs 2024-07-26T15:17:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:43 smithi112 ceph-mon[37501]: pgmap v5137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:43 smithi107 ceph-mon[32659]: pgmap v5137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:43.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:43 smithi132 ceph-mon[37580]: pgmap v5137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:44.244 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:44.243+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:44.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:44.243+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:44.245 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:44.243+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:45 smithi112 ceph-mon[37501]: pgmap v5138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:45 smithi107 ceph-mon[32659]: pgmap v5138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:45.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:45 smithi132 ceph-mon[37580]: pgmap v5138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:46.481+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:46.481+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:46.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:17:46.481+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8801.943872 secs 2024-07-26T15:17:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8801.943872 secs 2024-07-26T15:17:46.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8801.943872 secs 2024-07-26T15:17:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:47 smithi112 ceph-mon[37501]: pgmap v5139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:47 smithi107 ceph-mon[32659]: pgmap v5139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:47.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:47 smithi132 ceph-mon[37580]: pgmap v5139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:49 smithi112 ceph-mon[37501]: pgmap v5140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:49 smithi107 ceph-mon[32659]: pgmap v5140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:49.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:49 smithi132 ceph-mon[37580]: pgmap v5140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:51 smithi112 ceph-mon[37501]: pgmap v5141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8806.943990 secs 2024-07-26T15:17:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:51 smithi107 ceph-mon[32659]: pgmap v5141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8806.943990 secs 2024-07-26T15:17:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:51 smithi132 ceph-mon[37580]: pgmap v5141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8806.943990 secs 2024-07-26T15:17:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:53 smithi112 ceph-mon[37501]: pgmap v5142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:53.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:53 smithi107 ceph-mon[32659]: pgmap v5142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:53.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:53 smithi132 ceph-mon[37580]: pgmap v5142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:17:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:55 smithi112 ceph-mon[37501]: pgmap v5143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:55 smithi107 ceph-mon[32659]: pgmap v5143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:55 smithi132 ceph-mon[37580]: pgmap v5143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8811.944136 secs 2024-07-26T15:17:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8811.944136 secs 2024-07-26T15:17:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8811.944136 secs 2024-07-26T15:17:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:57 smithi112 ceph-mon[37501]: pgmap v5144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:57 smithi107 ceph-mon[32659]: pgmap v5144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:57.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:57 smithi132 ceph-mon[37580]: pgmap v5144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:59.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:59.248+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:59.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:59.248+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:17:59.250 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:17:59.248+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:59.541 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:59 smithi107 ceph-mon[32659]: pgmap v5145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:17:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:17:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:59 smithi112 ceph-mon[37501]: pgmap v5145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:17:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:17:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:59 smithi132 ceph-mon[37580]: pgmap v5145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:17:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:17:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:18:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:18:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:18:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:00 smithi112 ceph-mon[37501]: pgmap v5146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:18:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:00 smithi112 ceph-mon[37501]: pgmap v5147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:00.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:18:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:18:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:18:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:00 smithi107 ceph-mon[32659]: pgmap v5146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:18:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:00 smithi107 ceph-mon[32659]: pgmap v5147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:18:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:18:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:18:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:00 smithi132 ceph-mon[37580]: pgmap v5146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:18:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:00 smithi132 ceph-mon[37580]: pgmap v5147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:18:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:01.487+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:01.487+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:01.488 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:01.487+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8816.944246 secs 2024-07-26T15:18:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8816.944246 secs 2024-07-26T15:18:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8816.944246 secs 2024-07-26T15:18:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:02 smithi112 ceph-mon[37501]: pgmap v5148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:02 smithi107 ceph-mon[32659]: pgmap v5148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:02 smithi132 ceph-mon[37580]: pgmap v5148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:04 smithi107 ceph-mon[32659]: pgmap v5149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:04 smithi132 ceph-mon[37580]: pgmap v5149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:04 smithi112 ceph-mon[37501]: pgmap v5149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8821.944366 secs 2024-07-26T15:18:05.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8821.944366 secs 2024-07-26T15:18:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8821.944366 secs 2024-07-26T15:18:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:06 smithi107 ceph-mon[32659]: pgmap v5150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:06 smithi132 ceph-mon[37580]: pgmap v5150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:06 smithi112 ceph-mon[37501]: pgmap v5150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:08 smithi107 ceph-mon[32659]: pgmap v5151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:08 smithi132 ceph-mon[37580]: pgmap v5151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:08 smithi112 ceph-mon[37501]: pgmap v5151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:18:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:10 smithi107 ceph-mon[32659]: pgmap v5152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:18:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8826.944448 secs 2024-07-26T15:18:10.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:10 smithi132 ceph-mon[37580]: pgmap v5152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:18:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8826.944448 secs 2024-07-26T15:18:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:10 smithi112 ceph-mon[37501]: pgmap v5152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:18:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8826.944448 secs 2024-07-26T15:18:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:12 smithi107 ceph-mon[32659]: pgmap v5153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:12.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:12 smithi132 ceph-mon[37580]: pgmap v5153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:12 smithi112 ceph-mon[37501]: pgmap v5153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:14.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:14.132+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:18:14.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:14.133+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:18:14.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:14.135+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:18:14.137 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:14.136+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:18:14.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:14.264+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:14.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:14.264+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:14.265 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:14.264+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:14 smithi132 ceph-mon[37580]: pgmap v5154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:14 smithi112 ceph-mon[37501]: pgmap v5154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:14 smithi107 ceph-mon[32659]: pgmap v5154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8831.944569 secs 2024-07-26T15:18:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8831.944569 secs 2024-07-26T15:18:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8831.944569 secs 2024-07-26T15:18:16.321 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:16.321+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:18:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:16.321+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:18:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:16.322+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:18:16.322 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:16.322+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:18:16.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:16.503+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:16.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:16.503+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:16.504 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:16.503+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:16 smithi107 ceph-mon[32659]: pgmap v5155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:16 smithi132 ceph-mon[37580]: pgmap v5155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:16 smithi112 ceph-mon[37501]: pgmap v5155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:18 smithi107 ceph-mon[32659]: pgmap v5156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:18 smithi132 ceph-mon[37580]: pgmap v5156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:18 smithi112 ceph-mon[37501]: pgmap v5156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:20 smithi107 ceph-mon[32659]: pgmap v5157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8836.944696 secs 2024-07-26T15:18:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:20 smithi132 ceph-mon[37580]: pgmap v5157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8836.944696 secs 2024-07-26T15:18:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:20 smithi112 ceph-mon[37501]: pgmap v5157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8836.944696 secs 2024-07-26T15:18:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:22 smithi107 ceph-mon[32659]: pgmap v5158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:22 smithi132 ceph-mon[37580]: pgmap v5158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:22 smithi112 ceph-mon[37501]: pgmap v5158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:24 smithi107 ceph-mon[32659]: pgmap v5159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:24 smithi132 ceph-mon[37580]: pgmap v5159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:24 smithi112 ceph-mon[37501]: pgmap v5159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8841.944834 secs 2024-07-26T15:18:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8841.944834 secs 2024-07-26T15:18:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8841.944834 secs 2024-07-26T15:18:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:26 smithi107 ceph-mon[32659]: pgmap v5160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:26 smithi132 ceph-mon[37580]: pgmap v5160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:26 smithi112 ceph-mon[37501]: pgmap v5160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:28 smithi107 ceph-mon[32659]: pgmap v5161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:28 smithi132 ceph-mon[37580]: pgmap v5161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:28 smithi112 ceph-mon[37501]: pgmap v5161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:29.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:29.278+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:29.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:29.278+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:29.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:29.278+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:30 smithi107 ceph-mon[32659]: pgmap v5162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8846.944966 secs 2024-07-26T15:18:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:30 smithi132 ceph-mon[37580]: pgmap v5162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8846.944966 secs 2024-07-26T15:18:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:30 smithi112 ceph-mon[37501]: pgmap v5162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8846.944966 secs 2024-07-26T15:18:31.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:31.504+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:31.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:31.504+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:31.506 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:31.504+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:32 smithi107 ceph-mon[32659]: pgmap v5163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:32 smithi132 ceph-mon[37580]: pgmap v5163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:32 smithi112 ceph-mon[37501]: pgmap v5163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:33 smithi132 ceph-mon[37580]: pgmap v5164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:33 smithi112 ceph-mon[37501]: pgmap v5164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:33 smithi107 ceph-mon[32659]: pgmap v5164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8851.945101 secs 2024-07-26T15:18:35.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8851.945101 secs 2024-07-26T15:18:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8851.945101 secs 2024-07-26T15:18:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:36 smithi107 ceph-mon[32659]: pgmap v5165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:36 smithi132 ceph-mon[37580]: pgmap v5165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:36 smithi112 ceph-mon[37501]: pgmap v5165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:38 smithi107 ceph-mon[32659]: pgmap v5166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:38.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:38 smithi132 ceph-mon[37580]: pgmap v5166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:38 smithi112 ceph-mon[37501]: pgmap v5166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:40 smithi107 ceph-mon[32659]: pgmap v5167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8856.945223 secs 2024-07-26T15:18:40.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:40 smithi132 ceph-mon[37580]: pgmap v5167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8856.945223 secs 2024-07-26T15:18:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:40 smithi112 ceph-mon[37501]: pgmap v5167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8856.945223 secs 2024-07-26T15:18:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:42 smithi132 ceph-mon[37580]: pgmap v5168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:42 smithi112 ceph-mon[37501]: pgmap v5168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:42 smithi107 ceph-mon[32659]: pgmap v5168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:44.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:44.284+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:44.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:44.284+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:44.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:44.284+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:44.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:44 smithi132 ceph-mon[37580]: pgmap v5169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:44 smithi112 ceph-mon[37501]: pgmap v5169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:44 smithi107 ceph-mon[32659]: pgmap v5169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8861.945323 secs 2024-07-26T15:18:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8861.945323 secs 2024-07-26T15:18:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8861.945323 secs 2024-07-26T15:18:46.521 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:46.520+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:46.520+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:18:46.520+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:46 smithi112 ceph-mon[37501]: pgmap v5170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:46 smithi107 ceph-mon[32659]: pgmap v5170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:46 smithi132 ceph-mon[37580]: pgmap v5170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:48 smithi112 ceph-mon[37501]: pgmap v5171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:48 smithi107 ceph-mon[32659]: pgmap v5171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:49.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:48 smithi132 ceph-mon[37580]: pgmap v5171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:50 smithi112 ceph-mon[37501]: pgmap v5172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8866.945444 secs 2024-07-26T15:18:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:50 smithi107 ceph-mon[32659]: pgmap v5172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8866.945444 secs 2024-07-26T15:18:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:50 smithi132 ceph-mon[37580]: pgmap v5172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:18:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8866.945444 secs 2024-07-26T15:18:52.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:52 smithi107 ceph-mon[32659]: pgmap v5173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:52 smithi112 ceph-mon[37501]: pgmap v5173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:53.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:52 smithi132 ceph-mon[37580]: pgmap v5173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:54 smithi112 ceph-mon[37501]: pgmap v5174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:54 smithi107 ceph-mon[32659]: pgmap v5174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:54 smithi132 ceph-mon[37580]: pgmap v5174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8871.945508 secs 2024-07-26T15:18:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8871.945508 secs 2024-07-26T15:18:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8871.945508 secs 2024-07-26T15:18:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:56 smithi112 ceph-mon[37501]: pgmap v5175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:56 smithi107 ceph-mon[32659]: pgmap v5175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:56 smithi132 ceph-mon[37580]: pgmap v5175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:18:58 smithi112 ceph-mon[37501]: pgmap v5176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:18:58 smithi107 ceph-mon[32659]: pgmap v5176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:59.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:18:58 smithi132 ceph-mon[37580]: pgmap v5176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:59.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:59.292+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:18:59.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:59.292+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:59.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:18:59.292+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:00 smithi112 ceph-mon[37501]: pgmap v5177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:19:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:19:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:19:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8876.945625 secs 2024-07-26T15:19:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:00 smithi107 ceph-mon[32659]: pgmap v5177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:19:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:19:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:19:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8876.945625 secs 2024-07-26T15:19:01.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:00 smithi132 ceph-mon[37580]: pgmap v5177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:19:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:19:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:19:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8876.945625 secs 2024-07-26T15:19:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:01.536+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:01.536+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:01.536+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:19:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:19:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:19:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:19:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:19:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:19:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:19:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:19:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:19:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:02 smithi112 ceph-mon[37501]: pgmap v5178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T15:19:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:02 smithi112 ceph-mon[37501]: pgmap v5179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:19:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:02 smithi107 ceph-mon[32659]: pgmap v5178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T15:19:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:02 smithi107 ceph-mon[32659]: pgmap v5179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:19:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:02 smithi132 ceph-mon[37580]: pgmap v5178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T15:19:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:02 smithi132 ceph-mon[37580]: pgmap v5179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:19:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:04 smithi112 ceph-mon[37501]: pgmap v5180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:19:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:04 smithi107 ceph-mon[32659]: pgmap v5180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:19:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:04 smithi132 ceph-mon[37580]: pgmap v5180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:19:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8881.945747 secs 2024-07-26T15:19:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8881.945747 secs 2024-07-26T15:19:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8881.945747 secs 2024-07-26T15:19:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:06 smithi112 ceph-mon[37501]: pgmap v5181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:06 smithi107 ceph-mon[32659]: pgmap v5181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:07.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:06 smithi132 ceph-mon[37580]: pgmap v5181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:08 smithi112 ceph-mon[37501]: pgmap v5182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:08 smithi107 ceph-mon[32659]: pgmap v5182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:08 smithi132 ceph-mon[37580]: pgmap v5182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:10 smithi112 ceph-mon[37501]: pgmap v5183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8886.945855 secs 2024-07-26T15:19:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:10 smithi107 ceph-mon[32659]: pgmap v5183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8886.945855 secs 2024-07-26T15:19:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:10 smithi132 ceph-mon[37580]: pgmap v5183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8886.945855 secs 2024-07-26T15:19:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:12 smithi112 ceph-mon[37501]: pgmap v5184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:19:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:12 smithi107 ceph-mon[32659]: pgmap v5184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:19:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:12 smithi132 ceph-mon[37580]: pgmap v5184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:19:14.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:14.307+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:14.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:14.307+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:14.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:14.307+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:14 smithi112 ceph-mon[37501]: pgmap v5185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:14 smithi107 ceph-mon[32659]: pgmap v5185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:14 smithi132 ceph-mon[37580]: pgmap v5185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8891.945985 secs 2024-07-26T15:19:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8891.945985 secs 2024-07-26T15:19:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8891.945985 secs 2024-07-26T15:19:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:16.546+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:16.546+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:16.547 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:16.546+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:16 smithi112 ceph-mon[37501]: pgmap v5186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:16 smithi107 ceph-mon[32659]: pgmap v5186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:16 smithi132 ceph-mon[37580]: pgmap v5186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:18 smithi112 ceph-mon[37501]: pgmap v5187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:18 smithi107 ceph-mon[32659]: pgmap v5187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:18 smithi132 ceph-mon[37580]: pgmap v5187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:20 smithi107 ceph-mon[32659]: pgmap v5188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8896.946117 secs 2024-07-26T15:19:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:20 smithi132 ceph-mon[37580]: pgmap v5188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8896.946117 secs 2024-07-26T15:19:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:20 smithi112 ceph-mon[37501]: pgmap v5188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8896.946117 secs 2024-07-26T15:19:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:22 smithi107 ceph-mon[32659]: pgmap v5189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:23.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:22 smithi132 ceph-mon[37580]: pgmap v5189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:22 smithi112 ceph-mon[37501]: pgmap v5189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:24 smithi107 ceph-mon[32659]: pgmap v5190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:24 smithi132 ceph-mon[37580]: pgmap v5190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:24 smithi112 ceph-mon[37501]: pgmap v5190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8901.946270 secs 2024-07-26T15:19:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8901.946270 secs 2024-07-26T15:19:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8901.946270 secs 2024-07-26T15:19:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:26 smithi107 ceph-mon[32659]: pgmap v5191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:27.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:26 smithi132 ceph-mon[37580]: pgmap v5191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:26 smithi112 ceph-mon[37501]: pgmap v5191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:28 smithi107 ceph-mon[32659]: pgmap v5192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:28 smithi132 ceph-mon[37580]: pgmap v5192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:28 smithi112 ceph-mon[37501]: pgmap v5192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:29.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:29.307+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:29.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:29.307+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:29.309 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:29.307+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:31.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:30 smithi112 ceph-mon[37501]: pgmap v5193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:31.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8906.946353 secs 2024-07-26T15:19:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:30 smithi107 ceph-mon[32659]: pgmap v5193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8906.946353 secs 2024-07-26T15:19:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:30 smithi132 ceph-mon[37580]: pgmap v5193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8906.946353 secs 2024-07-26T15:19:31.562 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:31.561+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:31.561+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:31.563 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:31.561+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:32 smithi107 ceph-mon[32659]: pgmap v5194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:32 smithi132 ceph-mon[37580]: pgmap v5194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:32 smithi112 ceph-mon[37501]: pgmap v5194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:34 smithi107 ceph-mon[32659]: pgmap v5195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:34 smithi132 ceph-mon[37580]: pgmap v5195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:34 smithi112 ceph-mon[37501]: pgmap v5195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8911.946419 secs 2024-07-26T15:19:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8911.946419 secs 2024-07-26T15:19:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8911.946419 secs 2024-07-26T15:19:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:36 smithi107 ceph-mon[32659]: pgmap v5196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:36 smithi132 ceph-mon[37580]: pgmap v5196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:36 smithi112 ceph-mon[37501]: pgmap v5196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:38 smithi107 ceph-mon[32659]: pgmap v5197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:38 smithi132 ceph-mon[37580]: pgmap v5197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:39.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:38 smithi112 ceph-mon[37501]: pgmap v5197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:40 smithi107 ceph-mon[32659]: pgmap v5198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8916.946538 secs 2024-07-26T15:19:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:40 smithi132 ceph-mon[37580]: pgmap v5198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8916.946538 secs 2024-07-26T15:19:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:40 smithi112 ceph-mon[37501]: pgmap v5198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8916.946538 secs 2024-07-26T15:19:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:42 smithi107 ceph-mon[32659]: pgmap v5199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:42 smithi132 ceph-mon[37580]: pgmap v5199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:43.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:43.147+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:19:43.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:43.147+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:19:43.150 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:43.149+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:19:43.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:43.149+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:19:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:42 smithi112 ceph-mon[37501]: pgmap v5199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:44.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:44.318+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:44.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:44.318+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:44.320 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:44.318+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:44 smithi107 ceph-mon[32659]: pgmap v5200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:44 smithi132 ceph-mon[37580]: pgmap v5200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:44 smithi112 ceph-mon[37501]: pgmap v5200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:45.336+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:19:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:45.336+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:19:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:45.337+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:19:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:45.337+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:19:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8921.946658 secs 2024-07-26T15:19:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8921.946658 secs 2024-07-26T15:19:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8921.946658 secs 2024-07-26T15:19:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:46.577+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:46.577+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:19:46.577+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:46 smithi132 ceph-mon[37580]: pgmap v5201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:46 smithi112 ceph-mon[37501]: pgmap v5201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:46 smithi107 ceph-mon[32659]: pgmap v5201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:48 smithi132 ceph-mon[37580]: pgmap v5202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:48 smithi112 ceph-mon[37501]: pgmap v5202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:48 smithi107 ceph-mon[32659]: pgmap v5202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:49 smithi112 ceph-mon[37501]: pgmap v5203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:49 smithi107 ceph-mon[32659]: pgmap v5203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:49 smithi132 ceph-mon[37580]: pgmap v5203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:19:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8926.946780 secs 2024-07-26T15:19:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8926.946780 secs 2024-07-26T15:19:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8926.946780 secs 2024-07-26T15:19:52.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:51 smithi107 ceph-mon[32659]: pgmap v5204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:51 smithi112 ceph-mon[37501]: pgmap v5204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:51 smithi132 ceph-mon[37580]: pgmap v5204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:19:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:54 smithi112 ceph-mon[37501]: pgmap v5205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:54 smithi107 ceph-mon[32659]: pgmap v5205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:54.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:54 smithi132 ceph-mon[37580]: pgmap v5205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:19:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8931.946925 secs 2024-07-26T15:19:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:56 smithi112 ceph-mon[37501]: pgmap v5206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8931.946925 secs 2024-07-26T15:19:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:56 smithi107 ceph-mon[32659]: pgmap v5206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8931.946925 secs 2024-07-26T15:19:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:56 smithi132 ceph-mon[37580]: pgmap v5206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:19:58 smithi112 ceph-mon[37501]: pgmap v5207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:19:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:19:58 smithi107 ceph-mon[32659]: pgmap v5207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:19:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:19:58 smithi132 ceph-mon[37580]: pgmap v5207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:19:59.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:59.335+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:19:59.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:59.335+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:59.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:19:59.335+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:00 smithi112 ceph-mon[37501]: pgmap v5208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:00.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:20:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:00 smithi107 ceph-mon[32659]: pgmap v5208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:20:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:00 smithi132 ceph-mon[37580]: pgmap v5208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:20:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:01.592+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:01.592+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:01.592+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8936.947046 secs 2024-07-26T15:20:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8936.947046 secs 2024-07-26T15:20:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8936.947046 secs 2024-07-26T15:20:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:02 smithi112 ceph-mon[37501]: pgmap v5209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:20:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:20:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:02 smithi107 ceph-mon[32659]: pgmap v5209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:20:02.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:20:02.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:02 smithi132 ceph-mon[37580]: pgmap v5209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:20:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:20:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:04 smithi112 ceph-mon[37501]: pgmap v5210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:20:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:04 smithi107 ceph-mon[32659]: pgmap v5210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:20:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:04 smithi132 ceph-mon[37580]: pgmap v5210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:20:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8941.947184 secs 2024-07-26T15:20:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:05 smithi112 ceph-mon[37501]: pgmap v5211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8941.947184 secs 2024-07-26T15:20:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:05 smithi107 ceph-mon[32659]: pgmap v5211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8941.947184 secs 2024-07-26T15:20:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:05 smithi132 ceph-mon[37580]: pgmap v5211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:20:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:20:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:06 smithi112 ceph-mon[37501]: pgmap v5212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:06 smithi112 ceph-mon[37501]: pgmap v5213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T15:20:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:20:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:20:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:06 smithi107 ceph-mon[32659]: pgmap v5212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:06 smithi107 ceph-mon[32659]: pgmap v5213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T15:20:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:20:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:20:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:06 smithi132 ceph-mon[37580]: pgmap v5212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:06 smithi132 ceph-mon[37580]: pgmap v5213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T15:20:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:20:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:08 smithi107 ceph-mon[32659]: pgmap v5214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:20:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:08 smithi132 ceph-mon[37580]: pgmap v5214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:20:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:08 smithi112 ceph-mon[37501]: pgmap v5214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:20:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:10 smithi107 ceph-mon[32659]: pgmap v5215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T15:20:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8946.947275 secs 2024-07-26T15:20:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:10 smithi132 ceph-mon[37580]: pgmap v5215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T15:20:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8946.947275 secs 2024-07-26T15:20:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:10 smithi112 ceph-mon[37501]: pgmap v5215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T15:20:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8946.947275 secs 2024-07-26T15:20:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:12 smithi107 ceph-mon[32659]: pgmap v5216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:12 smithi132 ceph-mon[37580]: pgmap v5216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:12 smithi112 ceph-mon[37501]: pgmap v5216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:20:14.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:14.342+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:14.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:14.342+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:14.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:14.342+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:14 smithi107 ceph-mon[32659]: pgmap v5217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:20:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:14 smithi132 ceph-mon[37580]: pgmap v5217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:20:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:14 smithi112 ceph-mon[37501]: pgmap v5217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:20:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8951.947393 secs 2024-07-26T15:20:16.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8951.947393 secs 2024-07-26T15:20:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8951.947393 secs 2024-07-26T15:20:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:16.607+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:16.608 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:16.607+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:16.609 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:16.608+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:16 smithi107 ceph-mon[32659]: pgmap v5218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:20:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:16 smithi132 ceph-mon[37580]: pgmap v5218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:20:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:16 smithi112 ceph-mon[37501]: pgmap v5218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:20:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:18 smithi107 ceph-mon[32659]: pgmap v5219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:18 smithi132 ceph-mon[37580]: pgmap v5219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:18 smithi112 ceph-mon[37501]: pgmap v5219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:20 smithi107 ceph-mon[32659]: pgmap v5220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8956.947507 secs 2024-07-26T15:20:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:20 smithi132 ceph-mon[37580]: pgmap v5220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8956.947507 secs 2024-07-26T15:20:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:20 smithi112 ceph-mon[37501]: pgmap v5220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8956.947507 secs 2024-07-26T15:20:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:23 smithi107 ceph-mon[32659]: pgmap v5221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:23 smithi132 ceph-mon[37580]: pgmap v5221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:23 smithi112 ceph-mon[37501]: pgmap v5221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:25 smithi107 ceph-mon[32659]: pgmap v5222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:25 smithi132 ceph-mon[37580]: pgmap v5222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:25 smithi112 ceph-mon[37501]: pgmap v5222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8961.947612 secs 2024-07-26T15:20:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8961.947612 secs 2024-07-26T15:20:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8961.947612 secs 2024-07-26T15:20:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:27 smithi107 ceph-mon[32659]: pgmap v5223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:27 smithi132 ceph-mon[37580]: pgmap v5223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:27 smithi112 ceph-mon[37501]: pgmap v5223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:29 smithi107 ceph-mon[32659]: pgmap v5224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:29 smithi132 ceph-mon[37580]: pgmap v5224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:29.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:29.353+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:29.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:29.353+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:29.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:29.353+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:29 smithi112 ceph-mon[37501]: pgmap v5224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:31 smithi107 ceph-mon[32659]: pgmap v5225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8966.947745 secs 2024-07-26T15:20:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:31 smithi132 ceph-mon[37580]: pgmap v5225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8966.947745 secs 2024-07-26T15:20:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:31 smithi112 ceph-mon[37501]: pgmap v5225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8966.947745 secs 2024-07-26T15:20:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:31.622+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:31.622+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:31.624 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:31.622+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:33 smithi107 ceph-mon[32659]: pgmap v5226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:33 smithi132 ceph-mon[37580]: pgmap v5226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:33 smithi112 ceph-mon[37501]: pgmap v5226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:35 smithi132 ceph-mon[37580]: pgmap v5227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:35 smithi112 ceph-mon[37501]: pgmap v5227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:35 smithi107 ceph-mon[32659]: pgmap v5227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8971.947883 secs 2024-07-26T15:20:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8971.947883 secs 2024-07-26T15:20:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8971.947883 secs 2024-07-26T15:20:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:37 smithi112 ceph-mon[37501]: pgmap v5228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:37 smithi107 ceph-mon[32659]: pgmap v5228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:37.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:37 smithi132 ceph-mon[37580]: pgmap v5228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:39 smithi112 ceph-mon[37501]: pgmap v5229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:39 smithi107 ceph-mon[32659]: pgmap v5229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:39.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:39 smithi132 ceph-mon[37580]: pgmap v5229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:41.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:41 smithi132 ceph-mon[37580]: pgmap v5230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:41.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8976.948003 secs 2024-07-26T15:20:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:41 smithi112 ceph-mon[37501]: pgmap v5230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8976.948003 secs 2024-07-26T15:20:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:41 smithi107 ceph-mon[32659]: pgmap v5230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8976.948003 secs 2024-07-26T15:20:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:43 smithi112 ceph-mon[37501]: pgmap v5231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:43 smithi107 ceph-mon[32659]: pgmap v5231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:43 smithi132 ceph-mon[37580]: pgmap v5231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:44.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:44.362+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:44.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:44.362+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:44.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:44.362+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:45 smithi112 ceph-mon[37501]: pgmap v5232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:45 smithi107 ceph-mon[32659]: pgmap v5232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:45.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:45 smithi132 ceph-mon[37580]: pgmap v5232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8981.948111 secs 2024-07-26T15:20:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8981.948111 secs 2024-07-26T15:20:46.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8981.948111 secs 2024-07-26T15:20:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:46.638+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:46.638+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:46.640 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:20:46.638+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:47 smithi112 ceph-mon[37501]: pgmap v5233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:47 smithi107 ceph-mon[32659]: pgmap v5233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:48.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:47 smithi132 ceph-mon[37580]: pgmap v5233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:49 smithi112 ceph-mon[37501]: pgmap v5234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:49 smithi107 ceph-mon[32659]: pgmap v5234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:49 smithi132 ceph-mon[37580]: pgmap v5234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8986.948249 secs 2024-07-26T15:20:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8986.948249 secs 2024-07-26T15:20:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8986.948249 secs 2024-07-26T15:20:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:51 smithi112 ceph-mon[37501]: pgmap v5235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:51 smithi107 ceph-mon[32659]: pgmap v5235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:51 smithi132 ceph-mon[37580]: pgmap v5235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:53.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:52 smithi107 ceph-mon[32659]: pgmap v5236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:52 smithi112 ceph-mon[37501]: pgmap v5236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:52 smithi132 ceph-mon[37580]: pgmap v5236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:54 smithi107 ceph-mon[32659]: pgmap v5237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:55.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:54 smithi132 ceph-mon[37580]: pgmap v5237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:54 smithi112 ceph-mon[37501]: pgmap v5237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8991.948365 secs 2024-07-26T15:20:56.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8991.948365 secs 2024-07-26T15:20:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8991.948365 secs 2024-07-26T15:20:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:57 smithi107 ceph-mon[32659]: pgmap v5238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:57 smithi132 ceph-mon[37580]: pgmap v5238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:57 smithi112 ceph-mon[37501]: pgmap v5238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:20:59 smithi107 ceph-mon[32659]: pgmap v5239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:20:59 smithi132 ceph-mon[37580]: pgmap v5239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:59.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:59.370+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:59.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:59.370+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:59.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:20:59.370+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:20:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:20:59 smithi112 ceph-mon[37501]: pgmap v5239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:01 smithi107 ceph-mon[32659]: pgmap v5240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 8996.948500 secs 2024-07-26T15:21:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:01 smithi132 ceph-mon[37580]: pgmap v5240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 8996.948500 secs 2024-07-26T15:21:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:01 smithi112 ceph-mon[37501]: pgmap v5240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 8996.948500 secs 2024-07-26T15:21:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:01.654+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:01.655+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:01.655+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:03 smithi107 ceph-mon[32659]: pgmap v5241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:03 smithi132 ceph-mon[37580]: pgmap v5241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:03 smithi112 ceph-mon[37501]: pgmap v5241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:05 smithi107 ceph-mon[32659]: pgmap v5242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:05 smithi132 ceph-mon[37580]: pgmap v5242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:05 smithi112 ceph-mon[37501]: pgmap v5242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9001.948607 secs 2024-07-26T15:21:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9001.948607 secs 2024-07-26T15:21:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9001.948607 secs 2024-07-26T15:21:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:07 smithi107 ceph-mon[32659]: pgmap v5243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:21:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:07 smithi132 ceph-mon[37580]: pgmap v5243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:21:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:07 smithi112 ceph-mon[37501]: pgmap v5243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:21:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:21:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:21:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:08 smithi132 ceph-mon[37580]: pgmap v5244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:21:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:08 smithi132 ceph-mon[37580]: pgmap v5245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:21:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:21:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:21:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:08 smithi112 ceph-mon[37501]: pgmap v5244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:21:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:08 smithi112 ceph-mon[37501]: pgmap v5245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:21:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:21:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:21:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:08 smithi107 ceph-mon[32659]: pgmap v5244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:21:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:08 smithi107 ceph-mon[32659]: pgmap v5245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:21:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:10 smithi112 ceph-mon[37501]: pgmap v5246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:21:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:10 smithi107 ceph-mon[32659]: pgmap v5246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:21:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:10 smithi132 ceph-mon[37580]: pgmap v5246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:21:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9006.948727 secs 2024-07-26T15:21:11.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9006.948727 secs 2024-07-26T15:21:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9006.948727 secs 2024-07-26T15:21:11.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:11.664+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0670 msgr2=0x7ff0ec0c0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T15:21:11.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:11.664+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0670 msgr2=0x7ff0ec0c0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:21:11.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:11.664+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0670 0x7ff0ec0c0ee0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fa00 tx=0x7ff0f8067040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:21:11.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:11.664+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0670 0x7ff0ec0c0ee0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800fa00 tx=0x7ff0f8067040 comp rx=0 tx=0).stop 2024-07-26T15:21:11.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:11.664+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0670 msgr2=0x7ff0ec0c0ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:21:11.665 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:11.664+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0670 0x7ff0ec0c0ee0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:21:11.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:11.664+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:21:11.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:11.665+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:21:11.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:11.667+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f8014050 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:21:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:12 smithi112 ceph-mon[37501]: pgmap v5247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:21:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:12 smithi107 ceph-mon[32659]: pgmap v5247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:21:12.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:12 smithi132 ceph-mon[37580]: pgmap v5247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:21:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:13.771+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 msgr2=0x7efc5c0bfe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000264) for more than 900000000 us, fault. 2024-07-26T15:21:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:13.771+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7efc680aacd0 tx=0x7efc680e6040 comp rx=0 tx=0).stop 2024-07-26T15:21:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:13.771+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 msgr2=0x7efc5c0bfe40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:21:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:13.771+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:21:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:13.771+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:21:13.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:13.772+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:21:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:13.773+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c6a0 tx=0x7efc680b9000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:21:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:14.163+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:21:14.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:14.163+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:21:14.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:14.164+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:21:14.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:14.164+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x565421d73380 con 0x7ff104102930 2024-07-26T15:21:14.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:14.373+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:14.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:14.373+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:14.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:14.373+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:14 smithi112 ceph-mon[37501]: pgmap v5248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:21:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:14 smithi107 ceph-mon[32659]: pgmap v5248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:21:14.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:14 smithi132 ceph-mon[37580]: pgmap v5248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:21:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:16.353+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:21:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:16.353+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:21:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:16.354+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:21:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:16.354+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:21:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:16 smithi112 ceph-mon[37501]: pgmap v5249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:21:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9011.948875 secs 2024-07-26T15:21:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:16 smithi107 ceph-mon[32659]: pgmap v5249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:21:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9011.948875 secs 2024-07-26T15:21:16.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:16 smithi132 ceph-mon[37580]: pgmap v5249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:21:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9011.948875 secs 2024-07-26T15:21:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:16.670+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:16.670+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:16.671 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:16.670+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:18 smithi112 ceph-mon[37501]: pgmap v5250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:21:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:18 smithi107 ceph-mon[32659]: pgmap v5250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:21:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:18 smithi132 ceph-mon[37580]: pgmap v5250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:21:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:20 smithi112 ceph-mon[37501]: pgmap v5251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:20 smithi107 ceph-mon[32659]: pgmap v5251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:20 smithi132 ceph-mon[37580]: pgmap v5251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9016.948952 secs 2024-07-26T15:21:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9016.948952 secs 2024-07-26T15:21:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9016.948952 secs 2024-07-26T15:21:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:22 smithi107 ceph-mon[32659]: pgmap v5252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:22 smithi132 ceph-mon[37580]: pgmap v5252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:22 smithi112 ceph-mon[37501]: pgmap v5252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:24 smithi107 ceph-mon[32659]: pgmap v5253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:24 smithi132 ceph-mon[37580]: pgmap v5253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:24 smithi112 ceph-mon[37501]: pgmap v5253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:26 smithi107 ceph-mon[32659]: pgmap v5254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9021.949091 secs 2024-07-26T15:21:26.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:26 smithi132 ceph-mon[37580]: pgmap v5254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9021.949091 secs 2024-07-26T15:21:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:26 smithi112 ceph-mon[37501]: pgmap v5254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9021.949091 secs 2024-07-26T15:21:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:28 smithi107 ceph-mon[32659]: pgmap v5255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:28.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:28 smithi132 ceph-mon[37580]: pgmap v5255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:28 smithi112 ceph-mon[37501]: pgmap v5255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:29.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:29.389+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:29.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:29.389+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:29.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:29.389+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:30 smithi107 ceph-mon[32659]: pgmap v5256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:30 smithi132 ceph-mon[37580]: pgmap v5256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:30 smithi112 ceph-mon[37501]: pgmap v5256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9026.949237 secs 2024-07-26T15:21:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9026.949237 secs 2024-07-26T15:21:31.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9026.949237 secs 2024-07-26T15:21:31.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:31.673+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:31.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:31.673+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:31.675 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:31.673+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:32 smithi107 ceph-mon[32659]: pgmap v5257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:32 smithi132 ceph-mon[37580]: pgmap v5257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:32.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:32 smithi112 ceph-mon[37501]: pgmap v5257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:34 smithi107 ceph-mon[32659]: pgmap v5258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:34 smithi132 ceph-mon[37580]: pgmap v5258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:34 smithi112 ceph-mon[37501]: pgmap v5258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:36 smithi107 ceph-mon[32659]: pgmap v5259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9031.949319 secs 2024-07-26T15:21:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:36 smithi132 ceph-mon[37580]: pgmap v5259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9031.949319 secs 2024-07-26T15:21:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:36 smithi112 ceph-mon[37501]: pgmap v5259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9031.949319 secs 2024-07-26T15:21:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:38 smithi107 ceph-mon[32659]: pgmap v5260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:38.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:38 smithi132 ceph-mon[37580]: pgmap v5260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:38 smithi112 ceph-mon[37501]: pgmap v5260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:40 smithi107 ceph-mon[32659]: pgmap v5261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:40 smithi132 ceph-mon[37580]: pgmap v5261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:40 smithi112 ceph-mon[37501]: pgmap v5261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9036.949445 secs 2024-07-26T15:21:41.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9036.949445 secs 2024-07-26T15:21:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9036.949445 secs 2024-07-26T15:21:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:42 smithi107 ceph-mon[32659]: pgmap v5262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:42.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:42 smithi132 ceph-mon[37580]: pgmap v5262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:42 smithi112 ceph-mon[37501]: pgmap v5262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:44.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:44.404+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:44.404+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:44.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:44.404+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:44 smithi107 ceph-mon[32659]: pgmap v5263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:44 smithi132 ceph-mon[37580]: pgmap v5263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:44 smithi112 ceph-mon[37501]: pgmap v5263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:46 smithi132 ceph-mon[37580]: pgmap v5264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9041.949568 secs 2024-07-26T15:21:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:46.682+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:46.682+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:21:46.682+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:46 smithi112 ceph-mon[37501]: pgmap v5264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9041.949568 secs 2024-07-26T15:21:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:46 smithi107 ceph-mon[32659]: pgmap v5264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9041.949568 secs 2024-07-26T15:21:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:48 smithi132 ceph-mon[37580]: pgmap v5265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:48 smithi112 ceph-mon[37501]: pgmap v5265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:48 smithi107 ceph-mon[32659]: pgmap v5265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:50 smithi132 ceph-mon[37580]: pgmap v5266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:50 smithi112 ceph-mon[37501]: pgmap v5266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:50 smithi107 ceph-mon[32659]: pgmap v5266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9046.949663 secs 2024-07-26T15:21:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9046.949663 secs 2024-07-26T15:21:51.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9046.949663 secs 2024-07-26T15:21:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:52 smithi112 ceph-mon[37501]: pgmap v5267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:52 smithi107 ceph-mon[32659]: pgmap v5267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:52.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:52 smithi132 ceph-mon[37580]: pgmap v5267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:54.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:54 smithi107 ceph-mon[32659]: pgmap v5268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:54 smithi112 ceph-mon[37501]: pgmap v5268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:54 smithi132 ceph-mon[37580]: pgmap v5268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:21:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:56 smithi112 ceph-mon[37501]: pgmap v5269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9051.949789 secs 2024-07-26T15:21:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:56 smithi107 ceph-mon[32659]: pgmap v5269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9051.949789 secs 2024-07-26T15:21:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:56 smithi132 ceph-mon[37580]: pgmap v5269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9051.949789 secs 2024-07-26T15:21:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:21:58 smithi112 ceph-mon[37501]: pgmap v5270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:21:58 smithi107 ceph-mon[32659]: pgmap v5270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:58.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:21:58 smithi132 ceph-mon[37580]: pgmap v5270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:59.419+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:21:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:59.419+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:21:59.419+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:00 smithi112 ceph-mon[37501]: pgmap v5271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:00 smithi107 ceph-mon[32659]: pgmap v5271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:00.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:00 smithi132 ceph-mon[37580]: pgmap v5271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:01.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9056.949906 secs 2024-07-26T15:22:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:01.698+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:01.698+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:01.699 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:01.698+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9056.949906 secs 2024-07-26T15:22:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9056.949906 secs 2024-07-26T15:22:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:02 smithi112 ceph-mon[37501]: pgmap v5272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:02 smithi107 ceph-mon[32659]: pgmap v5272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:02 smithi132 ceph-mon[37580]: pgmap v5272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:04 smithi112 ceph-mon[37501]: pgmap v5273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:04 smithi107 ceph-mon[32659]: pgmap v5273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:04.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:04 smithi132 ceph-mon[37580]: pgmap v5273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:06 smithi112 ceph-mon[37501]: pgmap v5274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9061.950050 secs 2024-07-26T15:22:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:06 smithi107 ceph-mon[32659]: pgmap v5274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9061.950050 secs 2024-07-26T15:22:06.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:06 smithi132 ceph-mon[37580]: pgmap v5274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9061.950050 secs 2024-07-26T15:22:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:22:07.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:22:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:22:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:08 smithi112 ceph-mon[37501]: pgmap v5275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:22:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:22:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:22:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:08 smithi107 ceph-mon[32659]: pgmap v5275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:22:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:22:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:22:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:08 smithi132 ceph-mon[37580]: pgmap v5275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:22:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:22:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:22:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:09 smithi112 ceph-mon[37501]: pgmap v5276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:22:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:09 smithi112 ceph-mon[37501]: pgmap v5277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:22:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:09 smithi107 ceph-mon[32659]: pgmap v5276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:22:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:09 smithi107 ceph-mon[32659]: pgmap v5277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:22:09.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:09 smithi132 ceph-mon[37580]: pgmap v5276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:22:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:09 smithi132 ceph-mon[37580]: pgmap v5277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:22:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9066.950167 secs 2024-07-26T15:22:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:11 smithi112 ceph-mon[37501]: pgmap v5278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9066.950167 secs 2024-07-26T15:22:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:11 smithi107 ceph-mon[32659]: pgmap v5278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9066.950167 secs 2024-07-26T15:22:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:11 smithi132 ceph-mon[37580]: pgmap v5278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:13 smithi112 ceph-mon[37501]: pgmap v5279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:13 smithi107 ceph-mon[32659]: pgmap v5279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:13 smithi132 ceph-mon[37580]: pgmap v5279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:14.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:14.435+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:14.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:14.436+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:14.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:14.436+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:15 smithi112 ceph-mon[37501]: pgmap v5280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:15 smithi107 ceph-mon[32659]: pgmap v5280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:15 smithi132 ceph-mon[37580]: pgmap v5280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9071.950238 secs 2024-07-26T15:22:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:16.714+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:16.714+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:16.714+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9071.950238 secs 2024-07-26T15:22:16.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9071.950238 secs 2024-07-26T15:22:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:17 smithi112 ceph-mon[37501]: pgmap v5281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:22:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:17 smithi107 ceph-mon[32659]: pgmap v5281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:22:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:17 smithi132 ceph-mon[37580]: pgmap v5281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:22:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:19 smithi112 ceph-mon[37501]: pgmap v5282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:22:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:19 smithi107 ceph-mon[32659]: pgmap v5282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:22:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:19 smithi132 ceph-mon[37580]: pgmap v5282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:22:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9076.950368 secs 2024-07-26T15:22:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:21 smithi112 ceph-mon[37501]: pgmap v5283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9076.950368 secs 2024-07-26T15:22:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:21 smithi107 ceph-mon[32659]: pgmap v5283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9076.950368 secs 2024-07-26T15:22:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:21 smithi132 ceph-mon[37580]: pgmap v5283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:23 smithi112 ceph-mon[37501]: pgmap v5284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:23 smithi107 ceph-mon[32659]: pgmap v5284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:23 smithi132 ceph-mon[37580]: pgmap v5284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:25 smithi112 ceph-mon[37501]: pgmap v5285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:22:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:25 smithi107 ceph-mon[32659]: pgmap v5285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:22:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:25 smithi132 ceph-mon[37580]: pgmap v5285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:22:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9081.950494 secs 2024-07-26T15:22:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9081.950494 secs 2024-07-26T15:22:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9081.950494 secs 2024-07-26T15:22:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:27 smithi107 ceph-mon[32659]: pgmap v5286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:22:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:27 smithi132 ceph-mon[37580]: pgmap v5286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:22:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:27 smithi112 ceph-mon[37501]: pgmap v5286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:22:29.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:29.451+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:29.452 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:29.451+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:29.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:29.451+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:29 smithi107 ceph-mon[32659]: pgmap v5287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:22:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:29 smithi132 ceph-mon[37580]: pgmap v5287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:22:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:29 smithi112 ceph-mon[37501]: pgmap v5287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:22:31.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:31.723+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:31.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:31.723+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:31.724 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:31.723+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:31.752 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9086.950590 secs 2024-07-26T15:22:31.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:31 smithi112 ceph-mon[37501]: pgmap v5288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9086.950590 secs 2024-07-26T15:22:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:31 smithi107 ceph-mon[32659]: pgmap v5288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:31.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9086.950590 secs 2024-07-26T15:22:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:31 smithi132 ceph-mon[37580]: pgmap v5288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:33 smithi107 ceph-mon[32659]: pgmap v5289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:33.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:33 smithi132 ceph-mon[37580]: pgmap v5289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:33 smithi112 ceph-mon[37501]: pgmap v5289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:35 smithi107 ceph-mon[32659]: pgmap v5290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:35 smithi132 ceph-mon[37580]: pgmap v5290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:35 smithi112 ceph-mon[37501]: pgmap v5290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9091.950713 secs 2024-07-26T15:22:36.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9091.950713 secs 2024-07-26T15:22:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9091.950713 secs 2024-07-26T15:22:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:37 smithi107 ceph-mon[32659]: pgmap v5291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:37 smithi132 ceph-mon[37580]: pgmap v5291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:37 smithi112 ceph-mon[37501]: pgmap v5291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:39 smithi107 ceph-mon[32659]: pgmap v5292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:39 smithi132 ceph-mon[37580]: pgmap v5292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:39 smithi112 ceph-mon[37501]: pgmap v5292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9096.950828 secs 2024-07-26T15:22:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9096.950828 secs 2024-07-26T15:22:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9096.950828 secs 2024-07-26T15:22:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:41 smithi107 ceph-mon[32659]: pgmap v5293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:41.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:41 smithi132 ceph-mon[37580]: pgmap v5293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:41 smithi112 ceph-mon[37501]: pgmap v5293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:43 smithi107 ceph-mon[32659]: pgmap v5294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:43 smithi132 ceph-mon[37580]: pgmap v5294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:43 smithi112 ceph-mon[37501]: pgmap v5294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:44.463+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:44.465 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:44.463+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:44.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:44.463+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:45 smithi107 ceph-mon[32659]: pgmap v5295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:22:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9101.950889 secs 2024-07-26T15:22:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:45 smithi132 ceph-mon[37580]: pgmap v5295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:22:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9101.950889 secs 2024-07-26T15:22:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:45 smithi112 ceph-mon[37501]: pgmap v5295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:22:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9101.950889 secs 2024-07-26T15:22:46.739 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:46.737+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:46.740 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:46.738+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:46.740 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:22:46.738+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:47 smithi132 ceph-mon[37580]: pgmap v5296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:47 smithi112 ceph-mon[37501]: pgmap v5296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:47 smithi107 ceph-mon[32659]: pgmap v5296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:49 smithi132 ceph-mon[37580]: pgmap v5297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:49 smithi112 ceph-mon[37501]: pgmap v5297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:49 smithi107 ceph-mon[32659]: pgmap v5297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9106.951007 secs 2024-07-26T15:22:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9106.951007 secs 2024-07-26T15:22:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9106.951007 secs 2024-07-26T15:22:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:51 smithi112 ceph-mon[37501]: pgmap v5298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:51 smithi107 ceph-mon[32659]: pgmap v5298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:51 smithi132 ceph-mon[37580]: pgmap v5298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:22:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:53 smithi112 ceph-mon[37501]: pgmap v5299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:53 smithi107 ceph-mon[32659]: pgmap v5299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:54.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:53 smithi132 ceph-mon[37580]: pgmap v5299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:55.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:55 smithi107 ceph-mon[32659]: pgmap v5300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:22:55.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9111.951129 secs 2024-07-26T15:22:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:55 smithi112 ceph-mon[37501]: pgmap v5300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:22:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9111.951129 secs 2024-07-26T15:22:56.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:55 smithi132 ceph-mon[37580]: pgmap v5300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:22:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9111.951129 secs 2024-07-26T15:22:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:57 smithi112 ceph-mon[37501]: pgmap v5301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:57 smithi107 ceph-mon[32659]: pgmap v5301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:57 smithi132 ceph-mon[37580]: pgmap v5301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:59.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:59.474+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:59.475 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:59.474+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:59.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:22:59.474+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:22:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:22:59 smithi107 ceph-mon[32659]: pgmap v5302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:22:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:22:59 smithi112 ceph-mon[37501]: pgmap v5302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:00.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:22:59 smithi132 ceph-mon[37580]: pgmap v5302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9116.951241 secs 2024-07-26T15:23:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9116.951241 secs 2024-07-26T15:23:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9116.951241 secs 2024-07-26T15:23:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:01.752+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:01.752+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:01.753 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:01.752+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:01.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:01 smithi112 ceph-mon[37501]: pgmap v5303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:01 smithi107 ceph-mon[32659]: pgmap v5303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:01 smithi132 ceph-mon[37580]: pgmap v5303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:03 smithi112 ceph-mon[37501]: pgmap v5304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:04.026 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:03 smithi107 ceph-mon[32659]: pgmap v5304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:04.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:03 smithi132 ceph-mon[37580]: pgmap v5304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:05 smithi112 ceph-mon[37501]: pgmap v5305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9121.951358 secs 2024-07-26T15:23:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:05 smithi107 ceph-mon[32659]: pgmap v5305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9121.951358 secs 2024-07-26T15:23:06.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:05 smithi132 ceph-mon[37580]: pgmap v5305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9121.951358 secs 2024-07-26T15:23:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:07 smithi112 ceph-mon[37501]: pgmap v5306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:07 smithi107 ceph-mon[32659]: pgmap v5306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:08.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:07 smithi132 ceph-mon[37580]: pgmap v5306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:23:09.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:23:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:23:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:09 smithi112 ceph-mon[37501]: pgmap v5307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:23:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:23:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:23:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:09 smithi107 ceph-mon[32659]: pgmap v5307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:23:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:23:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:23:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:09 smithi132 ceph-mon[37580]: pgmap v5307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:23:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:23:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:23:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:10 smithi112 ceph-mon[37501]: pgmap v5308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:23:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:10 smithi112 ceph-mon[37501]: pgmap v5309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9126.951475 secs 2024-07-26T15:23:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:10 smithi107 ceph-mon[32659]: pgmap v5308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:23:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:10 smithi107 ceph-mon[32659]: pgmap v5309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9126.951475 secs 2024-07-26T15:23:11.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:10 smithi132 ceph-mon[37580]: pgmap v5308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:23:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:10 smithi132 ceph-mon[37580]: pgmap v5309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9126.951475 secs 2024-07-26T15:23:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:12 smithi112 ceph-mon[37501]: pgmap v5310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:12 smithi107 ceph-mon[32659]: pgmap v5310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:13.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:12 smithi132 ceph-mon[37580]: pgmap v5310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:14.480+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:14.481 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:14.480+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:14.480+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:14 smithi107 ceph-mon[32659]: pgmap v5311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:23:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:14 smithi112 ceph-mon[37501]: pgmap v5311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:23:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:14 smithi132 ceph-mon[37580]: pgmap v5311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:23:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9131.951589 secs 2024-07-26T15:23:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9131.951589 secs 2024-07-26T15:23:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9131.951589 secs 2024-07-26T15:23:16.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:16.762+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:16.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:16.762+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:16.763 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:16.762+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:16 smithi112 ceph-mon[37501]: pgmap v5312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:16 smithi107 ceph-mon[32659]: pgmap v5312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:16 smithi132 ceph-mon[37580]: pgmap v5312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:18 smithi112 ceph-mon[37501]: pgmap v5313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:18 smithi107 ceph-mon[32659]: pgmap v5313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:19.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:18 smithi132 ceph-mon[37580]: pgmap v5313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:20 smithi112 ceph-mon[37501]: pgmap v5314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:23:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9136.951712 secs 2024-07-26T15:23:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:20 smithi107 ceph-mon[32659]: pgmap v5314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:23:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9136.951712 secs 2024-07-26T15:23:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:20 smithi132 ceph-mon[37580]: pgmap v5314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:23:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9136.951712 secs 2024-07-26T15:23:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:22 smithi112 ceph-mon[37501]: pgmap v5315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:22 smithi107 ceph-mon[32659]: pgmap v5315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:22 smithi132 ceph-mon[37580]: pgmap v5315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:24 smithi112 ceph-mon[37501]: pgmap v5316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:24 smithi107 ceph-mon[32659]: pgmap v5316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:24 smithi132 ceph-mon[37580]: pgmap v5316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9141.951833 secs 2024-07-26T15:23:25.978 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9141.951833 secs 2024-07-26T15:23:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9141.951833 secs 2024-07-26T15:23:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:26 smithi112 ceph-mon[37501]: pgmap v5317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:26 smithi107 ceph-mon[32659]: pgmap v5317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:26 smithi132 ceph-mon[37580]: pgmap v5317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:28 smithi107 ceph-mon[32659]: pgmap v5318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:28 smithi132 ceph-mon[37580]: pgmap v5318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:28 smithi112 ceph-mon[37501]: pgmap v5318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:29.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:29.489+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:29.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:29.489+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:29.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:29.489+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:31.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:30 smithi112 ceph-mon[37501]: pgmap v5319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:31.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9146.951949 secs 2024-07-26T15:23:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:30 smithi107 ceph-mon[32659]: pgmap v5319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9146.951949 secs 2024-07-26T15:23:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:30 smithi132 ceph-mon[37580]: pgmap v5319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9146.951949 secs 2024-07-26T15:23:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:31.778+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:31.778+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:31.778+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:32 smithi107 ceph-mon[32659]: pgmap v5320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:23:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:32 smithi132 ceph-mon[37580]: pgmap v5320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:23:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:32 smithi112 ceph-mon[37501]: pgmap v5320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:23:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:34 smithi107 ceph-mon[32659]: pgmap v5321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:35.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:34 smithi132 ceph-mon[37580]: pgmap v5321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:34 smithi112 ceph-mon[37501]: pgmap v5321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9151.952046 secs 2024-07-26T15:23:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9151.952046 secs 2024-07-26T15:23:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9151.952046 secs 2024-07-26T15:23:36.999 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:36 smithi107 ceph-mon[32659]: pgmap v5322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:37.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:36 smithi132 ceph-mon[37580]: pgmap v5322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:36 smithi112 ceph-mon[37501]: pgmap v5322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:38 smithi107 ceph-mon[32659]: pgmap v5323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:38 smithi132 ceph-mon[37580]: pgmap v5323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:38 smithi112 ceph-mon[37501]: pgmap v5323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:40 smithi107 ceph-mon[32659]: pgmap v5324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9156.952146 secs 2024-07-26T15:23:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:40 smithi132 ceph-mon[37580]: pgmap v5324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9156.952146 secs 2024-07-26T15:23:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:40 smithi112 ceph-mon[37501]: pgmap v5324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9156.952146 secs 2024-07-26T15:23:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:42 smithi107 ceph-mon[32659]: pgmap v5325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:42 smithi132 ceph-mon[37580]: pgmap v5325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:42 smithi112 ceph-mon[37501]: pgmap v5325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:44.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:44.504+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:44.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:44.504+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:44.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:44.504+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:44 smithi107 ceph-mon[32659]: pgmap v5326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:44 smithi132 ceph-mon[37580]: pgmap v5326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:44 smithi112 ceph-mon[37501]: pgmap v5326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9161.952263 secs 2024-07-26T15:23:46.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9161.952263 secs 2024-07-26T15:23:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9161.952263 secs 2024-07-26T15:23:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:46.788+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:46.788+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:23:46.788+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:46 smithi107 ceph-mon[32659]: pgmap v5327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:46 smithi132 ceph-mon[37580]: pgmap v5327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:46 smithi112 ceph-mon[37501]: pgmap v5327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:48 smithi107 ceph-mon[32659]: pgmap v5328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:49.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:48 smithi132 ceph-mon[37580]: pgmap v5328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:48 smithi112 ceph-mon[37501]: pgmap v5328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:50 smithi107 ceph-mon[32659]: pgmap v5329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9166.952375 secs 2024-07-26T15:23:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:50 smithi132 ceph-mon[37580]: pgmap v5329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:51.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9166.952375 secs 2024-07-26T15:23:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:50 smithi112 ceph-mon[37501]: pgmap v5329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:23:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9166.952375 secs 2024-07-26T15:23:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:52 smithi132 ceph-mon[37580]: pgmap v5330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:52 smithi112 ceph-mon[37501]: pgmap v5330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:52 smithi107 ceph-mon[32659]: pgmap v5330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:54 smithi132 ceph-mon[37580]: pgmap v5331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:55.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:54 smithi107 ceph-mon[32659]: pgmap v5331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:54 smithi112 ceph-mon[37501]: pgmap v5331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9171.952494 secs 2024-07-26T15:23:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9171.952494 secs 2024-07-26T15:23:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9171.952494 secs 2024-07-26T15:23:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:56 smithi112 ceph-mon[37501]: pgmap v5332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:56 smithi107 ceph-mon[32659]: pgmap v5332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:56 smithi132 ceph-mon[37580]: pgmap v5332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:23:58 smithi112 ceph-mon[37501]: pgmap v5333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:23:58 smithi107 ceph-mon[32659]: pgmap v5333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:23:58 smithi132 ceph-mon[37580]: pgmap v5333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:59.520+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:59.520+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:23:59.522 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:23:59.520+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:00 smithi112 ceph-mon[37501]: pgmap v5334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9176.952613 secs 2024-07-26T15:24:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:00 smithi107 ceph-mon[32659]: pgmap v5334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9176.952613 secs 2024-07-26T15:24:01.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:00 smithi132 ceph-mon[37580]: pgmap v5334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9176.952613 secs 2024-07-26T15:24:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:01.798+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:01.798+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:01.798+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:02 smithi112 ceph-mon[37501]: pgmap v5335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:02 smithi107 ceph-mon[32659]: pgmap v5335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:03.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:02 smithi132 ceph-mon[37580]: pgmap v5335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:04 smithi112 ceph-mon[37501]: pgmap v5336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:04 smithi107 ceph-mon[32659]: pgmap v5336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:04 smithi132 ceph-mon[37580]: pgmap v5336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:06.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9181.952736 secs 2024-07-26T15:24:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9181.952736 secs 2024-07-26T15:24:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9181.952736 secs 2024-07-26T15:24:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:06 smithi112 ceph-mon[37501]: pgmap v5337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:06 smithi107 ceph-mon[32659]: pgmap v5337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:06 smithi132 ceph-mon[37580]: pgmap v5337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:08 smithi112 ceph-mon[37501]: pgmap v5338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:08 smithi107 ceph-mon[32659]: pgmap v5338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:08 smithi132 ceph-mon[37580]: pgmap v5338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:24:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:24:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:24:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:10 smithi112 ceph-mon[37501]: pgmap v5339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9186.952823 secs 2024-07-26T15:24:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:24:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:24:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:24:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:10 smithi107 ceph-mon[32659]: pgmap v5339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9186.952823 secs 2024-07-26T15:24:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:24:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:24:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:24:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:10 smithi132 ceph-mon[37580]: pgmap v5339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9186.952823 secs 2024-07-26T15:24:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:24:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:24:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:24:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:11 smithi112 ceph-mon[37501]: pgmap v5340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:24:12.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:11 smithi132 ceph-mon[37580]: pgmap v5340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:24:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:11 smithi107 ceph-mon[32659]: pgmap v5340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:24:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:12 smithi112 ceph-mon[37501]: pgmap v5341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:24:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:12 smithi107 ceph-mon[32659]: pgmap v5341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:24:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:12 smithi132 ceph-mon[37580]: pgmap v5341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:24:14.196 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:14.195+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:24:14.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:14.195+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:24:14.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:14.197+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:24:14.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:14.197+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:24:14.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:14.528+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:14.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:14.528+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:14.529 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:14.528+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:15 smithi107 ceph-mon[32659]: pgmap v5342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:24:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9191.952966 secs 2024-07-26T15:24:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:15 smithi132 ceph-mon[37580]: pgmap v5342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:24:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9191.952966 secs 2024-07-26T15:24:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:15 smithi112 ceph-mon[37501]: pgmap v5342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:24:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9191.952966 secs 2024-07-26T15:24:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:16.385+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:24:16.386 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:16.385+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:24:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:16.386+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:24:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:16.386+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:24:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:16.811+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:16.811+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:16.812 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:16.811+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:17 smithi107 ceph-mon[32659]: pgmap v5343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:24:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:17 smithi132 ceph-mon[37580]: pgmap v5343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:24:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:17 smithi112 ceph-mon[37501]: pgmap v5343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:24:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:19 smithi107 ceph-mon[32659]: pgmap v5344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:24:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:19 smithi132 ceph-mon[37580]: pgmap v5344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:24:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:19 smithi112 ceph-mon[37501]: pgmap v5344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:24:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9196.953054 secs 2024-07-26T15:24:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9196.953054 secs 2024-07-26T15:24:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9196.953054 secs 2024-07-26T15:24:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:21 smithi107 ceph-mon[32659]: pgmap v5345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:24:22.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:21 smithi132 ceph-mon[37580]: pgmap v5345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:24:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:21 smithi112 ceph-mon[37501]: pgmap v5345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:24:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:23 smithi107 ceph-mon[32659]: pgmap v5346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:23 smithi132 ceph-mon[37580]: pgmap v5346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:23 smithi112 ceph-mon[37501]: pgmap v5346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:25 smithi107 ceph-mon[32659]: pgmap v5347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9201.953175 secs 2024-07-26T15:24:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:25 smithi132 ceph-mon[37580]: pgmap v5347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9201.953175 secs 2024-07-26T15:24:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:25 smithi112 ceph-mon[37501]: pgmap v5347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9201.953175 secs 2024-07-26T15:24:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:27 smithi107 ceph-mon[32659]: pgmap v5348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:27 smithi132 ceph-mon[37580]: pgmap v5348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:27 smithi112 ceph-mon[37501]: pgmap v5348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:29.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:29.544+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:29.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:29.544+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:29.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:29.544+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:29 smithi107 ceph-mon[32659]: pgmap v5349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:30.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:29 smithi132 ceph-mon[37580]: pgmap v5349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:29 smithi112 ceph-mon[37501]: pgmap v5349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9206.953295 secs 2024-07-26T15:24:31.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9206.953295 secs 2024-07-26T15:24:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9206.953295 secs 2024-07-26T15:24:31.827 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:31.827+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:31.827+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:31.828 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:31.827+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:31 smithi132 ceph-mon[37580]: pgmap v5350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:31 smithi112 ceph-mon[37501]: pgmap v5350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:31 smithi107 ceph-mon[32659]: pgmap v5350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:33 smithi132 ceph-mon[37580]: pgmap v5351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:33 smithi112 ceph-mon[37501]: pgmap v5351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:33 smithi107 ceph-mon[32659]: pgmap v5351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:36.186 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:35 smithi107 ceph-mon[32659]: pgmap v5352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:36.186 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9211.953423 secs 2024-07-26T15:24:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:35 smithi112 ceph-mon[37501]: pgmap v5352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9211.953423 secs 2024-07-26T15:24:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:35 smithi132 ceph-mon[37580]: pgmap v5352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9211.953423 secs 2024-07-26T15:24:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:37 smithi112 ceph-mon[37501]: pgmap v5353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:37 smithi107 ceph-mon[32659]: pgmap v5353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:37 smithi132 ceph-mon[37580]: pgmap v5353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:39 smithi112 ceph-mon[37501]: pgmap v5354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:39 smithi107 ceph-mon[32659]: pgmap v5354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:39 smithi132 ceph-mon[37580]: pgmap v5354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9216.953543 secs 2024-07-26T15:24:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9216.953543 secs 2024-07-26T15:24:41.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9216.953543 secs 2024-07-26T15:24:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:41 smithi112 ceph-mon[37501]: pgmap v5355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:42.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:41 smithi132 ceph-mon[37580]: pgmap v5355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:41 smithi107 ceph-mon[32659]: pgmap v5355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:43 smithi112 ceph-mon[37501]: pgmap v5356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:43 smithi107 ceph-mon[32659]: pgmap v5356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:43 smithi132 ceph-mon[37580]: pgmap v5356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:44.547+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:44.547+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:44.547+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:45 smithi112 ceph-mon[37501]: pgmap v5357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9221.953668 secs 2024-07-26T15:24:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:45 smithi107 ceph-mon[32659]: pgmap v5357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9221.953668 secs 2024-07-26T15:24:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:45 smithi132 ceph-mon[37580]: pgmap v5357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9221.953668 secs 2024-07-26T15:24:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:46.841+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:46.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:46.841+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:46.843 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:24:46.841+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:48.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:47 smithi112 ceph-mon[37501]: pgmap v5358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:47 smithi107 ceph-mon[32659]: pgmap v5358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:47 smithi132 ceph-mon[37580]: pgmap v5358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:49 smithi112 ceph-mon[37501]: pgmap v5359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:49 smithi107 ceph-mon[32659]: pgmap v5359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:50.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:49 smithi132 ceph-mon[37580]: pgmap v5359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9226.953743 secs 2024-07-26T15:24:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9226.953743 secs 2024-07-26T15:24:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9226.953743 secs 2024-07-26T15:24:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:51 smithi112 ceph-mon[37501]: pgmap v5360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:51 smithi107 ceph-mon[32659]: pgmap v5360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:52.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:51 smithi132 ceph-mon[37580]: pgmap v5360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:52 smithi112 ceph-mon[37501]: pgmap v5361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:52 smithi107 ceph-mon[32659]: pgmap v5361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:52 smithi132 ceph-mon[37580]: pgmap v5361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:55 smithi107 ceph-mon[32659]: pgmap v5362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9231.953884 secs 2024-07-26T15:24:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:55 smithi132 ceph-mon[37580]: pgmap v5362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9231.953884 secs 2024-07-26T15:24:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:55 smithi112 ceph-mon[37501]: pgmap v5362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:24:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9231.953884 secs 2024-07-26T15:24:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:57 smithi107 ceph-mon[32659]: pgmap v5363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:24:58.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:57 smithi132 ceph-mon[37580]: pgmap v5363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:24:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:57 smithi112 ceph-mon[37501]: pgmap v5363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:24:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:59.562+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:24:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:59.562+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:59.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:24:59.562+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:24:59 smithi107 ceph-mon[32659]: pgmap v5364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:25:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:24:59 smithi132 ceph-mon[37580]: pgmap v5364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:25:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:24:59 smithi112 ceph-mon[37501]: pgmap v5364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:25:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9236.954002 secs 2024-07-26T15:25:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9236.954002 secs 2024-07-26T15:25:01.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9236.954002 secs 2024-07-26T15:25:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:01.857+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:01.857+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:01.857+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:01 smithi107 ceph-mon[32659]: pgmap v5365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:25:02.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:01 smithi132 ceph-mon[37580]: pgmap v5365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:25:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:01 smithi112 ceph-mon[37501]: pgmap v5365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:25:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:03 smithi132 ceph-mon[37580]: pgmap v5366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:25:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:03 smithi112 ceph-mon[37501]: pgmap v5366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:25:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:03 smithi107 ceph-mon[32659]: pgmap v5366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:25:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:05 smithi132 ceph-mon[37580]: pgmap v5367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:25:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9241.954144 secs 2024-07-26T15:25:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:05 smithi112 ceph-mon[37501]: pgmap v5367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:25:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9241.954144 secs 2024-07-26T15:25:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:05 smithi107 ceph-mon[32659]: pgmap v5367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:25:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9241.954144 secs 2024-07-26T15:25:08.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:07 smithi132 ceph-mon[37580]: pgmap v5368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:25:08.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:07 smithi107 ceph-mon[32659]: pgmap v5368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:25:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:07 smithi112 ceph-mon[37501]: pgmap v5368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:25:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:09 smithi112 ceph-mon[37501]: pgmap v5369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:09 smithi107 ceph-mon[32659]: pgmap v5369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:09 smithi132 ceph-mon[37580]: pgmap v5369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9246.954223 secs 2024-07-26T15:25:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9246.954223 secs 2024-07-26T15:25:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9246.954223 secs 2024-07-26T15:25:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:11 smithi112 ceph-mon[37501]: pgmap v5370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:25:12.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:11 smithi132 ceph-mon[37580]: pgmap v5370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:12.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:25:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:11 smithi107 ceph-mon[32659]: pgmap v5370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:25:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:25:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:25:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:25:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:25:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:25:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:25:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:25:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:25:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:25:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:13 smithi112 ceph-mon[37501]: pgmap v5371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:25:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:13 smithi107 ceph-mon[32659]: pgmap v5371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:25:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:13 smithi132 ceph-mon[37580]: pgmap v5371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:25:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:14.579+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:14.579+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:14.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:14.579+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:15 smithi112 ceph-mon[37501]: pgmap v5372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:25:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9251.954340 secs 2024-07-26T15:25:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:15 smithi107 ceph-mon[32659]: pgmap v5372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:25:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9251.954340 secs 2024-07-26T15:25:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:15 smithi132 ceph-mon[37580]: pgmap v5372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:25:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9251.954340 secs 2024-07-26T15:25:16.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:16.873+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:16.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:16.873+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:16.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:16.873+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:17 smithi112 ceph-mon[37501]: pgmap v5373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:25:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:17 smithi107 ceph-mon[32659]: pgmap v5373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:25:18.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:17 smithi132 ceph-mon[37580]: pgmap v5373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:25:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:19 smithi112 ceph-mon[37501]: pgmap v5374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:25:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:19 smithi107 ceph-mon[32659]: pgmap v5374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:25:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:19 smithi132 ceph-mon[37580]: pgmap v5374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:25:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9256.954465 secs 2024-07-26T15:25:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9256.954465 secs 2024-07-26T15:25:21.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9256.954465 secs 2024-07-26T15:25:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:21 smithi112 ceph-mon[37501]: pgmap v5375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:25:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:21 smithi107 ceph-mon[32659]: pgmap v5375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:25:22.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:21 smithi132 ceph-mon[37580]: pgmap v5375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:25:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:22 smithi112 ceph-mon[37501]: pgmap v5376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:25:23.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:22 smithi107 ceph-mon[32659]: pgmap v5376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:25:23.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:22 smithi132 ceph-mon[37580]: pgmap v5376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:25:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:24 smithi112 ceph-mon[37501]: pgmap v5377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:24 smithi107 ceph-mon[32659]: pgmap v5377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:25.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:24 smithi132 ceph-mon[37580]: pgmap v5377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9261.954601 secs 2024-07-26T15:25:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9261.954601 secs 2024-07-26T15:25:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9261.954601 secs 2024-07-26T15:25:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:26 smithi112 ceph-mon[37501]: pgmap v5378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:26 smithi107 ceph-mon[32659]: pgmap v5378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:26 smithi132 ceph-mon[37580]: pgmap v5378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:28 smithi107 ceph-mon[32659]: pgmap v5379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:28 smithi132 ceph-mon[37580]: pgmap v5379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:28 smithi112 ceph-mon[37501]: pgmap v5379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:29.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:29.594+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:29.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:29.594+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:29.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:29.594+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:31.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:30 smithi112 ceph-mon[37501]: pgmap v5380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:31.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9266.954706 secs 2024-07-26T15:25:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:30 smithi107 ceph-mon[32659]: pgmap v5380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9266.954706 secs 2024-07-26T15:25:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:30 smithi132 ceph-mon[37580]: pgmap v5380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9266.954706 secs 2024-07-26T15:25:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:31.882+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:31.882+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:31.882+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:32 smithi107 ceph-mon[32659]: pgmap v5381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:32 smithi132 ceph-mon[37580]: pgmap v5381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:32 smithi112 ceph-mon[37501]: pgmap v5381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:34 smithi107 ceph-mon[32659]: pgmap v5382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:34 smithi132 ceph-mon[37580]: pgmap v5382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:34 smithi112 ceph-mon[37501]: pgmap v5382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9271.954827 secs 2024-07-26T15:25:36.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9271.954827 secs 2024-07-26T15:25:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9271.954827 secs 2024-07-26T15:25:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:36 smithi107 ceph-mon[32659]: pgmap v5383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:36 smithi132 ceph-mon[37580]: pgmap v5383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:36 smithi112 ceph-mon[37501]: pgmap v5383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:38 smithi107 ceph-mon[32659]: pgmap v5384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:39.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:38 smithi132 ceph-mon[37580]: pgmap v5384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:38 smithi112 ceph-mon[37501]: pgmap v5384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:40 smithi107 ceph-mon[32659]: pgmap v5385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9276.954949 secs 2024-07-26T15:25:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:40 smithi132 ceph-mon[37580]: pgmap v5385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9276.954949 secs 2024-07-26T15:25:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:40 smithi112 ceph-mon[37501]: pgmap v5385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9276.954949 secs 2024-07-26T15:25:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:43 smithi107 ceph-mon[32659]: pgmap v5386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:43.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:43 smithi132 ceph-mon[37580]: pgmap v5386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:43 smithi112 ceph-mon[37501]: pgmap v5386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:44.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:44.604+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:44.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:44.604+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:44.606 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:44.604+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:45 smithi107 ceph-mon[32659]: pgmap v5387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:45 smithi132 ceph-mon[37580]: pgmap v5387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:45 smithi112 ceph-mon[37501]: pgmap v5387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9281.955342 secs 2024-07-26T15:25:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9281.955342 secs 2024-07-26T15:25:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9281.955342 secs 2024-07-26T15:25:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:46.890+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:46.890+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:25:46.890+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:47 smithi107 ceph-mon[32659]: pgmap v5388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:47 smithi132 ceph-mon[37580]: pgmap v5388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:47 smithi112 ceph-mon[37501]: pgmap v5388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:49 smithi107 ceph-mon[32659]: pgmap v5389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:49 smithi132 ceph-mon[37580]: pgmap v5389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:49 smithi112 ceph-mon[37501]: pgmap v5389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:51 smithi107 ceph-mon[32659]: pgmap v5390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9286.955404 secs 2024-07-26T15:25:51.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:51 smithi132 ceph-mon[37580]: pgmap v5390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9286.955404 secs 2024-07-26T15:25:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:51 smithi112 ceph-mon[37501]: pgmap v5390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9286.955404 secs 2024-07-26T15:25:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:53 smithi107 ceph-mon[32659]: pgmap v5391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:53.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:53 smithi132 ceph-mon[37580]: pgmap v5391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:53 smithi112 ceph-mon[37501]: pgmap v5391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:25:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:55 smithi107 ceph-mon[32659]: pgmap v5392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:55 smithi132 ceph-mon[37580]: pgmap v5392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:55 smithi112 ceph-mon[37501]: pgmap v5392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9291.955470 secs 2024-07-26T15:25:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9291.955470 secs 2024-07-26T15:25:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9291.955470 secs 2024-07-26T15:25:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:57 smithi132 ceph-mon[37580]: pgmap v5393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:57 smithi112 ceph-mon[37501]: pgmap v5393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:57 smithi107 ceph-mon[32659]: pgmap v5393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:25:59 smithi112 ceph-mon[37501]: pgmap v5394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:25:59 smithi107 ceph-mon[32659]: pgmap v5394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:25:59 smithi132 ceph-mon[37580]: pgmap v5394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:25:59.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:59.620+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:25:59.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:59.620+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:59.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:25:59.620+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:01 smithi112 ceph-mon[37501]: pgmap v5395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9296.955556 secs 2024-07-26T15:26:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:01 smithi107 ceph-mon[32659]: pgmap v5395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9296.955556 secs 2024-07-26T15:26:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:01 smithi132 ceph-mon[37580]: pgmap v5395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9296.955556 secs 2024-07-26T15:26:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:01.906+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:01.906+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:01.906+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:03 smithi112 ceph-mon[37501]: pgmap v5396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:03 smithi107 ceph-mon[32659]: pgmap v5396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:03.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:03 smithi132 ceph-mon[37580]: pgmap v5396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:05 smithi112 ceph-mon[37501]: pgmap v5397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:05 smithi107 ceph-mon[32659]: pgmap v5397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:05 smithi132 ceph-mon[37580]: pgmap v5397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9301.955676 secs 2024-07-26T15:26:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9301.955676 secs 2024-07-26T15:26:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9301.955676 secs 2024-07-26T15:26:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:07 smithi112 ceph-mon[37501]: pgmap v5398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:07 smithi107 ceph-mon[32659]: pgmap v5398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:07 smithi132 ceph-mon[37580]: pgmap v5398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:09.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:09 smithi112 ceph-mon[37501]: pgmap v5399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:09 smithi107 ceph-mon[32659]: pgmap v5399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:09 smithi132 ceph-mon[37580]: pgmap v5399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:11 smithi107 ceph-mon[32659]: pgmap v5400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9306.955790 secs 2024-07-26T15:26:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:11 smithi112 ceph-mon[37501]: pgmap v5400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9306.955790 secs 2024-07-26T15:26:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:11 smithi132 ceph-mon[37580]: pgmap v5400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9306.955790 secs 2024-07-26T15:26:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: pgmap v5401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:26:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: pgmap v5401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:26:13.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:26:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: pgmap v5401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:26:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:26:13.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:26:13.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:26:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:14 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:26:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:14 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:26:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:14 smithi112 ceph-mon[37501]: pgmap v5402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:26:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:14 smithi112 ceph-mon[37501]: pgmap v5403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:26:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:14 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:26:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:14 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:26:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:14 smithi107 ceph-mon[32659]: pgmap v5402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:26:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:14 smithi107 ceph-mon[32659]: pgmap v5403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:26:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:14 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:26:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:14 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:26:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:14 smithi132 ceph-mon[37580]: pgmap v5402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:26:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:14 smithi132 ceph-mon[37580]: pgmap v5403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:26:14.624 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:14.623+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:14.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:14.623+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:14.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:14.623+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:16 smithi112 ceph-mon[37501]: pgmap v5404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9311.955879 secs 2024-07-26T15:26:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:16 smithi107 ceph-mon[32659]: pgmap v5404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9311.955879 secs 2024-07-26T15:26:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:16 smithi132 ceph-mon[37580]: pgmap v5404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9311.955879 secs 2024-07-26T15:26:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:16.912+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:16.912+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:16.912+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:18 smithi112 ceph-mon[37501]: pgmap v5405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:18 smithi107 ceph-mon[32659]: pgmap v5405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:18.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:18 smithi132 ceph-mon[37580]: pgmap v5405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:20 smithi112 ceph-mon[37501]: pgmap v5406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:26:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:20 smithi107 ceph-mon[32659]: pgmap v5406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:26:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:20 smithi132 ceph-mon[37580]: pgmap v5406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:26:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9316.956021 secs 2024-07-26T15:26:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9316.956021 secs 2024-07-26T15:26:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9316.956021 secs 2024-07-26T15:26:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:22 smithi112 ceph-mon[37501]: pgmap v5407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:26:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:22 smithi107 ceph-mon[32659]: pgmap v5407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:26:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:22 smithi132 ceph-mon[37580]: pgmap v5407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:26:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:24 smithi112 ceph-mon[37501]: pgmap v5408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:26:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:24 smithi107 ceph-mon[32659]: pgmap v5408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:26:24.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:24 smithi132 ceph-mon[37580]: pgmap v5408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:26:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:26 smithi112 ceph-mon[37501]: pgmap v5409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9321.956155 secs 2024-07-26T15:26:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:26 smithi107 ceph-mon[32659]: pgmap v5409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9321.956155 secs 2024-07-26T15:26:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:26 smithi132 ceph-mon[37580]: pgmap v5409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9321.956155 secs 2024-07-26T15:26:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:28 smithi112 ceph-mon[37501]: pgmap v5410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:28 smithi107 ceph-mon[32659]: pgmap v5410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:28 smithi132 ceph-mon[37580]: pgmap v5410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:29.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:29.634+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:29.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:29.634+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:29.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:29.634+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:30 smithi112 ceph-mon[37501]: pgmap v5411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:30 smithi107 ceph-mon[32659]: pgmap v5411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:30 smithi132 ceph-mon[37580]: pgmap v5411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9326.956273 secs 2024-07-26T15:26:31.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9326.956273 secs 2024-07-26T15:26:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9326.956273 secs 2024-07-26T15:26:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:31.928+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:31.928+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:31.929 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:31.928+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:32 smithi107 ceph-mon[32659]: pgmap v5412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:32 smithi132 ceph-mon[37580]: pgmap v5412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:32 smithi112 ceph-mon[37501]: pgmap v5412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:34 smithi107 ceph-mon[32659]: pgmap v5413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:34.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:34 smithi132 ceph-mon[37580]: pgmap v5413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:34 smithi112 ceph-mon[37501]: pgmap v5413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:36 smithi107 ceph-mon[32659]: pgmap v5414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9331.956408 secs 2024-07-26T15:26:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:36 smithi132 ceph-mon[37580]: pgmap v5414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9331.956408 secs 2024-07-26T15:26:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:36 smithi112 ceph-mon[37501]: pgmap v5414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9331.956408 secs 2024-07-26T15:26:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:38 smithi107 ceph-mon[32659]: pgmap v5415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:38 smithi132 ceph-mon[37580]: pgmap v5415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:38 smithi112 ceph-mon[37501]: pgmap v5415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:40 smithi107 ceph-mon[32659]: pgmap v5416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:40 smithi132 ceph-mon[37580]: pgmap v5416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:40 smithi112 ceph-mon[37501]: pgmap v5416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9336.956492 secs 2024-07-26T15:26:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9336.956492 secs 2024-07-26T15:26:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9336.956492 secs 2024-07-26T15:26:42.443 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:42 smithi107 ceph-mon[32659]: pgmap v5417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:42 smithi132 ceph-mon[37580]: pgmap v5417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:42 smithi112 ceph-mon[37501]: pgmap v5417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:44 smithi107 ceph-mon[32659]: pgmap v5418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:44.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:44 smithi132 ceph-mon[37580]: pgmap v5418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:44.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:44.637+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:44.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:44.637+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:44.640 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:44.637+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:44 smithi112 ceph-mon[37501]: pgmap v5418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:46 smithi107 ceph-mon[32659]: pgmap v5419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9341.956639 secs 2024-07-26T15:26:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:46 smithi132 ceph-mon[37580]: pgmap v5419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9341.956639 secs 2024-07-26T15:26:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:46 smithi112 ceph-mon[37501]: pgmap v5419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9341.956639 secs 2024-07-26T15:26:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:46.936+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:46.936+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:26:46.936+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:48 smithi107 ceph-mon[32659]: pgmap v5420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:48 smithi132 ceph-mon[37580]: pgmap v5420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:48 smithi112 ceph-mon[37501]: pgmap v5420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:50 smithi107 ceph-mon[32659]: pgmap v5421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:50.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:50 smithi132 ceph-mon[37580]: pgmap v5421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:50 smithi112 ceph-mon[37501]: pgmap v5421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:26:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9346.956770 secs 2024-07-26T15:26:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9346.956770 secs 2024-07-26T15:26:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9346.956770 secs 2024-07-26T15:26:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:52 smithi107 ceph-mon[32659]: pgmap v5422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:52.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:52 smithi132 ceph-mon[37580]: pgmap v5422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:52 smithi112 ceph-mon[37501]: pgmap v5422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:54 smithi107 ceph-mon[32659]: pgmap v5423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:54.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:54 smithi132 ceph-mon[37580]: pgmap v5423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:54 smithi112 ceph-mon[37501]: pgmap v5423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:56 smithi107 ceph-mon[32659]: pgmap v5424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9351.956845 secs 2024-07-26T15:26:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:56 smithi132 ceph-mon[37580]: pgmap v5424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9351.956845 secs 2024-07-26T15:26:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:56 smithi112 ceph-mon[37501]: pgmap v5424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9351.956845 secs 2024-07-26T15:26:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:26:58 smithi132 ceph-mon[37580]: pgmap v5425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:26:58 smithi112 ceph-mon[37501]: pgmap v5425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:26:58 smithi107 ceph-mon[32659]: pgmap v5425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:26:59.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:59.648+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:26:59.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:59.648+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:59.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:26:59.648+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:00 smithi132 ceph-mon[37580]: pgmap v5426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:00 smithi112 ceph-mon[37501]: pgmap v5426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:00 smithi107 ceph-mon[32659]: pgmap v5426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:01.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9356.956964 secs 2024-07-26T15:27:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9356.956964 secs 2024-07-26T15:27:01.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9356.956964 secs 2024-07-26T15:27:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:01.952+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:01.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:01.952+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:01.954 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:01.952+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:02 smithi112 ceph-mon[37501]: pgmap v5427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:02 smithi107 ceph-mon[32659]: pgmap v5427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:02.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:02 smithi132 ceph-mon[37580]: pgmap v5427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:04 smithi112 ceph-mon[37501]: pgmap v5428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:04 smithi107 ceph-mon[32659]: pgmap v5428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:04 smithi132 ceph-mon[37580]: pgmap v5428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:06 smithi112 ceph-mon[37501]: pgmap v5429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9361.957075 secs 2024-07-26T15:27:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:06 smithi107 ceph-mon[32659]: pgmap v5429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9361.957075 secs 2024-07-26T15:27:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:06 smithi132 ceph-mon[37580]: pgmap v5429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9361.957075 secs 2024-07-26T15:27:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:08 smithi112 ceph-mon[37501]: pgmap v5430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:08 smithi107 ceph-mon[32659]: pgmap v5430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:08.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:08 smithi132 ceph-mon[37580]: pgmap v5430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:10 smithi112 ceph-mon[37501]: pgmap v5431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:10 smithi107 ceph-mon[32659]: pgmap v5431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:10 smithi132 ceph-mon[37580]: pgmap v5431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9366.957189 secs 2024-07-26T15:27:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9366.957189 secs 2024-07-26T15:27:11.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9366.957189 secs 2024-07-26T15:27:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:12 smithi112 ceph-mon[37501]: pgmap v5432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:12 smithi107 ceph-mon[32659]: pgmap v5432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:12 smithi132 ceph-mon[37580]: pgmap v5432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:27:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:27:13.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:27:14.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:14.227+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:27:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:14.227+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:27:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:14.227+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:27:14.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:14.227+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:27:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: pgmap v5433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:27:14.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:27:14.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:27:14.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:27:14.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:27:14.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:27:14.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:14.650+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:14.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:14.650+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:14.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:14.650+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: pgmap v5433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:27:14.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:27:14.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:27:14.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:27:14.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:27:14.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:27:14.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:27:14.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: pgmap v5433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:14.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:27:14.713 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:27:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:15 smithi112 ceph-mon[37501]: pgmap v5434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:27:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:15 smithi112 ceph-mon[37501]: pgmap v5435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:27:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:15 smithi107 ceph-mon[32659]: pgmap v5434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:27:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:15 smithi107 ceph-mon[32659]: pgmap v5435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:27:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:15 smithi132 ceph-mon[37580]: pgmap v5434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:27:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:15 smithi132 ceph-mon[37580]: pgmap v5435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:27:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:16.417+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:27:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:16.417+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:27:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:16.418+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:27:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:16.419+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:27:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9371.957296 secs 2024-07-26T15:27:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9371.957296 secs 2024-07-26T15:27:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9371.957296 secs 2024-07-26T15:27:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:16.964+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:16.964+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:16.964+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:17 smithi112 ceph-mon[37501]: pgmap v5436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:27:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:17 smithi107 ceph-mon[32659]: pgmap v5436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:27:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:17 smithi132 ceph-mon[37580]: pgmap v5436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:27:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:19 smithi112 ceph-mon[37501]: pgmap v5437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:27:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:19 smithi107 ceph-mon[32659]: pgmap v5437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:27:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:19 smithi132 ceph-mon[37580]: pgmap v5437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:27:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9376.957410 secs 2024-07-26T15:27:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:21 smithi112 ceph-mon[37501]: pgmap v5438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:27:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9376.957410 secs 2024-07-26T15:27:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:21 smithi107 ceph-mon[32659]: pgmap v5438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:27:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9376.957410 secs 2024-07-26T15:27:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:21 smithi132 ceph-mon[37580]: pgmap v5438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:27:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:23 smithi112 ceph-mon[37501]: pgmap v5439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:27:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:23 smithi107 ceph-mon[32659]: pgmap v5439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:27:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:23 smithi132 ceph-mon[37580]: pgmap v5439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:27:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:25 smithi112 ceph-mon[37501]: pgmap v5440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:27:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:25 smithi107 ceph-mon[32659]: pgmap v5440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:27:25.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:25 smithi132 ceph-mon[37580]: pgmap v5440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:27:26.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9381.957529 secs 2024-07-26T15:27:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9381.957529 secs 2024-07-26T15:27:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9381.957529 secs 2024-07-26T15:27:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:27 smithi112 ceph-mon[37501]: pgmap v5441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:27 smithi107 ceph-mon[32659]: pgmap v5441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:27 smithi132 ceph-mon[37580]: pgmap v5441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:29.666+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:29.666+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:29.667 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:29.666+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:29 smithi112 ceph-mon[37501]: pgmap v5442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:29 smithi107 ceph-mon[32659]: pgmap v5442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:29 smithi132 ceph-mon[37580]: pgmap v5442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9386.957648 secs 2024-07-26T15:27:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:31 smithi112 ceph-mon[37501]: pgmap v5443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9386.957648 secs 2024-07-26T15:27:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:31 smithi107 ceph-mon[32659]: pgmap v5443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9386.957648 secs 2024-07-26T15:27:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:31 smithi132 ceph-mon[37580]: pgmap v5443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:31.968+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:31.968+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:31.968+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:33 smithi112 ceph-mon[37501]: pgmap v5444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:33 smithi107 ceph-mon[32659]: pgmap v5444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:33 smithi132 ceph-mon[37580]: pgmap v5444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:35 smithi107 ceph-mon[32659]: pgmap v5445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:35 smithi132 ceph-mon[37580]: pgmap v5445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:35 smithi112 ceph-mon[37501]: pgmap v5445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9391.957783 secs 2024-07-26T15:27:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9391.957783 secs 2024-07-26T15:27:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9391.957783 secs 2024-07-26T15:27:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:37 smithi107 ceph-mon[32659]: pgmap v5446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:37 smithi132 ceph-mon[37580]: pgmap v5446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:37 smithi112 ceph-mon[37501]: pgmap v5446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:39 smithi107 ceph-mon[32659]: pgmap v5447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:39 smithi132 ceph-mon[37580]: pgmap v5447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:39 smithi112 ceph-mon[37501]: pgmap v5447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9396.957885 secs 2024-07-26T15:27:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:41 smithi107 ceph-mon[32659]: pgmap v5448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9396.957885 secs 2024-07-26T15:27:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:41 smithi132 ceph-mon[37580]: pgmap v5448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9396.957885 secs 2024-07-26T15:27:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:41 smithi112 ceph-mon[37501]: pgmap v5448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:43 smithi107 ceph-mon[32659]: pgmap v5449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:43 smithi132 ceph-mon[37580]: pgmap v5449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:43 smithi112 ceph-mon[37501]: pgmap v5449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:44.681+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:44.681+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:44.681+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:45 smithi107 ceph-mon[32659]: pgmap v5450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:45 smithi132 ceph-mon[37580]: pgmap v5450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:45 smithi112 ceph-mon[37501]: pgmap v5450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9401.958031 secs 2024-07-26T15:27:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9401.958031 secs 2024-07-26T15:27:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9401.958031 secs 2024-07-26T15:27:46.984 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:46.983+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:46.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:46.983+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:46.985 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:27:46.983+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:47 smithi107 ceph-mon[32659]: pgmap v5451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:47 smithi132 ceph-mon[37580]: pgmap v5451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:47 smithi112 ceph-mon[37501]: pgmap v5451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:49 smithi107 ceph-mon[32659]: pgmap v5452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:49 smithi132 ceph-mon[37580]: pgmap v5452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:49 smithi112 ceph-mon[37501]: pgmap v5452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9406.958151 secs 2024-07-26T15:27:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:51 smithi107 ceph-mon[32659]: pgmap v5453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9406.958151 secs 2024-07-26T15:27:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:51 smithi132 ceph-mon[37580]: pgmap v5453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9406.958151 secs 2024-07-26T15:27:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:51 smithi112 ceph-mon[37501]: pgmap v5453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:53 smithi107 ceph-mon[32659]: pgmap v5454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:53 smithi132 ceph-mon[37580]: pgmap v5454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:53 smithi112 ceph-mon[37501]: pgmap v5454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:27:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:55 smithi107 ceph-mon[32659]: pgmap v5455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:55 smithi132 ceph-mon[37580]: pgmap v5455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:55 smithi112 ceph-mon[37501]: pgmap v5455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:27:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9411.958277 secs 2024-07-26T15:27:56.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9411.958277 secs 2024-07-26T15:27:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9411.958277 secs 2024-07-26T15:27:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:57 smithi107 ceph-mon[32659]: pgmap v5456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:57 smithi132 ceph-mon[37580]: pgmap v5456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:57 smithi112 ceph-mon[37501]: pgmap v5456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:58.234+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:27:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:58.234+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:27:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:58.235+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:27:58.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:58.235+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:27:59.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:59.687+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:59.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:59.687+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:59.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:27:59.687+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:27:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:27:59 smithi107 ceph-mon[32659]: pgmap v5457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:27:59 smithi132 ceph-mon[37580]: pgmap v5457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:27:59 smithi112 ceph-mon[37501]: pgmap v5457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:00.425+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:28:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:00.425+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:28:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:00.426+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:28:00.427 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:00.426+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:28:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9416.958366 secs 2024-07-26T15:28:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:01 smithi107 ceph-mon[32659]: pgmap v5458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9416.958366 secs 2024-07-26T15:28:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:01 smithi132 ceph-mon[37580]: pgmap v5458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9416.958366 secs 2024-07-26T15:28:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:01 smithi112 ceph-mon[37501]: pgmap v5458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:01.998+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:01.998+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:01.998+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:03 smithi132 ceph-mon[37580]: pgmap v5459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:03 smithi112 ceph-mon[37501]: pgmap v5459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:03 smithi107 ceph-mon[32659]: pgmap v5459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:05 smithi132 ceph-mon[37580]: pgmap v5460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9421.958489 secs 2024-07-26T15:28:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:05 smithi112 ceph-mon[37501]: pgmap v5460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9421.958489 secs 2024-07-26T15:28:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:05 smithi107 ceph-mon[32659]: pgmap v5460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9421.958489 secs 2024-07-26T15:28:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:07 smithi132 ceph-mon[37580]: pgmap v5461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:07 smithi112 ceph-mon[37501]: pgmap v5461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:07 smithi107 ceph-mon[32659]: pgmap v5461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:09 smithi112 ceph-mon[37501]: pgmap v5462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:09 smithi107 ceph-mon[32659]: pgmap v5462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:09 smithi132 ceph-mon[37580]: pgmap v5462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9426.958572 secs 2024-07-26T15:28:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9426.958572 secs 2024-07-26T15:28:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9426.958572 secs 2024-07-26T15:28:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:11 smithi112 ceph-mon[37501]: pgmap v5463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:11 smithi107 ceph-mon[32659]: pgmap v5463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:11 smithi132 ceph-mon[37580]: pgmap v5463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:13 smithi112 ceph-mon[37501]: pgmap v5464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:13 smithi107 ceph-mon[32659]: pgmap v5464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:14.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:13 smithi132 ceph-mon[37580]: pgmap v5464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:14.688+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:14.688+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:14.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:14.688+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:14.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:28:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:28:15.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:28:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:15 smithi112 ceph-mon[37501]: pgmap v5465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9431.958631 secs 2024-07-26T15:28:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:28:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:28:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:28:15.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:15 smithi132 ceph-mon[37580]: pgmap v5465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:15.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9431.958631 secs 2024-07-26T15:28:15.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:28:15.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:28:15.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:28:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:15 smithi107 ceph-mon[32659]: pgmap v5465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9431.958631 secs 2024-07-26T15:28:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:28:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:28:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:28:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:16 smithi112 ceph-mon[37501]: pgmap v5466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:28:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:16 smithi112 ceph-mon[37501]: pgmap v5467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:17.015+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:17.015+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:17.015+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:16 smithi107 ceph-mon[32659]: pgmap v5466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:28:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:16 smithi107 ceph-mon[32659]: pgmap v5467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:16 smithi132 ceph-mon[37580]: pgmap v5466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:28:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:16 smithi132 ceph-mon[37580]: pgmap v5467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:18 smithi112 ceph-mon[37501]: pgmap v5468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:28:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:18 smithi107 ceph-mon[32659]: pgmap v5468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:28:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:18 smithi132 ceph-mon[37580]: pgmap v5468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:28:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:20 smithi112 ceph-mon[37501]: pgmap v5469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9436.958733 secs 2024-07-26T15:28:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:20 smithi107 ceph-mon[32659]: pgmap v5469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9436.958733 secs 2024-07-26T15:28:21.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:20 smithi132 ceph-mon[37580]: pgmap v5469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9436.958733 secs 2024-07-26T15:28:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:22 smithi112 ceph-mon[37501]: pgmap v5470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:22 smithi107 ceph-mon[32659]: pgmap v5470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:22 smithi132 ceph-mon[37580]: pgmap v5470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:24 smithi112 ceph-mon[37501]: pgmap v5471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:24 smithi107 ceph-mon[32659]: pgmap v5471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:24 smithi132 ceph-mon[37580]: pgmap v5471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:25.956 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9441.958857 secs 2024-07-26T15:28:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9441.958857 secs 2024-07-26T15:28:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9441.958857 secs 2024-07-26T15:28:26.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:26 smithi107 ceph-mon[32659]: pgmap v5472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:28:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:26 smithi112 ceph-mon[37501]: pgmap v5472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:28:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:26 smithi132 ceph-mon[37580]: pgmap v5472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:28:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:28 smithi112 ceph-mon[37501]: pgmap v5473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:28 smithi107 ceph-mon[32659]: pgmap v5473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:29.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:28 smithi132 ceph-mon[37580]: pgmap v5473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:29.697+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:29.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:29.697+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:29.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:29.697+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:30 smithi112 ceph-mon[37501]: pgmap v5474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:28:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9446.958939 secs 2024-07-26T15:28:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:30 smithi107 ceph-mon[32659]: pgmap v5474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:28:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9446.958939 secs 2024-07-26T15:28:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:30 smithi132 ceph-mon[37580]: pgmap v5474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:28:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9446.958939 secs 2024-07-26T15:28:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:32.030+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:32.030+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:32.031 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:32.030+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:32 smithi112 ceph-mon[37501]: pgmap v5475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:32 smithi107 ceph-mon[32659]: pgmap v5475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:33.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:32 smithi132 ceph-mon[37580]: pgmap v5475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:34 smithi112 ceph-mon[37501]: pgmap v5476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:34 smithi107 ceph-mon[32659]: pgmap v5476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:34 smithi132 ceph-mon[37580]: pgmap v5476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9451.959048 secs 2024-07-26T15:28:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9451.959048 secs 2024-07-26T15:28:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9451.959048 secs 2024-07-26T15:28:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:36 smithi112 ceph-mon[37501]: pgmap v5477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:36 smithi107 ceph-mon[32659]: pgmap v5477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:36 smithi132 ceph-mon[37580]: pgmap v5477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:38 smithi112 ceph-mon[37501]: pgmap v5478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:38 smithi107 ceph-mon[32659]: pgmap v5478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:38 smithi132 ceph-mon[37580]: pgmap v5478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:40 smithi112 ceph-mon[37501]: pgmap v5479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:28:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9456.959168 secs 2024-07-26T15:28:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:40 smithi107 ceph-mon[32659]: pgmap v5479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:28:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9456.959168 secs 2024-07-26T15:28:41.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:40 smithi132 ceph-mon[37580]: pgmap v5479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:28:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9456.959168 secs 2024-07-26T15:28:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:42 smithi112 ceph-mon[37501]: pgmap v5480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:42 smithi107 ceph-mon[32659]: pgmap v5480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:42 smithi132 ceph-mon[37580]: pgmap v5480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:44.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:44.704+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:44.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:44.705+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:44.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:44.705+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:44 smithi112 ceph-mon[37501]: pgmap v5481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:44 smithi107 ceph-mon[32659]: pgmap v5481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:44 smithi132 ceph-mon[37580]: pgmap v5481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:45.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9461.959311 secs 2024-07-26T15:28:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9461.959311 secs 2024-07-26T15:28:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9461.959311 secs 2024-07-26T15:28:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:46 smithi107 ceph-mon[32659]: pgmap v5482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:47.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:47.044+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:47.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:47.044+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:47.046 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:28:47.044+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:46 smithi132 ceph-mon[37580]: pgmap v5482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:46 smithi112 ceph-mon[37501]: pgmap v5482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:48 smithi107 ceph-mon[32659]: pgmap v5483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:49.065 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:48 smithi132 ceph-mon[37580]: pgmap v5483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:48 smithi112 ceph-mon[37501]: pgmap v5483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:50 smithi107 ceph-mon[32659]: pgmap v5484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:28:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9466.959433 secs 2024-07-26T15:28:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:50 smithi132 ceph-mon[37580]: pgmap v5484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:28:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9466.959433 secs 2024-07-26T15:28:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:50 smithi112 ceph-mon[37501]: pgmap v5484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:28:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9466.959433 secs 2024-07-26T15:28:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:52 smithi107 ceph-mon[32659]: pgmap v5485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:52 smithi132 ceph-mon[37580]: pgmap v5485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:52 smithi112 ceph-mon[37501]: pgmap v5485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:54 smithi107 ceph-mon[32659]: pgmap v5486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:54 smithi132 ceph-mon[37580]: pgmap v5486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:54 smithi112 ceph-mon[37501]: pgmap v5486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9471.959551 secs 2024-07-26T15:28:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9471.959551 secs 2024-07-26T15:28:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9471.959551 secs 2024-07-26T15:28:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:56 smithi107 ceph-mon[32659]: pgmap v5487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:56 smithi132 ceph-mon[37580]: pgmap v5487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:56 smithi112 ceph-mon[37501]: pgmap v5487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:28:58 smithi107 ceph-mon[32659]: pgmap v5488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:28:58 smithi132 ceph-mon[37580]: pgmap v5488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:28:58 smithi112 ceph-mon[37501]: pgmap v5488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:59.720+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:59.720+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:28:59.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:28:59.720+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:00 smithi107 ceph-mon[32659]: pgmap v5489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:29:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9476.959672 secs 2024-07-26T15:29:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:00 smithi132 ceph-mon[37580]: pgmap v5489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:29:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9476.959672 secs 2024-07-26T15:29:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:00 smithi112 ceph-mon[37501]: pgmap v5489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:29:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9476.959672 secs 2024-07-26T15:29:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:02.045+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:02.045+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:02.045+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:02 smithi107 ceph-mon[32659]: pgmap v5490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:02 smithi112 ceph-mon[37501]: pgmap v5490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:02 smithi132 ceph-mon[37580]: pgmap v5490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:04 smithi112 ceph-mon[37501]: pgmap v5491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:04 smithi107 ceph-mon[32659]: pgmap v5491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:05.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:04 smithi132 ceph-mon[37580]: pgmap v5491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9481.959772 secs 2024-07-26T15:29:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9481.959772 secs 2024-07-26T15:29:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9481.959772 secs 2024-07-26T15:29:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:06 smithi112 ceph-mon[37501]: pgmap v5492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:06 smithi107 ceph-mon[32659]: pgmap v5492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:06 smithi132 ceph-mon[37580]: pgmap v5492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:08 smithi112 ceph-mon[37501]: pgmap v5493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:08 smithi107 ceph-mon[32659]: pgmap v5493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:08 smithi132 ceph-mon[37580]: pgmap v5493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:10 smithi112 ceph-mon[37501]: pgmap v5494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:29:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9486.959896 secs 2024-07-26T15:29:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:10 smithi107 ceph-mon[32659]: pgmap v5494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:29:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9486.959896 secs 2024-07-26T15:29:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:10 smithi132 ceph-mon[37580]: pgmap v5494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:29:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9486.959896 secs 2024-07-26T15:29:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:12 smithi112 ceph-mon[37501]: pgmap v5495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:12 smithi107 ceph-mon[32659]: pgmap v5495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:12 smithi132 ceph-mon[37580]: pgmap v5495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:14.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:14.727+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:14.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:14.727+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:14.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:14.727+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:14 smithi112 ceph-mon[37501]: pgmap v5496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:29:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:14 smithi107 ceph-mon[32659]: pgmap v5496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:29:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:14 smithi132 ceph-mon[37580]: pgmap v5496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:29:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9491.960021 secs 2024-07-26T15:29:15.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9491.960021 secs 2024-07-26T15:29:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9491.960021 secs 2024-07-26T15:29:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:16 smithi112 ceph-mon[37501]: pgmap v5497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:29:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:29:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:29:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:29:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:16 smithi107 ceph-mon[32659]: pgmap v5497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:29:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:29:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:29:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:29:17.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:16 smithi132 ceph-mon[37580]: pgmap v5497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:29:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:29:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:29:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:29:17.061 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:17.061+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:17.061+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:17.061+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:29:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:29:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:29:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:29:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:29:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:29:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:29:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:29:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:29:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:18 smithi112 ceph-mon[37501]: pgmap v5498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:18 smithi107 ceph-mon[32659]: pgmap v5498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:18 smithi132 ceph-mon[37580]: pgmap v5498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:20 smithi112 ceph-mon[37501]: pgmap v5499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9496.960116 secs 2024-07-26T15:29:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:20 smithi107 ceph-mon[32659]: pgmap v5499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9496.960116 secs 2024-07-26T15:29:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:20 smithi132 ceph-mon[37580]: pgmap v5499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9496.960116 secs 2024-07-26T15:29:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:22 smithi112 ceph-mon[37501]: pgmap v5500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:29:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:22 smithi107 ceph-mon[32659]: pgmap v5500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:29:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:22 smithi132 ceph-mon[37580]: pgmap v5500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:29:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:24 smithi112 ceph-mon[37501]: pgmap v5501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:24 smithi107 ceph-mon[32659]: pgmap v5501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:24 smithi132 ceph-mon[37580]: pgmap v5501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9501.960260 secs 2024-07-26T15:29:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9501.960260 secs 2024-07-26T15:29:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9501.960260 secs 2024-07-26T15:29:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:26 smithi107 ceph-mon[32659]: pgmap v5502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:29:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:26 smithi132 ceph-mon[37580]: pgmap v5502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:29:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:26 smithi112 ceph-mon[37501]: pgmap v5502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:29:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:28 smithi107 ceph-mon[32659]: pgmap v5503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:29:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:28 smithi132 ceph-mon[37580]: pgmap v5503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:29:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:28 smithi112 ceph-mon[37501]: pgmap v5503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:29:29.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:29.742+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:29.742+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:29.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:29.742+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:30 smithi107 ceph-mon[32659]: pgmap v5504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9506.960382 secs 2024-07-26T15:29:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:30 smithi132 ceph-mon[37580]: pgmap v5504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9506.960382 secs 2024-07-26T15:29:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:30 smithi112 ceph-mon[37501]: pgmap v5504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9506.960382 secs 2024-07-26T15:29:32.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:32.076+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:32.077+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:32.077+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:33.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:32 smithi112 ceph-mon[37501]: pgmap v5505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:32 smithi107 ceph-mon[32659]: pgmap v5505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:32 smithi132 ceph-mon[37580]: pgmap v5505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:34 smithi107 ceph-mon[32659]: pgmap v5506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:34 smithi132 ceph-mon[37580]: pgmap v5506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:34 smithi112 ceph-mon[37501]: pgmap v5506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9511.960514 secs 2024-07-26T15:29:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9511.960514 secs 2024-07-26T15:29:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9511.960514 secs 2024-07-26T15:29:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:36 smithi107 ceph-mon[32659]: pgmap v5507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:37.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:36 smithi132 ceph-mon[37580]: pgmap v5507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:36 smithi112 ceph-mon[37501]: pgmap v5507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:38 smithi107 ceph-mon[32659]: pgmap v5508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:38 smithi132 ceph-mon[37580]: pgmap v5508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:38 smithi112 ceph-mon[37501]: pgmap v5508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:40 smithi107 ceph-mon[32659]: pgmap v5509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9516.960632 secs 2024-07-26T15:29:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:40 smithi132 ceph-mon[37580]: pgmap v5509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9516.960632 secs 2024-07-26T15:29:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:40 smithi112 ceph-mon[37501]: pgmap v5509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9516.960632 secs 2024-07-26T15:29:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:42 smithi107 ceph-mon[32659]: pgmap v5510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:29:43.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:42 smithi132 ceph-mon[37580]: pgmap v5510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:29:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:42 smithi112 ceph-mon[37501]: pgmap v5510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:29:44.749 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:44.747+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:44.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:44.747+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:44.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:44.747+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:44 smithi107 ceph-mon[32659]: pgmap v5511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:29:45.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:44 smithi132 ceph-mon[37580]: pgmap v5511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:29:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:44 smithi112 ceph-mon[37501]: pgmap v5511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:29:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9521.960758 secs 2024-07-26T15:29:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9521.960758 secs 2024-07-26T15:29:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9521.960758 secs 2024-07-26T15:29:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:46 smithi107 ceph-mon[32659]: pgmap v5512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:29:47.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:46 smithi132 ceph-mon[37580]: pgmap v5512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:47.091+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:47.091+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:29:47.091+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:46 smithi112 ceph-mon[37501]: pgmap v5512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:29:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:48 smithi132 ceph-mon[37580]: pgmap v5513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:48 smithi112 ceph-mon[37501]: pgmap v5513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:48 smithi107 ceph-mon[32659]: pgmap v5513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:50 smithi132 ceph-mon[37580]: pgmap v5514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9526.960886 secs 2024-07-26T15:29:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:50 smithi112 ceph-mon[37501]: pgmap v5514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9526.960886 secs 2024-07-26T15:29:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:50 smithi107 ceph-mon[32659]: pgmap v5514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9526.960886 secs 2024-07-26T15:29:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:52 smithi132 ceph-mon[37580]: pgmap v5515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:53.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:52 smithi112 ceph-mon[37501]: pgmap v5515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:52 smithi107 ceph-mon[32659]: pgmap v5515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:54 smithi112 ceph-mon[37501]: pgmap v5516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:54 smithi107 ceph-mon[32659]: pgmap v5516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:54 smithi132 ceph-mon[37580]: pgmap v5516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:56.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9531.961022 secs 2024-07-26T15:29:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9531.961022 secs 2024-07-26T15:29:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9531.961022 secs 2024-07-26T15:29:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:56 smithi112 ceph-mon[37501]: pgmap v5517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:56 smithi107 ceph-mon[32659]: pgmap v5517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:56 smithi132 ceph-mon[37580]: pgmap v5517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:29:58 smithi112 ceph-mon[37501]: pgmap v5518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:29:58 smithi107 ceph-mon[32659]: pgmap v5518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:59.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:29:58 smithi132 ceph-mon[37580]: pgmap v5518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:29:59.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:59.763+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:59.765 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:59.763+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:29:59.766 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:29:59.763+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:00 smithi112 ceph-mon[37501]: pgmap v5519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:30:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9536.961156 secs 2024-07-26T15:30:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:00 smithi107 ceph-mon[32659]: pgmap v5519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:30:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9536.961156 secs 2024-07-26T15:30:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:00 smithi132 ceph-mon[37580]: pgmap v5519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:30:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9536.961156 secs 2024-07-26T15:30:02.109 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:02.107+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:02.109 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:02.107+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:02.109 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:02.107+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:02 smithi112 ceph-mon[37501]: pgmap v5520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:30:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:02 smithi107 ceph-mon[32659]: pgmap v5520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:30:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:02 smithi132 ceph-mon[37580]: pgmap v5520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:30:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:04 smithi112 ceph-mon[37501]: pgmap v5521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:30:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:04 smithi107 ceph-mon[32659]: pgmap v5521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:30:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:04 smithi132 ceph-mon[37580]: pgmap v5521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:30:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9541.961287 secs 2024-07-26T15:30:06.210 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9541.961287 secs 2024-07-26T15:30:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9541.961287 secs 2024-07-26T15:30:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:06 smithi112 ceph-mon[37501]: pgmap v5522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:30:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:06 smithi107 ceph-mon[32659]: pgmap v5522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:30:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:06 smithi132 ceph-mon[37580]: pgmap v5522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:30:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:08 smithi112 ceph-mon[37501]: pgmap v5523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:30:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:08 smithi107 ceph-mon[32659]: pgmap v5523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:30:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:08 smithi132 ceph-mon[37580]: pgmap v5523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:30:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:10 smithi112 ceph-mon[37501]: pgmap v5524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9546.961373 secs 2024-07-26T15:30:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:10 smithi107 ceph-mon[32659]: pgmap v5524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9546.961373 secs 2024-07-26T15:30:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:10 smithi132 ceph-mon[37580]: pgmap v5524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9546.961373 secs 2024-07-26T15:30:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:12 smithi112 ceph-mon[37501]: pgmap v5525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:12 smithi107 ceph-mon[32659]: pgmap v5525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:12 smithi132 ceph-mon[37580]: pgmap v5525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:14.259+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:30:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:14.259+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:30:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:14.260+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:30:14.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:14.260+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:30:14.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:14.779+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:14.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:14.780+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:14.781 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:14.780+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:14 smithi107 ceph-mon[32659]: pgmap v5526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:14 smithi112 ceph-mon[37501]: pgmap v5526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:14 smithi132 ceph-mon[37580]: pgmap v5526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9551.961503 secs 2024-07-26T15:30:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9551.961503 secs 2024-07-26T15:30:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9551.961503 secs 2024-07-26T15:30:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:16.450+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:30:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:16.450+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:30:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:16.451+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:30:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:16.451+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:30:17.124 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:17.123+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:17.124+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:17.125 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:17.124+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:16 smithi112 ceph-mon[37501]: pgmap v5527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:16 smithi107 ceph-mon[32659]: pgmap v5527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:16 smithi132 ceph-mon[37580]: pgmap v5527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:30:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:30:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:30:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:18 smithi112 ceph-mon[37501]: pgmap v5528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:19.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:18 smithi107 ceph-mon[32659]: pgmap v5528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:19.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:19.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:19.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:18 smithi132 ceph-mon[37580]: pgmap v5528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:20 smithi112 ceph-mon[37501]: pgmap v5529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:20 smithi107 ceph-mon[32659]: pgmap v5529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:20 smithi132 ceph-mon[37580]: pgmap v5529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9556.961602 secs 2024-07-26T15:30:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9556.961602 secs 2024-07-26T15:30:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9556.961602 secs 2024-07-26T15:30:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:22 smithi112 ceph-mon[37501]: pgmap v5530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:30:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:30:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:30:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:22 smithi107 ceph-mon[32659]: pgmap v5530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:30:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:30:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:30:22.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:22 smithi132 ceph-mon[37580]: pgmap v5530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:30:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:30:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:30:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:30:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:23 smithi112 ceph-mon[37501]: pgmap v5531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:30:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:23 smithi112 ceph-mon[37501]: pgmap v5532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:30:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:23 smithi107 ceph-mon[32659]: pgmap v5531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:30:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:23 smithi107 ceph-mon[32659]: pgmap v5532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:30:23.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:23 smithi132 ceph-mon[37580]: pgmap v5531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:30:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:23 smithi132 ceph-mon[37580]: pgmap v5532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:30:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:25 smithi112 ceph-mon[37501]: pgmap v5533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:25 smithi107 ceph-mon[32659]: pgmap v5533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:25 smithi132 ceph-mon[37580]: pgmap v5533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:26.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9561.961746 secs 2024-07-26T15:30:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9561.961746 secs 2024-07-26T15:30:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9561.961746 secs 2024-07-26T15:30:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:27 smithi112 ceph-mon[37501]: pgmap v5534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:30:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:27 smithi107 ceph-mon[32659]: pgmap v5534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:30:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:27 smithi132 ceph-mon[37580]: pgmap v5534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:30:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:29 smithi112 ceph-mon[37501]: pgmap v5535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:29 smithi107 ceph-mon[32659]: pgmap v5535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:29.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:29.795+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:29.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:29.795+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:29.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:29.795+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:29 smithi132 ceph-mon[37580]: pgmap v5535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:31 smithi112 ceph-mon[37501]: pgmap v5536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:30:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9566.961886 secs 2024-07-26T15:30:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:31 smithi107 ceph-mon[32659]: pgmap v5536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:30:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9566.961886 secs 2024-07-26T15:30:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:31 smithi132 ceph-mon[37580]: pgmap v5536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:30:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9566.961886 secs 2024-07-26T15:30:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:32.138+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:32.139 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:32.138+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:32.140 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:32.138+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:33 smithi112 ceph-mon[37501]: pgmap v5537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:30:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:33 smithi107 ceph-mon[32659]: pgmap v5537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:30:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:33 smithi132 ceph-mon[37580]: pgmap v5537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:30:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:35 smithi112 ceph-mon[37501]: pgmap v5538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:35 smithi107 ceph-mon[32659]: pgmap v5538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:35 smithi132 ceph-mon[37580]: pgmap v5538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9571.962014 secs 2024-07-26T15:30:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9571.962014 secs 2024-07-26T15:30:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9571.962014 secs 2024-07-26T15:30:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:37 smithi112 ceph-mon[37501]: pgmap v5539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:37.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:37 smithi107 ceph-mon[32659]: pgmap v5539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:37 smithi132 ceph-mon[37580]: pgmap v5539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:39 smithi112 ceph-mon[37501]: pgmap v5540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:39 smithi107 ceph-mon[32659]: pgmap v5540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:39 smithi132 ceph-mon[37580]: pgmap v5540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:41 smithi112 ceph-mon[37501]: pgmap v5541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9576.962135 secs 2024-07-26T15:30:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:41 smithi107 ceph-mon[32659]: pgmap v5541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9576.962135 secs 2024-07-26T15:30:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:41 smithi132 ceph-mon[37580]: pgmap v5541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9576.962135 secs 2024-07-26T15:30:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:43 smithi112 ceph-mon[37501]: pgmap v5542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:43 smithi107 ceph-mon[32659]: pgmap v5542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:43 smithi132 ceph-mon[37580]: pgmap v5542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:44.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:44.810+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:44.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:44.810+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:44.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:44.810+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:45 smithi112 ceph-mon[37501]: pgmap v5543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:45 smithi107 ceph-mon[32659]: pgmap v5543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:45 smithi132 ceph-mon[37580]: pgmap v5543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9581.962202 secs 2024-07-26T15:30:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9581.962202 secs 2024-07-26T15:30:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9581.962202 secs 2024-07-26T15:30:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:47.149+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:47.149+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:30:47.149+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:47 smithi107 ceph-mon[32659]: pgmap v5544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:47 smithi132 ceph-mon[37580]: pgmap v5544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:47 smithi112 ceph-mon[37501]: pgmap v5544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:49 smithi107 ceph-mon[32659]: pgmap v5545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:49 smithi132 ceph-mon[37580]: pgmap v5545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:49 smithi112 ceph-mon[37501]: pgmap v5545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:51 smithi107 ceph-mon[32659]: pgmap v5546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9586.962330 secs 2024-07-26T15:30:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:51 smithi132 ceph-mon[37580]: pgmap v5546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9586.962330 secs 2024-07-26T15:30:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:51 smithi112 ceph-mon[37501]: pgmap v5546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9586.962330 secs 2024-07-26T15:30:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:53 smithi107 ceph-mon[32659]: pgmap v5547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:53 smithi132 ceph-mon[37580]: pgmap v5547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:53 smithi112 ceph-mon[37501]: pgmap v5547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:55 smithi107 ceph-mon[32659]: pgmap v5548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:55.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:55 smithi132 ceph-mon[37580]: pgmap v5548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:55 smithi112 ceph-mon[37501]: pgmap v5548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9591.962460 secs 2024-07-26T15:30:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9591.962460 secs 2024-07-26T15:30:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9591.962460 secs 2024-07-26T15:30:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:57 smithi107 ceph-mon[32659]: pgmap v5549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:57 smithi132 ceph-mon[37580]: pgmap v5549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:57 smithi112 ceph-mon[37501]: pgmap v5549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:30:59 smithi107 ceph-mon[32659]: pgmap v5550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:30:59 smithi132 ceph-mon[37580]: pgmap v5550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:59.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:59.826+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:59.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:59.826+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:59.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:30:59.826+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:30:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:30:59 smithi112 ceph-mon[37501]: pgmap v5550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:01 smithi107 ceph-mon[32659]: pgmap v5551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9596.962583 secs 2024-07-26T15:31:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:01 smithi132 ceph-mon[37580]: pgmap v5551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9596.962583 secs 2024-07-26T15:31:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:01 smithi112 ceph-mon[37501]: pgmap v5551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9596.962583 secs 2024-07-26T15:31:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:02.161+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:02.161+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:02.161+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:03 smithi107 ceph-mon[32659]: pgmap v5552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:03 smithi132 ceph-mon[37580]: pgmap v5552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:03 smithi112 ceph-mon[37501]: pgmap v5552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:05 smithi107 ceph-mon[32659]: pgmap v5553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:05 smithi132 ceph-mon[37580]: pgmap v5553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:05 smithi112 ceph-mon[37501]: pgmap v5553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9601.962701 secs 2024-07-26T15:31:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9601.962701 secs 2024-07-26T15:31:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9601.962701 secs 2024-07-26T15:31:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:07 smithi107 ceph-mon[32659]: pgmap v5554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:07 smithi132 ceph-mon[37580]: pgmap v5554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:07 smithi112 ceph-mon[37501]: pgmap v5554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:09 smithi132 ceph-mon[37580]: pgmap v5555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:09 smithi112 ceph-mon[37501]: pgmap v5555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:09 smithi107 ceph-mon[32659]: pgmap v5555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:11 smithi132 ceph-mon[37580]: pgmap v5556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9606.962818 secs 2024-07-26T15:31:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:11 smithi112 ceph-mon[37501]: pgmap v5556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9606.962818 secs 2024-07-26T15:31:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:11 smithi107 ceph-mon[32659]: pgmap v5556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9606.962818 secs 2024-07-26T15:31:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:13 smithi132 ceph-mon[37580]: pgmap v5557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:13 smithi112 ceph-mon[37501]: pgmap v5557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:13 smithi107 ceph-mon[32659]: pgmap v5557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:14.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:14.830+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:14.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:14.830+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:14.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:14.830+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:15 smithi112 ceph-mon[37501]: pgmap v5558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:15 smithi107 ceph-mon[32659]: pgmap v5558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:16.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:15 smithi132 ceph-mon[37580]: pgmap v5558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9611.962944 secs 2024-07-26T15:31:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9611.962944 secs 2024-07-26T15:31:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9611.962944 secs 2024-07-26T15:31:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:17.178+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:17.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:17.178+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:17.179 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:17.178+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:17 smithi112 ceph-mon[37501]: pgmap v5559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:17 smithi107 ceph-mon[32659]: pgmap v5559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:17 smithi132 ceph-mon[37580]: pgmap v5559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:19 smithi112 ceph-mon[37501]: pgmap v5560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:19 smithi107 ceph-mon[32659]: pgmap v5560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:20.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:19 smithi132 ceph-mon[37580]: pgmap v5560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9616.963058 secs 2024-07-26T15:31:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9616.963058 secs 2024-07-26T15:31:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9616.963058 secs 2024-07-26T15:31:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:21 smithi112 ceph-mon[37501]: pgmap v5561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:21 smithi107 ceph-mon[32659]: pgmap v5561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:21 smithi132 ceph-mon[37580]: pgmap v5561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:31:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:31:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:31:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:23 smithi112 ceph-mon[37501]: pgmap v5562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:31:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:31:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:31:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:23 smithi107 ceph-mon[32659]: pgmap v5562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:31:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:31:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:31:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:23 smithi132 ceph-mon[37580]: pgmap v5562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:31:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:31:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:31:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:24 smithi112 ceph-mon[37501]: pgmap v5563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:31:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:24 smithi112 ceph-mon[37501]: pgmap v5564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:31:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:24 smithi107 ceph-mon[32659]: pgmap v5563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:31:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:24 smithi107 ceph-mon[32659]: pgmap v5564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:31:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:24 smithi132 ceph-mon[37580]: pgmap v5563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:31:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:24 smithi132 ceph-mon[37580]: pgmap v5564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:31:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9621.963181 secs 2024-07-26T15:31:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9621.963181 secs 2024-07-26T15:31:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9621.963181 secs 2024-07-26T15:31:26.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:26 smithi107 ceph-mon[32659]: pgmap v5565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:31:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:26 smithi112 ceph-mon[37501]: pgmap v5565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:31:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:26 smithi132 ceph-mon[37580]: pgmap v5565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:31:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:28 smithi112 ceph-mon[37501]: pgmap v5566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:28 smithi107 ceph-mon[32659]: pgmap v5566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:28 smithi132 ceph-mon[37580]: pgmap v5566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:29.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:29.842+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:29.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:29.842+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:29.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:29.842+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:30 smithi112 ceph-mon[37501]: pgmap v5567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:31:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9626.963256 secs 2024-07-26T15:31:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:30 smithi107 ceph-mon[32659]: pgmap v5567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:31:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9626.963256 secs 2024-07-26T15:31:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:30 smithi132 ceph-mon[37580]: pgmap v5567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:31:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9626.963256 secs 2024-07-26T15:31:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:32.193+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:32.193+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:32.193+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:32 smithi112 ceph-mon[37501]: pgmap v5568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:31:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:32 smithi107 ceph-mon[32659]: pgmap v5568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:31:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:32 smithi132 ceph-mon[37580]: pgmap v5568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:31:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:34 smithi112 ceph-mon[37501]: pgmap v5569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:31:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:34 smithi107 ceph-mon[32659]: pgmap v5569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:31:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:34 smithi132 ceph-mon[37580]: pgmap v5569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:31:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9631.963336 secs 2024-07-26T15:31:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9631.963336 secs 2024-07-26T15:31:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9631.963336 secs 2024-07-26T15:31:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:36 smithi112 ceph-mon[37501]: pgmap v5570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:36 smithi107 ceph-mon[32659]: pgmap v5570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:36 smithi132 ceph-mon[37580]: pgmap v5570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:38 smithi112 ceph-mon[37501]: pgmap v5571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:38 smithi107 ceph-mon[32659]: pgmap v5571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:39.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:38 smithi132 ceph-mon[37580]: pgmap v5571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:40 smithi112 ceph-mon[37501]: pgmap v5572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9636.963452 secs 2024-07-26T15:31:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:40 smithi107 ceph-mon[32659]: pgmap v5572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9636.963452 secs 2024-07-26T15:31:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:40 smithi132 ceph-mon[37580]: pgmap v5572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9636.963452 secs 2024-07-26T15:31:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:42 smithi112 ceph-mon[37501]: pgmap v5573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:42 smithi107 ceph-mon[32659]: pgmap v5573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:42 smithi132 ceph-mon[37580]: pgmap v5573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:44.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:44.853+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:44.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:44.853+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:44.855 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:44.853+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:44 smithi112 ceph-mon[37501]: pgmap v5574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:44 smithi107 ceph-mon[32659]: pgmap v5574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:44 smithi132 ceph-mon[37580]: pgmap v5574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9641.963572 secs 2024-07-26T15:31:45.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9641.963572 secs 2024-07-26T15:31:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9641.963572 secs 2024-07-26T15:31:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:46 smithi107 ceph-mon[32659]: pgmap v5575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:46 smithi132 ceph-mon[37580]: pgmap v5575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:46 smithi112 ceph-mon[37501]: pgmap v5575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:47.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:47.208+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:47.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:47.208+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:47.210 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:31:47.208+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:48 smithi107 ceph-mon[32659]: pgmap v5576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:48 smithi132 ceph-mon[37580]: pgmap v5576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:48 smithi112 ceph-mon[37501]: pgmap v5576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:50 smithi107 ceph-mon[32659]: pgmap v5577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9646.963692 secs 2024-07-26T15:31:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:50 smithi132 ceph-mon[37580]: pgmap v5577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9646.963692 secs 2024-07-26T15:31:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:50 smithi112 ceph-mon[37501]: pgmap v5577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9646.963692 secs 2024-07-26T15:31:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:52 smithi107 ceph-mon[32659]: pgmap v5578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:52 smithi132 ceph-mon[37580]: pgmap v5578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:52 smithi112 ceph-mon[37501]: pgmap v5578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:54 smithi107 ceph-mon[32659]: pgmap v5579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:55.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:54 smithi132 ceph-mon[37580]: pgmap v5579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:54 smithi112 ceph-mon[37501]: pgmap v5579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9651.963837 secs 2024-07-26T15:31:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9651.963837 secs 2024-07-26T15:31:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9651.963837 secs 2024-07-26T15:31:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:56 smithi107 ceph-mon[32659]: pgmap v5580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:56 smithi132 ceph-mon[37580]: pgmap v5580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:56 smithi112 ceph-mon[37501]: pgmap v5580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:31:58 smithi107 ceph-mon[32659]: pgmap v5581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:31:58 smithi132 ceph-mon[37580]: pgmap v5581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:31:58 smithi112 ceph-mon[37501]: pgmap v5581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:59.870 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:59.868+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:59.868+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:31:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:31:59.868+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:00 smithi107 ceph-mon[32659]: pgmap v5582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9656.963974 secs 2024-07-26T15:32:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:00 smithi132 ceph-mon[37580]: pgmap v5582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9656.963974 secs 2024-07-26T15:32:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:00 smithi112 ceph-mon[37501]: pgmap v5582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9656.963974 secs 2024-07-26T15:32:02.225 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:02.224+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:02.224+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:02.224+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:02 smithi107 ceph-mon[32659]: pgmap v5583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:02 smithi132 ceph-mon[37580]: pgmap v5583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:02 smithi112 ceph-mon[37501]: pgmap v5583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:04 smithi107 ceph-mon[32659]: pgmap v5584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:04 smithi132 ceph-mon[37580]: pgmap v5584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:04 smithi112 ceph-mon[37501]: pgmap v5584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9661.964117 secs 2024-07-26T15:32:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9661.964117 secs 2024-07-26T15:32:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9661.964117 secs 2024-07-26T15:32:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:06 smithi107 ceph-mon[32659]: pgmap v5585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:06 smithi132 ceph-mon[37580]: pgmap v5585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:06 smithi112 ceph-mon[37501]: pgmap v5585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:08 smithi107 ceph-mon[32659]: pgmap v5586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:08 smithi132 ceph-mon[37580]: pgmap v5586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:09.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:08 smithi112 ceph-mon[37501]: pgmap v5586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:10 smithi107 ceph-mon[32659]: pgmap v5587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9666.964182 secs 2024-07-26T15:32:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:10 smithi132 ceph-mon[37580]: pgmap v5587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9666.964182 secs 2024-07-26T15:32:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:10 smithi112 ceph-mon[37501]: pgmap v5587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9666.964182 secs 2024-07-26T15:32:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:12 smithi132 ceph-mon[37580]: pgmap v5588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:12 smithi112 ceph-mon[37501]: pgmap v5588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:12 smithi107 ceph-mon[32659]: pgmap v5588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:14.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:14.885+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:14.886 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:14.885+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:14.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:14.885+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:14 smithi132 ceph-mon[37580]: pgmap v5589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:14 smithi112 ceph-mon[37501]: pgmap v5589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:14 smithi107 ceph-mon[32659]: pgmap v5589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9671.964305 secs 2024-07-26T15:32:16.240 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9671.964305 secs 2024-07-26T15:32:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9671.964305 secs 2024-07-26T15:32:17.171 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:16 smithi132 ceph-mon[37580]: pgmap v5590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:16 smithi112 ceph-mon[37501]: pgmap v5590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:17.240+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:17.240+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:17.240+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:16 smithi107 ceph-mon[32659]: pgmap v5590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:18 smithi112 ceph-mon[37501]: pgmap v5591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:18 smithi107 ceph-mon[32659]: pgmap v5591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:18 smithi132 ceph-mon[37580]: pgmap v5591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:20 smithi112 ceph-mon[37501]: pgmap v5592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9676.964419 secs 2024-07-26T15:32:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:20 smithi107 ceph-mon[32659]: pgmap v5592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9676.964419 secs 2024-07-26T15:32:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:20 smithi132 ceph-mon[37580]: pgmap v5592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9676.964419 secs 2024-07-26T15:32:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:22 smithi112 ceph-mon[37501]: pgmap v5593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:22 smithi107 ceph-mon[32659]: pgmap v5593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:22 smithi132 ceph-mon[37580]: pgmap v5593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:24.156 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:32:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:32:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:32:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:24 smithi112 ceph-mon[37501]: pgmap v5594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:32:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:32:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:32:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:24 smithi107 ceph-mon[32659]: pgmap v5594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:32:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:32:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:32:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:24 smithi132 ceph-mon[37580]: pgmap v5594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:32:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:32:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:32:26.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:25 smithi107 ceph-mon[32659]: pgmap v5595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:32:26.196 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:25 smithi107 ceph-mon[32659]: pgmap v5596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:26.196 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9681.964491 secs 2024-07-26T15:32:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:25 smithi112 ceph-mon[37501]: pgmap v5595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:32:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:25 smithi112 ceph-mon[37501]: pgmap v5596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9681.964491 secs 2024-07-26T15:32:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:25 smithi132 ceph-mon[37580]: pgmap v5595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:32:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:25 smithi132 ceph-mon[37580]: pgmap v5596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9681.964491 secs 2024-07-26T15:32:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:27 smithi112 ceph-mon[37501]: pgmap v5597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:32:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:27 smithi107 ceph-mon[32659]: pgmap v5597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:32:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:27 smithi132 ceph-mon[37580]: pgmap v5597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:32:29.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:29.892+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:29.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:29.892+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:29.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:29.892+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:29 smithi112 ceph-mon[37501]: pgmap v5598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:29 smithi107 ceph-mon[32659]: pgmap v5598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:29 smithi132 ceph-mon[37580]: pgmap v5598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9686.964611 secs 2024-07-26T15:32:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9686.964611 secs 2024-07-26T15:32:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9686.964611 secs 2024-07-26T15:32:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:31 smithi112 ceph-mon[37501]: pgmap v5599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:32.256+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:32.256+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:32.256+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:31 smithi107 ceph-mon[32659]: pgmap v5599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:31 smithi132 ceph-mon[37580]: pgmap v5599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:33 smithi112 ceph-mon[37501]: pgmap v5600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:32:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:33 smithi107 ceph-mon[32659]: pgmap v5600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:32:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:33 smithi132 ceph-mon[37580]: pgmap v5600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:32:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:35 smithi112 ceph-mon[37501]: pgmap v5601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:32:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9691.964705 secs 2024-07-26T15:32:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:35 smithi107 ceph-mon[32659]: pgmap v5601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:32:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9691.964705 secs 2024-07-26T15:32:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:35 smithi132 ceph-mon[37580]: pgmap v5601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:32:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9691.964705 secs 2024-07-26T15:32:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:37 smithi112 ceph-mon[37501]: pgmap v5602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:37 smithi107 ceph-mon[32659]: pgmap v5602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:37 smithi132 ceph-mon[37580]: pgmap v5602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:39 smithi112 ceph-mon[37501]: pgmap v5603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:39 smithi107 ceph-mon[32659]: pgmap v5603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:39 smithi132 ceph-mon[37580]: pgmap v5603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9696.964835 secs 2024-07-26T15:32:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9696.964835 secs 2024-07-26T15:32:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9696.964835 secs 2024-07-26T15:32:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:41 smithi112 ceph-mon[37501]: pgmap v5604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:41 smithi107 ceph-mon[32659]: pgmap v5604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:41 smithi132 ceph-mon[37580]: pgmap v5604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:43 smithi112 ceph-mon[37501]: pgmap v5605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:43 smithi107 ceph-mon[32659]: pgmap v5605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:43 smithi132 ceph-mon[37580]: pgmap v5605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:44.896+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:44.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:44.896+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:44.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:44.896+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:45 smithi112 ceph-mon[37501]: pgmap v5606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9701.964980 secs 2024-07-26T15:32:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:45 smithi107 ceph-mon[32659]: pgmap v5606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9701.964980 secs 2024-07-26T15:32:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:45 smithi132 ceph-mon[37580]: pgmap v5606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9701.964980 secs 2024-07-26T15:32:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:47.259+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:47.259+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:47.261 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:32:47.259+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:47 smithi112 ceph-mon[37501]: pgmap v5607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:47 smithi107 ceph-mon[32659]: pgmap v5607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:47 smithi132 ceph-mon[37580]: pgmap v5607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:49 smithi112 ceph-mon[37501]: pgmap v5608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:49 smithi107 ceph-mon[32659]: pgmap v5608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:49 smithi132 ceph-mon[37580]: pgmap v5608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9706.965135 secs 2024-07-26T15:32:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9706.965135 secs 2024-07-26T15:32:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9706.965135 secs 2024-07-26T15:32:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:51 smithi107 ceph-mon[32659]: pgmap v5609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:51 smithi132 ceph-mon[37580]: pgmap v5609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:51 smithi112 ceph-mon[37501]: pgmap v5609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:53 smithi107 ceph-mon[32659]: pgmap v5610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:53 smithi132 ceph-mon[37580]: pgmap v5610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:53 smithi112 ceph-mon[37501]: pgmap v5610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:55 smithi107 ceph-mon[32659]: pgmap v5611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9711.965259 secs 2024-07-26T15:32:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:55 smithi132 ceph-mon[37580]: pgmap v5611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9711.965259 secs 2024-07-26T15:32:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:55 smithi112 ceph-mon[37501]: pgmap v5611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9711.965259 secs 2024-07-26T15:32:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:57 smithi107 ceph-mon[32659]: pgmap v5612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:57 smithi132 ceph-mon[37580]: pgmap v5612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:57 smithi112 ceph-mon[37501]: pgmap v5612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:32:58 smithi107 ceph-mon[32659]: pgmap v5613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:32:58 smithi132 ceph-mon[37580]: pgmap v5613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:32:58 smithi112 ceph-mon[37501]: pgmap v5613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:59.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:59.912+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:32:59.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:59.912+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:59.914 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:32:59.912+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9716.965374 secs 2024-07-26T15:33:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:01 smithi112 ceph-mon[37501]: pgmap v5614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9716.965374 secs 2024-07-26T15:33:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:01 smithi107 ceph-mon[32659]: pgmap v5614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9716.965374 secs 2024-07-26T15:33:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:01 smithi132 ceph-mon[37580]: pgmap v5614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:02.275+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:02.275+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:02.277 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:02.275+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:03 smithi112 ceph-mon[37501]: pgmap v5615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:03 smithi107 ceph-mon[32659]: pgmap v5615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:03 smithi132 ceph-mon[37580]: pgmap v5615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:05 smithi107 ceph-mon[32659]: pgmap v5616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:05 smithi132 ceph-mon[37580]: pgmap v5616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:05 smithi112 ceph-mon[37501]: pgmap v5616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9721.965493 secs 2024-07-26T15:33:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9721.965493 secs 2024-07-26T15:33:06.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9721.965493 secs 2024-07-26T15:33:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:07 smithi107 ceph-mon[32659]: pgmap v5617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:07.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:07 smithi132 ceph-mon[37580]: pgmap v5617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:07 smithi112 ceph-mon[37501]: pgmap v5617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:09 smithi107 ceph-mon[32659]: pgmap v5618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:09 smithi132 ceph-mon[37580]: pgmap v5618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:09 smithi112 ceph-mon[37501]: pgmap v5618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9726.965640 secs 2024-07-26T15:33:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:11 smithi107 ceph-mon[32659]: pgmap v5619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9726.965640 secs 2024-07-26T15:33:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:11 smithi132 ceph-mon[37580]: pgmap v5619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9726.965640 secs 2024-07-26T15:33:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:11 smithi112 ceph-mon[37501]: pgmap v5619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:13 smithi107 ceph-mon[32659]: pgmap v5620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:13 smithi132 ceph-mon[37580]: pgmap v5620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:13 smithi112 ceph-mon[37501]: pgmap v5620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:14.290+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:33:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:14.291+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:33:14.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:14.291+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:33:14.293 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:14.292+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:33:14.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:14.928+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:14.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:14.928+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:14.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:14.928+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:15 smithi107 ceph-mon[32659]: pgmap v5621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:15.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:15 smithi132 ceph-mon[37580]: pgmap v5621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:15 smithi112 ceph-mon[37501]: pgmap v5621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:16.482+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:33:16.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:16.482+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:33:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:16.483+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:33:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:16.483+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:33:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9731.965778 secs 2024-07-26T15:33:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9731.965778 secs 2024-07-26T15:33:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9731.965778 secs 2024-07-26T15:33:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:17.291+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:17.292 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:17.291+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:17.293 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:17.291+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:17 smithi107 ceph-mon[32659]: pgmap v5622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:17 smithi132 ceph-mon[37580]: pgmap v5622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:17 smithi112 ceph-mon[37501]: pgmap v5622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:19 smithi107 ceph-mon[32659]: pgmap v5623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:19 smithi132 ceph-mon[37580]: pgmap v5623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:19 smithi112 ceph-mon[37501]: pgmap v5623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9736.965912 secs 2024-07-26T15:33:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:21 smithi107 ceph-mon[32659]: pgmap v5624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9736.965912 secs 2024-07-26T15:33:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:21 smithi132 ceph-mon[37580]: pgmap v5624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9736.965912 secs 2024-07-26T15:33:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:21 smithi112 ceph-mon[37501]: pgmap v5624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:23 smithi107 ceph-mon[32659]: pgmap v5625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:23 smithi132 ceph-mon[37580]: pgmap v5625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:23 smithi112 ceph-mon[37501]: pgmap v5625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:25 smithi107 ceph-mon[32659]: pgmap v5626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:33:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:25 smithi132 ceph-mon[37580]: pgmap v5626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:33:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:25 smithi112 ceph-mon[37501]: pgmap v5626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:33:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9741.966008 secs 2024-07-26T15:33:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:33:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:33:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:26 smithi107 ceph-mon[32659]: pgmap v5627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:33:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:26 smithi107 ceph-mon[32659]: pgmap v5628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:33:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:33:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9741.966008 secs 2024-07-26T15:33:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:33:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:33:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:26 smithi132 ceph-mon[37580]: pgmap v5627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:33:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:26 smithi132 ceph-mon[37580]: pgmap v5628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:33:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:33:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9741.966008 secs 2024-07-26T15:33:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:33:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:33:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:26 smithi112 ceph-mon[37501]: pgmap v5627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:33:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:26 smithi112 ceph-mon[37501]: pgmap v5628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:33:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:33:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:28 smithi112 ceph-mon[37501]: pgmap v5629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:33:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:28 smithi107 ceph-mon[32659]: pgmap v5629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:33:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:28 smithi132 ceph-mon[37580]: pgmap v5629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:33:29.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:29.943+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:29.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:29.943+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:29.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:29.943+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:30 smithi112 ceph-mon[37501]: pgmap v5630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:33:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9746.966129 secs 2024-07-26T15:33:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:30 smithi107 ceph-mon[32659]: pgmap v5630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:33:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9746.966129 secs 2024-07-26T15:33:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:30 smithi132 ceph-mon[37580]: pgmap v5630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:33:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9746.966129 secs 2024-07-26T15:33:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:32.307+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:32.307+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:32.308 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:32.307+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:32 smithi112 ceph-mon[37501]: pgmap v5631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:32 smithi107 ceph-mon[32659]: pgmap v5631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:32 smithi132 ceph-mon[37580]: pgmap v5631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:34 smithi112 ceph-mon[37501]: pgmap v5632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:33:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:34 smithi107 ceph-mon[32659]: pgmap v5632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:33:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:34 smithi132 ceph-mon[37580]: pgmap v5632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:33:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9751.966252 secs 2024-07-26T15:33:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9751.966252 secs 2024-07-26T15:33:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9751.966252 secs 2024-07-26T15:33:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:36 smithi112 ceph-mon[37501]: pgmap v5633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:33:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:36 smithi107 ceph-mon[32659]: pgmap v5633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:33:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:36 smithi132 ceph-mon[37580]: pgmap v5633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:33:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:38 smithi112 ceph-mon[37501]: pgmap v5634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:38 smithi107 ceph-mon[32659]: pgmap v5634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:38 smithi132 ceph-mon[37580]: pgmap v5634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:40 smithi112 ceph-mon[37501]: pgmap v5635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:33:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9756.966371 secs 2024-07-26T15:33:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:40 smithi107 ceph-mon[32659]: pgmap v5635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:33:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9756.966371 secs 2024-07-26T15:33:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:40 smithi132 ceph-mon[37580]: pgmap v5635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:33:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9756.966371 secs 2024-07-26T15:33:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:42 smithi107 ceph-mon[32659]: pgmap v5636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:33:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:42 smithi132 ceph-mon[37580]: pgmap v5636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:33:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:42 smithi112 ceph-mon[37501]: pgmap v5636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:33:44.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:44.951+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:44.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:44.951+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:44.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:44.951+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:44 smithi107 ceph-mon[32659]: pgmap v5637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:44 smithi132 ceph-mon[37580]: pgmap v5637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:44 smithi112 ceph-mon[37501]: pgmap v5637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9761.966468 secs 2024-07-26T15:33:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9761.966468 secs 2024-07-26T15:33:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9761.966468 secs 2024-07-26T15:33:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:46 smithi107 ceph-mon[32659]: pgmap v5638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:46 smithi132 ceph-mon[37580]: pgmap v5638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:46 smithi112 ceph-mon[37501]: pgmap v5638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:47.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:47.322+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:47.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:47.322+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:47.324 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:33:47.322+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:48 smithi107 ceph-mon[32659]: pgmap v5639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:48 smithi132 ceph-mon[37580]: pgmap v5639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:48 smithi112 ceph-mon[37501]: pgmap v5639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:50 smithi107 ceph-mon[32659]: pgmap v5640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:33:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9766.966607 secs 2024-07-26T15:33:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:50 smithi132 ceph-mon[37580]: pgmap v5640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:33:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9766.966607 secs 2024-07-26T15:33:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:50 smithi112 ceph-mon[37501]: pgmap v5640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:33:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9766.966607 secs 2024-07-26T15:33:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:52 smithi107 ceph-mon[32659]: pgmap v5641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:52 smithi132 ceph-mon[37580]: pgmap v5641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:52 smithi112 ceph-mon[37501]: pgmap v5641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:54 smithi107 ceph-mon[32659]: pgmap v5642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:54 smithi132 ceph-mon[37580]: pgmap v5642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:54 smithi112 ceph-mon[37501]: pgmap v5642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9771.966734 secs 2024-07-26T15:33:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9771.966734 secs 2024-07-26T15:33:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9771.966734 secs 2024-07-26T15:33:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:56 smithi107 ceph-mon[32659]: pgmap v5643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:56 smithi132 ceph-mon[37580]: pgmap v5643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:56 smithi112 ceph-mon[37501]: pgmap v5643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:33:58 smithi107 ceph-mon[32659]: pgmap v5644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:33:58 smithi132 ceph-mon[37580]: pgmap v5644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:33:58 smithi112 ceph-mon[37501]: pgmap v5644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:59.967+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:33:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:59.967+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:59.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:33:59.967+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:00 smithi107 ceph-mon[32659]: pgmap v5645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:34:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9776.966856 secs 2024-07-26T15:34:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:00 smithi132 ceph-mon[37580]: pgmap v5645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:34:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9776.966856 secs 2024-07-26T15:34:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:00 smithi112 ceph-mon[37501]: pgmap v5645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:34:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9776.966856 secs 2024-07-26T15:34:02.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:02.338+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:02.339 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:02.338+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:02.340 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:02.338+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:02 smithi107 ceph-mon[32659]: pgmap v5646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:02 smithi132 ceph-mon[37580]: pgmap v5646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:02 smithi112 ceph-mon[37501]: pgmap v5646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:04 smithi107 ceph-mon[32659]: pgmap v5647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:04 smithi132 ceph-mon[37580]: pgmap v5647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:04 smithi112 ceph-mon[37501]: pgmap v5647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9781.966996 secs 2024-07-26T15:34:06.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9781.966996 secs 2024-07-26T15:34:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9781.966996 secs 2024-07-26T15:34:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:06 smithi132 ceph-mon[37580]: pgmap v5648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:06 smithi112 ceph-mon[37501]: pgmap v5648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:06 smithi107 ceph-mon[32659]: pgmap v5648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:08 smithi132 ceph-mon[37580]: pgmap v5649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:08 smithi112 ceph-mon[37501]: pgmap v5649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:08 smithi107 ceph-mon[32659]: pgmap v5649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:10 smithi132 ceph-mon[37580]: pgmap v5650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:34:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9786.967095 secs 2024-07-26T15:34:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:10 smithi112 ceph-mon[37501]: pgmap v5650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:34:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9786.967095 secs 2024-07-26T15:34:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:10 smithi107 ceph-mon[32659]: pgmap v5650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:34:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9786.967095 secs 2024-07-26T15:34:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:12 smithi112 ceph-mon[37501]: pgmap v5651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:12 smithi107 ceph-mon[32659]: pgmap v5651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:12 smithi132 ceph-mon[37580]: pgmap v5651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:14.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:34:14.978+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:34:14.978+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:14.980 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:34:14.978+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:14 smithi112 ceph-mon[37501]: pgmap v5652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:14 smithi107 ceph-mon[32659]: pgmap v5652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:14 smithi132 ceph-mon[37580]: pgmap v5652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9791.967152 secs 2024-07-26T15:34:16.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9791.967152 secs 2024-07-26T15:34:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9791.967152 secs 2024-07-26T15:34:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:16 smithi112 ceph-mon[37501]: pgmap v5653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:16 smithi107 ceph-mon[32659]: pgmap v5653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:16 smithi132 ceph-mon[37580]: pgmap v5653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:17.354+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:17.354+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:17.354+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:18 smithi112 ceph-mon[37501]: pgmap v5654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:18 smithi107 ceph-mon[32659]: pgmap v5654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:18 smithi132 ceph-mon[37580]: pgmap v5654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:20 smithi112 ceph-mon[37501]: pgmap v5655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:34:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9796.967228 secs 2024-07-26T15:34:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:20 smithi107 ceph-mon[32659]: pgmap v5655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:34:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9796.967228 secs 2024-07-26T15:34:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:20 smithi132 ceph-mon[37580]: pgmap v5655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:34:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9796.967228 secs 2024-07-26T15:34:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:22 smithi112 ceph-mon[37501]: pgmap v5656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:22 smithi107 ceph-mon[32659]: pgmap v5656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:22 smithi132 ceph-mon[37580]: pgmap v5656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:24 smithi112 ceph-mon[37501]: pgmap v5657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:24 smithi107 ceph-mon[32659]: pgmap v5657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:24 smithi132 ceph-mon[37580]: pgmap v5657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:26.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9801.967298 secs 2024-07-26T15:34:26.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:34:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9801.967298 secs 2024-07-26T15:34:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:34:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9801.967298 secs 2024-07-26T15:34:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:34:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:26 smithi112 ceph-mon[37501]: pgmap v5658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:34:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:34:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:34:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:26 smithi107 ceph-mon[32659]: pgmap v5658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:34:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:34:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:34:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:26 smithi132 ceph-mon[37580]: pgmap v5658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:34:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:34:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:34:28.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:27 smithi107 ceph-mon[32659]: pgmap v5659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:34:28.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:27 smithi107 ceph-mon[32659]: pgmap v5660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:27 smithi112 ceph-mon[37501]: pgmap v5659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:34:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:27 smithi112 ceph-mon[37501]: pgmap v5660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:27 smithi132 ceph-mon[37580]: pgmap v5659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:34:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:27 smithi132 ceph-mon[37580]: pgmap v5660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:29.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:34:29.980+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:29.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:34:29.980+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:29.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:34:29.980+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:29 smithi112 ceph-mon[37501]: pgmap v5661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:34:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:29 smithi107 ceph-mon[32659]: pgmap v5661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:34:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:29 smithi132 ceph-mon[37580]: pgmap v5661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:34:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9806.967417 secs 2024-07-26T15:34:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9806.967417 secs 2024-07-26T15:34:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9806.967417 secs 2024-07-26T15:34:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:31 smithi112 ceph-mon[37501]: pgmap v5662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:34:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:31 smithi107 ceph-mon[32659]: pgmap v5662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:34:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:31 smithi132 ceph-mon[37580]: pgmap v5662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:34:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:32.369+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:32.370+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:32.370+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:33 smithi112 ceph-mon[37501]: pgmap v5663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:34:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:33 smithi107 ceph-mon[32659]: pgmap v5663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:34:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:33 smithi132 ceph-mon[37580]: pgmap v5663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:34:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:35 smithi112 ceph-mon[37501]: pgmap v5664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9811.967505 secs 2024-07-26T15:34:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:35 smithi107 ceph-mon[32659]: pgmap v5664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9811.967505 secs 2024-07-26T15:34:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:35 smithi132 ceph-mon[37580]: pgmap v5664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9811.967505 secs 2024-07-26T15:34:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:37 smithi112 ceph-mon[37501]: pgmap v5665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:34:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:37 smithi107 ceph-mon[32659]: pgmap v5665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:34:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:37 smithi132 ceph-mon[37580]: pgmap v5665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:34:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:39 smithi112 ceph-mon[37501]: pgmap v5666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:39 smithi107 ceph-mon[32659]: pgmap v5666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:39 smithi132 ceph-mon[37580]: pgmap v5666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9816.967664 secs 2024-07-26T15:34:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9816.967664 secs 2024-07-26T15:34:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9816.967664 secs 2024-07-26T15:34:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:41 smithi112 ceph-mon[37501]: pgmap v5667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:41 smithi107 ceph-mon[32659]: pgmap v5667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:41 smithi132 ceph-mon[37580]: pgmap v5667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:43 smithi112 ceph-mon[37501]: pgmap v5668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:43 smithi107 ceph-mon[32659]: pgmap v5668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:43 smithi132 ceph-mon[37580]: pgmap v5668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:44.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:34:44.994+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:44.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:34:44.994+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:44.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:34:44.994+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:45 smithi112 ceph-mon[37501]: pgmap v5669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9821.967769 secs 2024-07-26T15:34:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:45 smithi107 ceph-mon[32659]: pgmap v5669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9821.967769 secs 2024-07-26T15:34:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:45 smithi132 ceph-mon[37580]: pgmap v5669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9821.967769 secs 2024-07-26T15:34:47.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:47.378+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:47.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:47.378+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:34:47.380 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:34:47.378+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:47 smithi112 ceph-mon[37501]: pgmap v5670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:47 smithi107 ceph-mon[32659]: pgmap v5670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:47 smithi132 ceph-mon[37580]: pgmap v5670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:49 smithi107 ceph-mon[32659]: pgmap v5671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:49 smithi132 ceph-mon[37580]: pgmap v5671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:49 smithi112 ceph-mon[37501]: pgmap v5671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9826.967886 secs 2024-07-26T15:34:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9826.967886 secs 2024-07-26T15:34:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9826.967886 secs 2024-07-26T15:34:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:51 smithi107 ceph-mon[32659]: pgmap v5672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:51 smithi132 ceph-mon[37580]: pgmap v5672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:51 smithi112 ceph-mon[37501]: pgmap v5672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:53 smithi107 ceph-mon[32659]: pgmap v5673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:53 smithi132 ceph-mon[37580]: pgmap v5673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:53 smithi112 ceph-mon[37501]: pgmap v5673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:34:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:55 smithi107 ceph-mon[32659]: pgmap v5674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9831.968036 secs 2024-07-26T15:34:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:55 smithi132 ceph-mon[37580]: pgmap v5674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9831.968036 secs 2024-07-26T15:34:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:55 smithi112 ceph-mon[37501]: pgmap v5674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9831.968036 secs 2024-07-26T15:34:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:57 smithi107 ceph-mon[32659]: pgmap v5675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:57 smithi132 ceph-mon[37580]: pgmap v5675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:34:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:57 smithi112 ceph-mon[37501]: pgmap v5675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:00.010+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:00.010+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:00.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:00.010+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:34:59 smithi107 ceph-mon[32659]: pgmap v5676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:34:59 smithi132 ceph-mon[37580]: pgmap v5676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:34:59 smithi112 ceph-mon[37501]: pgmap v5676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9836.968107 secs 2024-07-26T15:35:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9836.968107 secs 2024-07-26T15:35:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9836.968107 secs 2024-07-26T15:35:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:02 smithi107 ceph-mon[32659]: pgmap v5677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:02 smithi132 ceph-mon[37580]: pgmap v5677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:02.392+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:02.392+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:02.394 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:02.392+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:02 smithi112 ceph-mon[37501]: pgmap v5677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:03 smithi107 ceph-mon[32659]: pgmap v5678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:03 smithi132 ceph-mon[37580]: pgmap v5678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:03 smithi112 ceph-mon[37501]: pgmap v5678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:06.153 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:05 smithi112 ceph-mon[37501]: pgmap v5679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:06.153 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9841.968192 secs 2024-07-26T15:35:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:05 smithi107 ceph-mon[32659]: pgmap v5679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9841.968192 secs 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:05 smithi132 ceph-mon[37580]: pgmap v5679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9841.968192 secs 2024-07-26T15:35:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:07 smithi112 ceph-mon[37501]: pgmap v5680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:07 smithi107 ceph-mon[32659]: pgmap v5680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:07 smithi132 ceph-mon[37580]: pgmap v5680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:09 smithi112 ceph-mon[37501]: pgmap v5681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:09 smithi107 ceph-mon[32659]: pgmap v5681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:09 smithi132 ceph-mon[37580]: pgmap v5681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9846.968246 secs 2024-07-26T15:35:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9846.968246 secs 2024-07-26T15:35:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9846.968246 secs 2024-07-26T15:35:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:11 smithi112 ceph-mon[37501]: pgmap v5682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:11 smithi107 ceph-mon[32659]: pgmap v5682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:11 smithi132 ceph-mon[37580]: pgmap v5682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:13 smithi112 ceph-mon[37501]: pgmap v5683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:13 smithi107 ceph-mon[32659]: pgmap v5683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:13 smithi132 ceph-mon[37580]: pgmap v5683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:15.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:15.011+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:15.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:15.011+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:15.013 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:15.011+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:16.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:15 smithi132 ceph-mon[37580]: pgmap v5684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:16.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9851.968371 secs 2024-07-26T15:35:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:15 smithi112 ceph-mon[37501]: pgmap v5684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9851.968371 secs 2024-07-26T15:35:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:15 smithi107 ceph-mon[32659]: pgmap v5684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9851.968371 secs 2024-07-26T15:35:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:17.408+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:17.408+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:17.409 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:17.408+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:17 smithi112 ceph-mon[37501]: pgmap v5685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:17 smithi107 ceph-mon[32659]: pgmap v5685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:17 smithi132 ceph-mon[37580]: pgmap v5685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:19 smithi112 ceph-mon[37501]: pgmap v5686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:19 smithi107 ceph-mon[32659]: pgmap v5686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:19 smithi132 ceph-mon[37580]: pgmap v5686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9856.968495 secs 2024-07-26T15:35:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9856.968495 secs 2024-07-26T15:35:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9856.968495 secs 2024-07-26T15:35:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:21 smithi112 ceph-mon[37501]: pgmap v5687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:21 smithi107 ceph-mon[32659]: pgmap v5687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:21 smithi132 ceph-mon[37580]: pgmap v5687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:23 smithi112 ceph-mon[37501]: pgmap v5688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:23 smithi107 ceph-mon[32659]: pgmap v5688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:23 smithi132 ceph-mon[37580]: pgmap v5688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:25 smithi112 ceph-mon[37501]: pgmap v5689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9861.968617 secs 2024-07-26T15:35:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:25 smithi107 ceph-mon[32659]: pgmap v5689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9861.968617 secs 2024-07-26T15:35:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:25 smithi132 ceph-mon[37580]: pgmap v5689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9861.968617 secs 2024-07-26T15:35:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:27 smithi112 ceph-mon[37501]: pgmap v5690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:35:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:35:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:35:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:27 smithi107 ceph-mon[32659]: pgmap v5690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:35:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:35:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:35:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:27 smithi132 ceph-mon[37580]: pgmap v5690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:35:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:35:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:35:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:28 smithi112 ceph-mon[37501]: pgmap v5691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:35:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:28 smithi112 ceph-mon[37501]: pgmap v5692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:35:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:35:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:28 smithi107 ceph-mon[32659]: pgmap v5691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:35:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:28 smithi107 ceph-mon[32659]: pgmap v5692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:35:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:35:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:28 smithi132 ceph-mon[37580]: pgmap v5691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:35:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:28 smithi132 ceph-mon[37580]: pgmap v5692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:35:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:35:30.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:30.027+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:30.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:30.027+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:30.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:30.027+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:31.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:30 smithi107 ceph-mon[32659]: pgmap v5693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:35:31.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9866.968736 secs 2024-07-26T15:35:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:30 smithi112 ceph-mon[37501]: pgmap v5693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:35:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9866.968736 secs 2024-07-26T15:35:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:30 smithi132 ceph-mon[37580]: pgmap v5693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:35:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9866.968736 secs 2024-07-26T15:35:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:32.424+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:32.424+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:32.424+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:32 smithi112 ceph-mon[37501]: pgmap v5694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:32 smithi107 ceph-mon[32659]: pgmap v5694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:32 smithi132 ceph-mon[37580]: pgmap v5694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:34 smithi112 ceph-mon[37501]: pgmap v5695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:35:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:34 smithi107 ceph-mon[32659]: pgmap v5695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:35:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:34 smithi132 ceph-mon[37580]: pgmap v5695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:35:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9871.968859 secs 2024-07-26T15:35:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9871.968859 secs 2024-07-26T15:35:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9871.968859 secs 2024-07-26T15:35:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:36 smithi112 ceph-mon[37501]: pgmap v5696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:35:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:36 smithi107 ceph-mon[32659]: pgmap v5696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:35:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:36 smithi132 ceph-mon[37580]: pgmap v5696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:35:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:38 smithi112 ceph-mon[37501]: pgmap v5697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:35:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:38 smithi107 ceph-mon[32659]: pgmap v5697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:35:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:38 smithi132 ceph-mon[37580]: pgmap v5697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:40 smithi112 ceph-mon[37501]: pgmap v5698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9876.968990 secs 2024-07-26T15:35:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:40 smithi107 ceph-mon[32659]: pgmap v5698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9876.968990 secs 2024-07-26T15:35:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:40 smithi132 ceph-mon[37580]: pgmap v5698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9876.968990 secs 2024-07-26T15:35:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:42 smithi112 ceph-mon[37501]: pgmap v5699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:42 smithi107 ceph-mon[32659]: pgmap v5699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:42 smithi132 ceph-mon[37580]: pgmap v5699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:45.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:45.041+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:45.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:45.041+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:45.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:35:45.041+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:44 smithi107 ceph-mon[32659]: pgmap v5700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:35:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:44 smithi132 ceph-mon[37580]: pgmap v5700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:35:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:44 smithi112 ceph-mon[37501]: pgmap v5700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:35:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9881.969154 secs 2024-07-26T15:35:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9881.969154 secs 2024-07-26T15:35:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9881.969154 secs 2024-07-26T15:35:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:46 smithi107 ceph-mon[32659]: pgmap v5701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:46 smithi132 ceph-mon[37580]: pgmap v5701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:47.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:47.433+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:47.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:47.433+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:35:47.435 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:35:47.433+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:46 smithi112 ceph-mon[37501]: pgmap v5701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:48 smithi107 ceph-mon[32659]: pgmap v5702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:48 smithi132 ceph-mon[37580]: pgmap v5702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:48 smithi112 ceph-mon[37501]: pgmap v5702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:50 smithi107 ceph-mon[32659]: pgmap v5703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9886.969279 secs 2024-07-26T15:35:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:50 smithi132 ceph-mon[37580]: pgmap v5703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9886.969279 secs 2024-07-26T15:35:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:50 smithi112 ceph-mon[37501]: pgmap v5703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9886.969279 secs 2024-07-26T15:35:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:52 smithi107 ceph-mon[32659]: pgmap v5704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:52 smithi132 ceph-mon[37580]: pgmap v5704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:52 smithi112 ceph-mon[37501]: pgmap v5704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:35:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:54 smithi107 ceph-mon[32659]: pgmap v5705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:54 smithi132 ceph-mon[37580]: pgmap v5705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:54 smithi112 ceph-mon[37501]: pgmap v5705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9891.969365 secs 2024-07-26T15:35:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9891.969365 secs 2024-07-26T15:35:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9891.969365 secs 2024-07-26T15:35:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:57 smithi107 ceph-mon[32659]: pgmap v5706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:57 smithi132 ceph-mon[37580]: pgmap v5706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:57 smithi112 ceph-mon[37501]: pgmap v5706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:35:59 smithi107 ceph-mon[32659]: pgmap v5707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:35:59 smithi132 ceph-mon[37580]: pgmap v5707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:35:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:35:59 smithi112 ceph-mon[37501]: pgmap v5707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:00.057+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:00.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:00.057+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:00.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:00.057+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:00 smithi107 ceph-mon[32659]: pgmap v5708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:00 smithi132 ceph-mon[37580]: pgmap v5708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:00 smithi112 ceph-mon[37501]: pgmap v5708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9896.969483 secs 2024-07-26T15:36:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9896.969483 secs 2024-07-26T15:36:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9896.969483 secs 2024-07-26T15:36:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:02 smithi107 ceph-mon[32659]: pgmap v5709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:02 smithi132 ceph-mon[37580]: pgmap v5709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:02 smithi112 ceph-mon[37501]: pgmap v5709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:02.449+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:02.449+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:02.449+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:05.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:04 smithi107 ceph-mon[32659]: pgmap v5710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:04 smithi132 ceph-mon[37580]: pgmap v5710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:04 smithi112 ceph-mon[37501]: pgmap v5710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9901.969610 secs 2024-07-26T15:36:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9901.969610 secs 2024-07-26T15:36:06.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9901.969610 secs 2024-07-26T15:36:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:06 smithi107 ceph-mon[32659]: pgmap v5711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:06 smithi132 ceph-mon[37580]: pgmap v5711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:06 smithi112 ceph-mon[37501]: pgmap v5711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:08 smithi107 ceph-mon[32659]: pgmap v5712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:08 smithi132 ceph-mon[37580]: pgmap v5712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:08 smithi112 ceph-mon[37501]: pgmap v5712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:10 smithi107 ceph-mon[32659]: pgmap v5713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9906.969671 secs 2024-07-26T15:36:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:10 smithi132 ceph-mon[37580]: pgmap v5713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9906.969671 secs 2024-07-26T15:36:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:10 smithi112 ceph-mon[37501]: pgmap v5713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9906.969671 secs 2024-07-26T15:36:11.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:11.667+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 msgr2=0x7ff0ec0c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T15:36:11.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:11.667+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 msgr2=0x7ff0ec0c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:36:11.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:11.667+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f8014050 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:36:11.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:11.667+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800f3e0 tx=0x7ff0f8014050 comp rx=0 tx=0).stop 2024-07-26T15:36:11.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:11.667+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 msgr2=0x7ff0ec0c40a0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T15:36:11.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:11.667+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 msgr2=0x7ff0ec0c40a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:36:11.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:11.667+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:36:11.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:11.668+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:36:11.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:11.668+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:36:11.672 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:11.671+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800ff40 tx=0x7ff0f8051050 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:36:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:12 smithi107 ceph-mon[32659]: pgmap v5714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:12 smithi132 ceph-mon[37580]: pgmap v5714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:13.321+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:36:13.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:13.321+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:36:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:13.322+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:36:13.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:13.322+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:36:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:12 smithi112 ceph-mon[37501]: pgmap v5714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:13.773+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T15:36:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:13.773+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:36:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:13.773+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c6a0 tx=0x7efc680b9000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:36:13.774 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:13.773+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7efc6800c6a0 tx=0x7efc680b9000 comp rx=0 tx=0).stop 2024-07-26T15:36:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:13.773+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c2f20 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T15:36:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:13.773+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 msgr2=0x7efc5c0c2f20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:36:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:13.773+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c1c50 0x7efc5c0c2f20 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:36:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:13.773+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:36:13.775 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:13.774+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:36:13.776 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:13.776+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7efc680aa8b0 tx=0x7efc680e9000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:36:14.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:14.321+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:36:14.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:14.321+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:36:14.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:14.322+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:36:14.323 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:14.322+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:36:15.059 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:15.058+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:15.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:15.058+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:15.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:15.058+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:14 smithi107 ceph-mon[32659]: pgmap v5715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:14 smithi132 ceph-mon[37580]: pgmap v5715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:14 smithi112 ceph-mon[37501]: pgmap v5715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:15.515+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:36:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:15.515+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:36:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:15.516+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:36:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:15.516+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:36:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9911.969827 secs 2024-07-26T15:36:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9911.969827 secs 2024-07-26T15:36:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9911.969827 secs 2024-07-26T15:36:16.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:16.515+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:36:16.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:16.515+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:36:16.516 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:16.516+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:36:16.517 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:16.516+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:36:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:17 smithi107 ceph-mon[32659]: pgmap v5716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:17 smithi132 ceph-mon[37580]: pgmap v5716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:17 smithi112 ceph-mon[37501]: pgmap v5716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:17.461+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:17.462+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:17.463 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:17.462+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:19 smithi107 ceph-mon[32659]: pgmap v5717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:19 smithi132 ceph-mon[37580]: pgmap v5717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:19 smithi112 ceph-mon[37501]: pgmap v5717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:20 smithi112 ceph-mon[37501]: pgmap v5718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:20 smithi107 ceph-mon[32659]: pgmap v5718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:20 smithi132 ceph-mon[37580]: pgmap v5718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9916.969956 secs 2024-07-26T15:36:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9916.969956 secs 2024-07-26T15:36:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9916.969956 secs 2024-07-26T15:36:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:22 smithi112 ceph-mon[37501]: pgmap v5719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:22 smithi107 ceph-mon[32659]: pgmap v5719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:22 smithi132 ceph-mon[37580]: pgmap v5719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:24 smithi107 ceph-mon[32659]: pgmap v5720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:24 smithi132 ceph-mon[37580]: pgmap v5720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:24 smithi112 ceph-mon[37501]: pgmap v5720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9921.970113 secs 2024-07-26T15:36:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9921.970113 secs 2024-07-26T15:36:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9921.970113 secs 2024-07-26T15:36:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:26 smithi107 ceph-mon[32659]: pgmap v5721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:27.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:26 smithi132 ceph-mon[37580]: pgmap v5721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:26 smithi112 ceph-mon[37501]: pgmap v5721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:28 smithi107 ceph-mon[32659]: pgmap v5722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:36:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:28 smithi132 ceph-mon[37580]: pgmap v5722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:36:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:28 smithi112 ceph-mon[37501]: pgmap v5722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:36:30.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:30.063+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:30.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:30.064+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:30.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:30.064+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:36:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:36:30.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:36:30.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:36:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:31 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:36:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:31 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:36:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:31 smithi107 ceph-mon[32659]: pgmap v5723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:36:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:31 smithi107 ceph-mon[32659]: pgmap v5724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:36:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9926.970217 secs 2024-07-26T15:36:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:31 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:36:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:31 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:36:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:31 smithi132 ceph-mon[37580]: pgmap v5723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:36:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:31 smithi132 ceph-mon[37580]: pgmap v5724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:36:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9926.970217 secs 2024-07-26T15:36:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:31 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:36:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:31 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:36:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:31 smithi112 ceph-mon[37501]: pgmap v5723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:36:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:31 smithi112 ceph-mon[37501]: pgmap v5724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:36:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9926.970217 secs 2024-07-26T15:36:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:32.466+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:32.466+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:32.466+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:33 smithi107 ceph-mon[32659]: pgmap v5725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:36:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:33 smithi132 ceph-mon[37580]: pgmap v5725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:36:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:33 smithi112 ceph-mon[37501]: pgmap v5725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:36:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:35 smithi107 ceph-mon[32659]: pgmap v5726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:36:35.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:35 smithi132 ceph-mon[37580]: pgmap v5726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:36:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:35 smithi112 ceph-mon[37501]: pgmap v5726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:36:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:36 smithi107 ceph-mon[32659]: pgmap v5727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9931.970335 secs 2024-07-26T15:36:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:36 smithi132 ceph-mon[37580]: pgmap v5727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9931.970335 secs 2024-07-26T15:36:36.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:36 smithi112 ceph-mon[37501]: pgmap v5727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:36.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9931.970335 secs 2024-07-26T15:36:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:38 smithi112 ceph-mon[37501]: pgmap v5728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:38 smithi107 ceph-mon[32659]: pgmap v5728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:38 smithi132 ceph-mon[37580]: pgmap v5728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:40 smithi112 ceph-mon[37501]: pgmap v5729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:36:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:40 smithi107 ceph-mon[32659]: pgmap v5729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:36:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:40 smithi132 ceph-mon[37580]: pgmap v5729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:36:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9936.970449 secs 2024-07-26T15:36:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9936.970449 secs 2024-07-26T15:36:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9936.970449 secs 2024-07-26T15:36:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:42 smithi112 ceph-mon[37501]: pgmap v5730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:42 smithi107 ceph-mon[32659]: pgmap v5730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:42 smithi132 ceph-mon[37580]: pgmap v5730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:44 smithi112 ceph-mon[37501]: pgmap v5731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:44 smithi107 ceph-mon[32659]: pgmap v5731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:44 smithi132 ceph-mon[37580]: pgmap v5731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:45.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:45.065+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:45.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:45.065+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:45.068 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:36:45.065+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:46 smithi112 ceph-mon[37501]: pgmap v5732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9941.970528 secs 2024-07-26T15:36:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:46 smithi107 ceph-mon[32659]: pgmap v5732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9941.970528 secs 2024-07-26T15:36:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:46 smithi132 ceph-mon[37580]: pgmap v5732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9941.970528 secs 2024-07-26T15:36:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:47.481+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:47.481+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:36:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:36:47.481+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:48 smithi112 ceph-mon[37501]: pgmap v5733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:48 smithi107 ceph-mon[32659]: pgmap v5733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:48 smithi132 ceph-mon[37580]: pgmap v5733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:50 smithi107 ceph-mon[32659]: pgmap v5734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:50 smithi132 ceph-mon[37580]: pgmap v5734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:50 smithi112 ceph-mon[37501]: pgmap v5734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9946.970640 secs 2024-07-26T15:36:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9946.970640 secs 2024-07-26T15:36:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9946.970640 secs 2024-07-26T15:36:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:52 smithi107 ceph-mon[32659]: pgmap v5735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:52 smithi132 ceph-mon[37580]: pgmap v5735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:52 smithi112 ceph-mon[37501]: pgmap v5735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:54 smithi107 ceph-mon[32659]: pgmap v5736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:54 smithi132 ceph-mon[37580]: pgmap v5736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:54 smithi112 ceph-mon[37501]: pgmap v5736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:56 smithi107 ceph-mon[32659]: pgmap v5737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9951.970789 secs 2024-07-26T15:36:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:56 smithi132 ceph-mon[37580]: pgmap v5737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9951.970789 secs 2024-07-26T15:36:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:56 smithi112 ceph-mon[37501]: pgmap v5737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9951.970789 secs 2024-07-26T15:36:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:36:58 smithi107 ceph-mon[32659]: pgmap v5738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:36:58 smithi132 ceph-mon[37580]: pgmap v5738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:36:58 smithi112 ceph-mon[37501]: pgmap v5738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:00.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:00.080+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:00.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:00.080+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:00.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:00.080+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:00 smithi107 ceph-mon[32659]: pgmap v5739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:00 smithi132 ceph-mon[37580]: pgmap v5739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:00 smithi112 ceph-mon[37501]: pgmap v5739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9956.970919 secs 2024-07-26T15:37:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9956.970919 secs 2024-07-26T15:37:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9956.970919 secs 2024-07-26T15:37:02.498 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:02.497+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:02.497+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:02.499 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:02.497+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:02 smithi107 ceph-mon[32659]: pgmap v5740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:02 smithi132 ceph-mon[37580]: pgmap v5740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:02 smithi112 ceph-mon[37501]: pgmap v5740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:04 smithi107 ceph-mon[32659]: pgmap v5741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:04 smithi132 ceph-mon[37580]: pgmap v5741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:04 smithi112 ceph-mon[37501]: pgmap v5741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:06 smithi107 ceph-mon[32659]: pgmap v5742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9961.971052 secs 2024-07-26T15:37:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:06 smithi132 ceph-mon[37580]: pgmap v5742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9961.971052 secs 2024-07-26T15:37:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:06 smithi112 ceph-mon[37501]: pgmap v5742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9961.971052 secs 2024-07-26T15:37:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:08 smithi107 ceph-mon[32659]: pgmap v5743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:08 smithi132 ceph-mon[37580]: pgmap v5743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:08 smithi112 ceph-mon[37501]: pgmap v5743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:10 smithi107 ceph-mon[32659]: pgmap v5744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:10 smithi132 ceph-mon[37580]: pgmap v5744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:10 smithi112 ceph-mon[37501]: pgmap v5744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9966.971187 secs 2024-07-26T15:37:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9966.971187 secs 2024-07-26T15:37:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9966.971187 secs 2024-07-26T15:37:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:12 smithi132 ceph-mon[37580]: pgmap v5745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:12 smithi112 ceph-mon[37501]: pgmap v5745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:12 smithi107 ceph-mon[32659]: pgmap v5745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:14 smithi112 ceph-mon[37501]: pgmap v5746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:14 smithi107 ceph-mon[32659]: pgmap v5746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:14 smithi132 ceph-mon[37580]: pgmap v5746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:15.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:15.096+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:15.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:15.096+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:15.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:15.096+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:16.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:16 smithi132 ceph-mon[37580]: pgmap v5747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:16.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9971.971272 secs 2024-07-26T15:37:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:16 smithi112 ceph-mon[37501]: pgmap v5747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9971.971272 secs 2024-07-26T15:37:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:16 smithi107 ceph-mon[32659]: pgmap v5747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9971.971272 secs 2024-07-26T15:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:17.502+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:17.502+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:17.503 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:17.502+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:18 smithi112 ceph-mon[37501]: pgmap v5748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:18 smithi107 ceph-mon[32659]: pgmap v5748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:18 smithi132 ceph-mon[37580]: pgmap v5748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:20 smithi107 ceph-mon[32659]: pgmap v5749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:20 smithi112 ceph-mon[37501]: pgmap v5749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:20 smithi132 ceph-mon[37580]: pgmap v5749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9976.971352 secs 2024-07-26T15:37:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9976.971352 secs 2024-07-26T15:37:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9976.971352 secs 2024-07-26T15:37:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:22 smithi112 ceph-mon[37501]: pgmap v5750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:22 smithi107 ceph-mon[32659]: pgmap v5750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:22 smithi132 ceph-mon[37580]: pgmap v5750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:24 smithi112 ceph-mon[37501]: pgmap v5751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:24 smithi107 ceph-mon[32659]: pgmap v5751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:24 smithi132 ceph-mon[37580]: pgmap v5751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:26 smithi112 ceph-mon[37501]: pgmap v5752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9981.971469 secs 2024-07-26T15:37:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:26 smithi107 ceph-mon[32659]: pgmap v5752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9981.971469 secs 2024-07-26T15:37:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:26 smithi132 ceph-mon[37580]: pgmap v5752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9981.971469 secs 2024-07-26T15:37:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:28 smithi112 ceph-mon[37501]: pgmap v5753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:28 smithi107 ceph-mon[32659]: pgmap v5753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:28 smithi132 ceph-mon[37580]: pgmap v5753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:37:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:37:29.809 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:37:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:30.110+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:30.110+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:30.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:30.110+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: pgmap v5754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:37:30.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:37:30.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:37:30.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:37:30.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:37:30.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:37:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: pgmap v5754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:37:30.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:37:30.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:37:30.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:37:30.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:37:30.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:37:30.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:37:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: pgmap v5754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:37:30.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:37:30.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:37:30.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:37:30.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:37:30.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:37:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:31 smithi112 ceph-mon[37501]: pgmap v5755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:37:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:31 smithi112 ceph-mon[37501]: pgmap v5756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9986.971579 secs 2024-07-26T15:37:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:31 smithi107 ceph-mon[32659]: pgmap v5755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:37:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:31 smithi107 ceph-mon[32659]: pgmap v5756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9986.971579 secs 2024-07-26T15:37:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:31 smithi132 ceph-mon[37580]: pgmap v5755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:37:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:31 smithi132 ceph-mon[37580]: pgmap v5756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9986.971579 secs 2024-07-26T15:37:32.518 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:32.518+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:32.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:32.518+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:32.519 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:32.518+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:33 smithi112 ceph-mon[37501]: pgmap v5757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:33 smithi107 ceph-mon[32659]: pgmap v5757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:33 smithi132 ceph-mon[37580]: pgmap v5757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:35 smithi112 ceph-mon[37501]: pgmap v5758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:35 smithi107 ceph-mon[32659]: pgmap v5758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:35 smithi132 ceph-mon[37580]: pgmap v5758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9991.971705 secs 2024-07-26T15:37:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9991.971705 secs 2024-07-26T15:37:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9991.971705 secs 2024-07-26T15:37:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:37 smithi112 ceph-mon[37501]: pgmap v5759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:37:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:37 smithi107 ceph-mon[32659]: pgmap v5759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:37:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:37 smithi132 ceph-mon[37580]: pgmap v5759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:37:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:39 smithi112 ceph-mon[37501]: pgmap v5760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:39 smithi107 ceph-mon[32659]: pgmap v5760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:39 smithi132 ceph-mon[37580]: pgmap v5760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 9996.971826 secs 2024-07-26T15:37:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:41 smithi112 ceph-mon[37501]: pgmap v5761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:37:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 9996.971826 secs 2024-07-26T15:37:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:41 smithi107 ceph-mon[32659]: pgmap v5761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:37:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 9996.971826 secs 2024-07-26T15:37:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:41 smithi132 ceph-mon[37580]: pgmap v5761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:37:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:43 smithi112 ceph-mon[37501]: pgmap v5762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:43 smithi107 ceph-mon[32659]: pgmap v5762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:43 smithi132 ceph-mon[37580]: pgmap v5762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:45.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:45.123+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:45.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:45.123+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:45.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:37:45.123+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:45 smithi112 ceph-mon[37501]: pgmap v5763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:45 smithi107 ceph-mon[32659]: pgmap v5763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:45 smithi132 ceph-mon[37580]: pgmap v5763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10001.971954 secs 2024-07-26T15:37:46.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10001.971954 secs 2024-07-26T15:37:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10001.971954 secs 2024-07-26T15:37:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:47.532+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:47.532+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:47.534 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:37:47.532+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:37:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:47 smithi112 ceph-mon[37501]: pgmap v5764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:47 smithi107 ceph-mon[32659]: pgmap v5764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:47 smithi132 ceph-mon[37580]: pgmap v5764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:49 smithi107 ceph-mon[32659]: pgmap v5765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:37:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:49 smithi132 ceph-mon[37580]: pgmap v5765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:37:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:49 smithi112 ceph-mon[37501]: pgmap v5765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:37:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10006.972099 secs 2024-07-26T15:37:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:51 smithi107 ceph-mon[32659]: pgmap v5766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:37:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10006.972099 secs 2024-07-26T15:37:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:51 smithi132 ceph-mon[37580]: pgmap v5766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:37:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10006.972099 secs 2024-07-26T15:37:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:51 smithi112 ceph-mon[37501]: pgmap v5766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:37:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:53 smithi107 ceph-mon[32659]: pgmap v5767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:53 smithi132 ceph-mon[37580]: pgmap v5767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:53 smithi112 ceph-mon[37501]: pgmap v5767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:55 smithi107 ceph-mon[32659]: pgmap v5768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:37:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:55 smithi132 ceph-mon[37580]: pgmap v5768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:37:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:55 smithi112 ceph-mon[37501]: pgmap v5768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:37:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10011.972221 secs 2024-07-26T15:37:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10011.972221 secs 2024-07-26T15:37:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10011.972221 secs 2024-07-26T15:37:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:57 smithi107 ceph-mon[32659]: pgmap v5769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:37:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:57 smithi132 ceph-mon[37580]: pgmap v5769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:37:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:57 smithi112 ceph-mon[37501]: pgmap v5769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:37:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:37:59 smithi107 ceph-mon[32659]: pgmap v5770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:37:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:37:59 smithi132 ceph-mon[37580]: pgmap v5770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:37:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:37:59 smithi112 ceph-mon[37501]: pgmap v5770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:38:00.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:00.139+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:00.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:00.139+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:00.141 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:00.139+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10016.972289 secs 2024-07-26T15:38:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:01 smithi107 ceph-mon[32659]: pgmap v5771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10016.972289 secs 2024-07-26T15:38:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:01 smithi132 ceph-mon[37580]: pgmap v5771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10016.972289 secs 2024-07-26T15:38:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:01 smithi112 ceph-mon[37501]: pgmap v5771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:02.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:02.548+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:02.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:02.548+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:02.550 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:02.548+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:03 smithi107 ceph-mon[32659]: pgmap v5772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:03 smithi132 ceph-mon[37580]: pgmap v5772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:03 smithi112 ceph-mon[37501]: pgmap v5772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:05 smithi107 ceph-mon[32659]: pgmap v5773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:38:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:05 smithi132 ceph-mon[37580]: pgmap v5773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:38:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:05 smithi112 ceph-mon[37501]: pgmap v5773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:38:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10021.972436 secs 2024-07-26T15:38:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10021.972436 secs 2024-07-26T15:38:06.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10021.972436 secs 2024-07-26T15:38:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:07 smithi107 ceph-mon[32659]: pgmap v5774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:07 smithi132 ceph-mon[37580]: pgmap v5774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:07 smithi112 ceph-mon[37501]: pgmap v5774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:09 smithi107 ceph-mon[32659]: pgmap v5775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:09 smithi132 ceph-mon[37580]: pgmap v5775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:09 smithi112 ceph-mon[37501]: pgmap v5775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10026.972549 secs 2024-07-26T15:38:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:11 smithi107 ceph-mon[32659]: pgmap v5776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10026.972549 secs 2024-07-26T15:38:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:11 smithi132 ceph-mon[37580]: pgmap v5776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10026.972549 secs 2024-07-26T15:38:11.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:11 smithi112 ceph-mon[37501]: pgmap v5776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:13 smithi107 ceph-mon[32659]: pgmap v5777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:13 smithi132 ceph-mon[37580]: pgmap v5777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:13 smithi112 ceph-mon[37501]: pgmap v5777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:15.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:15.155+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:15.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:15.155+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:15.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:15.155+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:15 smithi132 ceph-mon[37580]: pgmap v5778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:15 smithi112 ceph-mon[37501]: pgmap v5778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:15 smithi107 ceph-mon[32659]: pgmap v5778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10031.972664 secs 2024-07-26T15:38:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10031.972664 secs 2024-07-26T15:38:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10031.972664 secs 2024-07-26T15:38:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:17.564+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:17.564+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:17.565 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:17.564+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:17 smithi132 ceph-mon[37580]: pgmap v5779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:17 smithi112 ceph-mon[37501]: pgmap v5779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:17 smithi107 ceph-mon[32659]: pgmap v5779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:19 smithi112 ceph-mon[37501]: pgmap v5780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:19 smithi107 ceph-mon[32659]: pgmap v5780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:19 smithi132 ceph-mon[37580]: pgmap v5780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10036.972786 secs 2024-07-26T15:38:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:21 smithi112 ceph-mon[37501]: pgmap v5781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10036.972786 secs 2024-07-26T15:38:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:21 smithi107 ceph-mon[32659]: pgmap v5781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10036.972786 secs 2024-07-26T15:38:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:21 smithi132 ceph-mon[37580]: pgmap v5781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:23 smithi112 ceph-mon[37501]: pgmap v5782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:23 smithi107 ceph-mon[32659]: pgmap v5782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:23 smithi132 ceph-mon[37580]: pgmap v5782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:25 smithi112 ceph-mon[37501]: pgmap v5783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:25 smithi107 ceph-mon[32659]: pgmap v5783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:25 smithi132 ceph-mon[37580]: pgmap v5783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:26.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10041.972907 secs 2024-07-26T15:38:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10041.972907 secs 2024-07-26T15:38:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10041.972907 secs 2024-07-26T15:38:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:27 smithi112 ceph-mon[37501]: pgmap v5784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:27 smithi107 ceph-mon[32659]: pgmap v5784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:27 smithi132 ceph-mon[37580]: pgmap v5784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:29 smithi112 ceph-mon[37501]: pgmap v5785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:29 smithi107 ceph-mon[32659]: pgmap v5785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:29 smithi132 ceph-mon[37580]: pgmap v5785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:30.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:30.171+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:30.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:30.171+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:30.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:30.171+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:38:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:38:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:38:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10046.973039 secs 2024-07-26T15:38:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:31 smithi112 ceph-mon[37501]: pgmap v5786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:38:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:38:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:38:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10046.973039 secs 2024-07-26T15:38:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:31 smithi107 ceph-mon[32659]: pgmap v5786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:38:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:38:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:38:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10046.973039 secs 2024-07-26T15:38:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:31 smithi132 ceph-mon[37580]: pgmap v5786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:38:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:38:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:38:32.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:32.578+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:32.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:32.578+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:32.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:32.578+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:32 smithi112 ceph-mon[37501]: pgmap v5787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:38:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:32 smithi112 ceph-mon[37501]: pgmap v5788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:38:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:32 smithi107 ceph-mon[32659]: pgmap v5787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:38:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:32 smithi107 ceph-mon[32659]: pgmap v5788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:38:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:32 smithi132 ceph-mon[37580]: pgmap v5787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:38:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:32 smithi132 ceph-mon[37580]: pgmap v5788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:38:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:34 smithi112 ceph-mon[37501]: pgmap v5789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:38:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:34 smithi107 ceph-mon[32659]: pgmap v5789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:38:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:34 smithi132 ceph-mon[37580]: pgmap v5789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:38:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10051.973164 secs 2024-07-26T15:38:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:36 smithi112 ceph-mon[37501]: pgmap v5790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10051.973164 secs 2024-07-26T15:38:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:36 smithi107 ceph-mon[32659]: pgmap v5790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10051.973164 secs 2024-07-26T15:38:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:36 smithi132 ceph-mon[37580]: pgmap v5790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:38 smithi112 ceph-mon[37501]: pgmap v5791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:38:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:38 smithi107 ceph-mon[32659]: pgmap v5791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:38:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:38 smithi132 ceph-mon[37580]: pgmap v5791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:38:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:40 smithi112 ceph-mon[37501]: pgmap v5792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10056.973242 secs 2024-07-26T15:38:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:40 smithi107 ceph-mon[32659]: pgmap v5792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10056.973242 secs 2024-07-26T15:38:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:40 smithi132 ceph-mon[37580]: pgmap v5792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10056.973242 secs 2024-07-26T15:38:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:42 smithi112 ceph-mon[37501]: pgmap v5793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:38:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:42 smithi107 ceph-mon[32659]: pgmap v5793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:38:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:42 smithi132 ceph-mon[37580]: pgmap v5793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:38:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:44 smithi112 ceph-mon[37501]: pgmap v5794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:44 smithi107 ceph-mon[32659]: pgmap v5794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:44 smithi132 ceph-mon[37580]: pgmap v5794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:45.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:45.186+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:45.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:45.186+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:45.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:38:45.186+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10061.973381 secs 2024-07-26T15:38:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10061.973381 secs 2024-07-26T15:38:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10061.973381 secs 2024-07-26T15:38:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:46 smithi112 ceph-mon[37501]: pgmap v5795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:46.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:46 smithi132 ceph-mon[37580]: pgmap v5795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:46 smithi107 ceph-mon[32659]: pgmap v5795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:47.593+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:47.593+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:38:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:38:47.593+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:48 smithi112 ceph-mon[37501]: pgmap v5796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:48 smithi107 ceph-mon[32659]: pgmap v5796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:48 smithi132 ceph-mon[37580]: pgmap v5796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:50 smithi112 ceph-mon[37501]: pgmap v5797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:38:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10066.973501 secs 2024-07-26T15:38:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:50 smithi107 ceph-mon[32659]: pgmap v5797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:38:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10066.973501 secs 2024-07-26T15:38:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:50 smithi132 ceph-mon[37580]: pgmap v5797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:38:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10066.973501 secs 2024-07-26T15:38:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:52 smithi112 ceph-mon[37501]: pgmap v5798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:52 smithi107 ceph-mon[32659]: pgmap v5798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:52 smithi132 ceph-mon[37580]: pgmap v5798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:54 smithi112 ceph-mon[37501]: pgmap v5799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:54 smithi107 ceph-mon[32659]: pgmap v5799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:54 smithi132 ceph-mon[37580]: pgmap v5799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10071.973585 secs 2024-07-26T15:38:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10071.973585 secs 2024-07-26T15:38:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10071.973585 secs 2024-07-26T15:38:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:56 smithi107 ceph-mon[32659]: pgmap v5800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:56 smithi132 ceph-mon[37580]: pgmap v5800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:56 smithi112 ceph-mon[37501]: pgmap v5800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:38:58 smithi107 ceph-mon[32659]: pgmap v5801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:38:58 smithi132 ceph-mon[37580]: pgmap v5801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:38:58 smithi112 ceph-mon[37501]: pgmap v5801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:39:00.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:00.196+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:00.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:00.196+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:00.199 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:00.196+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:00 smithi107 ceph-mon[32659]: pgmap v5802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:39:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10076.973707 secs 2024-07-26T15:39:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:00 smithi132 ceph-mon[37580]: pgmap v5802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:39:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10076.973707 secs 2024-07-26T15:39:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:00 smithi112 ceph-mon[37501]: pgmap v5802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:39:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10076.973707 secs 2024-07-26T15:39:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:02.599+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:02.599+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:02.599+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:02 smithi107 ceph-mon[32659]: pgmap v5803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:02 smithi132 ceph-mon[37580]: pgmap v5803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:02 smithi112 ceph-mon[37501]: pgmap v5803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:04 smithi107 ceph-mon[32659]: pgmap v5804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:04 smithi132 ceph-mon[37580]: pgmap v5804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:04 smithi112 ceph-mon[37501]: pgmap v5804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10081.973812 secs 2024-07-26T15:39:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10081.973812 secs 2024-07-26T15:39:06.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10081.973812 secs 2024-07-26T15:39:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:06 smithi107 ceph-mon[32659]: pgmap v5805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:06 smithi132 ceph-mon[37580]: pgmap v5805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:06 smithi112 ceph-mon[37501]: pgmap v5805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:08 smithi107 ceph-mon[32659]: pgmap v5806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:39:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:08 smithi132 ceph-mon[37580]: pgmap v5806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:39:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:08 smithi112 ceph-mon[37501]: pgmap v5806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:39:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:10 smithi107 ceph-mon[32659]: pgmap v5807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:39:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10086.973930 secs 2024-07-26T15:39:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:10 smithi132 ceph-mon[37580]: pgmap v5807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:39:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10086.973930 secs 2024-07-26T15:39:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:10 smithi112 ceph-mon[37501]: pgmap v5807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:39:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10086.973930 secs 2024-07-26T15:39:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:12 smithi107 ceph-mon[32659]: pgmap v5808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:12 smithi132 ceph-mon[37580]: pgmap v5808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:12 smithi112 ceph-mon[37501]: pgmap v5808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:14.353+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:39:14.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:14.353+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:39:14.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:14.354+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:39:14.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:14.354+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:39:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:14 smithi107 ceph-mon[32659]: pgmap v5809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:14 smithi132 ceph-mon[37580]: pgmap v5809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:14 smithi112 ceph-mon[37501]: pgmap v5809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:15.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:15.210+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:15.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:15.210+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:15.211 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:15.210+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10091.974089 secs 2024-07-26T15:39:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10091.974089 secs 2024-07-26T15:39:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10091.974089 secs 2024-07-26T15:39:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:16.547+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:39:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:16.547+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:39:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:16.548+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:39:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:16.548+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:39:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:16 smithi107 ceph-mon[32659]: pgmap v5810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:16 smithi132 ceph-mon[37580]: pgmap v5810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:16 smithi112 ceph-mon[37501]: pgmap v5810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:17.609+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:17.609+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:17.610 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:17.609+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:18 smithi132 ceph-mon[37580]: pgmap v5811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:39:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:18 smithi112 ceph-mon[37501]: pgmap v5811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:39:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:18 smithi107 ceph-mon[32659]: pgmap v5811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:39:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:20 smithi132 ceph-mon[37580]: pgmap v5812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:39:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10096.974143 secs 2024-07-26T15:39:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:20 smithi112 ceph-mon[37501]: pgmap v5812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:39:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10096.974143 secs 2024-07-26T15:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:20 smithi107 ceph-mon[32659]: pgmap v5812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:39:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10096.974143 secs 2024-07-26T15:39:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:22 smithi132 ceph-mon[37580]: pgmap v5813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:22 smithi112 ceph-mon[37501]: pgmap v5813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:22 smithi107 ceph-mon[32659]: pgmap v5813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:24 smithi132 ceph-mon[37580]: pgmap v5814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:39:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:24 smithi112 ceph-mon[37501]: pgmap v5814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:39:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:24 smithi107 ceph-mon[32659]: pgmap v5814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:39:26.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10101.974280 secs 2024-07-26T15:39:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10101.974280 secs 2024-07-26T15:39:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10101.974280 secs 2024-07-26T15:39:27.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:26 smithi112 ceph-mon[37501]: pgmap v5815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:39:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:26 smithi107 ceph-mon[32659]: pgmap v5815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:39:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:26 smithi132 ceph-mon[37580]: pgmap v5815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:39:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:28 smithi112 ceph-mon[37501]: pgmap v5816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:39:29.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:28 smithi107 ceph-mon[32659]: pgmap v5816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:39:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:28 smithi132 ceph-mon[37580]: pgmap v5816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:39:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:30.215+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:30.215+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:30.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:30.215+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:30 smithi112 ceph-mon[37501]: pgmap v5817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:39:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10106.974408 secs 2024-07-26T15:39:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:30 smithi107 ceph-mon[32659]: pgmap v5817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:39:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10106.974408 secs 2024-07-26T15:39:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:30 smithi132 ceph-mon[37580]: pgmap v5817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:39:31.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10106.974408 secs 2024-07-26T15:39:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:39:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:39:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:39:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:32.625+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:32.625+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:32.625+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:32 smithi112 ceph-mon[37501]: pgmap v5818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:39:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:39:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:39:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:32 smithi107 ceph-mon[32659]: pgmap v5818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:39:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:39:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:39:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:32 smithi132 ceph-mon[37580]: pgmap v5818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:39:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:39:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:39:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:39:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:39:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:39:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:39:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:39:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:39:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:39:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:39:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:39:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:34 smithi112 ceph-mon[37501]: pgmap v5819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:39:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:34 smithi112 ceph-mon[37501]: pgmap v5820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:34 smithi107 ceph-mon[32659]: pgmap v5819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:39:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:34 smithi107 ceph-mon[32659]: pgmap v5820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:34 smithi132 ceph-mon[37580]: pgmap v5819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:39:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:34 smithi132 ceph-mon[37580]: pgmap v5820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10111.974533 secs 2024-07-26T15:39:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10111.974533 secs 2024-07-26T15:39:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10111.974533 secs 2024-07-26T15:39:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:36 smithi112 ceph-mon[37501]: pgmap v5821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:36 smithi107 ceph-mon[32659]: pgmap v5821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:36 smithi132 ceph-mon[37580]: pgmap v5821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:38 smithi112 ceph-mon[37501]: pgmap v5822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T15:39:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:38 smithi107 ceph-mon[32659]: pgmap v5822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T15:39:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:38 smithi132 ceph-mon[37580]: pgmap v5822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T15:39:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:40 smithi112 ceph-mon[37501]: pgmap v5823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10116.974657 secs 2024-07-26T15:39:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:40 smithi107 ceph-mon[32659]: pgmap v5823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10116.974657 secs 2024-07-26T15:39:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:40 smithi132 ceph-mon[37580]: pgmap v5823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10116.974657 secs 2024-07-26T15:39:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:42 smithi112 ceph-mon[37501]: pgmap v5824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:42 smithi107 ceph-mon[32659]: pgmap v5824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:42 smithi132 ceph-mon[37580]: pgmap v5824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:39:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:44 smithi112 ceph-mon[37501]: pgmap v5825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:39:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:45.231+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:45.231+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:45.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:39:45.231+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:45.233 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:44 smithi107 ceph-mon[32659]: pgmap v5825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:39:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:44 smithi132 ceph-mon[37580]: pgmap v5825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:39:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10121.974797 secs 2024-07-26T15:39:46.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10121.974797 secs 2024-07-26T15:39:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10121.974797 secs 2024-07-26T15:39:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:46 smithi112 ceph-mon[37501]: pgmap v5826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:46 smithi107 ceph-mon[32659]: pgmap v5826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:46 smithi132 ceph-mon[37580]: pgmap v5826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:47.639+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:47.641 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:47.639+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:47.642 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:39:47.639+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:39:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:48 smithi112 ceph-mon[37501]: pgmap v5827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:39:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:48 smithi107 ceph-mon[32659]: pgmap v5827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:39:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:48 smithi132 ceph-mon[37580]: pgmap v5827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:39:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:50 smithi112 ceph-mon[37501]: pgmap v5828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:39:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10126.974919 secs 2024-07-26T15:39:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:50 smithi107 ceph-mon[32659]: pgmap v5828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:39:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10126.974919 secs 2024-07-26T15:39:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:50 smithi132 ceph-mon[37580]: pgmap v5828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:39:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10126.974919 secs 2024-07-26T15:39:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:52 smithi112 ceph-mon[37501]: pgmap v5829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:52 smithi107 ceph-mon[32659]: pgmap v5829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:52 smithi132 ceph-mon[37580]: pgmap v5829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:54 smithi112 ceph-mon[37501]: pgmap v5830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:55.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:54 smithi107 ceph-mon[32659]: pgmap v5830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:54 smithi132 ceph-mon[37580]: pgmap v5830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10131.975060 secs 2024-07-26T15:39:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10131.975060 secs 2024-07-26T15:39:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10131.975060 secs 2024-07-26T15:39:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:56 smithi112 ceph-mon[37501]: pgmap v5831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:56 smithi107 ceph-mon[32659]: pgmap v5831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:56 smithi132 ceph-mon[37580]: pgmap v5831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:39:58 smithi107 ceph-mon[32659]: pgmap v5832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:39:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:39:58 smithi132 ceph-mon[37580]: pgmap v5832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:39:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:39:58 smithi112 ceph-mon[37501]: pgmap v5832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:00.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:00.246+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:00.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:00.246+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:00.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:00.246+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:00 smithi107 ceph-mon[32659]: pgmap v5833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:40:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10136.975197 secs 2024-07-26T15:40:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:00 smithi132 ceph-mon[37580]: pgmap v5833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:40:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10136.975197 secs 2024-07-26T15:40:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:00 smithi112 ceph-mon[37501]: pgmap v5833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:40:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10136.975197 secs 2024-07-26T15:40:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:02.645+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:02.645+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:02.647 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:02.645+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:02 smithi107 ceph-mon[32659]: pgmap v5834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:02 smithi132 ceph-mon[37580]: pgmap v5834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:02 smithi112 ceph-mon[37501]: pgmap v5834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:04 smithi107 ceph-mon[32659]: pgmap v5835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:04 smithi132 ceph-mon[37580]: pgmap v5835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:04 smithi112 ceph-mon[37501]: pgmap v5835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:06.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10141.975320 secs 2024-07-26T15:40:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10141.975320 secs 2024-07-26T15:40:06.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10141.975320 secs 2024-07-26T15:40:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:06 smithi107 ceph-mon[32659]: pgmap v5836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:06 smithi132 ceph-mon[37580]: pgmap v5836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:06 smithi112 ceph-mon[37501]: pgmap v5836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:08 smithi107 ceph-mon[32659]: pgmap v5837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:08 smithi132 ceph-mon[37580]: pgmap v5837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:09.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:08 smithi112 ceph-mon[37501]: pgmap v5837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:10 smithi107 ceph-mon[32659]: pgmap v5838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10146.975429 secs 2024-07-26T15:40:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:10 smithi132 ceph-mon[37580]: pgmap v5838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10146.975429 secs 2024-07-26T15:40:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:10 smithi112 ceph-mon[37501]: pgmap v5838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10146.975429 secs 2024-07-26T15:40:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:12 smithi107 ceph-mon[32659]: pgmap v5839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:12 smithi132 ceph-mon[37580]: pgmap v5839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:12 smithi112 ceph-mon[37501]: pgmap v5839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:15.262+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:15.262+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:15.264 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:15.262+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:15.264 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:15 smithi107 ceph-mon[32659]: pgmap v5840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:15 smithi132 ceph-mon[37580]: pgmap v5840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:15 smithi112 ceph-mon[37501]: pgmap v5840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10151.975533 secs 2024-07-26T15:40:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10151.975533 secs 2024-07-26T15:40:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10151.975533 secs 2024-07-26T15:40:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:17 smithi107 ceph-mon[32659]: pgmap v5841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:17 smithi132 ceph-mon[37580]: pgmap v5841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:17 smithi112 ceph-mon[37501]: pgmap v5841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:17.661+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:17.661+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:17.661+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:19 smithi107 ceph-mon[32659]: pgmap v5842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:19 smithi132 ceph-mon[37580]: pgmap v5842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:19 smithi112 ceph-mon[37501]: pgmap v5842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:21 smithi107 ceph-mon[32659]: pgmap v5843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10156.975656 secs 2024-07-26T15:40:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:21 smithi132 ceph-mon[37580]: pgmap v5843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10156.975656 secs 2024-07-26T15:40:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:21 smithi112 ceph-mon[37501]: pgmap v5843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10156.975656 secs 2024-07-26T15:40:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:23 smithi107 ceph-mon[32659]: pgmap v5844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:23 smithi132 ceph-mon[37580]: pgmap v5844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:23 smithi112 ceph-mon[37501]: pgmap v5844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:25 smithi107 ceph-mon[32659]: pgmap v5845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:25 smithi132 ceph-mon[37580]: pgmap v5845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:25 smithi112 ceph-mon[37501]: pgmap v5845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10161.975795 secs 2024-07-26T15:40:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10161.975795 secs 2024-07-26T15:40:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10161.975795 secs 2024-07-26T15:40:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:27 smithi107 ceph-mon[32659]: pgmap v5846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:27 smithi112 ceph-mon[37501]: pgmap v5846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:27 smithi132 ceph-mon[37580]: pgmap v5846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:28 smithi112 ceph-mon[37501]: pgmap v5847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:28 smithi107 ceph-mon[32659]: pgmap v5847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:28 smithi132 ceph-mon[37580]: pgmap v5847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:30.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:30.266+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:30.266+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:30.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:30.266+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:30 smithi107 ceph-mon[32659]: pgmap v5848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:30 smithi112 ceph-mon[37501]: pgmap v5848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:30 smithi132 ceph-mon[37580]: pgmap v5848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:40:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10166.975916 secs 2024-07-26T15:40:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10166.975916 secs 2024-07-26T15:40:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10166.975916 secs 2024-07-26T15:40:32.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:32.677+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:32.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:32.677+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:32.678 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:32.677+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:32 smithi112 ceph-mon[37501]: pgmap v5849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:32 smithi107 ceph-mon[32659]: pgmap v5849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:32 smithi132 ceph-mon[37580]: pgmap v5849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:34 smithi112 ceph-mon[37501]: pgmap v5850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:40:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.715 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:34 smithi132 ceph-mon[37580]: pgmap v5850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:34.715 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:40:34.715 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.715 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.715 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.716 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:34 smithi107 ceph-mon[32659]: pgmap v5850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:34.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:40:34.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:34.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10171.975986 secs 2024-07-26T15:40:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:36 smithi112 ceph-mon[37501]: pgmap v5851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10171.975986 secs 2024-07-26T15:40:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:36 smithi107 ceph-mon[32659]: pgmap v5851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10171.975986 secs 2024-07-26T15:40:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:36 smithi132 ceph-mon[37580]: pgmap v5851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:39 smithi107 ceph-mon[32659]: pgmap v5852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:39.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:39.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:40:39.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:40:39.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:39 smithi132 ceph-mon[37580]: pgmap v5852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:40:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:40:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:39 smithi112 ceph-mon[37501]: pgmap v5852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:40:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:40:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:40:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:40:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:40 smithi107 ceph-mon[32659]: pgmap v5853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:40:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:40 smithi107 ceph-mon[32659]: pgmap v5854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:40:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:40 smithi132 ceph-mon[37580]: pgmap v5853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:40:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:40 smithi132 ceph-mon[37580]: pgmap v5854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:40:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:40 smithi112 ceph-mon[37501]: pgmap v5853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:40:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:40 smithi112 ceph-mon[37501]: pgmap v5854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:40:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10176.976140 secs 2024-07-26T15:40:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10176.976140 secs 2024-07-26T15:40:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10176.976140 secs 2024-07-26T15:40:42.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:42 smithi107 ceph-mon[32659]: pgmap v5855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:40:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:42 smithi132 ceph-mon[37580]: pgmap v5855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:40:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:42 smithi112 ceph-mon[37501]: pgmap v5855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:40:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:44 smithi107 ceph-mon[32659]: pgmap v5856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:40:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:44 smithi132 ceph-mon[37580]: pgmap v5856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:40:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:44 smithi112 ceph-mon[37501]: pgmap v5856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:40:45.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:45.267+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:45.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:45.267+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:45.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:40:45.267+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:46 smithi107 ceph-mon[32659]: pgmap v5857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:40:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10181.976225 secs 2024-07-26T15:40:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:46 smithi132 ceph-mon[37580]: pgmap v5857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:40:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10181.976225 secs 2024-07-26T15:40:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:46 smithi112 ceph-mon[37501]: pgmap v5857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:40:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10181.976225 secs 2024-07-26T15:40:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:47.692+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:47.692+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:40:47.694 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:40:47.692+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:48 smithi107 ceph-mon[32659]: pgmap v5858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:40:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:48 smithi132 ceph-mon[37580]: pgmap v5858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:40:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:48 smithi112 ceph-mon[37501]: pgmap v5858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:40:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:50 smithi107 ceph-mon[32659]: pgmap v5859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:40:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:50 smithi132 ceph-mon[37580]: pgmap v5859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:40:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:50 smithi112 ceph-mon[37501]: pgmap v5859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:40:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10186.976295 secs 2024-07-26T15:40:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10186.976295 secs 2024-07-26T15:40:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10186.976295 secs 2024-07-26T15:40:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:52 smithi132 ceph-mon[37580]: pgmap v5860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:52 smithi112 ceph-mon[37501]: pgmap v5860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:52 smithi107 ceph-mon[32659]: pgmap v5860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:54 smithi112 ceph-mon[37501]: pgmap v5861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:40:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:54 smithi107 ceph-mon[32659]: pgmap v5861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:40:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:54 smithi132 ceph-mon[37580]: pgmap v5861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:40:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:56 smithi112 ceph-mon[37501]: pgmap v5862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:40:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10191.976422 secs 2024-07-26T15:40:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:56 smithi107 ceph-mon[32659]: pgmap v5862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:40:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10191.976422 secs 2024-07-26T15:40:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:56 smithi132 ceph-mon[37580]: pgmap v5862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:40:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10191.976422 secs 2024-07-26T15:40:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:58 smithi112 ceph-mon[37501]: pgmap v5863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:58 smithi107 ceph-mon[32659]: pgmap v5863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:58 smithi132 ceph-mon[37580]: pgmap v5863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:40:59 smithi112 ceph-mon[37501]: pgmap v5864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:40:59 smithi107 ceph-mon[32659]: pgmap v5864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:40:59 smithi132 ceph-mon[37580]: pgmap v5864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:00.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:00.282+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:00.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:00.282+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:00.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:00.282+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10196.976521 secs 2024-07-26T15:41:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:01 smithi112 ceph-mon[37501]: pgmap v5865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10196.976521 secs 2024-07-26T15:41:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:01 smithi107 ceph-mon[32659]: pgmap v5865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10196.976521 secs 2024-07-26T15:41:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:01 smithi132 ceph-mon[37580]: pgmap v5865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:02.708+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:02.708+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:02.708+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:03 smithi112 ceph-mon[37501]: pgmap v5866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:03 smithi107 ceph-mon[32659]: pgmap v5866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:03 smithi132 ceph-mon[37580]: pgmap v5866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:05 smithi112 ceph-mon[37501]: pgmap v5867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:05 smithi107 ceph-mon[32659]: pgmap v5867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:05 smithi132 ceph-mon[37580]: pgmap v5867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10201.976599 secs 2024-07-26T15:41:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10201.976599 secs 2024-07-26T15:41:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10201.976599 secs 2024-07-26T15:41:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:07 smithi112 ceph-mon[37501]: pgmap v5868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:07 smithi107 ceph-mon[32659]: pgmap v5868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:07 smithi132 ceph-mon[37580]: pgmap v5868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:09 smithi112 ceph-mon[37501]: pgmap v5869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:09 smithi107 ceph-mon[32659]: pgmap v5869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:09 smithi132 ceph-mon[37580]: pgmap v5869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10206.976717 secs 2024-07-26T15:41:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:11 smithi112 ceph-mon[37501]: pgmap v5870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10206.976717 secs 2024-07-26T15:41:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:11 smithi107 ceph-mon[32659]: pgmap v5870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10206.976717 secs 2024-07-26T15:41:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:11 smithi132 ceph-mon[37580]: pgmap v5870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:13 smithi112 ceph-mon[37501]: pgmap v5871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:13 smithi107 ceph-mon[32659]: pgmap v5871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:13 smithi132 ceph-mon[37580]: pgmap v5871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:15.291+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:15.291+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:15.291+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:15 smithi112 ceph-mon[37501]: pgmap v5872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:15 smithi107 ceph-mon[32659]: pgmap v5872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:15 smithi132 ceph-mon[37580]: pgmap v5872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10211.976849 secs 2024-07-26T15:41:16.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10211.976849 secs 2024-07-26T15:41:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10211.976849 secs 2024-07-26T15:41:17.725 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:17.724+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:17.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:17.724+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:17.726 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:17.724+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:17 smithi112 ceph-mon[37501]: pgmap v5873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:17 smithi107 ceph-mon[32659]: pgmap v5873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:17 smithi132 ceph-mon[37580]: pgmap v5873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:19 smithi112 ceph-mon[37501]: pgmap v5874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:19 smithi107 ceph-mon[32659]: pgmap v5874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:19 smithi132 ceph-mon[37580]: pgmap v5874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10216.976958 secs 2024-07-26T15:41:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10216.976958 secs 2024-07-26T15:41:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10216.976958 secs 2024-07-26T15:41:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:21 smithi112 ceph-mon[37501]: pgmap v5875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:21 smithi107 ceph-mon[32659]: pgmap v5875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:21 smithi132 ceph-mon[37580]: pgmap v5875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:23 smithi112 ceph-mon[37501]: pgmap v5876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:23 smithi107 ceph-mon[32659]: pgmap v5876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:23 smithi132 ceph-mon[37580]: pgmap v5876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:25 smithi112 ceph-mon[37501]: pgmap v5877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10221.977094 secs 2024-07-26T15:41:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:25 smithi107 ceph-mon[32659]: pgmap v5877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10221.977094 secs 2024-07-26T15:41:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:25 smithi132 ceph-mon[37580]: pgmap v5877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10221.977094 secs 2024-07-26T15:41:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:27 smithi107 ceph-mon[32659]: pgmap v5878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:41:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:27 smithi132 ceph-mon[37580]: pgmap v5878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:41:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:27 smithi112 ceph-mon[37501]: pgmap v5878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:41:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:29 smithi107 ceph-mon[32659]: pgmap v5879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:41:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:29 smithi132 ceph-mon[37580]: pgmap v5879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:41:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:29 smithi112 ceph-mon[37501]: pgmap v5879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:41:30.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:30.307+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:30.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:30.307+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:30.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:30.307+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10226.977214 secs 2024-07-26T15:41:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10226.977214 secs 2024-07-26T15:41:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10226.977214 secs 2024-07-26T15:41:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:31 smithi107 ceph-mon[32659]: pgmap v5880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:41:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:31 smithi132 ceph-mon[37580]: pgmap v5880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:41:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:31 smithi112 ceph-mon[37501]: pgmap v5880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:41:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:32.740+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:32.740+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:32.740+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:33 smithi107 ceph-mon[32659]: pgmap v5881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:41:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:33 smithi132 ceph-mon[37580]: pgmap v5881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:41:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:33 smithi112 ceph-mon[37501]: pgmap v5881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:41:36.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:35 smithi112 ceph-mon[37501]: pgmap v5882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:36.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10231.977311 secs 2024-07-26T15:41:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:35 smithi107 ceph-mon[32659]: pgmap v5882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10231.977311 secs 2024-07-26T15:41:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:35 smithi132 ceph-mon[37580]: pgmap v5882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:41:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10231.977311 secs 2024-07-26T15:41:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:37 smithi107 ceph-mon[32659]: pgmap v5883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:37 smithi132 ceph-mon[37580]: pgmap v5883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:37 smithi112 ceph-mon[37501]: pgmap v5883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:39 smithi107 ceph-mon[32659]: pgmap v5884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:41:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:39 smithi132 ceph-mon[37580]: pgmap v5884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:41:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:39 smithi112 ceph-mon[37501]: pgmap v5884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:41:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:41:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:41:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:40 smithi107 ceph-mon[32659]: pgmap v5885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:41:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:40 smithi107 ceph-mon[32659]: pgmap v5886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:41:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:41:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10236.977399 secs 2024-07-26T15:41:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:41:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:41:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:40 smithi132 ceph-mon[37580]: pgmap v5885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:41:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:40 smithi132 ceph-mon[37580]: pgmap v5886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:41:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:41:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10236.977399 secs 2024-07-26T15:41:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:41:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:41:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:40 smithi112 ceph-mon[37501]: pgmap v5885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:41:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:40 smithi112 ceph-mon[37501]: pgmap v5886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:41:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:41:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10236.977399 secs 2024-07-26T15:41:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:42 smithi112 ceph-mon[37501]: pgmap v5887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:42 smithi107 ceph-mon[32659]: pgmap v5887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:42 smithi132 ceph-mon[37580]: pgmap v5887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:44 smithi112 ceph-mon[37501]: pgmap v5888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:41:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:44 smithi107 ceph-mon[32659]: pgmap v5888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:41:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:44 smithi132 ceph-mon[37580]: pgmap v5888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:41:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:45.322+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:45.322+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:45.325 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:41:45.322+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10241.977485 secs 2024-07-26T15:41:46.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10241.977485 secs 2024-07-26T15:41:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10241.977485 secs 2024-07-26T15:41:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:46 smithi112 ceph-mon[37501]: pgmap v5889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:41:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:46 smithi107 ceph-mon[32659]: pgmap v5889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:41:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:46 smithi132 ceph-mon[37580]: pgmap v5889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:41:47.756 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:47.755+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:47.755+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:41:47.755+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:41:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:48 smithi112 ceph-mon[37501]: pgmap v5890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:41:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:48 smithi107 ceph-mon[32659]: pgmap v5890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:41:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:48 smithi132 ceph-mon[37580]: pgmap v5890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:41:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:50 smithi112 ceph-mon[37501]: pgmap v5891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:41:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10246.977631 secs 2024-07-26T15:41:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:50 smithi107 ceph-mon[32659]: pgmap v5891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:41:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10246.977631 secs 2024-07-26T15:41:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:50 smithi132 ceph-mon[37580]: pgmap v5891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:41:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10246.977631 secs 2024-07-26T15:41:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:52 smithi112 ceph-mon[37501]: pgmap v5892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:52 smithi107 ceph-mon[32659]: pgmap v5892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:52 smithi132 ceph-mon[37580]: pgmap v5892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:54 smithi112 ceph-mon[37501]: pgmap v5893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:54 smithi107 ceph-mon[32659]: pgmap v5893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:54 smithi132 ceph-mon[37580]: pgmap v5893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10251.977716 secs 2024-07-26T15:41:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10251.977716 secs 2024-07-26T15:41:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10251.977716 secs 2024-07-26T15:41:57.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:56 smithi107 ceph-mon[32659]: pgmap v5894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:56 smithi112 ceph-mon[37501]: pgmap v5894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:56 smithi132 ceph-mon[37580]: pgmap v5894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:41:58 smithi112 ceph-mon[37501]: pgmap v5895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:41:58 smithi107 ceph-mon[32659]: pgmap v5895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:41:58 smithi132 ceph-mon[37580]: pgmap v5895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:00.339 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:00.337+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:00.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:00.337+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:00.340 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:00.337+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:00 smithi112 ceph-mon[37501]: pgmap v5896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10256.977856 secs 2024-07-26T15:42:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:00 smithi107 ceph-mon[32659]: pgmap v5896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10256.977856 secs 2024-07-26T15:42:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:00 smithi132 ceph-mon[37580]: pgmap v5896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10256.977856 secs 2024-07-26T15:42:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:02.771+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:02.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:02.771+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:02.773 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:02.771+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:02 smithi112 ceph-mon[37501]: pgmap v5897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:02 smithi107 ceph-mon[32659]: pgmap v5897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:02 smithi132 ceph-mon[37580]: pgmap v5897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:04 smithi112 ceph-mon[37501]: pgmap v5898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:04 smithi107 ceph-mon[32659]: pgmap v5898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:04 smithi132 ceph-mon[37580]: pgmap v5898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10261.977964 secs 2024-07-26T15:42:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10261.977964 secs 2024-07-26T15:42:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10261.977964 secs 2024-07-26T15:42:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:06 smithi112 ceph-mon[37501]: pgmap v5899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:06 smithi107 ceph-mon[32659]: pgmap v5899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:06 smithi132 ceph-mon[37580]: pgmap v5899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:08 smithi112 ceph-mon[37501]: pgmap v5900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:08 smithi107 ceph-mon[32659]: pgmap v5900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:08 smithi132 ceph-mon[37580]: pgmap v5900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:10 smithi112 ceph-mon[37501]: pgmap v5901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10266.978100 secs 2024-07-26T15:42:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:10 smithi107 ceph-mon[32659]: pgmap v5901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10266.978100 secs 2024-07-26T15:42:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:10 smithi132 ceph-mon[37580]: pgmap v5901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10266.978100 secs 2024-07-26T15:42:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:12 smithi112 ceph-mon[37501]: pgmap v5902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:12 smithi107 ceph-mon[32659]: pgmap v5902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:12 smithi132 ceph-mon[37580]: pgmap v5902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:14.384+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:42:14.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:14.384+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:42:14.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:14.384+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:42:14.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:14.384+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:42:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:14 smithi112 ceph-mon[37501]: pgmap v5903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:14 smithi107 ceph-mon[32659]: pgmap v5903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:14 smithi132 ceph-mon[37580]: pgmap v5903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:15.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:15.352+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:15.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:15.352+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:15.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:15.352+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10271.978137 secs 2024-07-26T15:42:16.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10271.978137 secs 2024-07-26T15:42:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10271.978137 secs 2024-07-26T15:42:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:16.578+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:42:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:16.578+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:42:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:16.579+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:42:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:16.579+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:42:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:16 smithi112 ceph-mon[37501]: pgmap v5904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:16 smithi107 ceph-mon[32659]: pgmap v5904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:16 smithi132 ceph-mon[37580]: pgmap v5904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:17.787+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:17.787+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:17.787+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:18 smithi107 ceph-mon[32659]: pgmap v5905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:18 smithi132 ceph-mon[37580]: pgmap v5905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:18 smithi112 ceph-mon[37501]: pgmap v5905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:20 smithi107 ceph-mon[32659]: pgmap v5906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10276.978263 secs 2024-07-26T15:42:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:20 smithi132 ceph-mon[37580]: pgmap v5906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10276.978263 secs 2024-07-26T15:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:20 smithi112 ceph-mon[37501]: pgmap v5906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10276.978263 secs 2024-07-26T15:42:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:22 smithi107 ceph-mon[32659]: pgmap v5907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:22 smithi132 ceph-mon[37580]: pgmap v5907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:22 smithi112 ceph-mon[37501]: pgmap v5907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:24 smithi107 ceph-mon[32659]: pgmap v5908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:24 smithi132 ceph-mon[37580]: pgmap v5908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:24 smithi112 ceph-mon[37501]: pgmap v5908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10281.978349 secs 2024-07-26T15:42:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10281.978349 secs 2024-07-26T15:42:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10281.978349 secs 2024-07-26T15:42:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:26 smithi107 ceph-mon[32659]: pgmap v5909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:26 smithi132 ceph-mon[37580]: pgmap v5909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:26 smithi112 ceph-mon[37501]: pgmap v5909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:28 smithi107 ceph-mon[32659]: pgmap v5910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:28 smithi132 ceph-mon[37580]: pgmap v5910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:28 smithi112 ceph-mon[37501]: pgmap v5910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:30.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:30.368+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:30.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:30.368+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:30.370 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:30.368+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:30 smithi107 ceph-mon[32659]: pgmap v5911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10286.978477 secs 2024-07-26T15:42:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:30 smithi132 ceph-mon[37580]: pgmap v5911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10286.978477 secs 2024-07-26T15:42:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:30 smithi112 ceph-mon[37501]: pgmap v5911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10286.978477 secs 2024-07-26T15:42:32.803 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:32.802+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:32.803 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:32.802+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:32.803 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:32.802+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:32 smithi107 ceph-mon[32659]: pgmap v5912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:42:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:32 smithi132 ceph-mon[37580]: pgmap v5912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:42:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:32 smithi112 ceph-mon[37501]: pgmap v5912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:42:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:35 smithi107 ceph-mon[32659]: pgmap v5913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:42:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:35 smithi132 ceph-mon[37580]: pgmap v5913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:42:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:35 smithi112 ceph-mon[37501]: pgmap v5913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:42:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10291.978560 secs 2024-07-26T15:42:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10291.978560 secs 2024-07-26T15:42:36.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10291.978560 secs 2024-07-26T15:42:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:37 smithi107 ceph-mon[32659]: pgmap v5914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:42:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:37 smithi132 ceph-mon[37580]: pgmap v5914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:42:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:37 smithi112 ceph-mon[37501]: pgmap v5914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:42:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:39 smithi107 ceph-mon[32659]: pgmap v5915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:42:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:39 smithi132 ceph-mon[37580]: pgmap v5915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:42:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:39 smithi112 ceph-mon[37501]: pgmap v5915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:42:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:42:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:42:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:42:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:41 smithi107 ceph-mon[32659]: pgmap v5916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10296.978628 secs 2024-07-26T15:42:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:42:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:42:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:42:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:41 smithi132 ceph-mon[37580]: pgmap v5916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10296.978628 secs 2024-07-26T15:42:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:42:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:42:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:42:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:41 smithi112 ceph-mon[37501]: pgmap v5916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10296.978628 secs 2024-07-26T15:42:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:42:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:42:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:42:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:42 smithi132 ceph-mon[37580]: pgmap v5917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:42:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:42 smithi112 ceph-mon[37501]: pgmap v5917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:42:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:42 smithi107 ceph-mon[32659]: pgmap v5917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:42:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:44 smithi132 ceph-mon[37580]: pgmap v5918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:42:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:44 smithi112 ceph-mon[37501]: pgmap v5918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:42:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:44 smithi107 ceph-mon[32659]: pgmap v5918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:45.384+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:45.384+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:42:45.384+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:46 smithi132 ceph-mon[37580]: pgmap v5919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:42:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10301.978768 secs 2024-07-26T15:42:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:46 smithi112 ceph-mon[37501]: pgmap v5919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:42:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10301.978768 secs 2024-07-26T15:42:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:46 smithi107 ceph-mon[32659]: pgmap v5919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:42:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10301.978768 secs 2024-07-26T15:42:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:47.818+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:47.818+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:42:47.819 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:42:47.818+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:48 smithi132 ceph-mon[37580]: pgmap v5920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:42:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:48 smithi112 ceph-mon[37501]: pgmap v5920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:42:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:48 smithi107 ceph-mon[32659]: pgmap v5920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:42:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:50 smithi112 ceph-mon[37501]: pgmap v5921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:42:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:50 smithi107 ceph-mon[32659]: pgmap v5921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:42:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:50 smithi132 ceph-mon[37580]: pgmap v5921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:42:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10306.978897 secs 2024-07-26T15:42:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10306.978897 secs 2024-07-26T15:42:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10306.978897 secs 2024-07-26T15:42:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:52 smithi112 ceph-mon[37501]: pgmap v5922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:42:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:52 smithi107 ceph-mon[32659]: pgmap v5922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:42:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:52 smithi132 ceph-mon[37580]: pgmap v5922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:42:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:54 smithi112 ceph-mon[37501]: pgmap v5923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:54 smithi107 ceph-mon[32659]: pgmap v5923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:54 smithi132 ceph-mon[37580]: pgmap v5923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:56 smithi112 ceph-mon[37501]: pgmap v5924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10311.979043 secs 2024-07-26T15:42:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:56 smithi107 ceph-mon[32659]: pgmap v5924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10311.979043 secs 2024-07-26T15:42:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:56 smithi132 ceph-mon[37580]: pgmap v5924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10311.979043 secs 2024-07-26T15:42:58.347 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:58 smithi107 ceph-mon[32659]: pgmap v5925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:58 smithi112 ceph-mon[37501]: pgmap v5925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:58 smithi132 ceph-mon[37580]: pgmap v5925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:42:59 smithi112 ceph-mon[37501]: pgmap v5926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:42:59 smithi107 ceph-mon[32659]: pgmap v5926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:42:59 smithi132 ceph-mon[37580]: pgmap v5926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:00.396+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:00.396+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:00.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:00.396+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10316.979162 secs 2024-07-26T15:43:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10316.979162 secs 2024-07-26T15:43:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10316.979162 secs 2024-07-26T15:43:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:01 smithi107 ceph-mon[32659]: pgmap v5927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:01 smithi132 ceph-mon[37580]: pgmap v5927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:01 smithi112 ceph-mon[37501]: pgmap v5927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:02.833+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:02.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:02.833+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:02.835 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:02.833+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:03 smithi107 ceph-mon[32659]: pgmap v5928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:03 smithi132 ceph-mon[37580]: pgmap v5928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:03 smithi112 ceph-mon[37501]: pgmap v5928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:06.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:05 smithi112 ceph-mon[37501]: pgmap v5929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:06.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10321.979280 secs 2024-07-26T15:43:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:05 smithi107 ceph-mon[32659]: pgmap v5929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10321.979280 secs 2024-07-26T15:43:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:05 smithi132 ceph-mon[37580]: pgmap v5929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10321.979280 secs 2024-07-26T15:43:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:07 smithi107 ceph-mon[32659]: pgmap v5930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:07 smithi132 ceph-mon[37580]: pgmap v5930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:07 smithi112 ceph-mon[37501]: pgmap v5930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:10 smithi107 ceph-mon[32659]: pgmap v5931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:10 smithi132 ceph-mon[37580]: pgmap v5931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:10 smithi112 ceph-mon[37501]: pgmap v5931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10326.979363 secs 2024-07-26T15:43:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10326.979363 secs 2024-07-26T15:43:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10326.979363 secs 2024-07-26T15:43:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:12 smithi107 ceph-mon[32659]: pgmap v5932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:12 smithi132 ceph-mon[37580]: pgmap v5932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:12 smithi112 ceph-mon[37501]: pgmap v5932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:14 smithi107 ceph-mon[32659]: pgmap v5933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:14 smithi132 ceph-mon[37580]: pgmap v5933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:14 smithi112 ceph-mon[37501]: pgmap v5933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:15.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:15.412+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:15.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:15.412+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:15.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:15.412+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:16 smithi107 ceph-mon[32659]: pgmap v5934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10331.979479 secs 2024-07-26T15:43:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:16 smithi132 ceph-mon[37580]: pgmap v5934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10331.979479 secs 2024-07-26T15:43:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:16 smithi112 ceph-mon[37501]: pgmap v5934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10331.979479 secs 2024-07-26T15:43:17.849 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:17.848+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:17.850 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:17.848+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:17.850 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:17.849+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:18 smithi107 ceph-mon[32659]: pgmap v5935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:18 smithi132 ceph-mon[37580]: pgmap v5935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:18 smithi112 ceph-mon[37501]: pgmap v5935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:20 smithi107 ceph-mon[32659]: pgmap v5936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:20 smithi132 ceph-mon[37580]: pgmap v5936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:20 smithi112 ceph-mon[37501]: pgmap v5936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10336.979564 secs 2024-07-26T15:43:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10336.979564 secs 2024-07-26T15:43:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10336.979564 secs 2024-07-26T15:43:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:22 smithi132 ceph-mon[37580]: pgmap v5937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:22 smithi112 ceph-mon[37501]: pgmap v5937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:22 smithi107 ceph-mon[32659]: pgmap v5937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:24 smithi132 ceph-mon[37580]: pgmap v5938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:24 smithi112 ceph-mon[37501]: pgmap v5938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:24 smithi107 ceph-mon[32659]: pgmap v5938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:26 smithi112 ceph-mon[37501]: pgmap v5939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10341.979693 secs 2024-07-26T15:43:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:26 smithi107 ceph-mon[32659]: pgmap v5939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10341.979693 secs 2024-07-26T15:43:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:26 smithi132 ceph-mon[37580]: pgmap v5939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10341.979693 secs 2024-07-26T15:43:28.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:28 smithi107 ceph-mon[32659]: pgmap v5940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:28 smithi112 ceph-mon[37501]: pgmap v5940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:28 smithi132 ceph-mon[37580]: pgmap v5940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:30.428+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:30.428+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:30.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:30.428+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:30.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:30 smithi107 ceph-mon[32659]: pgmap v5941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:30 smithi112 ceph-mon[37501]: pgmap v5941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:30 smithi132 ceph-mon[37580]: pgmap v5941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10346.979795 secs 2024-07-26T15:43:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10346.979795 secs 2024-07-26T15:43:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10346.979795 secs 2024-07-26T15:43:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:32 smithi112 ceph-mon[37501]: pgmap v5942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:32 smithi107 ceph-mon[32659]: pgmap v5942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:32 smithi132 ceph-mon[37580]: pgmap v5942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:32.850+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:32.850+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:32.850+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:34 smithi112 ceph-mon[37501]: pgmap v5943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:34 smithi107 ceph-mon[32659]: pgmap v5943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:34 smithi132 ceph-mon[37580]: pgmap v5943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:36 smithi112 ceph-mon[37501]: pgmap v5944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10351.979892 secs 2024-07-26T15:43:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:36 smithi107 ceph-mon[32659]: pgmap v5944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10351.979892 secs 2024-07-26T15:43:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:36 smithi132 ceph-mon[37580]: pgmap v5944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10351.979892 secs 2024-07-26T15:43:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:38 smithi112 ceph-mon[37501]: pgmap v5945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:38 smithi107 ceph-mon[32659]: pgmap v5945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:38 smithi132 ceph-mon[37580]: pgmap v5945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:39 smithi112 ceph-mon[37501]: pgmap v5946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:39 smithi107 ceph-mon[32659]: pgmap v5946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:39 smithi132 ceph-mon[37580]: pgmap v5946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10356.980009 secs 2024-07-26T15:43:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10356.980009 secs 2024-07-26T15:43:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10356.980009 secs 2024-07-26T15:43:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:41 smithi107 ceph-mon[32659]: pgmap v5947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:43:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:41 smithi132 ceph-mon[37580]: pgmap v5947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:43:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:41 smithi112 ceph-mon[37501]: pgmap v5947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:43:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:43:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:43:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:42 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:43:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:43:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:43:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:42 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:43:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:43:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:43:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:42 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:43:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:43 smithi107 ceph-mon[32659]: pgmap v5948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:43:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:43 smithi132 ceph-mon[37580]: pgmap v5948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:43:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:43 smithi112 ceph-mon[37501]: pgmap v5948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:43:45.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:45.431+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:45.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:45.431+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:45.433 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:43:45.431+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:45 smithi107 ceph-mon[32659]: pgmap v5949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:43:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10361.980132 secs 2024-07-26T15:43:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:45 smithi132 ceph-mon[37580]: pgmap v5949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:43:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10361.980132 secs 2024-07-26T15:43:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:45 smithi112 ceph-mon[37501]: pgmap v5949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:43:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10361.980132 secs 2024-07-26T15:43:47.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:47.857+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:47.858 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:47.857+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:47.859 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:43:47.857+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:43:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:47 smithi107 ceph-mon[32659]: pgmap v5950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:43:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:47 smithi132 ceph-mon[37580]: pgmap v5950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:43:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:47 smithi112 ceph-mon[37501]: pgmap v5950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:43:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:49 smithi132 ceph-mon[37580]: pgmap v5951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:43:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:49 smithi112 ceph-mon[37501]: pgmap v5951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:43:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:49 smithi107 ceph-mon[32659]: pgmap v5951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:43:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10366.980249 secs 2024-07-26T15:43:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10366.980249 secs 2024-07-26T15:43:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10366.980249 secs 2024-07-26T15:43:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:51 smithi132 ceph-mon[37580]: pgmap v5952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:43:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:51 smithi112 ceph-mon[37501]: pgmap v5952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:43:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:51 smithi107 ceph-mon[32659]: pgmap v5952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:43:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:53 smithi112 ceph-mon[37501]: pgmap v5953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:43:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:53 smithi107 ceph-mon[32659]: pgmap v5953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:43:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:53 smithi132 ceph-mon[37580]: pgmap v5953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:43:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:55 smithi112 ceph-mon[37501]: pgmap v5954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10371.980393 secs 2024-07-26T15:43:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:55 smithi107 ceph-mon[32659]: pgmap v5954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10371.980393 secs 2024-07-26T15:43:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:55 smithi132 ceph-mon[37580]: pgmap v5954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10371.980393 secs 2024-07-26T15:43:58.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:57 smithi107 ceph-mon[32659]: pgmap v5955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:57 smithi112 ceph-mon[37501]: pgmap v5955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:57 smithi132 ceph-mon[37580]: pgmap v5955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:43:59 smithi112 ceph-mon[37501]: pgmap v5956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:43:59 smithi107 ceph-mon[32659]: pgmap v5956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:43:59 smithi132 ceph-mon[37580]: pgmap v5956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:00.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:00.446+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:00.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:00.446+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:00.448 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:00.446+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10376.980477 secs 2024-07-26T15:44:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10376.980477 secs 2024-07-26T15:44:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10376.980477 secs 2024-07-26T15:44:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:01 smithi112 ceph-mon[37501]: pgmap v5957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:01 smithi107 ceph-mon[32659]: pgmap v5957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:01 smithi132 ceph-mon[37580]: pgmap v5957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:02.872+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:02.872+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:02.874 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:02.872+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:03 smithi112 ceph-mon[37501]: pgmap v5958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:03 smithi107 ceph-mon[32659]: pgmap v5958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:03 smithi132 ceph-mon[37580]: pgmap v5958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:05 smithi112 ceph-mon[37501]: pgmap v5959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10381.980593 secs 2024-07-26T15:44:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:05 smithi107 ceph-mon[32659]: pgmap v5959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10381.980593 secs 2024-07-26T15:44:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:05 smithi132 ceph-mon[37580]: pgmap v5959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10381.980593 secs 2024-07-26T15:44:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:07 smithi112 ceph-mon[37501]: pgmap v5960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:07 smithi107 ceph-mon[32659]: pgmap v5960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:07 smithi132 ceph-mon[37580]: pgmap v5960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:09 smithi112 ceph-mon[37501]: pgmap v5961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:09 smithi107 ceph-mon[32659]: pgmap v5961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:09 smithi132 ceph-mon[37580]: pgmap v5961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10386.980694 secs 2024-07-26T15:44:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10386.980694 secs 2024-07-26T15:44:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10386.980694 secs 2024-07-26T15:44:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:11 smithi112 ceph-mon[37501]: pgmap v5962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:11 smithi107 ceph-mon[32659]: pgmap v5962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:11 smithi132 ceph-mon[37580]: pgmap v5962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:13 smithi112 ceph-mon[37501]: pgmap v5963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:13 smithi107 ceph-mon[32659]: pgmap v5963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:13 smithi132 ceph-mon[37580]: pgmap v5963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:15.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:15.460+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:15.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:15.460+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:15.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:15.460+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:15 smithi112 ceph-mon[37501]: pgmap v5964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10391.980763 secs 2024-07-26T15:44:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:15 smithi107 ceph-mon[32659]: pgmap v5964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10391.980763 secs 2024-07-26T15:44:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:15 smithi132 ceph-mon[37580]: pgmap v5964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10391.980763 secs 2024-07-26T15:44:17.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:17.874+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:17.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:17.874+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:17.875 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:17.874+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:17 smithi112 ceph-mon[37501]: pgmap v5965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:17 smithi107 ceph-mon[32659]: pgmap v5965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:17 smithi132 ceph-mon[37580]: pgmap v5965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:19 smithi112 ceph-mon[37501]: pgmap v5966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:19 smithi107 ceph-mon[32659]: pgmap v5966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:19 smithi132 ceph-mon[37580]: pgmap v5966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10396.980901 secs 2024-07-26T15:44:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10396.980901 secs 2024-07-26T15:44:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10396.980901 secs 2024-07-26T15:44:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:21 smithi112 ceph-mon[37501]: pgmap v5967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:21 smithi107 ceph-mon[32659]: pgmap v5967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:21 smithi132 ceph-mon[37580]: pgmap v5967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:23 smithi112 ceph-mon[37501]: pgmap v5968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:23 smithi107 ceph-mon[32659]: pgmap v5968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:23 smithi132 ceph-mon[37580]: pgmap v5968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:25 smithi112 ceph-mon[37501]: pgmap v5969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10401.981021 secs 2024-07-26T15:44:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:25 smithi107 ceph-mon[32659]: pgmap v5969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10401.981021 secs 2024-07-26T15:44:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:25 smithi132 ceph-mon[37580]: pgmap v5969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10401.981021 secs 2024-07-26T15:44:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:27 smithi112 ceph-mon[37501]: pgmap v5970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:27 smithi107 ceph-mon[32659]: pgmap v5970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:27 smithi132 ceph-mon[37580]: pgmap v5970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:28.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:28.408+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:44:28.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:28.408+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:44:28.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:28.409+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:44:28.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:28.409+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:44:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:29 smithi112 ceph-mon[37501]: pgmap v5971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:29 smithi107 ceph-mon[32659]: pgmap v5971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:29 smithi132 ceph-mon[37580]: pgmap v5971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:30.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:30.461+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:30.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:30.461+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:30.463 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:30.461+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:30.602+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:44:30.603 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:30.602+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:44:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:30.603+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:44:30.604 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:30.603+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:44:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10406.981143 secs 2024-07-26T15:44:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10406.981143 secs 2024-07-26T15:44:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10406.981143 secs 2024-07-26T15:44:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:31 smithi107 ceph-mon[32659]: pgmap v5972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:31 smithi132 ceph-mon[37580]: pgmap v5972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:31 smithi112 ceph-mon[37501]: pgmap v5972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:32.890+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:32.890+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:32.890+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:33 smithi107 ceph-mon[32659]: pgmap v5973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:33 smithi132 ceph-mon[37580]: pgmap v5973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:33 smithi112 ceph-mon[37501]: pgmap v5973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:36.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:35 smithi112 ceph-mon[37501]: pgmap v5974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:36.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10411.981230 secs 2024-07-26T15:44:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:35 smithi107 ceph-mon[32659]: pgmap v5974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10411.981230 secs 2024-07-26T15:44:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:35 smithi132 ceph-mon[37580]: pgmap v5974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10411.981230 secs 2024-07-26T15:44:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:37 smithi107 ceph-mon[32659]: pgmap v5975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:37 smithi132 ceph-mon[37580]: pgmap v5975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:37 smithi112 ceph-mon[37501]: pgmap v5975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:39 smithi107 ceph-mon[32659]: pgmap v5976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:39 smithi132 ceph-mon[37580]: pgmap v5976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:39 smithi112 ceph-mon[37501]: pgmap v5976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:44:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10416.981300 secs 2024-07-26T15:44:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10416.981300 secs 2024-07-26T15:44:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10416.981300 secs 2024-07-26T15:44:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:41 smithi107 ceph-mon[32659]: pgmap v5977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:41 smithi132 ceph-mon[37580]: pgmap v5977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:41 smithi112 ceph-mon[37501]: pgmap v5977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:44:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:44:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:44:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:44 smithi107 ceph-mon[32659]: pgmap v5978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:44:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:44:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:44 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:44:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:44 smithi132 ceph-mon[37580]: pgmap v5978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:44:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:44:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:44 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:44:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:44 smithi112 ceph-mon[37501]: pgmap v5978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:44:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:44:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:44 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:44:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:45 smithi107 ceph-mon[32659]: pgmap v5979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:44:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:45 smithi107 ceph-mon[32659]: pgmap v5980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:45 smithi132 ceph-mon[37580]: pgmap v5979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:44:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:45 smithi132 ceph-mon[37580]: pgmap v5980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:45 smithi112 ceph-mon[37501]: pgmap v5979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:44:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:45 smithi112 ceph-mon[37501]: pgmap v5980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:45.477+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:45.477+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:45.478 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:44:45.477+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10421.981387 secs 2024-07-26T15:44:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10421.981387 secs 2024-07-26T15:44:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10421.981387 secs 2024-07-26T15:44:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:47 smithi107 ceph-mon[32659]: pgmap v5981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:47 smithi132 ceph-mon[37580]: pgmap v5981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:47 smithi112 ceph-mon[37501]: pgmap v5981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:47.905+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:47.905+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:44:47.905+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:44:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:49 smithi107 ceph-mon[32659]: pgmap v5982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:49 smithi132 ceph-mon[37580]: pgmap v5982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:49 smithi112 ceph-mon[37501]: pgmap v5982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:51 smithi107 ceph-mon[32659]: pgmap v5983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10426.981516 secs 2024-07-26T15:44:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:51 smithi132 ceph-mon[37580]: pgmap v5983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10426.981516 secs 2024-07-26T15:44:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:51 smithi112 ceph-mon[37501]: pgmap v5983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10426.981516 secs 2024-07-26T15:44:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:53 smithi132 ceph-mon[37580]: pgmap v5984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:53 smithi112 ceph-mon[37501]: pgmap v5984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:53 smithi107 ceph-mon[32659]: pgmap v5984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:55 smithi132 ceph-mon[37580]: pgmap v5985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:44:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:55 smithi112 ceph-mon[37501]: pgmap v5985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:44:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:55 smithi107 ceph-mon[32659]: pgmap v5985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:44:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10431.981605 secs 2024-07-26T15:44:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10431.981605 secs 2024-07-26T15:44:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10431.981605 secs 2024-07-26T15:44:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:57 smithi112 ceph-mon[37501]: pgmap v5986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:57 smithi107 ceph-mon[32659]: pgmap v5986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:57 smithi132 ceph-mon[37580]: pgmap v5986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:59.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:44:59 smithi107 ceph-mon[32659]: pgmap v5987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:44:59 smithi112 ceph-mon[37501]: pgmap v5987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:44:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:44:59 smithi132 ceph-mon[37580]: pgmap v5987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:00.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:00.490+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:00.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:00.490+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:00.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:00.490+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:01 smithi112 ceph-mon[37501]: pgmap v5988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10436.981685 secs 2024-07-26T15:45:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:01 smithi107 ceph-mon[32659]: pgmap v5988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10436.981685 secs 2024-07-26T15:45:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:01 smithi132 ceph-mon[37580]: pgmap v5988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10436.981685 secs 2024-07-26T15:45:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:02.919+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:02.919+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:02.919+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:03 smithi112 ceph-mon[37501]: pgmap v5989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:45:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:03 smithi107 ceph-mon[32659]: pgmap v5989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:45:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:03 smithi132 ceph-mon[37580]: pgmap v5989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:45:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:05 smithi112 ceph-mon[37501]: pgmap v5990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:05 smithi107 ceph-mon[32659]: pgmap v5990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:05 smithi132 ceph-mon[37580]: pgmap v5990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:06.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10441.981754 secs 2024-07-26T15:45:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10441.981754 secs 2024-07-26T15:45:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10441.981754 secs 2024-07-26T15:45:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:07 smithi107 ceph-mon[32659]: pgmap v5991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:07 smithi112 ceph-mon[37501]: pgmap v5991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:07 smithi132 ceph-mon[37580]: pgmap v5991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:09.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:09 smithi112 ceph-mon[37501]: pgmap v5992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:09 smithi107 ceph-mon[32659]: pgmap v5992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:09 smithi132 ceph-mon[37580]: pgmap v5992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:11 smithi112 ceph-mon[37501]: pgmap v5993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10446.981876 secs 2024-07-26T15:45:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:11 smithi107 ceph-mon[32659]: pgmap v5993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10446.981876 secs 2024-07-26T15:45:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:11 smithi132 ceph-mon[37580]: pgmap v5993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10446.981876 secs 2024-07-26T15:45:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:13 smithi112 ceph-mon[37501]: pgmap v5994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:13 smithi107 ceph-mon[32659]: pgmap v5994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:13 smithi132 ceph-mon[37580]: pgmap v5994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:14.416+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:45:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:14.416+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:45:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:14.416+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:45:14.418 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:14.416+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:45:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:15 smithi112 ceph-mon[37501]: pgmap v5995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:15.505 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:15.504+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:15.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:15.504+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:15.506 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:15.504+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:15.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:15 smithi107 ceph-mon[32659]: pgmap v5995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:15 smithi132 ceph-mon[37580]: pgmap v5995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10451.982010 secs 2024-07-26T15:45:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10451.982010 secs 2024-07-26T15:45:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10451.982010 secs 2024-07-26T15:45:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:16.611+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:45:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:16.611+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:45:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:16.611+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:45:16.612 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:16.611+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:45:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:17 smithi112 ceph-mon[37501]: pgmap v5996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:17 smithi107 ceph-mon[32659]: pgmap v5996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:17 smithi132 ceph-mon[37580]: pgmap v5996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:17.936 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:17.935+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:17.936 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:17.935+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:17.936 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:17.935+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:18 smithi112 ceph-mon[37501]: pgmap v5997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:18 smithi107 ceph-mon[32659]: pgmap v5997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:18 smithi132 ceph-mon[37580]: pgmap v5997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:20 smithi112 ceph-mon[37501]: pgmap v5998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:20 smithi107 ceph-mon[32659]: pgmap v5998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:20 smithi132 ceph-mon[37580]: pgmap v5998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10456.982124 secs 2024-07-26T15:45:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10456.982124 secs 2024-07-26T15:45:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10456.982124 secs 2024-07-26T15:45:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:22 smithi112 ceph-mon[37501]: pgmap v5999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:22 smithi107 ceph-mon[32659]: pgmap v5999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:22 smithi132 ceph-mon[37580]: pgmap v5999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:24 smithi112 ceph-mon[37501]: pgmap v6000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:24 smithi107 ceph-mon[32659]: pgmap v6000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:24 smithi132 ceph-mon[37580]: pgmap v6000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10461.982244 secs 2024-07-26T15:45:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:26 smithi107 ceph-mon[32659]: pgmap v6001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10461.982244 secs 2024-07-26T15:45:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:26 smithi132 ceph-mon[37580]: pgmap v6001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10461.982244 secs 2024-07-26T15:45:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:26 smithi112 ceph-mon[37501]: pgmap v6001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:28 smithi107 ceph-mon[32659]: pgmap v6002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:28 smithi132 ceph-mon[37580]: pgmap v6002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:28 smithi112 ceph-mon[37501]: pgmap v6002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:30.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:30.520+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:30.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:30.520+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:30.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:30.520+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:30 smithi107 ceph-mon[32659]: pgmap v6003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:30 smithi132 ceph-mon[37580]: pgmap v6003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:30 smithi112 ceph-mon[37501]: pgmap v6003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10466.982361 secs 2024-07-26T15:45:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10466.982361 secs 2024-07-26T15:45:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10466.982361 secs 2024-07-26T15:45:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:32 smithi107 ceph-mon[32659]: pgmap v6004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:32 smithi132 ceph-mon[37580]: pgmap v6004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:32.948+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:32.948+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:32.948+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:32 smithi112 ceph-mon[37501]: pgmap v6004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:34 smithi107 ceph-mon[32659]: pgmap v6005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:34 smithi132 ceph-mon[37580]: pgmap v6005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:34 smithi112 ceph-mon[37501]: pgmap v6005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10471.982438 secs 2024-07-26T15:45:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:36 smithi107 ceph-mon[32659]: pgmap v6006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10471.982438 secs 2024-07-26T15:45:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:36 smithi132 ceph-mon[37580]: pgmap v6006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10471.982438 secs 2024-07-26T15:45:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:36 smithi112 ceph-mon[37501]: pgmap v6006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:38 smithi107 ceph-mon[32659]: pgmap v6007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:38 smithi132 ceph-mon[37580]: pgmap v6007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:38 smithi112 ceph-mon[37501]: pgmap v6007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:40 smithi132 ceph-mon[37580]: pgmap v6008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:40 smithi112 ceph-mon[37501]: pgmap v6008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:40 smithi107 ceph-mon[32659]: pgmap v6008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:45:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10476.982566 secs 2024-07-26T15:45:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10476.982566 secs 2024-07-26T15:45:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10476.982566 secs 2024-07-26T15:45:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:42 smithi132 ceph-mon[37580]: pgmap v6009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:45:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:42 smithi112 ceph-mon[37501]: pgmap v6009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:45:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:42 smithi107 ceph-mon[32659]: pgmap v6009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:45:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:44 smithi112 ceph-mon[37501]: pgmap v6010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:45:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:45:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:45:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:44 smithi107 ceph-mon[32659]: pgmap v6010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:45:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:45:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:45:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:44 smithi132 ceph-mon[37580]: pgmap v6010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:45:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:45:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:45:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:45:45.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:45.528+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:45.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:45.528+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:45.530 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:45:45.528+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:45 smithi107 ceph-mon[32659]: pgmap v6011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:45 smithi107 ceph-mon[32659]: pgmap v6012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:45:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:45:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:45 smithi112 ceph-mon[37501]: pgmap v6011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:45 smithi112 ceph-mon[37501]: pgmap v6012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:45:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:45:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:45 smithi132 ceph-mon[37580]: pgmap v6011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:45 smithi132 ceph-mon[37580]: pgmap v6012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:45:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:45:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10481.982666 secs 2024-07-26T15:45:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10481.982666 secs 2024-07-26T15:45:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10481.982666 secs 2024-07-26T15:45:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:47 smithi112 ceph-mon[37501]: pgmap v6013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:45:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:47.964+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:47.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:47.964+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:45:47.965 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:45:47.964+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:47 smithi107 ceph-mon[32659]: pgmap v6013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:45:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:47 smithi132 ceph-mon[37580]: pgmap v6013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:45:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:49 smithi112 ceph-mon[37501]: pgmap v6014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:45:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:49 smithi107 ceph-mon[32659]: pgmap v6014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:45:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:49 smithi132 ceph-mon[37580]: pgmap v6014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:45:51.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10486.982766 secs 2024-07-26T15:45:51.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:51 smithi132 ceph-mon[37580]: pgmap v6015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:45:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10486.982766 secs 2024-07-26T15:45:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:51 smithi112 ceph-mon[37501]: pgmap v6015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:45:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10486.982766 secs 2024-07-26T15:45:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:51 smithi107 ceph-mon[32659]: pgmap v6015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:45:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:53 smithi112 ceph-mon[37501]: pgmap v6016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:45:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:53 smithi107 ceph-mon[32659]: pgmap v6016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:45:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:53 smithi132 ceph-mon[37580]: pgmap v6016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:45:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:55 smithi112 ceph-mon[37501]: pgmap v6017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:45:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:55 smithi107 ceph-mon[32659]: pgmap v6017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:45:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:55 smithi132 ceph-mon[37580]: pgmap v6017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:45:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10491.982913 secs 2024-07-26T15:45:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10491.982913 secs 2024-07-26T15:45:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10491.982913 secs 2024-07-26T15:45:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:57 smithi112 ceph-mon[37501]: pgmap v6018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:57 smithi107 ceph-mon[32659]: pgmap v6018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:57 smithi132 ceph-mon[37580]: pgmap v6018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:45:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:45:59 smithi112 ceph-mon[37501]: pgmap v6019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:45:59 smithi107 ceph-mon[32659]: pgmap v6019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:45:59 smithi132 ceph-mon[37580]: pgmap v6019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:00.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:00.543+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:00.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:00.543+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:00.545 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:00.543+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10496.983014 secs 2024-07-26T15:46:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:01 smithi112 ceph-mon[37501]: pgmap v6020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10496.983014 secs 2024-07-26T15:46:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:01 smithi107 ceph-mon[32659]: pgmap v6020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10496.983014 secs 2024-07-26T15:46:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:01 smithi132 ceph-mon[37580]: pgmap v6020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:02.979+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:02.979+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:02.979+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:03 smithi112 ceph-mon[37501]: pgmap v6021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:03 smithi107 ceph-mon[32659]: pgmap v6021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:03 smithi132 ceph-mon[37580]: pgmap v6021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:05 smithi112 ceph-mon[37501]: pgmap v6022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:05 smithi107 ceph-mon[32659]: pgmap v6022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:05 smithi132 ceph-mon[37580]: pgmap v6022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10501.983135 secs 2024-07-26T15:46:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10501.983135 secs 2024-07-26T15:46:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10501.983135 secs 2024-07-26T15:46:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:07 smithi112 ceph-mon[37501]: pgmap v6023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:46:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:07 smithi107 ceph-mon[32659]: pgmap v6023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:46:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:07 smithi132 ceph-mon[37580]: pgmap v6023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:46:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:09 smithi112 ceph-mon[37501]: pgmap v6024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:09 smithi107 ceph-mon[32659]: pgmap v6024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:09 smithi132 ceph-mon[37580]: pgmap v6024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:11.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10506.983232 secs 2024-07-26T15:46:11.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:11 smithi112 ceph-mon[37501]: pgmap v6025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10506.983232 secs 2024-07-26T15:46:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:11 smithi107 ceph-mon[32659]: pgmap v6025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10506.983232 secs 2024-07-26T15:46:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:11 smithi132 ceph-mon[37580]: pgmap v6025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:13 smithi112 ceph-mon[37501]: pgmap v6026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:13 smithi107 ceph-mon[32659]: pgmap v6026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:13 smithi132 ceph-mon[37580]: pgmap v6026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:15.556+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:15.556+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:15.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:15.556+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:15 smithi112 ceph-mon[37501]: pgmap v6027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:15 smithi107 ceph-mon[32659]: pgmap v6027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:15 smithi132 ceph-mon[37580]: pgmap v6027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10511.983306 secs 2024-07-26T15:46:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10511.983306 secs 2024-07-26T15:46:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10511.983306 secs 2024-07-26T15:46:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:17.994+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:17.994+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:17.994+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:17 smithi107 ceph-mon[32659]: pgmap v6028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:17 smithi132 ceph-mon[37580]: pgmap v6028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:17 smithi112 ceph-mon[37501]: pgmap v6028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:19 smithi107 ceph-mon[32659]: pgmap v6029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:19 smithi132 ceph-mon[37580]: pgmap v6029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:19 smithi112 ceph-mon[37501]: pgmap v6029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10516.983391 secs 2024-07-26T15:46:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10516.983391 secs 2024-07-26T15:46:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10516.983391 secs 2024-07-26T15:46:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:21 smithi107 ceph-mon[32659]: pgmap v6030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:21 smithi132 ceph-mon[37580]: pgmap v6030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:21 smithi112 ceph-mon[37501]: pgmap v6030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:23 smithi107 ceph-mon[32659]: pgmap v6031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:23 smithi132 ceph-mon[37580]: pgmap v6031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:23 smithi112 ceph-mon[37501]: pgmap v6031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:25 smithi107 ceph-mon[32659]: pgmap v6032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10521.983503 secs 2024-07-26T15:46:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:25 smithi132 ceph-mon[37580]: pgmap v6032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10521.983503 secs 2024-07-26T15:46:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:25 smithi112 ceph-mon[37501]: pgmap v6032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10521.983503 secs 2024-07-26T15:46:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:27 smithi107 ceph-mon[32659]: pgmap v6033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:27 smithi132 ceph-mon[37580]: pgmap v6033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:27 smithi112 ceph-mon[37501]: pgmap v6033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:29 smithi107 ceph-mon[32659]: pgmap v6034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:29 smithi132 ceph-mon[37580]: pgmap v6034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:29 smithi112 ceph-mon[37501]: pgmap v6034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:30.572+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:30.572+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:30.572+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:30.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10526.983604 secs 2024-07-26T15:46:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10526.983604 secs 2024-07-26T15:46:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10526.983604 secs 2024-07-26T15:46:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:31 smithi107 ceph-mon[32659]: pgmap v6035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:31 smithi132 ceph-mon[37580]: pgmap v6035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:31 smithi112 ceph-mon[37501]: pgmap v6035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:32.995+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:32.995+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:32.995+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:33 smithi107 ceph-mon[32659]: pgmap v6036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:33 smithi132 ceph-mon[37580]: pgmap v6036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:33 smithi112 ceph-mon[37501]: pgmap v6036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:35 smithi107 ceph-mon[32659]: pgmap v6037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10531.983729 secs 2024-07-26T15:46:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:35 smithi132 ceph-mon[37580]: pgmap v6037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10531.983729 secs 2024-07-26T15:46:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:35 smithi112 ceph-mon[37501]: pgmap v6037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10531.983729 secs 2024-07-26T15:46:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:37 smithi107 ceph-mon[32659]: pgmap v6038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:37 smithi132 ceph-mon[37580]: pgmap v6038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:37 smithi112 ceph-mon[37501]: pgmap v6038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:39 smithi107 ceph-mon[32659]: pgmap v6039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:39 smithi132 ceph-mon[37580]: pgmap v6039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:39 smithi112 ceph-mon[37501]: pgmap v6039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10536.983869 secs 2024-07-26T15:46:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10536.983869 secs 2024-07-26T15:46:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10536.983869 secs 2024-07-26T15:46:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:41 smithi132 ceph-mon[37580]: pgmap v6040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:41 smithi112 ceph-mon[37501]: pgmap v6040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:41 smithi107 ceph-mon[32659]: pgmap v6040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:43 smithi132 ceph-mon[37580]: pgmap v6041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:43 smithi112 ceph-mon[37501]: pgmap v6041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:43 smithi107 ceph-mon[32659]: pgmap v6041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:46:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:46:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:46:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:45.587+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:45.587+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:45.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:46:45.587+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:45.936 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:45 smithi112 ceph-mon[37501]: pgmap v6042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:45.936 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10541.984018 secs 2024-07-26T15:46:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:45 smithi107 ceph-mon[32659]: pgmap v6042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10541.984018 secs 2024-07-26T15:46:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:45 smithi132 ceph-mon[37580]: pgmap v6042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:46:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10541.984018 secs 2024-07-26T15:46:47.973 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:47 smithi112 ceph-mon[37501]: pgmap v6043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:48.010+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:48.010+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:46:48.011 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:46:48.010+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:48.118 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:47 smithi107 ceph-mon[32659]: pgmap v6043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:47 smithi132 ceph-mon[37580]: pgmap v6043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:49 smithi107 ceph-mon[32659]: pgmap v6044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:49 smithi132 ceph-mon[37580]: pgmap v6044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:49 smithi112 ceph-mon[37501]: pgmap v6044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10546.984138 secs 2024-07-26T15:46:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10546.984138 secs 2024-07-26T15:46:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10546.984138 secs 2024-07-26T15:46:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: pgmap v6045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:46:52.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:46:52.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: pgmap v6045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:46:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:46:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: pgmap v6045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:46:52.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:46:52.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:46:52.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:46:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:52 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:46:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:52 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:46:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:52 smithi107 ceph-mon[32659]: pgmap v6046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:52 smithi107 ceph-mon[32659]: pgmap v6047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:46:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:52 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:46:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:52 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:46:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:52 smithi132 ceph-mon[37580]: pgmap v6046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:52 smithi132 ceph-mon[37580]: pgmap v6047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:46:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:52 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:46:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:52 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:46:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:52 smithi112 ceph-mon[37501]: pgmap v6046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:52 smithi112 ceph-mon[37501]: pgmap v6047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:46:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:54 smithi107 ceph-mon[32659]: pgmap v6048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:46:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:54 smithi132 ceph-mon[37580]: pgmap v6048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:46:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:54 smithi112 ceph-mon[37501]: pgmap v6048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:46:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10551.984261 secs 2024-07-26T15:46:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10551.984261 secs 2024-07-26T15:46:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10551.984261 secs 2024-07-26T15:46:56.920 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:56 smithi107 ceph-mon[32659]: pgmap v6049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:46:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:56 smithi132 ceph-mon[37580]: pgmap v6049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:46:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:56 smithi112 ceph-mon[37501]: pgmap v6049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:46:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:46:58 smithi132 ceph-mon[37580]: pgmap v6050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:46:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:46:58 smithi112 ceph-mon[37501]: pgmap v6050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:46:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:46:58 smithi107 ceph-mon[32659]: pgmap v6050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T15:47:00.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:00.587+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:00.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:00.587+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:00.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:00.587+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:00 smithi107 ceph-mon[32659]: pgmap v6051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10556.984363 secs 2024-07-26T15:47:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:00 smithi132 ceph-mon[37580]: pgmap v6051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10556.984363 secs 2024-07-26T15:47:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:00 smithi112 ceph-mon[37501]: pgmap v6051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10556.984363 secs 2024-07-26T15:47:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:03.025+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:03.025+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:03.025+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:02 smithi132 ceph-mon[37580]: pgmap v6052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:47:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:02 smithi112 ceph-mon[37501]: pgmap v6052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:47:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:02 smithi107 ceph-mon[32659]: pgmap v6052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:47:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:04 smithi112 ceph-mon[37501]: pgmap v6053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:04 smithi107 ceph-mon[32659]: pgmap v6053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:04 smithi132 ceph-mon[37580]: pgmap v6053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:06.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10561.984459 secs 2024-07-26T15:47:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10561.984459 secs 2024-07-26T15:47:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10561.984459 secs 2024-07-26T15:47:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:06 smithi112 ceph-mon[37501]: pgmap v6054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:06 smithi107 ceph-mon[32659]: pgmap v6054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:06 smithi132 ceph-mon[37580]: pgmap v6054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:08 smithi112 ceph-mon[37501]: pgmap v6055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:08 smithi107 ceph-mon[32659]: pgmap v6055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:08 smithi132 ceph-mon[37580]: pgmap v6055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:10 smithi112 ceph-mon[37501]: pgmap v6056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10566.984596 secs 2024-07-26T15:47:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:10 smithi107 ceph-mon[32659]: pgmap v6056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10566.984596 secs 2024-07-26T15:47:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:10 smithi132 ceph-mon[37580]: pgmap v6056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10566.984596 secs 2024-07-26T15:47:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:12 smithi112 ceph-mon[37501]: pgmap v6057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:47:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:12 smithi107 ceph-mon[32659]: pgmap v6057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:47:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:12 smithi132 ceph-mon[37580]: pgmap v6057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:47:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:14 smithi112 ceph-mon[37501]: pgmap v6058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:14 smithi107 ceph-mon[32659]: pgmap v6058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:14 smithi132 ceph-mon[37580]: pgmap v6058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:15.592+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:15.592+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:15.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:15.592+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10571.984666 secs 2024-07-26T15:47:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10571.984666 secs 2024-07-26T15:47:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10571.984666 secs 2024-07-26T15:47:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:16 smithi112 ceph-mon[37501]: pgmap v6059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:16 smithi107 ceph-mon[32659]: pgmap v6059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:16 smithi132 ceph-mon[37580]: pgmap v6059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:18.041+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:18.042 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:18.041+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:18.043 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:18.041+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:18 smithi112 ceph-mon[37501]: pgmap v6060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:18 smithi107 ceph-mon[32659]: pgmap v6060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:18 smithi132 ceph-mon[37580]: pgmap v6060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:20 smithi112 ceph-mon[37501]: pgmap v6061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T15:47:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10576.984812 secs 2024-07-26T15:47:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:20 smithi107 ceph-mon[32659]: pgmap v6061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T15:47:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10576.984812 secs 2024-07-26T15:47:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:20 smithi132 ceph-mon[37580]: pgmap v6061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T15:47:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10576.984812 secs 2024-07-26T15:47:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:22 smithi112 ceph-mon[37501]: pgmap v6062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:47:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:22 smithi107 ceph-mon[32659]: pgmap v6062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:47:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:22 smithi132 ceph-mon[37580]: pgmap v6062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:47:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:24 smithi112 ceph-mon[37501]: pgmap v6063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:47:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:24 smithi107 ceph-mon[32659]: pgmap v6063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:47:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:24 smithi132 ceph-mon[37580]: pgmap v6063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:47:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10581.984956 secs 2024-07-26T15:47:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10581.984956 secs 2024-07-26T15:47:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10581.984956 secs 2024-07-26T15:47:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:26 smithi112 ceph-mon[37501]: pgmap v6064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:26 smithi107 ceph-mon[32659]: pgmap v6064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:26 smithi132 ceph-mon[37580]: pgmap v6064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:28 smithi112 ceph-mon[37501]: pgmap v6065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:28 smithi107 ceph-mon[32659]: pgmap v6065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:28 smithi132 ceph-mon[37580]: pgmap v6065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:30.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:30.598+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:30.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:30.598+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:30.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:30.598+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:30 smithi112 ceph-mon[37501]: pgmap v6066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T15:47:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10586.985096 secs 2024-07-26T15:47:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:30 smithi107 ceph-mon[32659]: pgmap v6066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T15:47:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10586.985096 secs 2024-07-26T15:47:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:30 smithi132 ceph-mon[37580]: pgmap v6066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T15:47:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10586.985096 secs 2024-07-26T15:47:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:33.057+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:33.057+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:33.057+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:32 smithi112 ceph-mon[37501]: pgmap v6067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:32 smithi107 ceph-mon[32659]: pgmap v6067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:32 smithi132 ceph-mon[37580]: pgmap v6067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:34 smithi112 ceph-mon[37501]: pgmap v6068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:34 smithi107 ceph-mon[32659]: pgmap v6068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:34 smithi132 ceph-mon[37580]: pgmap v6068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10591.985179 secs 2024-07-26T15:47:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10591.985179 secs 2024-07-26T15:47:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10591.985179 secs 2024-07-26T15:47:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:36 smithi107 ceph-mon[32659]: pgmap v6069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:36 smithi132 ceph-mon[37580]: pgmap v6069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:36 smithi112 ceph-mon[37501]: pgmap v6069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:38 smithi107 ceph-mon[32659]: pgmap v6070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:38 smithi132 ceph-mon[37580]: pgmap v6070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:38 smithi112 ceph-mon[37501]: pgmap v6070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:40 smithi107 ceph-mon[32659]: pgmap v6071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10596.985289 secs 2024-07-26T15:47:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:40 smithi132 ceph-mon[37580]: pgmap v6071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10596.985289 secs 2024-07-26T15:47:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:40 smithi112 ceph-mon[37501]: pgmap v6071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10596.985289 secs 2024-07-26T15:47:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:42 smithi107 ceph-mon[32659]: pgmap v6072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:42 smithi132 ceph-mon[37580]: pgmap v6072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:42 smithi112 ceph-mon[37501]: pgmap v6072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:44 smithi107 ceph-mon[32659]: pgmap v6073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:44 smithi132 ceph-mon[37580]: pgmap v6073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:44 smithi112 ceph-mon[37501]: pgmap v6073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:45.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:45.614+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:45.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:45.614+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:45.615 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:47:45.614+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10601.985369 secs 2024-07-26T15:47:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10601.985369 secs 2024-07-26T15:47:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10601.985369 secs 2024-07-26T15:47:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:46 smithi107 ceph-mon[32659]: pgmap v6074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:46 smithi132 ceph-mon[37580]: pgmap v6074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:46 smithi112 ceph-mon[37501]: pgmap v6074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:48.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:48.064+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:48.066 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:48.064+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:47:48.064+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:47:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:48 smithi107 ceph-mon[32659]: pgmap v6075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:48 smithi132 ceph-mon[37580]: pgmap v6075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:48 smithi112 ceph-mon[37501]: pgmap v6075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:50 smithi107 ceph-mon[32659]: pgmap v6076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10606.985448 secs 2024-07-26T15:47:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:50 smithi132 ceph-mon[37580]: pgmap v6076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10606.985448 secs 2024-07-26T15:47:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:50 smithi112 ceph-mon[37501]: pgmap v6076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:47:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10606.985448 secs 2024-07-26T15:47:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:47:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:47:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:47:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: pgmap v6077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:47:53.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:53 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:47:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: pgmap v6077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:47:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:53 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:47:53.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: pgmap v6077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:53.525 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.525 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.525 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.525 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.525 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.525 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.525 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.525 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:53.525 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:47:53.526 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:53 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:47:55.210 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:55 smithi132 ceph-mon[37580]: pgmap v6078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:55 smithi107 ceph-mon[32659]: pgmap v6078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:55 smithi112 ceph-mon[37501]: pgmap v6078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:47:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:47:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: pgmap v6079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:47:56.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10611.985517 secs 2024-07-26T15:47:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:47:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:47:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:47:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:47:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:47:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:47:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: pgmap v6079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:47:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10611.985517 secs 2024-07-26T15:47:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:47:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:47:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:56.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:47:56.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:47:56.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:47:56.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:47:56.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: pgmap v6079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:56.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:47:56.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10611.985517 secs 2024-07-26T15:47:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:47:58 smithi112 ceph-mon[37501]: pgmap v6080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:47:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:47:58 smithi107 ceph-mon[32659]: pgmap v6080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:47:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:47:58 smithi132 ceph-mon[37580]: pgmap v6080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:48:00.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:00.617+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:00.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:00.617+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:00.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:00.617+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:00 smithi112 ceph-mon[37501]: pgmap v6081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:00 smithi107 ceph-mon[32659]: pgmap v6081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:00 smithi132 ceph-mon[37580]: pgmap v6081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:01.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10616.985618 secs 2024-07-26T15:48:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10616.985618 secs 2024-07-26T15:48:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10616.985618 secs 2024-07-26T15:48:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:02 smithi112 ceph-mon[37501]: pgmap v6082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:02 smithi107 ceph-mon[32659]: pgmap v6082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:02 smithi132 ceph-mon[37580]: pgmap v6082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:03.075+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:03.075+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:03.075+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:04 smithi112 ceph-mon[37501]: pgmap v6083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:48:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:04 smithi107 ceph-mon[32659]: pgmap v6083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:48:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:04 smithi132 ceph-mon[37580]: pgmap v6083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:48:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:06 smithi112 ceph-mon[37501]: pgmap v6084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10621.985745 secs 2024-07-26T15:48:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:06 smithi107 ceph-mon[32659]: pgmap v6084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10621.985745 secs 2024-07-26T15:48:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:06 smithi132 ceph-mon[37580]: pgmap v6084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10621.985745 secs 2024-07-26T15:48:08.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:08 smithi112 ceph-mon[37501]: pgmap v6085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:08 smithi107 ceph-mon[32659]: pgmap v6085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:08 smithi132 ceph-mon[37580]: pgmap v6085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:10 smithi112 ceph-mon[37501]: pgmap v6086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:10 smithi107 ceph-mon[32659]: pgmap v6086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:10 smithi132 ceph-mon[37580]: pgmap v6086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10626.985865 secs 2024-07-26T15:48:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10626.985865 secs 2024-07-26T15:48:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10626.985865 secs 2024-07-26T15:48:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:12 smithi112 ceph-mon[37501]: pgmap v6087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:12 smithi107 ceph-mon[32659]: pgmap v6087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:12 smithi132 ceph-mon[37580]: pgmap v6087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:14.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:14.447+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:48:14.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:14.447+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:48:14.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:14.448+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:48:14.450 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:14.448+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:48:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:14 smithi107 ceph-mon[32659]: pgmap v6088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:14 smithi112 ceph-mon[37501]: pgmap v6088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:14 smithi132 ceph-mon[37580]: pgmap v6088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:15.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:15.631+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:15.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:15.631+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:15.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:15.631+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:16.643+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:48:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:16.643+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:48:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:16.643+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:48:16.644 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:16.644+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:48:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10631.986003 secs 2024-07-26T15:48:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:16 smithi112 ceph-mon[37501]: pgmap v6089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10631.986003 secs 2024-07-26T15:48:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:16 smithi107 ceph-mon[32659]: pgmap v6089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10631.986003 secs 2024-07-26T15:48:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:16 smithi132 ceph-mon[37580]: pgmap v6089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:18.085+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:18.086+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:18.086+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:18 smithi112 ceph-mon[37501]: pgmap v6090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:18 smithi107 ceph-mon[32659]: pgmap v6090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:18 smithi132 ceph-mon[37580]: pgmap v6090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:20 smithi112 ceph-mon[37501]: pgmap v6091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:20 smithi107 ceph-mon[32659]: pgmap v6091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:20 smithi132 ceph-mon[37580]: pgmap v6091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10636.986104 secs 2024-07-26T15:48:21.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10636.986104 secs 2024-07-26T15:48:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10636.986104 secs 2024-07-26T15:48:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:22 smithi112 ceph-mon[37501]: pgmap v6092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:22.977 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:22 smithi132 ceph-mon[37580]: pgmap v6092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:22 smithi107 ceph-mon[32659]: pgmap v6092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:24 smithi112 ceph-mon[37501]: pgmap v6093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:24 smithi107 ceph-mon[32659]: pgmap v6093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:24 smithi132 ceph-mon[37580]: pgmap v6093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10641.986228 secs 2024-07-26T15:48:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:26 smithi112 ceph-mon[37501]: pgmap v6094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10641.986228 secs 2024-07-26T15:48:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:26 smithi107 ceph-mon[32659]: pgmap v6094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10641.986228 secs 2024-07-26T15:48:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:26 smithi132 ceph-mon[37580]: pgmap v6094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:28 smithi107 ceph-mon[32659]: pgmap v6095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:28 smithi132 ceph-mon[37580]: pgmap v6095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:29.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:28 smithi112 ceph-mon[37501]: pgmap v6095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:30.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:30.645+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:30.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:30.645+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:30.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:30.645+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:30 smithi107 ceph-mon[32659]: pgmap v6096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:30 smithi132 ceph-mon[37580]: pgmap v6096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:30 smithi112 ceph-mon[37501]: pgmap v6096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10646.986347 secs 2024-07-26T15:48:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10646.986347 secs 2024-07-26T15:48:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10646.986347 secs 2024-07-26T15:48:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:32 smithi107 ceph-mon[32659]: pgmap v6097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:32 smithi132 ceph-mon[37580]: pgmap v6097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:33.101+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:33.101+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:33.101+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:32 smithi112 ceph-mon[37501]: pgmap v6097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:48:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:34 smithi107 ceph-mon[32659]: pgmap v6098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:34 smithi132 ceph-mon[37580]: pgmap v6098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:34 smithi112 ceph-mon[37501]: pgmap v6098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10651.986439 secs 2024-07-26T15:48:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10651.986439 secs 2024-07-26T15:48:36.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10651.986439 secs 2024-07-26T15:48:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:36 smithi107 ceph-mon[32659]: pgmap v6099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:36 smithi132 ceph-mon[37580]: pgmap v6099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:36 smithi112 ceph-mon[37501]: pgmap v6099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:38 smithi107 ceph-mon[32659]: pgmap v6100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:38 smithi132 ceph-mon[37580]: pgmap v6100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:38 smithi112 ceph-mon[37501]: pgmap v6100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:40 smithi107 ceph-mon[32659]: pgmap v6101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10656.986535 secs 2024-07-26T15:48:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:40 smithi132 ceph-mon[37580]: pgmap v6101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10656.986535 secs 2024-07-26T15:48:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:40 smithi112 ceph-mon[37501]: pgmap v6101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10656.986535 secs 2024-07-26T15:48:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:42 smithi107 ceph-mon[32659]: pgmap v6102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:48:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:42 smithi132 ceph-mon[37580]: pgmap v6102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:48:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:42 smithi112 ceph-mon[37501]: pgmap v6102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:48:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:44 smithi107 ceph-mon[32659]: pgmap v6103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:44 smithi132 ceph-mon[37580]: pgmap v6103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:44 smithi112 ceph-mon[37501]: pgmap v6103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:45.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:45.649+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:45.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:45.649+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:45.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:48:45.649+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10661.986675 secs 2024-07-26T15:48:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10661.986675 secs 2024-07-26T15:48:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10661.986675 secs 2024-07-26T15:48:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:46 smithi107 ceph-mon[32659]: pgmap v6104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:46 smithi132 ceph-mon[37580]: pgmap v6104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:46 smithi112 ceph-mon[37501]: pgmap v6104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:48.116+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:48.116+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:48:48.118 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:48:48.116+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:48 smithi107 ceph-mon[32659]: pgmap v6105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:48 smithi132 ceph-mon[37580]: pgmap v6105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:48 smithi112 ceph-mon[37501]: pgmap v6105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:50 smithi132 ceph-mon[37580]: pgmap v6106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10666.986810 secs 2024-07-26T15:48:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:50 smithi112 ceph-mon[37501]: pgmap v6106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10666.986810 secs 2024-07-26T15:48:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:50 smithi107 ceph-mon[32659]: pgmap v6106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:48:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10666.986810 secs 2024-07-26T15:48:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:52 smithi132 ceph-mon[37580]: pgmap v6107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:52 smithi112 ceph-mon[37501]: pgmap v6107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:52 smithi107 ceph-mon[32659]: pgmap v6107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:54 smithi132 ceph-mon[37580]: pgmap v6108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:54 smithi112 ceph-mon[37501]: pgmap v6108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:54 smithi107 ceph-mon[32659]: pgmap v6108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10671.986962 secs 2024-07-26T15:48:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:48:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10671.986962 secs 2024-07-26T15:48:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:48:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10671.986962 secs 2024-07-26T15:48:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:48:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:56 smithi112 ceph-mon[37501]: pgmap v6109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:48:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:48:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:48:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:56 smithi107 ceph-mon[32659]: pgmap v6109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:48:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:48:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:48:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:56 smithi132 ceph-mon[37580]: pgmap v6109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:48:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:48:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:48:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:57 smithi112 ceph-mon[37501]: pgmap v6110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:48:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:57 smithi112 ceph-mon[37501]: pgmap v6111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:48:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:57 smithi107 ceph-mon[32659]: pgmap v6110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:48:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:57 smithi107 ceph-mon[32659]: pgmap v6111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:48:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:57 smithi132 ceph-mon[37580]: pgmap v6110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:48:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:57 smithi132 ceph-mon[37580]: pgmap v6111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:48:59 smithi112 ceph-mon[37501]: pgmap v6112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:49:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:48:59 smithi107 ceph-mon[32659]: pgmap v6112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:49:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:48:59 smithi132 ceph-mon[37580]: pgmap v6112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:49:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:00.664+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:00.664+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:00.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:00.664+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:01.018 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10676.987113 secs 2024-07-26T15:49:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10676.987113 secs 2024-07-26T15:49:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10676.987113 secs 2024-07-26T15:49:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:01 smithi112 ceph-mon[37501]: pgmap v6113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:49:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:01 smithi107 ceph-mon[32659]: pgmap v6113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:49:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:01 smithi132 ceph-mon[37580]: pgmap v6113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:49:03.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:03.128+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:03.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:03.128+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:03.130 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:03.128+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:03 smithi112 ceph-mon[37501]: pgmap v6114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:49:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:03 smithi107 ceph-mon[32659]: pgmap v6114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:49:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:03 smithi132 ceph-mon[37580]: pgmap v6114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:49:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:05 smithi112 ceph-mon[37501]: pgmap v6115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10681.987265 secs 2024-07-26T15:49:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:05 smithi107 ceph-mon[32659]: pgmap v6115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10681.987265 secs 2024-07-26T15:49:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:05 smithi132 ceph-mon[37580]: pgmap v6115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10681.987265 secs 2024-07-26T15:49:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:07 smithi112 ceph-mon[37501]: pgmap v6116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:49:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:07 smithi107 ceph-mon[32659]: pgmap v6116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:49:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:07 smithi132 ceph-mon[37580]: pgmap v6116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:49:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:09 smithi112 ceph-mon[37501]: pgmap v6117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:09 smithi107 ceph-mon[32659]: pgmap v6117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:09 smithi132 ceph-mon[37580]: pgmap v6117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10686.987385 secs 2024-07-26T15:49:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10686.987385 secs 2024-07-26T15:49:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10686.987385 secs 2024-07-26T15:49:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:11 smithi112 ceph-mon[37501]: pgmap v6118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:11 smithi107 ceph-mon[32659]: pgmap v6118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:11 smithi132 ceph-mon[37580]: pgmap v6118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:13 smithi112 ceph-mon[37501]: pgmap v6119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:13 smithi107 ceph-mon[32659]: pgmap v6119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:13 smithi132 ceph-mon[37580]: pgmap v6119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:15.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:15.680+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:15.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:15.680+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:15.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:15.680+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:15 smithi107 ceph-mon[32659]: pgmap v6120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10691.987532 secs 2024-07-26T15:49:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:15 smithi112 ceph-mon[37501]: pgmap v6120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10691.987532 secs 2024-07-26T15:49:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:15 smithi132 ceph-mon[37580]: pgmap v6120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10691.987532 secs 2024-07-26T15:49:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:18.139+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:18.139+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:18.140 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:18.139+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:17 smithi112 ceph-mon[37501]: pgmap v6121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:17 smithi107 ceph-mon[32659]: pgmap v6121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:17 smithi132 ceph-mon[37580]: pgmap v6121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:19 smithi112 ceph-mon[37501]: pgmap v6122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:19 smithi107 ceph-mon[32659]: pgmap v6122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:19 smithi132 ceph-mon[37580]: pgmap v6122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10696.987653 secs 2024-07-26T15:49:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10696.987653 secs 2024-07-26T15:49:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10696.987653 secs 2024-07-26T15:49:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:21 smithi112 ceph-mon[37501]: pgmap v6123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:22.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:21 smithi132 ceph-mon[37580]: pgmap v6123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:21 smithi107 ceph-mon[32659]: pgmap v6123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:23 smithi112 ceph-mon[37501]: pgmap v6124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:23 smithi107 ceph-mon[32659]: pgmap v6124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:23 smithi132 ceph-mon[37580]: pgmap v6124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:25 smithi112 ceph-mon[37501]: pgmap v6125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10701.987782 secs 2024-07-26T15:49:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:25 smithi107 ceph-mon[32659]: pgmap v6125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10701.987782 secs 2024-07-26T15:49:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:25 smithi132 ceph-mon[37580]: pgmap v6125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10701.987782 secs 2024-07-26T15:49:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:27 smithi112 ceph-mon[37501]: pgmap v6126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:27 smithi107 ceph-mon[32659]: pgmap v6126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:27 smithi132 ceph-mon[37580]: pgmap v6126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:29 smithi107 ceph-mon[32659]: pgmap v6127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:29 smithi132 ceph-mon[37580]: pgmap v6127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:29 smithi112 ceph-mon[37501]: pgmap v6127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:30.687+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:30.687+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:30.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:30.687+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:31.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10706.987909 secs 2024-07-26T15:49:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10706.987909 secs 2024-07-26T15:49:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10706.987909 secs 2024-07-26T15:49:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:31 smithi107 ceph-mon[32659]: pgmap v6128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:31 smithi132 ceph-mon[37580]: pgmap v6128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:31 smithi112 ceph-mon[37501]: pgmap v6128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:33.155+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:33.155+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:33.155+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:33 smithi107 ceph-mon[32659]: pgmap v6129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:33 smithi132 ceph-mon[37580]: pgmap v6129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:33 smithi112 ceph-mon[37501]: pgmap v6129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:36.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:35 smithi112 ceph-mon[37501]: pgmap v6130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:36.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10711.988052 secs 2024-07-26T15:49:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:35 smithi107 ceph-mon[32659]: pgmap v6130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10711.988052 secs 2024-07-26T15:49:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:35 smithi132 ceph-mon[37580]: pgmap v6130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10711.988052 secs 2024-07-26T15:49:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:37 smithi107 ceph-mon[32659]: pgmap v6131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:37 smithi132 ceph-mon[37580]: pgmap v6131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:37 smithi112 ceph-mon[37501]: pgmap v6131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:39 smithi107 ceph-mon[32659]: pgmap v6132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:39 smithi132 ceph-mon[37580]: pgmap v6132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:39 smithi112 ceph-mon[37501]: pgmap v6132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:49:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10716.988153 secs 2024-07-26T15:49:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10716.988153 secs 2024-07-26T15:49:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10716.988153 secs 2024-07-26T15:49:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:42 smithi107 ceph-mon[32659]: pgmap v6133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:42 smithi132 ceph-mon[37580]: pgmap v6133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:42 smithi112 ceph-mon[37501]: pgmap v6133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:44 smithi107 ceph-mon[32659]: pgmap v6134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:44 smithi132 ceph-mon[37580]: pgmap v6134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:44 smithi112 ceph-mon[37501]: pgmap v6134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:45.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:45.692+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:45.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:45.692+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:45.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:49:45.692+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:46 smithi107 ceph-mon[32659]: pgmap v6135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:49:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10721.988281 secs 2024-07-26T15:49:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:46 smithi132 ceph-mon[37580]: pgmap v6135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:49:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10721.988281 secs 2024-07-26T15:49:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:46 smithi112 ceph-mon[37501]: pgmap v6135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:49:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10721.988281 secs 2024-07-26T15:49:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:48.169+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:48.169+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:49:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:49:48.169+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:48 smithi107 ceph-mon[32659]: pgmap v6136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:49:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:48 smithi132 ceph-mon[37580]: pgmap v6136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:49:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:48 smithi112 ceph-mon[37501]: pgmap v6136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:49:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:50 smithi107 ceph-mon[32659]: pgmap v6137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:50 smithi132 ceph-mon[37580]: pgmap v6137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:50 smithi112 ceph-mon[37501]: pgmap v6137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10726.988370 secs 2024-07-26T15:49:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10726.988370 secs 2024-07-26T15:49:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10726.988370 secs 2024-07-26T15:49:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:52 smithi107 ceph-mon[32659]: pgmap v6138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:49:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:52 smithi132 ceph-mon[37580]: pgmap v6138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:49:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:52 smithi112 ceph-mon[37501]: pgmap v6138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:49:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:54 smithi132 ceph-mon[37580]: pgmap v6139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:54 smithi112 ceph-mon[37501]: pgmap v6139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:54 smithi107 ceph-mon[32659]: pgmap v6139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:49:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:56 smithi132 ceph-mon[37580]: pgmap v6140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10731.988494 secs 2024-07-26T15:49:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:56 smithi112 ceph-mon[37501]: pgmap v6140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10731.988494 secs 2024-07-26T15:49:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:56 smithi107 ceph-mon[32659]: pgmap v6140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:49:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10731.988494 secs 2024-07-26T15:49:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:49:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:49:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:49:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:58 smithi112 ceph-mon[37501]: pgmap v6141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:49:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:49:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:58 smithi107 ceph-mon[32659]: pgmap v6141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:49:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:49:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:58 smithi132 ceph-mon[37580]: pgmap v6141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:49:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:49:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:49:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:49:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:49:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:49:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:49:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:49:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:49:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:49:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:49:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:49:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:49:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:49:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:50:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:00 smithi112 ceph-mon[37501]: pgmap v6142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:50:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:50:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:00 smithi107 ceph-mon[32659]: pgmap v6142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:50:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:50:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:00 smithi132 ceph-mon[37580]: pgmap v6142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:50:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T15:50:00.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:00.694+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:00.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:00.694+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:00.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:00.694+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10736.988636 secs 2024-07-26T15:50:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10736.988636 secs 2024-07-26T15:50:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10736.988636 secs 2024-07-26T15:50:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:02 smithi112 ceph-mon[37501]: pgmap v6143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:50:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:02 smithi107 ceph-mon[32659]: pgmap v6143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:50:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:02 smithi132 ceph-mon[37580]: pgmap v6143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:50:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:03.180+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:03.180+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:03.180+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:04 smithi112 ceph-mon[37501]: pgmap v6144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:50:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:04 smithi107 ceph-mon[32659]: pgmap v6144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:50:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:04 smithi132 ceph-mon[37580]: pgmap v6144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:50:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:06 smithi112 ceph-mon[37501]: pgmap v6145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:50:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10741.988701 secs 2024-07-26T15:50:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:06 smithi107 ceph-mon[32659]: pgmap v6145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:50:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10741.988701 secs 2024-07-26T15:50:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:06 smithi132 ceph-mon[37580]: pgmap v6145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:50:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10741.988701 secs 2024-07-26T15:50:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:08 smithi112 ceph-mon[37501]: pgmap v6146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:08 smithi107 ceph-mon[32659]: pgmap v6146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:08 smithi132 ceph-mon[37580]: pgmap v6146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:09.553 INFO:tasks.workunit.client.0.smithi107.stderr:Terminated 2024-07-26T15:50:09.554 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-07-26T15:50:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:10 smithi132 ceph-mon[37580]: pgmap v6147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:10 smithi112 ceph-mon[37501]: pgmap v6147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:10 smithi107 ceph-mon[32659]: pgmap v6147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10746.988846 secs 2024-07-26T15:50:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10746.988846 secs 2024-07-26T15:50:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10746.988846 secs 2024-07-26T15:50:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:12 smithi112 ceph-mon[37501]: pgmap v6148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:12 smithi107 ceph-mon[32659]: pgmap v6148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:12 smithi132 ceph-mon[37580]: pgmap v6148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:14 smithi112 ceph-mon[37501]: pgmap v6149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:14 smithi107 ceph-mon[32659]: pgmap v6149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:14 smithi132 ceph-mon[37580]: pgmap v6149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:15.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:15.702+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:15.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:15.702+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:15.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:15.702+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:16 smithi112 ceph-mon[37501]: pgmap v6150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T15:50:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10751.988982 secs 2024-07-26T15:50:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:16 smithi107 ceph-mon[32659]: pgmap v6150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T15:50:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10751.988982 secs 2024-07-26T15:50:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:16 smithi132 ceph-mon[37580]: pgmap v6150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T15:50:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10751.988982 secs 2024-07-26T15:50:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:18.196+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:18.197+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:18.198 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:18.197+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:18 smithi112 ceph-mon[37501]: pgmap v6151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:50:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:18 smithi107 ceph-mon[32659]: pgmap v6151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:50:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:18 smithi132 ceph-mon[37580]: pgmap v6151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:50:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:20 smithi112 ceph-mon[37501]: pgmap v6152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:20 smithi107 ceph-mon[32659]: pgmap v6152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:20 smithi132 ceph-mon[37580]: pgmap v6152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10756.989129 secs 2024-07-26T15:50:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10756.989129 secs 2024-07-26T15:50:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10756.989129 secs 2024-07-26T15:50:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:22 smithi112 ceph-mon[37501]: pgmap v6153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:22.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:22 smithi132 ceph-mon[37580]: pgmap v6153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:22 smithi107 ceph-mon[32659]: pgmap v6153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:24 smithi112 ceph-mon[37501]: pgmap v6154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:24 smithi107 ceph-mon[32659]: pgmap v6154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:24 smithi132 ceph-mon[37580]: pgmap v6154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:26 smithi112 ceph-mon[37501]: pgmap v6155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10761.989279 secs 2024-07-26T15:50:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:26 smithi107 ceph-mon[32659]: pgmap v6155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10761.989279 secs 2024-07-26T15:50:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:26 smithi132 ceph-mon[37580]: pgmap v6155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10761.989279 secs 2024-07-26T15:50:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:28 smithi112 ceph-mon[37501]: pgmap v6156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:28 smithi107 ceph-mon[32659]: pgmap v6156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:28 smithi132 ceph-mon[37580]: pgmap v6156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:29 smithi107 ceph-mon[32659]: pgmap v6157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:29 smithi132 ceph-mon[37580]: pgmap v6157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:29 smithi112 ceph-mon[37501]: pgmap v6157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:30.711+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:30.711+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:30.713 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:30.711+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10766.989400 secs 2024-07-26T15:50:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:31 smithi112 ceph-mon[37501]: pgmap v6158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10766.989400 secs 2024-07-26T15:50:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:31 smithi107 ceph-mon[32659]: pgmap v6158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10766.989400 secs 2024-07-26T15:50:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:31 smithi132 ceph-mon[37580]: pgmap v6158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:33.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:33.212+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:33.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:33.212+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:33.213 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:33.212+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:33 smithi112 ceph-mon[37501]: pgmap v6159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:33 smithi107 ceph-mon[32659]: pgmap v6159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:33 smithi132 ceph-mon[37580]: pgmap v6159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:35 smithi112 ceph-mon[37501]: pgmap v6160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:35 smithi107 ceph-mon[32659]: pgmap v6160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:35 smithi132 ceph-mon[37580]: pgmap v6160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10771.989465 secs 2024-07-26T15:50:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10771.989465 secs 2024-07-26T15:50:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10771.989465 secs 2024-07-26T15:50:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:37 smithi107 ceph-mon[32659]: pgmap v6161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:37 smithi132 ceph-mon[37580]: pgmap v6161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:37 smithi112 ceph-mon[37501]: pgmap v6161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:39 smithi107 ceph-mon[32659]: pgmap v6162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:39 smithi132 ceph-mon[37580]: pgmap v6162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:39 smithi112 ceph-mon[37501]: pgmap v6162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10776.989562 secs 2024-07-26T15:50:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:41 smithi107 ceph-mon[32659]: pgmap v6163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10776.989562 secs 2024-07-26T15:50:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:41 smithi132 ceph-mon[37580]: pgmap v6163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10776.989562 secs 2024-07-26T15:50:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:41 smithi112 ceph-mon[37501]: pgmap v6163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:43 smithi107 ceph-mon[32659]: pgmap v6164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:43 smithi132 ceph-mon[37580]: pgmap v6164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:43 smithi112 ceph-mon[37501]: pgmap v6164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:45.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:45.727+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:45.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:45.727+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:45.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:50:45.727+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:45.729 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:45 smithi107 ceph-mon[32659]: pgmap v6165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:45 smithi132 ceph-mon[37580]: pgmap v6165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:45 smithi112 ceph-mon[37501]: pgmap v6165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10781.989688 secs 2024-07-26T15:50:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10781.989688 secs 2024-07-26T15:50:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10781.989688 secs 2024-07-26T15:50:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:47 smithi107 ceph-mon[32659]: pgmap v6166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:50:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:47 smithi132 ceph-mon[37580]: pgmap v6166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:50:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:47 smithi112 ceph-mon[37501]: pgmap v6166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:50:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:48.227+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:48.227+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:50:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:50:48.227+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:49 smithi107 ceph-mon[32659]: pgmap v6167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:49 smithi132 ceph-mon[37580]: pgmap v6167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:49 smithi112 ceph-mon[37501]: pgmap v6167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10786.989799 secs 2024-07-26T15:50:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:51 smithi107 ceph-mon[32659]: pgmap v6168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10786.989799 secs 2024-07-26T15:50:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:51 smithi132 ceph-mon[37580]: pgmap v6168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10786.989799 secs 2024-07-26T15:50:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:51 smithi112 ceph-mon[37501]: pgmap v6168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:53 smithi107 ceph-mon[32659]: pgmap v6169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:53 smithi132 ceph-mon[37580]: pgmap v6169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:53 smithi112 ceph-mon[37501]: pgmap v6169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:55 smithi107 ceph-mon[32659]: pgmap v6170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:55 smithi132 ceph-mon[37580]: pgmap v6170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:55 smithi112 ceph-mon[37501]: pgmap v6170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T15:50:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10791.989897 secs 2024-07-26T15:50:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10791.989897 secs 2024-07-26T15:50:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10791.989897 secs 2024-07-26T15:50:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:57 smithi132 ceph-mon[37580]: pgmap v6171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:57 smithi112 ceph-mon[37501]: pgmap v6171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:57 smithi107 ceph-mon[32659]: pgmap v6171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:50:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:59 smithi132 ceph-mon[37580]: pgmap v6172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:50:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:50:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:50:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:50:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:50:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:50:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:59 smithi112 ceph-mon[37501]: pgmap v6172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:50:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:50:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:50:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:50:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:50:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:50:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:59 smithi107 ceph-mon[32659]: pgmap v6172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:51:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:50:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:00.744 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:00.742+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:00.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:00.742+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:00.745 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:00.742+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10796.989977 secs 2024-07-26T15:51:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:01 smithi112 ceph-mon[37501]: pgmap v6173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10796.989977 secs 2024-07-26T15:51:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:01 smithi107 ceph-mon[32659]: pgmap v6173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10796.989977 secs 2024-07-26T15:51:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:01 smithi132 ceph-mon[37580]: pgmap v6173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:03.244 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:03.243+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:03.243+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:03.243+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:03 smithi112 ceph-mon[37501]: pgmap v6174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:51:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:51:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:03 smithi112 ceph-mon[37501]: pgmap v6175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:51:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:03 smithi112 ceph-mon[37501]: pgmap v6176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:51:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:03 smithi107 ceph-mon[32659]: pgmap v6174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:51:04.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:51:04.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:03 smithi107 ceph-mon[32659]: pgmap v6175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:51:04.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:03 smithi107 ceph-mon[32659]: pgmap v6176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:51:04.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:03 smithi132 ceph-mon[37580]: pgmap v6174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:51:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:51:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:03 smithi132 ceph-mon[37580]: pgmap v6175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:51:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:03 smithi132 ceph-mon[37580]: pgmap v6176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:51:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:51:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:05 smithi112 ceph-mon[37501]: pgmap v6177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:51:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10801.990098 secs 2024-07-26T15:51:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:05 smithi107 ceph-mon[32659]: pgmap v6177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:51:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10801.990098 secs 2024-07-26T15:51:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:05 smithi132 ceph-mon[37580]: pgmap v6177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:51:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10801.990098 secs 2024-07-26T15:51:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:07 smithi112 ceph-mon[37501]: pgmap v6178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T15:51:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:07 smithi107 ceph-mon[32659]: pgmap v6178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T15:51:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:07 smithi132 ceph-mon[37580]: pgmap v6178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T15:51:09.555 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-07-26T15:51:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:09 smithi112 ceph-mon[37501]: pgmap v6179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:51:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:09 smithi107 ceph-mon[32659]: pgmap v6179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:51:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:09 smithi132 ceph-mon[37580]: pgmap v6179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:51:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10806.990189 secs 2024-07-26T15:51:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10806.990189 secs 2024-07-26T15:51:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10806.990189 secs 2024-07-26T15:51:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:11.671+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T15:51:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:11.671+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:51:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:11.671+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800ff40 tx=0x7ff0f8051050 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:51:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:11.671+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800ff40 tx=0x7ff0f8051050 comp rx=0 tx=0).stop 2024-07-26T15:51:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:11.671+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c45e0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T15:51:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:11.671+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c45e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:51:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:11.671+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:51:11.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:11.671+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:51:11.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:11.671+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:51:11.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:11.673+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f806f040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:51:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:11 smithi112 ceph-mon[37501]: pgmap v6180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:51:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:11 smithi107 ceph-mon[32659]: pgmap v6180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:51:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:11 smithi132 ceph-mon[37580]: pgmap v6180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:51:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:13.776+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 msgr2=0x7efc5c0bfe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T15:51:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:13.776+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 msgr2=0x7efc5c0bfe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:51:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:13.776+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7efc680aa8b0 tx=0x7efc680e9000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:51:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:13.776+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7efc680aa8b0 tx=0x7efc680e9000 comp rx=0 tx=0).stop 2024-07-26T15:51:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:13.776+0000 7efc76800640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 msgr2=0x7efc5c0bfe40 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T15:51:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:13.776+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 msgr2=0x7efc5c0bfe40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:51:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:13.776+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0bfa30 0x7efc5c0bfe40 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:51:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:13.776+0000 7efc6f400640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2f20 0x7efc5c0c0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:51:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:13.777+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2f20 0x7efc5c0c0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:51:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:13.778+0000 7efc76800640 1 --2- 172.21.15.107:0/3520391650 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7efc5c0c2f20 0x7efc5c0c0380 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7efc5c003970 tx=0x7efc68094070 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:51:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:13 smithi112 ceph-mon[37501]: pgmap v6181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:51:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:13 smithi107 ceph-mon[32659]: pgmap v6181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:51:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:13 smithi132 ceph-mon[37580]: pgmap v6181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:51:14.483 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:14.482+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:51:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:14.482+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:51:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:14.483+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:51:14.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:14.483+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:51:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:15.748+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:15.748+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:15.748+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:15 smithi107 ceph-mon[32659]: pgmap v6182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10811.990321 secs 2024-07-26T15:51:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:15 smithi112 ceph-mon[37501]: pgmap v6182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10811.990321 secs 2024-07-26T15:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:15 smithi132 ceph-mon[37580]: pgmap v6182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10811.990321 secs 2024-07-26T15:51:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:16.675+0000 7efc6e000640 1 -- 172.21.15.107:0/3520391650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:51:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:16.675+0000 7efc5aa00640 1 -- 172.21.15.107:0/3975622631 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:51:16.682 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:16.681+0000 7efc5be00640 1 -- 172.21.15.107:0/3975622631 <== mon.1 v2:172.21.15.132:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7efc44006490 con 0x55aecd6b64d0 2024-07-26T15:51:16.683 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:16.681+0000 7efc6f400640 1 -- 172.21.15.107:0/3520391650 <== mon.0 v2:172.21.15.107:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc480065a0 con 0x7efc78103340 2024-07-26T15:51:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:17 smithi112 ceph-mon[37501]: pgmap v6183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:17 smithi107 ceph-mon[32659]: pgmap v6183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:17 smithi132 ceph-mon[37580]: pgmap v6183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:18.246+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:18.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:18.246+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:18.248 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:18.246+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:19 smithi112 ceph-mon[37501]: pgmap v6184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:20.282 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:19 smithi107 ceph-mon[32659]: pgmap v6184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:19 smithi132 ceph-mon[37580]: pgmap v6184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10816.990444 secs 2024-07-26T15:51:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10816.990444 secs 2024-07-26T15:51:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10816.990444 secs 2024-07-26T15:51:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:21 smithi112 ceph-mon[37501]: pgmap v6185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:21 smithi107 ceph-mon[32659]: pgmap v6185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:21 smithi132 ceph-mon[37580]: pgmap v6185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:23 smithi112 ceph-mon[37501]: pgmap v6186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:23 smithi107 ceph-mon[32659]: pgmap v6186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:23 smithi132 ceph-mon[37580]: pgmap v6186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:25 smithi112 ceph-mon[37501]: pgmap v6187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10821.990566 secs 2024-07-26T15:51:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:25 smithi107 ceph-mon[32659]: pgmap v6187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10821.990566 secs 2024-07-26T15:51:26.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:25 smithi132 ceph-mon[37580]: pgmap v6187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:26.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10821.990566 secs 2024-07-26T15:51:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:27 smithi112 ceph-mon[37501]: pgmap v6188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:27 smithi107 ceph-mon[32659]: pgmap v6188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:27 smithi132 ceph-mon[37580]: pgmap v6188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:29 smithi107 ceph-mon[32659]: pgmap v6189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:29 smithi132 ceph-mon[37580]: pgmap v6189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:29 smithi112 ceph-mon[37501]: pgmap v6189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:30.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:30.758+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:30.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:30.758+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:30.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:30.758+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10826.990706 secs 2024-07-26T15:51:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10826.990706 secs 2024-07-26T15:51:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10826.990706 secs 2024-07-26T15:51:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:31 smithi107 ceph-mon[32659]: pgmap v6190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:31 smithi132 ceph-mon[37580]: pgmap v6190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:31 smithi112 ceph-mon[37501]: pgmap v6190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:33.262+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:33.262+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:33.262+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:33 smithi107 ceph-mon[32659]: pgmap v6191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:33 smithi132 ceph-mon[37580]: pgmap v6191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:33 smithi112 ceph-mon[37501]: pgmap v6191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:36.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:35 smithi112 ceph-mon[37501]: pgmap v6192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:36.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10831.990787 secs 2024-07-26T15:51:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:35 smithi107 ceph-mon[32659]: pgmap v6192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10831.990787 secs 2024-07-26T15:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:35 smithi132 ceph-mon[37580]: pgmap v6192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10831.990787 secs 2024-07-26T15:51:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:37 smithi107 ceph-mon[32659]: pgmap v6193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:37 smithi132 ceph-mon[37580]: pgmap v6193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:37 smithi112 ceph-mon[37501]: pgmap v6193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:39 smithi107 ceph-mon[32659]: pgmap v6194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:39 smithi132 ceph-mon[37580]: pgmap v6194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:39 smithi112 ceph-mon[37501]: pgmap v6194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10836.990933 secs 2024-07-26T15:51:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10836.990933 secs 2024-07-26T15:51:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10836.990933 secs 2024-07-26T15:51:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:42 smithi107 ceph-mon[32659]: pgmap v6195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:42 smithi132 ceph-mon[37580]: pgmap v6195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:42 smithi112 ceph-mon[37501]: pgmap v6195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:44 smithi107 ceph-mon[32659]: pgmap v6196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:44 smithi132 ceph-mon[37580]: pgmap v6196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:44 smithi112 ceph-mon[37501]: pgmap v6196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:45.774+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:45.774+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:45.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:51:45.774+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:46 smithi107 ceph-mon[32659]: pgmap v6197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10841.991054 secs 2024-07-26T15:51:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:46 smithi132 ceph-mon[37580]: pgmap v6197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10841.991054 secs 2024-07-26T15:51:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:46 smithi112 ceph-mon[37501]: pgmap v6197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10841.991054 secs 2024-07-26T15:51:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:48.277+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:48.277+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:48.279 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:51:48.277+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:51:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:48 smithi107 ceph-mon[32659]: pgmap v6198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:48 smithi132 ceph-mon[37580]: pgmap v6198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:48 smithi112 ceph-mon[37501]: pgmap v6198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:50 smithi107 ceph-mon[32659]: pgmap v6199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:50 smithi132 ceph-mon[37580]: pgmap v6199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:50 smithi112 ceph-mon[37501]: pgmap v6199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10846.991181 secs 2024-07-26T15:51:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10846.991181 secs 2024-07-26T15:51:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10846.991181 secs 2024-07-26T15:51:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:52 smithi107 ceph-mon[32659]: pgmap v6200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:52 smithi132 ceph-mon[37580]: pgmap v6200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:52 smithi112 ceph-mon[37501]: pgmap v6200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:54 smithi132 ceph-mon[37580]: pgmap v6201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:54 smithi112 ceph-mon[37501]: pgmap v6201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:54 smithi107 ceph-mon[32659]: pgmap v6201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:51:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:56 smithi132 ceph-mon[37580]: pgmap v6202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10851.991262 secs 2024-07-26T15:51:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:56 smithi112 ceph-mon[37501]: pgmap v6202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10851.991262 secs 2024-07-26T15:51:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:56 smithi107 ceph-mon[32659]: pgmap v6202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10851.991262 secs 2024-07-26T15:51:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:51:58 smithi112 ceph-mon[37501]: pgmap v6203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:51:58 smithi107 ceph-mon[32659]: pgmap v6203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:51:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:51:58 smithi132 ceph-mon[37580]: pgmap v6203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:00 smithi112 ceph-mon[37501]: pgmap v6204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:00 smithi107 ceph-mon[32659]: pgmap v6204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:00 smithi132 ceph-mon[37580]: pgmap v6204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:00.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:00.789+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:52:00.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:00.789+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:00.791 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:00.789+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:52:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10856.991384 secs 2024-07-26T15:52:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10856.991384 secs 2024-07-26T15:52:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10856.991384 secs 2024-07-26T15:52:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:02 smithi112 ceph-mon[37501]: pgmap v6205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:02 smithi107 ceph-mon[32659]: pgmap v6205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:02 smithi132 ceph-mon[37580]: pgmap v6205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:52:03.292+0000 7efc77200640 1 -- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 msgr2=0x55aecda9ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:52:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:52:03.292+0000 7efc5be00640 1 client.25954 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:52:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi107.front.sepia.ceph.com.smithi107.stderr:2024-07-26T15:52:03.292+0000 7efc77200640 1 --2- 172.21.15.107:0/3975622631 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x55aecd695d80 0x55aecda9ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:52:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:52:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:52:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:04 smithi112 ceph-mon[37501]: pgmap v6206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:52:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:52:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:04 smithi107 ceph-mon[32659]: pgmap v6206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:52:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:52:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:04 smithi132 ceph-mon[37580]: pgmap v6206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:52:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:52:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:05 smithi112 ceph-mon[37501]: pgmap v6207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:52:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:05 smithi112 ceph-mon[37501]: pgmap v6208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:52:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:52:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:05 smithi107 ceph-mon[32659]: pgmap v6207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:52:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:05 smithi107 ceph-mon[32659]: pgmap v6208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:52:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:52:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:05 smithi132 ceph-mon[37580]: pgmap v6207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:52:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:05 smithi132 ceph-mon[37580]: pgmap v6208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:52:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:52:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10861.991504 secs 2024-07-26T15:52:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10861.991504 secs 2024-07-26T15:52:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10861.991504 secs 2024-07-26T15:52:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:07 smithi112 ceph-mon[37501]: pgmap v6209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:52:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:07 smithi107 ceph-mon[32659]: pgmap v6209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:52:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:07 smithi132 ceph-mon[37580]: pgmap v6209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:52:09.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:09 smithi112 ceph-mon[37501]: pgmap v6210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:52:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:09 smithi107 ceph-mon[32659]: pgmap v6210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:52:09.556 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2024-07-26T15:52:09.556 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-07-26T15:52:09.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-07-26T15:52:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:09 smithi132 ceph-mon[37580]: pgmap v6210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:52:10.628 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi107 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-07-26T15:52:10.836 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2f675414514647ca8566013e94eea4e3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi107 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-07-26T15:52:10.838 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-07-26T15:52:10.848 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi107.front.sepia.ceph.com 2024-07-26T15:52:10.901 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-07-26T15:52:10.910 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-07-26T15:52:10.910 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T15:52:10.910 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-07-26T15:52:10.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-07-26T15:52:11.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-07-26T15:52:11.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-07-26T15:52:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.378+0000 7f37e7e00640 1 -- 172.21.15.107:0/3502813628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e810b7c0 msgr2=0x7f37e8111ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T15:52:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.378+0000 7f37e7e00640 1 --2- 172.21.15.107:0/3502813628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e810b7c0 0x7f37e8111ff0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009be0 tx=0x7f37e002a6c0 comp rx=0 tx=0).stop 2024-07-26T15:52:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.378+0000 7f37e7e00640 1 -- 172.21.15.107:0/3502813628 shutdown_connections 2024-07-26T15:52:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.378+0000 7f37e7e00640 1 --2- 172.21.15.107:0/3502813628 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e810b7c0 0x7f37e8111ff0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:52:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.378+0000 7f37e7e00640 1 --2- 172.21.15.107:0/3502813628 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e810ae70 0x7f37e810b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:52:11.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.378+0000 7f37e7e00640 1 --2- 172.21.15.107:0/3502813628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37e8105540 0x7f37e8105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:52:11.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.378+0000 7f37e7e00640 1 -- 172.21.15.107:0/3502813628 >> 172.21.15.107:0/3502813628 conn(0x7f37e8100f20 msgr2=0x7f37e8103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T15:52:11.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.379+0000 7f37e7e00640 1 -- 172.21.15.107:0/3502813628 shutdown_connections 2024-07-26T15:52:11.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.379+0000 7f37e7e00640 1 -- 172.21.15.107:0/3502813628 wait complete. 2024-07-26T15:52:11.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.380+0000 7f37e7e00640 1 Processor -- start 2024-07-26T15:52:11.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.380+0000 7f37e7e00640 1 -- start start 2024-07-26T15:52:11.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e7e00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37e8105540 0x7f37e819be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:52:11.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e7e00640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e810ae70 0x7f37e819c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:52:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e7e00640 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e810b7c0 0x7f37e81a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:52:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e7e00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f37e8115470 con 0x7f37e8105540 2024-07-26T15:52:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e7e00640 1 -- --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81152f0 con 0x7f37e810ae70 2024-07-26T15:52:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e7e00640 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81155f0 con 0x7f37e810b7c0 2024-07-26T15:52:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e6a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37e8105540 0x7f37e819be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:52:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e6000640 1 --2- >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e810ae70 0x7f37e819c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:52:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e6a00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37e8105540 0x7f37e819be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:53014/0 (socket says 172.21.15.107:53014) 2024-07-26T15:52:11.385 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e6a00640 1 -- 172.21.15.107:0/4046735598 learned_addr learned my addr 172.21.15.107:0/4046735598 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-07-26T15:52:11.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e7400640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e810b7c0 0x7f37e81a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:52:11.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e6a00640 1 -- 172.21.15.107:0/4046735598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e810b7c0 msgr2=0x7f37e81a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T15:52:11.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e6a00640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37e810b7c0 0x7f37e81a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:52:11.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e6a00640 1 -- 172.21.15.107:0/4046735598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e810ae70 msgr2=0x7f37e819c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T15:52:11.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.381+0000 7f37e6a00640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] conn(0x7f37e810ae70 0x7f37e819c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:52:11.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.382+0000 7f37e6a00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e81a1da0 con 0x7f37e8105540 2024-07-26T15:52:11.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.382+0000 7f37e6a00640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37e8105540 0x7f37e819be70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f37d800e9c0 tx=0x7f37d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:52:11.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.382+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d800ccc0 con 0x7f37e8105540 2024-07-26T15:52:11.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.382+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37d80054d0 con 0x7f37e8105540 2024-07-26T15:52:11.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.382+0000 7f37e7e00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e81a2090 con 0x7f37e8105540 2024-07-26T15:52:11.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.382+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8010430 con 0x7f37e8105540 2024-07-26T15:52:11.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.382+0000 7f37e7e00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e81a8820 con 0x7f37e8105540 2024-07-26T15:52:11.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.383+0000 7f37e7e00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37e8105950 con 0x7f37e8105540 2024-07-26T15:52:11.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.386+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f37d8004f80 con 0x7f37e8105540 2024-07-26T15:52:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.388+0000 7f37eca00640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f37cc0b3dc0 0x7f37cc0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:52:11.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.389+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 11143+0+0 (secure 0 0 0) 0x7f37d80e4ea0 con 0x7f37e8105540 2024-07-26T15:52:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.389+0000 7f37e6000640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f37cc0b3dc0 0x7f37cc0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:52:11.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.390+0000 7f37e6000640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f37cc0b3dc0 0x7f37cc0b62d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f37e8103730 tx=0x7f37d4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:52:11.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.393+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f37d80ab710 con 0x7f37e8105540 2024-07-26T15:52:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:11 smithi112 ceph-mon[37501]: pgmap v6211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:52:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10866.991587 secs 2024-07-26T15:52:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:11 smithi107 ceph-mon[32659]: pgmap v6211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:52:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10866.991587 secs 2024-07-26T15:52:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:11 smithi132 ceph-mon[37580]: pgmap v6211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:52:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10866.991587 secs 2024-07-26T15:52:11.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:11.628+0000 7f37e7e00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f37e8063230 con 0x7f37cc0b3dc0 2024-07-26T15:52:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:13 smithi112 ceph-mon[37501]: from='client.15879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T15:52:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:13 smithi112 ceph-mon[37501]: pgmap v6212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:13 smithi107 ceph-mon[32659]: from='client.15879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T15:52:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:13 smithi107 ceph-mon[32659]: pgmap v6212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:13 smithi132 ceph-mon[37580]: from='client.15879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T15:52:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:13 smithi132 ceph-mon[37580]: pgmap v6212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:15 smithi112 ceph-mon[37501]: pgmap v6213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:52:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:15 smithi107 ceph-mon[32659]: pgmap v6213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:52:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:15 smithi132 ceph-mon[37580]: pgmap v6213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:52:15.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:15.794+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:52:15.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:15.794+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:52:15.796 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:15.794+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10871.991703 secs 2024-07-26T15:52:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10871.991703 secs 2024-07-26T15:52:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10871.991703 secs 2024-07-26T15:52:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:17 smithi112 ceph-mon[37501]: pgmap v6214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:17 smithi107 ceph-mon[32659]: pgmap v6214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:17 smithi132 ceph-mon[37580]: pgmap v6214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:19 smithi112 ceph-mon[37501]: pgmap v6215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:19 smithi107 ceph-mon[32659]: pgmap v6215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:19 smithi132 ceph-mon[37580]: pgmap v6215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:21 smithi112 ceph-mon[37501]: pgmap v6216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10876.991821 secs 2024-07-26T15:52:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:21 smithi107 ceph-mon[32659]: pgmap v6216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10876.991821 secs 2024-07-26T15:52:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:21 smithi132 ceph-mon[37580]: pgmap v6216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10876.991821 secs 2024-07-26T15:52:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:23 smithi112 ceph-mon[37501]: pgmap v6217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:23 smithi107 ceph-mon[32659]: pgmap v6217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:23 smithi132 ceph-mon[37580]: pgmap v6217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:25 smithi112 ceph-mon[37501]: pgmap v6218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:25 smithi107 ceph-mon[32659]: pgmap v6218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:25 smithi132 ceph-mon[37580]: pgmap v6218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10881.991945 secs 2024-07-26T15:52:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10881.991945 secs 2024-07-26T15:52:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10881.991945 secs 2024-07-26T15:52:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:27 smithi107 ceph-mon[32659]: pgmap v6219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:27 smithi132 ceph-mon[37580]: pgmap v6219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:27 smithi112 ceph-mon[37501]: pgmap v6219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:29 smithi107 ceph-mon[32659]: pgmap v6220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:29 smithi132 ceph-mon[37580]: pgmap v6220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:29 smithi112 ceph-mon[37501]: pgmap v6220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:30 smithi107 ceph-mon[32659]: pgmap v6221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:30 smithi132 ceph-mon[37580]: pgmap v6221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:30 smithi112 ceph-mon[37501]: pgmap v6221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:30.809+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:52:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:30.809+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:30.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:30.809+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:52:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10886.992087 secs 2024-07-26T15:52:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10886.992087 secs 2024-07-26T15:52:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10886.992087 secs 2024-07-26T15:52:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:32 smithi107 ceph-mon[32659]: pgmap v6222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:32 smithi132 ceph-mon[37580]: pgmap v6222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:32 smithi112 ceph-mon[37501]: pgmap v6222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:35 smithi112 ceph-mon[37501]: pgmap v6223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:35 smithi107 ceph-mon[32659]: pgmap v6223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:35 smithi132 ceph-mon[37580]: pgmap v6223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10891.992208 secs 2024-07-26T15:52:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10891.992208 secs 2024-07-26T15:52:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10891.992208 secs 2024-07-26T15:52:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:37 smithi112 ceph-mon[37501]: pgmap v6224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:37 smithi107 ceph-mon[32659]: pgmap v6224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:37 smithi132 ceph-mon[37580]: pgmap v6224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:39 smithi112 ceph-mon[37501]: pgmap v6225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:39 smithi107 ceph-mon[32659]: pgmap v6225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:39 smithi132 ceph-mon[37580]: pgmap v6225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:41 smithi112 ceph-mon[37501]: pgmap v6226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10896.992355 secs 2024-07-26T15:52:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:41 smithi107 ceph-mon[32659]: pgmap v6226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10896.992355 secs 2024-07-26T15:52:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:41 smithi132 ceph-mon[37580]: pgmap v6226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10896.992355 secs 2024-07-26T15:52:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:43 smithi112 ceph-mon[37501]: pgmap v6227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:43.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:43.498+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:52:43.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:43.498+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:52:43.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:43 smithi107 ceph-mon[32659]: pgmap v6227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:43.499+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:52:43.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:43.499+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:52:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:43 smithi132 ceph-mon[37580]: pgmap v6227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:45 smithi107 ceph-mon[32659]: pgmap v6228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:45 smithi132 ceph-mon[37580]: pgmap v6228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:45 smithi112 ceph-mon[37501]: pgmap v6228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:45.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:45.824+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:52:45.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:45.824+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:52:45.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:52:45.824+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10901.992440 secs 2024-07-26T15:52:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10901.992440 secs 2024-07-26T15:52:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10901.992440 secs 2024-07-26T15:52:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:47 smithi107 ceph-mon[32659]: pgmap v6229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:47 smithi132 ceph-mon[37580]: pgmap v6229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:47 smithi112 ceph-mon[37501]: pgmap v6229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:48 smithi107 ceph-mon[32659]: pgmap v6230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:48 smithi132 ceph-mon[37580]: pgmap v6230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:48 smithi112 ceph-mon[37501]: pgmap v6230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:51 smithi112 ceph-mon[37501]: pgmap v6231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10906.992567 secs 2024-07-26T15:52:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:51 smithi107 ceph-mon[32659]: pgmap v6231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10906.992567 secs 2024-07-26T15:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:51 smithi132 ceph-mon[37580]: pgmap v6231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10906.992567 secs 2024-07-26T15:52:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:53 smithi112 ceph-mon[37501]: pgmap v6232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:53 smithi107 ceph-mon[32659]: pgmap v6232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:53 smithi132 ceph-mon[37580]: pgmap v6232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:52:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:55 smithi112 ceph-mon[37501]: pgmap v6233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:55 smithi107 ceph-mon[32659]: pgmap v6233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:55 smithi132 ceph-mon[37580]: pgmap v6233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10911.992665 secs 2024-07-26T15:52:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10911.992665 secs 2024-07-26T15:52:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10911.992665 secs 2024-07-26T15:52:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:57 smithi107 ceph-mon[32659]: pgmap v6234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:57 smithi132 ceph-mon[37580]: pgmap v6234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:57 smithi112 ceph-mon[37501]: pgmap v6234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:52:59 smithi107 ceph-mon[32659]: pgmap v6235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:52:59 smithi132 ceph-mon[37580]: pgmap v6235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:52:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:52:59 smithi112 ceph-mon[37501]: pgmap v6235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:00.839+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:53:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:00.839+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:53:00.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:00.839+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:01 smithi107 ceph-mon[32659]: pgmap v6236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10916.992767 secs 2024-07-26T15:53:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:01 smithi132 ceph-mon[37580]: pgmap v6236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10916.992767 secs 2024-07-26T15:53:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:01 smithi112 ceph-mon[37501]: pgmap v6236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10916.992767 secs 2024-07-26T15:53:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:03 smithi107 ceph-mon[32659]: pgmap v6237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:03 smithi132 ceph-mon[37580]: pgmap v6237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:03 smithi112 ceph-mon[37501]: pgmap v6237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:04 smithi107 ceph-mon[32659]: pgmap v6238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:04 smithi132 ceph-mon[37580]: pgmap v6238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:04 smithi112 ceph-mon[37501]: pgmap v6238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:53:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:53:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:53:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:53:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:53:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:06 smithi107 ceph-mon[32659]: pgmap v6239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:53:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:06 smithi107 ceph-mon[32659]: pgmap v6240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:53:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10921.992888 secs 2024-07-26T15:53:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:53:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:53:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:06 smithi132 ceph-mon[37580]: pgmap v6239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:53:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:06 smithi132 ceph-mon[37580]: pgmap v6240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:53:06.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10921.992888 secs 2024-07-26T15:53:06.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:53:06.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:53:06.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:06 smithi112 ceph-mon[37501]: pgmap v6239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:53:06.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:06 smithi112 ceph-mon[37501]: pgmap v6240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:06.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:53:06.603 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10921.992888 secs 2024-07-26T15:53:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:08 smithi112 ceph-mon[37501]: pgmap v6241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:08 smithi107 ceph-mon[32659]: pgmap v6241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:08 smithi132 ceph-mon[37580]: pgmap v6241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:10 smithi112 ceph-mon[37501]: pgmap v6242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:10 smithi107 ceph-mon[32659]: pgmap v6242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:10 smithi132 ceph-mon[37580]: pgmap v6242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10926.993005 secs 2024-07-26T15:53:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10926.993005 secs 2024-07-26T15:53:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10926.993005 secs 2024-07-26T15:53:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:12 smithi112 ceph-mon[37501]: pgmap v6243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:53:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:12 smithi107 ceph-mon[32659]: pgmap v6243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:53:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:12 smithi132 ceph-mon[37580]: pgmap v6243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:53:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:14 smithi112 ceph-mon[37501]: pgmap v6244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:53:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:14 smithi107 ceph-mon[32659]: pgmap v6244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:53:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:14 smithi132 ceph-mon[37580]: pgmap v6244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:53:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:15.844+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:53:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:15.844+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:15.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:15.844+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:53:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:16 smithi112 ceph-mon[37501]: pgmap v6245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:53:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10931.993125 secs 2024-07-26T15:53:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:16 smithi107 ceph-mon[32659]: pgmap v6245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:53:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10931.993125 secs 2024-07-26T15:53:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:16 smithi132 ceph-mon[37580]: pgmap v6245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:53:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10931.993125 secs 2024-07-26T15:53:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:18 smithi107 ceph-mon[32659]: pgmap v6246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:18 smithi132 ceph-mon[37580]: pgmap v6246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:18 smithi112 ceph-mon[37501]: pgmap v6246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:20 smithi107 ceph-mon[32659]: pgmap v6247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:20 smithi132 ceph-mon[37580]: pgmap v6247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:20 smithi112 ceph-mon[37501]: pgmap v6247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10936.993255 secs 2024-07-26T15:53:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10936.993255 secs 2024-07-26T15:53:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10936.993255 secs 2024-07-26T15:53:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:22 smithi107 ceph-mon[32659]: pgmap v6248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:53:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:22 smithi132 ceph-mon[37580]: pgmap v6248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:53:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:22 smithi112 ceph-mon[37501]: pgmap v6248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:53:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:24 smithi107 ceph-mon[32659]: pgmap v6249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:24 smithi132 ceph-mon[37580]: pgmap v6249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:24 smithi112 ceph-mon[37501]: pgmap v6249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:26 smithi107 ceph-mon[32659]: pgmap v6250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10941.993391 secs 2024-07-26T15:53:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:26 smithi132 ceph-mon[37580]: pgmap v6250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10941.993391 secs 2024-07-26T15:53:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:26 smithi112 ceph-mon[37501]: pgmap v6250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10941.993391 secs 2024-07-26T15:53:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:28 smithi107 ceph-mon[32659]: pgmap v6251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:28 smithi132 ceph-mon[37580]: pgmap v6251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:28 smithi112 ceph-mon[37501]: pgmap v6251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:30.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:30.860+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:53:30.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:30.860+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:30.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:30.860+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:53:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:30 smithi107 ceph-mon[32659]: pgmap v6252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10946.993495 secs 2024-07-26T15:53:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:30 smithi132 ceph-mon[37580]: pgmap v6252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10946.993495 secs 2024-07-26T15:53:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:30 smithi112 ceph-mon[37501]: pgmap v6252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10946.993495 secs 2024-07-26T15:53:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:32 smithi107 ceph-mon[32659]: pgmap v6253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:32 smithi132 ceph-mon[37580]: pgmap v6253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:32 smithi112 ceph-mon[37501]: pgmap v6253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:34 smithi107 ceph-mon[32659]: pgmap v6254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:34 smithi132 ceph-mon[37580]: pgmap v6254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:35.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:34 smithi112 ceph-mon[37501]: pgmap v6254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10951.993575 secs 2024-07-26T15:53:36.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10951.993575 secs 2024-07-26T15:53:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10951.993575 secs 2024-07-26T15:53:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:36 smithi132 ceph-mon[37580]: pgmap v6255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:36 smithi112 ceph-mon[37501]: pgmap v6255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:36 smithi107 ceph-mon[32659]: pgmap v6255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:38 smithi132 ceph-mon[37580]: pgmap v6256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:38 smithi112 ceph-mon[37501]: pgmap v6256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:38 smithi107 ceph-mon[32659]: pgmap v6256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:40 smithi112 ceph-mon[37501]: pgmap v6257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10956.993669 secs 2024-07-26T15:53:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:40 smithi107 ceph-mon[32659]: pgmap v6257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10956.993669 secs 2024-07-26T15:53:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:40 smithi132 ceph-mon[37580]: pgmap v6257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10956.993669 secs 2024-07-26T15:53:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:42 smithi112 ceph-mon[37501]: pgmap v6258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:42 smithi107 ceph-mon[32659]: pgmap v6258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:42 smithi132 ceph-mon[37580]: pgmap v6258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:44 smithi112 ceph-mon[37501]: pgmap v6259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:44 smithi107 ceph-mon[32659]: pgmap v6259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:44 smithi132 ceph-mon[37580]: pgmap v6259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:45.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:45.860+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:53:45.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:45.860+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:45.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:53:45.860+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:53:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10961.993810 secs 2024-07-26T15:53:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10961.993810 secs 2024-07-26T15:53:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10961.993810 secs 2024-07-26T15:53:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:46 smithi112 ceph-mon[37501]: pgmap v6260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:46 smithi107 ceph-mon[32659]: pgmap v6260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:46 smithi132 ceph-mon[37580]: pgmap v6260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:48 smithi112 ceph-mon[37501]: pgmap v6261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:48 smithi107 ceph-mon[32659]: pgmap v6261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:48 smithi132 ceph-mon[37580]: pgmap v6261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:50 smithi112 ceph-mon[37501]: pgmap v6262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10966.993930 secs 2024-07-26T15:53:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:50 smithi107 ceph-mon[32659]: pgmap v6262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10966.993930 secs 2024-07-26T15:53:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:50 smithi132 ceph-mon[37580]: pgmap v6262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10966.993930 secs 2024-07-26T15:53:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:52 smithi112 ceph-mon[37501]: pgmap v6263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:52 smithi107 ceph-mon[32659]: pgmap v6263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:52 smithi132 ceph-mon[37580]: pgmap v6263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:54 smithi112 ceph-mon[37501]: pgmap v6264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:54 smithi107 ceph-mon[32659]: pgmap v6264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:54 smithi132 ceph-mon[37580]: pgmap v6264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10971.994047 secs 2024-07-26T15:53:56.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10971.994047 secs 2024-07-26T15:53:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10971.994047 secs 2024-07-26T15:53:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:56 smithi112 ceph-mon[37501]: pgmap v6265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:56 smithi107 ceph-mon[32659]: pgmap v6265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:56 smithi132 ceph-mon[37580]: pgmap v6265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:53:58 smithi112 ceph-mon[37501]: pgmap v6266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:53:58 smithi107 ceph-mon[32659]: pgmap v6266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:53:58 smithi132 ceph-mon[37580]: pgmap v6266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:00.875+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:54:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:00.875+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:00.875+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:54:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:00 smithi112 ceph-mon[37501]: pgmap v6267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10976.994194 secs 2024-07-26T15:54:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:00 smithi107 ceph-mon[32659]: pgmap v6267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10976.994194 secs 2024-07-26T15:54:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:00 smithi132 ceph-mon[37580]: pgmap v6267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10976.994194 secs 2024-07-26T15:54:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:02 smithi112 ceph-mon[37501]: pgmap v6268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:02 smithi107 ceph-mon[32659]: pgmap v6268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:02 smithi132 ceph-mon[37580]: pgmap v6268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:04 smithi112 ceph-mon[37501]: pgmap v6269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:04 smithi107 ceph-mon[32659]: pgmap v6269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:04 smithi132 ceph-mon[37580]: pgmap v6269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10981.994312 secs 2024-07-26T15:54:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:54:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10981.994312 secs 2024-07-26T15:54:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:54:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10981.994312 secs 2024-07-26T15:54:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:54:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:06 smithi112 ceph-mon[37501]: pgmap v6270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:54:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:54:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:54:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:06 smithi107 ceph-mon[32659]: pgmap v6270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:54:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:54:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:54:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:06 smithi132 ceph-mon[37580]: pgmap v6270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:54:07.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:54:07.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:54:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:07 smithi112 ceph-mon[37501]: pgmap v6271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:54:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:07 smithi112 ceph-mon[37501]: pgmap v6272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:07 smithi107 ceph-mon[32659]: pgmap v6271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:54:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:07 smithi107 ceph-mon[32659]: pgmap v6272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:07 smithi132 ceph-mon[37580]: pgmap v6271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:54:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:07 smithi132 ceph-mon[37580]: pgmap v6272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:09 smithi112 ceph-mon[37501]: pgmap v6273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:09 smithi107 ceph-mon[32659]: pgmap v6273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:09 smithi132 ceph-mon[37580]: pgmap v6273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10986.994401 secs 2024-07-26T15:54:11.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10986.994401 secs 2024-07-26T15:54:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10986.994401 secs 2024-07-26T15:54:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:11 smithi112 ceph-mon[37501]: pgmap v6274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:11 smithi107 ceph-mon[32659]: pgmap v6274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:11 smithi132 ceph-mon[37580]: pgmap v6274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:13 smithi107 ceph-mon[32659]: pgmap v6275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:13 smithi132 ceph-mon[37580]: pgmap v6275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:13 smithi112 ceph-mon[37501]: pgmap v6275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:14.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:14.514+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:54:14.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:14.515+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:54:14.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:14.515+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:54:14.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:14.515+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:54:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:15.891+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:54:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:15.891+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:54:15.893 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:15.891+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:15 smithi107 ceph-mon[32659]: pgmap v6276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10991.994538 secs 2024-07-26T15:54:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:15 smithi132 ceph-mon[37580]: pgmap v6276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10991.994538 secs 2024-07-26T15:54:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:15 smithi112 ceph-mon[37501]: pgmap v6276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10991.994538 secs 2024-07-26T15:54:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:17 smithi107 ceph-mon[32659]: pgmap v6277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:54:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:17 smithi132 ceph-mon[37580]: pgmap v6277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:54:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:17 smithi112 ceph-mon[37501]: pgmap v6277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:54:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:19 smithi107 ceph-mon[32659]: pgmap v6278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:19 smithi132 ceph-mon[37580]: pgmap v6278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:19 smithi112 ceph-mon[37501]: pgmap v6278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 10996.994632 secs 2024-07-26T15:54:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 10996.994632 secs 2024-07-26T15:54:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 10996.994632 secs 2024-07-26T15:54:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:21 smithi107 ceph-mon[32659]: pgmap v6279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:21 smithi132 ceph-mon[37580]: pgmap v6279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:21 smithi112 ceph-mon[37501]: pgmap v6279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:23 smithi107 ceph-mon[32659]: pgmap v6280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:23 smithi132 ceph-mon[37580]: pgmap v6280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:23 smithi112 ceph-mon[37501]: pgmap v6280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:26 smithi107 ceph-mon[32659]: pgmap v6281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11001.994697 secs 2024-07-26T15:54:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:26 smithi132 ceph-mon[37580]: pgmap v6281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11001.994697 secs 2024-07-26T15:54:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:26 smithi112 ceph-mon[37501]: pgmap v6281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11001.994697 secs 2024-07-26T15:54:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:28 smithi107 ceph-mon[32659]: pgmap v6282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:28 smithi132 ceph-mon[37580]: pgmap v6282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:28 smithi112 ceph-mon[37501]: pgmap v6282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:30 smithi107 ceph-mon[32659]: pgmap v6283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:30 smithi132 ceph-mon[37580]: pgmap v6283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:30 smithi112 ceph-mon[37501]: pgmap v6283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:30.902+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:54:30.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:30.902+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:30.904 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:30.902+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:54:31.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11006.994838 secs 2024-07-26T15:54:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11006.994838 secs 2024-07-26T15:54:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11006.994838 secs 2024-07-26T15:54:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:32 smithi107 ceph-mon[32659]: pgmap v6284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:32 smithi132 ceph-mon[37580]: pgmap v6284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:32 smithi112 ceph-mon[37501]: pgmap v6284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:34 smithi107 ceph-mon[32659]: pgmap v6285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:34 smithi132 ceph-mon[37580]: pgmap v6285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:34 smithi112 ceph-mon[37501]: pgmap v6285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:36 smithi132 ceph-mon[37580]: pgmap v6286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11011.994960 secs 2024-07-26T15:54:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:36 smithi112 ceph-mon[37501]: pgmap v6286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11011.994960 secs 2024-07-26T15:54:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:36 smithi107 ceph-mon[32659]: pgmap v6286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11011.994960 secs 2024-07-26T15:54:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:38 smithi132 ceph-mon[37580]: pgmap v6287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:38 smithi112 ceph-mon[37501]: pgmap v6287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:38 smithi107 ceph-mon[32659]: pgmap v6287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:40 smithi132 ceph-mon[37580]: pgmap v6288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:40 smithi112 ceph-mon[37501]: pgmap v6288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:40 smithi107 ceph-mon[32659]: pgmap v6288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11016.995102 secs 2024-07-26T15:54:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11016.995102 secs 2024-07-26T15:54:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11016.995102 secs 2024-07-26T15:54:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:42 smithi112 ceph-mon[37501]: pgmap v6289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:42 smithi107 ceph-mon[32659]: pgmap v6289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:42 smithi132 ceph-mon[37580]: pgmap v6289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:44 smithi112 ceph-mon[37501]: pgmap v6290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:44 smithi107 ceph-mon[32659]: pgmap v6290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:44 smithi132 ceph-mon[37580]: pgmap v6290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:45.917+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:54:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:45.917+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:54:45.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:54:45.917+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:46 smithi107 ceph-mon[32659]: pgmap v6291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11021.995218 secs 2024-07-26T15:54:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:46 smithi112 ceph-mon[37501]: pgmap v6291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11021.995218 secs 2024-07-26T15:54:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:46 smithi132 ceph-mon[37580]: pgmap v6291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11021.995218 secs 2024-07-26T15:54:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:48 smithi112 ceph-mon[37501]: pgmap v6292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:48 smithi107 ceph-mon[32659]: pgmap v6292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:48 smithi132 ceph-mon[37580]: pgmap v6292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:50 smithi112 ceph-mon[37501]: pgmap v6293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:50 smithi107 ceph-mon[32659]: pgmap v6293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:50 smithi132 ceph-mon[37580]: pgmap v6293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11026.995338 secs 2024-07-26T15:54:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11026.995338 secs 2024-07-26T15:54:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11026.995338 secs 2024-07-26T15:54:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:52 smithi112 ceph-mon[37501]: pgmap v6294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:52 smithi107 ceph-mon[32659]: pgmap v6294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:52 smithi132 ceph-mon[37580]: pgmap v6294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:54 smithi112 ceph-mon[37501]: pgmap v6295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:54 smithi107 ceph-mon[32659]: pgmap v6295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:54.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:54 smithi132 ceph-mon[37580]: pgmap v6295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:56 smithi112 ceph-mon[37501]: pgmap v6296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11031.995423 secs 2024-07-26T15:54:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:56 smithi107 ceph-mon[32659]: pgmap v6296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11031.995423 secs 2024-07-26T15:54:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:56 smithi132 ceph-mon[37580]: pgmap v6296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:54:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11031.995423 secs 2024-07-26T15:54:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:54:58 smithi107 ceph-mon[32659]: pgmap v6297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:54:58 smithi112 ceph-mon[37501]: pgmap v6297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:54:58 smithi132 ceph-mon[37580]: pgmap v6297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:00 smithi112 ceph-mon[37501]: pgmap v6298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:00 smithi107 ceph-mon[32659]: pgmap v6298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:00 smithi132 ceph-mon[37580]: pgmap v6298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:00.930 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:00.928+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:55:00.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:00.928+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:55:00.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:00.928+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11036.995508 secs 2024-07-26T15:55:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11036.995508 secs 2024-07-26T15:55:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11036.995508 secs 2024-07-26T15:55:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:02 smithi112 ceph-mon[37501]: pgmap v6299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:02 smithi107 ceph-mon[32659]: pgmap v6299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:02 smithi132 ceph-mon[37580]: pgmap v6299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:04 smithi112 ceph-mon[37501]: pgmap v6300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:04 smithi107 ceph-mon[32659]: pgmap v6300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:04 smithi132 ceph-mon[37580]: pgmap v6300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:06 smithi112 ceph-mon[37501]: pgmap v6301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:55:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11041.995629 secs 2024-07-26T15:55:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:06 smithi107 ceph-mon[32659]: pgmap v6301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:55:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11041.995629 secs 2024-07-26T15:55:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:06 smithi132 ceph-mon[37580]: pgmap v6301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:55:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11041.995629 secs 2024-07-26T15:55:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:55:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:55:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:55:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:08 smithi112 ceph-mon[37501]: pgmap v6302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:55:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:55:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:55:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:08 smithi107 ceph-mon[32659]: pgmap v6302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:55:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:55:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:55:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:08 smithi132 ceph-mon[37580]: pgmap v6302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:55:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:55:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:55:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:09 smithi112 ceph-mon[37501]: pgmap v6303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:55:09.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:09 smithi112 ceph-mon[37501]: pgmap v6304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:55:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:09 smithi107 ceph-mon[32659]: pgmap v6303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:55:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:09 smithi107 ceph-mon[32659]: pgmap v6304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:55:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:09 smithi132 ceph-mon[37580]: pgmap v6303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:55:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:09 smithi132 ceph-mon[37580]: pgmap v6304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:55:11.413 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:11.411+0000 7f37dea00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f37c00034c0 con 0x7f37e8105540 2024-07-26T15:55:11.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:11.412+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f37c00034c0 con 0x7f37e8105540 2024-07-26T15:55:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:11 smithi112 ceph-mon[37501]: pgmap v6305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:55:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11046.995728 secs 2024-07-26T15:55:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:11 smithi107 ceph-mon[32659]: pgmap v6305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:55:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11046.995728 secs 2024-07-26T15:55:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:11 smithi132 ceph-mon[37580]: pgmap v6305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:55:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11046.995728 secs 2024-07-26T15:55:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:13 smithi112 ceph-mon[37501]: pgmap v6306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:13 smithi107 ceph-mon[32659]: pgmap v6306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:13 smithi132 ceph-mon[37580]: pgmap v6306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:15 smithi112 ceph-mon[37501]: pgmap v6307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:55:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:15 smithi107 ceph-mon[32659]: pgmap v6307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:55:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:15 smithi132 ceph-mon[37580]: pgmap v6307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:55:15.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:15.944+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:55:15.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:15.945+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:55:15.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:15.945+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11051.995793 secs 2024-07-26T15:55:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11051.995793 secs 2024-07-26T15:55:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11051.995793 secs 2024-07-26T15:55:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:17 smithi107 ceph-mon[32659]: pgmap v6308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:55:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:17 smithi132 ceph-mon[37580]: pgmap v6308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:55:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:17 smithi112 ceph-mon[37501]: pgmap v6308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:55:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:19 smithi107 ceph-mon[32659]: pgmap v6309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:55:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:19 smithi132 ceph-mon[37580]: pgmap v6309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:55:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:19 smithi112 ceph-mon[37501]: pgmap v6309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:55:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:21 smithi107 ceph-mon[32659]: pgmap v6310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:55:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11056.995915 secs 2024-07-26T15:55:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:21 smithi132 ceph-mon[37580]: pgmap v6310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:55:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11056.995915 secs 2024-07-26T15:55:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:21 smithi112 ceph-mon[37501]: pgmap v6310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:55:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11056.995915 secs 2024-07-26T15:55:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:23 smithi107 ceph-mon[32659]: pgmap v6311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:55:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:23 smithi132 ceph-mon[37580]: pgmap v6311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:55:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:23 smithi112 ceph-mon[37501]: pgmap v6311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:55:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:25 smithi107 ceph-mon[32659]: pgmap v6312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:25 smithi132 ceph-mon[37580]: pgmap v6312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:25 smithi112 ceph-mon[37501]: pgmap v6312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11061.996049 secs 2024-07-26T15:55:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11061.996049 secs 2024-07-26T15:55:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11061.996049 secs 2024-07-26T15:55:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:27 smithi107 ceph-mon[32659]: pgmap v6313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:27 smithi132 ceph-mon[37580]: pgmap v6313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:27 smithi112 ceph-mon[37501]: pgmap v6313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:29 smithi107 ceph-mon[32659]: pgmap v6314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:29 smithi132 ceph-mon[37580]: pgmap v6314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:29 smithi112 ceph-mon[37501]: pgmap v6314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:30.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:30.960+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:55:30.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:30.960+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:55:30.962 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:30.960+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:31 smithi107 ceph-mon[32659]: pgmap v6315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11066.996188 secs 2024-07-26T15:55:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:31 smithi132 ceph-mon[37580]: pgmap v6315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:31.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11066.996188 secs 2024-07-26T15:55:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:31 smithi112 ceph-mon[37501]: pgmap v6315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11066.996188 secs 2024-07-26T15:55:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:32 smithi107 ceph-mon[32659]: pgmap v6316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:32 smithi132 ceph-mon[37580]: pgmap v6316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:32 smithi112 ceph-mon[37501]: pgmap v6316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:34 smithi132 ceph-mon[37580]: pgmap v6317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:34 smithi112 ceph-mon[37501]: pgmap v6317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:34 smithi107 ceph-mon[32659]: pgmap v6317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:36.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11071.996307 secs 2024-07-26T15:55:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11071.996307 secs 2024-07-26T15:55:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11071.996307 secs 2024-07-26T15:55:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:36 smithi112 ceph-mon[37501]: pgmap v6318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:36 smithi107 ceph-mon[32659]: pgmap v6318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:36 smithi132 ceph-mon[37580]: pgmap v6318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:38 smithi112 ceph-mon[37501]: pgmap v6319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:38 smithi107 ceph-mon[32659]: pgmap v6319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:38 smithi132 ceph-mon[37580]: pgmap v6319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:40 smithi112 ceph-mon[37501]: pgmap v6320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11076.996390 secs 2024-07-26T15:55:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:40 smithi107 ceph-mon[32659]: pgmap v6320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11076.996390 secs 2024-07-26T15:55:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:40 smithi132 ceph-mon[37580]: pgmap v6320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11076.996390 secs 2024-07-26T15:55:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:42 smithi112 ceph-mon[37501]: pgmap v6321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:42 smithi107 ceph-mon[32659]: pgmap v6321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:42 smithi132 ceph-mon[37580]: pgmap v6321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:45.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:44 smithi107 ceph-mon[32659]: pgmap v6322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:44 smithi112 ceph-mon[37501]: pgmap v6322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:44 smithi132 ceph-mon[37580]: pgmap v6322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:45.975+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:55:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:45.975+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:55:45.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:55:45.975+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11081.996492 secs 2024-07-26T15:55:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11081.996492 secs 2024-07-26T15:55:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11081.996492 secs 2024-07-26T15:55:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:46 smithi112 ceph-mon[37501]: pgmap v6323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:46 smithi107 ceph-mon[32659]: pgmap v6323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:46 smithi132 ceph-mon[37580]: pgmap v6323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:48 smithi112 ceph-mon[37501]: pgmap v6324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:48 smithi107 ceph-mon[32659]: pgmap v6324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:48 smithi132 ceph-mon[37580]: pgmap v6324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:50 smithi112 ceph-mon[37501]: pgmap v6325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11086.996580 secs 2024-07-26T15:55:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:50 smithi107 ceph-mon[32659]: pgmap v6325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11086.996580 secs 2024-07-26T15:55:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:50 smithi132 ceph-mon[37580]: pgmap v6325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11086.996580 secs 2024-07-26T15:55:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:52 smithi112 ceph-mon[37501]: pgmap v6326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:52 smithi107 ceph-mon[32659]: pgmap v6326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:52 smithi132 ceph-mon[37580]: pgmap v6326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:55:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:54 smithi112 ceph-mon[37501]: pgmap v6327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:54 smithi107 ceph-mon[32659]: pgmap v6327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:54 smithi132 ceph-mon[37580]: pgmap v6327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11091.996667 secs 2024-07-26T15:55:56.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11091.996667 secs 2024-07-26T15:55:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11091.996667 secs 2024-07-26T15:55:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:56 smithi112 ceph-mon[37501]: pgmap v6328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:56 smithi107 ceph-mon[32659]: pgmap v6328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:56 smithi132 ceph-mon[37580]: pgmap v6328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:55:58 smithi112 ceph-mon[37501]: pgmap v6329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:55:58 smithi107 ceph-mon[32659]: pgmap v6329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:55:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:55:58 smithi132 ceph-mon[37580]: pgmap v6329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:00.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:00.991+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:56:00.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:00.991+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:56:00.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:00.991+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:01.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:00 smithi107 ceph-mon[32659]: pgmap v6330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:01.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11096.996751 secs 2024-07-26T15:56:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:00 smithi112 ceph-mon[37501]: pgmap v6330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11096.996751 secs 2024-07-26T15:56:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:00 smithi132 ceph-mon[37580]: pgmap v6330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11096.996751 secs 2024-07-26T15:56:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:02 smithi112 ceph-mon[37501]: pgmap v6331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:02 smithi107 ceph-mon[32659]: pgmap v6331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:02 smithi132 ceph-mon[37580]: pgmap v6331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:04 smithi112 ceph-mon[37501]: pgmap v6332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:04 smithi107 ceph-mon[32659]: pgmap v6332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:04 smithi132 ceph-mon[37580]: pgmap v6332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11101.996855 secs 2024-07-26T15:56:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11101.996855 secs 2024-07-26T15:56:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11101.996855 secs 2024-07-26T15:56:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:06 smithi112 ceph-mon[37501]: pgmap v6333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:06 smithi107 ceph-mon[32659]: pgmap v6333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:07.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:06 smithi132 ceph-mon[37580]: pgmap v6333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:08 smithi112 ceph-mon[37501]: pgmap v6334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:56:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:08 smithi107 ceph-mon[32659]: pgmap v6334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:09.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:56:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:08 smithi132 ceph-mon[37580]: pgmap v6334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:56:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:56:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:56:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:09 smithi112 ceph-mon[37501]: pgmap v6335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:56:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:56:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:56:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:56:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:09 smithi107 ceph-mon[32659]: pgmap v6335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:56:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:56:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:56:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:56:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:09 smithi132 ceph-mon[37580]: pgmap v6335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:56:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:56:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11106.996985 secs 2024-07-26T15:56:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11106.996985 secs 2024-07-26T15:56:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11106.996985 secs 2024-07-26T15:56:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:11 smithi107 ceph-mon[32659]: pgmap v6336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:56:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:11 smithi132 ceph-mon[37580]: pgmap v6336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:56:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:11 smithi112 ceph-mon[37501]: pgmap v6336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:56:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:13 smithi107 ceph-mon[32659]: pgmap v6337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:56:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:13 smithi132 ceph-mon[37580]: pgmap v6337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:56:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:13 smithi112 ceph-mon[37501]: pgmap v6337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:56:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:15.998+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:56:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:15.998+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:56:16.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:15.998+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:15 smithi107 ceph-mon[32659]: pgmap v6338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:56:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11111.997105 secs 2024-07-26T15:56:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:15 smithi132 ceph-mon[37580]: pgmap v6338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:56:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11111.997105 secs 2024-07-26T15:56:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:15 smithi112 ceph-mon[37501]: pgmap v6338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:56:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11111.997105 secs 2024-07-26T15:56:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:17 smithi107 ceph-mon[32659]: pgmap v6339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:56:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:17 smithi132 ceph-mon[37580]: pgmap v6339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:56:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:17 smithi112 ceph-mon[37501]: pgmap v6339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:56:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:19 smithi107 ceph-mon[32659]: pgmap v6340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:56:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:19 smithi132 ceph-mon[37580]: pgmap v6340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:56:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:19 smithi112 ceph-mon[37501]: pgmap v6340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:56:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11116.997166 secs 2024-07-26T15:56:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11116.997166 secs 2024-07-26T15:56:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11116.997166 secs 2024-07-26T15:56:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:21 smithi107 ceph-mon[32659]: pgmap v6341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:21 smithi132 ceph-mon[37580]: pgmap v6341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:21 smithi112 ceph-mon[37501]: pgmap v6341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:24 smithi107 ceph-mon[32659]: pgmap v6342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:24 smithi132 ceph-mon[37580]: pgmap v6342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:24 smithi112 ceph-mon[37501]: pgmap v6342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:26 smithi107 ceph-mon[32659]: pgmap v6343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11121.997282 secs 2024-07-26T15:56:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:26 smithi132 ceph-mon[37580]: pgmap v6343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11121.997282 secs 2024-07-26T15:56:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:26 smithi112 ceph-mon[37501]: pgmap v6343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11121.997282 secs 2024-07-26T15:56:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:28 smithi107 ceph-mon[32659]: pgmap v6344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:28 smithi132 ceph-mon[37580]: pgmap v6344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:28 smithi112 ceph-mon[37501]: pgmap v6344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:30 smithi107 ceph-mon[32659]: pgmap v6345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:30 smithi132 ceph-mon[37580]: pgmap v6345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:30 smithi112 ceph-mon[37501]: pgmap v6345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:31.015 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:31.014+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:56:31.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:31.014+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:56:31.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:31.014+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11126.997380 secs 2024-07-26T15:56:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11126.997380 secs 2024-07-26T15:56:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11126.997380 secs 2024-07-26T15:56:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:32 smithi107 ceph-mon[32659]: pgmap v6346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:32 smithi132 ceph-mon[37580]: pgmap v6346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:32 smithi112 ceph-mon[37501]: pgmap v6346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:34 smithi107 ceph-mon[32659]: pgmap v6347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:34 smithi132 ceph-mon[37580]: pgmap v6347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:34 smithi112 ceph-mon[37501]: pgmap v6347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:36 smithi132 ceph-mon[37580]: pgmap v6348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11131.997502 secs 2024-07-26T15:56:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:36 smithi112 ceph-mon[37501]: pgmap v6348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11131.997502 secs 2024-07-26T15:56:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:36 smithi107 ceph-mon[32659]: pgmap v6348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11131.997502 secs 2024-07-26T15:56:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:38 smithi132 ceph-mon[37580]: pgmap v6349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:38 smithi112 ceph-mon[37501]: pgmap v6349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:38 smithi107 ceph-mon[32659]: pgmap v6349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:40 smithi132 ceph-mon[37580]: pgmap v6350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:40 smithi112 ceph-mon[37501]: pgmap v6350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:40 smithi107 ceph-mon[32659]: pgmap v6350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11136.997630 secs 2024-07-26T15:56:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11136.997630 secs 2024-07-26T15:56:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11136.997630 secs 2024-07-26T15:56:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:42 smithi112 ceph-mon[37501]: pgmap v6351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:42 smithi107 ceph-mon[32659]: pgmap v6351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:42 smithi132 ceph-mon[37580]: pgmap v6351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:44 smithi112 ceph-mon[37501]: pgmap v6352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:44 smithi107 ceph-mon[32659]: pgmap v6352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:44 smithi132 ceph-mon[37580]: pgmap v6352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:46.028+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:56:46.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:46.028+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:56:46.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:56:46.028+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:46 smithi107 ceph-mon[32659]: pgmap v6353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11141.997755 secs 2024-07-26T15:56:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:46 smithi112 ceph-mon[37501]: pgmap v6353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11141.997755 secs 2024-07-26T15:56:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:46 smithi132 ceph-mon[37580]: pgmap v6353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11141.997755 secs 2024-07-26T15:56:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:48 smithi112 ceph-mon[37501]: pgmap v6354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:48 smithi107 ceph-mon[32659]: pgmap v6354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:48 smithi132 ceph-mon[37580]: pgmap v6354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:50 smithi112 ceph-mon[37501]: pgmap v6355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:50 smithi107 ceph-mon[32659]: pgmap v6355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:50 smithi132 ceph-mon[37580]: pgmap v6355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11146.997875 secs 2024-07-26T15:56:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11146.997875 secs 2024-07-26T15:56:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11146.997875 secs 2024-07-26T15:56:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:52 smithi112 ceph-mon[37501]: pgmap v6356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:52 smithi107 ceph-mon[32659]: pgmap v6356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:52 smithi132 ceph-mon[37580]: pgmap v6356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:54 smithi112 ceph-mon[37501]: pgmap v6357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:54 smithi107 ceph-mon[32659]: pgmap v6357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:54.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:54 smithi132 ceph-mon[37580]: pgmap v6357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:56:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:56 smithi112 ceph-mon[37501]: pgmap v6358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11151.998009 secs 2024-07-26T15:56:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:56 smithi107 ceph-mon[32659]: pgmap v6358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11151.998009 secs 2024-07-26T15:56:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:56 smithi132 ceph-mon[37580]: pgmap v6358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11151.998009 secs 2024-07-26T15:56:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:56:58 smithi112 ceph-mon[37501]: pgmap v6359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:56:58 smithi107 ceph-mon[32659]: pgmap v6359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:56:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:56:58 smithi132 ceph-mon[37580]: pgmap v6359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:00 smithi112 ceph-mon[37501]: pgmap v6360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:57:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:00 smithi107 ceph-mon[32659]: pgmap v6360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:57:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:00 smithi132 ceph-mon[37580]: pgmap v6360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:57:01.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:01.035+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:57:01.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:01.035+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:57:01.038 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:01.035+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11156.998154 secs 2024-07-26T15:57:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11156.998154 secs 2024-07-26T15:57:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11156.998154 secs 2024-07-26T15:57:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:02 smithi112 ceph-mon[37501]: pgmap v6361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:57:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:02 smithi107 ceph-mon[32659]: pgmap v6361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:57:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:02 smithi132 ceph-mon[37580]: pgmap v6361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:57:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:04 smithi112 ceph-mon[37501]: pgmap v6362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:57:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:04 smithi107 ceph-mon[32659]: pgmap v6362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:57:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:04 smithi132 ceph-mon[37580]: pgmap v6362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:57:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:06 smithi112 ceph-mon[37501]: pgmap v6363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11161.998268 secs 2024-07-26T15:57:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:06 smithi107 ceph-mon[32659]: pgmap v6363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11161.998268 secs 2024-07-26T15:57:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:06 smithi132 ceph-mon[37580]: pgmap v6363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11161.998268 secs 2024-07-26T15:57:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:08 smithi112 ceph-mon[37501]: pgmap v6364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:08 smithi107 ceph-mon[32659]: pgmap v6364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:08 smithi132 ceph-mon[37580]: pgmap v6364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: pgmap v6365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:57:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:57:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:57:10.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:57:10.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:57:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: pgmap v6365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:57:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:57:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:57:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:57:10.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:57:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: pgmap v6365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:57:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:57:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:10.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:57:10.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:57:10.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:57:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:11 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:57:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:11 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:57:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:11 smithi112 ceph-mon[37501]: pgmap v6366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:57:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:11 smithi112 ceph-mon[37501]: pgmap v6367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11166.998340 secs 2024-07-26T15:57:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:11 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:57:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:11 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:57:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:11 smithi107 ceph-mon[32659]: pgmap v6366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:57:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:11 smithi107 ceph-mon[32659]: pgmap v6367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11166.998340 secs 2024-07-26T15:57:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:11 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T15:57:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:11 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T15:57:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:11 smithi132 ceph-mon[37580]: pgmap v6366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:57:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:11 smithi132 ceph-mon[37580]: pgmap v6367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11166.998340 secs 2024-07-26T15:57:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:13 smithi112 ceph-mon[37501]: pgmap v6368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:13 smithi107 ceph-mon[32659]: pgmap v6368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:13 smithi132 ceph-mon[37580]: pgmap v6368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:14.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:14.550+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T15:57:14.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:14.550+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T15:57:14.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:14.551+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T15:57:14.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:14.551+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T15:57:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:15 smithi112 ceph-mon[37501]: pgmap v6369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:15 smithi107 ceph-mon[32659]: pgmap v6369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:15 smithi132 ceph-mon[37580]: pgmap v6369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:16.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:16.052+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:57:16.053 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:16.052+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:57:16.054 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:16.052+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11171.998463 secs 2024-07-26T15:57:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11171.998463 secs 2024-07-26T15:57:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11171.998463 secs 2024-07-26T15:57:17.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:17 smithi107 ceph-mon[32659]: pgmap v6370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:57:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:17 smithi112 ceph-mon[37501]: pgmap v6370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:57:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:17 smithi132 ceph-mon[37580]: pgmap v6370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:57:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:19 smithi107 ceph-mon[32659]: pgmap v6371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:19 smithi132 ceph-mon[37580]: pgmap v6371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:19 smithi112 ceph-mon[37501]: pgmap v6371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:21 smithi107 ceph-mon[32659]: pgmap v6372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:57:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11176.998563 secs 2024-07-26T15:57:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:21 smithi132 ceph-mon[37580]: pgmap v6372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:57:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11176.998563 secs 2024-07-26T15:57:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:21 smithi112 ceph-mon[37501]: pgmap v6372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:57:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11176.998563 secs 2024-07-26T15:57:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:23 smithi107 ceph-mon[32659]: pgmap v6373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:23 smithi132 ceph-mon[37580]: pgmap v6373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:23 smithi112 ceph-mon[37501]: pgmap v6373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:25 smithi107 ceph-mon[32659]: pgmap v6374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:57:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:25 smithi132 ceph-mon[37580]: pgmap v6374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:57:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:25 smithi112 ceph-mon[37501]: pgmap v6374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:57:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11181.998685 secs 2024-07-26T15:57:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11181.998685 secs 2024-07-26T15:57:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11181.998685 secs 2024-07-26T15:57:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:27 smithi107 ceph-mon[32659]: pgmap v6375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:27 smithi132 ceph-mon[37580]: pgmap v6375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:27 smithi112 ceph-mon[37501]: pgmap v6375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:29 smithi107 ceph-mon[32659]: pgmap v6376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:29 smithi132 ceph-mon[37580]: pgmap v6376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:29 smithi112 ceph-mon[37501]: pgmap v6376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:31.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:31.067+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:57:31.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:31.067+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:57:31.069 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:31.067+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:31.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:31 smithi107 ceph-mon[32659]: pgmap v6377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:31.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11186.998804 secs 2024-07-26T15:57:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:31 smithi132 ceph-mon[37580]: pgmap v6377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11186.998804 secs 2024-07-26T15:57:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:31 smithi112 ceph-mon[37501]: pgmap v6377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11186.998804 secs 2024-07-26T15:57:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:33 smithi107 ceph-mon[32659]: pgmap v6378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:33 smithi132 ceph-mon[37580]: pgmap v6378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:33 smithi112 ceph-mon[37501]: pgmap v6378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:35 smithi107 ceph-mon[32659]: pgmap v6379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:57:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:35 smithi132 ceph-mon[37580]: pgmap v6379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:57:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:35 smithi112 ceph-mon[37501]: pgmap v6379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:57:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11191.998924 secs 2024-07-26T15:57:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11191.998924 secs 2024-07-26T15:57:36.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11191.998924 secs 2024-07-26T15:57:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:37 smithi107 ceph-mon[32659]: pgmap v6380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:37 smithi132 ceph-mon[37580]: pgmap v6380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:37 smithi112 ceph-mon[37501]: pgmap v6380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:39 smithi107 ceph-mon[32659]: pgmap v6381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:39 smithi132 ceph-mon[37580]: pgmap v6381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:39 smithi112 ceph-mon[37501]: pgmap v6381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:41 smithi107 ceph-mon[32659]: pgmap v6382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11196.999046 secs 2024-07-26T15:57:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:41 smithi132 ceph-mon[37580]: pgmap v6382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11196.999046 secs 2024-07-26T15:57:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:41 smithi112 ceph-mon[37501]: pgmap v6382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11196.999046 secs 2024-07-26T15:57:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:42 smithi107 ceph-mon[32659]: pgmap v6383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:42 smithi132 ceph-mon[37580]: pgmap v6383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:42 smithi112 ceph-mon[37501]: pgmap v6383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:57:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:45 smithi107 ceph-mon[32659]: pgmap v6384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:57:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:45 smithi132 ceph-mon[37580]: pgmap v6384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:57:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:45 smithi112 ceph-mon[37501]: pgmap v6384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:57:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:46.082+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:57:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:46.082+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:57:46.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:57:46.082+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11201.999129 secs 2024-07-26T15:57:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11201.999129 secs 2024-07-26T15:57:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11201.999129 secs 2024-07-26T15:57:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:47 smithi107 ceph-mon[32659]: pgmap v6385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:47.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:47 smithi132 ceph-mon[37580]: pgmap v6385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:47 smithi112 ceph-mon[37501]: pgmap v6385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:49 smithi107 ceph-mon[32659]: pgmap v6386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:57:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:49 smithi132 ceph-mon[37580]: pgmap v6386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:57:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:49 smithi112 ceph-mon[37501]: pgmap v6386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:57:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:51 smithi107 ceph-mon[32659]: pgmap v6387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:57:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11206.999245 secs 2024-07-26T15:57:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:51 smithi132 ceph-mon[37580]: pgmap v6387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:57:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11206.999245 secs 2024-07-26T15:57:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:51 smithi112 ceph-mon[37501]: pgmap v6387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:57:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11206.999245 secs 2024-07-26T15:57:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:53 smithi107 ceph-mon[32659]: pgmap v6388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:57:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:53 smithi132 ceph-mon[37580]: pgmap v6388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:57:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:53 smithi112 ceph-mon[37501]: pgmap v6388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:57:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:55 smithi107 ceph-mon[32659]: pgmap v6389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:57:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:55 smithi132 ceph-mon[37580]: pgmap v6389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:57:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:55 smithi112 ceph-mon[37501]: pgmap v6389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:57:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11211.999364 secs 2024-07-26T15:57:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11211.999364 secs 2024-07-26T15:57:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11211.999364 secs 2024-07-26T15:57:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:57 smithi107 ceph-mon[32659]: pgmap v6390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:57:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:57 smithi132 ceph-mon[37580]: pgmap v6390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:57:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:57 smithi112 ceph-mon[37501]: pgmap v6390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:57:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:57:59 smithi107 ceph-mon[32659]: pgmap v6391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:57:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:57:59 smithi132 ceph-mon[37580]: pgmap v6391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:57:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:57:59 smithi112 ceph-mon[37501]: pgmap v6391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T15:58:01.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:01.093+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:58:01.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:01.093+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:58:01.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:01.093+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:01.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:01 smithi107 ceph-mon[32659]: pgmap v6392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:58:01.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11216.999487 secs 2024-07-26T15:58:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:01 smithi132 ceph-mon[37580]: pgmap v6392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:58:01.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11216.999487 secs 2024-07-26T15:58:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:01 smithi112 ceph-mon[37501]: pgmap v6392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:58:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11216.999487 secs 2024-07-26T15:58:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:02 smithi132 ceph-mon[37580]: pgmap v6393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:02 smithi112 ceph-mon[37501]: pgmap v6393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:02 smithi107 ceph-mon[32659]: pgmap v6393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:05 smithi107 ceph-mon[32659]: pgmap v6394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:05 smithi132 ceph-mon[37580]: pgmap v6394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:05 smithi112 ceph-mon[37501]: pgmap v6394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11221.999604 secs 2024-07-26T15:58:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11221.999604 secs 2024-07-26T15:58:06.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11221.999604 secs 2024-07-26T15:58:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:07 smithi107 ceph-mon[32659]: pgmap v6395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:07 smithi132 ceph-mon[37580]: pgmap v6395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:07 smithi112 ceph-mon[37501]: pgmap v6395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:09 smithi107 ceph-mon[32659]: pgmap v6396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:09 smithi132 ceph-mon[37580]: pgmap v6396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:09 smithi112 ceph-mon[37501]: pgmap v6396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:11.445 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:11.444+0000 7f37dea00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f37c0005850 con 0x7f37e8105540 2024-07-26T15:58:11.446 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:11.444+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f37c0005850 con 0x7f37e8105540 2024-07-26T15:58:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: pgmap v6397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11226.999741 secs 2024-07-26T15:58:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:58:11.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: pgmap v6397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11226.999741 secs 2024-07-26T15:58:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:58:11.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: pgmap v6397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11226.999741 secs 2024-07-26T15:58:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:58:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.706 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.706 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:11.706 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:58:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:12 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:58:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:12 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:58:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:12 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:58:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:12 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:58:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:58:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:58:12.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:58:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:12 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:58:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:12 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:58:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:12 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:58:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:12 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:58:12.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:58:12.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:58:12.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:58:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:12 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T15:58:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:12 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T15:58:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:12 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T15:58:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:12 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T15:58:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:58:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:58:12.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:58:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:13 smithi107 ceph-mon[32659]: pgmap v6398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:58:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:13 smithi107 ceph-mon[32659]: pgmap v6399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:58:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:13 smithi132 ceph-mon[37580]: pgmap v6398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:58:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:13 smithi132 ceph-mon[37580]: pgmap v6399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:58:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:13 smithi112 ceph-mon[37501]: pgmap v6398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:58:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:13 smithi112 ceph-mon[37501]: pgmap v6399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:58:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:15 smithi107 ceph-mon[32659]: pgmap v6400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:15 smithi132 ceph-mon[37580]: pgmap v6400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:15 smithi112 ceph-mon[37501]: pgmap v6400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:16.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:16.109+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:58:16.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:16.109+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:58:16.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:16.109+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11231.999885 secs 2024-07-26T15:58:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11231.999885 secs 2024-07-26T15:58:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11231.999885 secs 2024-07-26T15:58:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:17 smithi107 ceph-mon[32659]: pgmap v6401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:17 smithi132 ceph-mon[37580]: pgmap v6401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:17 smithi112 ceph-mon[37501]: pgmap v6401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:18 smithi112 ceph-mon[37501]: pgmap v6402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:58:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:18 smithi107 ceph-mon[32659]: pgmap v6402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:58:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:18 smithi132 ceph-mon[37580]: pgmap v6402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:58:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:20 smithi112 ceph-mon[37501]: pgmap v6403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:20 smithi107 ceph-mon[32659]: pgmap v6403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:20 smithi132 ceph-mon[37580]: pgmap v6403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11237.000026 secs 2024-07-26T15:58:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11237.000026 secs 2024-07-26T15:58:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11237.000026 secs 2024-07-26T15:58:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:22 smithi112 ceph-mon[37501]: pgmap v6404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:58:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:22 smithi107 ceph-mon[32659]: pgmap v6404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:58:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:22 smithi132 ceph-mon[37580]: pgmap v6404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:58:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:24 smithi112 ceph-mon[37501]: pgmap v6405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:24 smithi107 ceph-mon[32659]: pgmap v6405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:24 smithi132 ceph-mon[37580]: pgmap v6405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:26 smithi112 ceph-mon[37501]: pgmap v6406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11242.000089 secs 2024-07-26T15:58:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:26 smithi107 ceph-mon[32659]: pgmap v6406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11242.000089 secs 2024-07-26T15:58:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:26 smithi132 ceph-mon[37580]: pgmap v6406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11242.000089 secs 2024-07-26T15:58:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:28 smithi112 ceph-mon[37501]: pgmap v6407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:28 smithi107 ceph-mon[32659]: pgmap v6407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:28 smithi132 ceph-mon[37580]: pgmap v6407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:30 smithi112 ceph-mon[37501]: pgmap v6408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:30 smithi107 ceph-mon[32659]: pgmap v6408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:30 smithi132 ceph-mon[37580]: pgmap v6408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:31.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:31.125+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:58:31.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:31.125+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:58:31.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:31.125+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11247.000215 secs 2024-07-26T15:58:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11247.000215 secs 2024-07-26T15:58:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11247.000215 secs 2024-07-26T15:58:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:32 smithi112 ceph-mon[37501]: pgmap v6409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:32 smithi107 ceph-mon[32659]: pgmap v6409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:32 smithi132 ceph-mon[37580]: pgmap v6409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:34 smithi112 ceph-mon[37501]: pgmap v6410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:34 smithi107 ceph-mon[32659]: pgmap v6410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:34 smithi132 ceph-mon[37580]: pgmap v6410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:36 smithi112 ceph-mon[37501]: pgmap v6411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11252.000298 secs 2024-07-26T15:58:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:36 smithi107 ceph-mon[32659]: pgmap v6411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11252.000298 secs 2024-07-26T15:58:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:36 smithi132 ceph-mon[37580]: pgmap v6411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11252.000298 secs 2024-07-26T15:58:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:38 smithi112 ceph-mon[37501]: pgmap v6412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:38 smithi107 ceph-mon[32659]: pgmap v6412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:38 smithi132 ceph-mon[37580]: pgmap v6412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:40 smithi112 ceph-mon[37501]: pgmap v6413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:40 smithi107 ceph-mon[32659]: pgmap v6413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:40 smithi132 ceph-mon[37580]: pgmap v6413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11257.000429 secs 2024-07-26T15:58:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11257.000429 secs 2024-07-26T15:58:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11257.000429 secs 2024-07-26T15:58:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:42 smithi112 ceph-mon[37501]: pgmap v6414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:42 smithi107 ceph-mon[32659]: pgmap v6414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:42 smithi132 ceph-mon[37580]: pgmap v6414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:44.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:44 smithi112 ceph-mon[37501]: pgmap v6415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:44 smithi107 ceph-mon[32659]: pgmap v6415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:44 smithi132 ceph-mon[37580]: pgmap v6415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:46.139+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:58:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:46.139+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:58:46.142 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:58:46.140+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11262.000515 secs 2024-07-26T15:58:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:46 smithi112 ceph-mon[37501]: pgmap v6416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11262.000515 secs 2024-07-26T15:58:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:46 smithi107 ceph-mon[32659]: pgmap v6416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11262.000515 secs 2024-07-26T15:58:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:46 smithi132 ceph-mon[37580]: pgmap v6416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:48 smithi107 ceph-mon[32659]: pgmap v6417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:48 smithi132 ceph-mon[37580]: pgmap v6417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:48 smithi112 ceph-mon[37501]: pgmap v6417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:50 smithi107 ceph-mon[32659]: pgmap v6418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:50 smithi132 ceph-mon[37580]: pgmap v6418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:50 smithi112 ceph-mon[37501]: pgmap v6418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11267.000635 secs 2024-07-26T15:58:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11267.000635 secs 2024-07-26T15:58:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11267.000635 secs 2024-07-26T15:58:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:52 smithi107 ceph-mon[32659]: pgmap v6419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:52 smithi132 ceph-mon[37580]: pgmap v6419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:52 smithi112 ceph-mon[37501]: pgmap v6419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:54 smithi107 ceph-mon[32659]: pgmap v6420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:54 smithi132 ceph-mon[37580]: pgmap v6420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:54 smithi112 ceph-mon[37501]: pgmap v6420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11272.000757 secs 2024-07-26T15:58:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:56 smithi107 ceph-mon[32659]: pgmap v6421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11272.000757 secs 2024-07-26T15:58:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:56 smithi132 ceph-mon[37580]: pgmap v6421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11272.000757 secs 2024-07-26T15:58:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:56 smithi112 ceph-mon[37501]: pgmap v6421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:58:58 smithi107 ceph-mon[32659]: pgmap v6422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:58:58 smithi132 ceph-mon[37580]: pgmap v6422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:58:58 smithi112 ceph-mon[37501]: pgmap v6422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:00 smithi107 ceph-mon[32659]: pgmap v6423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:00 smithi132 ceph-mon[37580]: pgmap v6423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:00 smithi112 ceph-mon[37501]: pgmap v6423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:01.146+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:59:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:01.146+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:59:01.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:01.146+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11277.000873 secs 2024-07-26T15:59:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11277.000873 secs 2024-07-26T15:59:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11277.000873 secs 2024-07-26T15:59:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:02 smithi107 ceph-mon[32659]: pgmap v6424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:59:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:02 smithi132 ceph-mon[37580]: pgmap v6424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:59:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:02 smithi112 ceph-mon[37501]: pgmap v6424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:59:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:04 smithi107 ceph-mon[32659]: pgmap v6425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:04 smithi132 ceph-mon[37580]: pgmap v6425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:04 smithi112 ceph-mon[37501]: pgmap v6425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11282.001018 secs 2024-07-26T15:59:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:06 smithi107 ceph-mon[32659]: pgmap v6426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11282.001018 secs 2024-07-26T15:59:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:06 smithi132 ceph-mon[37580]: pgmap v6426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11282.001018 secs 2024-07-26T15:59:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:06 smithi112 ceph-mon[37501]: pgmap v6426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:08 smithi107 ceph-mon[32659]: pgmap v6427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:08 smithi132 ceph-mon[37580]: pgmap v6427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:08 smithi112 ceph-mon[37501]: pgmap v6427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:10 smithi107 ceph-mon[32659]: pgmap v6428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:10 smithi132 ceph-mon[37580]: pgmap v6428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:10 smithi112 ceph-mon[37501]: pgmap v6428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11287.001137 secs 2024-07-26T15:59:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:59:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11287.001137 secs 2024-07-26T15:59:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:59:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11287.001137 secs 2024-07-26T15:59:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:59:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:12 smithi132 ceph-mon[37580]: pgmap v6429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:59:12.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:59:12.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:59:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:12 smithi112 ceph-mon[37501]: pgmap v6429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:59:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:59:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:59:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:12 smithi107 ceph-mon[32659]: pgmap v6429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:13.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:59:13.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:59:13.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T15:59:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:13 smithi112 ceph-mon[37501]: pgmap v6430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:59:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:13 smithi112 ceph-mon[37501]: pgmap v6431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:59:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:13 smithi107 ceph-mon[32659]: pgmap v6430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:59:14.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:13 smithi107 ceph-mon[32659]: pgmap v6431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:59:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:13 smithi132 ceph-mon[37580]: pgmap v6430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:59:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:13 smithi132 ceph-mon[37580]: pgmap v6431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:59:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:15 smithi112 ceph-mon[37501]: pgmap v6432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:15 smithi107 ceph-mon[32659]: pgmap v6432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:15 smithi132 ceph-mon[37580]: pgmap v6432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:16.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:16.162+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:59:16.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:16.162+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:59:16.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:16.162+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11292.001218 secs 2024-07-26T15:59:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11292.001218 secs 2024-07-26T15:59:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11292.001218 secs 2024-07-26T15:59:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:17 smithi112 ceph-mon[37501]: pgmap v6433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:17 smithi107 ceph-mon[32659]: pgmap v6433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:17 smithi132 ceph-mon[37580]: pgmap v6433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:19 smithi112 ceph-mon[37501]: pgmap v6434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:19 smithi107 ceph-mon[32659]: pgmap v6434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:19 smithi132 ceph-mon[37580]: pgmap v6434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11297.001336 secs 2024-07-26T15:59:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:21 smithi112 ceph-mon[37501]: pgmap v6435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11297.001336 secs 2024-07-26T15:59:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:21 smithi107 ceph-mon[32659]: pgmap v6435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11297.001336 secs 2024-07-26T15:59:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:21 smithi132 ceph-mon[37580]: pgmap v6435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:23 smithi112 ceph-mon[37501]: pgmap v6436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:59:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:23 smithi107 ceph-mon[32659]: pgmap v6436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:59:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:23 smithi132 ceph-mon[37580]: pgmap v6436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:59:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:25 smithi112 ceph-mon[37501]: pgmap v6437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:25 smithi107 ceph-mon[32659]: pgmap v6437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:25 smithi132 ceph-mon[37580]: pgmap v6437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11302.001435 secs 2024-07-26T15:59:26.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11302.001435 secs 2024-07-26T15:59:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11302.001435 secs 2024-07-26T15:59:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:27 smithi112 ceph-mon[37501]: pgmap v6438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:27 smithi107 ceph-mon[32659]: pgmap v6438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:27 smithi132 ceph-mon[37580]: pgmap v6438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:29 smithi112 ceph-mon[37501]: pgmap v6439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:30.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:29 smithi107 ceph-mon[32659]: pgmap v6439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:29 smithi132 ceph-mon[37580]: pgmap v6439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:31.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:31.164+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:59:31.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:31.164+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:59:31.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:31.164+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11307.001558 secs 2024-07-26T15:59:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:31 smithi112 ceph-mon[37501]: pgmap v6440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11307.001558 secs 2024-07-26T15:59:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:31 smithi107 ceph-mon[32659]: pgmap v6440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11307.001558 secs 2024-07-26T15:59:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:31 smithi132 ceph-mon[37580]: pgmap v6440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:33 smithi112 ceph-mon[37501]: pgmap v6441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:33 smithi107 ceph-mon[32659]: pgmap v6441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:33 smithi132 ceph-mon[37580]: pgmap v6441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:35 smithi112 ceph-mon[37501]: pgmap v6442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:35 smithi107 ceph-mon[32659]: pgmap v6442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:35 smithi132 ceph-mon[37580]: pgmap v6442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:36.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11312.001679 secs 2024-07-26T15:59:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11312.001679 secs 2024-07-26T15:59:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11312.001679 secs 2024-07-26T15:59:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:37 smithi112 ceph-mon[37501]: pgmap v6443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:37 smithi107 ceph-mon[32659]: pgmap v6443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:37 smithi132 ceph-mon[37580]: pgmap v6443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:39 smithi112 ceph-mon[37501]: pgmap v6444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:39 smithi107 ceph-mon[32659]: pgmap v6444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:39 smithi132 ceph-mon[37580]: pgmap v6444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11317.001783 secs 2024-07-26T15:59:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:41 smithi112 ceph-mon[37501]: pgmap v6445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11317.001783 secs 2024-07-26T15:59:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:41 smithi107 ceph-mon[32659]: pgmap v6445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11317.001783 secs 2024-07-26T15:59:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:41 smithi132 ceph-mon[37580]: pgmap v6445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:43 smithi112 ceph-mon[37501]: pgmap v6446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:43 smithi107 ceph-mon[32659]: pgmap v6446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:43 smithi132 ceph-mon[37580]: pgmap v6446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:45 smithi112 ceph-mon[37501]: pgmap v6447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:45 smithi107 ceph-mon[32659]: pgmap v6447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:45 smithi132 ceph-mon[37580]: pgmap v6447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:46.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:46.166+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T15:59:46.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:46.166+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T15:59:46.170 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T15:59:46.166+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11322.001912 secs 2024-07-26T15:59:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11322.001912 secs 2024-07-26T15:59:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11322.001912 secs 2024-07-26T15:59:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:47 smithi112 ceph-mon[37501]: pgmap v6448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:47 smithi107 ceph-mon[32659]: pgmap v6448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:47 smithi132 ceph-mon[37580]: pgmap v6448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:49 smithi112 ceph-mon[37501]: pgmap v6449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:49 smithi107 ceph-mon[32659]: pgmap v6449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:49 smithi132 ceph-mon[37580]: pgmap v6449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11327.002034 secs 2024-07-26T15:59:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:51 smithi107 ceph-mon[32659]: pgmap v6450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11327.002034 secs 2024-07-26T15:59:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:51 smithi132 ceph-mon[37580]: pgmap v6450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11327.002034 secs 2024-07-26T15:59:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:51 smithi112 ceph-mon[37501]: pgmap v6450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:53 smithi107 ceph-mon[32659]: pgmap v6451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:53 smithi132 ceph-mon[37580]: pgmap v6451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:53 smithi112 ceph-mon[37501]: pgmap v6451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:55 smithi107 ceph-mon[32659]: pgmap v6452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:55 smithi132 ceph-mon[37580]: pgmap v6452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:55 smithi112 ceph-mon[37501]: pgmap v6452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:59:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11332.002162 secs 2024-07-26T15:59:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11332.002162 secs 2024-07-26T15:59:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11332.002162 secs 2024-07-26T15:59:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:57 smithi107 ceph-mon[32659]: pgmap v6453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:57 smithi132 ceph-mon[37580]: pgmap v6453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:57 smithi112 ceph-mon[37501]: pgmap v6453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 15:59:59 smithi107 ceph-mon[32659]: pgmap v6454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 15:59:59 smithi132 ceph-mon[37580]: pgmap v6454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:00.081 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 15:59:59 smithi112 ceph-mon[37501]: pgmap v6454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:00:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:00:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:01.170+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:00:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:01.170+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:00:01.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:01.170+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:00:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11337.002249 secs 2024-07-26T16:00:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:01 smithi107 ceph-mon[32659]: pgmap v6455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11337.002249 secs 2024-07-26T16:00:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:01 smithi132 ceph-mon[37580]: pgmap v6455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11337.002249 secs 2024-07-26T16:00:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:01 smithi112 ceph-mon[37501]: pgmap v6455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:03 smithi107 ceph-mon[32659]: pgmap v6456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:03 smithi132 ceph-mon[37580]: pgmap v6456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:03 smithi112 ceph-mon[37501]: pgmap v6456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:05 smithi107 ceph-mon[32659]: pgmap v6457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:00:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:05 smithi132 ceph-mon[37580]: pgmap v6457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:00:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:05 smithi112 ceph-mon[37501]: pgmap v6457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:00:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11342.002370 secs 2024-07-26T16:00:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11342.002370 secs 2024-07-26T16:00:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11342.002370 secs 2024-07-26T16:00:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:07 smithi107 ceph-mon[32659]: pgmap v6458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:07 smithi132 ceph-mon[37580]: pgmap v6458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:07 smithi112 ceph-mon[37501]: pgmap v6458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:09 smithi107 ceph-mon[32659]: pgmap v6459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:09 smithi132 ceph-mon[37580]: pgmap v6459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:09 smithi112 ceph-mon[37501]: pgmap v6459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11347.002507 secs 2024-07-26T16:00:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:11 smithi107 ceph-mon[32659]: pgmap v6460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11347.002507 secs 2024-07-26T16:00:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:11 smithi132 ceph-mon[37580]: pgmap v6460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11347.002507 secs 2024-07-26T16:00:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:11 smithi112 ceph-mon[37501]: pgmap v6460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:00:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:00:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:00:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:13 smithi132 ceph-mon[37580]: pgmap v6461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:14.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:00:14.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:00:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:13 smithi112 ceph-mon[37501]: pgmap v6461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:00:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:00:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:13 smithi107 ceph-mon[32659]: pgmap v6461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:00:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:00:14.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:14.582+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:00:14.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:14.582+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:00:14.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:14.584+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:00:14.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:14.584+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:00:15.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:00:15.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:00:15.047 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:00:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:00:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:00:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:00:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:00:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:00:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:00:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:16.187+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:00:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:16.187+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:00:16.189 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:16.187+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:16.190 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:15 smithi107 ceph-mon[32659]: pgmap v6462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:00:16.190 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:15 smithi107 ceph-mon[32659]: pgmap v6463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:00:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:15 smithi112 ceph-mon[37501]: pgmap v6462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:00:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:15 smithi112 ceph-mon[37501]: pgmap v6463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:00:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:15 smithi132 ceph-mon[37580]: pgmap v6462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:00:16.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:15 smithi132 ceph-mon[37580]: pgmap v6463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:00:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11352.002609 secs 2024-07-26T16:00:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11352.002609 secs 2024-07-26T16:00:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11352.002609 secs 2024-07-26T16:00:18.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:17 smithi112 ceph-mon[37501]: pgmap v6464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:00:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:17 smithi107 ceph-mon[32659]: pgmap v6464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:00:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:17 smithi132 ceph-mon[37580]: pgmap v6464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:00:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:19 smithi112 ceph-mon[37501]: pgmap v6465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:19 smithi107 ceph-mon[32659]: pgmap v6465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:19 smithi132 ceph-mon[37580]: pgmap v6465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11357.002738 secs 2024-07-26T16:00:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11357.002738 secs 2024-07-26T16:00:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11357.002738 secs 2024-07-26T16:00:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:21 smithi112 ceph-mon[37501]: pgmap v6466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:21 smithi107 ceph-mon[32659]: pgmap v6466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:21 smithi132 ceph-mon[37580]: pgmap v6466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:23 smithi112 ceph-mon[37501]: pgmap v6467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:23 smithi107 ceph-mon[32659]: pgmap v6467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:23 smithi132 ceph-mon[37580]: pgmap v6467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:25 smithi112 ceph-mon[37501]: pgmap v6468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:00:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11362.002867 secs 2024-07-26T16:00:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:25 smithi107 ceph-mon[32659]: pgmap v6468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:00:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11362.002867 secs 2024-07-26T16:00:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:25 smithi132 ceph-mon[37580]: pgmap v6468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:00:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11362.002867 secs 2024-07-26T16:00:26.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:26.467+0000 7f37dea00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f37c0005b70 con 0x7f37e8105540 2024-07-26T16:00:26.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:26.468+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f37c0005b70 con 0x7f37e8105540 2024-07-26T16:00:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:27 smithi112 ceph-mon[37501]: pgmap v6469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:27 smithi107 ceph-mon[32659]: pgmap v6469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:27 smithi132 ceph-mon[37580]: pgmap v6469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:29 smithi112 ceph-mon[37501]: pgmap v6470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:29 smithi107 ceph-mon[32659]: pgmap v6470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:29 smithi132 ceph-mon[37580]: pgmap v6470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:31.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11367.002991 secs 2024-07-26T16:00:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11367.002991 secs 2024-07-26T16:00:31.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:31.202+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:00:31.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:31.202+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:00:31.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:31.202+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11367.002991 secs 2024-07-26T16:00:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:31 smithi112 ceph-mon[37501]: pgmap v6471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:31 smithi107 ceph-mon[32659]: pgmap v6471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:31 smithi132 ceph-mon[37580]: pgmap v6471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:33 smithi112 ceph-mon[37501]: pgmap v6472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:33 smithi107 ceph-mon[32659]: pgmap v6472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:33 smithi132 ceph-mon[37580]: pgmap v6472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:35 smithi112 ceph-mon[37501]: pgmap v6473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11372.003114 secs 2024-07-26T16:00:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:35 smithi107 ceph-mon[32659]: pgmap v6473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11372.003114 secs 2024-07-26T16:00:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:35 smithi132 ceph-mon[37580]: pgmap v6473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11372.003114 secs 2024-07-26T16:00:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:37 smithi112 ceph-mon[37501]: pgmap v6474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:37 smithi107 ceph-mon[32659]: pgmap v6474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:37 smithi132 ceph-mon[37580]: pgmap v6474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:39 smithi112 ceph-mon[37501]: pgmap v6475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:39 smithi107 ceph-mon[32659]: pgmap v6475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:39 smithi132 ceph-mon[37580]: pgmap v6475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11377.003252 secs 2024-07-26T16:00:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11377.003252 secs 2024-07-26T16:00:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11377.003252 secs 2024-07-26T16:00:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:41 smithi112 ceph-mon[37501]: pgmap v6476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:41 smithi107 ceph-mon[32659]: pgmap v6476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:41 smithi132 ceph-mon[37580]: pgmap v6476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:43 smithi112 ceph-mon[37501]: pgmap v6477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:43 smithi107 ceph-mon[32659]: pgmap v6477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:43 smithi132 ceph-mon[37580]: pgmap v6477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:45 smithi112 ceph-mon[37501]: pgmap v6478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11382.003336 secs 2024-07-26T16:00:46.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:46.210+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:00:46.212 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:46.210+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:00:46.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:46.210+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:46.213 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:45 smithi107 ceph-mon[32659]: pgmap v6478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:46.214 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11382.003336 secs 2024-07-26T16:00:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:45 smithi132 ceph-mon[37580]: pgmap v6478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:46.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11382.003336 secs 2024-07-26T16:00:48.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:47 smithi112 ceph-mon[37501]: pgmap v6479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:47 smithi107 ceph-mon[32659]: pgmap v6479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:47 smithi132 ceph-mon[37580]: pgmap v6479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:49 smithi112 ceph-mon[37501]: pgmap v6480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:49 smithi107 ceph-mon[32659]: pgmap v6480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:49 smithi132 ceph-mon[37580]: pgmap v6480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11387.003451 secs 2024-07-26T16:00:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11387.003451 secs 2024-07-26T16:00:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11387.003451 secs 2024-07-26T16:00:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:51 smithi107 ceph-mon[32659]: pgmap v6481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:51 smithi132 ceph-mon[37580]: pgmap v6481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:51 smithi112 ceph-mon[37501]: pgmap v6481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:54.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:53 smithi107 ceph-mon[32659]: pgmap v6482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:53 smithi132 ceph-mon[37580]: pgmap v6482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:53 smithi112 ceph-mon[37501]: pgmap v6482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:00:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:55 smithi107 ceph-mon[32659]: pgmap v6483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11392.003569 secs 2024-07-26T16:00:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:55 smithi132 ceph-mon[37580]: pgmap v6483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11392.003569 secs 2024-07-26T16:00:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:55 smithi112 ceph-mon[37501]: pgmap v6483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:00:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11392.003569 secs 2024-07-26T16:00:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:57 smithi107 ceph-mon[32659]: pgmap v6484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:57 smithi132 ceph-mon[37580]: pgmap v6484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:57 smithi112 ceph-mon[37501]: pgmap v6484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:58.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:58.591+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:00:58.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:58.591+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:00:58.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:58.591+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:00:58.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:00:58.591+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:01:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:00:59 smithi107 ceph-mon[32659]: pgmap v6485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:00:59 smithi132 ceph-mon[37580]: pgmap v6485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:00:59 smithi112 ceph-mon[37501]: pgmap v6485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:01.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:01.225+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:01:01.227 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:01.225+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:01:01.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:01.225+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11397.003727 secs 2024-07-26T16:01:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11397.003727 secs 2024-07-26T16:01:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11397.003727 secs 2024-07-26T16:01:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:01 smithi107 ceph-mon[32659]: pgmap v6486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:01 smithi132 ceph-mon[37580]: pgmap v6486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:01 smithi112 ceph-mon[37501]: pgmap v6486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:04 smithi107 ceph-mon[32659]: pgmap v6487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:04 smithi132 ceph-mon[37580]: pgmap v6487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:04 smithi112 ceph-mon[37501]: pgmap v6487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:06 smithi107 ceph-mon[32659]: pgmap v6488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11402.003880 secs 2024-07-26T16:01:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:06 smithi132 ceph-mon[37580]: pgmap v6488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11402.003880 secs 2024-07-26T16:01:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:06 smithi112 ceph-mon[37501]: pgmap v6488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11402.003880 secs 2024-07-26T16:01:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:08 smithi107 ceph-mon[32659]: pgmap v6489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:01:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:08 smithi132 ceph-mon[37580]: pgmap v6489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:01:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:08 smithi112 ceph-mon[37501]: pgmap v6489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:01:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:10 smithi107 ceph-mon[32659]: pgmap v6490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:10 smithi132 ceph-mon[37580]: pgmap v6490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:10 smithi112 ceph-mon[37501]: pgmap v6490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11407.004027 secs 2024-07-26T16:01:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11407.004027 secs 2024-07-26T16:01:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11407.004027 secs 2024-07-26T16:01:11.476 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:11.475+0000 7f37dea00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f37c0005ed0 con 0x7f37e8105540 2024-07-26T16:01:11.477 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:11.475+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f37c0005ed0 con 0x7f37e8105540 2024-07-26T16:01:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:12 smithi107 ceph-mon[32659]: pgmap v6491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:12 smithi132 ceph-mon[37580]: pgmap v6491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:12 smithi112 ceph-mon[37501]: pgmap v6491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:14 smithi107 ceph-mon[32659]: pgmap v6492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:14 smithi132 ceph-mon[37580]: pgmap v6492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:14 smithi112 ceph-mon[37501]: pgmap v6492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:01:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:01:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:01:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:16.241+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:01:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:16.241+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:16.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:16.241+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:01:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:16 smithi132 ceph-mon[37580]: pgmap v6493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11412.004117 secs 2024-07-26T16:01:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:16 smithi112 ceph-mon[37501]: pgmap v6493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11412.004117 secs 2024-07-26T16:01:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:16 smithi107 ceph-mon[32659]: pgmap v6493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11412.004117 secs 2024-07-26T16:01:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:17 smithi107 ceph-mon[32659]: pgmap v6494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:17 smithi132 ceph-mon[37580]: pgmap v6494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:17.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:17 smithi112 ceph-mon[37501]: pgmap v6494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:19.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:19.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:19 smithi107 ceph-mon[32659]: pgmap v6495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:01:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:01:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:19 smithi107 ceph-mon[32659]: pgmap v6496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:01:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:19 smithi132 ceph-mon[37580]: pgmap v6495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:01:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:01:20.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:19 smithi132 ceph-mon[37580]: pgmap v6496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:01:20.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:19 smithi112 ceph-mon[37501]: pgmap v6495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:01:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:01:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:19 smithi112 ceph-mon[37501]: pgmap v6496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:01:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:01:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11417.004219 secs 2024-07-26T16:01:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11417.004219 secs 2024-07-26T16:01:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11417.004219 secs 2024-07-26T16:01:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:21 smithi107 ceph-mon[32659]: pgmap v6497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:01:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:21 smithi132 ceph-mon[37580]: pgmap v6497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:01:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:21 smithi112 ceph-mon[37501]: pgmap v6497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:01:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:24 smithi112 ceph-mon[37501]: pgmap v6498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:01:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:24 smithi107 ceph-mon[32659]: pgmap v6498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:01:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:24 smithi132 ceph-mon[37580]: pgmap v6498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:01:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:26 smithi107 ceph-mon[32659]: pgmap v6499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:01:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11422.004335 secs 2024-07-26T16:01:26.363 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:26 smithi132 ceph-mon[37580]: pgmap v6499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:01:26.363 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11422.004335 secs 2024-07-26T16:01:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:26 smithi112 ceph-mon[37501]: pgmap v6499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:01:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11422.004335 secs 2024-07-26T16:01:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:28 smithi107 ceph-mon[32659]: pgmap v6500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:01:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:28 smithi112 ceph-mon[37501]: pgmap v6500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:01:28.556 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:28 smithi132 ceph-mon[37580]: pgmap v6500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:01:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:30 smithi112 ceph-mon[37501]: pgmap v6501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:01:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:30 smithi107 ceph-mon[32659]: pgmap v6501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:01:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:30 smithi132 ceph-mon[37580]: pgmap v6501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:01:31.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:31.247+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:01:31.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:31.248+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:01:31.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:31.248+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:31.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11427.004471 secs 2024-07-26T16:01:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11427.004471 secs 2024-07-26T16:01:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11427.004471 secs 2024-07-26T16:01:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:32 smithi112 ceph-mon[37501]: pgmap v6502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:32 smithi107 ceph-mon[32659]: pgmap v6502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:32 smithi132 ceph-mon[37580]: pgmap v6502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:34 smithi112 ceph-mon[37501]: pgmap v6503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:34 smithi107 ceph-mon[32659]: pgmap v6503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:34 smithi132 ceph-mon[37580]: pgmap v6503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:36 smithi112 ceph-mon[37501]: pgmap v6504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11432.004606 secs 2024-07-26T16:01:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:36 smithi107 ceph-mon[32659]: pgmap v6504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11432.004606 secs 2024-07-26T16:01:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:36 smithi132 ceph-mon[37580]: pgmap v6504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11432.004606 secs 2024-07-26T16:01:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:38 smithi132 ceph-mon[37580]: pgmap v6505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:38 smithi112 ceph-mon[37501]: pgmap v6505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:38 smithi107 ceph-mon[32659]: pgmap v6505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:40 smithi112 ceph-mon[37501]: pgmap v6506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:40 smithi107 ceph-mon[32659]: pgmap v6506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:40 smithi132 ceph-mon[37580]: pgmap v6506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11437.004768 secs 2024-07-26T16:01:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11437.004768 secs 2024-07-26T16:01:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11437.004768 secs 2024-07-26T16:01:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:42 smithi112 ceph-mon[37501]: pgmap v6507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:42 smithi107 ceph-mon[32659]: pgmap v6507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:42 smithi132 ceph-mon[37580]: pgmap v6507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:44 smithi112 ceph-mon[37501]: pgmap v6508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:44 smithi107 ceph-mon[32659]: pgmap v6508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:44 smithi132 ceph-mon[37580]: pgmap v6508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:46.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:46.249+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:01:46.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:46.249+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:01:46.253 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:01:46.250+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:46 smithi112 ceph-mon[37501]: pgmap v6509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11442.004899 secs 2024-07-26T16:01:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:46 smithi107 ceph-mon[32659]: pgmap v6509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11442.004899 secs 2024-07-26T16:01:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:46 smithi132 ceph-mon[37580]: pgmap v6509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11442.004899 secs 2024-07-26T16:01:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:48 smithi112 ceph-mon[37501]: pgmap v6510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:48 smithi107 ceph-mon[32659]: pgmap v6510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:48 smithi132 ceph-mon[37580]: pgmap v6510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:50 smithi112 ceph-mon[37501]: pgmap v6511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:50 smithi107 ceph-mon[32659]: pgmap v6511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:50 smithi132 ceph-mon[37580]: pgmap v6511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11447.005024 secs 2024-07-26T16:01:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11447.005024 secs 2024-07-26T16:01:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11447.005024 secs 2024-07-26T16:01:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:52 smithi112 ceph-mon[37501]: pgmap v6512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:52 smithi107 ceph-mon[32659]: pgmap v6512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:52 smithi132 ceph-mon[37580]: pgmap v6512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:54 smithi112 ceph-mon[37501]: pgmap v6513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:54 smithi107 ceph-mon[32659]: pgmap v6513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:54 smithi132 ceph-mon[37580]: pgmap v6513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:56 smithi112 ceph-mon[37501]: pgmap v6514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11452.005146 secs 2024-07-26T16:01:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:56 smithi107 ceph-mon[32659]: pgmap v6514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11452.005146 secs 2024-07-26T16:01:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:56 smithi132 ceph-mon[37580]: pgmap v6514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11452.005146 secs 2024-07-26T16:01:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:01:58 smithi112 ceph-mon[37501]: pgmap v6515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:01:58 smithi107 ceph-mon[32659]: pgmap v6515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:01:58 smithi132 ceph-mon[37580]: pgmap v6515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:00 smithi112 ceph-mon[37501]: pgmap v6516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:00 smithi107 ceph-mon[32659]: pgmap v6516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:00 smithi132 ceph-mon[37580]: pgmap v6516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:01.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:01.263+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:02:01.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:01.263+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:02:01.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:01.263+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11457.005258 secs 2024-07-26T16:02:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11457.005258 secs 2024-07-26T16:02:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11457.005258 secs 2024-07-26T16:02:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:02 smithi107 ceph-mon[32659]: pgmap v6517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:02 smithi132 ceph-mon[37580]: pgmap v6517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:02 smithi112 ceph-mon[37501]: pgmap v6517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:04 smithi107 ceph-mon[32659]: pgmap v6518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:04 smithi132 ceph-mon[37580]: pgmap v6518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:04 smithi112 ceph-mon[37501]: pgmap v6518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:06.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:06 smithi112 ceph-mon[37501]: pgmap v6519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:06.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11462.005374 secs 2024-07-26T16:02:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:06 smithi107 ceph-mon[32659]: pgmap v6519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11462.005374 secs 2024-07-26T16:02:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:06 smithi132 ceph-mon[37580]: pgmap v6519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11462.005374 secs 2024-07-26T16:02:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:08 smithi107 ceph-mon[32659]: pgmap v6520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:08 smithi132 ceph-mon[37580]: pgmap v6520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:08 smithi112 ceph-mon[37501]: pgmap v6520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:10 smithi107 ceph-mon[32659]: pgmap v6521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:10 smithi132 ceph-mon[37580]: pgmap v6521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:10 smithi112 ceph-mon[37501]: pgmap v6521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11467.005517 secs 2024-07-26T16:02:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11467.005517 secs 2024-07-26T16:02:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11467.005517 secs 2024-07-26T16:02:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:12 smithi107 ceph-mon[32659]: pgmap v6522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:12 smithi132 ceph-mon[37580]: pgmap v6522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:12 smithi112 ceph-mon[37501]: pgmap v6522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:14 smithi107 ceph-mon[32659]: pgmap v6523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:14 smithi132 ceph-mon[37580]: pgmap v6523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:14 smithi112 ceph-mon[37501]: pgmap v6523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:16.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:16.280+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:02:16.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:16.280+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:16.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:16.280+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:02:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:16 smithi107 ceph-mon[32659]: pgmap v6524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11472.005619 secs 2024-07-26T16:02:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:16 smithi132 ceph-mon[37580]: pgmap v6524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11472.005619 secs 2024-07-26T16:02:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:16 smithi112 ceph-mon[37501]: pgmap v6524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11472.005619 secs 2024-07-26T16:02:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:18 smithi107 ceph-mon[32659]: pgmap v6525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:18 smithi132 ceph-mon[37580]: pgmap v6525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:18 smithi112 ceph-mon[37501]: pgmap v6525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:20 smithi112 ceph-mon[37501]: pgmap v6526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:02:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:20 smithi107 ceph-mon[32659]: pgmap v6526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:02:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:20 smithi132 ceph-mon[37580]: pgmap v6526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:02:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:02:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:02:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:21 smithi112 ceph-mon[37501]: pgmap v6527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:02:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:02:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11477.005709 secs 2024-07-26T16:02:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:02:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:02:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:21 smithi107 ceph-mon[32659]: pgmap v6527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:02:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:02:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11477.005709 secs 2024-07-26T16:02:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:02:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:02:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:21 smithi132 ceph-mon[37580]: pgmap v6527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:02:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:02:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11477.005709 secs 2024-07-26T16:02:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:23 smithi112 ceph-mon[37501]: pgmap v6528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T16:02:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:23 smithi107 ceph-mon[32659]: pgmap v6528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T16:02:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:23 smithi132 ceph-mon[37580]: pgmap v6528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T16:02:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:25 smithi112 ceph-mon[37501]: pgmap v6529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T16:02:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:25 smithi107 ceph-mon[32659]: pgmap v6529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T16:02:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:25 smithi132 ceph-mon[37580]: pgmap v6529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4 KiB/s rd, 8 op/s 2024-07-26T16:02:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11482.005775 secs 2024-07-26T16:02:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:26 smithi112 ceph-mon[37501]: pgmap v6530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:02:26.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11482.005775 secs 2024-07-26T16:02:26.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:26 smithi132 ceph-mon[37580]: pgmap v6530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:02:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11482.005775 secs 2024-07-26T16:02:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:26 smithi107 ceph-mon[32659]: pgmap v6530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:02:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:29 smithi112 ceph-mon[37501]: pgmap v6531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:02:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:29 smithi107 ceph-mon[32659]: pgmap v6531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:02:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:29 smithi132 ceph-mon[37580]: pgmap v6531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:02:31.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:31.282+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:02:31.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:31.282+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:02:31.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:31.282+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:31 smithi107 ceph-mon[32659]: pgmap v6532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:02:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11487.005867 secs 2024-07-26T16:02:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:31 smithi112 ceph-mon[37501]: pgmap v6532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:02:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11487.005867 secs 2024-07-26T16:02:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:31 smithi132 ceph-mon[37580]: pgmap v6532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:02:31.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11487.005867 secs 2024-07-26T16:02:32.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:32 smithi112 ceph-mon[37501]: pgmap v6533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:32 smithi107 ceph-mon[32659]: pgmap v6533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:32 smithi132 ceph-mon[37580]: pgmap v6533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:35 smithi112 ceph-mon[37501]: pgmap v6534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:35 smithi107 ceph-mon[32659]: pgmap v6534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:35 smithi132 ceph-mon[37580]: pgmap v6534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:36.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11492.005963 secs 2024-07-26T16:02:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11492.005963 secs 2024-07-26T16:02:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11492.005963 secs 2024-07-26T16:02:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:37 smithi112 ceph-mon[37501]: pgmap v6535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:37 smithi107 ceph-mon[32659]: pgmap v6535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:37 smithi132 ceph-mon[37580]: pgmap v6535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:38 smithi112 ceph-mon[37501]: pgmap v6536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:38 smithi107 ceph-mon[32659]: pgmap v6536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:38 smithi132 ceph-mon[37580]: pgmap v6536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:41 smithi112 ceph-mon[37501]: pgmap v6537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11497.006044 secs 2024-07-26T16:02:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:41 smithi107 ceph-mon[32659]: pgmap v6537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11497.006044 secs 2024-07-26T16:02:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:41 smithi132 ceph-mon[37580]: pgmap v6537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11497.006044 secs 2024-07-26T16:02:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:42 smithi112 ceph-mon[37501]: pgmap v6538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:42.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:42 smithi107 ceph-mon[32659]: pgmap v6538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:42 smithi132 ceph-mon[37580]: pgmap v6538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:44.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:44 smithi112 ceph-mon[37501]: pgmap v6539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:44 smithi107 ceph-mon[32659]: pgmap v6539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:44 smithi132 ceph-mon[37580]: pgmap v6539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11502.006159 secs 2024-07-26T16:02:46.288 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:46.286+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:02:46.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:46.286+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:02:46.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:02:46.286+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:46.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11502.006159 secs 2024-07-26T16:02:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11502.006159 secs 2024-07-26T16:02:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:46 smithi112 ceph-mon[37501]: pgmap v6540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:46 smithi107 ceph-mon[32659]: pgmap v6540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:46 smithi132 ceph-mon[37580]: pgmap v6540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:48 smithi112 ceph-mon[37501]: pgmap v6541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:48 smithi107 ceph-mon[32659]: pgmap v6541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:48 smithi132 ceph-mon[37580]: pgmap v6541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:50 smithi112 ceph-mon[37501]: pgmap v6542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:50 smithi107 ceph-mon[32659]: pgmap v6542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:50 smithi132 ceph-mon[37580]: pgmap v6542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11507.006278 secs 2024-07-26T16:02:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11507.006278 secs 2024-07-26T16:02:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11507.006278 secs 2024-07-26T16:02:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:52 smithi112 ceph-mon[37501]: pgmap v6543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:52.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:52 smithi107 ceph-mon[32659]: pgmap v6543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:52 smithi132 ceph-mon[37580]: pgmap v6543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:54.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:54 smithi112 ceph-mon[37501]: pgmap v6544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:02:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:54 smithi107 ceph-mon[32659]: pgmap v6544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:02:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:54 smithi132 ceph-mon[37580]: pgmap v6544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:02:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11512.006386 secs 2024-07-26T16:02:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11512.006386 secs 2024-07-26T16:02:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11512.006386 secs 2024-07-26T16:02:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:56 smithi112 ceph-mon[37501]: pgmap v6545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:02:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:56 smithi107 ceph-mon[32659]: pgmap v6545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:02:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:56 smithi132 ceph-mon[37580]: pgmap v6545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:02:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:02:58 smithi112 ceph-mon[37501]: pgmap v6546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:02:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:02:58 smithi107 ceph-mon[32659]: pgmap v6546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:02:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:02:58 smithi132 ceph-mon[37580]: pgmap v6546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:03:00.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:00 smithi112 ceph-mon[37501]: pgmap v6547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:03:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:00 smithi107 ceph-mon[32659]: pgmap v6547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:03:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:00 smithi132 ceph-mon[37580]: pgmap v6547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:03:01.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:01.300+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:03:01.303 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:01.300+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:03:01.304 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:01.300+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:01.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11517.006483 secs 2024-07-26T16:03:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11517.006483 secs 2024-07-26T16:03:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11517.006483 secs 2024-07-26T16:03:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:02 smithi112 ceph-mon[37501]: pgmap v6548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:03:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:02 smithi107 ceph-mon[32659]: pgmap v6548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:03:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:02 smithi132 ceph-mon[37580]: pgmap v6548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:03:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:04 smithi112 ceph-mon[37501]: pgmap v6549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:03:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:04 smithi107 ceph-mon[32659]: pgmap v6549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:03:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:04 smithi132 ceph-mon[37580]: pgmap v6549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:03:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:05 smithi112 ceph-mon[37501]: 1 slow requests, 1 included below; oldest blocked for > 11522.006606 secs 2024-07-26T16:03:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:05 smithi112 ceph-mon[37501]: slow request 11522.006606 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T16:03:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:05 smithi107 ceph-mon[32659]: 1 slow requests, 1 included below; oldest blocked for > 11522.006606 secs 2024-07-26T16:03:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:05 smithi107 ceph-mon[32659]: slow request 11522.006606 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T16:03:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:05 smithi132 ceph-mon[37580]: 1 slow requests, 1 included below; oldest blocked for > 11522.006606 secs 2024-07-26T16:03:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:05 smithi132 ceph-mon[37580]: slow request 11522.006606 seconds old, received at 2024-07-26T12:51:03.333882+0000: client_request(client.15267:7393 rename #0x10000000299/builddefs #0x100000016d8/out 2024-07-26T12:51:03.332002+0000 caller_uid=1000, caller_gid=1316{6,36,1000,1316,}) currently acquired locks 2024-07-26T16:03:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:06 smithi112 ceph-mon[37501]: pgmap v6550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:06 smithi107 ceph-mon[32659]: pgmap v6550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:06 smithi132 ceph-mon[37580]: pgmap v6550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:08 smithi112 ceph-mon[37501]: pgmap v6551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:08.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:08 smithi107 ceph-mon[32659]: pgmap v6551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:08 smithi132 ceph-mon[37580]: pgmap v6551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:10 smithi112 ceph-mon[37501]: pgmap v6552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:10.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:10 smithi107 ceph-mon[32659]: pgmap v6552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:10 smithi132 ceph-mon[37580]: pgmap v6552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11527.006744 secs 2024-07-26T16:03:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11527.006744 secs 2024-07-26T16:03:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11527.006744 secs 2024-07-26T16:03:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:12 smithi112 ceph-mon[37501]: pgmap v6553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:12 smithi107 ceph-mon[32659]: pgmap v6553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:12 smithi132 ceph-mon[37580]: pgmap v6553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:14.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:14.617+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:03:14.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:14.617+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:03:14.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:14.617+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:03:14.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:14.617+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:03:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:14 smithi112 ceph-mon[37501]: pgmap v6554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:14 smithi107 ceph-mon[32659]: pgmap v6554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:14 smithi132 ceph-mon[37580]: pgmap v6554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11532.006845 secs 2024-07-26T16:03:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11532.006845 secs 2024-07-26T16:03:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11532.006845 secs 2024-07-26T16:03:16.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:16.317+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:03:16.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:16.317+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:03:16.319 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:16.317+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:16 smithi112 ceph-mon[37501]: pgmap v6555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:16 smithi107 ceph-mon[32659]: pgmap v6555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:16 smithi132 ceph-mon[37580]: pgmap v6555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:18 smithi112 ceph-mon[37501]: pgmap v6556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:18 smithi107 ceph-mon[32659]: pgmap v6556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:18 smithi132 ceph-mon[37580]: pgmap v6556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:20 smithi112 ceph-mon[37501]: pgmap v6557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:20 smithi107 ceph-mon[32659]: pgmap v6557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:20 smithi132 ceph-mon[37580]: pgmap v6557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11537.006986 secs 2024-07-26T16:03:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:03:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11537.006986 secs 2024-07-26T16:03:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:03:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11537.006986 secs 2024-07-26T16:03:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:03:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:03:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:03:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:22 smithi112 ceph-mon[37501]: pgmap v6558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:03:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:22 smithi112 ceph-mon[37501]: pgmap v6559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:03:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:03:22.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:03:22.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:03:22.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:22 smithi107 ceph-mon[32659]: pgmap v6558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:03:22.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:22 smithi107 ceph-mon[32659]: pgmap v6559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:03:22.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:03:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:03:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:03:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:22 smithi132 ceph-mon[37580]: pgmap v6558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:03:22.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:22 smithi132 ceph-mon[37580]: pgmap v6559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:03:22.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:03:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:24 smithi107 ceph-mon[32659]: pgmap v6560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:24 smithi132 ceph-mon[37580]: pgmap v6560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:24 smithi112 ceph-mon[37501]: pgmap v6560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11542.007127 secs 2024-07-26T16:03:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:26 smithi107 ceph-mon[32659]: pgmap v6561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11542.007127 secs 2024-07-26T16:03:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:26 smithi132 ceph-mon[37580]: pgmap v6561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11542.007127 secs 2024-07-26T16:03:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:26 smithi112 ceph-mon[37501]: pgmap v6561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:28 smithi107 ceph-mon[32659]: pgmap v6562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:03:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:28 smithi132 ceph-mon[37580]: pgmap v6562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:03:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:28 smithi112 ceph-mon[37501]: pgmap v6562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:03:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:30 smithi132 ceph-mon[37580]: pgmap v6563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:30.906 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:30 smithi107 ceph-mon[32659]: pgmap v6563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:30 smithi112 ceph-mon[37501]: pgmap v6563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:31.334 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:31.332+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:03:31.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:31.333+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:03:31.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:31.333+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11547.007213 secs 2024-07-26T16:03:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11547.007213 secs 2024-07-26T16:03:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11547.007213 secs 2024-07-26T16:03:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:32 smithi112 ceph-mon[37501]: pgmap v6564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:03:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:32 smithi107 ceph-mon[32659]: pgmap v6564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:03:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:32 smithi132 ceph-mon[37580]: pgmap v6564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:03:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:34 smithi112 ceph-mon[37501]: pgmap v6565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:34 smithi107 ceph-mon[32659]: pgmap v6565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:34 smithi132 ceph-mon[37580]: pgmap v6565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11552.007335 secs 2024-07-26T16:03:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:36 smithi112 ceph-mon[37501]: pgmap v6566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11552.007335 secs 2024-07-26T16:03:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:36 smithi107 ceph-mon[32659]: pgmap v6566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11552.007335 secs 2024-07-26T16:03:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:36 smithi132 ceph-mon[37580]: pgmap v6566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:38 smithi112 ceph-mon[37501]: pgmap v6567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:38 smithi107 ceph-mon[32659]: pgmap v6567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:38 smithi132 ceph-mon[37580]: pgmap v6567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:40 smithi112 ceph-mon[37501]: pgmap v6568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:03:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:40 smithi107 ceph-mon[32659]: pgmap v6568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:03:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:40 smithi132 ceph-mon[37580]: pgmap v6568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:03:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11557.007453 secs 2024-07-26T16:03:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11557.007453 secs 2024-07-26T16:03:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11557.007453 secs 2024-07-26T16:03:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:42 smithi112 ceph-mon[37501]: pgmap v6569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:03:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:42 smithi107 ceph-mon[32659]: pgmap v6569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:03:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:42 smithi132 ceph-mon[37580]: pgmap v6569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:03:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:44 smithi112 ceph-mon[37501]: pgmap v6570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:44 smithi107 ceph-mon[32659]: pgmap v6570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:44 smithi132 ceph-mon[37580]: pgmap v6570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:46.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:46.348+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:03:46.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:46.348+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:03:46.350 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:03:46.348+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11562.007594 secs 2024-07-26T16:03:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:46 smithi112 ceph-mon[37501]: pgmap v6571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11562.007594 secs 2024-07-26T16:03:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:46 smithi107 ceph-mon[32659]: pgmap v6571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11562.007594 secs 2024-07-26T16:03:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:46 smithi132 ceph-mon[37580]: pgmap v6571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:48 smithi112 ceph-mon[37501]: pgmap v6572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:48 smithi107 ceph-mon[32659]: pgmap v6572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:48 smithi132 ceph-mon[37580]: pgmap v6572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:50 smithi112 ceph-mon[37501]: pgmap v6573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:03:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:50 smithi107 ceph-mon[32659]: pgmap v6573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:03:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:50 smithi132 ceph-mon[37580]: pgmap v6573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:03:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11567.007732 secs 2024-07-26T16:03:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11567.007732 secs 2024-07-26T16:03:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11567.007732 secs 2024-07-26T16:03:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:52 smithi112 ceph-mon[37501]: pgmap v6574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:03:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:52 smithi107 ceph-mon[32659]: pgmap v6574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:03:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:52 smithi132 ceph-mon[37580]: pgmap v6574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:03:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:54 smithi112 ceph-mon[37501]: pgmap v6575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:54 smithi107 ceph-mon[32659]: pgmap v6575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:54 smithi132 ceph-mon[37580]: pgmap v6575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11572.007852 secs 2024-07-26T16:03:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:56 smithi112 ceph-mon[37501]: pgmap v6576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11572.007852 secs 2024-07-26T16:03:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:56 smithi107 ceph-mon[32659]: pgmap v6576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11572.007852 secs 2024-07-26T16:03:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:56 smithi132 ceph-mon[37580]: pgmap v6576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:03:58 smithi112 ceph-mon[37501]: pgmap v6577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:03:58 smithi107 ceph-mon[32659]: pgmap v6577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:03:58 smithi132 ceph-mon[37580]: pgmap v6577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:00 smithi112 ceph-mon[37501]: pgmap v6578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:04:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:00 smithi107 ceph-mon[32659]: pgmap v6578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:04:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:00 smithi132 ceph-mon[37580]: pgmap v6578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:04:01.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:01.363+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:04:01.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:01.363+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:04:01.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:01.363+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11577.007996 secs 2024-07-26T16:04:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11577.007996 secs 2024-07-26T16:04:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11577.007996 secs 2024-07-26T16:04:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:02 smithi112 ceph-mon[37501]: pgmap v6579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:02 smithi107 ceph-mon[32659]: pgmap v6579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:02 smithi132 ceph-mon[37580]: pgmap v6579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:04 smithi112 ceph-mon[37501]: pgmap v6580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:04 smithi107 ceph-mon[32659]: pgmap v6580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:04 smithi132 ceph-mon[37580]: pgmap v6580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11582.008138 secs 2024-07-26T16:04:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:06 smithi112 ceph-mon[37501]: pgmap v6581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11582.008138 secs 2024-07-26T16:04:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:06 smithi107 ceph-mon[32659]: pgmap v6581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11582.008138 secs 2024-07-26T16:04:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:06 smithi132 ceph-mon[37580]: pgmap v6581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:08 smithi112 ceph-mon[37501]: pgmap v6582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:08 smithi107 ceph-mon[32659]: pgmap v6582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:08 smithi132 ceph-mon[37580]: pgmap v6582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:10 smithi112 ceph-mon[37501]: pgmap v6583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:04:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:10 smithi107 ceph-mon[32659]: pgmap v6583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:04:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:10 smithi132 ceph-mon[37580]: pgmap v6583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:04:11.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:11.506+0000 7f37dea00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f37c00060f0 con 0x7f37e8105540 2024-07-26T16:04:11.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:11.507+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f37c00060f0 con 0x7f37e8105540 2024-07-26T16:04:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11587.008227 secs 2024-07-26T16:04:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11587.008227 secs 2024-07-26T16:04:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11587.008227 secs 2024-07-26T16:04:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:12 smithi107 ceph-mon[32659]: pgmap v6584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:12 smithi132 ceph-mon[37580]: pgmap v6584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:12 smithi112 ceph-mon[37501]: pgmap v6584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:14 smithi107 ceph-mon[32659]: pgmap v6585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:14 smithi132 ceph-mon[37580]: pgmap v6585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:14 smithi112 ceph-mon[37501]: pgmap v6585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:16.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:16.379+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:04:16.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:16.379+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:04:16.382 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:16.379+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:16.722 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11592.008308 secs 2024-07-26T16:04:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:16 smithi107 ceph-mon[32659]: pgmap v6586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11592.008308 secs 2024-07-26T16:04:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:16 smithi132 ceph-mon[37580]: pgmap v6586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11592.008308 secs 2024-07-26T16:04:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:16 smithi112 ceph-mon[37501]: pgmap v6586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:18 smithi107 ceph-mon[32659]: pgmap v6587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:18 smithi132 ceph-mon[37580]: pgmap v6587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:18 smithi112 ceph-mon[37501]: pgmap v6587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:20 smithi107 ceph-mon[32659]: pgmap v6588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:04:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:20 smithi132 ceph-mon[37580]: pgmap v6588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:04:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:20 smithi112 ceph-mon[37501]: pgmap v6588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:04:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11597.008434 secs 2024-07-26T16:04:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:04:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11597.008434 secs 2024-07-26T16:04:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:04:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11597.008434 secs 2024-07-26T16:04:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:04:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:22 smithi107 ceph-mon[32659]: pgmap v6589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:04:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:04:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:04:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:22 smithi132 ceph-mon[37580]: pgmap v6589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:04:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:04:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:04:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:22 smithi112 ceph-mon[37501]: pgmap v6589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:04:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:04:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:04:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:23 smithi107 ceph-mon[32659]: pgmap v6590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:04:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:23 smithi107 ceph-mon[32659]: pgmap v6591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:04:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:23 smithi132 ceph-mon[37580]: pgmap v6590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:04:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:23 smithi132 ceph-mon[37580]: pgmap v6591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:04:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:23 smithi112 ceph-mon[37501]: pgmap v6590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:04:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:23 smithi112 ceph-mon[37501]: pgmap v6591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:04:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:25 smithi107 ceph-mon[32659]: pgmap v6592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:04:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:25 smithi132 ceph-mon[37580]: pgmap v6592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:04:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:25 smithi112 ceph-mon[37501]: pgmap v6592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:04:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11602.008562 secs 2024-07-26T16:04:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11602.008562 secs 2024-07-26T16:04:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11602.008562 secs 2024-07-26T16:04:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:27 smithi107 ceph-mon[32659]: pgmap v6593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:27 smithi132 ceph-mon[37580]: pgmap v6593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:27 smithi112 ceph-mon[37501]: pgmap v6593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:29 smithi107 ceph-mon[32659]: pgmap v6594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:04:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:29 smithi132 ceph-mon[37580]: pgmap v6594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:04:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:29 smithi112 ceph-mon[37501]: pgmap v6594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:04:31.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:31.393+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:04:31.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:31.393+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:04:31.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:31.393+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11607.008697 secs 2024-07-26T16:04:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:31 smithi107 ceph-mon[32659]: pgmap v6595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:04:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11607.008697 secs 2024-07-26T16:04:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:31 smithi132 ceph-mon[37580]: pgmap v6595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:04:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11607.008697 secs 2024-07-26T16:04:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:31 smithi112 ceph-mon[37501]: pgmap v6595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:04:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:33 smithi107 ceph-mon[32659]: pgmap v6596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:04:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:33 smithi132 ceph-mon[37580]: pgmap v6596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:04:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:33 smithi112 ceph-mon[37501]: pgmap v6596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:04:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:35 smithi132 ceph-mon[37580]: pgmap v6597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:35 smithi112 ceph-mon[37501]: pgmap v6597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:35 smithi107 ceph-mon[32659]: pgmap v6597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11612.008815 secs 2024-07-26T16:04:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11612.008815 secs 2024-07-26T16:04:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11612.008815 secs 2024-07-26T16:04:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:37 smithi132 ceph-mon[37580]: pgmap v6598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:37 smithi112 ceph-mon[37501]: pgmap v6598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:37 smithi107 ceph-mon[32659]: pgmap v6598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:39 smithi112 ceph-mon[37501]: pgmap v6599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:39 smithi107 ceph-mon[32659]: pgmap v6599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:39 smithi132 ceph-mon[37580]: pgmap v6599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11617.008949 secs 2024-07-26T16:04:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:41 smithi112 ceph-mon[37501]: pgmap v6600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11617.008949 secs 2024-07-26T16:04:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:41 smithi107 ceph-mon[32659]: pgmap v6600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11617.008949 secs 2024-07-26T16:04:42.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:41 smithi132 ceph-mon[37580]: pgmap v6600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:43 smithi112 ceph-mon[37501]: pgmap v6601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:43 smithi107 ceph-mon[32659]: pgmap v6601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:43 smithi132 ceph-mon[37580]: pgmap v6601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:45 smithi112 ceph-mon[37501]: pgmap v6602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:45 smithi107 ceph-mon[32659]: pgmap v6602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:45 smithi132 ceph-mon[37580]: pgmap v6602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:46.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:46.409+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:04:46.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:46.409+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:04:46.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:04:46.409+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11622.009097 secs 2024-07-26T16:04:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11622.009097 secs 2024-07-26T16:04:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11622.009097 secs 2024-07-26T16:04:48.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:47 smithi112 ceph-mon[37501]: pgmap v6603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:48.218 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:47 smithi107 ceph-mon[32659]: pgmap v6603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:47 smithi132 ceph-mon[37580]: pgmap v6603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:49 smithi112 ceph-mon[37501]: pgmap v6604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:49 smithi107 ceph-mon[32659]: pgmap v6604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:49 smithi132 ceph-mon[37580]: pgmap v6604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11627.009216 secs 2024-07-26T16:04:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:51 smithi112 ceph-mon[37501]: pgmap v6605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11627.009216 secs 2024-07-26T16:04:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:51 smithi107 ceph-mon[32659]: pgmap v6605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11627.009216 secs 2024-07-26T16:04:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:51 smithi132 ceph-mon[37580]: pgmap v6605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:53 smithi112 ceph-mon[37501]: pgmap v6606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:53 smithi107 ceph-mon[32659]: pgmap v6606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:53 smithi132 ceph-mon[37580]: pgmap v6606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:04:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:55 smithi112 ceph-mon[37501]: pgmap v6607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:55 smithi107 ceph-mon[32659]: pgmap v6607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:55 smithi132 ceph-mon[37580]: pgmap v6607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11632.009336 secs 2024-07-26T16:04:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11632.009336 secs 2024-07-26T16:04:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11632.009336 secs 2024-07-26T16:04:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:57 smithi112 ceph-mon[37501]: pgmap v6608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:57 smithi107 ceph-mon[32659]: pgmap v6608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:04:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:57 smithi132 ceph-mon[37580]: pgmap v6608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:04:59 smithi112 ceph-mon[37501]: pgmap v6609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:04:59 smithi107 ceph-mon[32659]: pgmap v6609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:04:59 smithi132 ceph-mon[37580]: pgmap v6609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:01.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11637.009459 secs 2024-07-26T16:05:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11637.009459 secs 2024-07-26T16:05:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11637.009459 secs 2024-07-26T16:05:01.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:01.423+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:05:01.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:01.423+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:01.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:01.423+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:05:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:01 smithi112 ceph-mon[37501]: pgmap v6610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:01 smithi107 ceph-mon[32659]: pgmap v6610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:01 smithi132 ceph-mon[37580]: pgmap v6610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:03 smithi112 ceph-mon[37501]: pgmap v6611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:03 smithi107 ceph-mon[32659]: pgmap v6611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:03 smithi132 ceph-mon[37580]: pgmap v6611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:05 smithi112 ceph-mon[37501]: pgmap v6612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11642.009605 secs 2024-07-26T16:05:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:05 smithi107 ceph-mon[32659]: pgmap v6612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11642.009605 secs 2024-07-26T16:05:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:05 smithi132 ceph-mon[37580]: pgmap v6612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:06.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11642.009605 secs 2024-07-26T16:05:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:07 smithi112 ceph-mon[37501]: pgmap v6613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:07 smithi107 ceph-mon[32659]: pgmap v6613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:07 smithi132 ceph-mon[37580]: pgmap v6613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:09 smithi112 ceph-mon[37501]: pgmap v6614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:09 smithi107 ceph-mon[32659]: pgmap v6614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:09 smithi132 ceph-mon[37580]: pgmap v6614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11647.009704 secs 2024-07-26T16:05:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11647.009704 secs 2024-07-26T16:05:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11647.009704 secs 2024-07-26T16:05:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:11 smithi112 ceph-mon[37501]: pgmap v6615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:11 smithi107 ceph-mon[32659]: pgmap v6615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:11 smithi132 ceph-mon[37580]: pgmap v6615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:13 smithi107 ceph-mon[32659]: pgmap v6616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:13 smithi132 ceph-mon[37580]: pgmap v6616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:13 smithi112 ceph-mon[37501]: pgmap v6616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:15 smithi107 ceph-mon[32659]: pgmap v6617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11652.009811 secs 2024-07-26T16:05:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:15 smithi132 ceph-mon[37580]: pgmap v6617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11652.009811 secs 2024-07-26T16:05:16.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:16.439+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:05:16.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:16.439+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:16.442 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:16.439+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:05:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:15 smithi112 ceph-mon[37501]: pgmap v6617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11652.009811 secs 2024-07-26T16:05:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:18 smithi112 ceph-mon[37501]: pgmap v6618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:18 smithi107 ceph-mon[32659]: pgmap v6618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:18 smithi132 ceph-mon[37580]: pgmap v6618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:20 smithi107 ceph-mon[32659]: pgmap v6619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:20.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:20 smithi112 ceph-mon[37501]: pgmap v6619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:20 smithi132 ceph-mon[37580]: pgmap v6619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11657.009884 secs 2024-07-26T16:05:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11657.009884 secs 2024-07-26T16:05:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11657.009884 secs 2024-07-26T16:05:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:22 smithi112 ceph-mon[37501]: pgmap v6620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:22 smithi107 ceph-mon[32659]: pgmap v6620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:22 smithi132 ceph-mon[37580]: pgmap v6620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:05:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:05:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:05:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:24 smithi112 ceph-mon[37501]: pgmap v6621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:05:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:05:24.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:05:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:24 smithi107 ceph-mon[32659]: pgmap v6621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:24.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:05:24.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:05:24.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:05:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:24 smithi132 ceph-mon[37580]: pgmap v6621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:05:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:05:24.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:05:25.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:25 smithi112 ceph-mon[37501]: pgmap v6622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:05:25.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:25 smithi112 ceph-mon[37501]: pgmap v6623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:05:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:25 smithi107 ceph-mon[32659]: pgmap v6622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:05:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:25 smithi107 ceph-mon[32659]: pgmap v6623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:05:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:25 smithi132 ceph-mon[37580]: pgmap v6622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:05:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:25 smithi132 ceph-mon[37580]: pgmap v6623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:05:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11662.010007 secs 2024-07-26T16:05:26.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11662.010007 secs 2024-07-26T16:05:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11662.010007 secs 2024-07-26T16:05:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:27 smithi112 ceph-mon[37501]: pgmap v6624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:05:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:27 smithi107 ceph-mon[32659]: pgmap v6624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:05:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:27 smithi132 ceph-mon[37580]: pgmap v6624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:05:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:29 smithi112 ceph-mon[37501]: pgmap v6625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:05:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:29 smithi107 ceph-mon[32659]: pgmap v6625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:05:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:29 smithi132 ceph-mon[37580]: pgmap v6625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:05:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:31 smithi112 ceph-mon[37501]: pgmap v6626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11667.010125 secs 2024-07-26T16:05:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:31.455+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:05:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:31.455+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:31.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:31.455+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:05:31.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:31 smithi107 ceph-mon[32659]: pgmap v6626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:31.458 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11667.010125 secs 2024-07-26T16:05:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:31 smithi132 ceph-mon[37580]: pgmap v6626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11667.010125 secs 2024-07-26T16:05:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:33 smithi112 ceph-mon[37501]: pgmap v6627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:33 smithi107 ceph-mon[32659]: pgmap v6627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:33 smithi132 ceph-mon[37580]: pgmap v6627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:35 smithi112 ceph-mon[37501]: pgmap v6628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:05:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:35 smithi107 ceph-mon[32659]: pgmap v6628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:05:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:35 smithi132 ceph-mon[37580]: pgmap v6628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:05:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11672.010248 secs 2024-07-26T16:05:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11672.010248 secs 2024-07-26T16:05:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11672.010248 secs 2024-07-26T16:05:37.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:37 smithi112 ceph-mon[37501]: pgmap v6629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:37 smithi107 ceph-mon[32659]: pgmap v6629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:37 smithi132 ceph-mon[37580]: pgmap v6629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:39 smithi112 ceph-mon[37501]: pgmap v6630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:05:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:39 smithi107 ceph-mon[32659]: pgmap v6630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:05:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:39 smithi132 ceph-mon[37580]: pgmap v6630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:05:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:41 smithi112 ceph-mon[37501]: pgmap v6631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11677.010370 secs 2024-07-26T16:05:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:41 smithi107 ceph-mon[32659]: pgmap v6631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11677.010370 secs 2024-07-26T16:05:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:41 smithi132 ceph-mon[37580]: pgmap v6631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11677.010370 secs 2024-07-26T16:05:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:43 smithi112 ceph-mon[37501]: pgmap v6632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:43 smithi107 ceph-mon[32659]: pgmap v6632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:43 smithi132 ceph-mon[37580]: pgmap v6632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:45 smithi112 ceph-mon[37501]: pgmap v6633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:05:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:45 smithi107 ceph-mon[32659]: pgmap v6633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:05:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:45 smithi132 ceph-mon[37580]: pgmap v6633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:05:46.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:46.471+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:05:46.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:46.471+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:05:46.473 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:05:46.471+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:46.474 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11682.010495 secs 2024-07-26T16:05:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11682.010495 secs 2024-07-26T16:05:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11682.010495 secs 2024-07-26T16:05:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:47 smithi107 ceph-mon[32659]: pgmap v6634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:05:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:47 smithi132 ceph-mon[37580]: pgmap v6634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:05:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:47 smithi112 ceph-mon[37501]: pgmap v6634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:05:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:49 smithi107 ceph-mon[32659]: pgmap v6635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:05:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:49 smithi132 ceph-mon[37580]: pgmap v6635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:05:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:49 smithi112 ceph-mon[37501]: pgmap v6635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:05:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:51 smithi107 ceph-mon[32659]: pgmap v6636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11687.010583 secs 2024-07-26T16:05:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:51 smithi132 ceph-mon[37580]: pgmap v6636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11687.010583 secs 2024-07-26T16:05:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:51 smithi112 ceph-mon[37501]: pgmap v6636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11687.010583 secs 2024-07-26T16:05:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:53 smithi107 ceph-mon[32659]: pgmap v6637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:53 smithi132 ceph-mon[37580]: pgmap v6637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:53 smithi112 ceph-mon[37501]: pgmap v6637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:05:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:55 smithi107 ceph-mon[32659]: pgmap v6638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:55 smithi132 ceph-mon[37580]: pgmap v6638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:55 smithi112 ceph-mon[37501]: pgmap v6638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11692.010707 secs 2024-07-26T16:05:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11692.010707 secs 2024-07-26T16:05:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11692.010707 secs 2024-07-26T16:05:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:57 smithi107 ceph-mon[32659]: pgmap v6639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:57 smithi132 ceph-mon[37580]: pgmap v6639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:57 smithi112 ceph-mon[37501]: pgmap v6639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:05:59 smithi107 ceph-mon[32659]: pgmap v6640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:05:59 smithi132 ceph-mon[37580]: pgmap v6640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:05:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:05:59 smithi112 ceph-mon[37501]: pgmap v6640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:01.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:01.485+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:06:01.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:01.485+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:01.488 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:01.485+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:06:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:01 smithi107 ceph-mon[32659]: pgmap v6641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11697.010832 secs 2024-07-26T16:06:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:01 smithi132 ceph-mon[37580]: pgmap v6641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11697.010832 secs 2024-07-26T16:06:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:01 smithi112 ceph-mon[37501]: pgmap v6641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11697.010832 secs 2024-07-26T16:06:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:03 smithi107 ceph-mon[32659]: pgmap v6642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:03 smithi132 ceph-mon[37580]: pgmap v6642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:03 smithi112 ceph-mon[37501]: pgmap v6642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:05 smithi107 ceph-mon[32659]: pgmap v6643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:05 smithi132 ceph-mon[37580]: pgmap v6643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:05.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:05 smithi112 ceph-mon[37501]: pgmap v6643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11702.010955 secs 2024-07-26T16:06:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11702.010955 secs 2024-07-26T16:06:06.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11702.010955 secs 2024-07-26T16:06:07.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:07 smithi107 ceph-mon[32659]: pgmap v6644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:07 smithi132 ceph-mon[37580]: pgmap v6644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:07 smithi112 ceph-mon[37501]: pgmap v6644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:09 smithi132 ceph-mon[37580]: pgmap v6645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:09 smithi112 ceph-mon[37501]: pgmap v6645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:09 smithi107 ceph-mon[32659]: pgmap v6645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:11 smithi132 ceph-mon[37580]: pgmap v6646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:11.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11707.011095 secs 2024-07-26T16:06:11.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:11.673+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 msgr2=0x7ff0ec0c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T16:06:11.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:11.673+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 msgr2=0x7ff0ec0c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T16:06:11.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:11.673+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f806f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:06:11.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:11.673+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f806f040 comp rx=0 tx=0).stop 2024-07-26T16:06:11.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:11.673+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 msgr2=0x7ff0ec0c40a0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T16:06:11.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:11.673+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 msgr2=0x7ff0ec0c40a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:06:11.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:11.673+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1630 0x7ff0ec0c40a0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:06:11.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:11.673+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:06:11.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:11 smithi107 ceph-mon[32659]: pgmap v6646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:11.678 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11707.011095 secs 2024-07-26T16:06:11.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:11.674+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:06:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:11.676+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f806f040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:06:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:11 smithi112 ceph-mon[37501]: pgmap v6646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11707.011095 secs 2024-07-26T16:06:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:13 smithi132 ceph-mon[37580]: pgmap v6647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:13 smithi112 ceph-mon[37501]: pgmap v6647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:13 smithi107 ceph-mon[32659]: pgmap v6647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:14.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:14.648+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:06:14.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:14.648+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:06:14.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:14.648+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:06:14.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:14.648+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:06:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:15 smithi132 ceph-mon[37580]: pgmap v6648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:15 smithi112 ceph-mon[37501]: pgmap v6648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:15 smithi107 ceph-mon[32659]: pgmap v6648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:16.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:16.491+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:06:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:16.491+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:06:16.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:16.491+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11712.011208 secs 2024-07-26T16:06:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11712.011208 secs 2024-07-26T16:06:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11712.011208 secs 2024-07-26T16:06:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:17 smithi112 ceph-mon[37501]: pgmap v6649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:17 smithi107 ceph-mon[32659]: pgmap v6649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:17 smithi132 ceph-mon[37580]: pgmap v6649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:19 smithi112 ceph-mon[37501]: pgmap v6650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:19 smithi107 ceph-mon[32659]: pgmap v6650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:19 smithi132 ceph-mon[37580]: pgmap v6650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:21 smithi112 ceph-mon[37501]: pgmap v6651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11717.011326 secs 2024-07-26T16:06:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:21 smithi107 ceph-mon[32659]: pgmap v6651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11717.011326 secs 2024-07-26T16:06:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:21 smithi132 ceph-mon[37580]: pgmap v6651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:21.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11717.011326 secs 2024-07-26T16:06:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:23 smithi112 ceph-mon[37501]: pgmap v6652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:23 smithi107 ceph-mon[32659]: pgmap v6652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:23 smithi132 ceph-mon[37580]: pgmap v6652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:06:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:06:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:06:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:25 smithi112 ceph-mon[37501]: pgmap v6653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:06:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:06:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:06:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:25 smithi107 ceph-mon[32659]: pgmap v6653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:25.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:06:25.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:06:25.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:06:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:25 smithi132 ceph-mon[37580]: pgmap v6653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:06:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:06:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:06:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:26 smithi112 ceph-mon[37501]: pgmap v6654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:06:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:26 smithi112 ceph-mon[37501]: pgmap v6655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:06:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11722.011438 secs 2024-07-26T16:06:26.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:26 smithi132 ceph-mon[37580]: pgmap v6654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:06:26.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:26 smithi132 ceph-mon[37580]: pgmap v6655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:06:26.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11722.011438 secs 2024-07-26T16:06:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:26 smithi107 ceph-mon[32659]: pgmap v6654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:06:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:26 smithi107 ceph-mon[32659]: pgmap v6655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:06:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11722.011438 secs 2024-07-26T16:06:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:28 smithi112 ceph-mon[37501]: pgmap v6656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:28 smithi107 ceph-mon[32659]: pgmap v6656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:28 smithi132 ceph-mon[37580]: pgmap v6656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:30 smithi112 ceph-mon[37501]: pgmap v6657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:06:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:30 smithi107 ceph-mon[32659]: pgmap v6657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:06:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:30 smithi132 ceph-mon[37580]: pgmap v6657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:06:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:31.506+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:06:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:31.506+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:06:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:31.506+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:31.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11727.011576 secs 2024-07-26T16:06:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11727.011576 secs 2024-07-26T16:06:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11727.011576 secs 2024-07-26T16:06:32.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:32 smithi112 ceph-mon[37501]: pgmap v6658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:06:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:32 smithi107 ceph-mon[32659]: pgmap v6658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:06:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:32 smithi132 ceph-mon[37580]: pgmap v6658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:06:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:33 smithi112 ceph-mon[37501]: pgmap v6659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:06:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:33 smithi107 ceph-mon[32659]: pgmap v6659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:06:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:33 smithi132 ceph-mon[37580]: pgmap v6659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:06:36.404 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:36 smithi112 ceph-mon[37501]: pgmap v6660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:06:36.404 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11732.011709 secs 2024-07-26T16:06:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:36 smithi107 ceph-mon[32659]: pgmap v6660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:06:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11732.011709 secs 2024-07-26T16:06:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:36 smithi132 ceph-mon[37580]: pgmap v6660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:06:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11732.011709 secs 2024-07-26T16:06:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:38 smithi112 ceph-mon[37501]: pgmap v6661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:38.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:38 smithi107 ceph-mon[32659]: pgmap v6661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:38 smithi132 ceph-mon[37580]: pgmap v6661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:40 smithi112 ceph-mon[37501]: pgmap v6662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:40 smithi107 ceph-mon[32659]: pgmap v6662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:40 smithi132 ceph-mon[37580]: pgmap v6662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11737.011866 secs 2024-07-26T16:06:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11737.011866 secs 2024-07-26T16:06:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11737.011866 secs 2024-07-26T16:06:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:42 smithi112 ceph-mon[37501]: pgmap v6663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:42 smithi107 ceph-mon[32659]: pgmap v6663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:42 smithi132 ceph-mon[37580]: pgmap v6663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:44 smithi112 ceph-mon[37501]: pgmap v6664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:44 smithi107 ceph-mon[32659]: pgmap v6664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:44 smithi132 ceph-mon[37580]: pgmap v6664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:46 smithi112 ceph-mon[37501]: pgmap v6665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11742.012002 secs 2024-07-26T16:06:46.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:46.522+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:06:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:46.522+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:46.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:06:46.522+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:06:46.524 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:46 smithi107 ceph-mon[32659]: pgmap v6665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:46.524 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11742.012002 secs 2024-07-26T16:06:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:46 smithi132 ceph-mon[37580]: pgmap v6665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11742.012002 secs 2024-07-26T16:06:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:48 smithi112 ceph-mon[37501]: pgmap v6666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:48 smithi107 ceph-mon[32659]: pgmap v6666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:48 smithi132 ceph-mon[37580]: pgmap v6666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:50 smithi112 ceph-mon[37501]: pgmap v6667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:50 smithi107 ceph-mon[32659]: pgmap v6667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:50 smithi132 ceph-mon[37580]: pgmap v6667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11747.012120 secs 2024-07-26T16:06:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11747.012120 secs 2024-07-26T16:06:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11747.012120 secs 2024-07-26T16:06:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:52 smithi112 ceph-mon[37501]: pgmap v6668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:52 smithi107 ceph-mon[32659]: pgmap v6668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:52 smithi132 ceph-mon[37580]: pgmap v6668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:54 smithi112 ceph-mon[37501]: pgmap v6669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:54 smithi107 ceph-mon[32659]: pgmap v6669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:54 smithi132 ceph-mon[37580]: pgmap v6669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:06:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:56 smithi112 ceph-mon[37501]: pgmap v6670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11752.012222 secs 2024-07-26T16:06:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:56 smithi107 ceph-mon[32659]: pgmap v6670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11752.012222 secs 2024-07-26T16:06:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:56 smithi132 ceph-mon[37580]: pgmap v6670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11752.012222 secs 2024-07-26T16:06:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:06:58 smithi112 ceph-mon[37501]: pgmap v6671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:06:58 smithi107 ceph-mon[32659]: pgmap v6671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:06:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:06:58 smithi132 ceph-mon[37580]: pgmap v6671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:00 smithi112 ceph-mon[37501]: pgmap v6672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:00 smithi107 ceph-mon[32659]: pgmap v6672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:00 smithi132 ceph-mon[37580]: pgmap v6672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:01.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11757.012352 secs 2024-07-26T16:07:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11757.012352 secs 2024-07-26T16:07:01.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:01.531+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:07:01.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:01.531+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:01.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:01.531+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:07:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11757.012352 secs 2024-07-26T16:07:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:02 smithi112 ceph-mon[37501]: pgmap v6673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:02 smithi107 ceph-mon[32659]: pgmap v6673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:02 smithi132 ceph-mon[37580]: pgmap v6673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:04 smithi112 ceph-mon[37501]: pgmap v6674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:04 smithi107 ceph-mon[32659]: pgmap v6674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:04 smithi132 ceph-mon[37580]: pgmap v6674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:06 smithi107 ceph-mon[32659]: pgmap v6675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11762.012472 secs 2024-07-26T16:07:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:06 smithi132 ceph-mon[37580]: pgmap v6675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11762.012472 secs 2024-07-26T16:07:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:06 smithi112 ceph-mon[37501]: pgmap v6675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11762.012472 secs 2024-07-26T16:07:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:08 smithi107 ceph-mon[32659]: pgmap v6676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:08 smithi132 ceph-mon[37580]: pgmap v6676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:08 smithi112 ceph-mon[37501]: pgmap v6676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:10 smithi107 ceph-mon[32659]: pgmap v6677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:10 smithi132 ceph-mon[37580]: pgmap v6677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:10 smithi112 ceph-mon[37501]: pgmap v6677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:11.417+0000 7f37e6000640 1 -- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f37cc0b3dc0 msgr2=0x7f37cc0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900028056) for more than 900000000 us, fault. 2024-07-26T16:07:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:11.417+0000 7f37e6000640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f37cc0b3dc0 0x7f37cc0b62d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f37e8103730 tx=0x7f37d4005f90 comp rx=0 tx=0).stop 2024-07-26T16:07:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:11.417+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f37cc0b3dc0 msgr2=0x7f37cc0b62d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:07:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:11.417+0000 7f37eca00640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f37cc0b3dc0 0x7f37cc0b62d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:07:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:11.417+0000 7f37eca00640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f37cc0c0630 0x7f37cc0c2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:07:11.419 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:11.417+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f37d8015e80 con 0x7f37cc0c0630 2024-07-26T16:07:11.420 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:11.417+0000 7f37e6000640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f37cc0c0630 0x7f37cc0c2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:07:11.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:11.419+0000 7f37e6000640 1 --2- 172.21.15.107:0/4046735598 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7f37cc0c0630 0x7f37cc0c2a50 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f37d4005fd0 tx=0x7f37d4017000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:07:11.540 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:11.539+0000 7f37dea00640 1 -- 172.21.15.107:0/4046735598 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f37c0006330 con 0x7f37e8105540 2024-07-26T16:07:11.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:11.539+0000 7f37eca00640 1 -- 172.21.15.107:0/4046735598 <== mon.0 v2:172.21.15.107:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f37c0006330 con 0x7f37e8105540 2024-07-26T16:07:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11767.012600 secs 2024-07-26T16:07:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11767.012600 secs 2024-07-26T16:07:11.631 INFO:teuthology.orchestra.run.smithi107.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-07-26T16:07:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-07-26T16:07:11.668 ERROR:teuthology.run_tasks:Manager failed: full_sequential_finally Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/full_sequential_finally.py", line 45, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/'" 2024-07-26T16:07:11.675 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-07-26T16:07:11.685 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-07-26T16:07:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11767.012600 secs 2024-07-26T16:07:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:12 smithi107 ceph-mon[32659]: pgmap v6678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:12 smithi132 ceph-mon[37580]: pgmap v6678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:12 smithi112 ceph-mon[37501]: pgmap v6678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:13 smithi107 ceph-mon[32659]: from='client.15879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T16:07:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:13 smithi132 ceph-mon[37580]: from='client.15879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T16:07:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:13 smithi112 ceph-mon[37501]: from='client.15879 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T16:07:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:14 smithi107 ceph-mon[32659]: pgmap v6679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:14 smithi132 ceph-mon[37580]: pgmap v6679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:14 smithi112 ceph-mon[37501]: pgmap v6679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:16.540+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:07:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:16.540+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:16.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:16.540+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:07:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:16 smithi107 ceph-mon[32659]: pgmap v6680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11772.012719 secs 2024-07-26T16:07:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:16 smithi132 ceph-mon[37580]: pgmap v6680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11772.012719 secs 2024-07-26T16:07:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:16 smithi112 ceph-mon[37501]: pgmap v6680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11772.012719 secs 2024-07-26T16:07:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:18 smithi107 ceph-mon[32659]: pgmap v6681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:18 smithi132 ceph-mon[37580]: pgmap v6681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:18 smithi112 ceph-mon[37501]: pgmap v6681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:20 smithi107 ceph-mon[32659]: pgmap v6682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:20 smithi132 ceph-mon[37580]: pgmap v6682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:20 smithi112 ceph-mon[37501]: pgmap v6682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11777.012862 secs 2024-07-26T16:07:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11777.012862 secs 2024-07-26T16:07:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11777.012862 secs 2024-07-26T16:07:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:22 smithi107 ceph-mon[32659]: pgmap v6683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:22 smithi132 ceph-mon[37580]: pgmap v6683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:22 smithi112 ceph-mon[37501]: pgmap v6683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:24 smithi107 ceph-mon[32659]: pgmap v6684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:24 smithi132 ceph-mon[37580]: pgmap v6684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:24 smithi112 ceph-mon[37501]: pgmap v6684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:07:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:07:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:07:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: pgmap v6685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11782.012935 secs 2024-07-26T16:07:26.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:07:26.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:07:26.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:07:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: pgmap v6685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11782.012935 secs 2024-07-26T16:07:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:07:26.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:07:26.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:07:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: pgmap v6685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11782.012935 secs 2024-07-26T16:07:26.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:07:26.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:07:26.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:07:26.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:07:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:27 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:07:27.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:27 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:07:27.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:27 smithi132 ceph-mon[37580]: pgmap v6686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:07:27.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:27 smithi132 ceph-mon[37580]: pgmap v6687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:07:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:27 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:07:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:27 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:07:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:27 smithi112 ceph-mon[37501]: pgmap v6686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:07:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:27 smithi112 ceph-mon[37501]: pgmap v6687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:07:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:27 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:07:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:27 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:07:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:27 smithi107 ceph-mon[32659]: pgmap v6686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:07:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:27 smithi107 ceph-mon[32659]: pgmap v6687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:07:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:29 smithi112 ceph-mon[37501]: pgmap v6688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:29 smithi107 ceph-mon[32659]: pgmap v6688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:29 smithi132 ceph-mon[37580]: pgmap v6688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:31.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:31.555+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:07:31.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:31.555+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:07:31.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:31.555+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:31.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:31 smithi107 ceph-mon[32659]: pgmap v6689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:31.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11787.013114 secs 2024-07-26T16:07:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:31 smithi112 ceph-mon[37501]: pgmap v6689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11787.013114 secs 2024-07-26T16:07:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:31 smithi132 ceph-mon[37580]: pgmap v6689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11787.013114 secs 2024-07-26T16:07:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:33 smithi112 ceph-mon[37501]: pgmap v6690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:33 smithi107 ceph-mon[32659]: pgmap v6690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:33 smithi132 ceph-mon[37580]: pgmap v6690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:35 smithi112 ceph-mon[37501]: pgmap v6691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:35 smithi107 ceph-mon[32659]: pgmap v6691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:35 smithi132 ceph-mon[37580]: pgmap v6691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:36.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11792.013239 secs 2024-07-26T16:07:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11792.013239 secs 2024-07-26T16:07:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11792.013239 secs 2024-07-26T16:07:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:37 smithi112 ceph-mon[37501]: pgmap v6692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:07:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:37 smithi107 ceph-mon[32659]: pgmap v6692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:07:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:37 smithi132 ceph-mon[37580]: pgmap v6692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:07:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:39 smithi107 ceph-mon[32659]: pgmap v6693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:39 smithi112 ceph-mon[37501]: pgmap v6693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:39 smithi132 ceph-mon[37580]: pgmap v6693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:41 smithi112 ceph-mon[37501]: pgmap v6694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11797.013362 secs 2024-07-26T16:07:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:41 smithi107 ceph-mon[32659]: pgmap v6694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11797.013362 secs 2024-07-26T16:07:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:41 smithi132 ceph-mon[37580]: pgmap v6694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11797.013362 secs 2024-07-26T16:07:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:43 smithi112 ceph-mon[37501]: pgmap v6695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:43 smithi107 ceph-mon[32659]: pgmap v6695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:43 smithi132 ceph-mon[37580]: pgmap v6695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:45 smithi112 ceph-mon[37501]: pgmap v6696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:45 smithi107 ceph-mon[32659]: pgmap v6696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:45 smithi132 ceph-mon[37580]: pgmap v6696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:46.571+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:07:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:46.571+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:07:46.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:07:46.571+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11802.013485 secs 2024-07-26T16:07:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11802.013485 secs 2024-07-26T16:07:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11802.013485 secs 2024-07-26T16:07:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:47 smithi112 ceph-mon[37501]: pgmap v6697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:47 smithi107 ceph-mon[32659]: pgmap v6697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:47 smithi132 ceph-mon[37580]: pgmap v6697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:49 smithi112 ceph-mon[37501]: pgmap v6698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:49 smithi107 ceph-mon[32659]: pgmap v6698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:49 smithi132 ceph-mon[37580]: pgmap v6698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:51 smithi112 ceph-mon[37501]: pgmap v6699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11807.013632 secs 2024-07-26T16:07:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:51 smithi107 ceph-mon[32659]: pgmap v6699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11807.013632 secs 2024-07-26T16:07:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:51 smithi132 ceph-mon[37580]: pgmap v6699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11807.013632 secs 2024-07-26T16:07:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:53 smithi112 ceph-mon[37501]: pgmap v6700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:53 smithi107 ceph-mon[32659]: pgmap v6700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:53 smithi132 ceph-mon[37580]: pgmap v6700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:54 smithi107 ceph-mon[32659]: pgmap v6701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:54 smithi132 ceph-mon[37580]: pgmap v6701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:54 smithi112 ceph-mon[37501]: pgmap v6701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11812.013756 secs 2024-07-26T16:07:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11812.013756 secs 2024-07-26T16:07:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11812.013756 secs 2024-07-26T16:07:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:56 smithi107 ceph-mon[32659]: pgmap v6702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:56 smithi132 ceph-mon[37580]: pgmap v6702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:56 smithi112 ceph-mon[37501]: pgmap v6702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:07:59 smithi112 ceph-mon[37501]: pgmap v6703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:07:59 smithi107 ceph-mon[32659]: pgmap v6703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:07:59 smithi132 ceph-mon[37580]: pgmap v6703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:01.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:01.572+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:08:01.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:01.572+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:01.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:01.572+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:08:01.576 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:01 smithi107 ceph-mon[32659]: pgmap v6704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:01.576 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11817.013904 secs 2024-07-26T16:08:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:01 smithi112 ceph-mon[37501]: pgmap v6704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11817.013904 secs 2024-07-26T16:08:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:01 smithi132 ceph-mon[37580]: pgmap v6704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11817.013904 secs 2024-07-26T16:08:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:03 smithi112 ceph-mon[37501]: pgmap v6705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:08:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:03 smithi107 ceph-mon[32659]: pgmap v6705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:08:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:03 smithi132 ceph-mon[37580]: pgmap v6705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:08:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:05 smithi112 ceph-mon[37501]: pgmap v6706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:05 smithi107 ceph-mon[32659]: pgmap v6706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:05 smithi132 ceph-mon[37580]: pgmap v6706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:06.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11822.014039 secs 2024-07-26T16:08:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11822.014039 secs 2024-07-26T16:08:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11822.014039 secs 2024-07-26T16:08:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:07 smithi112 ceph-mon[37501]: pgmap v6707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:08:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:07 smithi107 ceph-mon[32659]: pgmap v6707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:08:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:07 smithi132 ceph-mon[37580]: pgmap v6707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:08:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:09 smithi112 ceph-mon[37501]: pgmap v6708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:08:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:09 smithi107 ceph-mon[32659]: pgmap v6708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:08:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:09 smithi132 ceph-mon[37580]: pgmap v6708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:08:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:11 smithi112 ceph-mon[37501]: pgmap v6709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:08:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11827.014164 secs 2024-07-26T16:08:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:11 smithi107 ceph-mon[32659]: pgmap v6709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:08:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11827.014164 secs 2024-07-26T16:08:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:11 smithi132 ceph-mon[37580]: pgmap v6709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:08:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11827.014164 secs 2024-07-26T16:08:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:13 smithi112 ceph-mon[37501]: pgmap v6710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:08:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:13 smithi107 ceph-mon[32659]: pgmap v6710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:08:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:13 smithi132 ceph-mon[37580]: pgmap v6710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:08:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:15 smithi112 ceph-mon[37501]: pgmap v6711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:15 smithi107 ceph-mon[32659]: pgmap v6711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:15 smithi132 ceph-mon[37580]: pgmap v6711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:16.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:16.582+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:08:16.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:16.582+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:08:16.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:16.582+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11832.014305 secs 2024-07-26T16:08:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11832.014305 secs 2024-07-26T16:08:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11832.014305 secs 2024-07-26T16:08:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:17 smithi112 ceph-mon[37501]: pgmap v6712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:17 smithi107 ceph-mon[32659]: pgmap v6712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:17 smithi132 ceph-mon[37580]: pgmap v6712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:19 smithi112 ceph-mon[37501]: pgmap v6713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:19 smithi107 ceph-mon[32659]: pgmap v6713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:19 smithi132 ceph-mon[37580]: pgmap v6713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:21 smithi112 ceph-mon[37501]: pgmap v6714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11837.014434 secs 2024-07-26T16:08:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:21 smithi107 ceph-mon[32659]: pgmap v6714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11837.014434 secs 2024-07-26T16:08:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:21 smithi132 ceph-mon[37580]: pgmap v6714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11837.014434 secs 2024-07-26T16:08:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:23 smithi112 ceph-mon[37501]: pgmap v6715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:08:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:23 smithi107 ceph-mon[32659]: pgmap v6715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:08:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:23 smithi132 ceph-mon[37580]: pgmap v6715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:08:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:25 smithi112 ceph-mon[37501]: pgmap v6716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:08:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:25 smithi107 ceph-mon[32659]: pgmap v6716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:08:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:25 smithi132 ceph-mon[37580]: pgmap v6716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:08:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11842.014561 secs 2024-07-26T16:08:26.710 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11842.014561 secs 2024-07-26T16:08:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11842.014561 secs 2024-07-26T16:08:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: pgmap v6717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:08:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.704 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: pgmap v6717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:08:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: pgmap v6717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:08:27.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:27.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:28 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:08:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:28 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:08:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:28 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:08:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:28 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:08:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:08:28.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:08:28.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:28 smithi112 ceph-mon[37501]: pgmap v6718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:08:28.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:28 smithi112 ceph-mon[37501]: pgmap v6719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:08:28.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:08:28.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:28 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:08:28.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:28 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:08:28.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:28 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:08:28.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:28 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:08:28.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:08:28.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:08:28.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:28 smithi107 ceph-mon[32659]: pgmap v6718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:08:28.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:28 smithi107 ceph-mon[32659]: pgmap v6719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:08:28.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:08:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:28 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:08:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:28 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:08:28.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:28 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:08:28.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:28 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:08:28.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:08:28.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:08:28.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:28 smithi132 ceph-mon[37580]: pgmap v6718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:08:28.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:28 smithi132 ceph-mon[37580]: pgmap v6719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:08:28.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:08:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:30 smithi107 ceph-mon[32659]: pgmap v6720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:30 smithi132 ceph-mon[37580]: pgmap v6720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:30 smithi112 ceph-mon[37501]: pgmap v6720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:31.600 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:31.598+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:08:31.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:31.598+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:08:31.601 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:31.599+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11847.014662 secs 2024-07-26T16:08:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11847.014662 secs 2024-07-26T16:08:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11847.014662 secs 2024-07-26T16:08:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:32 smithi107 ceph-mon[32659]: pgmap v6721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:08:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:32 smithi132 ceph-mon[37580]: pgmap v6721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:08:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:32 smithi112 ceph-mon[37501]: pgmap v6721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:08:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:34 smithi107 ceph-mon[32659]: pgmap v6722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:08:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:34 smithi132 ceph-mon[37580]: pgmap v6722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:08:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:34 smithi112 ceph-mon[37501]: pgmap v6722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:08:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11852.014810 secs 2024-07-26T16:08:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:36 smithi107 ceph-mon[32659]: pgmap v6723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11852.014810 secs 2024-07-26T16:08:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:36 smithi132 ceph-mon[37580]: pgmap v6723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11852.014810 secs 2024-07-26T16:08:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:36 smithi112 ceph-mon[37501]: pgmap v6723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:38.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:38 smithi107 ceph-mon[32659]: pgmap v6724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:08:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:38 smithi132 ceph-mon[37580]: pgmap v6724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:08:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:38 smithi112 ceph-mon[37501]: pgmap v6724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:08:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:40 smithi107 ceph-mon[32659]: pgmap v6725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:08:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:40 smithi132 ceph-mon[37580]: pgmap v6725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:08:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:40 smithi112 ceph-mon[37501]: pgmap v6725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:08:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11857.014939 secs 2024-07-26T16:08:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11857.014939 secs 2024-07-26T16:08:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11857.014939 secs 2024-07-26T16:08:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:42 smithi132 ceph-mon[37580]: pgmap v6726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:42 smithi112 ceph-mon[37501]: pgmap v6726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:42 smithi107 ceph-mon[32659]: pgmap v6726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:44 smithi112 ceph-mon[37501]: pgmap v6727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:44 smithi107 ceph-mon[32659]: pgmap v6727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:44 smithi132 ceph-mon[37580]: pgmap v6727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:46.614+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:08:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:46.614+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:46.616 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:08:46.614+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:08:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11862.015090 secs 2024-07-26T16:08:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:46 smithi112 ceph-mon[37501]: pgmap v6728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11862.015090 secs 2024-07-26T16:08:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:46 smithi107 ceph-mon[32659]: pgmap v6728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11862.015090 secs 2024-07-26T16:08:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:46 smithi132 ceph-mon[37580]: pgmap v6728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:48 smithi112 ceph-mon[37501]: pgmap v6729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:08:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:48 smithi107 ceph-mon[32659]: pgmap v6729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:08:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:48 smithi132 ceph-mon[37580]: pgmap v6729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:08:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:50 smithi112 ceph-mon[37501]: pgmap v6730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:08:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:50 smithi107 ceph-mon[32659]: pgmap v6730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:08:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:50 smithi132 ceph-mon[37580]: pgmap v6730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:08:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11867.015222 secs 2024-07-26T16:08:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11867.015222 secs 2024-07-26T16:08:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11867.015222 secs 2024-07-26T16:08:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:52 smithi112 ceph-mon[37501]: pgmap v6731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:52 smithi107 ceph-mon[32659]: pgmap v6731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:52 smithi132 ceph-mon[37580]: pgmap v6731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:08:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:54 smithi112 ceph-mon[37501]: pgmap v6732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:08:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:54 smithi107 ceph-mon[32659]: pgmap v6732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:08:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:54 smithi132 ceph-mon[37580]: pgmap v6732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:08:56.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11872.015351 secs 2024-07-26T16:08:56.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:56 smithi132 ceph-mon[37580]: pgmap v6733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:08:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11872.015351 secs 2024-07-26T16:08:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:56 smithi112 ceph-mon[37501]: pgmap v6733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:08:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11872.015351 secs 2024-07-26T16:08:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:56 smithi107 ceph-mon[32659]: pgmap v6733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:08:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:08:58 smithi112 ceph-mon[37501]: pgmap v6734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:08:59.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:08:58 smithi107 ceph-mon[32659]: pgmap v6734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:08:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:08:58 smithi132 ceph-mon[37580]: pgmap v6734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:09:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:00 smithi112 ceph-mon[37501]: pgmap v6735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:00 smithi107 ceph-mon[32659]: pgmap v6735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:00 smithi132 ceph-mon[37580]: pgmap v6735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:01.628+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:01.628+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:09:01.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:01.628+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:01.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11877.015439 secs 2024-07-26T16:09:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11877.015439 secs 2024-07-26T16:09:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11877.015439 secs 2024-07-26T16:09:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:02 smithi112 ceph-mon[37501]: pgmap v6736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:09:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:02 smithi107 ceph-mon[32659]: pgmap v6736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:09:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:02 smithi132 ceph-mon[37580]: pgmap v6736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:09:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:04 smithi112 ceph-mon[37501]: pgmap v6737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:09:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:04 smithi107 ceph-mon[32659]: pgmap v6737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:09:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:04 smithi132 ceph-mon[37580]: pgmap v6737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:09:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11882.015570 secs 2024-07-26T16:09:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:06 smithi112 ceph-mon[37501]: pgmap v6738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11882.015570 secs 2024-07-26T16:09:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:06 smithi107 ceph-mon[32659]: pgmap v6738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11882.015570 secs 2024-07-26T16:09:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:06 smithi132 ceph-mon[37580]: pgmap v6738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:08 smithi112 ceph-mon[37501]: pgmap v6739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:08 smithi107 ceph-mon[32659]: pgmap v6739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:08 smithi132 ceph-mon[37580]: pgmap v6739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:10 smithi112 ceph-mon[37501]: pgmap v6740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:10 smithi107 ceph-mon[32659]: pgmap v6740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:10 smithi132 ceph-mon[37580]: pgmap v6740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11887.015722 secs 2024-07-26T16:09:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11887.015722 secs 2024-07-26T16:09:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11887.015722 secs 2024-07-26T16:09:12.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:12 smithi107 ceph-mon[32659]: pgmap v6741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:09:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:12 smithi112 ceph-mon[37501]: pgmap v6741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:09:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:12 smithi132 ceph-mon[37580]: pgmap v6741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:09:13.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:13.680+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:09:13.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:13.680+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:09:13.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:13.680+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:09:13.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:13.680+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:09:14.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:14.680+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:09:14.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:14.680+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:09:14.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:14.680+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:09:14.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:14.681+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:09:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:14 smithi112 ceph-mon[37501]: pgmap v6742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:14 smithi107 ceph-mon[32659]: pgmap v6742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:14 smithi132 ceph-mon[37580]: pgmap v6742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:16.646 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:16.644+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:09:16.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:16.644+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:16.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:16.644+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:09:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11892.015866 secs 2024-07-26T16:09:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:16 smithi112 ceph-mon[37501]: pgmap v6743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11892.015866 secs 2024-07-26T16:09:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:16 smithi107 ceph-mon[32659]: pgmap v6743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11892.015866 secs 2024-07-26T16:09:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:16 smithi132 ceph-mon[37580]: pgmap v6743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:19 smithi112 ceph-mon[37501]: pgmap v6744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:19 smithi107 ceph-mon[32659]: pgmap v6744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:19 smithi132 ceph-mon[37580]: pgmap v6744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:21 smithi112 ceph-mon[37501]: pgmap v6745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11897.016017 secs 2024-07-26T16:09:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:21 smithi107 ceph-mon[32659]: pgmap v6745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11897.016017 secs 2024-07-26T16:09:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:21 smithi132 ceph-mon[37580]: pgmap v6745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11897.016017 secs 2024-07-26T16:09:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:23 smithi112 ceph-mon[37501]: pgmap v6746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:09:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:23 smithi107 ceph-mon[32659]: pgmap v6746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:09:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:23 smithi132 ceph-mon[37580]: pgmap v6746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:09:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:25 smithi107 ceph-mon[32659]: pgmap v6747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:25 smithi132 ceph-mon[37580]: pgmap v6747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:25 smithi112 ceph-mon[37501]: pgmap v6747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:26.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11902.016108 secs 2024-07-26T16:09:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11902.016108 secs 2024-07-26T16:09:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11902.016108 secs 2024-07-26T16:09:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:27 smithi107 ceph-mon[32659]: pgmap v6748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:27 smithi132 ceph-mon[37580]: pgmap v6748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:27 smithi112 ceph-mon[37501]: pgmap v6748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:09:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:09:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:09:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:29 smithi107 ceph-mon[32659]: pgmap v6749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:29.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:09:29.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:09:29.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:09:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:29 smithi132 ceph-mon[37580]: pgmap v6749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:09:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:09:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:09:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:29 smithi112 ceph-mon[37501]: pgmap v6749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:09:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:09:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:09:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:30 smithi107 ceph-mon[32659]: pgmap v6750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:09:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:30 smithi107 ceph-mon[32659]: pgmap v6751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:09:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:30 smithi132 ceph-mon[37580]: pgmap v6750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:09:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:30 smithi132 ceph-mon[37580]: pgmap v6751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:09:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:30 smithi112 ceph-mon[37501]: pgmap v6750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:09:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:30 smithi112 ceph-mon[37501]: pgmap v6751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:09:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11907.016229 secs 2024-07-26T16:09:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11907.016229 secs 2024-07-26T16:09:31.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:31.650+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:09:31.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:31.650+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:09:31.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:31.650+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11907.016229 secs 2024-07-26T16:09:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:32 smithi107 ceph-mon[32659]: pgmap v6752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:32 smithi132 ceph-mon[37580]: pgmap v6752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:32.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:32 smithi112 ceph-mon[37501]: pgmap v6752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:34 smithi107 ceph-mon[32659]: pgmap v6753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:09:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:34 smithi132 ceph-mon[37580]: pgmap v6753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:09:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:34 smithi112 ceph-mon[37501]: pgmap v6753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:09:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:36 smithi107 ceph-mon[32659]: pgmap v6754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11912.016355 secs 2024-07-26T16:09:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:36 smithi132 ceph-mon[37580]: pgmap v6754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11912.016355 secs 2024-07-26T16:09:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:36 smithi112 ceph-mon[37501]: pgmap v6754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11912.016355 secs 2024-07-26T16:09:38.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:38 smithi107 ceph-mon[32659]: pgmap v6755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:38 smithi132 ceph-mon[37580]: pgmap v6755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:38 smithi112 ceph-mon[37501]: pgmap v6755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:40 smithi107 ceph-mon[32659]: pgmap v6756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:09:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:40 smithi132 ceph-mon[37580]: pgmap v6756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:09:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:40 smithi112 ceph-mon[37501]: pgmap v6756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:09:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11917.016480 secs 2024-07-26T16:09:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11917.016480 secs 2024-07-26T16:09:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11917.016480 secs 2024-07-26T16:09:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:42 smithi132 ceph-mon[37580]: pgmap v6757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:42 smithi112 ceph-mon[37501]: pgmap v6757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:42 smithi107 ceph-mon[32659]: pgmap v6757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:44 smithi107 ceph-mon[32659]: pgmap v6758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:44 smithi132 ceph-mon[37580]: pgmap v6758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:44.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:44 smithi112 ceph-mon[37501]: pgmap v6758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:46 smithi132 ceph-mon[37580]: pgmap v6759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:46.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11922.016604 secs 2024-07-26T16:09:46.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:46.664+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:09:46.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:46.664+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:46.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:09:46.664+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:09:46.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:46 smithi107 ceph-mon[32659]: pgmap v6759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:46.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11922.016604 secs 2024-07-26T16:09:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:46 smithi112 ceph-mon[37501]: pgmap v6759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11922.016604 secs 2024-07-26T16:09:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:48 smithi112 ceph-mon[37501]: pgmap v6760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:09:48.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:48 smithi107 ceph-mon[32659]: pgmap v6760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:09:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:48 smithi132 ceph-mon[37580]: pgmap v6760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:09:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:50 smithi112 ceph-mon[37501]: pgmap v6761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:50 smithi107 ceph-mon[32659]: pgmap v6761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:50 smithi132 ceph-mon[37580]: pgmap v6761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11927.016757 secs 2024-07-26T16:09:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11927.016757 secs 2024-07-26T16:09:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11927.016757 secs 2024-07-26T16:09:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:52 smithi112 ceph-mon[37501]: pgmap v6762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:52 smithi107 ceph-mon[32659]: pgmap v6762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:52 smithi132 ceph-mon[37580]: pgmap v6762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:54.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:54 smithi112 ceph-mon[37501]: pgmap v6763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:54 smithi107 ceph-mon[32659]: pgmap v6763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:54 smithi132 ceph-mon[37580]: pgmap v6763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:09:56.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:56 smithi132 ceph-mon[37580]: pgmap v6764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:56.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11932.016895 secs 2024-07-26T16:09:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:56 smithi112 ceph-mon[37501]: pgmap v6764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11932.016895 secs 2024-07-26T16:09:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:56 smithi107 ceph-mon[32659]: pgmap v6764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:09:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11932.016895 secs 2024-07-26T16:09:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:09:58 smithi112 ceph-mon[37501]: pgmap v6765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:09:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:09:58 smithi107 ceph-mon[32659]: pgmap v6765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:09:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:09:58 smithi132 ceph-mon[37580]: pgmap v6765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:00.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:00 smithi112 ceph-mon[37501]: pgmap v6766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:00.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:10:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:00 smithi107 ceph-mon[32659]: pgmap v6766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:00.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:10:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:00 smithi132 ceph-mon[37580]: pgmap v6766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:10:01.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:01.674+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:10:01.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:01.674+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:01.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:01.674+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:10:01.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11937.016965 secs 2024-07-26T16:10:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11937.016965 secs 2024-07-26T16:10:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11937.016965 secs 2024-07-26T16:10:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:02 smithi112 ceph-mon[37501]: pgmap v6767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:02 smithi107 ceph-mon[32659]: pgmap v6767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:02 smithi132 ceph-mon[37580]: pgmap v6767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:04 smithi112 ceph-mon[37501]: pgmap v6768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:04 smithi107 ceph-mon[32659]: pgmap v6768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:04 smithi132 ceph-mon[37580]: pgmap v6768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:06 smithi107 ceph-mon[32659]: pgmap v6769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11942.017085 secs 2024-07-26T16:10:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:06 smithi112 ceph-mon[37501]: pgmap v6769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11942.017085 secs 2024-07-26T16:10:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:06 smithi132 ceph-mon[37580]: pgmap v6769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:06.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11942.017085 secs 2024-07-26T16:10:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:08 smithi112 ceph-mon[37501]: pgmap v6770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:08 smithi107 ceph-mon[32659]: pgmap v6770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:08 smithi132 ceph-mon[37580]: pgmap v6770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:10 smithi112 ceph-mon[37501]: pgmap v6771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:10 smithi107 ceph-mon[32659]: pgmap v6771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:10 smithi132 ceph-mon[37580]: pgmap v6771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11947.017211 secs 2024-07-26T16:10:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11947.017211 secs 2024-07-26T16:10:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11947.017211 secs 2024-07-26T16:10:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:12 smithi112 ceph-mon[37501]: pgmap v6772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:12 smithi107 ceph-mon[32659]: pgmap v6772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:12 smithi132 ceph-mon[37580]: pgmap v6772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:14 smithi112 ceph-mon[37501]: pgmap v6773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:14 smithi107 ceph-mon[32659]: pgmap v6773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:14 smithi132 ceph-mon[37580]: pgmap v6773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:16.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:16.689+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:10:16.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:16.689+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:16.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:16.689+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:10:16.693 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:16 smithi107 ceph-mon[32659]: pgmap v6774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:16.693 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11952.017274 secs 2024-07-26T16:10:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:16 smithi112 ceph-mon[37501]: pgmap v6774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11952.017274 secs 2024-07-26T16:10:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:16 smithi132 ceph-mon[37580]: pgmap v6774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:16.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11952.017274 secs 2024-07-26T16:10:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:18 smithi112 ceph-mon[37501]: pgmap v6775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:18 smithi107 ceph-mon[32659]: pgmap v6775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:18 smithi132 ceph-mon[37580]: pgmap v6775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:20 smithi112 ceph-mon[37501]: pgmap v6776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:20 smithi107 ceph-mon[32659]: pgmap v6776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:20 smithi132 ceph-mon[37580]: pgmap v6776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11957.017397 secs 2024-07-26T16:10:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11957.017397 secs 2024-07-26T16:10:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11957.017397 secs 2024-07-26T16:10:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:22 smithi112 ceph-mon[37501]: pgmap v6777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:22 smithi107 ceph-mon[32659]: pgmap v6777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:22 smithi132 ceph-mon[37580]: pgmap v6777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:23 smithi107 ceph-mon[32659]: pgmap v6778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:23 smithi132 ceph-mon[37580]: pgmap v6778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:23 smithi112 ceph-mon[37501]: pgmap v6778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:25 smithi107 ceph-mon[32659]: pgmap v6779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:25 smithi132 ceph-mon[37580]: pgmap v6779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:25 smithi112 ceph-mon[37501]: pgmap v6779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11962.017526 secs 2024-07-26T16:10:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11962.017526 secs 2024-07-26T16:10:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11962.017526 secs 2024-07-26T16:10:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:27 smithi107 ceph-mon[32659]: pgmap v6780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:27 smithi132 ceph-mon[37580]: pgmap v6780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:27 smithi112 ceph-mon[37501]: pgmap v6780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:10:30.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:29 smithi107 ceph-mon[32659]: pgmap v6781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:30.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:10:30.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:10:30.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:29 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:10:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:29 smithi132 ceph-mon[37580]: pgmap v6781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:10:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:10:30.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:29 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:10:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:29 smithi112 ceph-mon[37501]: pgmap v6781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:30.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:10:30.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:10:30.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:29 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:10:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:10:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:10:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:10:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:10:31.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:10:31.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:10:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:10:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:10:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:10:31.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:31.699+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:10:31.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:31.699+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:31.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:31.699+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:10:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11967.017668 secs 2024-07-26T16:10:32.061 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:31 smithi107 ceph-mon[32659]: pgmap v6782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:10:32.061 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:31 smithi107 ceph-mon[32659]: pgmap v6783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:10:32.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11967.017668 secs 2024-07-26T16:10:32.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:31 smithi132 ceph-mon[37580]: pgmap v6782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:10:32.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:31 smithi132 ceph-mon[37580]: pgmap v6783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:10:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11967.017668 secs 2024-07-26T16:10:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:31 smithi112 ceph-mon[37501]: pgmap v6782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:10:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:31 smithi112 ceph-mon[37501]: pgmap v6783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:10:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:33 smithi132 ceph-mon[37580]: pgmap v6784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:10:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:33 smithi112 ceph-mon[37501]: pgmap v6784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:10:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:33 smithi107 ceph-mon[32659]: pgmap v6784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:10:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:35 smithi132 ceph-mon[37580]: pgmap v6785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:10:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:35 smithi112 ceph-mon[37501]: pgmap v6785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:10:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:35 smithi107 ceph-mon[32659]: pgmap v6785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:10:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11972.017792 secs 2024-07-26T16:10:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11972.017792 secs 2024-07-26T16:10:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11972.017792 secs 2024-07-26T16:10:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:37 smithi112 ceph-mon[37501]: pgmap v6786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:10:38.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:37 smithi107 ceph-mon[32659]: pgmap v6786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:10:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:37 smithi132 ceph-mon[37580]: pgmap v6786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:10:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:39 smithi112 ceph-mon[37501]: pgmap v6787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:10:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:39 smithi107 ceph-mon[32659]: pgmap v6787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:10:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:39 smithi132 ceph-mon[37580]: pgmap v6787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:10:42.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11977.017927 secs 2024-07-26T16:10:42.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:41 smithi112 ceph-mon[37501]: pgmap v6788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:10:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11977.017927 secs 2024-07-26T16:10:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:41 smithi107 ceph-mon[32659]: pgmap v6788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:10:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11977.017927 secs 2024-07-26T16:10:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:41 smithi132 ceph-mon[37580]: pgmap v6788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:10:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:43 smithi112 ceph-mon[37501]: pgmap v6789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:43 smithi107 ceph-mon[32659]: pgmap v6789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:43 smithi132 ceph-mon[37580]: pgmap v6789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:46.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:45 smithi107 ceph-mon[32659]: pgmap v6790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:10:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:45 smithi112 ceph-mon[37501]: pgmap v6790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:10:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:45 smithi132 ceph-mon[37580]: pgmap v6790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:10:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:46.702+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:10:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:46.702+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:10:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:10:46.702+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11982.018046 secs 2024-07-26T16:10:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11982.018046 secs 2024-07-26T16:10:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11982.018046 secs 2024-07-26T16:10:48.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:47 smithi112 ceph-mon[37501]: pgmap v6791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:47 smithi107 ceph-mon[32659]: pgmap v6791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:47 smithi132 ceph-mon[37580]: pgmap v6791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:49 smithi112 ceph-mon[37501]: pgmap v6792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:49 smithi107 ceph-mon[32659]: pgmap v6792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:49 smithi132 ceph-mon[37580]: pgmap v6792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11987.018092 secs 2024-07-26T16:10:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:51 smithi112 ceph-mon[37501]: pgmap v6793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11987.018092 secs 2024-07-26T16:10:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:51 smithi107 ceph-mon[32659]: pgmap v6793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11987.018092 secs 2024-07-26T16:10:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:51 smithi132 ceph-mon[37580]: pgmap v6793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:53 smithi112 ceph-mon[37501]: pgmap v6794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:53 smithi107 ceph-mon[32659]: pgmap v6794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:53 smithi132 ceph-mon[37580]: pgmap v6794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:10:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:55 smithi112 ceph-mon[37501]: pgmap v6795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:10:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:55 smithi107 ceph-mon[32659]: pgmap v6795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:10:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:55 smithi132 ceph-mon[37580]: pgmap v6795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:10:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11992.018211 secs 2024-07-26T16:10:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11992.018211 secs 2024-07-26T16:10:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11992.018211 secs 2024-07-26T16:10:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:57 smithi112 ceph-mon[37501]: pgmap v6796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:57 smithi107 ceph-mon[32659]: pgmap v6796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:57 smithi132 ceph-mon[37580]: pgmap v6796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:11:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:10:59 smithi112 ceph-mon[37501]: pgmap v6797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:10:59 smithi107 ceph-mon[32659]: pgmap v6797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:10:59 smithi132 ceph-mon[37580]: pgmap v6797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:01.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:01.703+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:11:01.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:01.703+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:11:01.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:01.703+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:01.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 11997.018334 secs 2024-07-26T16:11:01.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:01 smithi107 ceph-mon[32659]: pgmap v6798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 11997.018334 secs 2024-07-26T16:11:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:01 smithi112 ceph-mon[37501]: pgmap v6798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 11997.018334 secs 2024-07-26T16:11:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:01 smithi132 ceph-mon[37580]: pgmap v6798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:03 smithi112 ceph-mon[37501]: pgmap v6799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:03 smithi107 ceph-mon[32659]: pgmap v6799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:03 smithi132 ceph-mon[37580]: pgmap v6799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:05 smithi112 ceph-mon[37501]: pgmap v6800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:05 smithi107 ceph-mon[32659]: pgmap v6800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:05 smithi132 ceph-mon[37580]: pgmap v6800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12002.018467 secs 2024-07-26T16:11:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12002.018467 secs 2024-07-26T16:11:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12002.018467 secs 2024-07-26T16:11:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:07 smithi112 ceph-mon[37501]: pgmap v6801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:11:08.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:07 smithi107 ceph-mon[32659]: pgmap v6801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:11:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:07 smithi132 ceph-mon[37580]: pgmap v6801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:11:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:09 smithi112 ceph-mon[37501]: pgmap v6802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:09 smithi107 ceph-mon[32659]: pgmap v6802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:09 smithi132 ceph-mon[37580]: pgmap v6802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12007.018553 secs 2024-07-26T16:11:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:11 smithi112 ceph-mon[37501]: pgmap v6803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12007.018553 secs 2024-07-26T16:11:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:11 smithi107 ceph-mon[32659]: pgmap v6803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:12.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12007.018553 secs 2024-07-26T16:11:12.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:11 smithi132 ceph-mon[37580]: pgmap v6803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:13 smithi112 ceph-mon[37501]: pgmap v6804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:13 smithi107 ceph-mon[32659]: pgmap v6804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:13 smithi132 ceph-mon[37580]: pgmap v6804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:15 smithi112 ceph-mon[37501]: pgmap v6805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12012.018681 secs 2024-07-26T16:11:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:15 smithi107 ceph-mon[32659]: pgmap v6805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12012.018681 secs 2024-07-26T16:11:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:15 smithi132 ceph-mon[37580]: pgmap v6805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12012.018681 secs 2024-07-26T16:11:16.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:16.713+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:11:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:16.713+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:11:16.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:16.713+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:18.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:17 smithi112 ceph-mon[37501]: pgmap v6806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:11:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:17 smithi107 ceph-mon[32659]: pgmap v6806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:11:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:17 smithi132 ceph-mon[37580]: pgmap v6806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:11:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:19 smithi107 ceph-mon[32659]: pgmap v6807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:11:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:19 smithi132 ceph-mon[37580]: pgmap v6807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:11:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:19 smithi112 ceph-mon[37501]: pgmap v6807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:11:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12017.018808 secs 2024-07-26T16:11:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12017.018808 secs 2024-07-26T16:11:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12017.018808 secs 2024-07-26T16:11:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:21 smithi107 ceph-mon[32659]: pgmap v6808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:11:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:21 smithi132 ceph-mon[37580]: pgmap v6808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:11:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:21 smithi112 ceph-mon[37501]: pgmap v6808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:11:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:23 smithi107 ceph-mon[32659]: pgmap v6809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:11:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:23 smithi132 ceph-mon[37580]: pgmap v6809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:11:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:23 smithi112 ceph-mon[37501]: pgmap v6809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:11:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:25 smithi107 ceph-mon[32659]: pgmap v6810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12022.018899 secs 2024-07-26T16:11:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:25 smithi132 ceph-mon[37580]: pgmap v6810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12022.018899 secs 2024-07-26T16:11:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:25 smithi112 ceph-mon[37501]: pgmap v6810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12022.018899 secs 2024-07-26T16:11:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:27 smithi107 ceph-mon[32659]: pgmap v6811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:11:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:27 smithi132 ceph-mon[37580]: pgmap v6811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:11:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:27 smithi112 ceph-mon[37501]: pgmap v6811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:11:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:29 smithi107 ceph-mon[32659]: pgmap v6812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:29 smithi132 ceph-mon[37580]: pgmap v6812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:29 smithi112 ceph-mon[37501]: pgmap v6812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12027.019054 secs 2024-07-26T16:11:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:11:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12027.019054 secs 2024-07-26T16:11:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:11:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12027.019054 secs 2024-07-26T16:11:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:11:31.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:31.729+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:11:31.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:31.729+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:11:31.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:31.729+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:32 smithi112 ceph-mon[37501]: pgmap v6813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:32 smithi107 ceph-mon[32659]: pgmap v6813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:32 smithi132 ceph-mon[37580]: pgmap v6813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:33 smithi107 ceph-mon[32659]: pgmap v6814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:33 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:33 smithi132 ceph-mon[37580]: pgmap v6814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:33 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:33 smithi112 ceph-mon[37501]: pgmap v6814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:33.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:33 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:35.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:35.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:36 smithi112 ceph-mon[37501]: pgmap v6815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12032.019179 secs 2024-07-26T16:11:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:11:36.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:11:36.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:36 smithi107 ceph-mon[32659]: pgmap v6815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12032.019179 secs 2024-07-26T16:11:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:11:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:11:36.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:36 smithi132 ceph-mon[37580]: pgmap v6815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:11:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12032.019179 secs 2024-07-26T16:11:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:11:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:11:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:11:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:37 smithi112 ceph-mon[37501]: pgmap v6816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:11:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:37 smithi112 ceph-mon[37501]: pgmap v6817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:11:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:37 smithi107 ceph-mon[32659]: pgmap v6816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:11:37.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:37 smithi107 ceph-mon[32659]: pgmap v6817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:11:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:37 smithi132 ceph-mon[37580]: pgmap v6816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:11:37.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:37 smithi132 ceph-mon[37580]: pgmap v6817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:11:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:39 smithi112 ceph-mon[37501]: pgmap v6818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:11:39.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:39 smithi107 ceph-mon[32659]: pgmap v6818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:11:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:39 smithi132 ceph-mon[37580]: pgmap v6818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:11:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:41 smithi112 ceph-mon[37501]: pgmap v6819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:11:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12037.019261 secs 2024-07-26T16:11:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:41 smithi107 ceph-mon[32659]: pgmap v6819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:11:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12037.019261 secs 2024-07-26T16:11:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:41 smithi132 ceph-mon[37580]: pgmap v6819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:11:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12037.019261 secs 2024-07-26T16:11:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:43 smithi112 ceph-mon[37501]: pgmap v6820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:11:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:43 smithi107 ceph-mon[32659]: pgmap v6820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:11:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:43 smithi132 ceph-mon[37580]: pgmap v6820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:11:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:45 smithi112 ceph-mon[37501]: pgmap v6821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:11:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:45 smithi107 ceph-mon[32659]: pgmap v6821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:11:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:45 smithi132 ceph-mon[37580]: pgmap v6821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:11:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12042.019378 secs 2024-07-26T16:11:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12042.019378 secs 2024-07-26T16:11:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12042.019378 secs 2024-07-26T16:11:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:46.745+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:11:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:46.745+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:11:46.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:11:46.745+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:47 smithi112 ceph-mon[37501]: pgmap v6822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:11:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:47 smithi107 ceph-mon[32659]: pgmap v6822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:11:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:47 smithi132 ceph-mon[37580]: pgmap v6822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:11:49.476 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:49 smithi107 ceph-mon[32659]: pgmap v6823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:49 smithi132 ceph-mon[37580]: pgmap v6823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:49 smithi112 ceph-mon[37501]: pgmap v6823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:51 smithi107 ceph-mon[32659]: pgmap v6824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:11:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12047.019496 secs 2024-07-26T16:11:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:51 smithi132 ceph-mon[37580]: pgmap v6824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:11:51.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12047.019496 secs 2024-07-26T16:11:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:51 smithi112 ceph-mon[37501]: pgmap v6824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:11:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12047.019496 secs 2024-07-26T16:11:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:53 smithi107 ceph-mon[32659]: pgmap v6825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:11:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:53 smithi132 ceph-mon[37580]: pgmap v6825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:11:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:53 smithi112 ceph-mon[37501]: pgmap v6825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:11:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:55 smithi107 ceph-mon[32659]: pgmap v6826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:55 smithi132 ceph-mon[37580]: pgmap v6826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:55 smithi112 ceph-mon[37501]: pgmap v6826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12052.019628 secs 2024-07-26T16:11:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12052.019628 secs 2024-07-26T16:11:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12052.019628 secs 2024-07-26T16:11:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:57 smithi107 ceph-mon[32659]: pgmap v6827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:57 smithi132 ceph-mon[37580]: pgmap v6827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:57 smithi112 ceph-mon[37501]: pgmap v6827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:11:59 smithi107 ceph-mon[32659]: pgmap v6828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:11:59 smithi132 ceph-mon[37580]: pgmap v6828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:11:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:11:59 smithi112 ceph-mon[37501]: pgmap v6828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:01 smithi107 ceph-mon[32659]: pgmap v6829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12057.019748 secs 2024-07-26T16:12:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:01 smithi132 ceph-mon[37580]: pgmap v6829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12057.019748 secs 2024-07-26T16:12:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:01 smithi112 ceph-mon[37501]: pgmap v6829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12057.019748 secs 2024-07-26T16:12:01.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:01.760+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:12:01.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:01.760+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:12:01.764 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:01.760+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:03 smithi107 ceph-mon[32659]: pgmap v6830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:03 smithi132 ceph-mon[37580]: pgmap v6830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:03 smithi112 ceph-mon[37501]: pgmap v6830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:05 smithi107 ceph-mon[32659]: pgmap v6831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:05 smithi132 ceph-mon[37580]: pgmap v6831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:05 smithi112 ceph-mon[37501]: pgmap v6831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12062.019866 secs 2024-07-26T16:12:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12062.019866 secs 2024-07-26T16:12:06.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12062.019866 secs 2024-07-26T16:12:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:07 smithi107 ceph-mon[32659]: pgmap v6832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:07 smithi132 ceph-mon[37580]: pgmap v6832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:07 smithi112 ceph-mon[37501]: pgmap v6832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:09 smithi107 ceph-mon[32659]: pgmap v6833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:09 smithi132 ceph-mon[37580]: pgmap v6833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:09 smithi112 ceph-mon[37501]: pgmap v6833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:11 smithi132 ceph-mon[37580]: pgmap v6834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12067.020021 secs 2024-07-26T16:12:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:11 smithi112 ceph-mon[37501]: pgmap v6834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12067.020021 secs 2024-07-26T16:12:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:11 smithi107 ceph-mon[32659]: pgmap v6834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12067.020021 secs 2024-07-26T16:12:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:13 smithi132 ceph-mon[37580]: pgmap v6835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:13 smithi112 ceph-mon[37501]: pgmap v6835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:13 smithi107 ceph-mon[32659]: pgmap v6835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:14.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:14.713+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:12:14.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:14.713+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:12:14.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:14.713+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:12:14.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:14.713+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:12:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:15 smithi132 ceph-mon[37580]: pgmap v6836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:15 smithi112 ceph-mon[37501]: pgmap v6836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:15.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:15 smithi107 ceph-mon[32659]: pgmap v6836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12072.020113 secs 2024-07-26T16:12:16.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:16.775+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:12:16.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:16.775+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:12:16.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:16.775+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:16.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12072.020113 secs 2024-07-26T16:12:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12072.020113 secs 2024-07-26T16:12:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:17 smithi112 ceph-mon[37501]: pgmap v6837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:17 smithi107 ceph-mon[32659]: pgmap v6837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:17 smithi132 ceph-mon[37580]: pgmap v6837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:19 smithi107 ceph-mon[32659]: pgmap v6838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:19 smithi112 ceph-mon[37501]: pgmap v6838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:19 smithi132 ceph-mon[37580]: pgmap v6838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:21 smithi112 ceph-mon[37501]: pgmap v6839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12077.020232 secs 2024-07-26T16:12:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:21 smithi107 ceph-mon[32659]: pgmap v6839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12077.020232 secs 2024-07-26T16:12:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:21 smithi132 ceph-mon[37580]: pgmap v6839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12077.020232 secs 2024-07-26T16:12:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:23 smithi112 ceph-mon[37501]: pgmap v6840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:23 smithi107 ceph-mon[32659]: pgmap v6840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:23 smithi132 ceph-mon[37580]: pgmap v6840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:25.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:25 smithi112 ceph-mon[37501]: pgmap v6841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:25 smithi107 ceph-mon[32659]: pgmap v6841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:25 smithi132 ceph-mon[37580]: pgmap v6841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12082.020348 secs 2024-07-26T16:12:26.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12082.020348 secs 2024-07-26T16:12:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12082.020348 secs 2024-07-26T16:12:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:27 smithi112 ceph-mon[37501]: pgmap v6842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:27 smithi107 ceph-mon[32659]: pgmap v6842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:27 smithi132 ceph-mon[37580]: pgmap v6842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:29 smithi112 ceph-mon[37501]: pgmap v6843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:29 smithi107 ceph-mon[32659]: pgmap v6843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:29 smithi132 ceph-mon[37580]: pgmap v6843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:31 smithi112 ceph-mon[37501]: pgmap v6844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12087.020460 secs 2024-07-26T16:12:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:31 smithi107 ceph-mon[32659]: pgmap v6844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12087.020460 secs 2024-07-26T16:12:31.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:31.791+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:12:31.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:31.791+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:12:31.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:31.791+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:31 smithi132 ceph-mon[37580]: pgmap v6844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12087.020460 secs 2024-07-26T16:12:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:33 smithi112 ceph-mon[37501]: pgmap v6845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:33.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:33 smithi107 ceph-mon[32659]: pgmap v6845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:33 smithi132 ceph-mon[37580]: pgmap v6845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:35 smithi107 ceph-mon[32659]: pgmap v6846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:35 smithi112 ceph-mon[37501]: pgmap v6846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:35 smithi132 ceph-mon[37580]: pgmap v6846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12092.020525 secs 2024-07-26T16:12:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:12:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12092.020525 secs 2024-07-26T16:12:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:12:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12092.020525 secs 2024-07-26T16:12:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:12:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:37 smithi112 ceph-mon[37501]: pgmap v6847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:12:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:12:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:12:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:37 smithi107 ceph-mon[32659]: pgmap v6847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:12:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:12:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:12:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:37 smithi132 ceph-mon[37580]: pgmap v6847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:12:37.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:12:37.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:12:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:38 smithi112 ceph-mon[37501]: pgmap v6848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:12:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:38 smithi112 ceph-mon[37501]: pgmap v6849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:12:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:38 smithi107 ceph-mon[32659]: pgmap v6848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:12:38.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:38 smithi107 ceph-mon[32659]: pgmap v6849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:12:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:38 smithi132 ceph-mon[37580]: pgmap v6848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:12:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:38 smithi132 ceph-mon[37580]: pgmap v6849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:12:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:40 smithi112 ceph-mon[37501]: pgmap v6850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:12:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:40 smithi107 ceph-mon[32659]: pgmap v6850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:12:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:40 smithi132 ceph-mon[37580]: pgmap v6850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:12:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12097.020651 secs 2024-07-26T16:12:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12097.020651 secs 2024-07-26T16:12:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12097.020651 secs 2024-07-26T16:12:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:42 smithi112 ceph-mon[37501]: pgmap v6851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:12:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:42 smithi107 ceph-mon[32659]: pgmap v6851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:12:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:42 smithi132 ceph-mon[37580]: pgmap v6851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:12:44.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:44 smithi112 ceph-mon[37501]: pgmap v6852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:12:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:44 smithi107 ceph-mon[32659]: pgmap v6852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:12:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:44 smithi132 ceph-mon[37580]: pgmap v6852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:12:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:46 smithi112 ceph-mon[37501]: pgmap v6853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:12:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12102.020790 secs 2024-07-26T16:12:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:46 smithi107 ceph-mon[32659]: pgmap v6853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:12:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12102.020790 secs 2024-07-26T16:12:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:46.807+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:12:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:46.807+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:12:46.807+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:12:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:46 smithi132 ceph-mon[37580]: pgmap v6853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:12:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12102.020790 secs 2024-07-26T16:12:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:48 smithi112 ceph-mon[37501]: pgmap v6854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:12:48.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:48 smithi107 ceph-mon[32659]: pgmap v6854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:12:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:48 smithi132 ceph-mon[37580]: pgmap v6854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:12:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:50 smithi112 ceph-mon[37501]: pgmap v6855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:50 smithi107 ceph-mon[32659]: pgmap v6855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:50 smithi132 ceph-mon[37580]: pgmap v6855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12107.020923 secs 2024-07-26T16:12:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12107.020923 secs 2024-07-26T16:12:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12107.020923 secs 2024-07-26T16:12:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:52 smithi107 ceph-mon[32659]: pgmap v6856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:52 smithi132 ceph-mon[37580]: pgmap v6856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:52 smithi112 ceph-mon[37501]: pgmap v6856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:54 smithi107 ceph-mon[32659]: pgmap v6857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:54 smithi132 ceph-mon[37580]: pgmap v6857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:54 smithi112 ceph-mon[37501]: pgmap v6857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:55 smithi107 ceph-mon[32659]: pgmap v6858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:55 smithi132 ceph-mon[37580]: pgmap v6858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:55 smithi112 ceph-mon[37501]: pgmap v6858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12112.021020 secs 2024-07-26T16:12:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12112.021020 secs 2024-07-26T16:12:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12112.021020 secs 2024-07-26T16:12:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:12:57 smithi107 ceph-mon[32659]: pgmap v6859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:12:57 smithi132 ceph-mon[37580]: pgmap v6859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:12:57 smithi112 ceph-mon[37501]: pgmap v6859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:00 smithi107 ceph-mon[32659]: pgmap v6860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:00 smithi132 ceph-mon[37580]: pgmap v6860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:00 smithi112 ceph-mon[37501]: pgmap v6860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12117.021137 secs 2024-07-26T16:13:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12117.021137 secs 2024-07-26T16:13:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12117.021137 secs 2024-07-26T16:13:01.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:01.822+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:13:01.824 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:01.822+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:01.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:01.822+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:13:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:02 smithi107 ceph-mon[32659]: pgmap v6861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:02 smithi132 ceph-mon[37580]: pgmap v6861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:02 smithi112 ceph-mon[37501]: pgmap v6861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:04.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:04 smithi107 ceph-mon[32659]: pgmap v6862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:04 smithi132 ceph-mon[37580]: pgmap v6862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:04 smithi112 ceph-mon[37501]: pgmap v6862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:06 smithi107 ceph-mon[32659]: pgmap v6863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12122.021258 secs 2024-07-26T16:13:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:06 smithi132 ceph-mon[37580]: pgmap v6863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12122.021258 secs 2024-07-26T16:13:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:06 smithi112 ceph-mon[37501]: pgmap v6863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12122.021258 secs 2024-07-26T16:13:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:08 smithi132 ceph-mon[37580]: pgmap v6864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:08 smithi112 ceph-mon[37501]: pgmap v6864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:08 smithi107 ceph-mon[32659]: pgmap v6864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:10 smithi132 ceph-mon[37580]: pgmap v6865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:10 smithi112 ceph-mon[37501]: pgmap v6865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:10 smithi107 ceph-mon[32659]: pgmap v6865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12127.021367 secs 2024-07-26T16:13:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12127.021367 secs 2024-07-26T16:13:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12127.021367 secs 2024-07-26T16:13:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:12 smithi112 ceph-mon[37501]: pgmap v6866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:12 smithi107 ceph-mon[32659]: pgmap v6866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:12 smithi132 ceph-mon[37580]: pgmap v6866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:14 smithi112 ceph-mon[37501]: pgmap v6867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:14 smithi107 ceph-mon[32659]: pgmap v6867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:14 smithi132 ceph-mon[37580]: pgmap v6867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:16 smithi112 ceph-mon[37501]: pgmap v6868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12132.021484 secs 2024-07-26T16:13:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:16 smithi107 ceph-mon[32659]: pgmap v6868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12132.021484 secs 2024-07-26T16:13:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:16 smithi132 ceph-mon[37580]: pgmap v6868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12132.021484 secs 2024-07-26T16:13:16.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:16.834+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:13:16.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:16.834+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:16.836 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:16.834+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:13:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:18 smithi132 ceph-mon[37580]: pgmap v6869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:18 smithi112 ceph-mon[37501]: pgmap v6869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:18 smithi107 ceph-mon[32659]: pgmap v6869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:20 smithi132 ceph-mon[37580]: pgmap v6870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:20 smithi112 ceph-mon[37501]: pgmap v6870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:20 smithi107 ceph-mon[32659]: pgmap v6870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12137.021605 secs 2024-07-26T16:13:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12137.021605 secs 2024-07-26T16:13:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12137.021605 secs 2024-07-26T16:13:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:22 smithi107 ceph-mon[32659]: pgmap v6871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:22 smithi112 ceph-mon[37501]: pgmap v6871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:22 smithi132 ceph-mon[37580]: pgmap v6871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:24 smithi112 ceph-mon[37501]: pgmap v6872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:24 smithi107 ceph-mon[32659]: pgmap v6872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:24 smithi132 ceph-mon[37580]: pgmap v6872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:26.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:26 smithi132 ceph-mon[37580]: pgmap v6873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:26.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12142.021744 secs 2024-07-26T16:13:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:26 smithi112 ceph-mon[37501]: pgmap v6873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12142.021744 secs 2024-07-26T16:13:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:26 smithi107 ceph-mon[32659]: pgmap v6873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12142.021744 secs 2024-07-26T16:13:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:28 smithi112 ceph-mon[37501]: pgmap v6874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:28 smithi107 ceph-mon[32659]: pgmap v6874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:28 smithi132 ceph-mon[37580]: pgmap v6874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:30 smithi112 ceph-mon[37501]: pgmap v6875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:30 smithi107 ceph-mon[32659]: pgmap v6875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:30 smithi132 ceph-mon[37580]: pgmap v6875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:31.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12147.021892 secs 2024-07-26T16:13:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12147.021892 secs 2024-07-26T16:13:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12147.021892 secs 2024-07-26T16:13:31.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:31.844+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:13:31.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:31.844+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:31.846 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:31.844+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:13:32.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:32 smithi112 ceph-mon[37501]: pgmap v6876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:32 smithi107 ceph-mon[32659]: pgmap v6876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:32 smithi132 ceph-mon[37580]: pgmap v6876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:34 smithi112 ceph-mon[37501]: pgmap v6877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:34 smithi107 ceph-mon[32659]: pgmap v6877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:34 smithi132 ceph-mon[37580]: pgmap v6877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:36 smithi112 ceph-mon[37501]: pgmap v6878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12152.021978 secs 2024-07-26T16:13:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:36 smithi107 ceph-mon[32659]: pgmap v6878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12152.021978 secs 2024-07-26T16:13:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:36 smithi132 ceph-mon[37580]: pgmap v6878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12152.021978 secs 2024-07-26T16:13:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:13:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:13:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:13:38.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:38 smithi107 ceph-mon[32659]: pgmap v6879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:38.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:13:38.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:13:38.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:13:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:38 smithi112 ceph-mon[37501]: pgmap v6879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:13:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:13:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:13:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:38 smithi132 ceph-mon[37580]: pgmap v6879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:13:38.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:13:38.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:13:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:39 smithi112 ceph-mon[37501]: pgmap v6880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:13:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:39 smithi112 ceph-mon[37501]: pgmap v6881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:13:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:39 smithi107 ceph-mon[32659]: pgmap v6880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:13:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:39 smithi107 ceph-mon[32659]: pgmap v6881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:13:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:39 smithi132 ceph-mon[37580]: pgmap v6880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:13:39.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:39 smithi132 ceph-mon[37580]: pgmap v6881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:13:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:41 smithi112 ceph-mon[37501]: pgmap v6882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12157.022105 secs 2024-07-26T16:13:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:41 smithi107 ceph-mon[32659]: pgmap v6882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12157.022105 secs 2024-07-26T16:13:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:41 smithi132 ceph-mon[37580]: pgmap v6882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12157.022105 secs 2024-07-26T16:13:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:43 smithi112 ceph-mon[37501]: pgmap v6883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:13:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:43 smithi107 ceph-mon[32659]: pgmap v6883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:13:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:43 smithi132 ceph-mon[37580]: pgmap v6883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:13:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:45 smithi112 ceph-mon[37501]: pgmap v6884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:45 smithi107 ceph-mon[32659]: pgmap v6884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:45 smithi132 ceph-mon[37580]: pgmap v6884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12162.022231 secs 2024-07-26T16:13:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12162.022231 secs 2024-07-26T16:13:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12162.022231 secs 2024-07-26T16:13:46.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:46.859+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:13:46.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:46.859+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:13:46.892 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:13:46.859+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:47 smithi112 ceph-mon[37501]: pgmap v6885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:13:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:47 smithi107 ceph-mon[32659]: pgmap v6885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:13:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:47 smithi132 ceph-mon[37580]: pgmap v6885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:13:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:49 smithi112 ceph-mon[37501]: pgmap v6886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:13:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:49 smithi107 ceph-mon[32659]: pgmap v6886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:13:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:49 smithi132 ceph-mon[37580]: pgmap v6886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:13:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:51 smithi112 ceph-mon[37501]: pgmap v6887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12167.022339 secs 2024-07-26T16:13:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:51 smithi107 ceph-mon[32659]: pgmap v6887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12167.022339 secs 2024-07-26T16:13:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:51 smithi132 ceph-mon[37580]: pgmap v6887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12167.022339 secs 2024-07-26T16:13:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:53 smithi112 ceph-mon[37501]: pgmap v6888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:53 smithi107 ceph-mon[32659]: pgmap v6888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:53 smithi132 ceph-mon[37580]: pgmap v6888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:13:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:55 smithi112 ceph-mon[37501]: pgmap v6889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:55 smithi107 ceph-mon[32659]: pgmap v6889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:55 smithi132 ceph-mon[37580]: pgmap v6889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:56.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12172.022459 secs 2024-07-26T16:13:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12172.022459 secs 2024-07-26T16:13:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12172.022459 secs 2024-07-26T16:13:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:57 smithi107 ceph-mon[32659]: pgmap v6890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:57 smithi132 ceph-mon[37580]: pgmap v6890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:57 smithi112 ceph-mon[37501]: pgmap v6890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:13:59 smithi107 ceph-mon[32659]: pgmap v6891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:13:59 smithi132 ceph-mon[37580]: pgmap v6891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:13:59 smithi112 ceph-mon[37501]: pgmap v6891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:01 smithi107 ceph-mon[32659]: pgmap v6892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12177.022585 secs 2024-07-26T16:14:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:01 smithi132 ceph-mon[37580]: pgmap v6892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12177.022585 secs 2024-07-26T16:14:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:01.874+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:14:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:01.874+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:14:01.876 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:01.874+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:01 smithi112 ceph-mon[37501]: pgmap v6892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12177.022585 secs 2024-07-26T16:14:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:03 smithi107 ceph-mon[32659]: pgmap v6893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:03 smithi132 ceph-mon[37580]: pgmap v6893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:03 smithi112 ceph-mon[37501]: pgmap v6893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:05.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:05 smithi107 ceph-mon[32659]: pgmap v6894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:05 smithi132 ceph-mon[37580]: pgmap v6894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:05 smithi112 ceph-mon[37501]: pgmap v6894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12182.022712 secs 2024-07-26T16:14:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12182.022712 secs 2024-07-26T16:14:06.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12182.022712 secs 2024-07-26T16:14:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:07 smithi107 ceph-mon[32659]: pgmap v6895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:07 smithi132 ceph-mon[37580]: pgmap v6895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:07 smithi112 ceph-mon[37501]: pgmap v6895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:08 smithi107 ceph-mon[32659]: pgmap v6896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:08 smithi132 ceph-mon[37580]: pgmap v6896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:08 smithi112 ceph-mon[37501]: pgmap v6896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:11 smithi107 ceph-mon[32659]: pgmap v6897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12187.022843 secs 2024-07-26T16:14:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:11 smithi132 ceph-mon[37580]: pgmap v6897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12187.022843 secs 2024-07-26T16:14:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:11 smithi112 ceph-mon[37501]: pgmap v6897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12187.022843 secs 2024-07-26T16:14:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:13 smithi107 ceph-mon[32659]: pgmap v6898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:13 smithi132 ceph-mon[37580]: pgmap v6898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:13 smithi112 ceph-mon[37501]: pgmap v6898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:15 smithi107 ceph-mon[32659]: pgmap v6899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:15 smithi132 ceph-mon[37580]: pgmap v6899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:15 smithi112 ceph-mon[37501]: pgmap v6899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12192.022966 secs 2024-07-26T16:14:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12192.022966 secs 2024-07-26T16:14:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12192.022966 secs 2024-07-26T16:14:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:16.878+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:14:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:16.878+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:14:16.880 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:16.878+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:17.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:17 smithi107 ceph-mon[32659]: pgmap v6900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:17 smithi132 ceph-mon[37580]: pgmap v6900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:17 smithi112 ceph-mon[37501]: pgmap v6900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:19 smithi107 ceph-mon[32659]: pgmap v6901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:19 smithi132 ceph-mon[37580]: pgmap v6901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:19 smithi112 ceph-mon[37501]: pgmap v6901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:21 smithi107 ceph-mon[32659]: pgmap v6902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12197.023097 secs 2024-07-26T16:14:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:21 smithi132 ceph-mon[37580]: pgmap v6902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12197.023097 secs 2024-07-26T16:14:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:21 smithi112 ceph-mon[37501]: pgmap v6902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12197.023097 secs 2024-07-26T16:14:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:23 smithi107 ceph-mon[32659]: pgmap v6903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:23 smithi132 ceph-mon[37580]: pgmap v6903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:23 smithi112 ceph-mon[37501]: pgmap v6903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:25 smithi107 ceph-mon[32659]: pgmap v6904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:25 smithi132 ceph-mon[37580]: pgmap v6904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:25 smithi112 ceph-mon[37501]: pgmap v6904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12202.023229 secs 2024-07-26T16:14:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12202.023229 secs 2024-07-26T16:14:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12202.023229 secs 2024-07-26T16:14:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:27 smithi107 ceph-mon[32659]: pgmap v6905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:27 smithi132 ceph-mon[37580]: pgmap v6905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:27 smithi112 ceph-mon[37501]: pgmap v6905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:29 smithi107 ceph-mon[32659]: pgmap v6906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:29 smithi132 ceph-mon[37580]: pgmap v6906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:29 smithi112 ceph-mon[37501]: pgmap v6906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:31 smithi132 ceph-mon[37580]: pgmap v6907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12207.023355 secs 2024-07-26T16:14:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:31 smithi112 ceph-mon[37501]: pgmap v6907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12207.023355 secs 2024-07-26T16:14:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:31 smithi107 ceph-mon[32659]: pgmap v6907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12207.023355 secs 2024-07-26T16:14:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:31.894+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:14:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:31.894+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:31.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:31.894+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:14:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:33 smithi132 ceph-mon[37580]: pgmap v6908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:33 smithi112 ceph-mon[37501]: pgmap v6908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:33 smithi107 ceph-mon[32659]: pgmap v6908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:35 smithi132 ceph-mon[37580]: pgmap v6909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:35 smithi112 ceph-mon[37501]: pgmap v6909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:35 smithi107 ceph-mon[32659]: pgmap v6909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:36.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12212.023503 secs 2024-07-26T16:14:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12212.023503 secs 2024-07-26T16:14:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12212.023503 secs 2024-07-26T16:14:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:37 smithi112 ceph-mon[37501]: pgmap v6910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:37.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:37 smithi107 ceph-mon[32659]: pgmap v6910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:37 smithi132 ceph-mon[37580]: pgmap v6910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:39 smithi112 ceph-mon[37501]: pgmap v6911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:14:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:39 smithi107 ceph-mon[32659]: pgmap v6911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:14:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:39 smithi132 ceph-mon[37580]: pgmap v6911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:14:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:14:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:14:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:14:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:40 smithi112 ceph-mon[37501]: pgmap v6912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:14:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:14:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:14:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:14:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:40 smithi107 ceph-mon[32659]: pgmap v6912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:14:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:14:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:14:40.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:14:40.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:40 smithi132 ceph-mon[37580]: pgmap v6912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:14:40.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:14:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12217.023653 secs 2024-07-26T16:14:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12217.023653 secs 2024-07-26T16:14:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12217.023653 secs 2024-07-26T16:14:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:42 smithi112 ceph-mon[37501]: pgmap v6913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:14:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:42 smithi107 ceph-mon[32659]: pgmap v6913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:14:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:42 smithi132 ceph-mon[37580]: pgmap v6913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:14:44.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:44 smithi112 ceph-mon[37501]: pgmap v6914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:14:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:44 smithi107 ceph-mon[32659]: pgmap v6914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:14:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:44 smithi132 ceph-mon[37580]: pgmap v6914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:14:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12222.023798 secs 2024-07-26T16:14:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:46 smithi112 ceph-mon[37501]: pgmap v6915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:14:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12222.023798 secs 2024-07-26T16:14:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:46 smithi107 ceph-mon[32659]: pgmap v6915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:14:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12222.023798 secs 2024-07-26T16:14:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:46 smithi132 ceph-mon[37580]: pgmap v6915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:14:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:46.909+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:14:46.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:46.910+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:46.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:14:46.910+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:14:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:48 smithi112 ceph-mon[37501]: pgmap v6916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:14:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:48 smithi107 ceph-mon[32659]: pgmap v6916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:14:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:48 smithi132 ceph-mon[37580]: pgmap v6916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:14:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:50 smithi112 ceph-mon[37501]: pgmap v6917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:14:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:50 smithi107 ceph-mon[32659]: pgmap v6917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:14:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:50 smithi132 ceph-mon[37580]: pgmap v6917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:14:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12227.023922 secs 2024-07-26T16:14:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12227.023922 secs 2024-07-26T16:14:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12227.023922 secs 2024-07-26T16:14:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:52 smithi107 ceph-mon[32659]: pgmap v6918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:14:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:52 smithi132 ceph-mon[37580]: pgmap v6918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:14:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:52 smithi112 ceph-mon[37501]: pgmap v6918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:14:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:54 smithi107 ceph-mon[32659]: pgmap v6919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:14:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:54 smithi132 ceph-mon[37580]: pgmap v6919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:14:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:54 smithi112 ceph-mon[37501]: pgmap v6919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:14:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12232.024047 secs 2024-07-26T16:14:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:56 smithi107 ceph-mon[32659]: pgmap v6920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:14:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12232.024047 secs 2024-07-26T16:14:56.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:56 smithi132 ceph-mon[37580]: pgmap v6920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:14:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12232.024047 secs 2024-07-26T16:14:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:56 smithi112 ceph-mon[37501]: pgmap v6920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:14:58.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:14:58 smithi107 ceph-mon[32659]: pgmap v6921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:14:58 smithi132 ceph-mon[37580]: pgmap v6921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:14:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:14:58 smithi112 ceph-mon[37501]: pgmap v6921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:00 smithi107 ceph-mon[32659]: pgmap v6922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T16:15:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:00 smithi132 ceph-mon[37580]: pgmap v6922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T16:15:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:00 smithi112 ceph-mon[37501]: pgmap v6922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T16:15:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12237.024152 secs 2024-07-26T16:15:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12237.024152 secs 2024-07-26T16:15:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:01.924+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:15:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:01.924+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:01.927 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:01.924+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:15:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12237.024152 secs 2024-07-26T16:15:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:02 smithi107 ceph-mon[32659]: pgmap v6923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:02 smithi132 ceph-mon[37580]: pgmap v6923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:02 smithi112 ceph-mon[37501]: pgmap v6923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:04 smithi107 ceph-mon[32659]: pgmap v6924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:04 smithi132 ceph-mon[37580]: pgmap v6924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:04 smithi112 ceph-mon[37501]: pgmap v6924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:06.752 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12242.024281 secs 2024-07-26T16:15:06.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:06 smithi112 ceph-mon[37501]: pgmap v6925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12242.024281 secs 2024-07-26T16:15:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:06 smithi107 ceph-mon[32659]: pgmap v6925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12242.024281 secs 2024-07-26T16:15:06.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:06 smithi132 ceph-mon[37580]: pgmap v6925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:07 smithi107 ceph-mon[32659]: pgmap v6926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:07 smithi132 ceph-mon[37580]: pgmap v6926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:07 smithi112 ceph-mon[37501]: pgmap v6926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:10 smithi112 ceph-mon[37501]: pgmap v6927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:10 smithi107 ceph-mon[32659]: pgmap v6927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:10 smithi132 ceph-mon[37580]: pgmap v6927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12247.024398 secs 2024-07-26T16:15:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12247.024398 secs 2024-07-26T16:15:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12247.024398 secs 2024-07-26T16:15:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:12 smithi107 ceph-mon[32659]: pgmap v6928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:12 smithi132 ceph-mon[37580]: pgmap v6928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:12 smithi112 ceph-mon[37501]: pgmap v6928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:14.744+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:15:14.747 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:14.744+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:15:14.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:14 smithi107 ceph-mon[32659]: pgmap v6929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:14.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:14.745+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:15:14.748 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:14.745+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:15:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:14 smithi132 ceph-mon[37580]: pgmap v6929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:14 smithi112 ceph-mon[37501]: pgmap v6929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12252.024524 secs 2024-07-26T16:15:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:16 smithi107 ceph-mon[32659]: pgmap v6930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12252.024524 secs 2024-07-26T16:15:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:16 smithi132 ceph-mon[37580]: pgmap v6930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:16.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:16.935+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:15:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:16.935+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:15:16.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:16.935+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12252.024524 secs 2024-07-26T16:15:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:16 smithi112 ceph-mon[37501]: pgmap v6930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:18 smithi107 ceph-mon[32659]: pgmap v6931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:18 smithi132 ceph-mon[37580]: pgmap v6931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:18 smithi112 ceph-mon[37501]: pgmap v6931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:20 smithi107 ceph-mon[32659]: pgmap v6932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:20 smithi132 ceph-mon[37580]: pgmap v6932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:20 smithi112 ceph-mon[37501]: pgmap v6932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12257.024669 secs 2024-07-26T16:15:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12257.024669 secs 2024-07-26T16:15:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12257.024669 secs 2024-07-26T16:15:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:22 smithi107 ceph-mon[32659]: pgmap v6933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:22 smithi132 ceph-mon[37580]: pgmap v6933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:22 smithi112 ceph-mon[37501]: pgmap v6933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:24 smithi107 ceph-mon[32659]: pgmap v6934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:24 smithi132 ceph-mon[37580]: pgmap v6934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:24 smithi112 ceph-mon[37501]: pgmap v6934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12262.024819 secs 2024-07-26T16:15:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:26 smithi107 ceph-mon[32659]: pgmap v6935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12262.024819 secs 2024-07-26T16:15:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:26 smithi132 ceph-mon[37580]: pgmap v6935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12262.024819 secs 2024-07-26T16:15:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:26 smithi112 ceph-mon[37501]: pgmap v6935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:28 smithi107 ceph-mon[32659]: pgmap v6936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:28 smithi132 ceph-mon[37580]: pgmap v6936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:28 smithi112 ceph-mon[37501]: pgmap v6936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:29 smithi107 ceph-mon[32659]: pgmap v6937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:29 smithi132 ceph-mon[37580]: pgmap v6937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:29 smithi112 ceph-mon[37501]: pgmap v6937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12267.024982 secs 2024-07-26T16:15:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12267.024982 secs 2024-07-26T16:15:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12267.024982 secs 2024-07-26T16:15:31.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:31.951+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:15:31.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:31.951+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:31.954 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:31.951+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:15:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:32 smithi107 ceph-mon[32659]: pgmap v6938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:32 smithi132 ceph-mon[37580]: pgmap v6938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:32 smithi112 ceph-mon[37501]: pgmap v6938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:34 smithi107 ceph-mon[32659]: pgmap v6939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:34 smithi132 ceph-mon[37580]: pgmap v6939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:34 smithi112 ceph-mon[37501]: pgmap v6939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:36.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12272.025111 secs 2024-07-26T16:15:36.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:36 smithi112 ceph-mon[37501]: pgmap v6940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12272.025111 secs 2024-07-26T16:15:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:36 smithi107 ceph-mon[32659]: pgmap v6940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12272.025111 secs 2024-07-26T16:15:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:36 smithi132 ceph-mon[37580]: pgmap v6940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:38 smithi107 ceph-mon[32659]: pgmap v6941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:38 smithi132 ceph-mon[37580]: pgmap v6941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:38 smithi112 ceph-mon[37501]: pgmap v6941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:40 smithi107 ceph-mon[32659]: pgmap v6942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:15:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:40 smithi132 ceph-mon[37580]: pgmap v6942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:15:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:40 smithi112 ceph-mon[37501]: pgmap v6942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:15:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:15:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12277.025229 secs 2024-07-26T16:15:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:15:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:15:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:15:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12277.025229 secs 2024-07-26T16:15:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:15:41.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:15:41.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:15:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12277.025229 secs 2024-07-26T16:15:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:15:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:15:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:15:42.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:42 smithi107 ceph-mon[32659]: pgmap v6943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:15:42.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:42 smithi107 ceph-mon[32659]: pgmap v6944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:15:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:42 smithi132 ceph-mon[37580]: pgmap v6943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:15:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:42 smithi132 ceph-mon[37580]: pgmap v6944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:15:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:42 smithi112 ceph-mon[37501]: pgmap v6943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:15:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:42 smithi112 ceph-mon[37501]: pgmap v6944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:15:44.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:44 smithi107 ceph-mon[32659]: pgmap v6945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:15:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:44 smithi132 ceph-mon[37580]: pgmap v6945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:15:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:44 smithi112 ceph-mon[37501]: pgmap v6945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:15:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:46 smithi107 ceph-mon[32659]: pgmap v6946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12282.025361 secs 2024-07-26T16:15:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:46 smithi132 ceph-mon[37580]: pgmap v6946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12282.025361 secs 2024-07-26T16:15:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:46 smithi112 ceph-mon[37501]: pgmap v6946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12282.025361 secs 2024-07-26T16:15:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:46.954+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:15:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:46.954+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:15:46.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:15:46.954+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:47 smithi107 ceph-mon[32659]: pgmap v6947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:47 smithi132 ceph-mon[37580]: pgmap v6947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:47 smithi112 ceph-mon[37501]: pgmap v6947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:49 smithi112 ceph-mon[37501]: pgmap v6948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:15:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:49 smithi107 ceph-mon[32659]: pgmap v6948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:15:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:49 smithi132 ceph-mon[37580]: pgmap v6948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:15:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12287.025444 secs 2024-07-26T16:15:51.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:51 smithi112 ceph-mon[37501]: pgmap v6949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:15:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12287.025444 secs 2024-07-26T16:15:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:51 smithi107 ceph-mon[32659]: pgmap v6949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:15:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12287.025444 secs 2024-07-26T16:15:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:51 smithi132 ceph-mon[37580]: pgmap v6949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:15:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:53 smithi112 ceph-mon[37501]: pgmap v6950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:53 smithi107 ceph-mon[32659]: pgmap v6950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:53 smithi132 ceph-mon[37580]: pgmap v6950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:15:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:55 smithi112 ceph-mon[37501]: pgmap v6951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:15:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:55 smithi107 ceph-mon[32659]: pgmap v6951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:15:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:55 smithi132 ceph-mon[37580]: pgmap v6951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:15:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12292.025526 secs 2024-07-26T16:15:56.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12292.025526 secs 2024-07-26T16:15:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12292.025526 secs 2024-07-26T16:15:57.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:57 smithi107 ceph-mon[32659]: pgmap v6952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:57 smithi112 ceph-mon[37501]: pgmap v6952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:57 smithi132 ceph-mon[37580]: pgmap v6952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:15:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:15:59 smithi112 ceph-mon[37501]: pgmap v6953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:15:59 smithi107 ceph-mon[32659]: pgmap v6953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:15:59 smithi132 ceph-mon[37580]: pgmap v6953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12297.025641 secs 2024-07-26T16:16:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:01 smithi112 ceph-mon[37501]: pgmap v6954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:01.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:01.969+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:16:01.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:01.969+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:01.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:01.969+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:16:01.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12297.025641 secs 2024-07-26T16:16:01.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:01 smithi107 ceph-mon[32659]: pgmap v6954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12297.025641 secs 2024-07-26T16:16:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:01 smithi132 ceph-mon[37580]: pgmap v6954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:03 smithi112 ceph-mon[37501]: pgmap v6955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:03 smithi107 ceph-mon[32659]: pgmap v6955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:03 smithi132 ceph-mon[37580]: pgmap v6955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:05 smithi112 ceph-mon[37501]: pgmap v6956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:05 smithi107 ceph-mon[32659]: pgmap v6956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:05 smithi132 ceph-mon[37580]: pgmap v6956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12302.025760 secs 2024-07-26T16:16:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12302.025760 secs 2024-07-26T16:16:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12302.025760 secs 2024-07-26T16:16:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:07 smithi112 ceph-mon[37501]: pgmap v6957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:07 smithi107 ceph-mon[32659]: pgmap v6957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:07 smithi132 ceph-mon[37580]: pgmap v6957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:09 smithi112 ceph-mon[37501]: pgmap v6958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:09 smithi107 ceph-mon[32659]: pgmap v6958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:09 smithi132 ceph-mon[37580]: pgmap v6958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12307.025902 secs 2024-07-26T16:16:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:11 smithi107 ceph-mon[32659]: pgmap v6959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12307.025902 secs 2024-07-26T16:16:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:11 smithi132 ceph-mon[37580]: pgmap v6959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12307.025902 secs 2024-07-26T16:16:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:11 smithi112 ceph-mon[37501]: pgmap v6959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:13 smithi107 ceph-mon[32659]: pgmap v6960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:13 smithi132 ceph-mon[37580]: pgmap v6960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:13 smithi112 ceph-mon[37501]: pgmap v6960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:15 smithi107 ceph-mon[32659]: pgmap v6961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:15 smithi132 ceph-mon[37580]: pgmap v6961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:15 smithi112 ceph-mon[37501]: pgmap v6961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:16.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:16.984+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:16:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:16.984+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:16.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:16.984+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:16:16.988 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12312.026022 secs 2024-07-26T16:16:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12312.026022 secs 2024-07-26T16:16:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12312.026022 secs 2024-07-26T16:16:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:17 smithi107 ceph-mon[32659]: pgmap v6962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:17 smithi132 ceph-mon[37580]: pgmap v6962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:17 smithi112 ceph-mon[37501]: pgmap v6962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:19 smithi107 ceph-mon[32659]: pgmap v6963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:20.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:19 smithi132 ceph-mon[37580]: pgmap v6963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:19 smithi112 ceph-mon[37501]: pgmap v6963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12317.026144 secs 2024-07-26T16:16:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:21 smithi107 ceph-mon[32659]: pgmap v6964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12317.026144 secs 2024-07-26T16:16:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:21 smithi132 ceph-mon[37580]: pgmap v6964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12317.026144 secs 2024-07-26T16:16:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:21 smithi112 ceph-mon[37501]: pgmap v6964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:23 smithi107 ceph-mon[32659]: pgmap v6965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:23 smithi132 ceph-mon[37580]: pgmap v6965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:23 smithi112 ceph-mon[37501]: pgmap v6965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:25 smithi107 ceph-mon[32659]: pgmap v6966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:25 smithi132 ceph-mon[37580]: pgmap v6966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:25 smithi112 ceph-mon[37501]: pgmap v6966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12322.026269 secs 2024-07-26T16:16:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12322.026269 secs 2024-07-26T16:16:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12322.026269 secs 2024-07-26T16:16:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:27 smithi107 ceph-mon[32659]: pgmap v6967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:27 smithi132 ceph-mon[37580]: pgmap v6967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:27 smithi112 ceph-mon[37501]: pgmap v6967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:30.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:29 smithi107 ceph-mon[32659]: pgmap v6968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:29 smithi132 ceph-mon[37580]: pgmap v6968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:29 smithi112 ceph-mon[37501]: pgmap v6968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:31.989+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:16:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:31.989+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:16:31.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:31.989+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12327.026389 secs 2024-07-26T16:16:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:31 smithi107 ceph-mon[32659]: pgmap v6969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12327.026389 secs 2024-07-26T16:16:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:31 smithi132 ceph-mon[37580]: pgmap v6969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12327.026389 secs 2024-07-26T16:16:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:31 smithi112 ceph-mon[37501]: pgmap v6969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:33 smithi107 ceph-mon[32659]: pgmap v6970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:33 smithi132 ceph-mon[37580]: pgmap v6970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:33 smithi112 ceph-mon[37501]: pgmap v6970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:35 smithi107 ceph-mon[32659]: pgmap v6971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:35 smithi132 ceph-mon[37580]: pgmap v6971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:35 smithi112 ceph-mon[37501]: pgmap v6971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:16:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12332.026539 secs 2024-07-26T16:16:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12332.026539 secs 2024-07-26T16:16:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12332.026539 secs 2024-07-26T16:16:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:37 smithi132 ceph-mon[37580]: pgmap v6972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:37 smithi112 ceph-mon[37501]: pgmap v6972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:37 smithi107 ceph-mon[32659]: pgmap v6972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:16:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:39 smithi132 ceph-mon[37580]: pgmap v6973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:39 smithi112 ceph-mon[37501]: pgmap v6973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:39 smithi107 ceph-mon[32659]: pgmap v6973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:16:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:16:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:16:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12337.026639 secs 2024-07-26T16:16:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:41 smithi112 ceph-mon[37501]: pgmap v6974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:16:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:41 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:16:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:41 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:16:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12337.026639 secs 2024-07-26T16:16:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:41 smithi107 ceph-mon[32659]: pgmap v6974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:16:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:41 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:16:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:41 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:16:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12337.026639 secs 2024-07-26T16:16:42.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:41 smithi132 ceph-mon[37580]: pgmap v6974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:42.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:16:42.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:41 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:16:42.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:41 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:16:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:42 smithi112 ceph-mon[37501]: pgmap v6975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:16:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:42 smithi112 ceph-mon[37501]: pgmap v6976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:42 smithi107 ceph-mon[32659]: pgmap v6975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:16:43.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:42 smithi107 ceph-mon[32659]: pgmap v6976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:42 smithi132 ceph-mon[37580]: pgmap v6975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:16:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:42 smithi132 ceph-mon[37580]: pgmap v6976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:44 smithi112 ceph-mon[37501]: pgmap v6977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:16:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:44 smithi107 ceph-mon[32659]: pgmap v6977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:16:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:44 smithi132 ceph-mon[37580]: pgmap v6977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:16:47.006 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:47.005+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:16:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:47.005+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:16:47.005+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:16:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12342.026740 secs 2024-07-26T16:16:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:46 smithi112 ceph-mon[37501]: pgmap v6978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:16:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12342.026740 secs 2024-07-26T16:16:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:46 smithi107 ceph-mon[32659]: pgmap v6978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:16:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12342.026740 secs 2024-07-26T16:16:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:46 smithi132 ceph-mon[37580]: pgmap v6978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:16:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:48 smithi112 ceph-mon[37501]: pgmap v6979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:16:49.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:48 smithi107 ceph-mon[32659]: pgmap v6979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:16:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:48 smithi132 ceph-mon[37580]: pgmap v6979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:16:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:50 smithi112 ceph-mon[37501]: pgmap v6980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:16:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:50 smithi107 ceph-mon[32659]: pgmap v6980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:16:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:50 smithi132 ceph-mon[37580]: pgmap v6980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:16:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12347.026883 secs 2024-07-26T16:16:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12347.026883 secs 2024-07-26T16:16:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12347.026883 secs 2024-07-26T16:16:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:52 smithi112 ceph-mon[37501]: pgmap v6981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:16:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:52 smithi107 ceph-mon[32659]: pgmap v6981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:16:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:52 smithi132 ceph-mon[37580]: pgmap v6981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:16:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:54 smithi112 ceph-mon[37501]: pgmap v6982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:54 smithi107 ceph-mon[32659]: pgmap v6982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:54 smithi132 ceph-mon[37580]: pgmap v6982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12352.027007 secs 2024-07-26T16:16:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:56 smithi112 ceph-mon[37501]: pgmap v6983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:57.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12352.027007 secs 2024-07-26T16:16:57.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:56 smithi107 ceph-mon[32659]: pgmap v6983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12352.027007 secs 2024-07-26T16:16:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:56 smithi132 ceph-mon[37580]: pgmap v6983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:16:58 smithi112 ceph-mon[37501]: pgmap v6984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:16:58 smithi107 ceph-mon[32659]: pgmap v6984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:16:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:16:58 smithi132 ceph-mon[37580]: pgmap v6984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:00 smithi112 ceph-mon[37501]: pgmap v6985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:17:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:00 smithi107 ceph-mon[32659]: pgmap v6985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:17:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:00 smithi132 ceph-mon[37580]: pgmap v6985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:17:02.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:02.011+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:17:02.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:02.011+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:02.014 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:02.011+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:17:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12357.027164 secs 2024-07-26T16:17:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12357.027164 secs 2024-07-26T16:17:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12357.027164 secs 2024-07-26T16:17:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:02 smithi112 ceph-mon[37501]: pgmap v6986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:02 smithi107 ceph-mon[32659]: pgmap v6986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:02 smithi132 ceph-mon[37580]: pgmap v6986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:04 smithi112 ceph-mon[37501]: pgmap v6987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:04 smithi107 ceph-mon[32659]: pgmap v6987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:04 smithi132 ceph-mon[37580]: pgmap v6987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12362.027289 secs 2024-07-26T16:17:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:06 smithi112 ceph-mon[37501]: pgmap v6988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12362.027289 secs 2024-07-26T16:17:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:06 smithi107 ceph-mon[32659]: pgmap v6988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12362.027289 secs 2024-07-26T16:17:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:06 smithi132 ceph-mon[37580]: pgmap v6988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:08 smithi112 ceph-mon[37501]: pgmap v6989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:09.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:08 smithi107 ceph-mon[32659]: pgmap v6989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:08 smithi132 ceph-mon[37580]: pgmap v6989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:10 smithi112 ceph-mon[37501]: pgmap v6990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:17:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:10 smithi107 ceph-mon[32659]: pgmap v6990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:17:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:10 smithi132 ceph-mon[37580]: pgmap v6990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:17:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12367.027386 secs 2024-07-26T16:17:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12367.027386 secs 2024-07-26T16:17:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12367.027386 secs 2024-07-26T16:17:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:12 smithi112 ceph-mon[37501]: pgmap v6991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:12 smithi107 ceph-mon[32659]: pgmap v6991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:12 smithi132 ceph-mon[37580]: pgmap v6991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:15.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:14 smithi112 ceph-mon[37501]: pgmap v6992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:15.227 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:14 smithi107 ceph-mon[32659]: pgmap v6992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:14 smithi132 ceph-mon[37580]: pgmap v6992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:17.025 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:17.022+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:17:17.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:17.022+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:17:17.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:17.022+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12372.027502 secs 2024-07-26T16:17:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:16 smithi112 ceph-mon[37501]: pgmap v6993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12372.027502 secs 2024-07-26T16:17:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:16 smithi107 ceph-mon[32659]: pgmap v6993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12372.027502 secs 2024-07-26T16:17:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:16 smithi132 ceph-mon[37580]: pgmap v6993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:19 smithi112 ceph-mon[37501]: pgmap v6994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:19 smithi107 ceph-mon[32659]: pgmap v6994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:19 smithi132 ceph-mon[37580]: pgmap v6994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:21 smithi112 ceph-mon[37501]: pgmap v6995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12377.027576 secs 2024-07-26T16:17:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:21 smithi107 ceph-mon[32659]: pgmap v6995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12377.027576 secs 2024-07-26T16:17:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:21 smithi132 ceph-mon[37580]: pgmap v6995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:21.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12377.027576 secs 2024-07-26T16:17:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:23 smithi112 ceph-mon[37501]: pgmap v6996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:17:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:23 smithi107 ceph-mon[32659]: pgmap v6996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:17:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:23 smithi132 ceph-mon[37580]: pgmap v6996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:17:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:25 smithi112 ceph-mon[37501]: pgmap v6997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:25.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:25 smithi107 ceph-mon[32659]: pgmap v6997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:25 smithi132 ceph-mon[37580]: pgmap v6997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12382.027698 secs 2024-07-26T16:17:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12382.027698 secs 2024-07-26T16:17:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12382.027698 secs 2024-07-26T16:17:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:27 smithi112 ceph-mon[37501]: pgmap v6998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:27.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:27 smithi132 ceph-mon[37580]: pgmap v6998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:27 smithi107 ceph-mon[32659]: pgmap v6998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:28.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:28.769+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:17:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:28.769+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:17:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:28.770+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:17:28.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:28.770+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:17:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:29 smithi112 ceph-mon[37501]: pgmap v6999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:29 smithi107 ceph-mon[32659]: pgmap v6999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:29 smithi132 ceph-mon[37580]: pgmap v6999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:31 smithi112 ceph-mon[37501]: pgmap v7000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12387.027832 secs 2024-07-26T16:17:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:31 smithi107 ceph-mon[32659]: pgmap v7000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12387.027832 secs 2024-07-26T16:17:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:31 smithi132 ceph-mon[37580]: pgmap v7000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12387.027832 secs 2024-07-26T16:17:32.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:32.038+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:17:32.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:32.038+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:32.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:32.038+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:17:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:33 smithi107 ceph-mon[32659]: pgmap v7001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:33 smithi132 ceph-mon[37580]: pgmap v7001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:33 smithi112 ceph-mon[37501]: pgmap v7001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:35 smithi107 ceph-mon[32659]: pgmap v7002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:35 smithi132 ceph-mon[37580]: pgmap v7002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:35 smithi112 ceph-mon[37501]: pgmap v7002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12392.027947 secs 2024-07-26T16:17:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12392.027947 secs 2024-07-26T16:17:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12392.027947 secs 2024-07-26T16:17:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:37 smithi107 ceph-mon[32659]: pgmap v7003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:37 smithi132 ceph-mon[37580]: pgmap v7003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:37.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:37 smithi112 ceph-mon[37501]: pgmap v7003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:39 smithi107 ceph-mon[32659]: pgmap v7004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:39 smithi132 ceph-mon[37580]: pgmap v7004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:39 smithi112 ceph-mon[37501]: pgmap v7004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:41 smithi107 ceph-mon[32659]: pgmap v7005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12397.028081 secs 2024-07-26T16:17:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:41 smithi132 ceph-mon[37580]: pgmap v7005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12397.028081 secs 2024-07-26T16:17:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:41 smithi112 ceph-mon[37501]: pgmap v7005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12397.028081 secs 2024-07-26T16:17:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:42 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:17:42.806 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:42 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:17:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:42 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:17:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:43 smithi107 ceph-mon[32659]: pgmap v7006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:43 smithi132 ceph-mon[37580]: pgmap v7006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:43.824 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:43 smithi112 ceph-mon[37501]: pgmap v7006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:17:45.570 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:45 smithi107 ceph-mon[32659]: pgmap v7007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:45.606 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:45 smithi112 ceph-mon[37501]: pgmap v7007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:45 smithi132 ceph-mon[37580]: pgmap v7007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12402.028211 secs 2024-07-26T16:17:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:46.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:46 smithi112 ceph-mon[37501]: pgmap v7008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12402.028211 secs 2024-07-26T16:17:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:46.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:46 smithi107 ceph-mon[32659]: pgmap v7008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:47.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:47.053+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:17:47.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:47.053+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:17:47.056 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:17:47.053+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12402.028211 secs 2024-07-26T16:17:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:47.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:47.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:46 smithi132 ceph-mon[37580]: pgmap v7008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: pgmap v7009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:17:49.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:17:49.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:17:49.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:17:49.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: pgmap v7010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:17:49.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: pgmap v7011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:17:49.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:49.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: pgmap v7009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:49.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:49.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:49.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:17:49.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:17:49.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:17:49.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:17:49.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: pgmap v7010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:17:49.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: pgmap v7011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:17:49.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: pgmap v7009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:17:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:49.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:17:49.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:17:49.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:17:49.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:17:49.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:17:49.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: pgmap v7010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:17:49.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: pgmap v7011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:17:49.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:17:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:51 smithi112 ceph-mon[37501]: pgmap v7012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:17:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12407.028296 secs 2024-07-26T16:17:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:51 smithi107 ceph-mon[32659]: pgmap v7012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:17:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12407.028296 secs 2024-07-26T16:17:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:51 smithi132 ceph-mon[37580]: pgmap v7012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:17:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12407.028296 secs 2024-07-26T16:17:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:53 smithi112 ceph-mon[37501]: pgmap v7013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T16:17:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:53 smithi107 ceph-mon[32659]: pgmap v7013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T16:17:53.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:53 smithi132 ceph-mon[37580]: pgmap v7013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2024-07-26T16:17:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:55 smithi112 ceph-mon[37501]: pgmap v7014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:17:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:55 smithi107 ceph-mon[32659]: pgmap v7014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:17:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:55 smithi132 ceph-mon[37580]: pgmap v7014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:17:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12412.028415 secs 2024-07-26T16:17:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12412.028415 secs 2024-07-26T16:17:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12412.028415 secs 2024-07-26T16:17:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:57 smithi112 ceph-mon[37501]: pgmap v7015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T16:17:57.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:57 smithi132 ceph-mon[37580]: pgmap v7015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T16:17:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:57 smithi107 ceph-mon[32659]: pgmap v7015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T16:17:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:17:59 smithi112 ceph-mon[37501]: pgmap v7016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:17:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:17:59 smithi107 ceph-mon[32659]: pgmap v7016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:17:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:17:59 smithi132 ceph-mon[37580]: pgmap v7016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:18:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:01 smithi112 ceph-mon[37501]: pgmap v7017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12417.028492 secs 2024-07-26T16:18:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:01 smithi107 ceph-mon[32659]: pgmap v7017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12417.028492 secs 2024-07-26T16:18:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:01 smithi132 ceph-mon[37580]: pgmap v7017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12417.028492 secs 2024-07-26T16:18:02.070 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:02.068+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:18:02.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:02.068+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:18:02.071 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:02.068+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:03 smithi112 ceph-mon[37501]: pgmap v7018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:03.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:03 smithi107 ceph-mon[32659]: pgmap v7018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:03 smithi132 ceph-mon[37580]: pgmap v7018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:05 smithi112 ceph-mon[37501]: pgmap v7019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:18:05.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:05 smithi107 ceph-mon[32659]: pgmap v7019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:18:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:05 smithi132 ceph-mon[37580]: pgmap v7019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:18:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12422.028645 secs 2024-07-26T16:18:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12422.028645 secs 2024-07-26T16:18:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12422.028645 secs 2024-07-26T16:18:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:07 smithi112 ceph-mon[37501]: pgmap v7020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:18:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:07 smithi107 ceph-mon[32659]: pgmap v7020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:18:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:07 smithi132 ceph-mon[37580]: pgmap v7020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:18:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:09 smithi112 ceph-mon[37501]: pgmap v7021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:18:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:09 smithi107 ceph-mon[32659]: pgmap v7021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:18:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:09 smithi132 ceph-mon[37580]: pgmap v7021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:18:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:11 smithi112 ceph-mon[37501]: pgmap v7022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12427.028702 secs 2024-07-26T16:18:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:11 smithi107 ceph-mon[32659]: pgmap v7022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12427.028702 secs 2024-07-26T16:18:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:11 smithi132 ceph-mon[37580]: pgmap v7022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:11.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12427.028702 secs 2024-07-26T16:18:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:13 smithi112 ceph-mon[37501]: pgmap v7023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:13 smithi107 ceph-mon[32659]: pgmap v7023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:13 smithi132 ceph-mon[37580]: pgmap v7023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:14.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:14.776+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:18:14.779 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:14.776+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:18:14.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:14.781+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:18:14.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:14.782+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:18:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:15 smithi112 ceph-mon[37501]: pgmap v7024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:15 smithi107 ceph-mon[32659]: pgmap v7024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:15 smithi132 ceph-mon[37580]: pgmap v7024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12432.028830 secs 2024-07-26T16:18:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12432.028830 secs 2024-07-26T16:18:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12432.028830 secs 2024-07-26T16:18:17.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:17.084+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:18:17.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:17.084+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:18:17.087 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:17.084+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:17 smithi112 ceph-mon[37501]: pgmap v7025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:17 smithi107 ceph-mon[32659]: pgmap v7025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:17 smithi132 ceph-mon[37580]: pgmap v7025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:19 smithi112 ceph-mon[37501]: pgmap v7026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:19 smithi107 ceph-mon[32659]: pgmap v7026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:19 smithi132 ceph-mon[37580]: pgmap v7026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:21 smithi112 ceph-mon[37501]: pgmap v7027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12437.028947 secs 2024-07-26T16:18:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:21 smithi107 ceph-mon[32659]: pgmap v7027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12437.028947 secs 2024-07-26T16:18:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:21 smithi132 ceph-mon[37580]: pgmap v7027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12437.028947 secs 2024-07-26T16:18:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:23 smithi112 ceph-mon[37501]: pgmap v7028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:23.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:23 smithi107 ceph-mon[32659]: pgmap v7028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:23 smithi132 ceph-mon[37580]: pgmap v7028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:25 smithi112 ceph-mon[37501]: pgmap v7029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:25 smithi107 ceph-mon[32659]: pgmap v7029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:25 smithi132 ceph-mon[37580]: pgmap v7029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12442.029086 secs 2024-07-26T16:18:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12442.029086 secs 2024-07-26T16:18:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12442.029086 secs 2024-07-26T16:18:27.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:27 smithi132 ceph-mon[37580]: pgmap v7030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:27 smithi107 ceph-mon[32659]: pgmap v7030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:27 smithi112 ceph-mon[37501]: pgmap v7030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:29.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:29 smithi107 ceph-mon[32659]: pgmap v7031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:18:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:29 smithi132 ceph-mon[37580]: pgmap v7031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:18:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:29 smithi112 ceph-mon[37501]: pgmap v7031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:18:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:31 smithi107 ceph-mon[32659]: pgmap v7032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:18:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12447.029187 secs 2024-07-26T16:18:31.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:31 smithi132 ceph-mon[37580]: pgmap v7032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:18:31.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12447.029187 secs 2024-07-26T16:18:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:31 smithi112 ceph-mon[37501]: pgmap v7032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:18:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12447.029187 secs 2024-07-26T16:18:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:32.094+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:18:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:32.094+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:18:32.096 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:32.094+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:33 smithi107 ceph-mon[32659]: pgmap v7033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:18:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:33 smithi132 ceph-mon[37580]: pgmap v7033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:18:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:33 smithi112 ceph-mon[37501]: pgmap v7033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:18:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:35 smithi107 ceph-mon[32659]: pgmap v7034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:18:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:35 smithi132 ceph-mon[37580]: pgmap v7034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:18:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:35 smithi112 ceph-mon[37501]: pgmap v7034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:18:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12452.029307 secs 2024-07-26T16:18:36.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12452.029307 secs 2024-07-26T16:18:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12452.029307 secs 2024-07-26T16:18:37.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:37 smithi107 ceph-mon[32659]: pgmap v7035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:18:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:37 smithi132 ceph-mon[37580]: pgmap v7035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:18:37.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:37 smithi112 ceph-mon[37501]: pgmap v7035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:18:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:39 smithi107 ceph-mon[32659]: pgmap v7036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:18:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:39 smithi132 ceph-mon[37580]: pgmap v7036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:18:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:39 smithi112 ceph-mon[37501]: pgmap v7036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:18:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:41 smithi107 ceph-mon[32659]: pgmap v7037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12457.029401 secs 2024-07-26T16:18:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:41 smithi132 ceph-mon[37580]: pgmap v7037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:41.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12457.029401 secs 2024-07-26T16:18:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:41 smithi112 ceph-mon[37501]: pgmap v7037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12457.029401 secs 2024-07-26T16:18:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:43 smithi107 ceph-mon[32659]: pgmap v7038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:43 smithi132 ceph-mon[37580]: pgmap v7038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:43 smithi112 ceph-mon[37501]: pgmap v7038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:45.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:45 smithi107 ceph-mon[32659]: pgmap v7039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:45 smithi132 ceph-mon[37580]: pgmap v7039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:45 smithi112 ceph-mon[37501]: pgmap v7039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12462.029524 secs 2024-07-26T16:18:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12462.029524 secs 2024-07-26T16:18:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12462.029524 secs 2024-07-26T16:18:47.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:47.110+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:18:47.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:47.110+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:18:47.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:18:47.110+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:47 smithi107 ceph-mon[32659]: pgmap v7040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:47 smithi132 ceph-mon[37580]: pgmap v7040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:47 smithi112 ceph-mon[37501]: pgmap v7040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:48 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:18:48.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:48 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:18:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:48 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:18:49.699 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: pgmap v7041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:49.699 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.700 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.700 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.700 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.700 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.700 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.700 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.700 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.700 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:18:49.701 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:49 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:18:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: pgmap v7041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:49.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:18:49.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:49 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:18:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: pgmap v7041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:18:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:50.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:50.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:50.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:50.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:50.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:50.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:50.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:18:50.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:49 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:18:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:50 smithi132 ceph-mon[37580]: pgmap v7042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:50 smithi112 ceph-mon[37501]: pgmap v7042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:50 smithi107 ceph-mon[32659]: pgmap v7042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:51.897 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12467.029683 secs 2024-07-26T16:18:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12467.029683 secs 2024-07-26T16:18:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12467.029683 secs 2024-07-26T16:18:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:18:53.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: pgmap v7043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:53.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:18:53.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:18:53.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:18:53.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:18:53.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:18:53.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: pgmap v7044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:18:53.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: pgmap v7045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:18:53.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:18:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:18:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: pgmap v7043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:18:53.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:18:53.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:18:53.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:18:53.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:18:53.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: pgmap v7044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:18:53.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: pgmap v7045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:18:53.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:18:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:18:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: pgmap v7043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:18:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:18:53.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:53.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:18:53.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:18:53.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:18:53.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:18:53.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: pgmap v7044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:18:53.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: pgmap v7045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:18:53.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:18:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:55 smithi112 ceph-mon[37501]: pgmap v7046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:55 smithi107 ceph-mon[32659]: pgmap v7046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:55 smithi132 ceph-mon[37580]: pgmap v7046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12472.029762 secs 2024-07-26T16:18:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12472.029762 secs 2024-07-26T16:18:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12472.029762 secs 2024-07-26T16:18:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:57 smithi112 ceph-mon[37501]: pgmap v7047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:18:57.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:57 smithi132 ceph-mon[37580]: pgmap v7047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:18:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:57 smithi107 ceph-mon[32659]: pgmap v7047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:18:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:18:59 smithi112 ceph-mon[37501]: pgmap v7048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:18:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:18:59 smithi107 ceph-mon[32659]: pgmap v7048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:18:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:18:59 smithi132 ceph-mon[37580]: pgmap v7048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:19:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:01 smithi112 ceph-mon[37501]: pgmap v7049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12477.029896 secs 2024-07-26T16:19:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:01 smithi107 ceph-mon[32659]: pgmap v7049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12477.029896 secs 2024-07-26T16:19:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:01 smithi132 ceph-mon[37580]: pgmap v7049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12477.029896 secs 2024-07-26T16:19:02.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:02.125+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:19:02.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:02.125+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:19:02.127 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:02.125+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:03 smithi112 ceph-mon[37501]: pgmap v7050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:19:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:03 smithi107 ceph-mon[32659]: pgmap v7050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:19:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:03 smithi132 ceph-mon[37580]: pgmap v7050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:19:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:05 smithi112 ceph-mon[37501]: pgmap v7051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:05 smithi107 ceph-mon[32659]: pgmap v7051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:05 smithi132 ceph-mon[37580]: pgmap v7051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12482.029958 secs 2024-07-26T16:19:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12482.029958 secs 2024-07-26T16:19:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12482.029958 secs 2024-07-26T16:19:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:07 smithi107 ceph-mon[32659]: pgmap v7052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:07 smithi132 ceph-mon[37580]: pgmap v7052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:07 smithi112 ceph-mon[37501]: pgmap v7052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:09 smithi107 ceph-mon[32659]: pgmap v7053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:09 smithi132 ceph-mon[37580]: pgmap v7053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:09 smithi112 ceph-mon[37501]: pgmap v7053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:11 smithi107 ceph-mon[32659]: pgmap v7054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12487.030101 secs 2024-07-26T16:19:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:11 smithi132 ceph-mon[37580]: pgmap v7054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12487.030101 secs 2024-07-26T16:19:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:11 smithi112 ceph-mon[37501]: pgmap v7054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12487.030101 secs 2024-07-26T16:19:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:13 smithi107 ceph-mon[32659]: pgmap v7055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:13 smithi132 ceph-mon[37580]: pgmap v7055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:13 smithi112 ceph-mon[37501]: pgmap v7055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:15 smithi107 ceph-mon[32659]: pgmap v7056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:15 smithi132 ceph-mon[37580]: pgmap v7056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:15 smithi112 ceph-mon[37501]: pgmap v7056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12492.030213 secs 2024-07-26T16:19:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12492.030213 secs 2024-07-26T16:19:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12492.030213 secs 2024-07-26T16:19:17.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:17.140+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:19:17.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:17.140+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:17.143 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:17.140+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:19:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:17 smithi107 ceph-mon[32659]: pgmap v7057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:17 smithi132 ceph-mon[37580]: pgmap v7057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:17 smithi112 ceph-mon[37501]: pgmap v7057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:19 smithi107 ceph-mon[32659]: pgmap v7058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:19 smithi132 ceph-mon[37580]: pgmap v7058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:19 smithi112 ceph-mon[37501]: pgmap v7058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:21 smithi107 ceph-mon[32659]: pgmap v7059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12497.030331 secs 2024-07-26T16:19:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:21 smithi132 ceph-mon[37580]: pgmap v7059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12497.030331 secs 2024-07-26T16:19:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:21 smithi112 ceph-mon[37501]: pgmap v7059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12497.030331 secs 2024-07-26T16:19:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:23 smithi107 ceph-mon[32659]: pgmap v7060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:23 smithi132 ceph-mon[37580]: pgmap v7060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:23 smithi112 ceph-mon[37501]: pgmap v7060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:25 smithi107 ceph-mon[32659]: pgmap v7061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:25 smithi132 ceph-mon[37580]: pgmap v7061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:25 smithi112 ceph-mon[37501]: pgmap v7061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12502.030446 secs 2024-07-26T16:19:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12502.030446 secs 2024-07-26T16:19:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12502.030446 secs 2024-07-26T16:19:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:27 smithi107 ceph-mon[32659]: pgmap v7062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:27 smithi132 ceph-mon[37580]: pgmap v7062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:27 smithi112 ceph-mon[37501]: pgmap v7062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:29 smithi107 ceph-mon[32659]: pgmap v7063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:29 smithi132 ceph-mon[37580]: pgmap v7063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:29 smithi112 ceph-mon[37501]: pgmap v7063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:31 smithi132 ceph-mon[37580]: pgmap v7064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:31.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12507.030528 secs 2024-07-26T16:19:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:31 smithi112 ceph-mon[37501]: pgmap v7064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12507.030528 secs 2024-07-26T16:19:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:31 smithi107 ceph-mon[32659]: pgmap v7064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12507.030528 secs 2024-07-26T16:19:32.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:32.146+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:19:32.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:32.146+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:32.148 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:32.146+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:19:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:33 smithi132 ceph-mon[37580]: pgmap v7065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:33 smithi112 ceph-mon[37501]: pgmap v7065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:33 smithi107 ceph-mon[32659]: pgmap v7065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:35 smithi132 ceph-mon[37580]: pgmap v7066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:35 smithi112 ceph-mon[37501]: pgmap v7066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:36.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:35 smithi107 ceph-mon[32659]: pgmap v7066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12512.030649 secs 2024-07-26T16:19:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:36 smithi112 ceph-mon[37501]: pgmap v7067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12512.030649 secs 2024-07-26T16:19:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:36 smithi107 ceph-mon[32659]: pgmap v7067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12512.030649 secs 2024-07-26T16:19:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:36 smithi132 ceph-mon[37580]: pgmap v7067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:39 smithi112 ceph-mon[37501]: pgmap v7068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:39 smithi107 ceph-mon[32659]: pgmap v7068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:39 smithi132 ceph-mon[37580]: pgmap v7068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12517.030758 secs 2024-07-26T16:19:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:41 smithi112 ceph-mon[37501]: pgmap v7069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12517.030758 secs 2024-07-26T16:19:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:41 smithi107 ceph-mon[32659]: pgmap v7069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12517.030758 secs 2024-07-26T16:19:41.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:41 smithi132 ceph-mon[37580]: pgmap v7069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:43 smithi112 ceph-mon[37501]: pgmap v7070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:43 smithi107 ceph-mon[32659]: pgmap v7070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:43 smithi132 ceph-mon[37580]: pgmap v7070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:45 smithi112 ceph-mon[37501]: pgmap v7071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:45 smithi107 ceph-mon[32659]: pgmap v7071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:45 smithi132 ceph-mon[37580]: pgmap v7071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:19:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12522.030879 secs 2024-07-26T16:19:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12522.030879 secs 2024-07-26T16:19:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12522.030879 secs 2024-07-26T16:19:47.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:47.162+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:19:47.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:47.162+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:47.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:19:47.162+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:19:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:47 smithi107 ceph-mon[32659]: pgmap v7072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:47.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:47 smithi132 ceph-mon[37580]: pgmap v7072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:47 smithi112 ceph-mon[37501]: pgmap v7072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:49.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:49 smithi107 ceph-mon[32659]: pgmap v7073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:49 smithi132 ceph-mon[37580]: pgmap v7073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:49 smithi112 ceph-mon[37501]: pgmap v7073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12527.030970 secs 2024-07-26T16:19:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:51 smithi107 ceph-mon[32659]: pgmap v7074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12527.030970 secs 2024-07-26T16:19:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:51 smithi132 ceph-mon[37580]: pgmap v7074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12527.030970 secs 2024-07-26T16:19:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:51 smithi112 ceph-mon[37501]: pgmap v7074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:53 smithi107 ceph-mon[32659]: pgmap v7075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:19:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:53 smithi132 ceph-mon[37580]: pgmap v7075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:53.814 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:19:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:53 smithi112 ceph-mon[37501]: pgmap v7075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:19:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:19:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:19:54.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:19:54.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:19:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:19:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:19:54.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:19:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:19:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:19:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:19:55.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:55 smithi107 ceph-mon[32659]: pgmap v7076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:19:55.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:55 smithi107 ceph-mon[32659]: pgmap v7077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:19:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:55 smithi132 ceph-mon[37580]: pgmap v7076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:19:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:55 smithi132 ceph-mon[37580]: pgmap v7077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:19:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:55 smithi112 ceph-mon[37501]: pgmap v7076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:19:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:55 smithi112 ceph-mon[37501]: pgmap v7077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:19:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12532.031098 secs 2024-07-26T16:19:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12532.031098 secs 2024-07-26T16:19:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12532.031098 secs 2024-07-26T16:19:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:57 smithi107 ceph-mon[32659]: pgmap v7078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:19:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:57 smithi132 ceph-mon[37580]: pgmap v7078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:19:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:57 smithi112 ceph-mon[37501]: pgmap v7078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:19:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:19:59 smithi107 ceph-mon[32659]: pgmap v7079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:19:59.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:19:59 smithi132 ceph-mon[37580]: pgmap v7079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:19:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:19:59 smithi112 ceph-mon[37501]: pgmap v7079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:20:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:20:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:20:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:20:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:01 smithi107 ceph-mon[32659]: pgmap v7080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12537.031218 secs 2024-07-26T16:20:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:01 smithi132 ceph-mon[37580]: pgmap v7080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12537.031218 secs 2024-07-26T16:20:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:01 smithi112 ceph-mon[37501]: pgmap v7080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12537.031218 secs 2024-07-26T16:20:02.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:02.176+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:20:02.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:02.176+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:02.179 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:02.176+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:20:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:03 smithi107 ceph-mon[32659]: pgmap v7081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:03 smithi132 ceph-mon[37580]: pgmap v7081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:03 smithi112 ceph-mon[37501]: pgmap v7081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:05 smithi107 ceph-mon[32659]: pgmap v7082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:20:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:05 smithi132 ceph-mon[37580]: pgmap v7082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:20:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:05 smithi112 ceph-mon[37501]: pgmap v7082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:20:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12542.031330 secs 2024-07-26T16:20:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12542.031330 secs 2024-07-26T16:20:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12542.031330 secs 2024-07-26T16:20:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:07 smithi132 ceph-mon[37580]: pgmap v7083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:07.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:07 smithi107 ceph-mon[32659]: pgmap v7083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:07 smithi112 ceph-mon[37501]: pgmap v7083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:09 smithi132 ceph-mon[37580]: pgmap v7084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:09 smithi112 ceph-mon[37501]: pgmap v7084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:09 smithi107 ceph-mon[32659]: pgmap v7084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:11 smithi112 ceph-mon[37501]: pgmap v7085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12547.031464 secs 2024-07-26T16:20:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:11 smithi107 ceph-mon[32659]: pgmap v7085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12547.031464 secs 2024-07-26T16:20:12.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:11 smithi132 ceph-mon[37580]: pgmap v7085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:12.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12547.031464 secs 2024-07-26T16:20:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:12 smithi112 ceph-mon[37501]: pgmap v7086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:12 smithi107 ceph-mon[32659]: pgmap v7086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:12 smithi132 ceph-mon[37580]: pgmap v7086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:14 smithi112 ceph-mon[37501]: pgmap v7087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:14 smithi107 ceph-mon[32659]: pgmap v7087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:14 smithi132 ceph-mon[37580]: pgmap v7087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12552.031582 secs 2024-07-26T16:20:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:16 smithi112 ceph-mon[37501]: pgmap v7088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12552.031582 secs 2024-07-26T16:20:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:16 smithi107 ceph-mon[32659]: pgmap v7088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12552.031582 secs 2024-07-26T16:20:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:16 smithi132 ceph-mon[37580]: pgmap v7088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:17.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:17.177+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:20:17.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:17.177+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:17.183 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:17.177+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:20:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:18 smithi112 ceph-mon[37501]: pgmap v7089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:18 smithi107 ceph-mon[32659]: pgmap v7089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:18 smithi132 ceph-mon[37580]: pgmap v7089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:20 smithi112 ceph-mon[37501]: pgmap v7090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:20 smithi107 ceph-mon[32659]: pgmap v7090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:20 smithi132 ceph-mon[37580]: pgmap v7090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12557.031704 secs 2024-07-26T16:20:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12557.031704 secs 2024-07-26T16:20:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12557.031704 secs 2024-07-26T16:20:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:22 smithi112 ceph-mon[37501]: pgmap v7091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:22 smithi107 ceph-mon[32659]: pgmap v7091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:22 smithi132 ceph-mon[37580]: pgmap v7091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:24 smithi112 ceph-mon[37501]: pgmap v7092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:24 smithi107 ceph-mon[32659]: pgmap v7092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:24 smithi132 ceph-mon[37580]: pgmap v7092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12562.031828 secs 2024-07-26T16:20:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:26 smithi112 ceph-mon[37501]: pgmap v7093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12562.031828 secs 2024-07-26T16:20:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:26 smithi107 ceph-mon[32659]: pgmap v7093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12562.031828 secs 2024-07-26T16:20:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:26 smithi132 ceph-mon[37580]: pgmap v7093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:28 smithi112 ceph-mon[37501]: pgmap v7094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:29.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:28 smithi107 ceph-mon[32659]: pgmap v7094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:28 smithi132 ceph-mon[37580]: pgmap v7094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:30 smithi112 ceph-mon[37501]: pgmap v7095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:30 smithi107 ceph-mon[32659]: pgmap v7095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:30 smithi132 ceph-mon[37580]: pgmap v7095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:31.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12567.031926 secs 2024-07-26T16:20:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12567.031926 secs 2024-07-26T16:20:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12567.031926 secs 2024-07-26T16:20:32.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:32.192+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:20:32.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:32.192+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:20:32.194 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:32.192+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:32 smithi112 ceph-mon[37501]: pgmap v7096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:32 smithi107 ceph-mon[32659]: pgmap v7096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:32 smithi132 ceph-mon[37580]: pgmap v7096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:34 smithi112 ceph-mon[37501]: pgmap v7097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:34 smithi107 ceph-mon[32659]: pgmap v7097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:34 smithi132 ceph-mon[37580]: pgmap v7097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12572.032049 secs 2024-07-26T16:20:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:36 smithi112 ceph-mon[37501]: pgmap v7098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12572.032049 secs 2024-07-26T16:20:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:36 smithi107 ceph-mon[32659]: pgmap v7098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12572.032049 secs 2024-07-26T16:20:37.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:36 smithi132 ceph-mon[37580]: pgmap v7098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:38 smithi112 ceph-mon[37501]: pgmap v7099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:38 smithi107 ceph-mon[32659]: pgmap v7099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:38 smithi132 ceph-mon[37580]: pgmap v7099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:40 smithi112 ceph-mon[37501]: pgmap v7100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:40 smithi107 ceph-mon[32659]: pgmap v7100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:40 smithi132 ceph-mon[37580]: pgmap v7100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12577.032141 secs 2024-07-26T16:20:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12577.032141 secs 2024-07-26T16:20:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12577.032141 secs 2024-07-26T16:20:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:42 smithi112 ceph-mon[37501]: pgmap v7101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:42 smithi107 ceph-mon[32659]: pgmap v7101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:42 smithi132 ceph-mon[37580]: pgmap v7101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:44 smithi112 ceph-mon[37501]: pgmap v7102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:20:45.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:44 smithi107 ceph-mon[32659]: pgmap v7102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:20:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:44 smithi132 ceph-mon[37580]: pgmap v7102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:20:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12582.032253 secs 2024-07-26T16:20:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:46 smithi112 ceph-mon[37501]: pgmap v7103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:20:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12582.032253 secs 2024-07-26T16:20:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:46 smithi107 ceph-mon[32659]: pgmap v7103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:20:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12582.032253 secs 2024-07-26T16:20:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:46 smithi132 ceph-mon[37580]: pgmap v7103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:20:47.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:47.207+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:20:47.209 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:47.207+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:20:47.210 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:20:47.207+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:49.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:48 smithi107 ceph-mon[32659]: pgmap v7104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:20:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:48 smithi132 ceph-mon[37580]: pgmap v7104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:20:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:48 smithi112 ceph-mon[37501]: pgmap v7104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:20:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:50 smithi107 ceph-mon[32659]: pgmap v7105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:50 smithi132 ceph-mon[37580]: pgmap v7105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:50 smithi112 ceph-mon[37501]: pgmap v7105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:20:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12587.032370 secs 2024-07-26T16:20:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12587.032370 secs 2024-07-26T16:20:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12587.032370 secs 2024-07-26T16:20:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:52 smithi107 ceph-mon[32659]: pgmap v7106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:20:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:52 smithi132 ceph-mon[37580]: pgmap v7106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:20:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:52 smithi112 ceph-mon[37501]: pgmap v7106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:20:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:54 smithi107 ceph-mon[32659]: pgmap v7107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:20:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:20:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:20:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:20:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:54 smithi132 ceph-mon[37580]: pgmap v7107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:20:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:20:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:20:55.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:20:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:54 smithi112 ceph-mon[37501]: pgmap v7107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:20:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:20:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:20:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:20:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:20:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:20:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:55 smithi107 ceph-mon[32659]: pgmap v7108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:20:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:55 smithi107 ceph-mon[32659]: pgmap v7109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:20:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:20:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:20:56.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:20:56.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:55 smithi132 ceph-mon[37580]: pgmap v7108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:20:56.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:55 smithi132 ceph-mon[37580]: pgmap v7109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:20:56.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:20:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:20:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:20:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:55 smithi112 ceph-mon[37501]: pgmap v7108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:20:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:55 smithi112 ceph-mon[37501]: pgmap v7109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:20:56.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:20:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12592.032504 secs 2024-07-26T16:20:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12592.032504 secs 2024-07-26T16:20:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12592.032504 secs 2024-07-26T16:20:58.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:20:57 smithi107 ceph-mon[32659]: pgmap v7110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:20:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:20:57 smithi132 ceph-mon[37580]: pgmap v7110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:20:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:20:57 smithi112 ceph-mon[37501]: pgmap v7110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:21:00.764 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:00 smithi132 ceph-mon[37580]: pgmap v7111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:00 smithi107 ceph-mon[32659]: pgmap v7111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:00 smithi112 ceph-mon[37501]: pgmap v7111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12597.032620 secs 2024-07-26T16:21:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12597.032620 secs 2024-07-26T16:21:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12597.032620 secs 2024-07-26T16:21:02.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:02.213+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:21:02.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:02.213+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:21:02.217 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:02.213+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:02 smithi107 ceph-mon[32659]: pgmap v7112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:02 smithi132 ceph-mon[37580]: pgmap v7112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:02 smithi112 ceph-mon[37501]: pgmap v7112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:04 smithi107 ceph-mon[32659]: pgmap v7113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:04 smithi132 ceph-mon[37580]: pgmap v7113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:04 smithi112 ceph-mon[37501]: pgmap v7113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12602.032738 secs 2024-07-26T16:21:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:06 smithi107 ceph-mon[32659]: pgmap v7114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:21:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12602.032738 secs 2024-07-26T16:21:06.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:06 smithi132 ceph-mon[37580]: pgmap v7114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:21:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12602.032738 secs 2024-07-26T16:21:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:06 smithi112 ceph-mon[37501]: pgmap v7114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:21:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:08 smithi107 ceph-mon[32659]: pgmap v7115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:08 smithi132 ceph-mon[37580]: pgmap v7115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:08 smithi112 ceph-mon[37501]: pgmap v7115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:10.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:10 smithi112 ceph-mon[37501]: pgmap v7116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:10 smithi107 ceph-mon[32659]: pgmap v7116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:10 smithi132 ceph-mon[37580]: pgmap v7116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:11.676+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T16:21:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:11.676+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T16:21:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:11.676+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f806f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:21:11.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:11.676+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f806f040 comp rx=0 tx=0).stop 2024-07-26T16:21:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:11.676+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c45e0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T16:21:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:11.676+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c45e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:21:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:11.676+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c45e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:21:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:11.676+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:21:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:11.677+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:21:11.681 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:11.679+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8014050 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:21:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12607.032802 secs 2024-07-26T16:21:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12607.032802 secs 2024-07-26T16:21:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12607.032802 secs 2024-07-26T16:21:12.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:12 smithi107 ceph-mon[32659]: pgmap v7117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:12 smithi132 ceph-mon[37580]: pgmap v7117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:12 smithi112 ceph-mon[37501]: pgmap v7117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:14.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:14 smithi107 ceph-mon[32659]: pgmap v7118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:14.810 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:14.807+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:21:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:14 smithi132 ceph-mon[37580]: pgmap v7118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:14.808+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:21:14.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:14.810+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:21:14.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:14.811+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:21:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:14 smithi112 ceph-mon[37501]: pgmap v7118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12612.032957 secs 2024-07-26T16:21:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:16 smithi107 ceph-mon[32659]: pgmap v7119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12612.032957 secs 2024-07-26T16:21:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:16 smithi132 ceph-mon[37580]: pgmap v7119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12612.032957 secs 2024-07-26T16:21:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:16 smithi112 ceph-mon[37501]: pgmap v7119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:17.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:17.217+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:21:17.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:17.217+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:21:17.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:17.217+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:18 smithi107 ceph-mon[32659]: pgmap v7120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:18 smithi132 ceph-mon[37580]: pgmap v7120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:18 smithi112 ceph-mon[37501]: pgmap v7120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:20 smithi107 ceph-mon[32659]: pgmap v7121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:20 smithi132 ceph-mon[37580]: pgmap v7121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:20 smithi112 ceph-mon[37501]: pgmap v7121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12617.033095 secs 2024-07-26T16:21:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12617.033095 secs 2024-07-26T16:21:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12617.033095 secs 2024-07-26T16:21:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:22 smithi132 ceph-mon[37580]: pgmap v7122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:22 smithi112 ceph-mon[37501]: pgmap v7122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:22 smithi107 ceph-mon[32659]: pgmap v7122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:24 smithi112 ceph-mon[37501]: pgmap v7123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:24 smithi107 ceph-mon[32659]: pgmap v7123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:25.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:24 smithi132 ceph-mon[37580]: pgmap v7123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12622.033153 secs 2024-07-26T16:21:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:26 smithi112 ceph-mon[37501]: pgmap v7124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12622.033153 secs 2024-07-26T16:21:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:26 smithi107 ceph-mon[32659]: pgmap v7124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12622.033153 secs 2024-07-26T16:21:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:26 smithi132 ceph-mon[37580]: pgmap v7124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:28 smithi112 ceph-mon[37501]: pgmap v7125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:28 smithi107 ceph-mon[32659]: pgmap v7125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:28 smithi132 ceph-mon[37580]: pgmap v7125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:30.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:30 smithi132 ceph-mon[37580]: pgmap v7126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:30 smithi112 ceph-mon[37501]: pgmap v7126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:30 smithi107 ceph-mon[32659]: pgmap v7126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:31.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12627.033265 secs 2024-07-26T16:21:31.945 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:31 smithi107 ceph-mon[32659]: pgmap v7127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12627.033265 secs 2024-07-26T16:21:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:31 smithi112 ceph-mon[37501]: pgmap v7127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12627.033265 secs 2024-07-26T16:21:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:31 smithi132 ceph-mon[37580]: pgmap v7127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:32.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:32.234+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:21:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:32.234+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:21:32.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:32.234+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:34 smithi107 ceph-mon[32659]: pgmap v7128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:34 smithi132 ceph-mon[37580]: pgmap v7128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:34 smithi112 ceph-mon[37501]: pgmap v7128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12632.033386 secs 2024-07-26T16:21:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:36 smithi107 ceph-mon[32659]: pgmap v7129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12632.033386 secs 2024-07-26T16:21:36.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:36 smithi132 ceph-mon[37580]: pgmap v7129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12632.033386 secs 2024-07-26T16:21:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:36 smithi112 ceph-mon[37501]: pgmap v7129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:38.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:38 smithi107 ceph-mon[32659]: pgmap v7130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:38 smithi132 ceph-mon[37580]: pgmap v7130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:38 smithi112 ceph-mon[37501]: pgmap v7130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:40.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:40 smithi112 ceph-mon[37501]: pgmap v7131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:40 smithi107 ceph-mon[32659]: pgmap v7131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:40 smithi132 ceph-mon[37580]: pgmap v7131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12637.033516 secs 2024-07-26T16:21:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12637.033516 secs 2024-07-26T16:21:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12637.033516 secs 2024-07-26T16:21:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:42 smithi107 ceph-mon[32659]: pgmap v7132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:42 smithi132 ceph-mon[37580]: pgmap v7132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:42 smithi112 ceph-mon[37501]: pgmap v7132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:44 smithi132 ceph-mon[37580]: pgmap v7133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:44.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:44 smithi112 ceph-mon[37501]: pgmap v7133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:45.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:44 smithi107 ceph-mon[32659]: pgmap v7133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12642.033595 secs 2024-07-26T16:21:46.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:46 smithi132 ceph-mon[37580]: pgmap v7134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12642.033595 secs 2024-07-26T16:21:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:46 smithi112 ceph-mon[37501]: pgmap v7134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12642.033595 secs 2024-07-26T16:21:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:46 smithi107 ceph-mon[32659]: pgmap v7134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:47.249+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:47.249+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:47.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:21:47.249+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:21:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:48 smithi132 ceph-mon[37580]: pgmap v7135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:48 smithi112 ceph-mon[37501]: pgmap v7135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:49.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:48 smithi107 ceph-mon[32659]: pgmap v7135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:50 smithi132 ceph-mon[37580]: pgmap v7136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:50 smithi112 ceph-mon[37501]: pgmap v7136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:50 smithi107 ceph-mon[32659]: pgmap v7136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:21:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12647.033736 secs 2024-07-26T16:21:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12647.033736 secs 2024-07-26T16:21:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12647.033736 secs 2024-07-26T16:21:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:52 smithi112 ceph-mon[37501]: pgmap v7137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:52 smithi107 ceph-mon[32659]: pgmap v7137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:52 smithi132 ceph-mon[37580]: pgmap v7137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:54 smithi112 ceph-mon[37501]: pgmap v7138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:54 smithi107 ceph-mon[32659]: pgmap v7138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:54 smithi132 ceph-mon[37580]: pgmap v7138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:56.853 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12652.033862 secs 2024-07-26T16:21:56.853 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:56 smithi112 ceph-mon[37501]: pgmap v7139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:56.854 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:21:56.854 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.854 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.854 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.854 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12652.033862 secs 2024-07-26T16:21:56.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:56 smithi107 ceph-mon[32659]: pgmap v7139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:56.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:21:56.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.896 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12652.033862 secs 2024-07-26T16:21:56.896 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:56 smithi132 ceph-mon[37580]: pgmap v7139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:56.896 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:21:56.896 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.896 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.896 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:56.897 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:58 smithi107 ceph-mon[32659]: pgmap v7140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:58.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:21:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:58 smithi132 ceph-mon[37580]: pgmap v7140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:21:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:58 smithi112 ceph-mon[37501]: pgmap v7140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:21:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:21:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:21:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:22:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:22:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:22:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:00 smithi107 ceph-mon[32659]: pgmap v7141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:22:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:22:00.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:22:00.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:00 smithi132 ceph-mon[37580]: pgmap v7141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:22:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:22:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:22:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:00 smithi112 ceph-mon[37501]: pgmap v7141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:22:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:22:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:22:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:01 smithi107 ceph-mon[32659]: pgmap v7142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:22:01.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:01 smithi107 ceph-mon[32659]: pgmap v7143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:22:01.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:22:01.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12657.033998 secs 2024-07-26T16:22:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:22:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:22:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:01 smithi132 ceph-mon[37580]: pgmap v7142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:22:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:01 smithi132 ceph-mon[37580]: pgmap v7143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:22:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:22:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12657.033998 secs 2024-07-26T16:22:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:22:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:22:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:01 smithi112 ceph-mon[37501]: pgmap v7142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:22:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:01 smithi112 ceph-mon[37501]: pgmap v7143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:22:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:22:01.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12657.033998 secs 2024-07-26T16:22:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:02.264+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:22:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:02.264+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:02.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:02.264+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:22:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:03 smithi112 ceph-mon[37501]: pgmap v7144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T16:22:03.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:03 smithi107 ceph-mon[32659]: pgmap v7144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T16:22:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:03 smithi132 ceph-mon[37580]: pgmap v7144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T16:22:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:05 smithi112 ceph-mon[37501]: pgmap v7145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:05 smithi107 ceph-mon[32659]: pgmap v7145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:05 smithi132 ceph-mon[37580]: pgmap v7145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12662.034135 secs 2024-07-26T16:22:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12662.034135 secs 2024-07-26T16:22:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12662.034135 secs 2024-07-26T16:22:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:07 smithi112 ceph-mon[37501]: pgmap v7146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T16:22:07.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:07 smithi107 ceph-mon[32659]: pgmap v7146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T16:22:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:07 smithi132 ceph-mon[37580]: pgmap v7146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T16:22:09.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:09 smithi112 ceph-mon[37501]: pgmap v7147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:22:09.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:09 smithi107 ceph-mon[32659]: pgmap v7147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:22:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:09 smithi132 ceph-mon[37580]: pgmap v7147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:22:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:11 smithi112 ceph-mon[37501]: pgmap v7148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:22:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12667.034213 secs 2024-07-26T16:22:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:11 smithi107 ceph-mon[32659]: pgmap v7148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:22:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12667.034213 secs 2024-07-26T16:22:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:11 smithi132 ceph-mon[37580]: pgmap v7148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:22:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12667.034213 secs 2024-07-26T16:22:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:13 smithi112 ceph-mon[37501]: pgmap v7149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:13 smithi107 ceph-mon[32659]: pgmap v7149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:13 smithi132 ceph-mon[37580]: pgmap v7149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:15 smithi112 ceph-mon[37501]: pgmap v7150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:15 smithi107 ceph-mon[32659]: pgmap v7150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:15 smithi132 ceph-mon[37580]: pgmap v7150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12672.034329 secs 2024-07-26T16:22:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12672.034329 secs 2024-07-26T16:22:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12672.034329 secs 2024-07-26T16:22:17.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:17.279+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:22:17.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:17.279+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:17.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:17.279+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:22:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:17 smithi107 ceph-mon[32659]: pgmap v7151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:17 smithi132 ceph-mon[37580]: pgmap v7151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:17 smithi112 ceph-mon[37501]: pgmap v7151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:19 smithi107 ceph-mon[32659]: pgmap v7152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:19 smithi132 ceph-mon[37580]: pgmap v7152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:19 smithi112 ceph-mon[37501]: pgmap v7152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:21 smithi107 ceph-mon[32659]: pgmap v7153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12677.034448 secs 2024-07-26T16:22:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:21 smithi132 ceph-mon[37580]: pgmap v7153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12677.034448 secs 2024-07-26T16:22:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:21 smithi112 ceph-mon[37501]: pgmap v7153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12677.034448 secs 2024-07-26T16:22:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:23 smithi107 ceph-mon[32659]: pgmap v7154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:23 smithi132 ceph-mon[37580]: pgmap v7154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:23 smithi112 ceph-mon[37501]: pgmap v7154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:25 smithi107 ceph-mon[32659]: pgmap v7155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:25 smithi132 ceph-mon[37580]: pgmap v7155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:25 smithi112 ceph-mon[37501]: pgmap v7155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12682.034571 secs 2024-07-26T16:22:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12682.034571 secs 2024-07-26T16:22:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12682.034571 secs 2024-07-26T16:22:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:27 smithi107 ceph-mon[32659]: pgmap v7156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:27.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:27 smithi132 ceph-mon[37580]: pgmap v7156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:27 smithi112 ceph-mon[37501]: pgmap v7156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:29 smithi107 ceph-mon[32659]: pgmap v7157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:29 smithi132 ceph-mon[37580]: pgmap v7157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:29 smithi112 ceph-mon[37501]: pgmap v7157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:31 smithi107 ceph-mon[32659]: pgmap v7158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12687.034683 secs 2024-07-26T16:22:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:31 smithi132 ceph-mon[37580]: pgmap v7158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12687.034683 secs 2024-07-26T16:22:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:31 smithi112 ceph-mon[37501]: pgmap v7158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12687.034683 secs 2024-07-26T16:22:32.297 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:32.296+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:22:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:32.296+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:22:32.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:32.296+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:33.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:33 smithi107 ceph-mon[32659]: pgmap v7159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:33 smithi132 ceph-mon[37580]: pgmap v7159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:33 smithi112 ceph-mon[37501]: pgmap v7159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:35 smithi107 ceph-mon[32659]: pgmap v7160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:35 smithi132 ceph-mon[37580]: pgmap v7160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:35 smithi112 ceph-mon[37501]: pgmap v7160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12692.034821 secs 2024-07-26T16:22:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12692.034821 secs 2024-07-26T16:22:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12692.034821 secs 2024-07-26T16:22:37.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:37 smithi107 ceph-mon[32659]: pgmap v7161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:37 smithi132 ceph-mon[37580]: pgmap v7161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:37 smithi112 ceph-mon[37501]: pgmap v7161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:39 smithi107 ceph-mon[32659]: pgmap v7162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:39 smithi132 ceph-mon[37580]: pgmap v7162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:39 smithi112 ceph-mon[37501]: pgmap v7162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:41 smithi132 ceph-mon[37580]: pgmap v7163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12697.034949 secs 2024-07-26T16:22:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:41 smithi112 ceph-mon[37501]: pgmap v7163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12697.034949 secs 2024-07-26T16:22:41.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:41 smithi107 ceph-mon[32659]: pgmap v7163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:41.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12697.034949 secs 2024-07-26T16:22:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:43 smithi132 ceph-mon[37580]: pgmap v7164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:43 smithi112 ceph-mon[37501]: pgmap v7164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:43 smithi107 ceph-mon[32659]: pgmap v7164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:45 smithi132 ceph-mon[37580]: pgmap v7165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:45 smithi112 ceph-mon[37501]: pgmap v7165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:45 smithi107 ceph-mon[32659]: pgmap v7165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12702.035094 secs 2024-07-26T16:22:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12702.035094 secs 2024-07-26T16:22:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12702.035094 secs 2024-07-26T16:22:47.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:47.311+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:22:47.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:47.311+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:47.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:22:47.311+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:22:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:47 smithi112 ceph-mon[37501]: pgmap v7166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:47 smithi107 ceph-mon[32659]: pgmap v7166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:47 smithi132 ceph-mon[37580]: pgmap v7166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:49 smithi112 ceph-mon[37501]: pgmap v7167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:49 smithi107 ceph-mon[32659]: pgmap v7167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:49 smithi132 ceph-mon[37580]: pgmap v7167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:51 smithi112 ceph-mon[37501]: pgmap v7168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12707.035210 secs 2024-07-26T16:22:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:51 smithi107 ceph-mon[32659]: pgmap v7168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12707.035210 secs 2024-07-26T16:22:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:51 smithi132 ceph-mon[37580]: pgmap v7168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12707.035210 secs 2024-07-26T16:22:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:53 smithi112 ceph-mon[37501]: pgmap v7169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:53 smithi107 ceph-mon[32659]: pgmap v7169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:53 smithi132 ceph-mon[37580]: pgmap v7169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:55 smithi112 ceph-mon[37501]: pgmap v7170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:55.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:55 smithi107 ceph-mon[32659]: pgmap v7170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:55 smithi132 ceph-mon[37580]: pgmap v7170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12712.035289 secs 2024-07-26T16:22:56.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12712.035289 secs 2024-07-26T16:22:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12712.035289 secs 2024-07-26T16:22:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:57 smithi112 ceph-mon[37501]: pgmap v7171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:57 smithi107 ceph-mon[32659]: pgmap v7171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:57 smithi132 ceph-mon[37580]: pgmap v7171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:22:59 smithi112 ceph-mon[37501]: pgmap v7172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:22:59 smithi107 ceph-mon[32659]: pgmap v7172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:22:59 smithi132 ceph-mon[37580]: pgmap v7172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:00.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:23:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:23:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:23:01.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:01 smithi107 ceph-mon[32659]: pgmap v7173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:01.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12717.035393 secs 2024-07-26T16:23:01.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:23:01.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:23:01.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:23:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:01 smithi112 ceph-mon[37501]: pgmap v7173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12717.035393 secs 2024-07-26T16:23:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:23:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:23:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:23:01.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:01 smithi132 ceph-mon[37580]: pgmap v7173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:01.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12717.035393 secs 2024-07-26T16:23:01.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:23:01.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:23:01.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:23:02.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:02.326+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:23:02.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:02.326+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:02.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:02.326+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:23:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:02 smithi112 ceph-mon[37501]: pgmap v7174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:23:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:02 smithi112 ceph-mon[37501]: pgmap v7175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:23:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:02 smithi107 ceph-mon[32659]: pgmap v7174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:23:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:02 smithi107 ceph-mon[32659]: pgmap v7175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:23:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:02 smithi132 ceph-mon[37580]: pgmap v7174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:23:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:02 smithi132 ceph-mon[37580]: pgmap v7175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:23:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:04 smithi112 ceph-mon[37501]: pgmap v7176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:23:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:04 smithi107 ceph-mon[32659]: pgmap v7176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:23:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:04 smithi132 ceph-mon[37580]: pgmap v7176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:23:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:06 smithi112 ceph-mon[37501]: pgmap v7177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:23:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12722.035498 secs 2024-07-26T16:23:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:06 smithi107 ceph-mon[32659]: pgmap v7177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:23:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12722.035498 secs 2024-07-26T16:23:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:06 smithi132 ceph-mon[37580]: pgmap v7177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:23:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12722.035498 secs 2024-07-26T16:23:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:08 smithi112 ceph-mon[37501]: pgmap v7178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:08.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:08 smithi107 ceph-mon[32659]: pgmap v7178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:08 smithi132 ceph-mon[37580]: pgmap v7178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:10 smithi112 ceph-mon[37501]: pgmap v7179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:23:10.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:10 smithi107 ceph-mon[32659]: pgmap v7179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:23:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:10 smithi132 ceph-mon[37580]: pgmap v7179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:23:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12727.035624 secs 2024-07-26T16:23:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12727.035624 secs 2024-07-26T16:23:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12727.035624 secs 2024-07-26T16:23:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:12 smithi112 ceph-mon[37501]: pgmap v7180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:23:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:12 smithi107 ceph-mon[32659]: pgmap v7180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:23:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:12 smithi132 ceph-mon[37580]: pgmap v7180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:23:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:14 smithi112 ceph-mon[37501]: pgmap v7181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:14.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:14 smithi107 ceph-mon[32659]: pgmap v7181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:14 smithi132 ceph-mon[37580]: pgmap v7181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:16 smithi112 ceph-mon[37501]: pgmap v7182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12732.035718 secs 2024-07-26T16:23:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:16 smithi107 ceph-mon[32659]: pgmap v7182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12732.035718 secs 2024-07-26T16:23:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:16 smithi132 ceph-mon[37580]: pgmap v7182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12732.035718 secs 2024-07-26T16:23:17.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:17.342+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:23:17.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:17.342+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:23:17.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:17.342+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:18 smithi112 ceph-mon[37501]: pgmap v7183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:18.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:18 smithi107 ceph-mon[32659]: pgmap v7183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:18 smithi132 ceph-mon[37580]: pgmap v7183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:20 smithi112 ceph-mon[37501]: pgmap v7184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:20 smithi107 ceph-mon[32659]: pgmap v7184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:20 smithi132 ceph-mon[37580]: pgmap v7184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12737.035851 secs 2024-07-26T16:23:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12737.035851 secs 2024-07-26T16:23:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12737.035851 secs 2024-07-26T16:23:22.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:22 smithi107 ceph-mon[32659]: pgmap v7185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:22 smithi132 ceph-mon[37580]: pgmap v7185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:22 smithi112 ceph-mon[37501]: pgmap v7185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:24.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:24 smithi107 ceph-mon[32659]: pgmap v7186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:24 smithi132 ceph-mon[37580]: pgmap v7186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:24 smithi112 ceph-mon[37501]: pgmap v7186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:26 smithi107 ceph-mon[32659]: pgmap v7187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12742.035984 secs 2024-07-26T16:23:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:26 smithi132 ceph-mon[37580]: pgmap v7187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12742.035984 secs 2024-07-26T16:23:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:26 smithi112 ceph-mon[37501]: pgmap v7187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12742.035984 secs 2024-07-26T16:23:28.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:28 smithi107 ceph-mon[32659]: pgmap v7188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:28 smithi132 ceph-mon[37580]: pgmap v7188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:28 smithi112 ceph-mon[37501]: pgmap v7188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:30 smithi107 ceph-mon[32659]: pgmap v7189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:30 smithi132 ceph-mon[37580]: pgmap v7189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:30 smithi112 ceph-mon[37501]: pgmap v7189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12747.036101 secs 2024-07-26T16:23:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12747.036101 secs 2024-07-26T16:23:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12747.036101 secs 2024-07-26T16:23:32.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:32.358+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:23:32.360 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:32.358+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:23:32.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:32.358+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:32.725 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:32 smithi107 ceph-mon[32659]: pgmap v7190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:32 smithi132 ceph-mon[37580]: pgmap v7190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:32 smithi112 ceph-mon[37501]: pgmap v7190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:34.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:34 smithi107 ceph-mon[32659]: pgmap v7191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:34 smithi132 ceph-mon[37580]: pgmap v7191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:34 smithi112 ceph-mon[37501]: pgmap v7191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:36 smithi107 ceph-mon[32659]: pgmap v7192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12752.036226 secs 2024-07-26T16:23:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:36 smithi132 ceph-mon[37580]: pgmap v7192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12752.036226 secs 2024-07-26T16:23:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:36 smithi112 ceph-mon[37501]: pgmap v7192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12752.036226 secs 2024-07-26T16:23:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:38 smithi107 ceph-mon[32659]: pgmap v7193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:38 smithi132 ceph-mon[37580]: pgmap v7193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:38 smithi112 ceph-mon[37501]: pgmap v7193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:40 smithi107 ceph-mon[32659]: pgmap v7194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:40 smithi132 ceph-mon[37580]: pgmap v7194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:40 smithi112 ceph-mon[37501]: pgmap v7194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12757.036334 secs 2024-07-26T16:23:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12757.036334 secs 2024-07-26T16:23:41.854 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12757.036334 secs 2024-07-26T16:23:42.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:42 smithi107 ceph-mon[32659]: pgmap v7195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:42 smithi132 ceph-mon[37580]: pgmap v7195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:42 smithi112 ceph-mon[37501]: pgmap v7195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:44.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:44 smithi107 ceph-mon[32659]: pgmap v7196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:44 smithi132 ceph-mon[37580]: pgmap v7196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:44 smithi112 ceph-mon[37501]: pgmap v7196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:46 smithi132 ceph-mon[37580]: pgmap v7197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:46.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12762.036475 secs 2024-07-26T16:23:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:46 smithi112 ceph-mon[37501]: pgmap v7197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12762.036475 secs 2024-07-26T16:23:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:46 smithi107 ceph-mon[32659]: pgmap v7197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12762.036475 secs 2024-07-26T16:23:47.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:47.363+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:23:47.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:47.363+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:23:47.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:23:47.363+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:48 smithi132 ceph-mon[37580]: pgmap v7198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:48 smithi112 ceph-mon[37501]: pgmap v7198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:49.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:48 smithi107 ceph-mon[32659]: pgmap v7198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:50 smithi132 ceph-mon[37580]: pgmap v7199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:50 smithi112 ceph-mon[37501]: pgmap v7199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:51.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:50 smithi107 ceph-mon[32659]: pgmap v7199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:23:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12767.036601 secs 2024-07-26T16:23:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12767.036601 secs 2024-07-26T16:23:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12767.036601 secs 2024-07-26T16:23:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:52 smithi112 ceph-mon[37501]: pgmap v7200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:52 smithi107 ceph-mon[32659]: pgmap v7200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:52 smithi132 ceph-mon[37580]: pgmap v7200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:23:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:54 smithi112 ceph-mon[37501]: pgmap v7201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:54 smithi107 ceph-mon[32659]: pgmap v7201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:54 smithi132 ceph-mon[37580]: pgmap v7201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:56 smithi112 ceph-mon[37501]: pgmap v7202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12772.036748 secs 2024-07-26T16:23:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:56 smithi107 ceph-mon[32659]: pgmap v7202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12772.036748 secs 2024-07-26T16:23:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:56 smithi132 ceph-mon[37580]: pgmap v7202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12772.036748 secs 2024-07-26T16:23:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:23:58 smithi112 ceph-mon[37501]: pgmap v7203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:59.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:23:58 smithi107 ceph-mon[32659]: pgmap v7203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:23:58 smithi132 ceph-mon[37580]: pgmap v7203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:00 smithi112 ceph-mon[37501]: pgmap v7204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:24:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:00 smithi107 ceph-mon[32659]: pgmap v7204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:24:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:00 smithi132 ceph-mon[37580]: pgmap v7204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:24:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12777.036858 secs 2024-07-26T16:24:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:24:01.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12777.036858 secs 2024-07-26T16:24:01.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:24:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12777.036858 secs 2024-07-26T16:24:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:24:02.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:02.377+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:24:02.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:02.377+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:24:02.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:02.377+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:02 smithi107 ceph-mon[32659]: pgmap v7205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:24:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:24:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:24:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:24:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:02 smithi112 ceph-mon[37501]: pgmap v7205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:24:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:24:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:24:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:24:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:02 smithi132 ceph-mon[37580]: pgmap v7205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:24:03.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:24:03.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:24:03.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:24:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:03 smithi112 ceph-mon[37501]: pgmap v7206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:24:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:03 smithi112 ceph-mon[37501]: pgmap v7207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:03 smithi107 ceph-mon[32659]: pgmap v7206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:24:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:03 smithi107 ceph-mon[32659]: pgmap v7207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:03 smithi132 ceph-mon[37580]: pgmap v7206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:24:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:03 smithi132 ceph-mon[37580]: pgmap v7207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:04 smithi112 ceph-mon[37501]: pgmap v7208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:04 smithi107 ceph-mon[32659]: pgmap v7208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:04 smithi132 ceph-mon[37580]: pgmap v7208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12782.036987 secs 2024-07-26T16:24:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12782.036987 secs 2024-07-26T16:24:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12782.036987 secs 2024-07-26T16:24:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:07 smithi112 ceph-mon[37501]: pgmap v7209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:07.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:07 smithi107 ceph-mon[32659]: pgmap v7209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:07 smithi132 ceph-mon[37580]: pgmap v7209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:09 smithi112 ceph-mon[37501]: pgmap v7210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:24:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:09 smithi107 ceph-mon[32659]: pgmap v7210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:24:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:09 smithi132 ceph-mon[37580]: pgmap v7210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:24:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12787.037119 secs 2024-07-26T16:24:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:11 smithi112 ceph-mon[37501]: pgmap v7211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12787.037119 secs 2024-07-26T16:24:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:11 smithi107 ceph-mon[32659]: pgmap v7211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12787.037119 secs 2024-07-26T16:24:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:11 smithi132 ceph-mon[37580]: pgmap v7211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:24:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:13 smithi112 ceph-mon[37501]: pgmap v7212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:24:14.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:13 smithi107 ceph-mon[32659]: pgmap v7212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:24:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:13 smithi132 ceph-mon[37580]: pgmap v7212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:24:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:14.839+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:24:14.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:14.839+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:24:14.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:14.839+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:24:14.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:14.840+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:24:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:15 smithi112 ceph-mon[37501]: pgmap v7213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:15 smithi107 ceph-mon[32659]: pgmap v7213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:15 smithi132 ceph-mon[37580]: pgmap v7213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12792.037249 secs 2024-07-26T16:24:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12792.037249 secs 2024-07-26T16:24:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12792.037249 secs 2024-07-26T16:24:17.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:17.393+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:24:17.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:17.393+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:24:17.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:17.393+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:17 smithi107 ceph-mon[32659]: pgmap v7214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:17 smithi112 ceph-mon[37501]: pgmap v7214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:17 smithi132 ceph-mon[37580]: pgmap v7214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:18 smithi112 ceph-mon[37501]: pgmap v7215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:18 smithi107 ceph-mon[32659]: pgmap v7215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:18 smithi132 ceph-mon[37580]: pgmap v7215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12797.037348 secs 2024-07-26T16:24:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:21 smithi112 ceph-mon[37501]: pgmap v7216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12797.037348 secs 2024-07-26T16:24:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:21 smithi107 ceph-mon[32659]: pgmap v7216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12797.037348 secs 2024-07-26T16:24:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:21 smithi132 ceph-mon[37580]: pgmap v7216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:23 smithi112 ceph-mon[37501]: pgmap v7217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:23 smithi107 ceph-mon[32659]: pgmap v7217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:23 smithi132 ceph-mon[37580]: pgmap v7217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:25 smithi112 ceph-mon[37501]: pgmap v7218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:25 smithi107 ceph-mon[32659]: pgmap v7218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:25 smithi132 ceph-mon[37580]: pgmap v7218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12802.037477 secs 2024-07-26T16:24:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12802.037477 secs 2024-07-26T16:24:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12802.037477 secs 2024-07-26T16:24:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:27 smithi112 ceph-mon[37501]: pgmap v7219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:27 smithi107 ceph-mon[32659]: pgmap v7219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:27 smithi132 ceph-mon[37580]: pgmap v7219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:29 smithi112 ceph-mon[37501]: pgmap v7220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:29 smithi107 ceph-mon[32659]: pgmap v7220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:29 smithi132 ceph-mon[37580]: pgmap v7220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:31.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12807.037636 secs 2024-07-26T16:24:31.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:31 smithi132 ceph-mon[37580]: pgmap v7221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12807.037636 secs 2024-07-26T16:24:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:31 smithi112 ceph-mon[37501]: pgmap v7221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:31.962 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12807.037636 secs 2024-07-26T16:24:31.963 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:31 smithi107 ceph-mon[32659]: pgmap v7221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:32.410+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:24:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:32.410+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:32.410+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:24:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:33 smithi112 ceph-mon[37501]: pgmap v7222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:33 smithi107 ceph-mon[32659]: pgmap v7222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:33 smithi132 ceph-mon[37580]: pgmap v7222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:35 smithi112 ceph-mon[37501]: pgmap v7223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:35 smithi107 ceph-mon[32659]: pgmap v7223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:35 smithi132 ceph-mon[37580]: pgmap v7223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12812.037765 secs 2024-07-26T16:24:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:36 smithi112 ceph-mon[37501]: pgmap v7224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12812.037765 secs 2024-07-26T16:24:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:36 smithi107 ceph-mon[32659]: pgmap v7224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12812.037765 secs 2024-07-26T16:24:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:36 smithi132 ceph-mon[37580]: pgmap v7224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:39 smithi112 ceph-mon[37501]: pgmap v7225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:39 smithi107 ceph-mon[32659]: pgmap v7225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:39 smithi132 ceph-mon[37580]: pgmap v7225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12817.037840 secs 2024-07-26T16:24:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:41 smithi112 ceph-mon[37501]: pgmap v7226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12817.037840 secs 2024-07-26T16:24:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:41 smithi107 ceph-mon[32659]: pgmap v7226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12817.037840 secs 2024-07-26T16:24:42.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:41 smithi132 ceph-mon[37580]: pgmap v7226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:43 smithi112 ceph-mon[37501]: pgmap v7227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:43 smithi107 ceph-mon[32659]: pgmap v7227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:43 smithi132 ceph-mon[37580]: pgmap v7227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:45 smithi112 ceph-mon[37501]: pgmap v7228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:45 smithi107 ceph-mon[32659]: pgmap v7228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:45 smithi132 ceph-mon[37580]: pgmap v7228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:46.945 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12822.037966 secs 2024-07-26T16:24:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12822.037966 secs 2024-07-26T16:24:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12822.037966 secs 2024-07-26T16:24:47.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:47.412+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:24:47.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:47.412+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:47.414 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:24:47.412+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:24:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:47 smithi107 ceph-mon[32659]: pgmap v7229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:47 smithi112 ceph-mon[37501]: pgmap v7229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:47 smithi132 ceph-mon[37580]: pgmap v7229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:24:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:49 smithi112 ceph-mon[37501]: pgmap v7230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:49 smithi107 ceph-mon[32659]: pgmap v7230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:49 smithi132 ceph-mon[37580]: pgmap v7230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:50 smithi112 ceph-mon[37501]: pgmap v7231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:51.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:50 smithi107 ceph-mon[32659]: pgmap v7231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:50 smithi132 ceph-mon[37580]: pgmap v7231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12827.038049 secs 2024-07-26T16:24:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12827.038049 secs 2024-07-26T16:24:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12827.038049 secs 2024-07-26T16:24:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:52 smithi112 ceph-mon[37501]: pgmap v7232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:52 smithi107 ceph-mon[32659]: pgmap v7232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:52 smithi132 ceph-mon[37580]: pgmap v7232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:24:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:55 smithi112 ceph-mon[37501]: pgmap v7233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:55 smithi107 ceph-mon[32659]: pgmap v7233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:55 smithi132 ceph-mon[37580]: pgmap v7233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:24:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12832.038177 secs 2024-07-26T16:24:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12832.038177 secs 2024-07-26T16:24:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12832.038177 secs 2024-07-26T16:24:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:57 smithi112 ceph-mon[37501]: pgmap v7234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:24:58.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:57 smithi107 ceph-mon[32659]: pgmap v7234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:24:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:57 smithi132 ceph-mon[37580]: pgmap v7234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:24:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:24:59 smithi112 ceph-mon[37501]: pgmap v7235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:24:59 smithi107 ceph-mon[32659]: pgmap v7235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:24:59 smithi132 ceph-mon[37580]: pgmap v7235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12837.038321 secs 2024-07-26T16:25:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:01 smithi112 ceph-mon[37501]: pgmap v7236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:25:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12837.038321 secs 2024-07-26T16:25:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:01 smithi107 ceph-mon[32659]: pgmap v7236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:25:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12837.038321 secs 2024-07-26T16:25:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:01 smithi132 ceph-mon[37580]: pgmap v7236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:25:02.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:02.427+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:25:02.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:02.427+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:02.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:02.427+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:25:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:02 smithi107 ceph-mon[32659]: pgmap v7237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:02 smithi112 ceph-mon[37501]: pgmap v7237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:02 smithi132 ceph-mon[37580]: pgmap v7237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:25:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:25:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:25:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:25:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:25:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:04 smithi112 ceph-mon[37501]: pgmap v7238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:25:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:04 smithi112 ceph-mon[37501]: pgmap v7239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:25:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:25:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:25:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:25:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:04 smithi107 ceph-mon[32659]: pgmap v7238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:25:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:04 smithi107 ceph-mon[32659]: pgmap v7239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:25:05.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:25:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:25:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:25:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:04 smithi132 ceph-mon[37580]: pgmap v7238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:25:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:04 smithi132 ceph-mon[37580]: pgmap v7239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:25:05.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:25:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12842.038453 secs 2024-07-26T16:25:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:06 smithi107 ceph-mon[32659]: pgmap v7240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:25:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12842.038453 secs 2024-07-26T16:25:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:06 smithi132 ceph-mon[37580]: pgmap v7240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:25:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12842.038453 secs 2024-07-26T16:25:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:06 smithi112 ceph-mon[37501]: pgmap v7240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:25:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:08 smithi107 ceph-mon[32659]: pgmap v7241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:25:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:08 smithi132 ceph-mon[37580]: pgmap v7241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:25:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:08 smithi112 ceph-mon[37501]: pgmap v7241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:25:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:10 smithi107 ceph-mon[32659]: pgmap v7242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:10 smithi132 ceph-mon[37580]: pgmap v7242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:10 smithi112 ceph-mon[37501]: pgmap v7242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12847.038548 secs 2024-07-26T16:25:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12847.038548 secs 2024-07-26T16:25:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12847.038548 secs 2024-07-26T16:25:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:12 smithi132 ceph-mon[37580]: pgmap v7243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:12 smithi112 ceph-mon[37501]: pgmap v7243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:12 smithi107 ceph-mon[32659]: pgmap v7243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:14 smithi132 ceph-mon[37580]: pgmap v7244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:25:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:14 smithi112 ceph-mon[37501]: pgmap v7244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:25:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:14 smithi107 ceph-mon[32659]: pgmap v7244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:25:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12852.038674 secs 2024-07-26T16:25:17.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:16 smithi132 ceph-mon[37580]: pgmap v7245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12852.038674 secs 2024-07-26T16:25:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:16 smithi112 ceph-mon[37501]: pgmap v7245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12852.038674 secs 2024-07-26T16:25:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:16 smithi107 ceph-mon[32659]: pgmap v7245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:17.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:17.435+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:25:17.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:17.435+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:17.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:17.435+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:25:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:18 smithi112 ceph-mon[37501]: pgmap v7246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:18 smithi107 ceph-mon[32659]: pgmap v7246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:19.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:18 smithi132 ceph-mon[37580]: pgmap v7246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:20 smithi107 ceph-mon[32659]: pgmap v7247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:25:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:20 smithi132 ceph-mon[37580]: pgmap v7247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:25:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:20 smithi112 ceph-mon[37501]: pgmap v7247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:25:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12857.038816 secs 2024-07-26T16:25:22.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12857.038816 secs 2024-07-26T16:25:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12857.038816 secs 2024-07-26T16:25:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:22 smithi132 ceph-mon[37580]: pgmap v7248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:22 smithi112 ceph-mon[37501]: pgmap v7248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:22 smithi107 ceph-mon[32659]: pgmap v7248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:25.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:24 smithi132 ceph-mon[37580]: pgmap v7249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:24 smithi112 ceph-mon[37501]: pgmap v7249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:24 smithi107 ceph-mon[32659]: pgmap v7249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:27.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12862.038943 secs 2024-07-26T16:25:27.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:26 smithi132 ceph-mon[37580]: pgmap v7250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12862.038943 secs 2024-07-26T16:25:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:26 smithi112 ceph-mon[37501]: pgmap v7250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12862.038943 secs 2024-07-26T16:25:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:26 smithi107 ceph-mon[32659]: pgmap v7250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:28 smithi112 ceph-mon[37501]: pgmap v7251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:29.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:28 smithi107 ceph-mon[32659]: pgmap v7251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:28 smithi132 ceph-mon[37580]: pgmap v7251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:31.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:30 smithi107 ceph-mon[32659]: pgmap v7252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:31.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:30 smithi132 ceph-mon[37580]: pgmap v7252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:30 smithi112 ceph-mon[37501]: pgmap v7252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12867.039087 secs 2024-07-26T16:25:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12867.039087 secs 2024-07-26T16:25:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12867.039087 secs 2024-07-26T16:25:32.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:32.451+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:25:32.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:32.451+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:32.454 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:32.451+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:25:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:32 smithi112 ceph-mon[37501]: pgmap v7253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:32 smithi107 ceph-mon[32659]: pgmap v7253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:32 smithi132 ceph-mon[37580]: pgmap v7253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:34 smithi112 ceph-mon[37501]: pgmap v7254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:34 smithi107 ceph-mon[32659]: pgmap v7254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:34 smithi132 ceph-mon[37580]: pgmap v7254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12872.039214 secs 2024-07-26T16:25:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:36 smithi112 ceph-mon[37501]: pgmap v7255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:37.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12872.039214 secs 2024-07-26T16:25:37.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:36 smithi107 ceph-mon[32659]: pgmap v7255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12872.039214 secs 2024-07-26T16:25:37.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:36 smithi132 ceph-mon[37580]: pgmap v7255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:39.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:38 smithi112 ceph-mon[37501]: pgmap v7256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:39.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:38 smithi107 ceph-mon[32659]: pgmap v7256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:38 smithi132 ceph-mon[37580]: pgmap v7256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:40 smithi112 ceph-mon[37501]: pgmap v7257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:40 smithi107 ceph-mon[32659]: pgmap v7257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:40 smithi132 ceph-mon[37580]: pgmap v7257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12877.039362 secs 2024-07-26T16:25:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12877.039362 secs 2024-07-26T16:25:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12877.039362 secs 2024-07-26T16:25:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:42 smithi112 ceph-mon[37501]: pgmap v7258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:43.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:42 smithi107 ceph-mon[32659]: pgmap v7258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:42 smithi132 ceph-mon[37580]: pgmap v7258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:43.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:43.854+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:25:43.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:43.854+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:25:43.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:43.855+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:25:43.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:43.855+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:25:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:44 smithi112 ceph-mon[37501]: pgmap v7259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:45.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:44 smithi107 ceph-mon[32659]: pgmap v7259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:44 smithi132 ceph-mon[37580]: pgmap v7259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12882.039502 secs 2024-07-26T16:25:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:46 smithi112 ceph-mon[37501]: pgmap v7260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12882.039502 secs 2024-07-26T16:25:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:46 smithi107 ceph-mon[32659]: pgmap v7260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12882.039502 secs 2024-07-26T16:25:47.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:46 smithi132 ceph-mon[37580]: pgmap v7260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:47.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:47.467+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:25:47.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:47.467+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:47.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:25:47.467+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:25:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:48 smithi112 ceph-mon[37501]: pgmap v7261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:48 smithi107 ceph-mon[32659]: pgmap v7261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:48 smithi132 ceph-mon[37580]: pgmap v7261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:25:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:50 smithi112 ceph-mon[37501]: pgmap v7262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:50 smithi107 ceph-mon[32659]: pgmap v7262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:50 smithi132 ceph-mon[37580]: pgmap v7262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:25:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12887.039631 secs 2024-07-26T16:25:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12887.039631 secs 2024-07-26T16:25:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12887.039631 secs 2024-07-26T16:25:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:52 smithi112 ceph-mon[37501]: pgmap v7263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:25:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:52 smithi107 ceph-mon[32659]: pgmap v7263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:25:53.315 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:52 smithi132 ceph-mon[37580]: pgmap v7263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:25:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:54 smithi112 ceph-mon[37501]: pgmap v7264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:55.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:54 smithi107 ceph-mon[32659]: pgmap v7264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:54 smithi132 ceph-mon[37580]: pgmap v7264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12892.039780 secs 2024-07-26T16:25:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:56 smithi112 ceph-mon[37501]: pgmap v7265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:25:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12892.039780 secs 2024-07-26T16:25:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:56 smithi107 ceph-mon[32659]: pgmap v7265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:25:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12892.039780 secs 2024-07-26T16:25:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:56 smithi132 ceph-mon[37580]: pgmap v7265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:25:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:25:58 smithi112 ceph-mon[37501]: pgmap v7266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:59.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:25:58 smithi107 ceph-mon[32659]: pgmap v7266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:25:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:25:58 smithi132 ceph-mon[37580]: pgmap v7266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:26:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:00 smithi112 ceph-mon[37501]: pgmap v7267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:00 smithi107 ceph-mon[32659]: pgmap v7267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:00 smithi132 ceph-mon[37580]: pgmap v7267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12897.039900 secs 2024-07-26T16:26:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12897.039900 secs 2024-07-26T16:26:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12897.039900 secs 2024-07-26T16:26:02.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:02.481+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:26:02.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:02.481+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:02.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:02.481+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:26:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:02 smithi112 ceph-mon[37501]: pgmap v7268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:26:03.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:02 smithi107 ceph-mon[32659]: pgmap v7268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:26:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:02 smithi132 ceph-mon[37580]: pgmap v7268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:26:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:26:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:26:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:26:05.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:04 smithi107 ceph-mon[32659]: pgmap v7269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:05.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:26:05.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:26:05.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:26:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:04 smithi132 ceph-mon[37580]: pgmap v7269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:26:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:26:05.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:26:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:04 smithi112 ceph-mon[37501]: pgmap v7269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:26:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:26:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:26:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:05 smithi107 ceph-mon[32659]: pgmap v7270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:26:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:05 smithi107 ceph-mon[32659]: pgmap v7271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:05 smithi132 ceph-mon[37580]: pgmap v7270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:26:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:05 smithi132 ceph-mon[37580]: pgmap v7271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:05 smithi112 ceph-mon[37501]: pgmap v7270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:26:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:05 smithi112 ceph-mon[37501]: pgmap v7271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12902.040027 secs 2024-07-26T16:26:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12902.040027 secs 2024-07-26T16:26:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12902.040027 secs 2024-07-26T16:26:08.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:07 smithi107 ceph-mon[32659]: pgmap v7272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:07 smithi132 ceph-mon[37580]: pgmap v7272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:07 smithi112 ceph-mon[37501]: pgmap v7272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:09 smithi107 ceph-mon[32659]: pgmap v7273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:26:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:09 smithi132 ceph-mon[37580]: pgmap v7273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:26:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:09 smithi112 ceph-mon[37501]: pgmap v7273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:26:12.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12907.040144 secs 2024-07-26T16:26:12.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:11 smithi112 ceph-mon[37501]: pgmap v7274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:26:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12907.040144 secs 2024-07-26T16:26:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:11 smithi107 ceph-mon[32659]: pgmap v7274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:26:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12907.040144 secs 2024-07-26T16:26:12.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:11 smithi132 ceph-mon[37580]: pgmap v7274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:26:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:13 smithi107 ceph-mon[32659]: pgmap v7275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:26:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:13 smithi132 ceph-mon[37580]: pgmap v7275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:26:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:13 smithi112 ceph-mon[37501]: pgmap v7275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:26:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:15 smithi107 ceph-mon[32659]: pgmap v7276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:26:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:15 smithi132 ceph-mon[37580]: pgmap v7276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:26:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:15 smithi112 ceph-mon[37501]: pgmap v7276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:26:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12912.040245 secs 2024-07-26T16:26:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12912.040245 secs 2024-07-26T16:26:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12912.040245 secs 2024-07-26T16:26:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:17.487+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:26:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:17.487+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:17.490 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:17.487+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:26:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:18 smithi107 ceph-mon[32659]: pgmap v7277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:18 smithi132 ceph-mon[37580]: pgmap v7277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:18 smithi112 ceph-mon[37501]: pgmap v7277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:20 smithi107 ceph-mon[32659]: pgmap v7278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:20 smithi132 ceph-mon[37580]: pgmap v7278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:20 smithi112 ceph-mon[37501]: pgmap v7278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12917.040341 secs 2024-07-26T16:26:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:22 smithi107 ceph-mon[32659]: pgmap v7279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12917.040341 secs 2024-07-26T16:26:22.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:22 smithi132 ceph-mon[37580]: pgmap v7279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12917.040341 secs 2024-07-26T16:26:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:22 smithi112 ceph-mon[37501]: pgmap v7279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:24.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:24 smithi107 ceph-mon[32659]: pgmap v7280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:24 smithi132 ceph-mon[37580]: pgmap v7280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:24 smithi112 ceph-mon[37501]: pgmap v7280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:26 smithi107 ceph-mon[32659]: pgmap v7281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:26 smithi132 ceph-mon[37580]: pgmap v7281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:26 smithi112 ceph-mon[37501]: pgmap v7281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12922.040410 secs 2024-07-26T16:26:27.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12922.040410 secs 2024-07-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12922.040410 secs 2024-07-26T16:26:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:28 smithi132 ceph-mon[37580]: pgmap v7282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:28 smithi112 ceph-mon[37501]: pgmap v7282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:28.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:28 smithi107 ceph-mon[32659]: pgmap v7282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:30 smithi132 ceph-mon[37580]: pgmap v7283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:30 smithi112 ceph-mon[37501]: pgmap v7283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:30 smithi107 ceph-mon[32659]: pgmap v7283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:32.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:32 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12927.040539 secs 2024-07-26T16:26:32.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:32 smithi107 ceph-mon[32659]: pgmap v7284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:32 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12927.040539 secs 2024-07-26T16:26:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:32 smithi112 ceph-mon[37501]: pgmap v7284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:32.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:32.489+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:26:32.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:32.489+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:32.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:32.489+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:26:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:32 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12927.040539 secs 2024-07-26T16:26:32.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:32 smithi132 ceph-mon[37580]: pgmap v7284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:34 smithi112 ceph-mon[37501]: pgmap v7285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:34.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:34 smithi107 ceph-mon[32659]: pgmap v7285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:34 smithi132 ceph-mon[37580]: pgmap v7285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:36 smithi112 ceph-mon[37501]: pgmap v7286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12932.040632 secs 2024-07-26T16:26:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:36 smithi107 ceph-mon[32659]: pgmap v7286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12932.040632 secs 2024-07-26T16:26:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:36 smithi132 ceph-mon[37580]: pgmap v7286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12932.040632 secs 2024-07-26T16:26:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:38 smithi112 ceph-mon[37501]: pgmap v7287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:38 smithi107 ceph-mon[32659]: pgmap v7287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:38 smithi132 ceph-mon[37580]: pgmap v7287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:40 smithi112 ceph-mon[37501]: pgmap v7288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:40.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:40 smithi107 ceph-mon[32659]: pgmap v7288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:40 smithi132 ceph-mon[37580]: pgmap v7288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12937.040763 secs 2024-07-26T16:26:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12937.040763 secs 2024-07-26T16:26:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12937.040763 secs 2024-07-26T16:26:42.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:42 smithi112 ceph-mon[37501]: pgmap v7289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:42 smithi107 ceph-mon[32659]: pgmap v7289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:42 smithi132 ceph-mon[37580]: pgmap v7289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:44 smithi112 ceph-mon[37501]: pgmap v7290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:44.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:44 smithi107 ceph-mon[32659]: pgmap v7290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:44 smithi132 ceph-mon[37580]: pgmap v7290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:46 smithi112 ceph-mon[37501]: pgmap v7291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:26:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12942.040914 secs 2024-07-26T16:26:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:46 smithi107 ceph-mon[32659]: pgmap v7291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:26:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12942.040914 secs 2024-07-26T16:26:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:46 smithi132 ceph-mon[37580]: pgmap v7291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:26:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12942.040914 secs 2024-07-26T16:26:47.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:47.502+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:26:47.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:47.502+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:26:47.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:26:47.502+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:48 smithi112 ceph-mon[37501]: pgmap v7292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:26:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:48 smithi107 ceph-mon[32659]: pgmap v7292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:26:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:48 smithi132 ceph-mon[37580]: pgmap v7292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:26:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:50 smithi112 ceph-mon[37501]: pgmap v7293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:50.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:50 smithi107 ceph-mon[32659]: pgmap v7293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:50 smithi132 ceph-mon[37580]: pgmap v7293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12947.041034 secs 2024-07-26T16:26:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12947.041034 secs 2024-07-26T16:26:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12947.041034 secs 2024-07-26T16:26:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:52 smithi112 ceph-mon[37501]: pgmap v7294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:26:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:52 smithi107 ceph-mon[32659]: pgmap v7294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:26:52.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:52 smithi132 ceph-mon[37580]: pgmap v7294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:26:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:54 smithi112 ceph-mon[37501]: pgmap v7295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:54 smithi107 ceph-mon[32659]: pgmap v7295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:54 smithi132 ceph-mon[37580]: pgmap v7295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:56 smithi112 ceph-mon[37501]: pgmap v7296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12952.041171 secs 2024-07-26T16:26:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:56 smithi107 ceph-mon[32659]: pgmap v7296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12952.041171 secs 2024-07-26T16:26:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:56 smithi132 ceph-mon[37580]: pgmap v7296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12952.041171 secs 2024-07-26T16:26:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:26:58 smithi112 ceph-mon[37501]: pgmap v7297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:58.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:26:58 smithi107 ceph-mon[32659]: pgmap v7297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:26:58 smithi132 ceph-mon[37580]: pgmap v7297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:00 smithi112 ceph-mon[37501]: pgmap v7298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:00 smithi107 ceph-mon[32659]: pgmap v7298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:00 smithi132 ceph-mon[37580]: pgmap v7298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12957.041268 secs 2024-07-26T16:27:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12957.041268 secs 2024-07-26T16:27:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12957.041268 secs 2024-07-26T16:27:02.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:02 smithi107 ceph-mon[32659]: pgmap v7299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:02 smithi112 ceph-mon[37501]: pgmap v7299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:02.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:02 smithi132 ceph-mon[37580]: pgmap v7299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:02.519 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:02.516+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:27:02.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:02.516+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:02.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:02.516+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:27:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:04 smithi112 ceph-mon[37501]: pgmap v7300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:04 smithi107 ceph-mon[32659]: pgmap v7300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:04 smithi132 ceph-mon[37580]: pgmap v7300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:27:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:27:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:27:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:06 smithi112 ceph-mon[37501]: pgmap v7301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12962.041335 secs 2024-07-26T16:27:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:27:06.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:27:06.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:27:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:06 smithi107 ceph-mon[32659]: pgmap v7301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12962.041335 secs 2024-07-26T16:27:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:27:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:27:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:27:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:06 smithi132 ceph-mon[37580]: pgmap v7301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:06.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12962.041335 secs 2024-07-26T16:27:06.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:27:06.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:27:06.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:27:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:07 smithi112 ceph-mon[37501]: pgmap v7302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:27:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:07 smithi112 ceph-mon[37501]: pgmap v7303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:27:07.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:07 smithi107 ceph-mon[32659]: pgmap v7302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:27:07.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:07 smithi107 ceph-mon[32659]: pgmap v7303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:27:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:07 smithi132 ceph-mon[37580]: pgmap v7302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:27:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:07 smithi132 ceph-mon[37580]: pgmap v7303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:27:09.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:09 smithi107 ceph-mon[32659]: pgmap v7304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:27:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:09 smithi132 ceph-mon[37580]: pgmap v7304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:27:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:09 smithi112 ceph-mon[37501]: pgmap v7304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:27:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:11 smithi107 ceph-mon[32659]: pgmap v7305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12967.041458 secs 2024-07-26T16:27:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:11 smithi132 ceph-mon[37580]: pgmap v7305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:11.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12967.041458 secs 2024-07-26T16:27:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:11 smithi112 ceph-mon[37501]: pgmap v7305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12967.041458 secs 2024-07-26T16:27:13.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:13 smithi112 ceph-mon[37501]: pgmap v7306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:13 smithi107 ceph-mon[32659]: pgmap v7306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:13 smithi132 ceph-mon[37580]: pgmap v7306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:14.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:14.870+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:27:14.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:14.870+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:27:14.873 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:14.871+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:27:14.874 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:14.871+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:27:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:15 smithi107 ceph-mon[32659]: pgmap v7307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:27:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:15 smithi132 ceph-mon[37580]: pgmap v7307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:27:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:15 smithi112 ceph-mon[37501]: pgmap v7307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:27:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12972.041549 secs 2024-07-26T16:27:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12972.041549 secs 2024-07-26T16:27:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12972.041549 secs 2024-07-26T16:27:17.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:17.524+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:27:17.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:17.524+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:17.527 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:17.524+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:27:17.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:17 smithi107 ceph-mon[32659]: pgmap v7308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:27:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:17 smithi132 ceph-mon[37580]: pgmap v7308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:27:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:17 smithi112 ceph-mon[37501]: pgmap v7308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:27:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:19 smithi107 ceph-mon[32659]: pgmap v7309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:19 smithi132 ceph-mon[37580]: pgmap v7309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:19 smithi112 ceph-mon[37501]: pgmap v7309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:21 smithi107 ceph-mon[32659]: pgmap v7310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12977.041678 secs 2024-07-26T16:27:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:21 smithi132 ceph-mon[37580]: pgmap v7310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12977.041678 secs 2024-07-26T16:27:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:21 smithi112 ceph-mon[37501]: pgmap v7310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12977.041678 secs 2024-07-26T16:27:23.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:23 smithi107 ceph-mon[32659]: pgmap v7311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:23 smithi132 ceph-mon[37580]: pgmap v7311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:23 smithi112 ceph-mon[37501]: pgmap v7311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:25 smithi107 ceph-mon[32659]: pgmap v7312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:25 smithi132 ceph-mon[37580]: pgmap v7312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:25 smithi112 ceph-mon[37501]: pgmap v7312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12982.041824 secs 2024-07-26T16:27:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12982.041824 secs 2024-07-26T16:27:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12982.041824 secs 2024-07-26T16:27:27.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:27 smithi107 ceph-mon[32659]: pgmap v7313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:27 smithi132 ceph-mon[37580]: pgmap v7313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:27 smithi112 ceph-mon[37501]: pgmap v7313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:29.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:29 smithi107 ceph-mon[32659]: pgmap v7314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:29 smithi132 ceph-mon[37580]: pgmap v7314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:29 smithi112 ceph-mon[37501]: pgmap v7314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:31 smithi107 ceph-mon[32659]: pgmap v7315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12987.041970 secs 2024-07-26T16:27:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:31 smithi132 ceph-mon[37580]: pgmap v7315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12987.041970 secs 2024-07-26T16:27:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:31 smithi112 ceph-mon[37501]: pgmap v7315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12987.041970 secs 2024-07-26T16:27:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:32.540+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:27:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:32.540+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:27:32.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:32.540+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:33.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:33 smithi107 ceph-mon[32659]: pgmap v7316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:33 smithi132 ceph-mon[37580]: pgmap v7316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:33 smithi112 ceph-mon[37501]: pgmap v7316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:35 smithi107 ceph-mon[32659]: pgmap v7317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:35 smithi132 ceph-mon[37580]: pgmap v7317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:35 smithi112 ceph-mon[37501]: pgmap v7317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12992.042120 secs 2024-07-26T16:27:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12992.042120 secs 2024-07-26T16:27:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12992.042120 secs 2024-07-26T16:27:37.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:37 smithi132 ceph-mon[37580]: pgmap v7318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:37 smithi112 ceph-mon[37501]: pgmap v7318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:37 smithi107 ceph-mon[32659]: pgmap v7318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:39 smithi132 ceph-mon[37580]: pgmap v7319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:39 smithi112 ceph-mon[37501]: pgmap v7319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:39 smithi107 ceph-mon[32659]: pgmap v7319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:41 smithi112 ceph-mon[37501]: pgmap v7320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:41.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 12997.042261 secs 2024-07-26T16:27:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:41 smithi107 ceph-mon[32659]: pgmap v7320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 12997.042261 secs 2024-07-26T16:27:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:41 smithi132 ceph-mon[37580]: pgmap v7320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:41.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 12997.042261 secs 2024-07-26T16:27:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:43 smithi112 ceph-mon[37501]: pgmap v7321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:43 smithi107 ceph-mon[32659]: pgmap v7321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:43 smithi132 ceph-mon[37580]: pgmap v7321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:45 smithi112 ceph-mon[37501]: pgmap v7322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:45 smithi107 ceph-mon[32659]: pgmap v7322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:45 smithi132 ceph-mon[37580]: pgmap v7322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13002.042347 secs 2024-07-26T16:27:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13002.042347 secs 2024-07-26T16:27:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13002.042347 secs 2024-07-26T16:27:47.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:47.555+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:27:47.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:47.555+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:27:47.558 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:27:47.556+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:47 smithi112 ceph-mon[37501]: pgmap v7323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:47 smithi107 ceph-mon[32659]: pgmap v7323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:47 smithi132 ceph-mon[37580]: pgmap v7323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:49 smithi112 ceph-mon[37501]: pgmap v7324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:49.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:49 smithi107 ceph-mon[32659]: pgmap v7324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:49 smithi132 ceph-mon[37580]: pgmap v7324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:51 smithi112 ceph-mon[37501]: pgmap v7325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13007.042414 secs 2024-07-26T16:27:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:51 smithi107 ceph-mon[32659]: pgmap v7325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13007.042414 secs 2024-07-26T16:27:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:51 smithi132 ceph-mon[37580]: pgmap v7325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13007.042414 secs 2024-07-26T16:27:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:53 smithi112 ceph-mon[37501]: pgmap v7326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:53 smithi107 ceph-mon[32659]: pgmap v7326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:53 smithi132 ceph-mon[37580]: pgmap v7326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:55 smithi112 ceph-mon[37501]: pgmap v7327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:55.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:55 smithi107 ceph-mon[32659]: pgmap v7327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:55 smithi132 ceph-mon[37580]: pgmap v7327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13012.042543 secs 2024-07-26T16:27:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13012.042543 secs 2024-07-26T16:27:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13012.042543 secs 2024-07-26T16:27:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:57 smithi112 ceph-mon[37501]: pgmap v7328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:57.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:57 smithi107 ceph-mon[32659]: pgmap v7328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:57 smithi132 ceph-mon[37580]: pgmap v7328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:27:59 smithi112 ceph-mon[37501]: pgmap v7329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:27:59 smithi107 ceph-mon[32659]: pgmap v7329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:27:59 smithi132 ceph-mon[37580]: pgmap v7329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:01 smithi112 ceph-mon[37501]: pgmap v7330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13017.042619 secs 2024-07-26T16:28:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:01 smithi107 ceph-mon[32659]: pgmap v7330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13017.042619 secs 2024-07-26T16:28:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:01 smithi132 ceph-mon[37580]: pgmap v7330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13017.042619 secs 2024-07-26T16:28:02.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:02.570+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:28:02.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:02.570+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:28:02.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:02.570+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:03 smithi112 ceph-mon[37501]: pgmap v7331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:03 smithi107 ceph-mon[32659]: pgmap v7331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:03 smithi132 ceph-mon[37580]: pgmap v7331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:05 smithi112 ceph-mon[37501]: pgmap v7332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:05.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:05 smithi107 ceph-mon[32659]: pgmap v7332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:05 smithi132 ceph-mon[37580]: pgmap v7332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13022.042729 secs 2024-07-26T16:28:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:28:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13022.042729 secs 2024-07-26T16:28:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:28:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13022.042729 secs 2024-07-26T16:28:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:28:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: pgmap v7333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:28:07.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:28:07.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:28:07.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: pgmap v7333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:07.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:28:07.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:28:07.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:28:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: pgmap v7333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:07.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:28:07.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:28:07.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:28:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:08 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:28:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:08 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:28:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:08 smithi112 ceph-mon[37501]: pgmap v7334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:28:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:08 smithi112 ceph-mon[37501]: pgmap v7335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:08.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:08 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:28:08.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:08 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:28:08.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:08 smithi107 ceph-mon[32659]: pgmap v7334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:28:08.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:08 smithi107 ceph-mon[32659]: pgmap v7335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:08 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:28:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:08 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:28:08.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:08 smithi132 ceph-mon[37580]: pgmap v7334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:28:08.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:08 smithi132 ceph-mon[37580]: pgmap v7335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:10 smithi112 ceph-mon[37501]: pgmap v7336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:10.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:10 smithi107 ceph-mon[32659]: pgmap v7336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:10 smithi132 ceph-mon[37580]: pgmap v7336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13027.042849 secs 2024-07-26T16:28:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13027.042849 secs 2024-07-26T16:28:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13027.042849 secs 2024-07-26T16:28:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:12 smithi112 ceph-mon[37501]: pgmap v7337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:28:12.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:12 smithi107 ceph-mon[32659]: pgmap v7337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:28:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:12 smithi132 ceph-mon[37580]: pgmap v7337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:28:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:14 smithi112 ceph-mon[37501]: pgmap v7338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:14 smithi107 ceph-mon[32659]: pgmap v7338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:14 smithi132 ceph-mon[37580]: pgmap v7338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:16 smithi107 ceph-mon[32659]: pgmap v7339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13032.043001 secs 2024-07-26T16:28:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:16 smithi132 ceph-mon[37580]: pgmap v7339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13032.043001 secs 2024-07-26T16:28:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:16 smithi112 ceph-mon[37501]: pgmap v7339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13032.043001 secs 2024-07-26T16:28:17.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:17.586+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:28:17.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:17.586+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:28:17.589 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:17.586+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:18.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:18 smithi107 ceph-mon[32659]: pgmap v7340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:28:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:18 smithi132 ceph-mon[37580]: pgmap v7340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:28:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:18 smithi112 ceph-mon[37501]: pgmap v7340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:28:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:20 smithi107 ceph-mon[32659]: pgmap v7341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:20 smithi132 ceph-mon[37580]: pgmap v7341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:20 smithi112 ceph-mon[37501]: pgmap v7341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13037.043138 secs 2024-07-26T16:28:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13037.043138 secs 2024-07-26T16:28:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13037.043138 secs 2024-07-26T16:28:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:22 smithi107 ceph-mon[32659]: pgmap v7342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:22 smithi132 ceph-mon[37580]: pgmap v7342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:22 smithi112 ceph-mon[37501]: pgmap v7342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:24.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:24 smithi107 ceph-mon[32659]: pgmap v7343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:24 smithi132 ceph-mon[37580]: pgmap v7343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:24 smithi112 ceph-mon[37501]: pgmap v7343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:26 smithi107 ceph-mon[32659]: pgmap v7344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13042.043233 secs 2024-07-26T16:28:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:26 smithi132 ceph-mon[37580]: pgmap v7344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13042.043233 secs 2024-07-26T16:28:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:26 smithi112 ceph-mon[37501]: pgmap v7344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13042.043233 secs 2024-07-26T16:28:28.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:28 smithi107 ceph-mon[32659]: pgmap v7345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:28 smithi132 ceph-mon[37580]: pgmap v7345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:28 smithi112 ceph-mon[37501]: pgmap v7345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:30 smithi107 ceph-mon[32659]: pgmap v7346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:30 smithi132 ceph-mon[37580]: pgmap v7346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:30 smithi112 ceph-mon[37501]: pgmap v7346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13047.043357 secs 2024-07-26T16:28:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13047.043357 secs 2024-07-26T16:28:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13047.043357 secs 2024-07-26T16:28:32.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:32.590+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:28:32.592 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:32.590+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:32.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:32.590+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:28:32.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:32 smithi107 ceph-mon[32659]: pgmap v7347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:28:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:32 smithi132 ceph-mon[37580]: pgmap v7347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:28:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:32 smithi112 ceph-mon[37501]: pgmap v7347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:28:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:33 smithi107 ceph-mon[32659]: pgmap v7348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:33 smithi132 ceph-mon[37580]: pgmap v7348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:33 smithi112 ceph-mon[37501]: pgmap v7348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:36 smithi107 ceph-mon[32659]: pgmap v7349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13052.043436 secs 2024-07-26T16:28:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:36 smithi132 ceph-mon[37580]: pgmap v7349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13052.043436 secs 2024-07-26T16:28:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:36 smithi112 ceph-mon[37501]: pgmap v7349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13052.043436 secs 2024-07-26T16:28:38.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:38 smithi107 ceph-mon[32659]: pgmap v7350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:38 smithi132 ceph-mon[37580]: pgmap v7350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:38 smithi112 ceph-mon[37501]: pgmap v7350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:40.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:40 smithi107 ceph-mon[32659]: pgmap v7351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:40 smithi132 ceph-mon[37580]: pgmap v7351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:40 smithi112 ceph-mon[37501]: pgmap v7351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13057.043558 secs 2024-07-26T16:28:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13057.043558 secs 2024-07-26T16:28:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13057.043558 secs 2024-07-26T16:28:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:42 smithi132 ceph-mon[37580]: pgmap v7352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:42 smithi112 ceph-mon[37501]: pgmap v7352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:42.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:42 smithi107 ceph-mon[32659]: pgmap v7352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:44 smithi132 ceph-mon[37580]: pgmap v7353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:44.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:44 smithi112 ceph-mon[37501]: pgmap v7353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:44.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:44 smithi107 ceph-mon[32659]: pgmap v7353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:46 smithi132 ceph-mon[37580]: pgmap v7354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:46.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13062.043694 secs 2024-07-26T16:28:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:46 smithi112 ceph-mon[37501]: pgmap v7354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13062.043694 secs 2024-07-26T16:28:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:46 smithi107 ceph-mon[32659]: pgmap v7354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13062.043694 secs 2024-07-26T16:28:47.593 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:47.591+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:28:47.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:47.592+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:28:47.594 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:28:47.592+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:48 smithi112 ceph-mon[37501]: pgmap v7355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:48 smithi107 ceph-mon[32659]: pgmap v7355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:48 smithi132 ceph-mon[37580]: pgmap v7355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:50 smithi112 ceph-mon[37501]: pgmap v7356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:50 smithi107 ceph-mon[32659]: pgmap v7356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:50 smithi132 ceph-mon[37580]: pgmap v7356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13067.043841 secs 2024-07-26T16:28:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13067.043841 secs 2024-07-26T16:28:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13067.043841 secs 2024-07-26T16:28:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:52 smithi112 ceph-mon[37501]: pgmap v7357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:52.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:52 smithi107 ceph-mon[32659]: pgmap v7357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:52 smithi132 ceph-mon[37580]: pgmap v7357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:54 smithi107 ceph-mon[32659]: pgmap v7358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:54.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:54 smithi112 ceph-mon[37501]: pgmap v7358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:54 smithi132 ceph-mon[37580]: pgmap v7358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:56 smithi112 ceph-mon[37501]: pgmap v7359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13072.043954 secs 2024-07-26T16:28:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:56 smithi107 ceph-mon[32659]: pgmap v7359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13072.043954 secs 2024-07-26T16:28:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:56 smithi132 ceph-mon[37580]: pgmap v7359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13072.043954 secs 2024-07-26T16:28:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:28:58 smithi112 ceph-mon[37501]: pgmap v7360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:28:58 smithi107 ceph-mon[32659]: pgmap v7360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:28:58 smithi132 ceph-mon[37580]: pgmap v7360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:29:00.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:00 smithi112 ceph-mon[37501]: pgmap v7361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:00 smithi107 ceph-mon[32659]: pgmap v7361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:00 smithi132 ceph-mon[37580]: pgmap v7361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13077.044085 secs 2024-07-26T16:29:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13077.044085 secs 2024-07-26T16:29:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13077.044085 secs 2024-07-26T16:29:02.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:02.600+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:29:02.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:02.600+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:29:02.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:02.600+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:02.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:02 smithi107 ceph-mon[32659]: pgmap v7362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:02 smithi112 ceph-mon[37501]: pgmap v7362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:02.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:02 smithi132 ceph-mon[37580]: pgmap v7362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:04 smithi112 ceph-mon[37501]: pgmap v7363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:04 smithi107 ceph-mon[32659]: pgmap v7363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:04 smithi132 ceph-mon[37580]: pgmap v7363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:06 smithi112 ceph-mon[37501]: pgmap v7364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13082.044173 secs 2024-07-26T16:29:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:06 smithi107 ceph-mon[32659]: pgmap v7364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13082.044173 secs 2024-07-26T16:29:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:06 smithi132 ceph-mon[37580]: pgmap v7364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13082.044173 secs 2024-07-26T16:29:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: pgmap v7365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:29:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:29:08.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.704 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:29:08.704 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:29:08.704 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:29:08.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: pgmap v7365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:29:08.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:29:08.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:29:08.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:29:08.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: pgmap v7365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:29:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:09 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:29:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:09 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:29:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:09 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:29:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:09 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:29:09.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:09 smithi112 ceph-mon[37501]: pgmap v7366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:29:09.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:09 smithi112 ceph-mon[37501]: pgmap v7367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:09 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:29:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:09 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:29:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:09 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:29:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:09 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:29:09.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:09 smithi107 ceph-mon[32659]: pgmap v7366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:29:09.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:09 smithi107 ceph-mon[32659]: pgmap v7367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:09 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:29:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:09 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:29:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:09 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:29:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:09 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:29:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:09 smithi132 ceph-mon[37580]: pgmap v7366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:29:09.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:09 smithi132 ceph-mon[37580]: pgmap v7367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:11 smithi112 ceph-mon[37501]: pgmap v7368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13087.044266 secs 2024-07-26T16:29:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:11 smithi107 ceph-mon[32659]: pgmap v7368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13087.044266 secs 2024-07-26T16:29:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:11 smithi132 ceph-mon[37580]: pgmap v7368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13087.044266 secs 2024-07-26T16:29:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:13 smithi112 ceph-mon[37501]: pgmap v7369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:29:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:13 smithi107 ceph-mon[32659]: pgmap v7369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:29:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:13 smithi132 ceph-mon[37580]: pgmap v7369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:29:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:15 smithi112 ceph-mon[37501]: pgmap v7370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:15 smithi107 ceph-mon[32659]: pgmap v7370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:15 smithi132 ceph-mon[37580]: pgmap v7370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13092.044377 secs 2024-07-26T16:29:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13092.044377 secs 2024-07-26T16:29:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13092.044377 secs 2024-07-26T16:29:17.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:17.616+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:29:17.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:17.616+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:17.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:17.616+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:29:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:17 smithi112 ceph-mon[37501]: pgmap v7371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:29:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:17 smithi107 ceph-mon[32659]: pgmap v7371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:29:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:17 smithi132 ceph-mon[37580]: pgmap v7371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:29:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:18 smithi107 ceph-mon[32659]: pgmap v7372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:29:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:18 smithi132 ceph-mon[37580]: pgmap v7372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:29:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:18 smithi112 ceph-mon[37501]: pgmap v7372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:29:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13097.044505 secs 2024-07-26T16:29:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:21 smithi112 ceph-mon[37501]: pgmap v7373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13097.044505 secs 2024-07-26T16:29:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:21 smithi107 ceph-mon[32659]: pgmap v7373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13097.044505 secs 2024-07-26T16:29:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:21 smithi132 ceph-mon[37580]: pgmap v7373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:23 smithi112 ceph-mon[37501]: pgmap v7374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:23.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:23 smithi107 ceph-mon[32659]: pgmap v7374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:23 smithi132 ceph-mon[37580]: pgmap v7374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:25 smithi112 ceph-mon[37501]: pgmap v7375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:25 smithi107 ceph-mon[32659]: pgmap v7375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:25 smithi132 ceph-mon[37580]: pgmap v7375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13102.044638 secs 2024-07-26T16:29:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13102.044638 secs 2024-07-26T16:29:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13102.044638 secs 2024-07-26T16:29:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:27 smithi107 ceph-mon[32659]: pgmap v7376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:27 smithi132 ceph-mon[37580]: pgmap v7376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:27 smithi112 ceph-mon[37501]: pgmap v7376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:29 smithi107 ceph-mon[32659]: pgmap v7377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:29 smithi132 ceph-mon[37580]: pgmap v7377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:29 smithi112 ceph-mon[37501]: pgmap v7377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13107.044776 secs 2024-07-26T16:29:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:31 smithi107 ceph-mon[32659]: pgmap v7378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13107.044776 secs 2024-07-26T16:29:31.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:31 smithi132 ceph-mon[37580]: pgmap v7378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13107.044776 secs 2024-07-26T16:29:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:31 smithi112 ceph-mon[37501]: pgmap v7378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:32.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:32.631+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:29:32.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:32.631+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:32.633 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:32.631+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:29:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:33 smithi107 ceph-mon[32659]: pgmap v7379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:33 smithi132 ceph-mon[37580]: pgmap v7379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:33 smithi112 ceph-mon[37501]: pgmap v7379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:35 smithi107 ceph-mon[32659]: pgmap v7380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:35 smithi132 ceph-mon[37580]: pgmap v7380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:35 smithi112 ceph-mon[37501]: pgmap v7380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13112.044913 secs 2024-07-26T16:29:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13112.044913 secs 2024-07-26T16:29:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13112.044913 secs 2024-07-26T16:29:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:37 smithi107 ceph-mon[32659]: pgmap v7381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:37 smithi132 ceph-mon[37580]: pgmap v7381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:37 smithi112 ceph-mon[37501]: pgmap v7381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:39 smithi107 ceph-mon[32659]: pgmap v7382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:39 smithi132 ceph-mon[37580]: pgmap v7382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:39 smithi112 ceph-mon[37501]: pgmap v7382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13117.045011 secs 2024-07-26T16:29:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:41 smithi107 ceph-mon[32659]: pgmap v7383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13117.045011 secs 2024-07-26T16:29:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:41 smithi132 ceph-mon[37580]: pgmap v7383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13117.045011 secs 2024-07-26T16:29:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:41 smithi112 ceph-mon[37501]: pgmap v7383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:43 smithi107 ceph-mon[32659]: pgmap v7384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:43 smithi132 ceph-mon[37580]: pgmap v7384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:43 smithi112 ceph-mon[37501]: pgmap v7384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:45 smithi107 ceph-mon[32659]: pgmap v7385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:45 smithi132 ceph-mon[37580]: pgmap v7385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:45 smithi112 ceph-mon[37501]: pgmap v7385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13122.045129 secs 2024-07-26T16:29:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13122.045129 secs 2024-07-26T16:29:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13122.045129 secs 2024-07-26T16:29:47.644 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:47.642+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:29:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:47.642+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:29:47.645 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:29:47.642+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:47 smithi107 ceph-mon[32659]: pgmap v7386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:47 smithi132 ceph-mon[37580]: pgmap v7386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:47 smithi112 ceph-mon[37501]: pgmap v7386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:49.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:49 smithi107 ceph-mon[32659]: pgmap v7387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:49 smithi132 ceph-mon[37580]: pgmap v7387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:49 smithi112 ceph-mon[37501]: pgmap v7387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:51.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13127.045249 secs 2024-07-26T16:29:51.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:51 smithi132 ceph-mon[37580]: pgmap v7388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13127.045249 secs 2024-07-26T16:29:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:51 smithi112 ceph-mon[37501]: pgmap v7388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13127.045249 secs 2024-07-26T16:29:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:51 smithi107 ceph-mon[32659]: pgmap v7388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:53.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:53 smithi132 ceph-mon[37580]: pgmap v7389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:53 smithi112 ceph-mon[37501]: pgmap v7389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:53 smithi107 ceph-mon[32659]: pgmap v7389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:29:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:55 smithi132 ceph-mon[37580]: pgmap v7390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:55 smithi112 ceph-mon[37501]: pgmap v7390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:55 smithi107 ceph-mon[32659]: pgmap v7390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:29:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13132.045370 secs 2024-07-26T16:29:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13132.045370 secs 2024-07-26T16:29:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13132.045370 secs 2024-07-26T16:29:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:57 smithi112 ceph-mon[37501]: pgmap v7391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:57 smithi107 ceph-mon[32659]: pgmap v7391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:57 smithi132 ceph-mon[37580]: pgmap v7391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:29:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:29:59 smithi112 ceph-mon[37501]: pgmap v7392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:29:59 smithi107 ceph-mon[32659]: pgmap v7392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:29:59 smithi132 ceph-mon[37580]: pgmap v7392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:30:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:30:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:30:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13137.045486 secs 2024-07-26T16:30:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:01 smithi112 ceph-mon[37501]: pgmap v7393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13137.045486 secs 2024-07-26T16:30:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:01 smithi107 ceph-mon[32659]: pgmap v7393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13137.045486 secs 2024-07-26T16:30:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:01 smithi132 ceph-mon[37580]: pgmap v7393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:02.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:02.657+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:30:02.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:02.657+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:30:02.660 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:02.657+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:03 smithi112 ceph-mon[37501]: pgmap v7394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:03 smithi107 ceph-mon[32659]: pgmap v7394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:03 smithi132 ceph-mon[37580]: pgmap v7394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:05 smithi112 ceph-mon[37501]: pgmap v7395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:05 smithi107 ceph-mon[32659]: pgmap v7395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:05 smithi132 ceph-mon[37580]: pgmap v7395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13142.045600 secs 2024-07-26T16:30:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13142.045600 secs 2024-07-26T16:30:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13142.045600 secs 2024-07-26T16:30:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:07 smithi112 ceph-mon[37501]: pgmap v7396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:07 smithi107 ceph-mon[32659]: pgmap v7396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:07 smithi132 ceph-mon[37580]: pgmap v7396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:30:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:30:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:30:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:09 smithi112 ceph-mon[37501]: pgmap v7397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:30:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:30:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:30:10.018 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:09 smithi132 ceph-mon[37580]: pgmap v7397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:10.018 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:30:10.018 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:30:10.018 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:30:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:09 smithi107 ceph-mon[32659]: pgmap v7397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:30:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:30:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:30:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:10 smithi112 ceph-mon[37501]: pgmap v7398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:30:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:10 smithi112 ceph-mon[37501]: pgmap v7399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:30:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:10 smithi107 ceph-mon[32659]: pgmap v7398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:30:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:10 smithi107 ceph-mon[32659]: pgmap v7399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:30:11.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:10 smithi132 ceph-mon[37580]: pgmap v7398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:30:11.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:10 smithi132 ceph-mon[37580]: pgmap v7399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:30:11.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13147.045727 secs 2024-07-26T16:30:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13147.045727 secs 2024-07-26T16:30:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13147.045727 secs 2024-07-26T16:30:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:12 smithi112 ceph-mon[37501]: pgmap v7400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:12 smithi107 ceph-mon[32659]: pgmap v7400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:12 smithi132 ceph-mon[37580]: pgmap v7400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:14.906 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:14.902+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:30:14.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:14.902+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:30:14.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:14.903+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:30:14.907 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:14.903+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:30:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:14 smithi112 ceph-mon[37501]: pgmap v7401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:30:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:14 smithi107 ceph-mon[32659]: pgmap v7401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:30:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:14 smithi132 ceph-mon[37580]: pgmap v7401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:30:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13152.045867 secs 2024-07-26T16:30:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:16 smithi112 ceph-mon[37501]: pgmap v7402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:30:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13152.045867 secs 2024-07-26T16:30:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:16 smithi107 ceph-mon[32659]: pgmap v7402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:30:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13152.045867 secs 2024-07-26T16:30:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:16 smithi132 ceph-mon[37580]: pgmap v7402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:30:17.675 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:17.673+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:30:17.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:17.673+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:30:17.676 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:17.673+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:18 smithi112 ceph-mon[37501]: pgmap v7403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:30:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:18 smithi107 ceph-mon[32659]: pgmap v7403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:30:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:18 smithi132 ceph-mon[37580]: pgmap v7403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:30:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:20 smithi112 ceph-mon[37501]: pgmap v7404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:30:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:20 smithi107 ceph-mon[32659]: pgmap v7404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:30:21.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:20 smithi132 ceph-mon[37580]: pgmap v7404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:30:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13157.045990 secs 2024-07-26T16:30:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13157.045990 secs 2024-07-26T16:30:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13157.045990 secs 2024-07-26T16:30:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:22 smithi112 ceph-mon[37501]: pgmap v7405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:30:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:22 smithi107 ceph-mon[32659]: pgmap v7405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:30:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:22 smithi132 ceph-mon[37580]: pgmap v7405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:30:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:23 smithi107 ceph-mon[32659]: pgmap v7406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:30:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:23 smithi132 ceph-mon[37580]: pgmap v7406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:30:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:23 smithi112 ceph-mon[37501]: pgmap v7406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:30:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13162.046110 secs 2024-07-26T16:30:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:26 smithi112 ceph-mon[37501]: pgmap v7407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:30:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13162.046110 secs 2024-07-26T16:30:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:26 smithi107 ceph-mon[32659]: pgmap v7407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:30:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13162.046110 secs 2024-07-26T16:30:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:26 smithi132 ceph-mon[37580]: pgmap v7407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:30:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:28 smithi112 ceph-mon[37501]: pgmap v7408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:30:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:28 smithi107 ceph-mon[32659]: pgmap v7408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:30:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:28 smithi132 ceph-mon[37580]: pgmap v7408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:30:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:30 smithi112 ceph-mon[37501]: pgmap v7409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:30 smithi107 ceph-mon[32659]: pgmap v7409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:30 smithi132 ceph-mon[37580]: pgmap v7409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13167.046231 secs 2024-07-26T16:30:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13167.046231 secs 2024-07-26T16:30:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13167.046231 secs 2024-07-26T16:30:32.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:32.689+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:30:32.691 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:32.689+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:30:32.692 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:32.689+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:32.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:32 smithi107 ceph-mon[32659]: pgmap v7410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:30:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:32 smithi112 ceph-mon[37501]: pgmap v7410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:30:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:32 smithi132 ceph-mon[37580]: pgmap v7410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:30:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:34 smithi112 ceph-mon[37501]: pgmap v7411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:34 smithi107 ceph-mon[32659]: pgmap v7411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:34 smithi132 ceph-mon[37580]: pgmap v7411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13172.046352 secs 2024-07-26T16:30:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:36 smithi112 ceph-mon[37501]: pgmap v7412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13172.046352 secs 2024-07-26T16:30:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:36 smithi107 ceph-mon[32659]: pgmap v7412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:37.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13172.046352 secs 2024-07-26T16:30:37.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:36 smithi132 ceph-mon[37580]: pgmap v7412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:38 smithi112 ceph-mon[37501]: pgmap v7413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:38 smithi107 ceph-mon[32659]: pgmap v7413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:38 smithi132 ceph-mon[37580]: pgmap v7413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:39 smithi107 ceph-mon[32659]: pgmap v7414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:39 smithi132 ceph-mon[37580]: pgmap v7414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:39 smithi112 ceph-mon[37501]: pgmap v7414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13177.046471 secs 2024-07-26T16:30:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13177.046471 secs 2024-07-26T16:30:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13177.046471 secs 2024-07-26T16:30:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:42 smithi112 ceph-mon[37501]: pgmap v7415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:30:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:42 smithi107 ceph-mon[32659]: pgmap v7415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:30:42.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:42 smithi132 ceph-mon[37580]: pgmap v7415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:30:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:44 smithi112 ceph-mon[37501]: pgmap v7416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:45.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:44 smithi107 ceph-mon[32659]: pgmap v7416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:44 smithi132 ceph-mon[37580]: pgmap v7416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13182.046532 secs 2024-07-26T16:30:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:46 smithi112 ceph-mon[37501]: pgmap v7417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13182.046532 secs 2024-07-26T16:30:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:46 smithi107 ceph-mon[32659]: pgmap v7417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13182.046532 secs 2024-07-26T16:30:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:46 smithi132 ceph-mon[37580]: pgmap v7417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:47.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:47.705+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:30:47.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:47.705+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:30:47.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:30:47.705+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:48 smithi112 ceph-mon[37501]: pgmap v7418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:49.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:48 smithi107 ceph-mon[32659]: pgmap v7418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:48 smithi132 ceph-mon[37580]: pgmap v7418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:50 smithi112 ceph-mon[37501]: pgmap v7419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:51.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:50 smithi107 ceph-mon[32659]: pgmap v7419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:50 smithi132 ceph-mon[37580]: pgmap v7419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:30:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13187.046649 secs 2024-07-26T16:30:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13187.046649 secs 2024-07-26T16:30:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13187.046649 secs 2024-07-26T16:30:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:52 smithi112 ceph-mon[37501]: pgmap v7420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:30:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:52 smithi107 ceph-mon[32659]: pgmap v7420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:30:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:52 smithi132 ceph-mon[37580]: pgmap v7420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:30:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:53 smithi107 ceph-mon[32659]: pgmap v7421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:53 smithi132 ceph-mon[37580]: pgmap v7421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:53 smithi112 ceph-mon[37501]: pgmap v7421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13192.046770 secs 2024-07-26T16:30:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:56 smithi112 ceph-mon[37501]: pgmap v7422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13192.046770 secs 2024-07-26T16:30:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:56 smithi107 ceph-mon[32659]: pgmap v7422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13192.046770 secs 2024-07-26T16:30:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:56 smithi132 ceph-mon[37580]: pgmap v7422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:30:58 smithi112 ceph-mon[37501]: pgmap v7423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:59.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:30:58 smithi107 ceph-mon[32659]: pgmap v7423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:30:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:30:58 smithi132 ceph-mon[37580]: pgmap v7423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:00 smithi112 ceph-mon[37501]: pgmap v7424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:00 smithi107 ceph-mon[32659]: pgmap v7424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:00 smithi132 ceph-mon[37580]: pgmap v7424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13197.046887 secs 2024-07-26T16:31:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13197.046887 secs 2024-07-26T16:31:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13197.046887 secs 2024-07-26T16:31:02.722 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:02.719+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:31:02.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:02.719+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:31:02.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:02.719+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:02.945 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:02 smithi107 ceph-mon[32659]: pgmap v7425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:02 smithi112 ceph-mon[37501]: pgmap v7425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:02 smithi132 ceph-mon[37580]: pgmap v7425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:03 smithi107 ceph-mon[32659]: pgmap v7426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:03 smithi132 ceph-mon[37580]: pgmap v7426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:03 smithi112 ceph-mon[37501]: pgmap v7426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13202.047027 secs 2024-07-26T16:31:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:06 smithi112 ceph-mon[37501]: pgmap v7427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13202.047027 secs 2024-07-26T16:31:07.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:06 smithi107 ceph-mon[32659]: pgmap v7427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13202.047027 secs 2024-07-26T16:31:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:06 smithi132 ceph-mon[37580]: pgmap v7427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:08 smithi112 ceph-mon[37501]: pgmap v7428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:09.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:08 smithi107 ceph-mon[32659]: pgmap v7428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:08 smithi132 ceph-mon[37580]: pgmap v7428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:10 smithi112 ceph-mon[37501]: pgmap v7429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:31:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:31:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:31:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:10 smithi107 ceph-mon[32659]: pgmap v7429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:31:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:31:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:31:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:10 smithi132 ceph-mon[37580]: pgmap v7429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:11.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:31:11.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:31:11.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:31:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13207.047152 secs 2024-07-26T16:31:11.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:31:11.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:31:11.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:31:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13207.047152 secs 2024-07-26T16:31:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:31:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:31:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:31:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13207.047152 secs 2024-07-26T16:31:12.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:31:12.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:31:12.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:31:13.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:12 smithi107 ceph-mon[32659]: pgmap v7430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T16:31:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:12 smithi132 ceph-mon[37580]: pgmap v7430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T16:31:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:12 smithi112 ceph-mon[37501]: pgmap v7430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T16:31:14.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:13 smithi107 ceph-mon[32659]: pgmap v7431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:31:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:13 smithi132 ceph-mon[37580]: pgmap v7431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:31:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:13 smithi112 ceph-mon[37501]: pgmap v7431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:31:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:16 smithi112 ceph-mon[37501]: pgmap v7432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:31:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13212.047275 secs 2024-07-26T16:31:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:16 smithi107 ceph-mon[32659]: pgmap v7432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:31:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13212.047275 secs 2024-07-26T16:31:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:16 smithi132 ceph-mon[37580]: pgmap v7432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:31:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13212.047275 secs 2024-07-26T16:31:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:17.735+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:31:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:17.735+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:31:17.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:17.735+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:18 smithi112 ceph-mon[37501]: pgmap v7433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:31:18.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:18 smithi107 ceph-mon[32659]: pgmap v7433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:31:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:18 smithi132 ceph-mon[37580]: pgmap v7433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:31:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:20 smithi112 ceph-mon[37501]: pgmap v7434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:31:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:20 smithi107 ceph-mon[32659]: pgmap v7434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:31:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:20 smithi132 ceph-mon[37580]: pgmap v7434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:31:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13217.047387 secs 2024-07-26T16:31:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13217.047387 secs 2024-07-26T16:31:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13217.047387 secs 2024-07-26T16:31:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:22 smithi112 ceph-mon[37501]: pgmap v7435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T16:31:22.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:22 smithi107 ceph-mon[32659]: pgmap v7435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T16:31:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:22 smithi132 ceph-mon[37580]: pgmap v7435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T16:31:24.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:24 smithi107 ceph-mon[32659]: pgmap v7436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:24 smithi132 ceph-mon[37580]: pgmap v7436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:24 smithi112 ceph-mon[37501]: pgmap v7436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:26 smithi107 ceph-mon[32659]: pgmap v7437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13222.047511 secs 2024-07-26T16:31:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:26 smithi132 ceph-mon[37580]: pgmap v7437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:26.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13222.047511 secs 2024-07-26T16:31:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:26 smithi112 ceph-mon[37501]: pgmap v7437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13222.047511 secs 2024-07-26T16:31:28.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:28 smithi107 ceph-mon[32659]: pgmap v7438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:28 smithi132 ceph-mon[37580]: pgmap v7438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:28 smithi112 ceph-mon[37501]: pgmap v7438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:30 smithi107 ceph-mon[32659]: pgmap v7439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:31:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:30 smithi132 ceph-mon[37580]: pgmap v7439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:31:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:30 smithi112 ceph-mon[37501]: pgmap v7439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:31:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13227.047636 secs 2024-07-26T16:31:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13227.047636 secs 2024-07-26T16:31:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13227.047636 secs 2024-07-26T16:31:32.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:32.740+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:31:32.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:32.740+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:32.743 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:32.740+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:31:32.743 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:32 smithi107 ceph-mon[32659]: pgmap v7440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:32 smithi132 ceph-mon[37580]: pgmap v7440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:32 smithi112 ceph-mon[37501]: pgmap v7440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:34 smithi107 ceph-mon[32659]: pgmap v7441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:34 smithi132 ceph-mon[37580]: pgmap v7441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:34 smithi112 ceph-mon[37501]: pgmap v7441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:36.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:36 smithi107 ceph-mon[32659]: pgmap v7442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:36.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13232.047771 secs 2024-07-26T16:31:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:36 smithi132 ceph-mon[37580]: pgmap v7442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13232.047771 secs 2024-07-26T16:31:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:36 smithi112 ceph-mon[37501]: pgmap v7442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13232.047771 secs 2024-07-26T16:31:38.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:38 smithi107 ceph-mon[32659]: pgmap v7443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:38 smithi132 ceph-mon[37580]: pgmap v7443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:38 smithi112 ceph-mon[37501]: pgmap v7443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:40 smithi107 ceph-mon[32659]: pgmap v7444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:31:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:40 smithi132 ceph-mon[37580]: pgmap v7444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:31:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:40 smithi112 ceph-mon[37501]: pgmap v7444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:31:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13237.047907 secs 2024-07-26T16:31:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13237.047907 secs 2024-07-26T16:31:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13237.047907 secs 2024-07-26T16:31:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:42 smithi107 ceph-mon[32659]: pgmap v7445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:42 smithi132 ceph-mon[37580]: pgmap v7445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:42 smithi112 ceph-mon[37501]: pgmap v7445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:44.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:44 smithi107 ceph-mon[32659]: pgmap v7446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:44 smithi132 ceph-mon[37580]: pgmap v7446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:44 smithi112 ceph-mon[37501]: pgmap v7446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:46 smithi107 ceph-mon[32659]: pgmap v7447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13242.047979 secs 2024-07-26T16:31:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:46 smithi132 ceph-mon[37580]: pgmap v7447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13242.047979 secs 2024-07-26T16:31:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:46 smithi112 ceph-mon[37501]: pgmap v7447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13242.047979 secs 2024-07-26T16:31:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:47.756+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:31:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:47.756+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:31:47.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:31:47.756+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:48.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:48 smithi107 ceph-mon[32659]: pgmap v7448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:48 smithi132 ceph-mon[37580]: pgmap v7448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:48 smithi112 ceph-mon[37501]: pgmap v7448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:50 smithi107 ceph-mon[32659]: pgmap v7449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:31:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:50 smithi132 ceph-mon[37580]: pgmap v7449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:31:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:50 smithi112 ceph-mon[37501]: pgmap v7449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:31:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13247.048098 secs 2024-07-26T16:31:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13247.048098 secs 2024-07-26T16:31:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13247.048098 secs 2024-07-26T16:31:52.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:52 smithi132 ceph-mon[37580]: pgmap v7450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:52 smithi112 ceph-mon[37501]: pgmap v7450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:52 smithi107 ceph-mon[32659]: pgmap v7450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:54 smithi112 ceph-mon[37501]: pgmap v7451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:54 smithi107 ceph-mon[32659]: pgmap v7451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:54 smithi132 ceph-mon[37580]: pgmap v7451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13252.048211 secs 2024-07-26T16:31:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:56 smithi112 ceph-mon[37501]: pgmap v7452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13252.048211 secs 2024-07-26T16:31:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:56 smithi107 ceph-mon[32659]: pgmap v7452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13252.048211 secs 2024-07-26T16:31:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:56 smithi132 ceph-mon[37580]: pgmap v7452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:31:58 smithi112 ceph-mon[37501]: pgmap v7453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:59.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:31:58 smithi107 ceph-mon[32659]: pgmap v7453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:31:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:31:58 smithi132 ceph-mon[37580]: pgmap v7453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:00 smithi112 ceph-mon[37501]: pgmap v7454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:32:00.987 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:00 smithi107 ceph-mon[32659]: pgmap v7454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:32:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:00 smithi132 ceph-mon[37580]: pgmap v7454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:32:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13257.048321 secs 2024-07-26T16:32:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13257.048321 secs 2024-07-26T16:32:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13257.048321 secs 2024-07-26T16:32:02.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:02.770+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:32:02.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:02.770+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:02.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:02.770+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:32:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:02 smithi112 ceph-mon[37501]: pgmap v7455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:02 smithi107 ceph-mon[32659]: pgmap v7455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:02 smithi132 ceph-mon[37580]: pgmap v7455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:04 smithi112 ceph-mon[37501]: pgmap v7456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:04 smithi107 ceph-mon[32659]: pgmap v7456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:04 smithi132 ceph-mon[37580]: pgmap v7456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:06.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13262.048442 secs 2024-07-26T16:32:06.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:06 smithi132 ceph-mon[37580]: pgmap v7457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13262.048442 secs 2024-07-26T16:32:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:06 smithi112 ceph-mon[37501]: pgmap v7457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13262.048442 secs 2024-07-26T16:32:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:06 smithi107 ceph-mon[32659]: pgmap v7457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:08 smithi112 ceph-mon[37501]: pgmap v7458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:09.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:08 smithi107 ceph-mon[32659]: pgmap v7458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:08 smithi132 ceph-mon[37580]: pgmap v7458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:10 smithi112 ceph-mon[37501]: pgmap v7459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:32:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:10 smithi107 ceph-mon[32659]: pgmap v7459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:32:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:10 smithi132 ceph-mon[37580]: pgmap v7459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:32:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13267.048517 secs 2024-07-26T16:32:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:32:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13267.048517 secs 2024-07-26T16:32:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:32:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13267.048517 secs 2024-07-26T16:32:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:32:12.792 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:12 smithi132 ceph-mon[37580]: pgmap v7460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:12.792 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:12.793 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:12.793 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:12.793 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:12 smithi112 ceph-mon[37501]: pgmap v7460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:12.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:12.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:12.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:12.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:13.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:12 smithi107 ceph-mon[32659]: pgmap v7460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:13.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:13.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:13.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:13.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:14 smithi112 ceph-mon[37501]: pgmap v7461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:14 smithi107 ceph-mon[32659]: pgmap v7461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:14.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:14 smithi132 ceph-mon[37580]: pgmap v7461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:14.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:16 smithi107 ceph-mon[32659]: pgmap v7462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:16.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:16 smithi132 ceph-mon[37580]: pgmap v7462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:16 smithi112 ceph-mon[37501]: pgmap v7462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13272.048643 secs 2024-07-26T16:32:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:32:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:32:17.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:17 smithi112 ceph-mon[37501]: pgmap v7463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:17.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13272.048643 secs 2024-07-26T16:32:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:32:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:32:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:17 smithi107 ceph-mon[32659]: pgmap v7463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13272.048643 secs 2024-07-26T16:32:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:32:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:32:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:17 smithi132 ceph-mon[37580]: pgmap v7463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:32:17.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:17.786+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:32:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:17.786+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:17.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:17.786+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:32:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:19 smithi112 ceph-mon[37501]: pgmap v7464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:19 smithi107 ceph-mon[32659]: pgmap v7464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:19 smithi132 ceph-mon[37580]: pgmap v7464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:21 smithi112 ceph-mon[37501]: pgmap v7465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13277.048762 secs 2024-07-26T16:32:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:21 smithi107 ceph-mon[32659]: pgmap v7465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13277.048762 secs 2024-07-26T16:32:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:21 smithi132 ceph-mon[37580]: pgmap v7465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13277.048762 secs 2024-07-26T16:32:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:23 smithi112 ceph-mon[37501]: pgmap v7466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:32:23.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:23 smithi107 ceph-mon[32659]: pgmap v7466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:32:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:23 smithi132 ceph-mon[37580]: pgmap v7466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:32:25.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:25 smithi112 ceph-mon[37501]: pgmap v7467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:25 smithi107 ceph-mon[32659]: pgmap v7467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:25 smithi132 ceph-mon[37580]: pgmap v7467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13282.048888 secs 2024-07-26T16:32:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13282.048888 secs 2024-07-26T16:32:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13282.048888 secs 2024-07-26T16:32:27.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:27 smithi112 ceph-mon[37501]: pgmap v7468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:27.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:27 smithi107 ceph-mon[32659]: pgmap v7468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:27 smithi132 ceph-mon[37580]: pgmap v7468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:29.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:29 smithi107 ceph-mon[32659]: pgmap v7469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:29 smithi132 ceph-mon[37580]: pgmap v7469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:29 smithi112 ceph-mon[37501]: pgmap v7469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:31 smithi107 ceph-mon[32659]: pgmap v7470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13287.049005 secs 2024-07-26T16:32:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:31 smithi132 ceph-mon[37580]: pgmap v7470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13287.049005 secs 2024-07-26T16:32:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:31 smithi112 ceph-mon[37501]: pgmap v7470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13287.049005 secs 2024-07-26T16:32:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:32.802+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:32:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:32.802+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:32:32.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:32.802+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:33.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:33 smithi107 ceph-mon[32659]: pgmap v7471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:33 smithi132 ceph-mon[37580]: pgmap v7471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:33 smithi112 ceph-mon[37501]: pgmap v7471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:35 smithi107 ceph-mon[32659]: pgmap v7472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:35 smithi132 ceph-mon[37580]: pgmap v7472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:35 smithi112 ceph-mon[37501]: pgmap v7472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13292.049127 secs 2024-07-26T16:32:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13292.049127 secs 2024-07-26T16:32:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13292.049127 secs 2024-07-26T16:32:37.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:37 smithi107 ceph-mon[32659]: pgmap v7473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:37 smithi132 ceph-mon[37580]: pgmap v7473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:37 smithi112 ceph-mon[37501]: pgmap v7473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:39 smithi107 ceph-mon[32659]: pgmap v7474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:39 smithi132 ceph-mon[37580]: pgmap v7474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:39 smithi112 ceph-mon[37501]: pgmap v7474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:41 smithi107 ceph-mon[32659]: pgmap v7475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13297.049249 secs 2024-07-26T16:32:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:41 smithi132 ceph-mon[37580]: pgmap v7475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13297.049249 secs 2024-07-26T16:32:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:41 smithi112 ceph-mon[37501]: pgmap v7475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13297.049249 secs 2024-07-26T16:32:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:43 smithi107 ceph-mon[32659]: pgmap v7476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:43 smithi132 ceph-mon[37580]: pgmap v7476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:43 smithi112 ceph-mon[37501]: pgmap v7476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:45.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:45 smithi107 ceph-mon[32659]: pgmap v7477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:45 smithi132 ceph-mon[37580]: pgmap v7477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:45 smithi112 ceph-mon[37501]: pgmap v7477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13302.049367 secs 2024-07-26T16:32:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13302.049367 secs 2024-07-26T16:32:46.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13302.049367 secs 2024-07-26T16:32:47.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:47 smithi107 ceph-mon[32659]: pgmap v7478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:47 smithi132 ceph-mon[37580]: pgmap v7478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:47 smithi112 ceph-mon[37501]: pgmap v7478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:47.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:47.809+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:32:47.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:47.810+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:32:47.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:32:47.810+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:49 smithi107 ceph-mon[32659]: pgmap v7479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:49 smithi132 ceph-mon[37580]: pgmap v7479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:49 smithi112 ceph-mon[37501]: pgmap v7479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:51 smithi107 ceph-mon[32659]: pgmap v7480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13307.049485 secs 2024-07-26T16:32:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:51 smithi132 ceph-mon[37580]: pgmap v7480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:51.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13307.049485 secs 2024-07-26T16:32:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:51 smithi112 ceph-mon[37501]: pgmap v7480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13307.049485 secs 2024-07-26T16:32:53.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:53 smithi107 ceph-mon[32659]: pgmap v7481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:53 smithi132 ceph-mon[37580]: pgmap v7481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:53 smithi112 ceph-mon[37501]: pgmap v7481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:32:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:55 smithi107 ceph-mon[32659]: pgmap v7482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:55 smithi132 ceph-mon[37580]: pgmap v7482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:55 smithi112 ceph-mon[37501]: pgmap v7482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13312.049611 secs 2024-07-26T16:32:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13312.049611 secs 2024-07-26T16:32:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13312.049611 secs 2024-07-26T16:32:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:57 smithi132 ceph-mon[37580]: pgmap v7483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:57 smithi112 ceph-mon[37501]: pgmap v7483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:57.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:57 smithi107 ceph-mon[32659]: pgmap v7483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:32:59 smithi112 ceph-mon[37501]: pgmap v7484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:32:59 smithi107 ceph-mon[32659]: pgmap v7484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:32:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:32:59 smithi132 ceph-mon[37580]: pgmap v7484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:01 smithi112 ceph-mon[37501]: pgmap v7485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13317.049714 secs 2024-07-26T16:33:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:01 smithi107 ceph-mon[32659]: pgmap v7485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13317.049714 secs 2024-07-26T16:33:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:01 smithi132 ceph-mon[37580]: pgmap v7485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13317.049714 secs 2024-07-26T16:33:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:02.824+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:33:02.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:02.824+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:02.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:02.824+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:33:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:03 smithi112 ceph-mon[37501]: pgmap v7486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:03 smithi107 ceph-mon[32659]: pgmap v7486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:03 smithi132 ceph-mon[37580]: pgmap v7486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:05 smithi112 ceph-mon[37501]: pgmap v7487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:05.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:05 smithi107 ceph-mon[32659]: pgmap v7487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:05 smithi132 ceph-mon[37580]: pgmap v7487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13322.049832 secs 2024-07-26T16:33:06.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13322.049832 secs 2024-07-26T16:33:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13322.049832 secs 2024-07-26T16:33:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:07 smithi112 ceph-mon[37501]: pgmap v7488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:07.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:07 smithi107 ceph-mon[32659]: pgmap v7488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:07 smithi132 ceph-mon[37580]: pgmap v7488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:09 smithi112 ceph-mon[37501]: pgmap v7489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:09 smithi107 ceph-mon[32659]: pgmap v7489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:09 smithi132 ceph-mon[37580]: pgmap v7489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:11 smithi112 ceph-mon[37501]: pgmap v7490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13327.049953 secs 2024-07-26T16:33:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:11 smithi107 ceph-mon[32659]: pgmap v7490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13327.049953 secs 2024-07-26T16:33:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:11 smithi132 ceph-mon[37580]: pgmap v7490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13327.049953 secs 2024-07-26T16:33:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:13 smithi112 ceph-mon[37501]: pgmap v7491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:13.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:13 smithi107 ceph-mon[32659]: pgmap v7491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:13 smithi132 ceph-mon[37580]: pgmap v7491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:33:14.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:14.935+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:33:14.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:14.935+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:33:14.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:14.935+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:33:14.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:14.935+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:33:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:15 smithi112 ceph-mon[37501]: pgmap v7492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:15 smithi107 ceph-mon[32659]: pgmap v7492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:15 smithi132 ceph-mon[37580]: pgmap v7492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13332.050094 secs 2024-07-26T16:33:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:33:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13332.050094 secs 2024-07-26T16:33:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:33:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13332.050094 secs 2024-07-26T16:33:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:33:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:17 smithi112 ceph-mon[37501]: pgmap v7493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:33:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:33:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:33:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:17 smithi107 ceph-mon[32659]: pgmap v7493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:33:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:33:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:33:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:17 smithi132 ceph-mon[37580]: pgmap v7493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:33:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:33:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:33:17.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:17.840+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:33:17.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:17.840+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:33:17.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:17.840+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:18 smithi112 ceph-mon[37501]: pgmap v7494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:33:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:18 smithi112 ceph-mon[37501]: pgmap v7495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:18.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:18 smithi107 ceph-mon[32659]: pgmap v7494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:33:18.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:18 smithi107 ceph-mon[32659]: pgmap v7495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:18 smithi132 ceph-mon[37580]: pgmap v7494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:33:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:18 smithi132 ceph-mon[37580]: pgmap v7495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:20 smithi112 ceph-mon[37501]: pgmap v7496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:20 smithi107 ceph-mon[32659]: pgmap v7496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:20 smithi132 ceph-mon[37580]: pgmap v7496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13337.050226 secs 2024-07-26T16:33:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13337.050226 secs 2024-07-26T16:33:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13337.050226 secs 2024-07-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:22 smithi112 ceph-mon[37501]: pgmap v7497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:33:22.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:22 smithi107 ceph-mon[32659]: pgmap v7497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:33:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:22 smithi132 ceph-mon[37580]: pgmap v7497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:33:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:24 smithi112 ceph-mon[37501]: pgmap v7498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:24.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:24 smithi107 ceph-mon[32659]: pgmap v7498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:24 smithi132 ceph-mon[37580]: pgmap v7498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:26 smithi112 ceph-mon[37501]: pgmap v7499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13342.050355 secs 2024-07-26T16:33:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:26 smithi107 ceph-mon[32659]: pgmap v7499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13342.050355 secs 2024-07-26T16:33:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:26 smithi132 ceph-mon[37580]: pgmap v7499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13342.050355 secs 2024-07-26T16:33:28.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:28 smithi112 ceph-mon[37501]: pgmap v7500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:33:28.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:28 smithi107 ceph-mon[32659]: pgmap v7500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:33:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:28 smithi132 ceph-mon[37580]: pgmap v7500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:33:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:30 smithi112 ceph-mon[37501]: pgmap v7501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:33:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:30 smithi107 ceph-mon[32659]: pgmap v7501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:33:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:30 smithi132 ceph-mon[37580]: pgmap v7501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:33:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13347.050479 secs 2024-07-26T16:33:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13347.050479 secs 2024-07-26T16:33:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13347.050479 secs 2024-07-26T16:33:32.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:32 smithi107 ceph-mon[32659]: pgmap v7502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:32 smithi132 ceph-mon[37580]: pgmap v7502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:32.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:32.842+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:33:32.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:32.842+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:33:32.844 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:32.842+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:32 smithi112 ceph-mon[37501]: pgmap v7502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:34 smithi107 ceph-mon[32659]: pgmap v7503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:33:34.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:34 smithi132 ceph-mon[37580]: pgmap v7503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:33:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:34 smithi112 ceph-mon[37501]: pgmap v7503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:33:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:36 smithi107 ceph-mon[32659]: pgmap v7504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:33:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13352.050558 secs 2024-07-26T16:33:36.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:36 smithi132 ceph-mon[37580]: pgmap v7504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:33:36.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13352.050558 secs 2024-07-26T16:33:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:36 smithi112 ceph-mon[37501]: pgmap v7504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:33:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13352.050558 secs 2024-07-26T16:33:38.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:38 smithi107 ceph-mon[32659]: pgmap v7505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:33:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:38 smithi132 ceph-mon[37580]: pgmap v7505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:33:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:38 smithi112 ceph-mon[37501]: pgmap v7505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:33:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:40 smithi107 ceph-mon[32659]: pgmap v7506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:33:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:40 smithi132 ceph-mon[37580]: pgmap v7506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:33:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:40 smithi112 ceph-mon[37501]: pgmap v7506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:33:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13357.050682 secs 2024-07-26T16:33:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13357.050682 secs 2024-07-26T16:33:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13357.050682 secs 2024-07-26T16:33:42.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:42 smithi107 ceph-mon[32659]: pgmap v7507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:33:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:42 smithi132 ceph-mon[37580]: pgmap v7507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:33:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:42 smithi112 ceph-mon[37501]: pgmap v7507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:33:44.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:44 smithi107 ceph-mon[32659]: pgmap v7508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:33:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:44 smithi132 ceph-mon[37580]: pgmap v7508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:33:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:44 smithi112 ceph-mon[37501]: pgmap v7508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:33:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:46 smithi107 ceph-mon[32659]: pgmap v7509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13362.050828 secs 2024-07-26T16:33:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:46 smithi132 ceph-mon[37580]: pgmap v7509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13362.050828 secs 2024-07-26T16:33:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:46 smithi112 ceph-mon[37501]: pgmap v7509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13362.050828 secs 2024-07-26T16:33:47.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:47.849+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:33:47.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:47.849+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:47.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:47.849+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:33:48.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:48 smithi107 ceph-mon[32659]: pgmap v7510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:48 smithi132 ceph-mon[37580]: pgmap v7510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:48 smithi112 ceph-mon[37501]: pgmap v7510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:50 smithi107 ceph-mon[32659]: pgmap v7511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:33:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:50 smithi132 ceph-mon[37580]: pgmap v7511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:33:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:50 smithi112 ceph-mon[37501]: pgmap v7511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:33:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13367.050972 secs 2024-07-26T16:33:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13367.050972 secs 2024-07-26T16:33:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13367.050972 secs 2024-07-26T16:33:52.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:52 smithi107 ceph-mon[32659]: pgmap v7512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:52 smithi132 ceph-mon[37580]: pgmap v7512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:52 smithi112 ceph-mon[37501]: pgmap v7512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:54.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:54 smithi107 ceph-mon[32659]: pgmap v7513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:54 smithi132 ceph-mon[37580]: pgmap v7513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:54 smithi112 ceph-mon[37501]: pgmap v7513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:56 smithi107 ceph-mon[32659]: pgmap v7514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13372.051095 secs 2024-07-26T16:33:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:56 smithi132 ceph-mon[37580]: pgmap v7514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13372.051095 secs 2024-07-26T16:33:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:56 smithi112 ceph-mon[37501]: pgmap v7514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13372.051095 secs 2024-07-26T16:33:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:33:58 smithi132 ceph-mon[37580]: pgmap v7515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:58.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:58.943+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:33:58.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:58.943+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:33:58.946 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:33:58 smithi107 ceph-mon[32659]: pgmap v7515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:33:58.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:58.944+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:33:58.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:33:58.944+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:33:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:33:58 smithi112 ceph-mon[37501]: pgmap v7515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:00.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:00 smithi132 ceph-mon[37580]: pgmap v7516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:00 smithi112 ceph-mon[37501]: pgmap v7516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:00 smithi107 ceph-mon[32659]: pgmap v7516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13377.051221 secs 2024-07-26T16:34:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13377.051221 secs 2024-07-26T16:34:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13377.051221 secs 2024-07-26T16:34:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:02.864+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:34:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:02.864+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:34:02.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:02.864+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:02.867 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:02 smithi107 ceph-mon[32659]: pgmap v7517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:02 smithi112 ceph-mon[37501]: pgmap v7517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:02 smithi132 ceph-mon[37580]: pgmap v7517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:04.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:04 smithi107 ceph-mon[32659]: pgmap v7518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:04 smithi112 ceph-mon[37501]: pgmap v7518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:04 smithi132 ceph-mon[37580]: pgmap v7518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:06.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:06 smithi132 ceph-mon[37580]: pgmap v7519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:06.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13382.051292 secs 2024-07-26T16:34:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:06 smithi112 ceph-mon[37501]: pgmap v7519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13382.051292 secs 2024-07-26T16:34:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:06 smithi107 ceph-mon[32659]: pgmap v7519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13382.051292 secs 2024-07-26T16:34:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:08 smithi112 ceph-mon[37501]: pgmap v7520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:09.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:08 smithi107 ceph-mon[32659]: pgmap v7520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:09.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:08 smithi132 ceph-mon[37580]: pgmap v7520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:10 smithi112 ceph-mon[37501]: pgmap v7521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:10 smithi107 ceph-mon[32659]: pgmap v7521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:10 smithi132 ceph-mon[37580]: pgmap v7521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13387.051423 secs 2024-07-26T16:34:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13387.051423 secs 2024-07-26T16:34:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13387.051423 secs 2024-07-26T16:34:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:12 smithi112 ceph-mon[37501]: pgmap v7522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:12 smithi107 ceph-mon[32659]: pgmap v7522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:12 smithi132 ceph-mon[37580]: pgmap v7522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:14 smithi112 ceph-mon[37501]: pgmap v7523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:14 smithi107 ceph-mon[32659]: pgmap v7523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:14 smithi132 ceph-mon[37580]: pgmap v7523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:16 smithi112 ceph-mon[37501]: pgmap v7524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13392.051529 secs 2024-07-26T16:34:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:16 smithi107 ceph-mon[32659]: pgmap v7524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13392.051529 secs 2024-07-26T16:34:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:16 smithi132 ceph-mon[37580]: pgmap v7524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13392.051529 secs 2024-07-26T16:34:17.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:17.879+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:34:17.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:17.879+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:34:17.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:17.879+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:17.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:34:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:34:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:34:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:18 smithi112 ceph-mon[37501]: pgmap v7525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:34:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:34:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:34:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:18 smithi107 ceph-mon[32659]: pgmap v7525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:34:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:34:19.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:34:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:18 smithi132 ceph-mon[37580]: pgmap v7525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:19.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:34:19.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:34:19.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:34:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:19 smithi112 ceph-mon[37501]: pgmap v7526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:34:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:19 smithi112 ceph-mon[37501]: pgmap v7527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:34:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:19 smithi107 ceph-mon[32659]: pgmap v7526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:34:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:19 smithi107 ceph-mon[32659]: pgmap v7527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:34:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:19 smithi132 ceph-mon[37580]: pgmap v7526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:34:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:19 smithi132 ceph-mon[37580]: pgmap v7527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:34:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13397.051659 secs 2024-07-26T16:34:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:21 smithi112 ceph-mon[37501]: pgmap v7528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13397.051659 secs 2024-07-26T16:34:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:21 smithi107 ceph-mon[32659]: pgmap v7528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13397.051659 secs 2024-07-26T16:34:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:21 smithi132 ceph-mon[37580]: pgmap v7528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:23 smithi112 ceph-mon[37501]: pgmap v7529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:34:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:23 smithi107 ceph-mon[32659]: pgmap v7529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:34:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:23 smithi132 ceph-mon[37580]: pgmap v7529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:34:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:25 smithi112 ceph-mon[37501]: pgmap v7530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:25 smithi107 ceph-mon[32659]: pgmap v7530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:25 smithi132 ceph-mon[37580]: pgmap v7530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13402.051722 secs 2024-07-26T16:34:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13402.051722 secs 2024-07-26T16:34:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13402.051722 secs 2024-07-26T16:34:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:27 smithi112 ceph-mon[37501]: pgmap v7531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:34:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:27 smithi107 ceph-mon[32659]: pgmap v7531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:34:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:27 smithi132 ceph-mon[37580]: pgmap v7531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:34:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:29 smithi112 ceph-mon[37501]: pgmap v7532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:34:30.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:29 smithi107 ceph-mon[32659]: pgmap v7532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:34:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:29 smithi132 ceph-mon[37580]: pgmap v7532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:34:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13407.051846 secs 2024-07-26T16:34:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:31 smithi112 ceph-mon[37501]: pgmap v7533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13407.051846 secs 2024-07-26T16:34:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:31 smithi107 ceph-mon[32659]: pgmap v7533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13407.051846 secs 2024-07-26T16:34:32.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:31 smithi132 ceph-mon[37580]: pgmap v7533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:32.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:32.896+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:34:32.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:32.896+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:34:32.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:32.896+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:33 smithi112 ceph-mon[37501]: pgmap v7534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:33 smithi107 ceph-mon[32659]: pgmap v7534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:33 smithi132 ceph-mon[37580]: pgmap v7534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:35 smithi112 ceph-mon[37501]: pgmap v7535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:35 smithi107 ceph-mon[32659]: pgmap v7535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:35 smithi132 ceph-mon[37580]: pgmap v7535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13412.051997 secs 2024-07-26T16:34:36.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13412.051997 secs 2024-07-26T16:34:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13412.051997 secs 2024-07-26T16:34:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:37 smithi107 ceph-mon[32659]: pgmap v7536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:37 smithi132 ceph-mon[37580]: pgmap v7536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:37 smithi112 ceph-mon[37501]: pgmap v7536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:39 smithi107 ceph-mon[32659]: pgmap v7537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:39 smithi132 ceph-mon[37580]: pgmap v7537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:39 smithi112 ceph-mon[37501]: pgmap v7537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13417.052148 secs 2024-07-26T16:34:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:41 smithi107 ceph-mon[32659]: pgmap v7538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13417.052148 secs 2024-07-26T16:34:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:41 smithi132 ceph-mon[37580]: pgmap v7538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13417.052148 secs 2024-07-26T16:34:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:41 smithi112 ceph-mon[37501]: pgmap v7538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:43 smithi107 ceph-mon[32659]: pgmap v7539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:43 smithi132 ceph-mon[37580]: pgmap v7539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:43 smithi112 ceph-mon[37501]: pgmap v7539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:46.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:45 smithi112 ceph-mon[37501]: pgmap v7540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:45 smithi107 ceph-mon[32659]: pgmap v7540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:45 smithi132 ceph-mon[37580]: pgmap v7540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13422.052266 secs 2024-07-26T16:34:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13422.052266 secs 2024-07-26T16:34:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13422.052266 secs 2024-07-26T16:34:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:47.906+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:34:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:47.906+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:34:47.908 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:34:47.906+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:47 smithi107 ceph-mon[32659]: pgmap v7541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:47 smithi132 ceph-mon[37580]: pgmap v7541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:48.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:47 smithi112 ceph-mon[37501]: pgmap v7541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:49 smithi107 ceph-mon[32659]: pgmap v7542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:49 smithi132 ceph-mon[37580]: pgmap v7542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:49 smithi112 ceph-mon[37501]: pgmap v7542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:50 smithi132 ceph-mon[37580]: pgmap v7543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:50 smithi112 ceph-mon[37501]: pgmap v7543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:50 smithi107 ceph-mon[32659]: pgmap v7543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13427.052335 secs 2024-07-26T16:34:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13427.052335 secs 2024-07-26T16:34:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13427.052335 secs 2024-07-26T16:34:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:52 smithi112 ceph-mon[37501]: pgmap v7544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:52 smithi107 ceph-mon[32659]: pgmap v7544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:52 smithi132 ceph-mon[37580]: pgmap v7544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:55 smithi112 ceph-mon[37501]: pgmap v7545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:55 smithi107 ceph-mon[32659]: pgmap v7545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:55 smithi132 ceph-mon[37580]: pgmap v7545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:34:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13432.052468 secs 2024-07-26T16:34:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13432.052468 secs 2024-07-26T16:34:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13432.052468 secs 2024-07-26T16:34:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:57 smithi112 ceph-mon[37501]: pgmap v7546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:58.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:57 smithi107 ceph-mon[32659]: pgmap v7546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:57 smithi132 ceph-mon[37580]: pgmap v7546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:34:59 smithi112 ceph-mon[37501]: pgmap v7547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:34:59 smithi107 ceph-mon[32659]: pgmap v7547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:34:59 smithi132 ceph-mon[37580]: pgmap v7547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13437.052592 secs 2024-07-26T16:35:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:01 smithi112 ceph-mon[37501]: pgmap v7548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13437.052592 secs 2024-07-26T16:35:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:01 smithi107 ceph-mon[32659]: pgmap v7548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13437.052592 secs 2024-07-26T16:35:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:01 smithi132 ceph-mon[37580]: pgmap v7548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:02.920+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:35:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:02.920+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:35:02.923 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:02.920+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:03 smithi112 ceph-mon[37501]: pgmap v7549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:03 smithi107 ceph-mon[32659]: pgmap v7549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:03 smithi132 ceph-mon[37580]: pgmap v7549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:05 smithi112 ceph-mon[37501]: pgmap v7550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:35:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:05 smithi107 ceph-mon[32659]: pgmap v7550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:35:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:05 smithi132 ceph-mon[37580]: pgmap v7550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:35:06.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13442.052727 secs 2024-07-26T16:35:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13442.052727 secs 2024-07-26T16:35:06.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13442.052727 secs 2024-07-26T16:35:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:07 smithi112 ceph-mon[37501]: pgmap v7551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:35:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:07 smithi107 ceph-mon[32659]: pgmap v7551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:35:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:07 smithi132 ceph-mon[37580]: pgmap v7551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:35:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:09 smithi112 ceph-mon[37501]: pgmap v7552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:09 smithi107 ceph-mon[32659]: pgmap v7552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:09 smithi132 ceph-mon[37580]: pgmap v7552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13447.052843 secs 2024-07-26T16:35:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:11 smithi112 ceph-mon[37501]: pgmap v7553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13447.052843 secs 2024-07-26T16:35:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:11 smithi107 ceph-mon[32659]: pgmap v7553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13447.052843 secs 2024-07-26T16:35:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:11 smithi132 ceph-mon[37580]: pgmap v7553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:13 smithi112 ceph-mon[37501]: pgmap v7554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:14.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:13 smithi107 ceph-mon[32659]: pgmap v7554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:13 smithi132 ceph-mon[37580]: pgmap v7554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:15 smithi112 ceph-mon[37501]: pgmap v7555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:35:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:15 smithi107 ceph-mon[32659]: pgmap v7555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:35:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:15 smithi132 ceph-mon[37580]: pgmap v7555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:35:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13452.052969 secs 2024-07-26T16:35:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13452.052969 secs 2024-07-26T16:35:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13452.052969 secs 2024-07-26T16:35:17.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:17.936+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:35:17.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:17.936+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:35:17.939 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:17.936+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:17.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:17 smithi107 ceph-mon[32659]: pgmap v7556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:17 smithi112 ceph-mon[37501]: pgmap v7556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:17 smithi132 ceph-mon[37580]: pgmap v7556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:18 smithi112 ceph-mon[37501]: pgmap v7557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:35:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:18 smithi107 ceph-mon[32659]: pgmap v7557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:35:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:18 smithi132 ceph-mon[37580]: pgmap v7557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:35:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:35:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:35:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:19 smithi112 ceph-mon[37501]: pgmap v7558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:35:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:19 smithi112 ceph-mon[37501]: pgmap v7559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:35:20.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:35:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:35:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:35:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:19 smithi107 ceph-mon[32659]: pgmap v7558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:35:20.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:19 smithi107 ceph-mon[32659]: pgmap v7559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:35:20.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:35:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:35:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:35:20.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:19 smithi132 ceph-mon[37580]: pgmap v7558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:35:20.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:19 smithi132 ceph-mon[37580]: pgmap v7559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:35:20.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:35:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13457.053115 secs 2024-07-26T16:35:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13457.053115 secs 2024-07-26T16:35:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13457.053115 secs 2024-07-26T16:35:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:22 smithi107 ceph-mon[32659]: pgmap v7560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:22 smithi132 ceph-mon[37580]: pgmap v7560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:22 smithi112 ceph-mon[37501]: pgmap v7560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:23 smithi112 ceph-mon[37501]: pgmap v7561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:35:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:23 smithi107 ceph-mon[32659]: pgmap v7561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:35:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:23 smithi132 ceph-mon[37580]: pgmap v7561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:35:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13462.053233 secs 2024-07-26T16:35:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:26 smithi107 ceph-mon[32659]: pgmap v7562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:35:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13462.053233 secs 2024-07-26T16:35:27.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:26 smithi132 ceph-mon[37580]: pgmap v7562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:35:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13462.053233 secs 2024-07-26T16:35:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:26 smithi112 ceph-mon[37501]: pgmap v7562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:35:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:27 smithi132 ceph-mon[37580]: pgmap v7563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:35:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:27 smithi112 ceph-mon[37501]: pgmap v7563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:35:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:27 smithi107 ceph-mon[32659]: pgmap v7563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:35:31.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:30 smithi107 ceph-mon[32659]: pgmap v7564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:35:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:30 smithi132 ceph-mon[37580]: pgmap v7564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:35:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:30 smithi112 ceph-mon[37501]: pgmap v7564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:35:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13467.053295 secs 2024-07-26T16:35:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:31 smithi107 ceph-mon[32659]: pgmap v7565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:35:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13467.053295 secs 2024-07-26T16:35:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:31 smithi132 ceph-mon[37580]: pgmap v7565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:35:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13467.053295 secs 2024-07-26T16:35:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:31 smithi112 ceph-mon[37501]: pgmap v7565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:35:32.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:32.945+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:35:32.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:32.945+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:35:32.947 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:32.945+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:34 smithi107 ceph-mon[32659]: pgmap v7566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:34 smithi132 ceph-mon[37580]: pgmap v7566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:34 smithi112 ceph-mon[37501]: pgmap v7566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:35 smithi107 ceph-mon[32659]: pgmap v7567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:35 smithi132 ceph-mon[37580]: pgmap v7567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:35 smithi112 ceph-mon[37501]: pgmap v7567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13472.053425 secs 2024-07-26T16:35:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13472.053425 secs 2024-07-26T16:35:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13472.053425 secs 2024-07-26T16:35:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:37 smithi107 ceph-mon[32659]: pgmap v7568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:37 smithi132 ceph-mon[37580]: pgmap v7568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:37 smithi112 ceph-mon[37501]: pgmap v7568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:39 smithi132 ceph-mon[37580]: pgmap v7569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:39 smithi112 ceph-mon[37501]: pgmap v7569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:39 smithi107 ceph-mon[32659]: pgmap v7569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13477.053552 secs 2024-07-26T16:35:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13477.053552 secs 2024-07-26T16:35:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13477.053552 secs 2024-07-26T16:35:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:42 smithi107 ceph-mon[32659]: pgmap v7570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:42 smithi132 ceph-mon[37580]: pgmap v7570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:42 smithi112 ceph-mon[37501]: pgmap v7570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:43 smithi132 ceph-mon[37580]: pgmap v7571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:43 smithi112 ceph-mon[37501]: pgmap v7571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:43 smithi107 ceph-mon[32659]: pgmap v7571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:45 smithi132 ceph-mon[37580]: pgmap v7572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:45 smithi112 ceph-mon[37501]: pgmap v7572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:45 smithi107 ceph-mon[32659]: pgmap v7572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13482.053694 secs 2024-07-26T16:35:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13482.053694 secs 2024-07-26T16:35:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13482.053694 secs 2024-07-26T16:35:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:47.960+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:35:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:47.960+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:35:47.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:35:47.961+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:48.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:47 smithi112 ceph-mon[37501]: pgmap v7573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:47 smithi107 ceph-mon[32659]: pgmap v7573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:47 smithi132 ceph-mon[37580]: pgmap v7573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:49 smithi132 ceph-mon[37580]: pgmap v7574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:49 smithi112 ceph-mon[37501]: pgmap v7574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:50.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:49 smithi107 ceph-mon[32659]: pgmap v7574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13487.053815 secs 2024-07-26T16:35:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13487.053815 secs 2024-07-26T16:35:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13487.053815 secs 2024-07-26T16:35:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:52 smithi107 ceph-mon[32659]: pgmap v7575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:52 smithi132 ceph-mon[37580]: pgmap v7575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:52 smithi112 ceph-mon[37501]: pgmap v7575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:53 smithi112 ceph-mon[37501]: pgmap v7576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:54.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:53 smithi107 ceph-mon[32659]: pgmap v7576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:53 smithi132 ceph-mon[37580]: pgmap v7576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:55 smithi112 ceph-mon[37501]: pgmap v7577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:55 smithi107 ceph-mon[32659]: pgmap v7577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:55 smithi132 ceph-mon[37580]: pgmap v7577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13492.053918 secs 2024-07-26T16:35:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13492.053918 secs 2024-07-26T16:35:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13492.053918 secs 2024-07-26T16:35:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:57 smithi112 ceph-mon[37501]: pgmap v7578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:58.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:57 smithi107 ceph-mon[32659]: pgmap v7578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:57 smithi132 ceph-mon[37580]: pgmap v7578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:35:59 smithi112 ceph-mon[37501]: pgmap v7579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:35:59 smithi107 ceph-mon[32659]: pgmap v7579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:35:59 smithi132 ceph-mon[37580]: pgmap v7579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13497.054051 secs 2024-07-26T16:36:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13497.054051 secs 2024-07-26T16:36:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13497.054051 secs 2024-07-26T16:36:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:02 smithi107 ceph-mon[32659]: pgmap v7580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:02 smithi132 ceph-mon[37580]: pgmap v7580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:02 smithi112 ceph-mon[37501]: pgmap v7580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:02.975+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:36:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:02.975+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:02.975+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:36:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:03 smithi112 ceph-mon[37501]: pgmap v7581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:04.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:03 smithi107 ceph-mon[32659]: pgmap v7581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:03 smithi132 ceph-mon[37580]: pgmap v7581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:06.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:05 smithi107 ceph-mon[32659]: pgmap v7582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:06.114 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:05 smithi132 ceph-mon[37580]: pgmap v7582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:05 smithi112 ceph-mon[37501]: pgmap v7582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13502.054165 secs 2024-07-26T16:36:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13502.054165 secs 2024-07-26T16:36:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13502.054165 secs 2024-07-26T16:36:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:07 smithi112 ceph-mon[37501]: pgmap v7583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:07 smithi107 ceph-mon[32659]: pgmap v7583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:07 smithi132 ceph-mon[37580]: pgmap v7583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:09 smithi112 ceph-mon[37501]: pgmap v7584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:09 smithi107 ceph-mon[32659]: pgmap v7584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:09 smithi132 ceph-mon[37580]: pgmap v7584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13507.054279 secs 2024-07-26T16:36:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13507.054279 secs 2024-07-26T16:36:11.682 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:11.679+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T16:36:11.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:11.679+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T16:36:11.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:11.679+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8014050 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:36:11.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:11.679+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8014050 comp rx=0 tx=0).stop 2024-07-26T16:36:11.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:11.679+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T16:36:11.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:11.679+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:36:11.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:11.679+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:36:11.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:11.679+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:36:11.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:11.680+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:36:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:11.682+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f806f040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:36:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13507.054279 secs 2024-07-26T16:36:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:12 smithi107 ceph-mon[32659]: pgmap v7585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:12.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:12 smithi132 ceph-mon[37580]: pgmap v7585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:12 smithi112 ceph-mon[37501]: pgmap v7585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:13 smithi112 ceph-mon[37501]: pgmap v7586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:13 smithi107 ceph-mon[32659]: pgmap v7586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:13 smithi132 ceph-mon[37580]: pgmap v7586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:14.967+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:36:14.969 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:14.967+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:36:14.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:14.967+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:36:14.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:14.967+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:36:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:15 smithi112 ceph-mon[37501]: pgmap v7587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:15 smithi107 ceph-mon[32659]: pgmap v7587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:15 smithi132 ceph-mon[37580]: pgmap v7587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13512.054395 secs 2024-07-26T16:36:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13512.054395 secs 2024-07-26T16:36:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13512.054395 secs 2024-07-26T16:36:17.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:17.982+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:36:17.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:17.982+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:17.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:17.982+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:36:18.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:17 smithi112 ceph-mon[37501]: pgmap v7588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:17 smithi107 ceph-mon[32659]: pgmap v7588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:17 smithi132 ceph-mon[37580]: pgmap v7588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:19 smithi112 ceph-mon[37501]: pgmap v7589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:19 smithi107 ceph-mon[32659]: pgmap v7589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:19 smithi132 ceph-mon[37580]: pgmap v7589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:36:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:36:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:36:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13517.054484 secs 2024-07-26T16:36:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:36:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:36:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:21 smithi112 ceph-mon[37501]: pgmap v7590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T16:36:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:21 smithi112 ceph-mon[37501]: pgmap v7591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:36:22.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:36:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13517.054484 secs 2024-07-26T16:36:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:36:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:36:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:21 smithi107 ceph-mon[32659]: pgmap v7590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T16:36:22.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:21 smithi107 ceph-mon[32659]: pgmap v7591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:36:22.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:36:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13517.054484 secs 2024-07-26T16:36:22.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:36:22.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:36:22.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:21 smithi132 ceph-mon[37580]: pgmap v7590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T16:36:22.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:21 smithi132 ceph-mon[37580]: pgmap v7591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:36:22.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:36:24.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:23 smithi107 ceph-mon[32659]: pgmap v7592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:36:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:23 smithi132 ceph-mon[37580]: pgmap v7592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:36:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:23 smithi112 ceph-mon[37501]: pgmap v7592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:36:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:25 smithi107 ceph-mon[32659]: pgmap v7593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:36:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:25 smithi132 ceph-mon[37580]: pgmap v7593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:36:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:25 smithi112 ceph-mon[37501]: pgmap v7593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:36:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13522.054580 secs 2024-07-26T16:36:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13522.054580 secs 2024-07-26T16:36:27.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13522.054580 secs 2024-07-26T16:36:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:27 smithi107 ceph-mon[32659]: pgmap v7594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:36:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:27 smithi132 ceph-mon[37580]: pgmap v7594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:36:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:27 smithi112 ceph-mon[37501]: pgmap v7594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:36:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:29 smithi107 ceph-mon[32659]: pgmap v7595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:36:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:29 smithi132 ceph-mon[37580]: pgmap v7595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:36:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:29 smithi112 ceph-mon[37501]: pgmap v7595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:36:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13527.054715 secs 2024-07-26T16:36:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:31 smithi107 ceph-mon[32659]: pgmap v7596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:36:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13527.054715 secs 2024-07-26T16:36:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:31 smithi132 ceph-mon[37580]: pgmap v7596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:36:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13527.054715 secs 2024-07-26T16:36:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:31 smithi112 ceph-mon[37501]: pgmap v7596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:36:33.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:32.999+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:36:33.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:32.999+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:33.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:32.999+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:36:34.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:33 smithi107 ceph-mon[32659]: pgmap v7597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:33 smithi132 ceph-mon[37580]: pgmap v7597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:33 smithi112 ceph-mon[37501]: pgmap v7597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:36.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:35 smithi107 ceph-mon[32659]: pgmap v7598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:35 smithi132 ceph-mon[37580]: pgmap v7598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:35 smithi112 ceph-mon[37501]: pgmap v7598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:37.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13532.054861 secs 2024-07-26T16:36:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13532.054861 secs 2024-07-26T16:36:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13532.054861 secs 2024-07-26T16:36:38.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:38 smithi107 ceph-mon[32659]: pgmap v7599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:38 smithi132 ceph-mon[37580]: pgmap v7599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:38 smithi112 ceph-mon[37501]: pgmap v7599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:40 smithi107 ceph-mon[32659]: pgmap v7600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:40 smithi132 ceph-mon[37580]: pgmap v7600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:40 smithi112 ceph-mon[37501]: pgmap v7600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13537.055009 secs 2024-07-26T16:36:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:42 smithi107 ceph-mon[32659]: pgmap v7601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:36:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13537.055009 secs 2024-07-26T16:36:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:42 smithi132 ceph-mon[37580]: pgmap v7601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:36:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13537.055009 secs 2024-07-26T16:36:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:42 smithi112 ceph-mon[37501]: pgmap v7601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:36:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:44 smithi107 ceph-mon[32659]: pgmap v7602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:44 smithi132 ceph-mon[37580]: pgmap v7602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:44 smithi112 ceph-mon[37501]: pgmap v7602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:46 smithi107 ceph-mon[32659]: pgmap v7603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:36:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:46 smithi132 ceph-mon[37580]: pgmap v7603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:36:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:46 smithi112 ceph-mon[37501]: pgmap v7603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:36:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13542.055142 secs 2024-07-26T16:36:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13542.055142 secs 2024-07-26T16:36:47.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13542.055142 secs 2024-07-26T16:36:48.016 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:48.014+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:36:48.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:48.014+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:48.017 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:36:48.014+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:36:48.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:48 smithi107 ceph-mon[32659]: pgmap v7604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:36:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:48 smithi132 ceph-mon[37580]: pgmap v7604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:36:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:48 smithi112 ceph-mon[37501]: pgmap v7604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:36:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:50 smithi132 ceph-mon[37580]: pgmap v7605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:50 smithi112 ceph-mon[37501]: pgmap v7605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:50.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:50 smithi107 ceph-mon[32659]: pgmap v7605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13547.055260 secs 2024-07-26T16:36:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:52 smithi112 ceph-mon[37501]: pgmap v7606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13547.055260 secs 2024-07-26T16:36:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:52 smithi107 ceph-mon[32659]: pgmap v7606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13547.055260 secs 2024-07-26T16:36:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:52 smithi132 ceph-mon[37580]: pgmap v7606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:54 smithi112 ceph-mon[37501]: pgmap v7607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:54 smithi107 ceph-mon[32659]: pgmap v7607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:54.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:54 smithi132 ceph-mon[37580]: pgmap v7607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:56 smithi112 ceph-mon[37501]: pgmap v7608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:56 smithi107 ceph-mon[32659]: pgmap v7608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:56 smithi132 ceph-mon[37580]: pgmap v7608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13552.055323 secs 2024-07-26T16:36:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13552.055323 secs 2024-07-26T16:36:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13552.055323 secs 2024-07-26T16:36:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:36:58 smithi112 ceph-mon[37501]: pgmap v7609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:36:58 smithi107 ceph-mon[32659]: pgmap v7609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:36:58 smithi132 ceph-mon[37580]: pgmap v7609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:00 smithi112 ceph-mon[37501]: pgmap v7610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:00 smithi107 ceph-mon[32659]: pgmap v7610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:00 smithi132 ceph-mon[37580]: pgmap v7610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13557.055445 secs 2024-07-26T16:37:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:02 smithi112 ceph-mon[37501]: pgmap v7611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13557.055445 secs 2024-07-26T16:37:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:02 smithi107 ceph-mon[32659]: pgmap v7611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13557.055445 secs 2024-07-26T16:37:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:02 smithi132 ceph-mon[37580]: pgmap v7611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:03.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:03.029+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:37:03.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:03.029+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:37:03.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:03.029+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:04 smithi112 ceph-mon[37501]: pgmap v7612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:04 smithi107 ceph-mon[32659]: pgmap v7612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:04 smithi132 ceph-mon[37580]: pgmap v7612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:06 smithi112 ceph-mon[37501]: pgmap v7613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:06 smithi107 ceph-mon[32659]: pgmap v7613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:06 smithi132 ceph-mon[37580]: pgmap v7613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13562.055558 secs 2024-07-26T16:37:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13562.055558 secs 2024-07-26T16:37:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13562.055558 secs 2024-07-26T16:37:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:08 smithi112 ceph-mon[37501]: pgmap v7614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:08 smithi107 ceph-mon[32659]: pgmap v7614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:08 smithi132 ceph-mon[37580]: pgmap v7614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:10 smithi112 ceph-mon[37501]: pgmap v7615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:10 smithi107 ceph-mon[32659]: pgmap v7615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:10 smithi132 ceph-mon[37580]: pgmap v7615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13567.055655 secs 2024-07-26T16:37:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:12 smithi112 ceph-mon[37501]: pgmap v7616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13567.055655 secs 2024-07-26T16:37:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:12 smithi107 ceph-mon[32659]: pgmap v7616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13567.055655 secs 2024-07-26T16:37:12.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:12 smithi132 ceph-mon[37580]: pgmap v7616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:14 smithi112 ceph-mon[37501]: pgmap v7617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:14 smithi107 ceph-mon[32659]: pgmap v7617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:14 smithi132 ceph-mon[37580]: pgmap v7617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:16 smithi112 ceph-mon[37501]: pgmap v7618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:16 smithi107 ceph-mon[32659]: pgmap v7618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:16 smithi132 ceph-mon[37580]: pgmap v7618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13572.055772 secs 2024-07-26T16:37:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13572.055772 secs 2024-07-26T16:37:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13572.055772 secs 2024-07-26T16:37:18.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:18.045+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:37:18.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:18.045+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:37:18.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:18.045+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:18 smithi112 ceph-mon[37501]: pgmap v7619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:18.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:18 smithi107 ceph-mon[32659]: pgmap v7619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:18 smithi132 ceph-mon[37580]: pgmap v7619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:20 smithi112 ceph-mon[37501]: pgmap v7620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:20 smithi107 ceph-mon[32659]: pgmap v7620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:20 smithi132 ceph-mon[37580]: pgmap v7620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13577.055865 secs 2024-07-26T16:37:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:37:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13577.055865 secs 2024-07-26T16:37:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:37:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13577.055865 secs 2024-07-26T16:37:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:37:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:22 smithi112 ceph-mon[37501]: pgmap v7621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:37:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:37:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:37:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:22 smithi107 ceph-mon[32659]: pgmap v7621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:37:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:37:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:37:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:22 smithi132 ceph-mon[37580]: pgmap v7621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:37:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:37:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:37:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:23 smithi112 ceph-mon[37501]: pgmap v7622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:37:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:23 smithi112 ceph-mon[37501]: pgmap v7623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:23.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:23 smithi107 ceph-mon[32659]: pgmap v7622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:37:23.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:23 smithi107 ceph-mon[32659]: pgmap v7623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:23 smithi132 ceph-mon[37580]: pgmap v7622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:37:23.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:23 smithi132 ceph-mon[37580]: pgmap v7623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:25 smithi107 ceph-mon[32659]: pgmap v7624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:37:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:25 smithi132 ceph-mon[37580]: pgmap v7624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:37:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:25 smithi112 ceph-mon[37501]: pgmap v7624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:37:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13582.056004 secs 2024-07-26T16:37:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13582.056004 secs 2024-07-26T16:37:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13582.056004 secs 2024-07-26T16:37:27.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:27 smithi107 ceph-mon[32659]: pgmap v7625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:37:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:27 smithi132 ceph-mon[37580]: pgmap v7625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:37:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:27 smithi112 ceph-mon[37501]: pgmap v7625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:37:29.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:29 smithi107 ceph-mon[32659]: pgmap v7626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:37:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:29 smithi132 ceph-mon[37580]: pgmap v7626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:37:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:29 smithi112 ceph-mon[37501]: pgmap v7626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:37:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:31 smithi107 ceph-mon[32659]: pgmap v7627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:37:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13587.056105 secs 2024-07-26T16:37:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:31 smithi132 ceph-mon[37580]: pgmap v7627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:37:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13587.056105 secs 2024-07-26T16:37:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:31 smithi112 ceph-mon[37501]: pgmap v7627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:37:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13587.056105 secs 2024-07-26T16:37:33.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:33.061+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:37:33.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:33.061+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:33.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:33.061+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:37:33.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:33 smithi107 ceph-mon[32659]: pgmap v7628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:37:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:33 smithi132 ceph-mon[37580]: pgmap v7628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:37:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:33 smithi112 ceph-mon[37501]: pgmap v7628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:37:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:35 smithi107 ceph-mon[32659]: pgmap v7629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:35 smithi132 ceph-mon[37580]: pgmap v7629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:35 smithi112 ceph-mon[37501]: pgmap v7629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13592.056218 secs 2024-07-26T16:37:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13592.056218 secs 2024-07-26T16:37:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13592.056218 secs 2024-07-26T16:37:37.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:37 smithi107 ceph-mon[32659]: pgmap v7630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:37 smithi132 ceph-mon[37580]: pgmap v7630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:37 smithi112 ceph-mon[37501]: pgmap v7630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:39.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:39 smithi107 ceph-mon[32659]: pgmap v7631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:39 smithi132 ceph-mon[37580]: pgmap v7631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:39 smithi112 ceph-mon[37501]: pgmap v7631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:41 smithi107 ceph-mon[32659]: pgmap v7632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13597.056341 secs 2024-07-26T16:37:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:41 smithi132 ceph-mon[37580]: pgmap v7632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13597.056341 secs 2024-07-26T16:37:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:41 smithi112 ceph-mon[37501]: pgmap v7632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13597.056341 secs 2024-07-26T16:37:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:43 smithi107 ceph-mon[32659]: pgmap v7633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:43 smithi132 ceph-mon[37580]: pgmap v7633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:43 smithi112 ceph-mon[37501]: pgmap v7633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:45.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:45 smithi107 ceph-mon[32659]: pgmap v7634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:45 smithi132 ceph-mon[37580]: pgmap v7634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:45 smithi112 ceph-mon[37501]: pgmap v7634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13602.056427 secs 2024-07-26T16:37:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13602.056427 secs 2024-07-26T16:37:46.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13602.056427 secs 2024-07-26T16:37:47.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:47 smithi107 ceph-mon[32659]: pgmap v7635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:47 smithi132 ceph-mon[37580]: pgmap v7635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:47 smithi112 ceph-mon[37501]: pgmap v7635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:48.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:48.076+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:37:48.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:48.077+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:37:48.079 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:37:48.077+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:49 smithi132 ceph-mon[37580]: pgmap v7636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:49 smithi112 ceph-mon[37501]: pgmap v7636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:49 smithi107 ceph-mon[32659]: pgmap v7636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:51 smithi132 ceph-mon[37580]: pgmap v7637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:51.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13607.056510 secs 2024-07-26T16:37:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:51 smithi112 ceph-mon[37501]: pgmap v7637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13607.056510 secs 2024-07-26T16:37:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:51 smithi107 ceph-mon[32659]: pgmap v7637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13607.056510 secs 2024-07-26T16:37:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:53 smithi132 ceph-mon[37580]: pgmap v7638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:53 smithi112 ceph-mon[37501]: pgmap v7638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:53 smithi107 ceph-mon[32659]: pgmap v7638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:55 smithi112 ceph-mon[37501]: pgmap v7639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:55 smithi107 ceph-mon[32659]: pgmap v7639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:55 smithi132 ceph-mon[37580]: pgmap v7639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13612.056631 secs 2024-07-26T16:37:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13612.056631 secs 2024-07-26T16:37:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13612.056631 secs 2024-07-26T16:37:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:57 smithi112 ceph-mon[37501]: pgmap v7640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:57 smithi107 ceph-mon[32659]: pgmap v7640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:57 smithi132 ceph-mon[37580]: pgmap v7640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:37:59 smithi112 ceph-mon[37501]: pgmap v7641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:37:59 smithi107 ceph-mon[32659]: pgmap v7641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:37:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:37:59 smithi132 ceph-mon[37580]: pgmap v7641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:01 smithi112 ceph-mon[37501]: pgmap v7642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13617.056773 secs 2024-07-26T16:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:01 smithi107 ceph-mon[32659]: pgmap v7642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13617.056773 secs 2024-07-26T16:38:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:01 smithi132 ceph-mon[37580]: pgmap v7642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13617.056773 secs 2024-07-26T16:38:03.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:03.091+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:38:03.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:03.091+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:03.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:03.091+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:38:03.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:03 smithi107 ceph-mon[32659]: pgmap v7643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:03 smithi112 ceph-mon[37501]: pgmap v7643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:03 smithi132 ceph-mon[37580]: pgmap v7643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:05 smithi112 ceph-mon[37501]: pgmap v7644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:05.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:05 smithi107 ceph-mon[32659]: pgmap v7644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:05 smithi132 ceph-mon[37580]: pgmap v7644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:06.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13622.056888 secs 2024-07-26T16:38:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13622.056888 secs 2024-07-26T16:38:06.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13622.056888 secs 2024-07-26T16:38:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:07 smithi112 ceph-mon[37501]: pgmap v7645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:07.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:07 smithi107 ceph-mon[32659]: pgmap v7645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:07 smithi132 ceph-mon[37580]: pgmap v7645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:09 smithi112 ceph-mon[37501]: pgmap v7646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:09 smithi107 ceph-mon[32659]: pgmap v7646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:09 smithi132 ceph-mon[37580]: pgmap v7646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:11 smithi112 ceph-mon[37501]: pgmap v7647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13627.057010 secs 2024-07-26T16:38:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:11 smithi107 ceph-mon[32659]: pgmap v7647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13627.057010 secs 2024-07-26T16:38:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:11 smithi132 ceph-mon[37580]: pgmap v7647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13627.057010 secs 2024-07-26T16:38:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:13 smithi112 ceph-mon[37501]: pgmap v7648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:13.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:13 smithi107 ceph-mon[32659]: pgmap v7648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:13 smithi132 ceph-mon[37580]: pgmap v7648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:15 smithi112 ceph-mon[37501]: pgmap v7649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:15 smithi107 ceph-mon[32659]: pgmap v7649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:15 smithi132 ceph-mon[37580]: pgmap v7649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13632.057125 secs 2024-07-26T16:38:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13632.057125 secs 2024-07-26T16:38:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13632.057125 secs 2024-07-26T16:38:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:17 smithi112 ceph-mon[37501]: pgmap v7650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:17 smithi107 ceph-mon[32659]: pgmap v7650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:17 smithi132 ceph-mon[37580]: pgmap v7650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:18.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:18.099+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:38:18.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:18.099+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:38:18.102 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:18.099+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:19 smithi112 ceph-mon[37501]: pgmap v7651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:19 smithi107 ceph-mon[32659]: pgmap v7651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:19 smithi132 ceph-mon[37580]: pgmap v7651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:21 smithi112 ceph-mon[37501]: pgmap v7652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13637.057217 secs 2024-07-26T16:38:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:21 smithi107 ceph-mon[32659]: pgmap v7652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13637.057217 secs 2024-07-26T16:38:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:21 smithi132 ceph-mon[37580]: pgmap v7652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13637.057217 secs 2024-07-26T16:38:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:38:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:38:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:38:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: pgmap v7653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:38:23.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:38:23.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:38:23.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: pgmap v7653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:23.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:38:23.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:38:23.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:38:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: pgmap v7653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:23.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:38:23.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:38:23.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:38:23.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:38:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:24 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:38:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:24 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:38:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:24 smithi112 ceph-mon[37501]: pgmap v7654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:38:24.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:24 smithi112 ceph-mon[37501]: pgmap v7655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:38:24.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:24 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:38:24.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:24 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:38:24.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:24 smithi107 ceph-mon[32659]: pgmap v7654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:38:24.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:24 smithi107 ceph-mon[32659]: pgmap v7655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:38:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:24 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:38:24.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:24 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:38:24.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:24 smithi132 ceph-mon[37580]: pgmap v7654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:38:24.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:24 smithi132 ceph-mon[37580]: pgmap v7655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:38:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:26 smithi112 ceph-mon[37501]: pgmap v7656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:38:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13642.057339 secs 2024-07-26T16:38:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:26 smithi107 ceph-mon[32659]: pgmap v7656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:38:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13642.057339 secs 2024-07-26T16:38:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:26 smithi132 ceph-mon[37580]: pgmap v7656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:38:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13642.057339 secs 2024-07-26T16:38:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:28 smithi112 ceph-mon[37501]: pgmap v7657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:28.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:28 smithi107 ceph-mon[32659]: pgmap v7657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:28 smithi132 ceph-mon[37580]: pgmap v7657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:30 smithi112 ceph-mon[37501]: pgmap v7658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:38:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:30 smithi107 ceph-mon[32659]: pgmap v7658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:38:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:30 smithi132 ceph-mon[37580]: pgmap v7658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:38:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13647.057456 secs 2024-07-26T16:38:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13647.057456 secs 2024-07-26T16:38:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13647.057456 secs 2024-07-26T16:38:32.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:32 smithi107 ceph-mon[32659]: pgmap v7659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:38:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:32 smithi132 ceph-mon[37580]: pgmap v7659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:38:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:32 smithi112 ceph-mon[37501]: pgmap v7659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:38:33.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:33.113+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:38:33.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:33.113+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:33.115 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:33.113+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:38:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:34 smithi107 ceph-mon[32659]: pgmap v7660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:38:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:34 smithi132 ceph-mon[37580]: pgmap v7660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:38:34.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:34 smithi112 ceph-mon[37501]: pgmap v7660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:38:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:36 smithi107 ceph-mon[32659]: pgmap v7661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13652.057578 secs 2024-07-26T16:38:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:36 smithi132 ceph-mon[37580]: pgmap v7661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13652.057578 secs 2024-07-26T16:38:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:36 smithi112 ceph-mon[37501]: pgmap v7661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13652.057578 secs 2024-07-26T16:38:38.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:38 smithi107 ceph-mon[32659]: pgmap v7662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:38 smithi132 ceph-mon[37580]: pgmap v7662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:38 smithi112 ceph-mon[37501]: pgmap v7662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:40 smithi107 ceph-mon[32659]: pgmap v7663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:40 smithi132 ceph-mon[37580]: pgmap v7663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:40 smithi112 ceph-mon[37501]: pgmap v7663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13657.057698 secs 2024-07-26T16:38:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13657.057698 secs 2024-07-26T16:38:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13657.057698 secs 2024-07-26T16:38:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:42 smithi107 ceph-mon[32659]: pgmap v7664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:42 smithi132 ceph-mon[37580]: pgmap v7664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:42 smithi112 ceph-mon[37501]: pgmap v7664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:44.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:44 smithi107 ceph-mon[32659]: pgmap v7665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:44 smithi132 ceph-mon[37580]: pgmap v7665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:44 smithi112 ceph-mon[37501]: pgmap v7665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:46 smithi107 ceph-mon[32659]: pgmap v7666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13662.057817 secs 2024-07-26T16:38:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:46 smithi132 ceph-mon[37580]: pgmap v7666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13662.057817 secs 2024-07-26T16:38:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:46 smithi112 ceph-mon[37501]: pgmap v7666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13662.057817 secs 2024-07-26T16:38:48.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:48.114+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:38:48.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:48.114+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:48.116 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:38:48.114+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:38:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:48 smithi107 ceph-mon[32659]: pgmap v7667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:48 smithi132 ceph-mon[37580]: pgmap v7667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:48 smithi112 ceph-mon[37501]: pgmap v7667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:50 smithi107 ceph-mon[32659]: pgmap v7668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:50 smithi132 ceph-mon[37580]: pgmap v7668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:50 smithi112 ceph-mon[37501]: pgmap v7668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:38:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13667.057920 secs 2024-07-26T16:38:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13667.057920 secs 2024-07-26T16:38:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13667.057920 secs 2024-07-26T16:38:52.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:52 smithi107 ceph-mon[32659]: pgmap v7669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:52 smithi132 ceph-mon[37580]: pgmap v7669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:52 smithi112 ceph-mon[37501]: pgmap v7669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:54.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:54 smithi107 ceph-mon[32659]: pgmap v7670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:54 smithi132 ceph-mon[37580]: pgmap v7670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:54 smithi112 ceph-mon[37501]: pgmap v7670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:56 smithi132 ceph-mon[37580]: pgmap v7671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:56.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13672.058090 secs 2024-07-26T16:38:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:56 smithi112 ceph-mon[37501]: pgmap v7671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13672.058090 secs 2024-07-26T16:38:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:56 smithi107 ceph-mon[32659]: pgmap v7671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13672.058090 secs 2024-07-26T16:38:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:38:58 smithi132 ceph-mon[37580]: pgmap v7672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:38:58 smithi112 ceph-mon[37501]: pgmap v7672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:38:59.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:38:58 smithi107 ceph-mon[32659]: pgmap v7672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:00 smithi132 ceph-mon[37580]: pgmap v7673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:00 smithi112 ceph-mon[37501]: pgmap v7673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:01.047 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:00 smithi107 ceph-mon[32659]: pgmap v7673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13677.058207 secs 2024-07-26T16:39:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13677.058207 secs 2024-07-26T16:39:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13677.058207 secs 2024-07-26T16:39:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:02 smithi112 ceph-mon[37501]: pgmap v7674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:02 smithi107 ceph-mon[32659]: pgmap v7674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:02 smithi132 ceph-mon[37580]: pgmap v7674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:03.118+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:39:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:03.118+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:03.120 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:03.118+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:39:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:04 smithi112 ceph-mon[37501]: pgmap v7675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:04 smithi107 ceph-mon[32659]: pgmap v7675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:04 smithi132 ceph-mon[37580]: pgmap v7675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:06.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:06 smithi107 ceph-mon[32659]: pgmap v7676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:06.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13682.058326 secs 2024-07-26T16:39:06.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:06 smithi132 ceph-mon[37580]: pgmap v7676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:06.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13682.058326 secs 2024-07-26T16:39:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:06 smithi112 ceph-mon[37501]: pgmap v7676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13682.058326 secs 2024-07-26T16:39:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:08 smithi112 ceph-mon[37501]: pgmap v7677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:09.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:08 smithi107 ceph-mon[32659]: pgmap v7677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:08 smithi132 ceph-mon[37580]: pgmap v7677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:10 smithi112 ceph-mon[37501]: pgmap v7678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:10 smithi107 ceph-mon[32659]: pgmap v7678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:11.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:10 smithi132 ceph-mon[37580]: pgmap v7678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13687.058443 secs 2024-07-26T16:39:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13687.058443 secs 2024-07-26T16:39:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13687.058443 secs 2024-07-26T16:39:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:12 smithi112 ceph-mon[37501]: pgmap v7679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:12 smithi107 ceph-mon[32659]: pgmap v7679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:12 smithi132 ceph-mon[37580]: pgmap v7679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:14 smithi112 ceph-mon[37501]: pgmap v7680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:14.997+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:39:15.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:14.997+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:39:15.001 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:14 smithi107 ceph-mon[32659]: pgmap v7680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:14.998+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:39:15.001 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:14.998+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:39:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:14 smithi132 ceph-mon[37580]: pgmap v7680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:16 smithi112 ceph-mon[37501]: pgmap v7681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13692.058559 secs 2024-07-26T16:39:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:16 smithi107 ceph-mon[32659]: pgmap v7681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13692.058559 secs 2024-07-26T16:39:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:16 smithi132 ceph-mon[37580]: pgmap v7681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13692.058559 secs 2024-07-26T16:39:18.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:18.133+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:39:18.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:18.133+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:18.136 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:18.133+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:39:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:18 smithi112 ceph-mon[37501]: pgmap v7682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:18 smithi107 ceph-mon[32659]: pgmap v7682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:18 smithi132 ceph-mon[37580]: pgmap v7682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:20 smithi112 ceph-mon[37501]: pgmap v7683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:20 smithi107 ceph-mon[32659]: pgmap v7683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:20 smithi132 ceph-mon[37580]: pgmap v7683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13697.058696 secs 2024-07-26T16:39:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13697.058696 secs 2024-07-26T16:39:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13697.058696 secs 2024-07-26T16:39:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:22 smithi112 ceph-mon[37501]: pgmap v7684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:22 smithi107 ceph-mon[32659]: pgmap v7684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:22 smithi132 ceph-mon[37580]: pgmap v7684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:39:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:39:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:39:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: pgmap v7685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:24.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:39:24.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:39:24.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:39:25.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: pgmap v7685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:25.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:39:25.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:39:25.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:39:25.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: pgmap v7685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:25.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:25.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:39:25.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:39:25.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:39:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:25 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:39:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:25 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:39:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:25 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:39:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:25 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:39:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:25 smithi112 ceph-mon[37501]: pgmap v7686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:39:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:25 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:39:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:25 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:39:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:25 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:39:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:25 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:39:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:25 smithi107 ceph-mon[32659]: pgmap v7686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:39:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:25 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:39:26.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:25 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:39:26.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:25 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:39:26.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:25 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:39:26.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:25 smithi132 ceph-mon[37580]: pgmap v7686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:39:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13702.058831 secs 2024-07-26T16:39:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13702.058831 secs 2024-07-26T16:39:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13702.058831 secs 2024-07-26T16:39:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:27 smithi112 ceph-mon[37501]: pgmap v7687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:39:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:27 smithi107 ceph-mon[32659]: pgmap v7687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:39:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:27 smithi132 ceph-mon[37580]: pgmap v7687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:39:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:29 smithi112 ceph-mon[37501]: pgmap v7688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:39:30.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:29 smithi107 ceph-mon[32659]: pgmap v7688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:39:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:29 smithi132 ceph-mon[37580]: pgmap v7688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:39:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13707.058962 secs 2024-07-26T16:39:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:31 smithi107 ceph-mon[32659]: pgmap v7689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:39:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13707.058962 secs 2024-07-26T16:39:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:31 smithi132 ceph-mon[37580]: pgmap v7689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:39:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13707.058962 secs 2024-07-26T16:39:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:31 smithi112 ceph-mon[37501]: pgmap v7689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:39:33.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:33.149+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:39:33.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:33.149+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:33.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:33.149+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:39:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:33 smithi107 ceph-mon[32659]: pgmap v7690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:39:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:33 smithi132 ceph-mon[37580]: pgmap v7690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:39:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:33 smithi112 ceph-mon[37501]: pgmap v7690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:39:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:35 smithi107 ceph-mon[32659]: pgmap v7691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:39:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:35 smithi132 ceph-mon[37580]: pgmap v7691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:39:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:35 smithi112 ceph-mon[37501]: pgmap v7691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:39:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13712.059062 secs 2024-07-26T16:39:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13712.059062 secs 2024-07-26T16:39:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13712.059062 secs 2024-07-26T16:39:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:37 smithi107 ceph-mon[32659]: pgmap v7692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:37 smithi132 ceph-mon[37580]: pgmap v7692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:37 smithi112 ceph-mon[37501]: pgmap v7692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:39 smithi107 ceph-mon[32659]: pgmap v7693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:39 smithi132 ceph-mon[37580]: pgmap v7693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:39 smithi112 ceph-mon[37501]: pgmap v7693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13717.059183 secs 2024-07-26T16:39:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:41 smithi107 ceph-mon[32659]: pgmap v7694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13717.059183 secs 2024-07-26T16:39:42.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:41 smithi132 ceph-mon[37580]: pgmap v7694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13717.059183 secs 2024-07-26T16:39:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:41 smithi112 ceph-mon[37501]: pgmap v7694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:43 smithi107 ceph-mon[32659]: pgmap v7695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:43 smithi132 ceph-mon[37580]: pgmap v7695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:43 smithi112 ceph-mon[37501]: pgmap v7695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:46.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:45 smithi112 ceph-mon[37501]: pgmap v7696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:39:46.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:45 smithi107 ceph-mon[32659]: pgmap v7696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:39:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:45 smithi132 ceph-mon[37580]: pgmap v7696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:39:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13722.059287 secs 2024-07-26T16:39:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13722.059287 secs 2024-07-26T16:39:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13722.059287 secs 2024-07-26T16:39:48.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:47 smithi107 ceph-mon[32659]: pgmap v7697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:47 smithi132 ceph-mon[37580]: pgmap v7697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:48.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:48.151+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:39:48.153 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:48.151+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:39:48.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:39:48.151+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:48.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:47 smithi112 ceph-mon[37501]: pgmap v7697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:49 smithi107 ceph-mon[32659]: pgmap v7698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:49 smithi132 ceph-mon[37580]: pgmap v7698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:49 smithi112 ceph-mon[37501]: pgmap v7698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13727.059405 secs 2024-07-26T16:39:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:51 smithi107 ceph-mon[32659]: pgmap v7699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13727.059405 secs 2024-07-26T16:39:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:51 smithi132 ceph-mon[37580]: pgmap v7699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13727.059405 secs 2024-07-26T16:39:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:51 smithi112 ceph-mon[37501]: pgmap v7699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:53 smithi107 ceph-mon[32659]: pgmap v7700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:53 smithi132 ceph-mon[37580]: pgmap v7700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:53 smithi112 ceph-mon[37501]: pgmap v7700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:55 smithi107 ceph-mon[32659]: pgmap v7701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:39:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:55 smithi132 ceph-mon[37580]: pgmap v7701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:39:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:55 smithi112 ceph-mon[37501]: pgmap v7701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:39:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13732.059491 secs 2024-07-26T16:39:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13732.059491 secs 2024-07-26T16:39:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13732.059491 secs 2024-07-26T16:39:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:57 smithi132 ceph-mon[37580]: pgmap v7702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:57 smithi112 ceph-mon[37501]: pgmap v7702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:58.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:57 smithi107 ceph-mon[32659]: pgmap v7702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:39:58 smithi112 ceph-mon[37501]: pgmap v7703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:39:58 smithi107 ceph-mon[32659]: pgmap v7703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:39:58 smithi132 ceph-mon[37580]: pgmap v7703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:40:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:40:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:40:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:01 smithi112 ceph-mon[37501]: pgmap v7704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:01 smithi107 ceph-mon[32659]: pgmap v7704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:01 smithi132 ceph-mon[37580]: pgmap v7704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13737.059629 secs 2024-07-26T16:40:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13737.059629 secs 2024-07-26T16:40:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13737.059629 secs 2024-07-26T16:40:03.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:03.163+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:40:03.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:03.163+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:40:03.165 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:03.163+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:03 smithi112 ceph-mon[37501]: pgmap v7705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:03 smithi107 ceph-mon[32659]: pgmap v7705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:03 smithi132 ceph-mon[37580]: pgmap v7705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:05 smithi112 ceph-mon[37501]: pgmap v7706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:05 smithi107 ceph-mon[32659]: pgmap v7706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:05 smithi132 ceph-mon[37580]: pgmap v7706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13742.059768 secs 2024-07-26T16:40:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13742.059768 secs 2024-07-26T16:40:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13742.059768 secs 2024-07-26T16:40:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:07 smithi112 ceph-mon[37501]: pgmap v7707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:40:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:07 smithi107 ceph-mon[32659]: pgmap v7707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:40:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:07 smithi132 ceph-mon[37580]: pgmap v7707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:40:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:09 smithi112 ceph-mon[37501]: pgmap v7708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:10.018 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:09 smithi132 ceph-mon[37580]: pgmap v7708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:09 smithi107 ceph-mon[32659]: pgmap v7708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13747.059853 secs 2024-07-26T16:40:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:11 smithi112 ceph-mon[37501]: pgmap v7709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13747.059853 secs 2024-07-26T16:40:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:11 smithi107 ceph-mon[32659]: pgmap v7709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13747.059853 secs 2024-07-26T16:40:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:11 smithi132 ceph-mon[37580]: pgmap v7709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:13 smithi112 ceph-mon[37501]: pgmap v7710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:14.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:13 smithi107 ceph-mon[32659]: pgmap v7710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:13 smithi132 ceph-mon[37580]: pgmap v7710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:15 smithi112 ceph-mon[37501]: pgmap v7711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:15 smithi107 ceph-mon[32659]: pgmap v7711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:15 smithi132 ceph-mon[37580]: pgmap v7711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13752.059934 secs 2024-07-26T16:40:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13752.059934 secs 2024-07-26T16:40:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13752.059934 secs 2024-07-26T16:40:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:17 smithi112 ceph-mon[37501]: pgmap v7712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:40:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:17 smithi107 ceph-mon[32659]: pgmap v7712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:40:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:17 smithi132 ceph-mon[37580]: pgmap v7712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:40:18.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:18.171+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:40:18.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:18.171+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:40:18.178 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:18.171+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:19 smithi112 ceph-mon[37501]: pgmap v7713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:19 smithi107 ceph-mon[32659]: pgmap v7713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:19 smithi132 ceph-mon[37580]: pgmap v7713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13757.060085 secs 2024-07-26T16:40:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:21 smithi112 ceph-mon[37501]: pgmap v7714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13757.060085 secs 2024-07-26T16:40:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:21 smithi107 ceph-mon[32659]: pgmap v7714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13757.060085 secs 2024-07-26T16:40:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:21 smithi132 ceph-mon[37580]: pgmap v7714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:23 smithi112 ceph-mon[37501]: pgmap v7715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:23 smithi107 ceph-mon[32659]: pgmap v7715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:23 smithi132 ceph-mon[37580]: pgmap v7715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:25 smithi112 ceph-mon[37501]: pgmap v7716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:40:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:40:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:40:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:25 smithi107 ceph-mon[32659]: pgmap v7716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:40:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:40:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:40:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:25 smithi132 ceph-mon[37580]: pgmap v7716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:26.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:40:26.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:40:26.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:40:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13762.060201 secs 2024-07-26T16:40:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:26 smithi112 ceph-mon[37501]: pgmap v7717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:40:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:26 smithi112 ceph-mon[37501]: pgmap v7718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:40:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13762.060201 secs 2024-07-26T16:40:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:26 smithi107 ceph-mon[32659]: pgmap v7717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:40:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:26 smithi107 ceph-mon[32659]: pgmap v7718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:40:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13762.060201 secs 2024-07-26T16:40:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:26 smithi132 ceph-mon[37580]: pgmap v7717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:40:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:26 smithi132 ceph-mon[37580]: pgmap v7718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:27.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:40:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:28 smithi112 ceph-mon[37501]: pgmap v7719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:40:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:28 smithi107 ceph-mon[32659]: pgmap v7719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:40:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:28 smithi132 ceph-mon[37580]: pgmap v7719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:40:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:30 smithi112 ceph-mon[37501]: pgmap v7720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:30 smithi107 ceph-mon[32659]: pgmap v7720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:30 smithi132 ceph-mon[37580]: pgmap v7720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13767.060320 secs 2024-07-26T16:40:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13767.060320 secs 2024-07-26T16:40:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13767.060320 secs 2024-07-26T16:40:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:32 smithi112 ceph-mon[37501]: pgmap v7721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:32.987 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:32 smithi107 ceph-mon[32659]: pgmap v7721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:32 smithi132 ceph-mon[37580]: pgmap v7721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:33.186+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:40:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:33.186+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:33.186+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:40:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:34 smithi107 ceph-mon[32659]: pgmap v7722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:40:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:34 smithi132 ceph-mon[37580]: pgmap v7722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:40:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:34 smithi112 ceph-mon[37501]: pgmap v7722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:40:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13772.060435 secs 2024-07-26T16:40:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:36 smithi107 ceph-mon[32659]: pgmap v7723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:40:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13772.060435 secs 2024-07-26T16:40:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:36 smithi132 ceph-mon[37580]: pgmap v7723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:40:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13772.060435 secs 2024-07-26T16:40:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:36 smithi112 ceph-mon[37501]: pgmap v7723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:40:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:38 smithi107 ceph-mon[32659]: pgmap v7724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:38 smithi132 ceph-mon[37580]: pgmap v7724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:38 smithi112 ceph-mon[37501]: pgmap v7724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:40 smithi107 ceph-mon[32659]: pgmap v7725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:40 smithi132 ceph-mon[37580]: pgmap v7725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:40 smithi112 ceph-mon[37501]: pgmap v7725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13777.060556 secs 2024-07-26T16:40:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13777.060556 secs 2024-07-26T16:40:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13777.060556 secs 2024-07-26T16:40:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:42 smithi107 ceph-mon[32659]: pgmap v7726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:42 smithi132 ceph-mon[37580]: pgmap v7726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:42 smithi112 ceph-mon[37501]: pgmap v7726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:45.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:44 smithi107 ceph-mon[32659]: pgmap v7727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:44 smithi132 ceph-mon[37580]: pgmap v7727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:44 smithi112 ceph-mon[37501]: pgmap v7727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:47.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13782.060682 secs 2024-07-26T16:40:47.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:46 smithi112 ceph-mon[37501]: pgmap v7728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13782.060682 secs 2024-07-26T16:40:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:46 smithi107 ceph-mon[32659]: pgmap v7728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13782.060682 secs 2024-07-26T16:40:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:46 smithi132 ceph-mon[37580]: pgmap v7728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:48.202 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:48.200+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:40:48.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:48.201+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:48.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:40:48.201+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:40:49.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:48 smithi107 ceph-mon[32659]: pgmap v7729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:48 smithi132 ceph-mon[37580]: pgmap v7729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:48 smithi112 ceph-mon[37501]: pgmap v7729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:51.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:50 smithi107 ceph-mon[32659]: pgmap v7730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:50 smithi132 ceph-mon[37580]: pgmap v7730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:50 smithi112 ceph-mon[37501]: pgmap v7730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:40:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13787.060799 secs 2024-07-26T16:40:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13787.060799 secs 2024-07-26T16:40:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13787.060799 secs 2024-07-26T16:40:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:52 smithi107 ceph-mon[32659]: pgmap v7731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:52 smithi132 ceph-mon[37580]: pgmap v7731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:52 smithi112 ceph-mon[37501]: pgmap v7731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:54 smithi107 ceph-mon[32659]: pgmap v7732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:54 smithi132 ceph-mon[37580]: pgmap v7732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:54 smithi112 ceph-mon[37501]: pgmap v7732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13792.060915 secs 2024-07-26T16:40:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:56 smithi107 ceph-mon[32659]: pgmap v7733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13792.060915 secs 2024-07-26T16:40:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:56 smithi132 ceph-mon[37580]: pgmap v7733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13792.060915 secs 2024-07-26T16:40:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:56 smithi112 ceph-mon[37501]: pgmap v7733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:59.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:40:58 smithi107 ceph-mon[32659]: pgmap v7734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:40:58 smithi132 ceph-mon[37580]: pgmap v7734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:40:58 smithi112 ceph-mon[37501]: pgmap v7734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:00 smithi107 ceph-mon[32659]: pgmap v7735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:00 smithi132 ceph-mon[37580]: pgmap v7735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:00 smithi112 ceph-mon[37501]: pgmap v7735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13797.061052 secs 2024-07-26T16:41:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13797.061052 secs 2024-07-26T16:41:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13797.061052 secs 2024-07-26T16:41:03.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:02 smithi132 ceph-mon[37580]: pgmap v7736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:41:03.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:02 smithi107 ceph-mon[32659]: pgmap v7736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:41:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:02 smithi112 ceph-mon[37501]: pgmap v7736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:41:03.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:03.216+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:41:03.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:03.216+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:41:03.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:03.216+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:04 smithi112 ceph-mon[37501]: pgmap v7737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:05.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:04 smithi107 ceph-mon[32659]: pgmap v7737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:04 smithi132 ceph-mon[37580]: pgmap v7737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:07.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13802.061165 secs 2024-07-26T16:41:07.096 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:06 smithi107 ceph-mon[32659]: pgmap v7738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:07.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13802.061165 secs 2024-07-26T16:41:07.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:06 smithi132 ceph-mon[37580]: pgmap v7738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13802.061165 secs 2024-07-26T16:41:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:06 smithi112 ceph-mon[37501]: pgmap v7738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:08 smithi112 ceph-mon[37501]: pgmap v7739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:09.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:08 smithi107 ceph-mon[32659]: pgmap v7739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:08 smithi132 ceph-mon[37580]: pgmap v7739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:09 smithi112 ceph-mon[37501]: pgmap v7740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:09 smithi107 ceph-mon[32659]: pgmap v7740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:09 smithi132 ceph-mon[37580]: pgmap v7740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13807.061291 secs 2024-07-26T16:41:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13807.061291 secs 2024-07-26T16:41:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13807.061291 secs 2024-07-26T16:41:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:12 smithi107 ceph-mon[32659]: pgmap v7741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:41:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:12 smithi132 ceph-mon[37580]: pgmap v7741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:41:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:12 smithi112 ceph-mon[37501]: pgmap v7741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:41:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:14 smithi107 ceph-mon[32659]: pgmap v7742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:41:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:14 smithi132 ceph-mon[37580]: pgmap v7742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:41:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:14 smithi112 ceph-mon[37501]: pgmap v7742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:41:17.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13812.061412 secs 2024-07-26T16:41:17.004 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:16 smithi112 ceph-mon[37501]: pgmap v7743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:41:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13812.061412 secs 2024-07-26T16:41:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:16 smithi107 ceph-mon[32659]: pgmap v7743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:41:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13812.061412 secs 2024-07-26T16:41:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:16 smithi132 ceph-mon[37580]: pgmap v7743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:41:18.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:18.218+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:41:18.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:18.218+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:41:18.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:18.218+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:18 smithi107 ceph-mon[32659]: pgmap v7744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:41:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:18 smithi132 ceph-mon[37580]: pgmap v7744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:41:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:18 smithi112 ceph-mon[37501]: pgmap v7744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:41:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:20 smithi107 ceph-mon[32659]: pgmap v7745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:20 smithi132 ceph-mon[37580]: pgmap v7745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:20 smithi112 ceph-mon[37501]: pgmap v7745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13817.061545 secs 2024-07-26T16:41:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13817.061545 secs 2024-07-26T16:41:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13817.061545 secs 2024-07-26T16:41:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:22 smithi107 ceph-mon[32659]: pgmap v7746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:41:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:22 smithi132 ceph-mon[37580]: pgmap v7746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:41:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:22 smithi112 ceph-mon[37501]: pgmap v7746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:41:25.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:24 smithi107 ceph-mon[32659]: pgmap v7747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:24 smithi132 ceph-mon[37580]: pgmap v7747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:24 smithi112 ceph-mon[37501]: pgmap v7747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13822.061662 secs 2024-07-26T16:41:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:26 smithi107 ceph-mon[32659]: pgmap v7748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:41:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:41:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:26 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:41:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13822.061662 secs 2024-07-26T16:41:27.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:26 smithi132 ceph-mon[37580]: pgmap v7748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:27.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:41:27.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:41:27.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:26 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:41:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13822.061662 secs 2024-07-26T16:41:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:26 smithi112 ceph-mon[37501]: pgmap v7748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:41:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:41:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:26 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:41:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:41:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:41:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:41:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:41:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:41:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:41:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:41:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:41:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:41:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:28 smithi107 ceph-mon[32659]: pgmap v7749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:28 smithi107 ceph-mon[32659]: pgmap v7750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:28 smithi132 ceph-mon[37580]: pgmap v7749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:29.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:28 smithi132 ceph-mon[37580]: pgmap v7750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:28 smithi112 ceph-mon[37501]: pgmap v7749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:28 smithi112 ceph-mon[37501]: pgmap v7750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:30 smithi132 ceph-mon[37580]: pgmap v7751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:30 smithi112 ceph-mon[37501]: pgmap v7751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:30 smithi107 ceph-mon[32659]: pgmap v7751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13827.061786 secs 2024-07-26T16:41:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13827.061786 secs 2024-07-26T16:41:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13827.061786 secs 2024-07-26T16:41:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:32 smithi112 ceph-mon[37501]: pgmap v7752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:33.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:33.221+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:41:33.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:33.221+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:41:33.224 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:33.221+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:33.224 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:32 smithi107 ceph-mon[32659]: pgmap v7752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:32 smithi132 ceph-mon[37580]: pgmap v7752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:34 smithi112 ceph-mon[37501]: pgmap v7753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:34 smithi107 ceph-mon[32659]: pgmap v7753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:34 smithi132 ceph-mon[37580]: pgmap v7753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:37.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13832.061907 secs 2024-07-26T16:41:37.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:36 smithi107 ceph-mon[32659]: pgmap v7754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:37.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13832.061907 secs 2024-07-26T16:41:37.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:36 smithi132 ceph-mon[37580]: pgmap v7754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13832.061907 secs 2024-07-26T16:41:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:36 smithi112 ceph-mon[37501]: pgmap v7754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:41:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:38 smithi112 ceph-mon[37501]: pgmap v7755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:41:39.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:38 smithi107 ceph-mon[32659]: pgmap v7755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:41:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:38 smithi132 ceph-mon[37580]: pgmap v7755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:41:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:39 smithi112 ceph-mon[37501]: pgmap v7756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:39 smithi107 ceph-mon[32659]: pgmap v7756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:39 smithi132 ceph-mon[37580]: pgmap v7756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13837.062026 secs 2024-07-26T16:41:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13837.062026 secs 2024-07-26T16:41:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13837.062026 secs 2024-07-26T16:41:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:42 smithi107 ceph-mon[32659]: pgmap v7757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:41:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:42 smithi132 ceph-mon[37580]: pgmap v7757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:41:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:42 smithi112 ceph-mon[37501]: pgmap v7757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:41:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:44 smithi112 ceph-mon[37501]: pgmap v7758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:45.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:44 smithi107 ceph-mon[32659]: pgmap v7758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:44 smithi132 ceph-mon[37580]: pgmap v7758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13842.062152 secs 2024-07-26T16:41:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:46 smithi112 ceph-mon[37501]: pgmap v7759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13842.062152 secs 2024-07-26T16:41:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:46 smithi107 ceph-mon[32659]: pgmap v7759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13842.062152 secs 2024-07-26T16:41:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:46 smithi132 ceph-mon[37580]: pgmap v7759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:48.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:48.237+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:41:48.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:48.237+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:48.239 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:41:48.237+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:41:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:48 smithi112 ceph-mon[37501]: pgmap v7760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:49.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:48 smithi107 ceph-mon[32659]: pgmap v7760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:48 smithi132 ceph-mon[37580]: pgmap v7760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:50 smithi112 ceph-mon[37501]: pgmap v7761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:51.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:50 smithi107 ceph-mon[32659]: pgmap v7761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:50 smithi132 ceph-mon[37580]: pgmap v7761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:41:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13847.062253 secs 2024-07-26T16:41:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13847.062253 secs 2024-07-26T16:41:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13847.062253 secs 2024-07-26T16:41:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:52 smithi112 ceph-mon[37501]: pgmap v7762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:41:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:52 smithi107 ceph-mon[32659]: pgmap v7762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:41:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:52 smithi132 ceph-mon[37580]: pgmap v7762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:41:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:54 smithi112 ceph-mon[37501]: pgmap v7763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:54 smithi107 ceph-mon[32659]: pgmap v7763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:54 smithi132 ceph-mon[37580]: pgmap v7763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13852.062336 secs 2024-07-26T16:41:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:56 smithi112 ceph-mon[37501]: pgmap v7764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13852.062336 secs 2024-07-26T16:41:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:56 smithi107 ceph-mon[32659]: pgmap v7764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13852.062336 secs 2024-07-26T16:41:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:56 smithi132 ceph-mon[37580]: pgmap v7764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:41:58 smithi112 ceph-mon[37501]: pgmap v7765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:41:58 smithi107 ceph-mon[32659]: pgmap v7765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:41:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:41:58 smithi132 ceph-mon[37580]: pgmap v7765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:00 smithi112 ceph-mon[37501]: pgmap v7766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:00 smithi107 ceph-mon[32659]: pgmap v7766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:00 smithi132 ceph-mon[37580]: pgmap v7766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13857.062450 secs 2024-07-26T16:42:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13857.062450 secs 2024-07-26T16:42:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13857.062450 secs 2024-07-26T16:42:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:02 smithi112 ceph-mon[37501]: pgmap v7767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:02 smithi107 ceph-mon[32659]: pgmap v7767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:02 smithi132 ceph-mon[37580]: pgmap v7767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:03.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:03.253+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:42:03.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:03.253+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:03.255 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:03.253+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:42:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:04 smithi112 ceph-mon[37501]: pgmap v7768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:04 smithi107 ceph-mon[32659]: pgmap v7768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:04 smithi132 ceph-mon[37580]: pgmap v7768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13862.062592 secs 2024-07-26T16:42:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:06 smithi112 ceph-mon[37501]: pgmap v7769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13862.062592 secs 2024-07-26T16:42:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:06 smithi107 ceph-mon[32659]: pgmap v7769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13862.062592 secs 2024-07-26T16:42:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:06 smithi132 ceph-mon[37580]: pgmap v7769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:08 smithi112 ceph-mon[37501]: pgmap v7770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:09.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:08 smithi107 ceph-mon[32659]: pgmap v7770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:08 smithi132 ceph-mon[37580]: pgmap v7770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:10 smithi112 ceph-mon[37501]: pgmap v7771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:10 smithi107 ceph-mon[32659]: pgmap v7771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:10 smithi132 ceph-mon[37580]: pgmap v7771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13867.062731 secs 2024-07-26T16:42:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13867.062731 secs 2024-07-26T16:42:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13867.062731 secs 2024-07-26T16:42:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:12 smithi112 ceph-mon[37501]: pgmap v7772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:42:13.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:12 smithi107 ceph-mon[32659]: pgmap v7772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:42:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:12 smithi132 ceph-mon[37580]: pgmap v7772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:42:14.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:14.029+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:42:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:14.029+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:42:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:14.029+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:42:14.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:14.029+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:42:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:14 smithi112 ceph-mon[37501]: pgmap v7773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:42:15.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:15.029+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:42:15.032 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:15.029+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:42:15.033 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:14 smithi107 ceph-mon[32659]: pgmap v7773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:42:15.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:15.030+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:42:15.033 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:15.030+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:42:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:14 smithi132 ceph-mon[37580]: pgmap v7773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:42:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13872.062850 secs 2024-07-26T16:42:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:16 smithi112 ceph-mon[37501]: pgmap v7774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:42:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13872.062850 secs 2024-07-26T16:42:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:16 smithi107 ceph-mon[32659]: pgmap v7774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:42:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13872.062850 secs 2024-07-26T16:42:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:16 smithi132 ceph-mon[37580]: pgmap v7774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:42:18.261 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:18.259+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:42:18.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:18.259+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:42:18.262 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:18.259+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:18 smithi112 ceph-mon[37501]: pgmap v7775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:42:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:18 smithi107 ceph-mon[32659]: pgmap v7775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:42:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:18 smithi132 ceph-mon[37580]: pgmap v7775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:42:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:20 smithi112 ceph-mon[37501]: pgmap v7776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:20 smithi107 ceph-mon[32659]: pgmap v7776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:20 smithi132 ceph-mon[37580]: pgmap v7776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13877.062970 secs 2024-07-26T16:42:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13877.062970 secs 2024-07-26T16:42:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13877.062970 secs 2024-07-26T16:42:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:22 smithi112 ceph-mon[37501]: pgmap v7777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:22 smithi107 ceph-mon[32659]: pgmap v7777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:22 smithi132 ceph-mon[37580]: pgmap v7777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:24 smithi112 ceph-mon[37501]: pgmap v7778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:25.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:24 smithi107 ceph-mon[32659]: pgmap v7778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:24 smithi132 ceph-mon[37580]: pgmap v7778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13882.063087 secs 2024-07-26T16:42:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:26 smithi112 ceph-mon[37501]: pgmap v7779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13882.063087 secs 2024-07-26T16:42:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:26 smithi107 ceph-mon[32659]: pgmap v7779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13882.063087 secs 2024-07-26T16:42:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:26 smithi132 ceph-mon[37580]: pgmap v7779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:28.985 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:28 smithi112 ceph-mon[37501]: pgmap v7780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:28.985 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:42:28.985 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:28.985 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:28.986 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:28.986 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:29.007 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:28 smithi132 ceph-mon[37580]: pgmap v7780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:29.007 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:42:29.007 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:29.007 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:29.007 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:29.007 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:28 smithi107 ceph-mon[32659]: pgmap v7780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:42:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:30 smithi112 ceph-mon[37501]: pgmap v7781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:30.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:30 smithi107 ceph-mon[32659]: pgmap v7781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:30 smithi132 ceph-mon[37580]: pgmap v7781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13887.063182 secs 2024-07-26T16:42:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13887.063182 secs 2024-07-26T16:42:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13887.063182 secs 2024-07-26T16:42:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:32.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:32.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:32.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:32 smithi107 ceph-mon[32659]: pgmap v7782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:42:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:42:33.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:32 smithi132 ceph-mon[37580]: pgmap v7782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:33.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:33.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:42:33.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:42:33.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:32 smithi112 ceph-mon[37501]: pgmap v7782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:42:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:42:33.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:42:33.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:33.275+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:42:33.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:33.275+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:42:33.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:33.275+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:33 smithi107 ceph-mon[32659]: pgmap v7783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:42:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:33 smithi107 ceph-mon[32659]: pgmap v7784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:42:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:33 smithi132 ceph-mon[37580]: pgmap v7783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:42:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:33 smithi132 ceph-mon[37580]: pgmap v7784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:42:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:33 smithi112 ceph-mon[37501]: pgmap v7783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:42:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:33 smithi112 ceph-mon[37501]: pgmap v7784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:42:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:35 smithi107 ceph-mon[32659]: pgmap v7785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:42:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:35 smithi132 ceph-mon[37580]: pgmap v7785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:42:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:35 smithi112 ceph-mon[37501]: pgmap v7785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:42:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13892.063301 secs 2024-07-26T16:42:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13892.063301 secs 2024-07-26T16:42:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13892.063301 secs 2024-07-26T16:42:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:37 smithi107 ceph-mon[32659]: pgmap v7786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:37 smithi132 ceph-mon[37580]: pgmap v7786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:38.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:37 smithi112 ceph-mon[37501]: pgmap v7786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:39 smithi107 ceph-mon[32659]: pgmap v7787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:42:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:39 smithi132 ceph-mon[37580]: pgmap v7787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:42:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:39 smithi112 ceph-mon[37501]: pgmap v7787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:42:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13897.063416 secs 2024-07-26T16:42:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:41 smithi107 ceph-mon[32659]: pgmap v7788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:42:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13897.063416 secs 2024-07-26T16:42:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:41 smithi132 ceph-mon[37580]: pgmap v7788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:42:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13897.063416 secs 2024-07-26T16:42:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:41 smithi112 ceph-mon[37501]: pgmap v7788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:42:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:43 smithi107 ceph-mon[32659]: pgmap v7789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:42:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:43 smithi132 ceph-mon[37580]: pgmap v7789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:42:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:43 smithi112 ceph-mon[37501]: pgmap v7789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:42:46.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:45 smithi107 ceph-mon[32659]: pgmap v7790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:45 smithi132 ceph-mon[37580]: pgmap v7790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:45 smithi112 ceph-mon[37501]: pgmap v7790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13902.063548 secs 2024-07-26T16:42:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13902.063548 secs 2024-07-26T16:42:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13902.063548 secs 2024-07-26T16:42:48.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:47 smithi107 ceph-mon[32659]: pgmap v7791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:47 smithi132 ceph-mon[37580]: pgmap v7791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:48.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:47 smithi112 ceph-mon[37501]: pgmap v7791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:48.277+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:42:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:48.277+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:42:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:42:48.277+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:50.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:49 smithi107 ceph-mon[32659]: pgmap v7792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:50.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:49 smithi132 ceph-mon[37580]: pgmap v7792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:49 smithi112 ceph-mon[37501]: pgmap v7792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13907.063651 secs 2024-07-26T16:42:52.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:51 smithi107 ceph-mon[32659]: pgmap v7793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13907.063651 secs 2024-07-26T16:42:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:51 smithi132 ceph-mon[37580]: pgmap v7793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13907.063651 secs 2024-07-26T16:42:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:51 smithi112 ceph-mon[37501]: pgmap v7793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:53 smithi107 ceph-mon[32659]: pgmap v7794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:53 smithi132 ceph-mon[37580]: pgmap v7794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:53 smithi112 ceph-mon[37501]: pgmap v7794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:42:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:55 smithi107 ceph-mon[32659]: pgmap v7795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:55 smithi132 ceph-mon[37580]: pgmap v7795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:55 smithi112 ceph-mon[37501]: pgmap v7795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:42:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13912.063790 secs 2024-07-26T16:42:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13912.063790 secs 2024-07-26T16:42:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13912.063790 secs 2024-07-26T16:42:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:57 smithi132 ceph-mon[37580]: pgmap v7796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:57 smithi112 ceph-mon[37501]: pgmap v7796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:42:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:57 smithi107 ceph-mon[32659]: pgmap v7796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:42:59 smithi132 ceph-mon[37580]: pgmap v7797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:42:59 smithi112 ceph-mon[37501]: pgmap v7797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:42:59 smithi107 ceph-mon[32659]: pgmap v7797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13917.063921 secs 2024-07-26T16:43:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:01 smithi112 ceph-mon[37501]: pgmap v7798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13917.063921 secs 2024-07-26T16:43:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:01 smithi107 ceph-mon[32659]: pgmap v7798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13917.063921 secs 2024-07-26T16:43:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:01 smithi132 ceph-mon[37580]: pgmap v7798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:03.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:03.292+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:43:03.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:03.293+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:43:03.295 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:03.293+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:03 smithi112 ceph-mon[37501]: pgmap v7799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:04.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:03 smithi107 ceph-mon[32659]: pgmap v7799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:03 smithi132 ceph-mon[37580]: pgmap v7799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:05 smithi112 ceph-mon[37501]: pgmap v7800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:43:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:05 smithi107 ceph-mon[32659]: pgmap v7800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:43:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:05 smithi132 ceph-mon[37580]: pgmap v7800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:43:07.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13922.064044 secs 2024-07-26T16:43:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13922.064044 secs 2024-07-26T16:43:07.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13922.064044 secs 2024-07-26T16:43:08.123 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:07 smithi107 ceph-mon[32659]: pgmap v7801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:07 smithi112 ceph-mon[37501]: pgmap v7801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:07 smithi132 ceph-mon[37580]: pgmap v7801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:09 smithi112 ceph-mon[37501]: pgmap v7802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:10.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:09 smithi107 ceph-mon[32659]: pgmap v7802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:09 smithi132 ceph-mon[37580]: pgmap v7802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13927.064164 secs 2024-07-26T16:43:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:11 smithi112 ceph-mon[37501]: pgmap v7803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13927.064164 secs 2024-07-26T16:43:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:11 smithi107 ceph-mon[32659]: pgmap v7803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13927.064164 secs 2024-07-26T16:43:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:11 smithi132 ceph-mon[37580]: pgmap v7803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:12 smithi112 ceph-mon[37501]: pgmap v7804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:12 smithi107 ceph-mon[32659]: pgmap v7804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:12 smithi132 ceph-mon[37580]: pgmap v7804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:15 smithi112 ceph-mon[37501]: pgmap v7805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:43:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:15 smithi107 ceph-mon[32659]: pgmap v7805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:43:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:15 smithi132 ceph-mon[37580]: pgmap v7805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:43:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13932.064299 secs 2024-07-26T16:43:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13932.064299 secs 2024-07-26T16:43:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13932.064299 secs 2024-07-26T16:43:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:17 smithi112 ceph-mon[37501]: pgmap v7806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:18.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:17 smithi107 ceph-mon[32659]: pgmap v7806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:17 smithi132 ceph-mon[37580]: pgmap v7806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:18.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:18.296+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:43:18.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:18.296+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:18.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:18.296+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:43:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:19 smithi112 ceph-mon[37501]: pgmap v7807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:19 smithi107 ceph-mon[32659]: pgmap v7807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:19 smithi132 ceph-mon[37580]: pgmap v7807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13937.064427 secs 2024-07-26T16:43:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:21 smithi112 ceph-mon[37501]: pgmap v7808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13937.064427 secs 2024-07-26T16:43:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:21 smithi107 ceph-mon[32659]: pgmap v7808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13937.064427 secs 2024-07-26T16:43:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:21 smithi132 ceph-mon[37580]: pgmap v7808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:23 smithi112 ceph-mon[37501]: pgmap v7809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:23.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:23 smithi107 ceph-mon[32659]: pgmap v7809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:23 smithi132 ceph-mon[37580]: pgmap v7809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:25 smithi112 ceph-mon[37501]: pgmap v7810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:43:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:25 smithi107 ceph-mon[32659]: pgmap v7810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:43:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:25 smithi132 ceph-mon[37580]: pgmap v7810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:43:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13942.064555 secs 2024-07-26T16:43:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13942.064555 secs 2024-07-26T16:43:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13942.064555 secs 2024-07-26T16:43:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:27 smithi107 ceph-mon[32659]: pgmap v7811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:43:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:27 smithi132 ceph-mon[37580]: pgmap v7811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:43:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:27 smithi112 ceph-mon[37501]: pgmap v7811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:43:30.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:29 smithi107 ceph-mon[32659]: pgmap v7812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:29 smithi132 ceph-mon[37580]: pgmap v7812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:29 smithi112 ceph-mon[37501]: pgmap v7812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13947.064685 secs 2024-07-26T16:43:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:31 smithi107 ceph-mon[32659]: pgmap v7813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13947.064685 secs 2024-07-26T16:43:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:31 smithi132 ceph-mon[37580]: pgmap v7813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13947.064685 secs 2024-07-26T16:43:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:31 smithi112 ceph-mon[37501]: pgmap v7813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:33.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:33.311+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:43:33.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:33.311+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:33.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:33.311+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:43:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:33 smithi107 ceph-mon[32659]: pgmap v7814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:43:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:33 smithi132 ceph-mon[37580]: pgmap v7814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:43:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:33 smithi112 ceph-mon[37501]: pgmap v7814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:43:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:34 smithi107 ceph-mon[32659]: pgmap v7815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:34 smithi107 ceph-mon[32659]: pgmap v7816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:43:35.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:43:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:43:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:43:35.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:34 smithi132 ceph-mon[37580]: pgmap v7815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:43:35.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:34 smithi132 ceph-mon[37580]: pgmap v7816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:43:35.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:34 smithi112 ceph-mon[37501]: pgmap v7815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:34 smithi112 ceph-mon[37501]: pgmap v7816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:43:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:43:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:35 smithi112 ceph-mon[37501]: pgmap v7817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:43:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:35 smithi107 ceph-mon[32659]: pgmap v7817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:43:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:35 smithi132 ceph-mon[37580]: pgmap v7817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:43:37.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13952.064779 secs 2024-07-26T16:43:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13952.064779 secs 2024-07-26T16:43:37.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13952.064779 secs 2024-07-26T16:43:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:37 smithi112 ceph-mon[37501]: pgmap v7818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:43:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:37 smithi107 ceph-mon[32659]: pgmap v7818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:43:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:37 smithi132 ceph-mon[37580]: pgmap v7818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:43:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:39 smithi112 ceph-mon[37501]: pgmap v7819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:43:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:39 smithi107 ceph-mon[32659]: pgmap v7819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:43:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:39 smithi132 ceph-mon[37580]: pgmap v7819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:43:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13957.064876 secs 2024-07-26T16:43:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13957.064876 secs 2024-07-26T16:43:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13957.064876 secs 2024-07-26T16:43:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:42 smithi107 ceph-mon[32659]: pgmap v7820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:42 smithi132 ceph-mon[37580]: pgmap v7820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:42 smithi112 ceph-mon[37501]: pgmap v7820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:43 smithi112 ceph-mon[37501]: pgmap v7821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:43:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:43 smithi107 ceph-mon[32659]: pgmap v7821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:43:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:43 smithi132 ceph-mon[37580]: pgmap v7821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:43:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:45 smithi112 ceph-mon[37501]: pgmap v7822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:45 smithi107 ceph-mon[32659]: pgmap v7822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:45 smithi132 ceph-mon[37580]: pgmap v7822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13962.065003 secs 2024-07-26T16:43:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13962.065003 secs 2024-07-26T16:43:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13962.065003 secs 2024-07-26T16:43:48.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:47 smithi112 ceph-mon[37501]: pgmap v7823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:48.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:47 smithi107 ceph-mon[32659]: pgmap v7823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:47 smithi132 ceph-mon[37580]: pgmap v7823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:48.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:48.327+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:43:48.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:48.328+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:43:48.331 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:43:48.328+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:49 smithi112 ceph-mon[37501]: pgmap v7824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:50.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:49 smithi107 ceph-mon[32659]: pgmap v7824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:49 smithi132 ceph-mon[37580]: pgmap v7824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13967.065107 secs 2024-07-26T16:43:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13967.065107 secs 2024-07-26T16:43:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13967.065107 secs 2024-07-26T16:43:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:52 smithi107 ceph-mon[32659]: pgmap v7825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:52 smithi132 ceph-mon[37580]: pgmap v7825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:52 smithi112 ceph-mon[37501]: pgmap v7825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:43:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:53 smithi112 ceph-mon[37501]: pgmap v7826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:54.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:53 smithi107 ceph-mon[32659]: pgmap v7826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:53 smithi132 ceph-mon[37580]: pgmap v7826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:55 smithi112 ceph-mon[37501]: pgmap v7827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:55 smithi107 ceph-mon[32659]: pgmap v7827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:55 smithi132 ceph-mon[37580]: pgmap v7827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13972.065237 secs 2024-07-26T16:43:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13972.065237 secs 2024-07-26T16:43:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13972.065237 secs 2024-07-26T16:43:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:57 smithi112 ceph-mon[37501]: pgmap v7828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:57 smithi107 ceph-mon[32659]: pgmap v7828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:43:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:57 smithi132 ceph-mon[37580]: pgmap v7828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:43:59 smithi112 ceph-mon[37501]: pgmap v7829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:43:59 smithi107 ceph-mon[32659]: pgmap v7829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:43:59 smithi132 ceph-mon[37580]: pgmap v7829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13977.065357 secs 2024-07-26T16:44:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13977.065357 secs 2024-07-26T16:44:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13977.065357 secs 2024-07-26T16:44:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:02 smithi107 ceph-mon[32659]: pgmap v7830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:02 smithi132 ceph-mon[37580]: pgmap v7830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:02 smithi112 ceph-mon[37501]: pgmap v7830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:03.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:03.343+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:44:03.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:03.343+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:03.346 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:03.343+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:44:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:03 smithi112 ceph-mon[37501]: pgmap v7831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:04.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:03 smithi107 ceph-mon[32659]: pgmap v7831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:03 smithi132 ceph-mon[37580]: pgmap v7831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:05 smithi112 ceph-mon[37501]: pgmap v7832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:05 smithi107 ceph-mon[32659]: pgmap v7832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:05 smithi132 ceph-mon[37580]: pgmap v7832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:07.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13982.065455 secs 2024-07-26T16:44:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13982.065455 secs 2024-07-26T16:44:07.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13982.065455 secs 2024-07-26T16:44:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:07 smithi112 ceph-mon[37501]: pgmap v7833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:08.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:07 smithi107 ceph-mon[32659]: pgmap v7833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:07 smithi132 ceph-mon[37580]: pgmap v7833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:09 smithi112 ceph-mon[37501]: pgmap v7834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:09 smithi107 ceph-mon[32659]: pgmap v7834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:09 smithi132 ceph-mon[37580]: pgmap v7834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13987.065557 secs 2024-07-26T16:44:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13987.065557 secs 2024-07-26T16:44:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13987.065557 secs 2024-07-26T16:44:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:12 smithi107 ceph-mon[32659]: pgmap v7835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:12 smithi132 ceph-mon[37580]: pgmap v7835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:12 smithi112 ceph-mon[37501]: pgmap v7835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:13 smithi112 ceph-mon[37501]: pgmap v7836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:13 smithi107 ceph-mon[32659]: pgmap v7836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:13 smithi132 ceph-mon[37580]: pgmap v7836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:16.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:15 smithi112 ceph-mon[37501]: pgmap v7837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:15 smithi107 ceph-mon[32659]: pgmap v7837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:15 smithi132 ceph-mon[37580]: pgmap v7837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13992.065641 secs 2024-07-26T16:44:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13992.065641 secs 2024-07-26T16:44:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13992.065641 secs 2024-07-26T16:44:18.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:17 smithi112 ceph-mon[37501]: pgmap v7838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:17 smithi107 ceph-mon[32659]: pgmap v7838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:17 smithi132 ceph-mon[37580]: pgmap v7838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:18.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:18.355+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:44:18.357 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:18.355+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:44:18.358 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:18.355+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:19 smithi112 ceph-mon[37501]: pgmap v7839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:19 smithi107 ceph-mon[32659]: pgmap v7839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:19 smithi132 ceph-mon[37580]: pgmap v7839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 13997.065770 secs 2024-07-26T16:44:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 13997.065770 secs 2024-07-26T16:44:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 13997.065770 secs 2024-07-26T16:44:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:22 smithi107 ceph-mon[32659]: pgmap v7840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:22 smithi132 ceph-mon[37580]: pgmap v7840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:22 smithi112 ceph-mon[37501]: pgmap v7840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:23 smithi112 ceph-mon[37501]: pgmap v7841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:24.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:23 smithi107 ceph-mon[32659]: pgmap v7841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:23 smithi132 ceph-mon[37580]: pgmap v7841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:25 smithi112 ceph-mon[37501]: pgmap v7842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:25 smithi107 ceph-mon[32659]: pgmap v7842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:25 smithi132 ceph-mon[37580]: pgmap v7842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14002.065831 secs 2024-07-26T16:44:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14002.065831 secs 2024-07-26T16:44:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14002.065831 secs 2024-07-26T16:44:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:27 smithi112 ceph-mon[37501]: pgmap v7843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:27 smithi107 ceph-mon[32659]: pgmap v7843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:27 smithi132 ceph-mon[37580]: pgmap v7843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:29 smithi112 ceph-mon[37501]: pgmap v7844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:29 smithi107 ceph-mon[32659]: pgmap v7844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:29 smithi132 ceph-mon[37580]: pgmap v7844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14007.065957 secs 2024-07-26T16:44:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14007.065957 secs 2024-07-26T16:44:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14007.065957 secs 2024-07-26T16:44:32.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:32 smithi107 ceph-mon[32659]: pgmap v7845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:32 smithi132 ceph-mon[37580]: pgmap v7845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:32.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:32 smithi112 ceph-mon[37501]: pgmap v7845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:33.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:33.362+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:44:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:33.362+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:33.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:33.362+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:44:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:33 smithi112 ceph-mon[37501]: pgmap v7846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:33 smithi107 ceph-mon[32659]: pgmap v7846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:33 smithi132 ceph-mon[37580]: pgmap v7846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:44:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:44:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:35 smithi112 ceph-mon[37501]: pgmap v7847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:35 smithi112 ceph-mon[37501]: pgmap v7848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:44:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:44:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:44:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:35 smithi107 ceph-mon[32659]: pgmap v7847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:44:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:35 smithi107 ceph-mon[32659]: pgmap v7848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:44:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:44:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:44:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:35 smithi132 ceph-mon[37580]: pgmap v7847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:44:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:35 smithi132 ceph-mon[37580]: pgmap v7848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:44:37.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14012.066119 secs 2024-07-26T16:44:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14012.066119 secs 2024-07-26T16:44:37.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14012.066119 secs 2024-07-26T16:44:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:37 smithi112 ceph-mon[37501]: pgmap v7849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:37 smithi107 ceph-mon[32659]: pgmap v7849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:37 smithi132 ceph-mon[37580]: pgmap v7849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:40 smithi132 ceph-mon[37580]: pgmap v7850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:44:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:40 smithi112 ceph-mon[37501]: pgmap v7850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:44:40.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:40 smithi107 ceph-mon[32659]: pgmap v7850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:44:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14017.066249 secs 2024-07-26T16:44:42.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:42 smithi132 ceph-mon[37580]: pgmap v7851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:44:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14017.066249 secs 2024-07-26T16:44:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:42 smithi112 ceph-mon[37501]: pgmap v7851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:44:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14017.066249 secs 2024-07-26T16:44:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:42 smithi107 ceph-mon[32659]: pgmap v7851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:44:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:44 smithi112 ceph-mon[37501]: pgmap v7852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:44:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:44 smithi107 ceph-mon[32659]: pgmap v7852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:44:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:44 smithi132 ceph-mon[37580]: pgmap v7852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:44:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:46 smithi112 ceph-mon[37501]: pgmap v7853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:44:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:46 smithi107 ceph-mon[32659]: pgmap v7853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:44:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:46 smithi132 ceph-mon[37580]: pgmap v7853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:44:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14022.066383 secs 2024-07-26T16:44:47.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14022.066383 secs 2024-07-26T16:44:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14022.066383 secs 2024-07-26T16:44:48.376 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:48.374+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:44:48.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:48.374+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:48.377 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:44:48.374+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:44:48.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:48 smithi107 ceph-mon[32659]: pgmap v7854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:48 smithi112 ceph-mon[37501]: pgmap v7854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:48 smithi132 ceph-mon[37580]: pgmap v7854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:50 smithi112 ceph-mon[37501]: pgmap v7855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:50.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:50 smithi107 ceph-mon[32659]: pgmap v7855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:50 smithi132 ceph-mon[37580]: pgmap v7855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14027.066512 secs 2024-07-26T16:44:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:52 smithi112 ceph-mon[37501]: pgmap v7856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14027.066512 secs 2024-07-26T16:44:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:52 smithi107 ceph-mon[32659]: pgmap v7856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14027.066512 secs 2024-07-26T16:44:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:52 smithi132 ceph-mon[37580]: pgmap v7856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:54 smithi112 ceph-mon[37501]: pgmap v7857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:54 smithi107 ceph-mon[32659]: pgmap v7857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:54 smithi132 ceph-mon[37580]: pgmap v7857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:56 smithi112 ceph-mon[37501]: pgmap v7858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:56 smithi107 ceph-mon[32659]: pgmap v7858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:56 smithi132 ceph-mon[37580]: pgmap v7858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:57.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14032.066666 secs 2024-07-26T16:44:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14032.066666 secs 2024-07-26T16:44:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14032.066666 secs 2024-07-26T16:44:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:44:58 smithi112 ceph-mon[37501]: pgmap v7859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:44:58 smithi107 ceph-mon[32659]: pgmap v7859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:44:58 smithi132 ceph-mon[37580]: pgmap v7859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:00 smithi112 ceph-mon[37501]: pgmap v7860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:00 smithi107 ceph-mon[32659]: pgmap v7860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:00 smithi132 ceph-mon[37580]: pgmap v7860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14037.066794 secs 2024-07-26T16:45:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:02 smithi112 ceph-mon[37501]: pgmap v7861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14037.066794 secs 2024-07-26T16:45:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:02 smithi107 ceph-mon[32659]: pgmap v7861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14037.066794 secs 2024-07-26T16:45:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:02 smithi132 ceph-mon[37580]: pgmap v7861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:03.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:03.379+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:45:03.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:03.379+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:03.381 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:03.379+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:45:04.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:04 smithi112 ceph-mon[37501]: pgmap v7862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:04 smithi107 ceph-mon[32659]: pgmap v7862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:04 smithi132 ceph-mon[37580]: pgmap v7862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:06 smithi112 ceph-mon[37501]: pgmap v7863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:06 smithi107 ceph-mon[32659]: pgmap v7863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:06 smithi132 ceph-mon[37580]: pgmap v7863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:07.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14042.066919 secs 2024-07-26T16:45:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14042.066919 secs 2024-07-26T16:45:07.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14042.066919 secs 2024-07-26T16:45:08.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:08 smithi112 ceph-mon[37501]: pgmap v7864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:08 smithi107 ceph-mon[32659]: pgmap v7864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:08 smithi132 ceph-mon[37580]: pgmap v7864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:10 smithi112 ceph-mon[37501]: pgmap v7865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:10 smithi107 ceph-mon[32659]: pgmap v7865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:10 smithi132 ceph-mon[37580]: pgmap v7865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14047.067062 secs 2024-07-26T16:45:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:12 smithi112 ceph-mon[37501]: pgmap v7866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14047.067062 secs 2024-07-26T16:45:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:12 smithi107 ceph-mon[32659]: pgmap v7866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14047.067062 secs 2024-07-26T16:45:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:12 smithi132 ceph-mon[37580]: pgmap v7866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:14 smithi112 ceph-mon[37501]: pgmap v7867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:14 smithi107 ceph-mon[32659]: pgmap v7867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:14 smithi132 ceph-mon[37580]: pgmap v7867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:15.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:15.061+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:45:15.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:15.061+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:45:15.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:15.061+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:45:15.064 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:15.061+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:45:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:16 smithi112 ceph-mon[37501]: pgmap v7868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:16.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:16 smithi107 ceph-mon[32659]: pgmap v7868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:16 smithi132 ceph-mon[37580]: pgmap v7868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14052.067182 secs 2024-07-26T16:45:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14052.067182 secs 2024-07-26T16:45:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14052.067182 secs 2024-07-26T16:45:18.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:18.383+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:45:18.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:18.383+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:18.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:18.383+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:45:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:18 smithi112 ceph-mon[37501]: pgmap v7869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:18 smithi107 ceph-mon[32659]: pgmap v7869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:18.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:18 smithi132 ceph-mon[37580]: pgmap v7869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:20 smithi107 ceph-mon[32659]: pgmap v7870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:20 smithi132 ceph-mon[37580]: pgmap v7870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:20 smithi112 ceph-mon[37501]: pgmap v7870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14057.067247 secs 2024-07-26T16:45:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:22 smithi107 ceph-mon[32659]: pgmap v7871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14057.067247 secs 2024-07-26T16:45:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:22 smithi132 ceph-mon[37580]: pgmap v7871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14057.067247 secs 2024-07-26T16:45:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:22 smithi112 ceph-mon[37501]: pgmap v7871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:24.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:24 smithi107 ceph-mon[32659]: pgmap v7872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:24 smithi132 ceph-mon[37580]: pgmap v7872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:24 smithi112 ceph-mon[37501]: pgmap v7872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:26 smithi107 ceph-mon[32659]: pgmap v7873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:26 smithi132 ceph-mon[37580]: pgmap v7873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:26 smithi112 ceph-mon[37501]: pgmap v7873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:27.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14062.067314 secs 2024-07-26T16:45:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14062.067314 secs 2024-07-26T16:45:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14062.067314 secs 2024-07-26T16:45:28.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:28 smithi107 ceph-mon[32659]: pgmap v7874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:28 smithi132 ceph-mon[37580]: pgmap v7874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:28 smithi112 ceph-mon[37501]: pgmap v7874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:30 smithi107 ceph-mon[32659]: pgmap v7875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:30 smithi132 ceph-mon[37580]: pgmap v7875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:30 smithi112 ceph-mon[37501]: pgmap v7875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14067.067414 secs 2024-07-26T16:45:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14067.067414 secs 2024-07-26T16:45:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14067.067414 secs 2024-07-26T16:45:32.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:32 smithi107 ceph-mon[32659]: pgmap v7876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:32 smithi132 ceph-mon[37580]: pgmap v7876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:32.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:32 smithi112 ceph-mon[37501]: pgmap v7876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:33.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:33.399+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:45:33.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:33.399+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:45:33.402 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:33.399+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:34.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:34 smithi107 ceph-mon[32659]: pgmap v7877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:34 smithi132 ceph-mon[37580]: pgmap v7877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:34 smithi112 ceph-mon[37501]: pgmap v7877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:45:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:45:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:45:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:36 smithi107 ceph-mon[32659]: pgmap v7878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14072.067504 secs 2024-07-26T16:45:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:45:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:45:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:45:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:36 smithi132 ceph-mon[37580]: pgmap v7878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14072.067504 secs 2024-07-26T16:45:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:45:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:45:36.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:45:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:36 smithi112 ceph-mon[37501]: pgmap v7878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14072.067504 secs 2024-07-26T16:45:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:45:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:45:36.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:45:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:37 smithi132 ceph-mon[37580]: pgmap v7879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:37.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:37 smithi132 ceph-mon[37580]: pgmap v7880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:45:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:37 smithi112 ceph-mon[37501]: pgmap v7879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:37 smithi112 ceph-mon[37501]: pgmap v7880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:45:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:37 smithi107 ceph-mon[32659]: pgmap v7879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:37 smithi107 ceph-mon[32659]: pgmap v7880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:45:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:39 smithi132 ceph-mon[37580]: pgmap v7881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:45:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:39 smithi112 ceph-mon[37501]: pgmap v7881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:45:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:39 smithi107 ceph-mon[32659]: pgmap v7881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:45:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:41 smithi132 ceph-mon[37580]: pgmap v7882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:45:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14077.067658 secs 2024-07-26T16:45:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:41 smithi112 ceph-mon[37501]: pgmap v7882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:45:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14077.067658 secs 2024-07-26T16:45:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:41 smithi107 ceph-mon[32659]: pgmap v7882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:45:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14077.067658 secs 2024-07-26T16:45:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:43 smithi132 ceph-mon[37580]: pgmap v7883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:45:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:43 smithi112 ceph-mon[37501]: pgmap v7883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:45:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:43 smithi107 ceph-mon[32659]: pgmap v7883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:45:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:45 smithi112 ceph-mon[37501]: pgmap v7884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:45:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:45 smithi107 ceph-mon[32659]: pgmap v7884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:45:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:45 smithi132 ceph-mon[37580]: pgmap v7884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:45:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14082.067784 secs 2024-07-26T16:45:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14082.067784 secs 2024-07-26T16:45:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14082.067784 secs 2024-07-26T16:45:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:47 smithi112 ceph-mon[37501]: pgmap v7885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:45:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:47 smithi107 ceph-mon[32659]: pgmap v7885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:45:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:47 smithi132 ceph-mon[37580]: pgmap v7885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:45:48.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:48.408+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:45:48.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:48.408+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:48.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:45:48.408+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:45:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:49 smithi112 ceph-mon[37501]: pgmap v7886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:50.026 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:49 smithi132 ceph-mon[37580]: pgmap v7886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:50.026 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:49 smithi107 ceph-mon[32659]: pgmap v7886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:51 smithi112 ceph-mon[37501]: pgmap v7887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14087.067918 secs 2024-07-26T16:45:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:51 smithi107 ceph-mon[32659]: pgmap v7887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14087.067918 secs 2024-07-26T16:45:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:51 smithi132 ceph-mon[37580]: pgmap v7887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:51.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14087.067918 secs 2024-07-26T16:45:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:53 smithi112 ceph-mon[37501]: pgmap v7888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:53 smithi107 ceph-mon[32659]: pgmap v7888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:53 smithi132 ceph-mon[37580]: pgmap v7888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:45:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:55 smithi112 ceph-mon[37501]: pgmap v7889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:55.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:55 smithi107 ceph-mon[32659]: pgmap v7889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:55 smithi132 ceph-mon[37580]: pgmap v7889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:45:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14092.068040 secs 2024-07-26T16:45:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14092.068040 secs 2024-07-26T16:45:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14092.068040 secs 2024-07-26T16:45:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:57 smithi112 ceph-mon[37501]: pgmap v7890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:57.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:57 smithi107 ceph-mon[32659]: pgmap v7890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:57 smithi132 ceph-mon[37580]: pgmap v7890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:45:59 smithi112 ceph-mon[37501]: pgmap v7891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:45:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:45:59 smithi107 ceph-mon[32659]: pgmap v7891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:45:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:45:59 smithi132 ceph-mon[37580]: pgmap v7891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:01 smithi112 ceph-mon[37501]: pgmap v7892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:46:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14097.068160 secs 2024-07-26T16:46:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:01 smithi107 ceph-mon[32659]: pgmap v7892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:46:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14097.068160 secs 2024-07-26T16:46:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:01 smithi132 ceph-mon[37580]: pgmap v7892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:46:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14097.068160 secs 2024-07-26T16:46:03.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:03.423+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:46:03.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:03.423+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:03.426 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:03.423+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:46:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:03 smithi112 ceph-mon[37501]: pgmap v7893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T16:46:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:03 smithi107 ceph-mon[32659]: pgmap v7893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T16:46:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:03 smithi132 ceph-mon[37580]: pgmap v7893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T16:46:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:05 smithi112 ceph-mon[37501]: pgmap v7894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:46:05.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:05 smithi107 ceph-mon[32659]: pgmap v7894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:46:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:05 smithi132 ceph-mon[37580]: pgmap v7894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:46:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14102.068244 secs 2024-07-26T16:46:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14102.068244 secs 2024-07-26T16:46:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14102.068244 secs 2024-07-26T16:46:07.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:07 smithi107 ceph-mon[32659]: pgmap v7895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:46:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:07 smithi112 ceph-mon[37501]: pgmap v7895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:46:07.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:07 smithi132 ceph-mon[37580]: pgmap v7895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:46:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:09 smithi112 ceph-mon[37501]: pgmap v7896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:46:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:09 smithi107 ceph-mon[32659]: pgmap v7896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:46:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:09 smithi132 ceph-mon[37580]: pgmap v7896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:46:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:11 smithi112 ceph-mon[37501]: pgmap v7897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14107.068345 secs 2024-07-26T16:46:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:11 smithi107 ceph-mon[32659]: pgmap v7897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14107.068345 secs 2024-07-26T16:46:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:11 smithi132 ceph-mon[37580]: pgmap v7897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:11.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14107.068345 secs 2024-07-26T16:46:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:13 smithi112 ceph-mon[37501]: pgmap v7898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:13.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:13 smithi107 ceph-mon[32659]: pgmap v7898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:13 smithi132 ceph-mon[37580]: pgmap v7898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:15 smithi112 ceph-mon[37501]: pgmap v7899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:15 smithi107 ceph-mon[32659]: pgmap v7899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:15 smithi132 ceph-mon[37580]: pgmap v7899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14112.068461 secs 2024-07-26T16:46:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14112.068461 secs 2024-07-26T16:46:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14112.068461 secs 2024-07-26T16:46:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:17 smithi112 ceph-mon[37501]: pgmap v7900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:17 smithi107 ceph-mon[32659]: pgmap v7900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:17 smithi132 ceph-mon[37580]: pgmap v7900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:18.438+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:46:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:18.438+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:18.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:18.438+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:46:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:19 smithi107 ceph-mon[32659]: pgmap v7901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:19 smithi132 ceph-mon[37580]: pgmap v7901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:19 smithi112 ceph-mon[37501]: pgmap v7901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:21.752 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:21 smithi112 ceph-mon[37501]: pgmap v7902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:21.752 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14117.068599 secs 2024-07-26T16:46:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:21 smithi107 ceph-mon[32659]: pgmap v7902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14117.068599 secs 2024-07-26T16:46:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:21 smithi132 ceph-mon[37580]: pgmap v7902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:21.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14117.068599 secs 2024-07-26T16:46:23.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:23 smithi107 ceph-mon[32659]: pgmap v7903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:23 smithi132 ceph-mon[37580]: pgmap v7903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:23 smithi112 ceph-mon[37501]: pgmap v7903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:25.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:25 smithi107 ceph-mon[32659]: pgmap v7904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:25 smithi132 ceph-mon[37580]: pgmap v7904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:25 smithi112 ceph-mon[37501]: pgmap v7904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14122.068666 secs 2024-07-26T16:46:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14122.068666 secs 2024-07-26T16:46:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14122.068666 secs 2024-07-26T16:46:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:27 smithi107 ceph-mon[32659]: pgmap v7905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:27 smithi132 ceph-mon[37580]: pgmap v7905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:27 smithi112 ceph-mon[37501]: pgmap v7905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:29 smithi107 ceph-mon[32659]: pgmap v7906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:29 smithi132 ceph-mon[37580]: pgmap v7906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:29 smithi112 ceph-mon[37501]: pgmap v7906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:31 smithi107 ceph-mon[32659]: pgmap v7907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14127.068782 secs 2024-07-26T16:46:31.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:31 smithi132 ceph-mon[37580]: pgmap v7907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:31.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14127.068782 secs 2024-07-26T16:46:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:31 smithi112 ceph-mon[37501]: pgmap v7907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14127.068782 secs 2024-07-26T16:46:33.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:33.454+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:46:33.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:33.454+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:33.457 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:33.454+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:46:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:33 smithi107 ceph-mon[32659]: pgmap v7908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:33 smithi132 ceph-mon[37580]: pgmap v7908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:33 smithi112 ceph-mon[37501]: pgmap v7908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:35 smithi107 ceph-mon[32659]: pgmap v7909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:35 smithi132 ceph-mon[37580]: pgmap v7909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:35 smithi112 ceph-mon[37501]: pgmap v7909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14132.068899 secs 2024-07-26T16:46:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:46:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14132.068899 secs 2024-07-26T16:46:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:46:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14132.068899 secs 2024-07-26T16:46:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:46:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:37 smithi107 ceph-mon[32659]: pgmap v7910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:46:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:46:37.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:37 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:46:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:37 smithi132 ceph-mon[37580]: pgmap v7910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:46:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:46:37.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:37 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:46:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:37 smithi112 ceph-mon[37501]: pgmap v7910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:46:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:46:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:37 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:46:38.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:38 smithi107 ceph-mon[32659]: pgmap v7911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:46:38.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:38 smithi107 ceph-mon[32659]: pgmap v7912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:38 smithi132 ceph-mon[37580]: pgmap v7911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:46:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:38 smithi132 ceph-mon[37580]: pgmap v7912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:38 smithi112 ceph-mon[37501]: pgmap v7911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:46:38.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:38 smithi112 ceph-mon[37501]: pgmap v7912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:40 smithi107 ceph-mon[32659]: pgmap v7913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:40 smithi132 ceph-mon[37580]: pgmap v7913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:40 smithi112 ceph-mon[37501]: pgmap v7913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14137.069018 secs 2024-07-26T16:46:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14137.069018 secs 2024-07-26T16:46:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14137.069018 secs 2024-07-26T16:46:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:42 smithi132 ceph-mon[37580]: pgmap v7914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:42 smithi112 ceph-mon[37501]: pgmap v7914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:43.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:42 smithi107 ceph-mon[32659]: pgmap v7914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:44 smithi132 ceph-mon[37580]: pgmap v7915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:44 smithi112 ceph-mon[37501]: pgmap v7915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:45.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:44 smithi107 ceph-mon[32659]: pgmap v7915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:46 smithi112 ceph-mon[37501]: pgmap v7916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14142.069169 secs 2024-07-26T16:46:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:46 smithi107 ceph-mon[32659]: pgmap v7916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14142.069169 secs 2024-07-26T16:46:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:46 smithi132 ceph-mon[37580]: pgmap v7916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:47.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14142.069169 secs 2024-07-26T16:46:48.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:48.466+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:46:48.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:48.466+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:48.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:46:48.466+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:46:48.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:48 smithi107 ceph-mon[32659]: pgmap v7917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:46:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:48 smithi112 ceph-mon[37501]: pgmap v7917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:46:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:48 smithi132 ceph-mon[37580]: pgmap v7917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:46:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:50 smithi112 ceph-mon[37501]: pgmap v7918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:51.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:50 smithi107 ceph-mon[32659]: pgmap v7918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:50 smithi132 ceph-mon[37580]: pgmap v7918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:51.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14147.069281 secs 2024-07-26T16:46:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14147.069281 secs 2024-07-26T16:46:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14147.069281 secs 2024-07-26T16:46:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:52 smithi112 ceph-mon[37501]: pgmap v7919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:52 smithi107 ceph-mon[32659]: pgmap v7919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:52 smithi132 ceph-mon[37580]: pgmap v7919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:46:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:54 smithi112 ceph-mon[37501]: pgmap v7920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:54 smithi107 ceph-mon[32659]: pgmap v7920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:54 smithi132 ceph-mon[37580]: pgmap v7920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:56 smithi112 ceph-mon[37501]: pgmap v7921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14152.069354 secs 2024-07-26T16:46:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:56 smithi107 ceph-mon[32659]: pgmap v7921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14152.069354 secs 2024-07-26T16:46:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:56 smithi132 ceph-mon[37580]: pgmap v7921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14152.069354 secs 2024-07-26T16:46:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:46:58 smithi112 ceph-mon[37501]: pgmap v7922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:59.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:46:58 smithi107 ceph-mon[32659]: pgmap v7922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:46:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:46:58 smithi132 ceph-mon[37580]: pgmap v7922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:00 smithi112 ceph-mon[37501]: pgmap v7923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:00 smithi107 ceph-mon[32659]: pgmap v7923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:00 smithi132 ceph-mon[37580]: pgmap v7923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14157.069432 secs 2024-07-26T16:47:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14157.069432 secs 2024-07-26T16:47:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14157.069432 secs 2024-07-26T16:47:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:02 smithi112 ceph-mon[37501]: pgmap v7924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:02 smithi107 ceph-mon[32659]: pgmap v7924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:02 smithi132 ceph-mon[37580]: pgmap v7924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:03.482+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:47:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:03.482+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:47:03.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:03.482+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:47:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:04 smithi112 ceph-mon[37501]: pgmap v7925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:04 smithi107 ceph-mon[32659]: pgmap v7925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:04 smithi132 ceph-mon[37580]: pgmap v7925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:06 smithi112 ceph-mon[37501]: pgmap v7926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14162.069554 secs 2024-07-26T16:47:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:06 smithi107 ceph-mon[32659]: pgmap v7926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14162.069554 secs 2024-07-26T16:47:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:06 smithi132 ceph-mon[37580]: pgmap v7926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:07.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14162.069554 secs 2024-07-26T16:47:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:08 smithi112 ceph-mon[37501]: pgmap v7927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:09.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:08 smithi107 ceph-mon[32659]: pgmap v7927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:08 smithi132 ceph-mon[37580]: pgmap v7927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:10 smithi112 ceph-mon[37501]: pgmap v7928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:10 smithi107 ceph-mon[32659]: pgmap v7928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:10 smithi132 ceph-mon[37580]: pgmap v7928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14167.069673 secs 2024-07-26T16:47:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14167.069673 secs 2024-07-26T16:47:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14167.069673 secs 2024-07-26T16:47:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:12 smithi112 ceph-mon[37501]: pgmap v7929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:13.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:12 smithi107 ceph-mon[32659]: pgmap v7929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:12 smithi132 ceph-mon[37580]: pgmap v7929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:14 smithi112 ceph-mon[37501]: pgmap v7930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:14 smithi107 ceph-mon[32659]: pgmap v7930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:14 smithi132 ceph-mon[37580]: pgmap v7930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:16 smithi112 ceph-mon[37501]: pgmap v7931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14172.069793 secs 2024-07-26T16:47:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:16 smithi107 ceph-mon[32659]: pgmap v7931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14172.069793 secs 2024-07-26T16:47:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:16 smithi132 ceph-mon[37580]: pgmap v7931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:17.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14172.069793 secs 2024-07-26T16:47:18.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:18.497+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:47:18.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:18.497+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:47:18.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:18.497+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:47:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:18 smithi107 ceph-mon[32659]: pgmap v7932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:18 smithi112 ceph-mon[37501]: pgmap v7932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:18 smithi132 ceph-mon[37580]: pgmap v7932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:20 smithi112 ceph-mon[37501]: pgmap v7933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:20 smithi107 ceph-mon[32659]: pgmap v7933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:20 smithi132 ceph-mon[37580]: pgmap v7933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14177.069915 secs 2024-07-26T16:47:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14177.069915 secs 2024-07-26T16:47:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14177.069915 secs 2024-07-26T16:47:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:22 smithi107 ceph-mon[32659]: pgmap v7934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:22 smithi132 ceph-mon[37580]: pgmap v7934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:22 smithi112 ceph-mon[37501]: pgmap v7934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:25.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:24 smithi107 ceph-mon[32659]: pgmap v7935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:24 smithi132 ceph-mon[37580]: pgmap v7935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:24 smithi112 ceph-mon[37501]: pgmap v7935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:26 smithi107 ceph-mon[32659]: pgmap v7936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14182.070035 secs 2024-07-26T16:47:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:26 smithi132 ceph-mon[37580]: pgmap v7936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14182.070035 secs 2024-07-26T16:47:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:26 smithi112 ceph-mon[37501]: pgmap v7936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14182.070035 secs 2024-07-26T16:47:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:28 smithi107 ceph-mon[32659]: pgmap v7937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:28 smithi132 ceph-mon[37580]: pgmap v7937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:28 smithi112 ceph-mon[37501]: pgmap v7937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:30 smithi107 ceph-mon[32659]: pgmap v7938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:30 smithi132 ceph-mon[37580]: pgmap v7938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:30 smithi112 ceph-mon[37501]: pgmap v7938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14187.070160 secs 2024-07-26T16:47:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14187.070160 secs 2024-07-26T16:47:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14187.070160 secs 2024-07-26T16:47:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:32 smithi107 ceph-mon[32659]: pgmap v7939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:32 smithi132 ceph-mon[37580]: pgmap v7939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:32 smithi112 ceph-mon[37501]: pgmap v7939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:33.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:33.498+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:47:33.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:33.498+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:47:33.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:33.498+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:47:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:34 smithi107 ceph-mon[32659]: pgmap v7940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:34 smithi132 ceph-mon[37580]: pgmap v7940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:34 smithi112 ceph-mon[37501]: pgmap v7940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:36 smithi107 ceph-mon[32659]: pgmap v7941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14192.070276 secs 2024-07-26T16:47:37.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:36 smithi132 ceph-mon[37580]: pgmap v7941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:37.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14192.070276 secs 2024-07-26T16:47:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:36 smithi112 ceph-mon[37501]: pgmap v7941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14192.070276 secs 2024-07-26T16:47:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:37 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:47:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:37 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:47:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:37 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:47:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:38 smithi107 ceph-mon[32659]: pgmap v7942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:47:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:38 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:47:39.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:38 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:47:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:38 smithi132 ceph-mon[37580]: pgmap v7942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:47:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:38 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:47:39.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:38 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:47:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:38 smithi112 ceph-mon[37501]: pgmap v7942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:47:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:38 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:47:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:38 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:47:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:39 smithi107 ceph-mon[32659]: pgmap v7943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:47:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:39 smithi107 ceph-mon[32659]: pgmap v7944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:47:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:39 smithi132 ceph-mon[37580]: pgmap v7943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:47:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:39 smithi132 ceph-mon[37580]: pgmap v7944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:47:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:39 smithi112 ceph-mon[37501]: pgmap v7943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:47:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:39 smithi112 ceph-mon[37501]: pgmap v7944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:47:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14197.070419 secs 2024-07-26T16:47:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:41 smithi107 ceph-mon[32659]: pgmap v7945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:47:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14197.070419 secs 2024-07-26T16:47:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:41 smithi132 ceph-mon[37580]: pgmap v7945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:47:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14197.070419 secs 2024-07-26T16:47:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:41 smithi112 ceph-mon[37501]: pgmap v7945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:47:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:43 smithi132 ceph-mon[37580]: pgmap v7946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:47:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:43 smithi112 ceph-mon[37501]: pgmap v7946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:47:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:43 smithi107 ceph-mon[32659]: pgmap v7946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:47:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:45 smithi132 ceph-mon[37580]: pgmap v7947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:47:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:45 smithi112 ceph-mon[37501]: pgmap v7947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:47:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:45 smithi107 ceph-mon[32659]: pgmap v7947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:47:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14202.070492 secs 2024-07-26T16:47:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14202.070492 secs 2024-07-26T16:47:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14202.070492 secs 2024-07-26T16:47:48.198 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:47 smithi107 ceph-mon[32659]: pgmap v7948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:47 smithi112 ceph-mon[37501]: pgmap v7948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:47 smithi132 ceph-mon[37580]: pgmap v7948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:48.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:48.502+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:47:48.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:48.502+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:47:48.504 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:47:48.502+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:47:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:49 smithi112 ceph-mon[37501]: pgmap v7949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:47:50.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:49 smithi107 ceph-mon[32659]: pgmap v7949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:47:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:49 smithi132 ceph-mon[37580]: pgmap v7949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:47:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14207.070586 secs 2024-07-26T16:47:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:51 smithi112 ceph-mon[37501]: pgmap v7950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14207.070586 secs 2024-07-26T16:47:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:51 smithi107 ceph-mon[32659]: pgmap v7950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14207.070586 secs 2024-07-26T16:47:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:51 smithi132 ceph-mon[37580]: pgmap v7950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:47:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:53 smithi112 ceph-mon[37501]: pgmap v7951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:54.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:53 smithi107 ceph-mon[32659]: pgmap v7951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:53 smithi132 ceph-mon[37580]: pgmap v7951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:47:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:55 smithi112 ceph-mon[37501]: pgmap v7952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:47:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:55 smithi107 ceph-mon[32659]: pgmap v7952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:47:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:55 smithi132 ceph-mon[37580]: pgmap v7952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:47:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14212.070740 secs 2024-07-26T16:47:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14212.070740 secs 2024-07-26T16:47:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14212.070740 secs 2024-07-26T16:47:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:57 smithi112 ceph-mon[37501]: pgmap v7953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:57 smithi107 ceph-mon[32659]: pgmap v7953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:47:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:57 smithi132 ceph-mon[37580]: pgmap v7953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:47:59 smithi112 ceph-mon[37501]: pgmap v7954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:48:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:47:59 smithi107 ceph-mon[32659]: pgmap v7954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:48:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:47:59 smithi132 ceph-mon[37580]: pgmap v7954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:48:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14217.070872 secs 2024-07-26T16:48:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:01 smithi112 ceph-mon[37501]: pgmap v7955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:48:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14217.070872 secs 2024-07-26T16:48:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:01 smithi107 ceph-mon[32659]: pgmap v7955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:48:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14217.070872 secs 2024-07-26T16:48:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:01 smithi132 ceph-mon[37580]: pgmap v7955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:48:03.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:03.506+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:48:03.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:03.506+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:48:03.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:03.506+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:48:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:03 smithi112 ceph-mon[37501]: pgmap v7956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:48:04.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:03 smithi107 ceph-mon[32659]: pgmap v7956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:48:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:03 smithi132 ceph-mon[37580]: pgmap v7956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:48:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:05 smithi112 ceph-mon[37501]: pgmap v7957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:05 smithi107 ceph-mon[32659]: pgmap v7957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:05 smithi132 ceph-mon[37580]: pgmap v7957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14222.070993 secs 2024-07-26T16:48:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14222.070993 secs 2024-07-26T16:48:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14222.070993 secs 2024-07-26T16:48:08.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:07 smithi107 ceph-mon[32659]: pgmap v7958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:48:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:07 smithi112 ceph-mon[37501]: pgmap v7958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:48:08.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:07 smithi132 ceph-mon[37580]: pgmap v7958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:48:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:09 smithi112 ceph-mon[37501]: pgmap v7959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:48:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:09 smithi107 ceph-mon[32659]: pgmap v7959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:48:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:09 smithi132 ceph-mon[37580]: pgmap v7959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:48:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14227.071115 secs 2024-07-26T16:48:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:11 smithi112 ceph-mon[37501]: pgmap v7960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14227.071115 secs 2024-07-26T16:48:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:11 smithi107 ceph-mon[32659]: pgmap v7960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14227.071115 secs 2024-07-26T16:48:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:11 smithi132 ceph-mon[37580]: pgmap v7960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:13 smithi112 ceph-mon[37501]: pgmap v7961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:13 smithi107 ceph-mon[32659]: pgmap v7961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:13 smithi132 ceph-mon[37580]: pgmap v7961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:15.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:15.091+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:48:15.094 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:15.091+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:48:15.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:15.092+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:48:15.095 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:15.092+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:48:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:15 smithi112 ceph-mon[37501]: pgmap v7962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:15 smithi107 ceph-mon[32659]: pgmap v7962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:15 smithi132 ceph-mon[37580]: pgmap v7962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14232.071255 secs 2024-07-26T16:48:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14232.071255 secs 2024-07-26T16:48:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14232.071255 secs 2024-07-26T16:48:18.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:17 smithi112 ceph-mon[37501]: pgmap v7963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:17 smithi107 ceph-mon[32659]: pgmap v7963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:17 smithi132 ceph-mon[37580]: pgmap v7963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:18.523 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:18.521+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:48:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:18.521+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:48:18.524 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:18.521+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:48:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:18 smithi112 ceph-mon[37501]: pgmap v7964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:18 smithi107 ceph-mon[32659]: pgmap v7964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:18 smithi132 ceph-mon[37580]: pgmap v7964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14237.071314 secs 2024-07-26T16:48:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:21 smithi112 ceph-mon[37501]: pgmap v7965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14237.071314 secs 2024-07-26T16:48:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:21 smithi107 ceph-mon[32659]: pgmap v7965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14237.071314 secs 2024-07-26T16:48:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:21 smithi132 ceph-mon[37580]: pgmap v7965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:23 smithi112 ceph-mon[37501]: pgmap v7966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:23 smithi107 ceph-mon[32659]: pgmap v7966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:23 smithi132 ceph-mon[37580]: pgmap v7966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:25 smithi112 ceph-mon[37501]: pgmap v7967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:25 smithi107 ceph-mon[32659]: pgmap v7967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:26.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:25 smithi132 ceph-mon[37580]: pgmap v7967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14242.071436 secs 2024-07-26T16:48:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14242.071436 secs 2024-07-26T16:48:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14242.071436 secs 2024-07-26T16:48:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:27 smithi112 ceph-mon[37501]: pgmap v7968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:27 smithi107 ceph-mon[32659]: pgmap v7968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:27 smithi132 ceph-mon[37580]: pgmap v7968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:29 smithi112 ceph-mon[37501]: pgmap v7969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:30.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:29 smithi107 ceph-mon[32659]: pgmap v7969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:29 smithi132 ceph-mon[37580]: pgmap v7969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14247.071556 secs 2024-07-26T16:48:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:31 smithi112 ceph-mon[37501]: pgmap v7970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14247.071556 secs 2024-07-26T16:48:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:31 smithi107 ceph-mon[32659]: pgmap v7970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14247.071556 secs 2024-07-26T16:48:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:31 smithi132 ceph-mon[37580]: pgmap v7970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:33.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:33.531+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:48:33.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:33.531+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:48:33.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:33.531+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:48:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:33 smithi107 ceph-mon[32659]: pgmap v7971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:33 smithi112 ceph-mon[37501]: pgmap v7971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:34.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:33 smithi132 ceph-mon[37580]: pgmap v7971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:35 smithi112 ceph-mon[37501]: pgmap v7972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:35 smithi107 ceph-mon[32659]: pgmap v7972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:35 smithi132 ceph-mon[37580]: pgmap v7972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14252.071671 secs 2024-07-26T16:48:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14252.071671 secs 2024-07-26T16:48:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14252.071671 secs 2024-07-26T16:48:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:37 smithi112 ceph-mon[37501]: pgmap v7973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:37 smithi107 ceph-mon[32659]: pgmap v7973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:37 smithi132 ceph-mon[37580]: pgmap v7973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:39.920 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:39 smithi112 ceph-mon[37501]: pgmap v7974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:39.920 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:48:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:39 smithi107 ceph-mon[32659]: pgmap v7974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:48:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:39 smithi132 ceph-mon[37580]: pgmap v7974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:48:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14257.071791 secs 2024-07-26T16:48:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:41 smithi107 ceph-mon[32659]: pgmap v7975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14257.071791 secs 2024-07-26T16:48:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:41 smithi132 ceph-mon[37580]: pgmap v7975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:42.073 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14257.071791 secs 2024-07-26T16:48:42.074 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:41 smithi112 ceph-mon[37501]: pgmap v7975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:48:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:43 smithi112 ceph-mon[37501]: pgmap v7976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:43 smithi107 ceph-mon[32659]: pgmap v7976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:43 smithi132 ceph-mon[37580]: pgmap v7976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:48:44.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:44.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: pgmap v7977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:45.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:45.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:45.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:45.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:45.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:45.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:45.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:45.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:45.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:48:45.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:48:45.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:46.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: pgmap v7977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:46.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:46.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:46.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:48:46.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:48:46.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: pgmap v7977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:48:46.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:46.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:46.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:48:46.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:48:46.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:48:46.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:48:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:46 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:48:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:46 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:48:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:46 smithi112 ceph-mon[37501]: pgmap v7978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:48:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:46 smithi112 ceph-mon[37501]: pgmap v7979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:48:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14262.071918 secs 2024-07-26T16:48:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:46 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:48:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:46 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:48:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:46 smithi107 ceph-mon[32659]: pgmap v7978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:48:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:46 smithi107 ceph-mon[32659]: pgmap v7979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:48:47.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14262.071918 secs 2024-07-26T16:48:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:46 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:48:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:46 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:48:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:46 smithi132 ceph-mon[37580]: pgmap v7978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:48:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:46 smithi132 ceph-mon[37580]: pgmap v7979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:48:47.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14262.071918 secs 2024-07-26T16:48:48.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:48.546+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:48:48.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:48.546+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:48:48.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:48:48.546+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:48:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:48 smithi112 ceph-mon[37501]: pgmap v7980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T16:48:49.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:48 smithi107 ceph-mon[32659]: pgmap v7980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T16:48:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:48 smithi132 ceph-mon[37580]: pgmap v7980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T16:48:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:50 smithi112 ceph-mon[37501]: pgmap v7981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:48:51.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:50 smithi107 ceph-mon[32659]: pgmap v7981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:48:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:50 smithi132 ceph-mon[37580]: pgmap v7981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:48:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14267.072061 secs 2024-07-26T16:48:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14267.072061 secs 2024-07-26T16:48:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14267.072061 secs 2024-07-26T16:48:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:52 smithi112 ceph-mon[37501]: pgmap v7982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T16:48:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:52 smithi107 ceph-mon[32659]: pgmap v7982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T16:48:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:52 smithi132 ceph-mon[37580]: pgmap v7982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T16:48:54.730 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:54 smithi112 ceph-mon[37501]: pgmap v7983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:48:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:54 smithi107 ceph-mon[32659]: pgmap v7983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:48:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:54 smithi132 ceph-mon[37580]: pgmap v7983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:48:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:56 smithi112 ceph-mon[37501]: pgmap v7984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:48:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14272.072178 secs 2024-07-26T16:48:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:56 smithi107 ceph-mon[32659]: pgmap v7984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:48:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14272.072178 secs 2024-07-26T16:48:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:56 smithi132 ceph-mon[37580]: pgmap v7984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:48:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14272.072178 secs 2024-07-26T16:48:58.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:48:58 smithi112 ceph-mon[37501]: pgmap v7985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:59.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:48:58 smithi107 ceph-mon[32659]: pgmap v7985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:48:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:48:58 smithi132 ceph-mon[37580]: pgmap v7985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:00 smithi112 ceph-mon[37501]: pgmap v7986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:00 smithi107 ceph-mon[32659]: pgmap v7986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:00 smithi132 ceph-mon[37580]: pgmap v7986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14277.072288 secs 2024-07-26T16:49:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14277.072288 secs 2024-07-26T16:49:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14277.072288 secs 2024-07-26T16:49:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:02 smithi112 ceph-mon[37501]: pgmap v7987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:02 smithi107 ceph-mon[32659]: pgmap v7987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:02 smithi132 ceph-mon[37580]: pgmap v7987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:03.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:03.562+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:49:03.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:03.562+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:49:03.565 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:03.562+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:49:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:04 smithi112 ceph-mon[37501]: pgmap v7988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:05.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:04 smithi107 ceph-mon[32659]: pgmap v7988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:04 smithi132 ceph-mon[37580]: pgmap v7988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:06 smithi112 ceph-mon[37501]: pgmap v7989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14282.072410 secs 2024-07-26T16:49:07.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:06 smithi107 ceph-mon[32659]: pgmap v7989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:07.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14282.072410 secs 2024-07-26T16:49:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:06 smithi132 ceph-mon[37580]: pgmap v7989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14282.072410 secs 2024-07-26T16:49:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:08 smithi112 ceph-mon[37501]: pgmap v7990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:09.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:08 smithi107 ceph-mon[32659]: pgmap v7990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:08 smithi132 ceph-mon[37580]: pgmap v7990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:10 smithi107 ceph-mon[32659]: pgmap v7991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:10 smithi132 ceph-mon[37580]: pgmap v7991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:10 smithi112 ceph-mon[37501]: pgmap v7991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14287.072533 secs 2024-07-26T16:49:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14287.072533 secs 2024-07-26T16:49:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14287.072533 secs 2024-07-26T16:49:13.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:12 smithi107 ceph-mon[32659]: pgmap v7992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:12 smithi132 ceph-mon[37580]: pgmap v7992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:12 smithi112 ceph-mon[37501]: pgmap v7992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:14 smithi107 ceph-mon[32659]: pgmap v7993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:14 smithi132 ceph-mon[37580]: pgmap v7993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:14 smithi112 ceph-mon[37501]: pgmap v7993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:16 smithi107 ceph-mon[32659]: pgmap v7994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14292.072655 secs 2024-07-26T16:49:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:16 smithi132 ceph-mon[37580]: pgmap v7994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14292.072655 secs 2024-07-26T16:49:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:16 smithi112 ceph-mon[37501]: pgmap v7994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14292.072655 secs 2024-07-26T16:49:18.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:18.576+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:49:18.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:18.576+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:49:18.579 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:18.576+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:49:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:18 smithi107 ceph-mon[32659]: pgmap v7995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:18 smithi132 ceph-mon[37580]: pgmap v7995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:18 smithi112 ceph-mon[37501]: pgmap v7995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:20 smithi107 ceph-mon[32659]: pgmap v7996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:20 smithi132 ceph-mon[37580]: pgmap v7996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:20 smithi112 ceph-mon[37501]: pgmap v7996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14297.072776 secs 2024-07-26T16:49:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14297.072776 secs 2024-07-26T16:49:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14297.072776 secs 2024-07-26T16:49:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:22 smithi107 ceph-mon[32659]: pgmap v7997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:22 smithi132 ceph-mon[37580]: pgmap v7997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:23.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:22 smithi112 ceph-mon[37501]: pgmap v7997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:25.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:24 smithi107 ceph-mon[32659]: pgmap v7998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:24 smithi132 ceph-mon[37580]: pgmap v7998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:24 smithi112 ceph-mon[37501]: pgmap v7998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:26 smithi107 ceph-mon[32659]: pgmap v7999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14302.072918 secs 2024-07-26T16:49:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:26 smithi132 ceph-mon[37580]: pgmap v7999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14302.072918 secs 2024-07-26T16:49:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:26 smithi112 ceph-mon[37501]: pgmap v7999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14302.072918 secs 2024-07-26T16:49:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:28 smithi107 ceph-mon[32659]: pgmap v8000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:28 smithi132 ceph-mon[37580]: pgmap v8000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:28 smithi112 ceph-mon[37501]: pgmap v8000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:30 smithi107 ceph-mon[32659]: pgmap v8001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:30 smithi132 ceph-mon[37580]: pgmap v8001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:30 smithi112 ceph-mon[37501]: pgmap v8001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14307.073043 secs 2024-07-26T16:49:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14307.073043 secs 2024-07-26T16:49:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14307.073043 secs 2024-07-26T16:49:33.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:32 smithi107 ceph-mon[32659]: pgmap v8002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:32 smithi132 ceph-mon[37580]: pgmap v8002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:32 smithi112 ceph-mon[37501]: pgmap v8002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:33.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:33.592+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:49:33.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:33.592+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:49:33.595 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:33.592+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:49:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:34 smithi132 ceph-mon[37580]: pgmap v8003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:34 smithi112 ceph-mon[37501]: pgmap v8003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:34 smithi107 ceph-mon[32659]: pgmap v8003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:36 smithi132 ceph-mon[37580]: pgmap v8004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:37.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14312.073146 secs 2024-07-26T16:49:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:36 smithi112 ceph-mon[37501]: pgmap v8004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14312.073146 secs 2024-07-26T16:49:37.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:36 smithi107 ceph-mon[32659]: pgmap v8004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:37.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14312.073146 secs 2024-07-26T16:49:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:38 smithi132 ceph-mon[37580]: pgmap v8005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:38 smithi112 ceph-mon[37501]: pgmap v8005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:39.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:38 smithi107 ceph-mon[32659]: pgmap v8005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:41.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:40 smithi107 ceph-mon[32659]: pgmap v8006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:40 smithi112 ceph-mon[37501]: pgmap v8006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:40 smithi132 ceph-mon[37580]: pgmap v8006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14317.073265 secs 2024-07-26T16:49:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14317.073265 secs 2024-07-26T16:49:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14317.073265 secs 2024-07-26T16:49:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:42 smithi112 ceph-mon[37501]: pgmap v8007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:43.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:42 smithi107 ceph-mon[32659]: pgmap v8007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:42 smithi132 ceph-mon[37580]: pgmap v8007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:49:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:44 smithi112 ceph-mon[37501]: pgmap v8008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:44 smithi107 ceph-mon[32659]: pgmap v8008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:44 smithi132 ceph-mon[37580]: pgmap v8008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:46.182 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:49:46.183 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.183 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.183 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.183 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.183 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.183 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.183 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.184 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:49:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:49:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:46.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:46 smithi112 ceph-mon[37501]: pgmap v8009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14322.073344 secs 2024-07-26T16:49:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:46 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:49:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:46 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:49:47.271 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:46 smithi132 ceph-mon[37580]: pgmap v8009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:47.271 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14322.073344 secs 2024-07-26T16:49:47.271 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:46 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:49:47.272 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:46 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:49:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:46 smithi107 ceph-mon[32659]: pgmap v8009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14322.073344 secs 2024-07-26T16:49:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:46 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T16:49:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:46 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T16:49:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:48.608+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:49:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:48.608+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:49:48.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:49:48.608+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:49:49.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:48 smithi107 ceph-mon[32659]: pgmap v8010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:48 smithi132 ceph-mon[37580]: pgmap v8010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:48 smithi112 ceph-mon[37501]: pgmap v8010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:49:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: pgmap v8011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:50.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:49:50.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:49:50.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:49:50.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:49:50.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:49:50.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:49:50.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: pgmap v8012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:49:50.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: pgmap v8013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:49:50.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:49:50.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: pgmap v8011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:50.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:49:50.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:49:50.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:49:50.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:49:50.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:49:50.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:49:50.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: pgmap v8012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:49:50.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: pgmap v8013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:49:50.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:49:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: pgmap v8011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:49:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:49:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:49:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:49:50.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T16:49:50.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T16:49:50.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:49:50.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:49:50.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: pgmap v8012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:49:50.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: pgmap v8013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:49:50.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:49:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14327.073419 secs 2024-07-26T16:49:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:52 smithi107 ceph-mon[32659]: pgmap v8014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:49:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14327.073419 secs 2024-07-26T16:49:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:52 smithi132 ceph-mon[37580]: pgmap v8014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:49:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14327.073419 secs 2024-07-26T16:49:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:52 smithi112 ceph-mon[37501]: pgmap v8014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:49:54.502 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:54 smithi112 ceph-mon[37501]: pgmap v8015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:49:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:54 smithi107 ceph-mon[32659]: pgmap v8015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:49:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:54 smithi132 ceph-mon[37580]: pgmap v8015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:49:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:56 smithi107 ceph-mon[32659]: pgmap v8016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:49:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:56 smithi132 ceph-mon[37580]: pgmap v8016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:49:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:56 smithi112 ceph-mon[37501]: pgmap v8016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:49:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14332.073518 secs 2024-07-26T16:49:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14332.073518 secs 2024-07-26T16:49:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14332.073518 secs 2024-07-26T16:49:58.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:49:58 smithi107 ceph-mon[32659]: pgmap v8017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:49:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:49:58 smithi132 ceph-mon[37580]: pgmap v8017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:49:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:49:58 smithi112 ceph-mon[37501]: pgmap v8017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:50:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:00 smithi112 ceph-mon[37501]: pgmap v8018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:50:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:50:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:00 smithi107 ceph-mon[32659]: pgmap v8018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:50:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:50:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:00 smithi132 ceph-mon[37580]: pgmap v8018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:50:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T16:50:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14337.073641 secs 2024-07-26T16:50:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:02 smithi107 ceph-mon[32659]: pgmap v8019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14337.073641 secs 2024-07-26T16:50:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:02 smithi132 ceph-mon[37580]: pgmap v8019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14337.073641 secs 2024-07-26T16:50:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:02 smithi112 ceph-mon[37501]: pgmap v8019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:03.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:03.624+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:50:03.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:03.624+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:50:03.626 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:03.624+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:50:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:04 smithi107 ceph-mon[32659]: pgmap v8020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:04 smithi132 ceph-mon[37580]: pgmap v8020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:04 smithi112 ceph-mon[37501]: pgmap v8020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:06 smithi107 ceph-mon[32659]: pgmap v8021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:06 smithi132 ceph-mon[37580]: pgmap v8021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:06 smithi112 ceph-mon[37501]: pgmap v8021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14342.073770 secs 2024-07-26T16:50:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14342.073770 secs 2024-07-26T16:50:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14342.073770 secs 2024-07-26T16:50:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:08 smithi107 ceph-mon[32659]: pgmap v8022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:08 smithi132 ceph-mon[37580]: pgmap v8022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:08 smithi112 ceph-mon[37501]: pgmap v8022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:10 smithi107 ceph-mon[32659]: pgmap v8023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:10 smithi132 ceph-mon[37580]: pgmap v8023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:10 smithi112 ceph-mon[37501]: pgmap v8023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14347.073873 secs 2024-07-26T16:50:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14347.073873 secs 2024-07-26T16:50:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14347.073873 secs 2024-07-26T16:50:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:12 smithi132 ceph-mon[37580]: pgmap v8024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:12 smithi112 ceph-mon[37501]: pgmap v8024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:12.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:12 smithi107 ceph-mon[32659]: pgmap v8024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:14 smithi132 ceph-mon[37580]: pgmap v8025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:14 smithi112 ceph-mon[37501]: pgmap v8025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:14.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:14 smithi107 ceph-mon[32659]: pgmap v8025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:16 smithi112 ceph-mon[37501]: pgmap v8026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14352.073972 secs 2024-07-26T16:50:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:16 smithi107 ceph-mon[32659]: pgmap v8026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14352.073972 secs 2024-07-26T16:50:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:16 smithi132 ceph-mon[37580]: pgmap v8026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14352.073972 secs 2024-07-26T16:50:18.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:18.639+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:50:18.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:18.639+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:50:18.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:18.639+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:50:18.643 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:18 smithi107 ceph-mon[32659]: pgmap v8027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:18 smithi112 ceph-mon[37501]: pgmap v8027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:18 smithi132 ceph-mon[37580]: pgmap v8027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:20 smithi112 ceph-mon[37501]: pgmap v8028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:20 smithi107 ceph-mon[32659]: pgmap v8028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:20 smithi132 ceph-mon[37580]: pgmap v8028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:21.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14357.074108 secs 2024-07-26T16:50:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14357.074108 secs 2024-07-26T16:50:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14357.074108 secs 2024-07-26T16:50:22.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:22 smithi112 ceph-mon[37501]: pgmap v8029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:22 smithi107 ceph-mon[32659]: pgmap v8029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:22 smithi132 ceph-mon[37580]: pgmap v8029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:24 smithi112 ceph-mon[37501]: pgmap v8030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:24.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:24 smithi107 ceph-mon[32659]: pgmap v8030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:24 smithi132 ceph-mon[37580]: pgmap v8030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:26 smithi112 ceph-mon[37501]: pgmap v8031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14362.074238 secs 2024-07-26T16:50:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:26 smithi107 ceph-mon[32659]: pgmap v8031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14362.074238 secs 2024-07-26T16:50:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:26 smithi132 ceph-mon[37580]: pgmap v8031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:26.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14362.074238 secs 2024-07-26T16:50:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:28 smithi112 ceph-mon[37501]: pgmap v8032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:28.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:28 smithi107 ceph-mon[32659]: pgmap v8032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:28 smithi132 ceph-mon[37580]: pgmap v8032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:29.115+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:50:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:29.115+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:50:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:29.115+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:50:29.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:29.115+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:50:30.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:30 smithi112 ceph-mon[37501]: pgmap v8033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:30 smithi107 ceph-mon[32659]: pgmap v8033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:30 smithi132 ceph-mon[37580]: pgmap v8033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14367.074381 secs 2024-07-26T16:50:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14367.074381 secs 2024-07-26T16:50:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14367.074381 secs 2024-07-26T16:50:32.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:32 smithi112 ceph-mon[37501]: pgmap v8034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:32.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:32 smithi107 ceph-mon[32659]: pgmap v8034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:32 smithi132 ceph-mon[37580]: pgmap v8034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:33.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:33.655+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:50:33.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:33.655+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:50:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:33.655+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:50:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:34 smithi112 ceph-mon[37501]: pgmap v8035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:34.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:34 smithi107 ceph-mon[32659]: pgmap v8035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:34 smithi132 ceph-mon[37580]: pgmap v8035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:36 smithi112 ceph-mon[37501]: pgmap v8036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14372.074511 secs 2024-07-26T16:50:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:36 smithi107 ceph-mon[32659]: pgmap v8036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14372.074511 secs 2024-07-26T16:50:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:36 smithi132 ceph-mon[37580]: pgmap v8036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:36.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14372.074511 secs 2024-07-26T16:50:38.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:38 smithi112 ceph-mon[37501]: pgmap v8037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:38.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:38 smithi107 ceph-mon[32659]: pgmap v8037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:38 smithi132 ceph-mon[37580]: pgmap v8037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:40 smithi112 ceph-mon[37501]: pgmap v8038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:40 smithi107 ceph-mon[32659]: pgmap v8038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:40 smithi132 ceph-mon[37580]: pgmap v8038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:41.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14377.074604 secs 2024-07-26T16:50:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14377.074604 secs 2024-07-26T16:50:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14377.074604 secs 2024-07-26T16:50:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:42 smithi112 ceph-mon[37501]: pgmap v8039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:42.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:42 smithi107 ceph-mon[32659]: pgmap v8039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:42 smithi132 ceph-mon[37580]: pgmap v8039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:50:44.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:44 smithi112 ceph-mon[37501]: pgmap v8040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:44.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:44 smithi107 ceph-mon[32659]: pgmap v8040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:44 smithi132 ceph-mon[37580]: pgmap v8040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:46 smithi112 ceph-mon[37501]: pgmap v8041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14382.074723 secs 2024-07-26T16:50:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:46 smithi107 ceph-mon[32659]: pgmap v8041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14382.074723 secs 2024-07-26T16:50:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:46 smithi132 ceph-mon[37580]: pgmap v8041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14382.074723 secs 2024-07-26T16:50:48.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:48.671+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:50:48.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:48.671+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:50:48.674 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:50:48.671+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:50:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:48 smithi112 ceph-mon[37501]: pgmap v8042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:48.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:48 smithi107 ceph-mon[32659]: pgmap v8042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:48 smithi132 ceph-mon[37580]: pgmap v8042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:50:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:50:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:50:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:50:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:50 smithi112 ceph-mon[37501]: pgmap v8043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:50:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:50:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:50:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:50 smithi107 ceph-mon[32659]: pgmap v8043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:50:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:50:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:50:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:50 smithi132 ceph-mon[37580]: pgmap v8043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:50:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:50:50.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:50:50.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:50:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:51 smithi112 ceph-mon[37501]: pgmap v8044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:50:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:51 smithi112 ceph-mon[37501]: pgmap v8045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:50:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14387.074845 secs 2024-07-26T16:50:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:51 smithi107 ceph-mon[32659]: pgmap v8044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:50:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:51 smithi107 ceph-mon[32659]: pgmap v8045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:50:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14387.074845 secs 2024-07-26T16:50:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:51 smithi132 ceph-mon[37580]: pgmap v8044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:50:51.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:51 smithi132 ceph-mon[37580]: pgmap v8045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:50:51.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14387.074845 secs 2024-07-26T16:50:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:53 smithi107 ceph-mon[32659]: pgmap v8046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:50:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:53 smithi132 ceph-mon[37580]: pgmap v8046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:50:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:53 smithi112 ceph-mon[37501]: pgmap v8046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:50:55.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:55 smithi112 ceph-mon[37501]: pgmap v8047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:50:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:55 smithi107 ceph-mon[32659]: pgmap v8047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:50:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:55 smithi132 ceph-mon[37580]: pgmap v8047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:50:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14392.074966 secs 2024-07-26T16:50:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14392.074966 secs 2024-07-26T16:50:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14392.074966 secs 2024-07-26T16:50:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:57 smithi107 ceph-mon[32659]: pgmap v8048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:50:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:57 smithi132 ceph-mon[37580]: pgmap v8048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:50:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:57 smithi112 ceph-mon[37501]: pgmap v8048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:50:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:50:59 smithi107 ceph-mon[32659]: pgmap v8049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:50:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:50:59 smithi132 ceph-mon[37580]: pgmap v8049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:50:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:50:59 smithi112 ceph-mon[37501]: pgmap v8049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:51:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:01 smithi107 ceph-mon[32659]: pgmap v8050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:51:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14397.075080 secs 2024-07-26T16:51:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:01 smithi132 ceph-mon[37580]: pgmap v8050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:51:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14397.075080 secs 2024-07-26T16:51:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:01 smithi112 ceph-mon[37501]: pgmap v8050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:51:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14397.075080 secs 2024-07-26T16:51:03.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:03.681+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:51:03.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:03.681+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:51:03.683 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:03.681+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:51:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:03 smithi107 ceph-mon[32659]: pgmap v8051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:03 smithi132 ceph-mon[37580]: pgmap v8051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:03 smithi112 ceph-mon[37501]: pgmap v8051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:05.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:05 smithi107 ceph-mon[32659]: pgmap v8052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:05 smithi132 ceph-mon[37580]: pgmap v8052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:05 smithi112 ceph-mon[37501]: pgmap v8052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14402.075195 secs 2024-07-26T16:51:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14402.075195 secs 2024-07-26T16:51:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14402.075195 secs 2024-07-26T16:51:07.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:07 smithi107 ceph-mon[32659]: pgmap v8053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:07 smithi132 ceph-mon[37580]: pgmap v8053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:07.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:07 smithi112 ceph-mon[37501]: pgmap v8053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:09 smithi107 ceph-mon[32659]: pgmap v8054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:09 smithi132 ceph-mon[37580]: pgmap v8054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:09.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:09 smithi112 ceph-mon[37501]: pgmap v8054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:11.683+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T16:51:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:11.683+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T16:51:11.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:11.683+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f806f040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:51:11.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:11.683+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f806f040 comp rx=0 tx=0).stop 2024-07-26T16:51:11.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:11.683+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T16:51:11.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:11.683+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:51:11.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:11.683+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:51:11.686 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:11.683+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:51:11.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:11 smithi107 ceph-mon[32659]: pgmap v8055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:11.687 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14407.075310 secs 2024-07-26T16:51:11.687 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:11.684+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:51:11.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:11.686+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8079040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:51:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:11 smithi132 ceph-mon[37580]: pgmap v8055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14407.075310 secs 2024-07-26T16:51:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:11 smithi112 ceph-mon[37501]: pgmap v8055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14407.075310 secs 2024-07-26T16:51:13.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:13 smithi107 ceph-mon[32659]: pgmap v8056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:13.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:13 smithi132 ceph-mon[37580]: pgmap v8056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:13 smithi112 ceph-mon[37501]: pgmap v8056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:15.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:15.123+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:51:15.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:15.123+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:51:15.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:15.123+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:51:15.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:15.123+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:51:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:15 smithi132 ceph-mon[37580]: pgmap v8057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:15 smithi112 ceph-mon[37501]: pgmap v8057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:15 smithi107 ceph-mon[32659]: pgmap v8057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14412.075389 secs 2024-07-26T16:51:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14412.075389 secs 2024-07-26T16:51:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14412.075389 secs 2024-07-26T16:51:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:17 smithi112 ceph-mon[37501]: pgmap v8058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:17 smithi107 ceph-mon[32659]: pgmap v8058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:17 smithi132 ceph-mon[37580]: pgmap v8058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:18.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:18.682+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:51:18.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:18.682+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:51:18.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:18.682+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:51:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:19 smithi112 ceph-mon[37501]: pgmap v8059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:19 smithi107 ceph-mon[32659]: pgmap v8059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:19 smithi132 ceph-mon[37580]: pgmap v8059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:21 smithi112 ceph-mon[37501]: pgmap v8060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14417.075507 secs 2024-07-26T16:51:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:21 smithi107 ceph-mon[32659]: pgmap v8060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14417.075507 secs 2024-07-26T16:51:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:21 smithi132 ceph-mon[37580]: pgmap v8060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14417.075507 secs 2024-07-26T16:51:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:23 smithi112 ceph-mon[37501]: pgmap v8061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:23 smithi107 ceph-mon[32659]: pgmap v8061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:23 smithi132 ceph-mon[37580]: pgmap v8061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:25 smithi112 ceph-mon[37501]: pgmap v8062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:25 smithi107 ceph-mon[32659]: pgmap v8062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:25 smithi132 ceph-mon[37580]: pgmap v8062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14422.075646 secs 2024-07-26T16:51:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14422.075646 secs 2024-07-26T16:51:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14422.075646 secs 2024-07-26T16:51:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:27 smithi112 ceph-mon[37501]: pgmap v8063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:27 smithi107 ceph-mon[32659]: pgmap v8063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:27 smithi132 ceph-mon[37580]: pgmap v8063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:29 smithi112 ceph-mon[37501]: pgmap v8064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:51:30.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:29 smithi107 ceph-mon[32659]: pgmap v8064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:51:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:29 smithi132 ceph-mon[37580]: pgmap v8064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:51:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:31 smithi112 ceph-mon[37501]: pgmap v8065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14427.075726 secs 2024-07-26T16:51:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:31 smithi107 ceph-mon[32659]: pgmap v8065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14427.075726 secs 2024-07-26T16:51:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:31 smithi132 ceph-mon[37580]: pgmap v8065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14427.075726 secs 2024-07-26T16:51:33.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:33.698+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:51:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:33.698+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:51:33.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:33.698+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:51:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:33 smithi112 ceph-mon[37501]: pgmap v8066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:33 smithi107 ceph-mon[32659]: pgmap v8066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:33 smithi132 ceph-mon[37580]: pgmap v8066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:35 smithi112 ceph-mon[37501]: pgmap v8067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:35 smithi107 ceph-mon[32659]: pgmap v8067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:35 smithi132 ceph-mon[37580]: pgmap v8067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14432.075846 secs 2024-07-26T16:51:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14432.075846 secs 2024-07-26T16:51:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14432.075846 secs 2024-07-26T16:51:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:37 smithi112 ceph-mon[37501]: pgmap v8068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:37 smithi107 ceph-mon[32659]: pgmap v8068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:37 smithi132 ceph-mon[37580]: pgmap v8068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:39 smithi112 ceph-mon[37501]: pgmap v8069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:39 smithi107 ceph-mon[32659]: pgmap v8069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:39 smithi132 ceph-mon[37580]: pgmap v8069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:41 smithi112 ceph-mon[37501]: pgmap v8070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14437.075982 secs 2024-07-26T16:51:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:41 smithi107 ceph-mon[32659]: pgmap v8070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14437.075982 secs 2024-07-26T16:51:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:41 smithi132 ceph-mon[37580]: pgmap v8070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14437.075982 secs 2024-07-26T16:51:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:43 smithi112 ceph-mon[37501]: pgmap v8071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:44.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:43 smithi107 ceph-mon[32659]: pgmap v8071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:43 smithi132 ceph-mon[37580]: pgmap v8071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:45 smithi112 ceph-mon[37501]: pgmap v8072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:45 smithi107 ceph-mon[32659]: pgmap v8072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:45 smithi132 ceph-mon[37580]: pgmap v8072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14442.076089 secs 2024-07-26T16:51:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14442.076089 secs 2024-07-26T16:51:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14442.076089 secs 2024-07-26T16:51:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:47 smithi112 ceph-mon[37501]: pgmap v8073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:48.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:47 smithi107 ceph-mon[32659]: pgmap v8073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:47 smithi132 ceph-mon[37580]: pgmap v8073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:51:48.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:48.714+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:51:48.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:48.714+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:51:48.716 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:51:48.714+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:51:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:49 smithi112 ceph-mon[37501]: pgmap v8074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:49 smithi107 ceph-mon[32659]: pgmap v8074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:49 smithi132 ceph-mon[37580]: pgmap v8074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:51:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:51:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:51:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:51:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:51 smithi112 ceph-mon[37501]: pgmap v8075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14447.076210 secs 2024-07-26T16:51:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:51:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:51:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:51 smithi107 ceph-mon[32659]: pgmap v8075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14447.076210 secs 2024-07-26T16:51:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:51:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:51:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:51 smithi132 ceph-mon[37580]: pgmap v8075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:51:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14447.076210 secs 2024-07-26T16:51:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:51:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:51:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:51:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:51:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:51:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:51:53.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:51:53.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:51:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:51:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:51:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:51:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:53 smithi107 ceph-mon[32659]: pgmap v8076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:51:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:53 smithi107 ceph-mon[32659]: pgmap v8077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:51:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:53 smithi132 ceph-mon[37580]: pgmap v8076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:51:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:53 smithi132 ceph-mon[37580]: pgmap v8077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:51:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:53 smithi112 ceph-mon[37501]: pgmap v8076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:51:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:53 smithi112 ceph-mon[37501]: pgmap v8077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:51:56.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:55 smithi112 ceph-mon[37501]: pgmap v8078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:51:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:55 smithi107 ceph-mon[32659]: pgmap v8078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:51:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:55 smithi132 ceph-mon[37580]: pgmap v8078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:51:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14452.076301 secs 2024-07-26T16:51:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14452.076301 secs 2024-07-26T16:51:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14452.076301 secs 2024-07-26T16:51:58.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:57 smithi107 ceph-mon[32659]: pgmap v8079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:51:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:57 smithi132 ceph-mon[37580]: pgmap v8079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:51:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:57 smithi112 ceph-mon[37501]: pgmap v8079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:52:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:51:59 smithi107 ceph-mon[32659]: pgmap v8080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:52:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:51:59 smithi132 ceph-mon[37580]: pgmap v8080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:52:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:51:59 smithi112 ceph-mon[37501]: pgmap v8080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:52:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:01 smithi107 ceph-mon[32659]: pgmap v8081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:52:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14457.076377 secs 2024-07-26T16:52:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:01 smithi132 ceph-mon[37580]: pgmap v8081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:52:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14457.076377 secs 2024-07-26T16:52:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:01 smithi112 ceph-mon[37501]: pgmap v8081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:52:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14457.076377 secs 2024-07-26T16:52:03.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:03.721+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:52:03.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:03.721+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:52:03.725 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:03.721+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:52:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:03 smithi107 ceph-mon[32659]: pgmap v8082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:52:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:03 smithi132 ceph-mon[37580]: pgmap v8082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:52:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:03 smithi112 ceph-mon[37501]: pgmap v8082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:52:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:05 smithi107 ceph-mon[32659]: pgmap v8083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:05 smithi132 ceph-mon[37580]: pgmap v8083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:05 smithi112 ceph-mon[37501]: pgmap v8083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14462.076490 secs 2024-07-26T16:52:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14462.076490 secs 2024-07-26T16:52:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14462.076490 secs 2024-07-26T16:52:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:07 smithi107 ceph-mon[32659]: pgmap v8084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:07 smithi132 ceph-mon[37580]: pgmap v8084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:07 smithi112 ceph-mon[37501]: pgmap v8084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:09 smithi107 ceph-mon[32659]: pgmap v8085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:09 smithi132 ceph-mon[37580]: pgmap v8085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:09 smithi112 ceph-mon[37501]: pgmap v8085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:11 smithi107 ceph-mon[32659]: pgmap v8086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14467.076606 secs 2024-07-26T16:52:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:11 smithi132 ceph-mon[37580]: pgmap v8086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14467.076606 secs 2024-07-26T16:52:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:11 smithi112 ceph-mon[37501]: pgmap v8086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14467.076606 secs 2024-07-26T16:52:14.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:13 smithi107 ceph-mon[32659]: pgmap v8087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:13 smithi132 ceph-mon[37580]: pgmap v8087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:13 smithi112 ceph-mon[37501]: pgmap v8087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:15 smithi107 ceph-mon[32659]: pgmap v8088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:15 smithi132 ceph-mon[37580]: pgmap v8088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:15 smithi112 ceph-mon[37501]: pgmap v8088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14472.076704 secs 2024-07-26T16:52:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14472.076704 secs 2024-07-26T16:52:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14472.076704 secs 2024-07-26T16:52:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:17 smithi132 ceph-mon[37580]: pgmap v8089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:18.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:17 smithi112 ceph-mon[37501]: pgmap v8089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:17 smithi107 ceph-mon[32659]: pgmap v8089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:18.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:18.730+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:52:18.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:18.730+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:52:18.733 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:18.730+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:52:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:19 smithi132 ceph-mon[37580]: pgmap v8090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:19 smithi112 ceph-mon[37501]: pgmap v8090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:19 smithi107 ceph-mon[32659]: pgmap v8090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:21 smithi132 ceph-mon[37580]: pgmap v8091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14477.076836 secs 2024-07-26T16:52:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:21 smithi112 ceph-mon[37501]: pgmap v8091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14477.076836 secs 2024-07-26T16:52:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:21 smithi107 ceph-mon[32659]: pgmap v8091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14477.076836 secs 2024-07-26T16:52:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:23 smithi112 ceph-mon[37501]: pgmap v8092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:24.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:23 smithi107 ceph-mon[32659]: pgmap v8092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:23 smithi132 ceph-mon[37580]: pgmap v8092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:25 smithi112 ceph-mon[37501]: pgmap v8093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:25 smithi107 ceph-mon[32659]: pgmap v8093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:25 smithi132 ceph-mon[37580]: pgmap v8093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14482.076980 secs 2024-07-26T16:52:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14482.076980 secs 2024-07-26T16:52:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14482.076980 secs 2024-07-26T16:52:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:27 smithi112 ceph-mon[37501]: pgmap v8094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:28.250 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:27 smithi107 ceph-mon[32659]: pgmap v8094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:27 smithi132 ceph-mon[37580]: pgmap v8094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:29 smithi112 ceph-mon[37501]: pgmap v8095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:29 smithi107 ceph-mon[32659]: pgmap v8095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:29 smithi132 ceph-mon[37580]: pgmap v8095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:31 smithi112 ceph-mon[37501]: pgmap v8096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14487.077042 secs 2024-07-26T16:52:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:31 smithi107 ceph-mon[32659]: pgmap v8096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14487.077042 secs 2024-07-26T16:52:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:31 smithi132 ceph-mon[37580]: pgmap v8096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14487.077042 secs 2024-07-26T16:52:33.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:33.734+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:52:33.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:33.734+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:52:33.737 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:33.734+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:52:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:33 smithi107 ceph-mon[32659]: pgmap v8097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:33 smithi112 ceph-mon[37501]: pgmap v8097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:33 smithi132 ceph-mon[37580]: pgmap v8097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:35 smithi112 ceph-mon[37501]: pgmap v8098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:35 smithi107 ceph-mon[32659]: pgmap v8098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:35 smithi132 ceph-mon[37580]: pgmap v8098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14492.077164 secs 2024-07-26T16:52:37.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14492.077164 secs 2024-07-26T16:52:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14492.077164 secs 2024-07-26T16:52:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:37 smithi112 ceph-mon[37501]: pgmap v8099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:38.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:37 smithi107 ceph-mon[32659]: pgmap v8099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:37 smithi132 ceph-mon[37580]: pgmap v8099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:39 smithi112 ceph-mon[37501]: pgmap v8100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:39 smithi107 ceph-mon[32659]: pgmap v8100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:39 smithi132 ceph-mon[37580]: pgmap v8100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:41 smithi112 ceph-mon[37501]: pgmap v8101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14497.077287 secs 2024-07-26T16:52:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:41 smithi107 ceph-mon[32659]: pgmap v8101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14497.077287 secs 2024-07-26T16:52:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:41 smithi132 ceph-mon[37580]: pgmap v8101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14497.077287 secs 2024-07-26T16:52:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:43 smithi112 ceph-mon[37501]: pgmap v8102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:43 smithi107 ceph-mon[32659]: pgmap v8102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:43 smithi132 ceph-mon[37580]: pgmap v8102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:45 smithi112 ceph-mon[37501]: pgmap v8103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:45 smithi107 ceph-mon[32659]: pgmap v8103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:45 smithi132 ceph-mon[37580]: pgmap v8103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14502.077412 secs 2024-07-26T16:52:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14502.077412 secs 2024-07-26T16:52:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14502.077412 secs 2024-07-26T16:52:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:47 smithi112 ceph-mon[37501]: pgmap v8104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:48.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:47 smithi107 ceph-mon[32659]: pgmap v8104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:47 smithi132 ceph-mon[37580]: pgmap v8104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:48.750+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:52:48.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:48.750+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:52:48.753 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:52:48.750+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:52:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:49 smithi112 ceph-mon[37501]: pgmap v8105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:50.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:49 smithi107 ceph-mon[32659]: pgmap v8105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:49 smithi132 ceph-mon[37580]: pgmap v8105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:51 smithi112 ceph-mon[37501]: pgmap v8106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14507.077527 secs 2024-07-26T16:52:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:51 smithi107 ceph-mon[32659]: pgmap v8106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14507.077527 secs 2024-07-26T16:52:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:51 smithi132 ceph-mon[37580]: pgmap v8106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:52.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14507.077527 secs 2024-07-26T16:52:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:52:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:52:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:52:53.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:53.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:53 smithi112 ceph-mon[37501]: pgmap v8107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:54.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:53 smithi107 ceph-mon[32659]: pgmap v8107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:53 smithi132 ceph-mon[37580]: pgmap v8107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:52:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:55 smithi107 ceph-mon[32659]: pgmap v8108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:55 smithi132 ceph-mon[37580]: pgmap v8108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:55 smithi112 ceph-mon[37501]: pgmap v8108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:55.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14512.077675 secs 2024-07-26T16:52:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14512.077675 secs 2024-07-26T16:52:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14512.077675 secs 2024-07-26T16:52:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:57 smithi112 ceph-mon[37501]: pgmap v8109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:52:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:52:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:57 smithi112 ceph-mon[37501]: pgmap v8110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:52:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:57 smithi112 ceph-mon[37501]: pgmap v8111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T16:52:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:58.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:57 smithi107 ceph-mon[32659]: pgmap v8109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:58.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:52:58.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:52:58.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:57 smithi107 ceph-mon[32659]: pgmap v8110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:52:58.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:57 smithi107 ceph-mon[32659]: pgmap v8111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T16:52:58.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:57 smithi132 ceph-mon[37580]: pgmap v8109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:52:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:52:58.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:52:58.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:57 smithi132 ceph-mon[37580]: pgmap v8110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:52:58.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:57 smithi132 ceph-mon[37580]: pgmap v8111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T16:52:58.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:52:59.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:52:59 smithi107 ceph-mon[32659]: pgmap v8112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:52:59 smithi132 ceph-mon[37580]: pgmap v8112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:52:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:52:59 smithi112 ceph-mon[37501]: pgmap v8112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:01 smithi107 ceph-mon[32659]: pgmap v8113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:53:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:01 smithi132 ceph-mon[37580]: pgmap v8113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:53:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:01 smithi112 ceph-mon[37501]: pgmap v8113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:53:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14517.077743 secs 2024-07-26T16:53:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14517.077743 secs 2024-07-26T16:53:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14517.077743 secs 2024-07-26T16:53:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:03 smithi107 ceph-mon[32659]: pgmap v8114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:53:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:03 smithi132 ceph-mon[37580]: pgmap v8114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:53:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:03 smithi112 ceph-mon[37501]: pgmap v8114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:53:03.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:03.766+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:53:03.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:03.766+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:53:03.768 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:03.766+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:53:05.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:05 smithi107 ceph-mon[32659]: pgmap v8115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:53:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:05 smithi132 ceph-mon[37580]: pgmap v8115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:53:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:05 smithi112 ceph-mon[37501]: pgmap v8115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:53:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14522.077865 secs 2024-07-26T16:53:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14522.077865 secs 2024-07-26T16:53:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14522.077865 secs 2024-07-26T16:53:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:07 smithi112 ceph-mon[37501]: pgmap v8116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:53:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:07 smithi107 ceph-mon[32659]: pgmap v8116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:53:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:07 smithi132 ceph-mon[37580]: pgmap v8116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:53:09.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:09 smithi107 ceph-mon[32659]: pgmap v8117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:09 smithi132 ceph-mon[37580]: pgmap v8117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:09.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:09 smithi112 ceph-mon[37501]: pgmap v8117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:11 smithi107 ceph-mon[32659]: pgmap v8118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:11 smithi132 ceph-mon[37580]: pgmap v8118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:11 smithi112 ceph-mon[37501]: pgmap v8118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14527.078007 secs 2024-07-26T16:53:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14527.078007 secs 2024-07-26T16:53:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14527.078007 secs 2024-07-26T16:53:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:13 smithi132 ceph-mon[37580]: pgmap v8119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:13 smithi112 ceph-mon[37501]: pgmap v8119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:13 smithi107 ceph-mon[32659]: pgmap v8119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:15 smithi107 ceph-mon[32659]: pgmap v8120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:15 smithi132 ceph-mon[37580]: pgmap v8120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:15 smithi112 ceph-mon[37501]: pgmap v8120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14532.078152 secs 2024-07-26T16:53:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14532.078152 secs 2024-07-26T16:53:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14532.078152 secs 2024-07-26T16:53:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:17 smithi112 ceph-mon[37501]: pgmap v8121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:17 smithi107 ceph-mon[32659]: pgmap v8121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:17 smithi132 ceph-mon[37580]: pgmap v8121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:18.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:18.774+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:53:18.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:18.774+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:53:18.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:18.774+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:53:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:19 smithi112 ceph-mon[37501]: pgmap v8122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:19 smithi107 ceph-mon[32659]: pgmap v8122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:19 smithi132 ceph-mon[37580]: pgmap v8122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:21 smithi107 ceph-mon[32659]: pgmap v8123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:21 smithi132 ceph-mon[37580]: pgmap v8123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:21 smithi112 ceph-mon[37501]: pgmap v8123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14537.078271 secs 2024-07-26T16:53:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14537.078271 secs 2024-07-26T16:53:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14537.078271 secs 2024-07-26T16:53:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:23 smithi132 ceph-mon[37580]: pgmap v8124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:23 smithi112 ceph-mon[37501]: pgmap v8124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:23 smithi107 ceph-mon[32659]: pgmap v8124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:25 smithi107 ceph-mon[32659]: pgmap v8125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:25 smithi132 ceph-mon[37580]: pgmap v8125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:25.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:25 smithi112 ceph-mon[37501]: pgmap v8125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14542.078338 secs 2024-07-26T16:53:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14542.078338 secs 2024-07-26T16:53:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14542.078338 secs 2024-07-26T16:53:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:27 smithi112 ceph-mon[37501]: pgmap v8126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:27 smithi107 ceph-mon[32659]: pgmap v8126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:27 smithi132 ceph-mon[37580]: pgmap v8126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:29.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:29 smithi107 ceph-mon[32659]: pgmap v8127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:29 smithi132 ceph-mon[37580]: pgmap v8127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:29 smithi112 ceph-mon[37501]: pgmap v8127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:31 smithi107 ceph-mon[32659]: pgmap v8128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:31 smithi132 ceph-mon[37580]: pgmap v8128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:31 smithi112 ceph-mon[37501]: pgmap v8128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:32 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14547.078422 secs 2024-07-26T16:53:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:32 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14547.078422 secs 2024-07-26T16:53:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:32 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14547.078422 secs 2024-07-26T16:53:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:33 smithi132 ceph-mon[37580]: pgmap v8129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:33 smithi112 ceph-mon[37501]: pgmap v8129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:33.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:33 smithi107 ceph-mon[32659]: pgmap v8129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:33.785 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:33.783+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:53:33.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:33.783+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:53:33.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:33.783+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:53:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:35 smithi107 ceph-mon[32659]: pgmap v8130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:35 smithi132 ceph-mon[37580]: pgmap v8130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:35 smithi112 ceph-mon[37501]: pgmap v8130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14552.078504 secs 2024-07-26T16:53:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14552.078504 secs 2024-07-26T16:53:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14552.078504 secs 2024-07-26T16:53:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:37 smithi112 ceph-mon[37501]: pgmap v8131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:37 smithi107 ceph-mon[32659]: pgmap v8131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:37 smithi132 ceph-mon[37580]: pgmap v8131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:39.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:39 smithi107 ceph-mon[32659]: pgmap v8132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:39 smithi132 ceph-mon[37580]: pgmap v8132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:39 smithi112 ceph-mon[37501]: pgmap v8132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:41 smithi107 ceph-mon[32659]: pgmap v8133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:41 smithi132 ceph-mon[37580]: pgmap v8133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:41 smithi112 ceph-mon[37501]: pgmap v8133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14557.078619 secs 2024-07-26T16:53:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14557.078619 secs 2024-07-26T16:53:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14557.078619 secs 2024-07-26T16:53:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:43 smithi132 ceph-mon[37580]: pgmap v8134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:43 smithi112 ceph-mon[37501]: pgmap v8134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:43 smithi107 ceph-mon[32659]: pgmap v8134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:45 smithi107 ceph-mon[32659]: pgmap v8135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:45 smithi132 ceph-mon[37580]: pgmap v8135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:45 smithi112 ceph-mon[37501]: pgmap v8135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14562.078746 secs 2024-07-26T16:53:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14562.078746 secs 2024-07-26T16:53:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14562.078746 secs 2024-07-26T16:53:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:47 smithi112 ceph-mon[37501]: pgmap v8136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:47 smithi107 ceph-mon[32659]: pgmap v8136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:47 smithi132 ceph-mon[37580]: pgmap v8136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:53:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:48.796+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:53:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:48.796+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:53:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:53:48.796+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:53:49.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:49 smithi107 ceph-mon[32659]: pgmap v8137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:49 smithi132 ceph-mon[37580]: pgmap v8137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:49 smithi112 ceph-mon[37501]: pgmap v8137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:53:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:51 smithi107 ceph-mon[32659]: pgmap v8138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:51 smithi132 ceph-mon[37580]: pgmap v8138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:51 smithi112 ceph-mon[37501]: pgmap v8138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:53:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14567.078867 secs 2024-07-26T16:53:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14567.078867 secs 2024-07-26T16:53:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14567.078867 secs 2024-07-26T16:53:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:53 smithi132 ceph-mon[37580]: pgmap v8139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:53:53.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:53 smithi112 ceph-mon[37501]: pgmap v8139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:53:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:53 smithi107 ceph-mon[32659]: pgmap v8139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:53:55.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:55 smithi107 ceph-mon[32659]: pgmap v8140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:53:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:55 smithi132 ceph-mon[37580]: pgmap v8140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:53:55.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:55 smithi112 ceph-mon[37501]: pgmap v8140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:53:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14572.079004 secs 2024-07-26T16:53:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14572.079004 secs 2024-07-26T16:53:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14572.079004 secs 2024-07-26T16:53:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:57 smithi112 ceph-mon[37501]: pgmap v8141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:53:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:53:57.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:57 smithi107 ceph-mon[32659]: pgmap v8141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:53:57.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:53:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:57 smithi132 ceph-mon[37580]: pgmap v8141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:53:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:53:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:53:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:53:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:53:58.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:53:58.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:53:58.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:53:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:53:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:53:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:53:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:59 smithi112 ceph-mon[37501]: pgmap v8142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:53:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:53:59 smithi112 ceph-mon[37501]: pgmap v8143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:53:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:59 smithi107 ceph-mon[32659]: pgmap v8142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:53:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:53:59 smithi107 ceph-mon[32659]: pgmap v8143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:53:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:59 smithi132 ceph-mon[37580]: pgmap v8142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:53:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:53:59 smithi132 ceph-mon[37580]: pgmap v8143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:54:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:01 smithi112 ceph-mon[37501]: pgmap v8144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:54:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14577.079135 secs 2024-07-26T16:54:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:01 smithi107 ceph-mon[32659]: pgmap v8144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:54:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14577.079135 secs 2024-07-26T16:54:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:01 smithi132 ceph-mon[37580]: pgmap v8144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:54:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14577.079135 secs 2024-07-26T16:54:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:03 smithi112 ceph-mon[37501]: pgmap v8145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:54:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:03 smithi107 ceph-mon[32659]: pgmap v8145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:54:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:03 smithi132 ceph-mon[37580]: pgmap v8145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:54:03.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:03.812+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:54:03.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:03.812+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:54:03.814 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:03.812+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:54:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:05 smithi112 ceph-mon[37501]: pgmap v8146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:54:05.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:05 smithi107 ceph-mon[32659]: pgmap v8146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:54:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:05 smithi132 ceph-mon[37580]: pgmap v8146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:54:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14582.079267 secs 2024-07-26T16:54:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14582.079267 secs 2024-07-26T16:54:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14582.079267 secs 2024-07-26T16:54:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:07 smithi112 ceph-mon[37501]: pgmap v8147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:54:07.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:07 smithi107 ceph-mon[32659]: pgmap v8147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:54:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:07 smithi132 ceph-mon[37580]: pgmap v8147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:54:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:09 smithi112 ceph-mon[37501]: pgmap v8148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:54:09.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:09 smithi107 ceph-mon[32659]: pgmap v8148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:54:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:09 smithi132 ceph-mon[37580]: pgmap v8148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:54:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:11 smithi112 ceph-mon[37501]: pgmap v8149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14587.079358 secs 2024-07-26T16:54:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:11 smithi107 ceph-mon[32659]: pgmap v8149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14587.079358 secs 2024-07-26T16:54:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:11 smithi132 ceph-mon[37580]: pgmap v8149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14587.079358 secs 2024-07-26T16:54:13.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:13 smithi112 ceph-mon[37501]: pgmap v8150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:13.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:13 smithi107 ceph-mon[32659]: pgmap v8150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:13 smithi132 ceph-mon[37580]: pgmap v8150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:15.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:15.154+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:54:15.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:15.154+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:54:15.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:15.155+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:54:15.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:15.155+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:54:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:15 smithi112 ceph-mon[37501]: pgmap v8151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:15 smithi107 ceph-mon[32659]: pgmap v8151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:15 smithi132 ceph-mon[37580]: pgmap v8151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14592.079480 secs 2024-07-26T16:54:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14592.079480 secs 2024-07-26T16:54:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14592.079480 secs 2024-07-26T16:54:17.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:17 smithi107 ceph-mon[32659]: pgmap v8152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:17 smithi132 ceph-mon[37580]: pgmap v8152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:17 smithi112 ceph-mon[37501]: pgmap v8152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:18.826+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:54:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:18.826+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:54:18.829 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:18.826+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:54:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:19 smithi107 ceph-mon[32659]: pgmap v8153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:19 smithi132 ceph-mon[37580]: pgmap v8153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:19 smithi112 ceph-mon[37501]: pgmap v8153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:21 smithi107 ceph-mon[32659]: pgmap v8154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14597.079592 secs 2024-07-26T16:54:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:21 smithi132 ceph-mon[37580]: pgmap v8154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14597.079592 secs 2024-07-26T16:54:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:21 smithi112 ceph-mon[37501]: pgmap v8154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14597.079592 secs 2024-07-26T16:54:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:23 smithi107 ceph-mon[32659]: pgmap v8155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:23 smithi132 ceph-mon[37580]: pgmap v8155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:23 smithi112 ceph-mon[37501]: pgmap v8155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:25 smithi107 ceph-mon[32659]: pgmap v8156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:25 smithi132 ceph-mon[37580]: pgmap v8156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:25 smithi112 ceph-mon[37501]: pgmap v8156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14602.079731 secs 2024-07-26T16:54:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14602.079731 secs 2024-07-26T16:54:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14602.079731 secs 2024-07-26T16:54:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:27 smithi107 ceph-mon[32659]: pgmap v8157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:27 smithi132 ceph-mon[37580]: pgmap v8157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:27.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:27 smithi112 ceph-mon[37501]: pgmap v8157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:29 smithi107 ceph-mon[32659]: pgmap v8158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:29 smithi132 ceph-mon[37580]: pgmap v8158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:29 smithi112 ceph-mon[37501]: pgmap v8158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:31 smithi107 ceph-mon[32659]: pgmap v8159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14607.079867 secs 2024-07-26T16:54:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:31 smithi132 ceph-mon[37580]: pgmap v8159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14607.079867 secs 2024-07-26T16:54:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:31 smithi112 ceph-mon[37501]: pgmap v8159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14607.079867 secs 2024-07-26T16:54:33.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:33 smithi107 ceph-mon[32659]: pgmap v8160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:33 smithi132 ceph-mon[37580]: pgmap v8160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:33.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:33.842+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:54:33.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:33.842+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:54:33.845 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:33.842+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:54:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:33 smithi112 ceph-mon[37501]: pgmap v8160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:35.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:35 smithi107 ceph-mon[32659]: pgmap v8161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:35 smithi132 ceph-mon[37580]: pgmap v8161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:35 smithi112 ceph-mon[37501]: pgmap v8161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14612.080002 secs 2024-07-26T16:54:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14612.080002 secs 2024-07-26T16:54:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14612.080002 secs 2024-07-26T16:54:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:37 smithi132 ceph-mon[37580]: pgmap v8162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:37 smithi112 ceph-mon[37501]: pgmap v8162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:37 smithi107 ceph-mon[32659]: pgmap v8162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:39.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:39 smithi132 ceph-mon[37580]: pgmap v8163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:39 smithi112 ceph-mon[37501]: pgmap v8163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:39 smithi107 ceph-mon[32659]: pgmap v8163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:41 smithi132 ceph-mon[37580]: pgmap v8164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:41.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14617.080149 secs 2024-07-26T16:54:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:41 smithi112 ceph-mon[37501]: pgmap v8164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14617.080149 secs 2024-07-26T16:54:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:41 smithi107 ceph-mon[32659]: pgmap v8164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14617.080149 secs 2024-07-26T16:54:43.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:43 smithi107 ceph-mon[32659]: pgmap v8165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:43 smithi112 ceph-mon[37501]: pgmap v8165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:43 smithi132 ceph-mon[37580]: pgmap v8165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:45 smithi112 ceph-mon[37501]: pgmap v8166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:45 smithi107 ceph-mon[32659]: pgmap v8166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:45 smithi132 ceph-mon[37580]: pgmap v8166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14622.080231 secs 2024-07-26T16:54:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14622.080231 secs 2024-07-26T16:54:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14622.080231 secs 2024-07-26T16:54:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:47 smithi112 ceph-mon[37501]: pgmap v8167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:48.027 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:47 smithi107 ceph-mon[32659]: pgmap v8167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:47 smithi132 ceph-mon[37580]: pgmap v8167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:48.849+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:54:48.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:48.849+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:54:48.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:54:48.849+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:54:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:49 smithi112 ceph-mon[37501]: pgmap v8168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:49 smithi107 ceph-mon[32659]: pgmap v8168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:49 smithi132 ceph-mon[37580]: pgmap v8168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:54:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:51 smithi112 ceph-mon[37501]: pgmap v8169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14627.080306 secs 2024-07-26T16:54:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:51 smithi107 ceph-mon[32659]: pgmap v8169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14627.080306 secs 2024-07-26T16:54:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:51 smithi132 ceph-mon[37580]: pgmap v8169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14627.080306 secs 2024-07-26T16:54:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:53 smithi112 ceph-mon[37501]: pgmap v8170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:53 smithi107 ceph-mon[32659]: pgmap v8170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:53 smithi132 ceph-mon[37580]: pgmap v8170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:54:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:55 smithi112 ceph-mon[37501]: pgmap v8171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:54:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:55 smithi107 ceph-mon[32659]: pgmap v8171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:54:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:55 smithi132 ceph-mon[37580]: pgmap v8171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T16:54:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14632.080396 secs 2024-07-26T16:54:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14632.080396 secs 2024-07-26T16:54:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14632.080396 secs 2024-07-26T16:54:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:57 smithi112 ceph-mon[37501]: pgmap v8172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:58.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:57 smithi107 ceph-mon[32659]: pgmap v8172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:58.083 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:57 smithi132 ceph-mon[37580]: pgmap v8172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:54:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:54:59.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:54:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:54:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:59 smithi112 ceph-mon[37501]: pgmap v8173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:54:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:54:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:54:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:54:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:55:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:59 smithi107 ceph-mon[32659]: pgmap v8173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:55:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:55:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:55:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:54:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:55:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:59 smithi132 ceph-mon[37580]: pgmap v8173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:55:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:55:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:55:00.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:54:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:55:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:00 smithi112 ceph-mon[37501]: pgmap v8174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:55:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:00 smithi107 ceph-mon[32659]: pgmap v8174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:55:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:00 smithi132 ceph-mon[37580]: pgmap v8174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:55:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14637.080533 secs 2024-07-26T16:55:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14637.080533 secs 2024-07-26T16:55:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14637.080533 secs 2024-07-26T16:55:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:02 smithi112 ceph-mon[37501]: pgmap v8175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:55:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:02 smithi107 ceph-mon[32659]: pgmap v8175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:55:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:02 smithi132 ceph-mon[37580]: pgmap v8175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:55:03.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:03.857+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:55:03.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:03.857+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:55:03.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:03.857+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:55:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:04 smithi112 ceph-mon[37501]: pgmap v8176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:55:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:04 smithi107 ceph-mon[32659]: pgmap v8176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:55:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:04 smithi132 ceph-mon[37580]: pgmap v8176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:55:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14642.080655 secs 2024-07-26T16:55:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:06 smithi112 ceph-mon[37501]: pgmap v8177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:55:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14642.080655 secs 2024-07-26T16:55:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:06 smithi107 ceph-mon[32659]: pgmap v8177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:55:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14642.080655 secs 2024-07-26T16:55:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:06 smithi132 ceph-mon[37580]: pgmap v8177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:55:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:08 smithi112 ceph-mon[37501]: pgmap v8178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:55:09.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:08 smithi107 ceph-mon[32659]: pgmap v8178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:55:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:08 smithi132 ceph-mon[37580]: pgmap v8178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:55:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:10 smithi112 ceph-mon[37501]: pgmap v8179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:55:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:10 smithi107 ceph-mon[32659]: pgmap v8179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:55:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:10 smithi132 ceph-mon[37580]: pgmap v8179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:55:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14647.080798 secs 2024-07-26T16:55:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14647.080798 secs 2024-07-26T16:55:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14647.080798 secs 2024-07-26T16:55:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:12 smithi112 ceph-mon[37501]: pgmap v8180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:55:13.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:12 smithi107 ceph-mon[32659]: pgmap v8180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:55:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:12 smithi132 ceph-mon[37580]: pgmap v8180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:55:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:14 smithi112 ceph-mon[37501]: pgmap v8181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:14 smithi107 ceph-mon[32659]: pgmap v8181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:14 smithi132 ceph-mon[37580]: pgmap v8181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14652.080921 secs 2024-07-26T16:55:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:16 smithi107 ceph-mon[32659]: pgmap v8182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14652.080921 secs 2024-07-26T16:55:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:16 smithi132 ceph-mon[37580]: pgmap v8182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14652.080921 secs 2024-07-26T16:55:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:16 smithi112 ceph-mon[37501]: pgmap v8182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:18.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:18.859+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:55:18.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:18.859+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:55:18.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:18.859+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:55:18.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:18 smithi107 ceph-mon[32659]: pgmap v8183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:18 smithi132 ceph-mon[37580]: pgmap v8183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:18 smithi112 ceph-mon[37501]: pgmap v8183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:20 smithi107 ceph-mon[32659]: pgmap v8184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:20 smithi132 ceph-mon[37580]: pgmap v8184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:20 smithi112 ceph-mon[37501]: pgmap v8184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14657.081057 secs 2024-07-26T16:55:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14657.081057 secs 2024-07-26T16:55:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14657.081057 secs 2024-07-26T16:55:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:22 smithi107 ceph-mon[32659]: pgmap v8185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:55:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:22 smithi132 ceph-mon[37580]: pgmap v8185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:55:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:22 smithi112 ceph-mon[37501]: pgmap v8185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:55:25.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:24 smithi107 ceph-mon[32659]: pgmap v8186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:24 smithi132 ceph-mon[37580]: pgmap v8186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:24 smithi112 ceph-mon[37501]: pgmap v8186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:27.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14662.081179 secs 2024-07-26T16:55:27.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:26 smithi112 ceph-mon[37501]: pgmap v8187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14662.081179 secs 2024-07-26T16:55:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:26 smithi107 ceph-mon[32659]: pgmap v8187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14662.081179 secs 2024-07-26T16:55:27.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:26 smithi132 ceph-mon[37580]: pgmap v8187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:28 smithi107 ceph-mon[32659]: pgmap v8188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:28 smithi132 ceph-mon[37580]: pgmap v8188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:28 smithi112 ceph-mon[37501]: pgmap v8188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:30 smithi107 ceph-mon[32659]: pgmap v8189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:30 smithi132 ceph-mon[37580]: pgmap v8189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:30 smithi112 ceph-mon[37501]: pgmap v8189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14667.081299 secs 2024-07-26T16:55:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14667.081299 secs 2024-07-26T16:55:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14667.081299 secs 2024-07-26T16:55:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:32 smithi107 ceph-mon[32659]: pgmap v8190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:55:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:32 smithi132 ceph-mon[37580]: pgmap v8190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:55:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:32 smithi112 ceph-mon[37501]: pgmap v8190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:55:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:33.865+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:55:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:33.865+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:55:33.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:33.865+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:55:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:34 smithi107 ceph-mon[32659]: pgmap v8191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:34 smithi132 ceph-mon[37580]: pgmap v8191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:34 smithi112 ceph-mon[37501]: pgmap v8191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14672.081382 secs 2024-07-26T16:55:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:36 smithi107 ceph-mon[32659]: pgmap v8192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14672.081382 secs 2024-07-26T16:55:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:36 smithi132 ceph-mon[37580]: pgmap v8192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14672.081382 secs 2024-07-26T16:55:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:36 smithi112 ceph-mon[37501]: pgmap v8192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:38 smithi107 ceph-mon[32659]: pgmap v8193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:38 smithi132 ceph-mon[37580]: pgmap v8193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:38 smithi112 ceph-mon[37501]: pgmap v8193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:40 smithi107 ceph-mon[32659]: pgmap v8194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:40 smithi132 ceph-mon[37580]: pgmap v8194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:40 smithi112 ceph-mon[37501]: pgmap v8194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14677.081454 secs 2024-07-26T16:55:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14677.081454 secs 2024-07-26T16:55:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14677.081454 secs 2024-07-26T16:55:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:42 smithi132 ceph-mon[37580]: pgmap v8195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:55:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:42 smithi112 ceph-mon[37501]: pgmap v8195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:55:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:42 smithi107 ceph-mon[32659]: pgmap v8195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:55:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:44 smithi132 ceph-mon[37580]: pgmap v8196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:45.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:44 smithi107 ceph-mon[32659]: pgmap v8196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:44 smithi112 ceph-mon[37501]: pgmap v8196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14682.081555 secs 2024-07-26T16:55:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:46 smithi112 ceph-mon[37501]: pgmap v8197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14682.081555 secs 2024-07-26T16:55:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:46 smithi107 ceph-mon[32659]: pgmap v8197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14682.081555 secs 2024-07-26T16:55:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:46 smithi132 ceph-mon[37580]: pgmap v8197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:48.878 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:48.876+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:55:48.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:48.877+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:55:48.879 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:55:48.877+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:55:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:48 smithi112 ceph-mon[37501]: pgmap v8198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:49.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:48 smithi107 ceph-mon[32659]: pgmap v8198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:48 smithi132 ceph-mon[37580]: pgmap v8198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:50 smithi112 ceph-mon[37501]: pgmap v8199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:50 smithi107 ceph-mon[32659]: pgmap v8199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:50 smithi132 ceph-mon[37580]: pgmap v8199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:55:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14687.081654 secs 2024-07-26T16:55:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14687.081654 secs 2024-07-26T16:55:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14687.081654 secs 2024-07-26T16:55:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:52 smithi112 ceph-mon[37501]: pgmap v8200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:55:53.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:52 smithi107 ceph-mon[32659]: pgmap v8200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:55:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:52 smithi132 ceph-mon[37580]: pgmap v8200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:55:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:54 smithi112 ceph-mon[37501]: pgmap v8201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:54 smithi107 ceph-mon[32659]: pgmap v8201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:54 smithi132 ceph-mon[37580]: pgmap v8201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14692.081791 secs 2024-07-26T16:55:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:56 smithi112 ceph-mon[37501]: pgmap v8202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14692.081791 secs 2024-07-26T16:55:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:56 smithi107 ceph-mon[32659]: pgmap v8202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14692.081791 secs 2024-07-26T16:55:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:56 smithi132 ceph-mon[37580]: pgmap v8202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:55:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:58 smithi112 ceph-mon[37501]: pgmap v8203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:59.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:58 smithi107 ceph-mon[32659]: pgmap v8203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:55:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:58 smithi132 ceph-mon[37580]: pgmap v8203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:55:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:56:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:55:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:56:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:55:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:56:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:00 smithi112 ceph-mon[37501]: pgmap v8204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:56:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:56:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:56:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:56:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:00 smithi107 ceph-mon[32659]: pgmap v8204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:56:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:56:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:56:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:56:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:00 smithi132 ceph-mon[37580]: pgmap v8204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:56:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:56:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:56:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:56:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14697.081927 secs 2024-07-26T16:56:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:01 smithi112 ceph-mon[37501]: pgmap v8205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:56:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:01 smithi112 ceph-mon[37501]: pgmap v8206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:56:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14697.081927 secs 2024-07-26T16:56:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:01 smithi107 ceph-mon[32659]: pgmap v8205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:56:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:01 smithi107 ceph-mon[32659]: pgmap v8206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:56:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14697.081927 secs 2024-07-26T16:56:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:01 smithi132 ceph-mon[37580]: pgmap v8205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:56:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:01 smithi132 ceph-mon[37580]: pgmap v8206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:56:03.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:03.892+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:56:03.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:03.892+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:56:03.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:03.892+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:56:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:03 smithi112 ceph-mon[37501]: pgmap v8207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:56:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:03 smithi107 ceph-mon[32659]: pgmap v8207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:56:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:03 smithi132 ceph-mon[37580]: pgmap v8207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:56:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:05 smithi112 ceph-mon[37501]: pgmap v8208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:56:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:05 smithi107 ceph-mon[32659]: pgmap v8208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:56:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:05 smithi132 ceph-mon[37580]: pgmap v8208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:56:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14702.082049 secs 2024-07-26T16:56:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14702.082049 secs 2024-07-26T16:56:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14702.082049 secs 2024-07-26T16:56:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:07 smithi112 ceph-mon[37501]: pgmap v8209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:07 smithi107 ceph-mon[32659]: pgmap v8209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:07 smithi132 ceph-mon[37580]: pgmap v8209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:09 smithi112 ceph-mon[37501]: pgmap v8210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:56:10.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:09 smithi107 ceph-mon[32659]: pgmap v8210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:56:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:09 smithi132 ceph-mon[37580]: pgmap v8210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:56:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14707.082143 secs 2024-07-26T16:56:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:11 smithi112 ceph-mon[37501]: pgmap v8211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:56:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14707.082143 secs 2024-07-26T16:56:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:11 smithi107 ceph-mon[32659]: pgmap v8211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:56:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14707.082143 secs 2024-07-26T16:56:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:11 smithi132 ceph-mon[37580]: pgmap v8211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:56:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:13 smithi112 ceph-mon[37501]: pgmap v8212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:56:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:13 smithi107 ceph-mon[32659]: pgmap v8212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:56:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:13 smithi132 ceph-mon[37580]: pgmap v8212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:56:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:15 smithi112 ceph-mon[37501]: pgmap v8213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:56:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:15 smithi107 ceph-mon[32659]: pgmap v8213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:56:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:15 smithi132 ceph-mon[37580]: pgmap v8213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:56:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14712.082259 secs 2024-07-26T16:56:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14712.082259 secs 2024-07-26T16:56:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14712.082259 secs 2024-07-26T16:56:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:17 smithi112 ceph-mon[37501]: pgmap v8214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T16:56:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:17 smithi107 ceph-mon[32659]: pgmap v8214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T16:56:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:17 smithi132 ceph-mon[37580]: pgmap v8214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T16:56:18.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:18.893+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:56:18.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:18.893+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:56:18.896 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:18.893+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:56:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:19 smithi112 ceph-mon[37501]: pgmap v8215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:56:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:19 smithi107 ceph-mon[32659]: pgmap v8215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:56:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:19 smithi132 ceph-mon[37580]: pgmap v8215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:56:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14717.082347 secs 2024-07-26T16:56:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:21 smithi107 ceph-mon[32659]: pgmap v8216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14717.082347 secs 2024-07-26T16:56:22.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:21 smithi132 ceph-mon[37580]: pgmap v8216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14717.082347 secs 2024-07-26T16:56:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:21 smithi112 ceph-mon[37501]: pgmap v8216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:24.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:23 smithi107 ceph-mon[32659]: pgmap v8217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:23 smithi132 ceph-mon[37580]: pgmap v8217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:23 smithi112 ceph-mon[37501]: pgmap v8217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:25 smithi107 ceph-mon[32659]: pgmap v8218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:25 smithi132 ceph-mon[37580]: pgmap v8218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:25 smithi112 ceph-mon[37501]: pgmap v8218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14722.082463 secs 2024-07-26T16:56:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14722.082463 secs 2024-07-26T16:56:27.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14722.082463 secs 2024-07-26T16:56:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:27 smithi107 ceph-mon[32659]: pgmap v8219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:27 smithi132 ceph-mon[37580]: pgmap v8219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:27 smithi112 ceph-mon[37501]: pgmap v8219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:29 smithi107 ceph-mon[32659]: pgmap v8220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:29 smithi132 ceph-mon[37580]: pgmap v8220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:29 smithi112 ceph-mon[37501]: pgmap v8220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14727.082530 secs 2024-07-26T16:56:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:31 smithi107 ceph-mon[32659]: pgmap v8221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14727.082530 secs 2024-07-26T16:56:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:31 smithi132 ceph-mon[37580]: pgmap v8221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14727.082530 secs 2024-07-26T16:56:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:31 smithi112 ceph-mon[37501]: pgmap v8221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:33.911 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:33.909+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:56:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:33.909+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:56:33.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:33.909+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:56:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:33 smithi107 ceph-mon[32659]: pgmap v8222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:33 smithi132 ceph-mon[37580]: pgmap v8222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:33 smithi112 ceph-mon[37501]: pgmap v8222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:35 smithi107 ceph-mon[32659]: pgmap v8223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:35 smithi132 ceph-mon[37580]: pgmap v8223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:35 smithi112 ceph-mon[37501]: pgmap v8223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:37.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14732.082656 secs 2024-07-26T16:56:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14732.082656 secs 2024-07-26T16:56:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14732.082656 secs 2024-07-26T16:56:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:38 smithi107 ceph-mon[32659]: pgmap v8224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:38 smithi132 ceph-mon[37580]: pgmap v8224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:38 smithi112 ceph-mon[37501]: pgmap v8224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:40.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:40 smithi107 ceph-mon[32659]: pgmap v8225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:40 smithi132 ceph-mon[37580]: pgmap v8225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:40 smithi112 ceph-mon[37501]: pgmap v8225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:41 smithi107 ceph-mon[32659]: pgmap v8226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:41 smithi132 ceph-mon[37580]: pgmap v8226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:41 smithi112 ceph-mon[37501]: pgmap v8226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14737.082779 secs 2024-07-26T16:56:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14737.082779 secs 2024-07-26T16:56:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14737.082779 secs 2024-07-26T16:56:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:43 smithi132 ceph-mon[37580]: pgmap v8227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:43 smithi112 ceph-mon[37501]: pgmap v8227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:43 smithi107 ceph-mon[32659]: pgmap v8227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:45 smithi112 ceph-mon[37501]: pgmap v8228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:46.014 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:45 smithi132 ceph-mon[37580]: pgmap v8228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:45 smithi107 ceph-mon[32659]: pgmap v8228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14742.082891 secs 2024-07-26T16:56:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14742.082891 secs 2024-07-26T16:56:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14742.082891 secs 2024-07-26T16:56:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:47 smithi112 ceph-mon[37501]: pgmap v8229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:48.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:47 smithi107 ceph-mon[32659]: pgmap v8229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:47 smithi132 ceph-mon[37580]: pgmap v8229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:48.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:48.919+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:56:48.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:48.919+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:56:48.921 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:56:48.919+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:56:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:49 smithi112 ceph-mon[37501]: pgmap v8230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:49 smithi107 ceph-mon[32659]: pgmap v8230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:49 smithi132 ceph-mon[37580]: pgmap v8230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:51.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14747.083008 secs 2024-07-26T16:56:51.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:51 smithi107 ceph-mon[32659]: pgmap v8231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14747.083008 secs 2024-07-26T16:56:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:51 smithi112 ceph-mon[37501]: pgmap v8231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14747.083008 secs 2024-07-26T16:56:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:51 smithi132 ceph-mon[37580]: pgmap v8231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:56:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:53 smithi112 ceph-mon[37501]: pgmap v8232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:53 smithi107 ceph-mon[32659]: pgmap v8232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:53 smithi132 ceph-mon[37580]: pgmap v8232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:56:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:55 smithi112 ceph-mon[37501]: pgmap v8233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:55 smithi107 ceph-mon[32659]: pgmap v8233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:55 smithi132 ceph-mon[37580]: pgmap v8233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14752.083130 secs 2024-07-26T16:56:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14752.083130 secs 2024-07-26T16:56:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14752.083130 secs 2024-07-26T16:56:58.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:57 smithi107 ceph-mon[32659]: pgmap v8234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:57 smithi132 ceph-mon[37580]: pgmap v8234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:56:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:57 smithi112 ceph-mon[37501]: pgmap v8234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:56:59 smithi107 ceph-mon[32659]: pgmap v8235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:56:59 smithi132 ceph-mon[37580]: pgmap v8235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:56:59 smithi112 ceph-mon[37501]: pgmap v8235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14757.083243 secs 2024-07-26T16:57:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:01 smithi107 ceph-mon[32659]: pgmap v8236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:57:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14757.083243 secs 2024-07-26T16:57:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:01 smithi132 ceph-mon[37580]: pgmap v8236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:57:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14757.083243 secs 2024-07-26T16:57:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:01 smithi112 ceph-mon[37501]: pgmap v8236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:57:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:57:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:57:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:02 smithi107 ceph-mon[32659]: pgmap v8237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:57:03.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:02 smithi107 ceph-mon[32659]: pgmap v8238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:03.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:57:03.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:57:03.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:57:03.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:02 smithi132 ceph-mon[37580]: pgmap v8237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:57:03.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:02 smithi132 ceph-mon[37580]: pgmap v8238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:03.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:57:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:57:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:57:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:02 smithi112 ceph-mon[37501]: pgmap v8237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:57:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:02 smithi112 ceph-mon[37501]: pgmap v8238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:57:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:03.922+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:57:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:03.922+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:57:03.924 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:03.922+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:57:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:04 smithi112 ceph-mon[37501]: pgmap v8239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:57:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:04 smithi107 ceph-mon[32659]: pgmap v8239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:57:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:04 smithi132 ceph-mon[37580]: pgmap v8239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:57:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14762.083358 secs 2024-07-26T16:57:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:06 smithi112 ceph-mon[37501]: pgmap v8240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:57:07.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14762.083358 secs 2024-07-26T16:57:07.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:06 smithi107 ceph-mon[32659]: pgmap v8240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:57:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14762.083358 secs 2024-07-26T16:57:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:06 smithi132 ceph-mon[37580]: pgmap v8240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:57:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:08 smithi112 ceph-mon[37501]: pgmap v8241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:57:09.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:08 smithi107 ceph-mon[32659]: pgmap v8241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:57:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:08 smithi132 ceph-mon[37580]: pgmap v8241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:57:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:10 smithi112 ceph-mon[37501]: pgmap v8242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:57:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:10 smithi107 ceph-mon[32659]: pgmap v8242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:57:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:10 smithi132 ceph-mon[37580]: pgmap v8242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:57:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14767.083434 secs 2024-07-26T16:57:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14767.083434 secs 2024-07-26T16:57:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14767.083434 secs 2024-07-26T16:57:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:12 smithi112 ceph-mon[37501]: pgmap v8243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:57:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:12 smithi107 ceph-mon[32659]: pgmap v8243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:57:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:12 smithi132 ceph-mon[37580]: pgmap v8243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:57:15.187 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:15.186+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:57:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:15.186+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:57:15.188 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:14 smithi107 ceph-mon[32659]: pgmap v8244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:15.186+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:57:15.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:15.186+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:57:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:14 smithi112 ceph-mon[37501]: pgmap v8244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:14 smithi132 ceph-mon[37580]: pgmap v8244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14772.083562 secs 2024-07-26T16:57:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:16 smithi112 ceph-mon[37501]: pgmap v8245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14772.083562 secs 2024-07-26T16:57:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:16 smithi107 ceph-mon[32659]: pgmap v8245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14772.083562 secs 2024-07-26T16:57:17.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:16 smithi132 ceph-mon[37580]: pgmap v8245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:18.929+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:57:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:18.929+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:57:18.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:18.929+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:57:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:18 smithi112 ceph-mon[37501]: pgmap v8246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:18 smithi107 ceph-mon[32659]: pgmap v8246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:18 smithi132 ceph-mon[37580]: pgmap v8246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:20 smithi112 ceph-mon[37501]: pgmap v8247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:20 smithi107 ceph-mon[32659]: pgmap v8247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:20 smithi132 ceph-mon[37580]: pgmap v8247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14777.083668 secs 2024-07-26T16:57:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14777.083668 secs 2024-07-26T16:57:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14777.083668 secs 2024-07-26T16:57:23.163 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:22 smithi107 ceph-mon[32659]: pgmap v8248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:22 smithi112 ceph-mon[37501]: pgmap v8248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:22 smithi132 ceph-mon[37580]: pgmap v8248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:24 smithi112 ceph-mon[37501]: pgmap v8249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:24 smithi107 ceph-mon[32659]: pgmap v8249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:24 smithi132 ceph-mon[37580]: pgmap v8249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14782.083725 secs 2024-07-26T16:57:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:26 smithi112 ceph-mon[37501]: pgmap v8250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14782.083725 secs 2024-07-26T16:57:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:26 smithi107 ceph-mon[32659]: pgmap v8250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14782.083725 secs 2024-07-26T16:57:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:26 smithi132 ceph-mon[37580]: pgmap v8250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:28 smithi112 ceph-mon[37501]: pgmap v8251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:29.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:28 smithi107 ceph-mon[32659]: pgmap v8251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:28 smithi132 ceph-mon[37580]: pgmap v8251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:30 smithi112 ceph-mon[37501]: pgmap v8252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:30 smithi107 ceph-mon[32659]: pgmap v8252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:30 smithi132 ceph-mon[37580]: pgmap v8252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14787.083846 secs 2024-07-26T16:57:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14787.083846 secs 2024-07-26T16:57:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14787.083846 secs 2024-07-26T16:57:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:32 smithi112 ceph-mon[37501]: pgmap v8253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:32 smithi107 ceph-mon[32659]: pgmap v8253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:32 smithi132 ceph-mon[37580]: pgmap v8253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:33.937 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:33.934+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:57:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:33.934+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:57:33.938 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:33.934+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:57:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:34 smithi112 ceph-mon[37501]: pgmap v8254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:34 smithi107 ceph-mon[32659]: pgmap v8254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:34 smithi132 ceph-mon[37580]: pgmap v8254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14792.083993 secs 2024-07-26T16:57:37.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:36 smithi107 ceph-mon[32659]: pgmap v8255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14792.083993 secs 2024-07-26T16:57:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:36 smithi132 ceph-mon[37580]: pgmap v8255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14792.083993 secs 2024-07-26T16:57:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:36 smithi112 ceph-mon[37501]: pgmap v8255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:38 smithi107 ceph-mon[32659]: pgmap v8256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:38 smithi132 ceph-mon[37580]: pgmap v8256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:38 smithi112 ceph-mon[37501]: pgmap v8256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:40 smithi107 ceph-mon[32659]: pgmap v8257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:40 smithi132 ceph-mon[37580]: pgmap v8257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:40 smithi112 ceph-mon[37501]: pgmap v8257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14797.084108 secs 2024-07-26T16:57:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14797.084108 secs 2024-07-26T16:57:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14797.084108 secs 2024-07-26T16:57:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:42 smithi107 ceph-mon[32659]: pgmap v8258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:42 smithi132 ceph-mon[37580]: pgmap v8258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:42 smithi112 ceph-mon[37501]: pgmap v8258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:45.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:44 smithi107 ceph-mon[32659]: pgmap v8259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:57:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:44 smithi132 ceph-mon[37580]: pgmap v8259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:57:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:44 smithi112 ceph-mon[37501]: pgmap v8259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:57:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14802.084219 secs 2024-07-26T16:57:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:46 smithi107 ceph-mon[32659]: pgmap v8260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:57:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14802.084219 secs 2024-07-26T16:57:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:46 smithi132 ceph-mon[37580]: pgmap v8260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:57:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14802.084219 secs 2024-07-26T16:57:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:46 smithi112 ceph-mon[37501]: pgmap v8260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:57:48.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:48.943+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:57:48.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:48.943+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:57:48.946 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:57:48.943+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:57:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:48 smithi107 ceph-mon[32659]: pgmap v8261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:57:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:48 smithi132 ceph-mon[37580]: pgmap v8261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:57:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:48 smithi112 ceph-mon[37501]: pgmap v8261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:57:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:51 smithi107 ceph-mon[32659]: pgmap v8262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:51 smithi132 ceph-mon[37580]: pgmap v8262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:51 smithi112 ceph-mon[37501]: pgmap v8262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14807.084323 secs 2024-07-26T16:57:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14807.084323 secs 2024-07-26T16:57:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14807.084323 secs 2024-07-26T16:57:53.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:53 smithi107 ceph-mon[32659]: pgmap v8263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:53 smithi132 ceph-mon[37580]: pgmap v8263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:53.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:53 smithi112 ceph-mon[37501]: pgmap v8263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:57:55.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:55 smithi107 ceph-mon[32659]: pgmap v8264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:55 smithi132 ceph-mon[37580]: pgmap v8264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:55.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:55 smithi112 ceph-mon[37501]: pgmap v8264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14812.084401 secs 2024-07-26T16:57:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:57 smithi107 ceph-mon[32659]: pgmap v8265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14812.084401 secs 2024-07-26T16:57:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:57 smithi132 ceph-mon[37580]: pgmap v8265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14812.084401 secs 2024-07-26T16:57:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:57 smithi112 ceph-mon[37501]: pgmap v8265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:57:59.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:57:59 smithi107 ceph-mon[32659]: pgmap v8266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:57:59 smithi132 ceph-mon[37580]: pgmap v8266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:57:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:57:59 smithi112 ceph-mon[37501]: pgmap v8266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:00 smithi112 ceph-mon[37501]: pgmap v8267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:00 smithi107 ceph-mon[32659]: pgmap v8267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:00 smithi132 ceph-mon[37580]: pgmap v8267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14817.084515 secs 2024-07-26T16:58:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14817.084515 secs 2024-07-26T16:58:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14817.084515 secs 2024-07-26T16:58:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:02 smithi112 ceph-mon[37501]: pgmap v8268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:58:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:02 smithi107 ceph-mon[32659]: pgmap v8268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:58:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:02 smithi132 ceph-mon[37580]: pgmap v8268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:58:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:58:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:58:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:03 smithi112 ceph-mon[37501]: pgmap v8269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:58:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:03 smithi112 ceph-mon[37501]: pgmap v8270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:58:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:58:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:58:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:58:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:03 smithi107 ceph-mon[32659]: pgmap v8269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:58:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:03 smithi107 ceph-mon[32659]: pgmap v8270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:58:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:58:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:58:03.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:58:03.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:03 smithi132 ceph-mon[37580]: pgmap v8269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:58:03.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:03 smithi132 ceph-mon[37580]: pgmap v8270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:58:03.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:58:03.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:03.951+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:58:03.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:03.951+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:58:03.953 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:03.951+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:58:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:05 smithi112 ceph-mon[37501]: pgmap v8271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:58:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:05 smithi107 ceph-mon[32659]: pgmap v8271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:58:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:05 smithi132 ceph-mon[37580]: pgmap v8271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:58:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14822.084615 secs 2024-07-26T16:58:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14822.084615 secs 2024-07-26T16:58:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14822.084615 secs 2024-07-26T16:58:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:07 smithi112 ceph-mon[37501]: pgmap v8272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:07 smithi107 ceph-mon[32659]: pgmap v8272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:07 smithi132 ceph-mon[37580]: pgmap v8272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:09.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:09 smithi112 ceph-mon[37501]: pgmap v8273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:58:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:09 smithi107 ceph-mon[32659]: pgmap v8273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:58:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:09 smithi132 ceph-mon[37580]: pgmap v8273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:58:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:11 smithi112 ceph-mon[37501]: pgmap v8274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:58:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:11 smithi107 ceph-mon[32659]: pgmap v8274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:58:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:11 smithi132 ceph-mon[37580]: pgmap v8274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:58:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14827.084733 secs 2024-07-26T16:58:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14827.084733 secs 2024-07-26T16:58:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14827.084733 secs 2024-07-26T16:58:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:13 smithi112 ceph-mon[37501]: pgmap v8275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:58:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:13 smithi107 ceph-mon[32659]: pgmap v8275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:58:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:13 smithi132 ceph-mon[37580]: pgmap v8275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:58:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:15 smithi112 ceph-mon[37501]: pgmap v8276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:15 smithi107 ceph-mon[32659]: pgmap v8276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:15 smithi132 ceph-mon[37580]: pgmap v8276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14832.084848 secs 2024-07-26T16:58:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14832.084848 secs 2024-07-26T16:58:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14832.084848 secs 2024-07-26T16:58:17.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:17 smithi107 ceph-mon[32659]: pgmap v8277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:17 smithi112 ceph-mon[37501]: pgmap v8277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:17.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:17 smithi132 ceph-mon[37580]: pgmap v8277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:18.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:18.962+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:58:18.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:18.962+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:58:18.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:18.962+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:58:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:19 smithi107 ceph-mon[32659]: pgmap v8278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:19 smithi112 ceph-mon[37501]: pgmap v8278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:19 smithi132 ceph-mon[37580]: pgmap v8278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:21 smithi112 ceph-mon[37501]: pgmap v8279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:21 smithi107 ceph-mon[32659]: pgmap v8279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:21 smithi132 ceph-mon[37580]: pgmap v8279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14837.084968 secs 2024-07-26T16:58:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14837.084968 secs 2024-07-26T16:58:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14837.084968 secs 2024-07-26T16:58:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:23 smithi112 ceph-mon[37501]: pgmap v8280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:23 smithi107 ceph-mon[32659]: pgmap v8280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:23 smithi132 ceph-mon[37580]: pgmap v8280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:25.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:25 smithi112 ceph-mon[37501]: pgmap v8281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:25 smithi107 ceph-mon[32659]: pgmap v8281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:25 smithi132 ceph-mon[37580]: pgmap v8281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14842.085103 secs 2024-07-26T16:58:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14842.085103 secs 2024-07-26T16:58:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14842.085103 secs 2024-07-26T16:58:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:27 smithi112 ceph-mon[37501]: pgmap v8282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:27 smithi107 ceph-mon[32659]: pgmap v8282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:27 smithi132 ceph-mon[37580]: pgmap v8282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:29 smithi112 ceph-mon[37501]: pgmap v8283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:29 smithi107 ceph-mon[32659]: pgmap v8283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:29 smithi132 ceph-mon[37580]: pgmap v8283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:31 smithi112 ceph-mon[37501]: pgmap v8284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:31 smithi107 ceph-mon[32659]: pgmap v8284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:31 smithi132 ceph-mon[37580]: pgmap v8284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:32 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14847.085225 secs 2024-07-26T16:58:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:32 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14847.085225 secs 2024-07-26T16:58:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:32 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14847.085225 secs 2024-07-26T16:58:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:33 smithi112 ceph-mon[37501]: pgmap v8285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:33 smithi107 ceph-mon[32659]: pgmap v8285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:33 smithi132 ceph-mon[37580]: pgmap v8285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:33.973+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:58:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:33.973+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:58:33.976 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:33.973+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:58:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:35 smithi112 ceph-mon[37501]: pgmap v8286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:35 smithi107 ceph-mon[32659]: pgmap v8286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:35 smithi132 ceph-mon[37580]: pgmap v8286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14852.085353 secs 2024-07-26T16:58:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14852.085353 secs 2024-07-26T16:58:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14852.085353 secs 2024-07-26T16:58:37.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:37 smithi112 ceph-mon[37501]: pgmap v8287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:37 smithi107 ceph-mon[32659]: pgmap v8287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:37 smithi132 ceph-mon[37580]: pgmap v8287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:39 smithi112 ceph-mon[37501]: pgmap v8288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:39 smithi107 ceph-mon[32659]: pgmap v8288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:39 smithi132 ceph-mon[37580]: pgmap v8288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:41 smithi112 ceph-mon[37501]: pgmap v8289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:41 smithi107 ceph-mon[32659]: pgmap v8289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:41 smithi132 ceph-mon[37580]: pgmap v8289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14857.085426 secs 2024-07-26T16:58:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14857.085426 secs 2024-07-26T16:58:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14857.085426 secs 2024-07-26T16:58:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:43 smithi112 ceph-mon[37501]: pgmap v8290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:43 smithi107 ceph-mon[32659]: pgmap v8290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:43 smithi132 ceph-mon[37580]: pgmap v8290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:44.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:44.201+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T16:58:44.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:44.201+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T16:58:44.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:44.201+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T16:58:44.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:44.201+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T16:58:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:45 smithi112 ceph-mon[37501]: pgmap v8291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:45 smithi107 ceph-mon[32659]: pgmap v8291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:45 smithi132 ceph-mon[37580]: pgmap v8291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14862.085550 secs 2024-07-26T16:58:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14862.085550 secs 2024-07-26T16:58:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14862.085550 secs 2024-07-26T16:58:47.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:47 smithi107 ceph-mon[32659]: pgmap v8292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:47 smithi112 ceph-mon[37501]: pgmap v8292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:47.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:47 smithi132 ceph-mon[37580]: pgmap v8292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:48.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:48.989+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:58:48.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:48.989+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:58:48.992 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:58:48.989+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:58:49.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:49 smithi107 ceph-mon[32659]: pgmap v8293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:49 smithi112 ceph-mon[37501]: pgmap v8293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:49 smithi132 ceph-mon[37580]: pgmap v8293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:51 smithi112 ceph-mon[37501]: pgmap v8294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:51 smithi107 ceph-mon[32659]: pgmap v8294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:51 smithi132 ceph-mon[37580]: pgmap v8294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:58:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14867.085704 secs 2024-07-26T16:58:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14867.085704 secs 2024-07-26T16:58:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14867.085704 secs 2024-07-26T16:58:53.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:53 smithi112 ceph-mon[37501]: pgmap v8295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:53.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:53 smithi107 ceph-mon[32659]: pgmap v8295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:53 smithi132 ceph-mon[37580]: pgmap v8295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:55.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:55 smithi112 ceph-mon[37501]: pgmap v8296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:55 smithi107 ceph-mon[32659]: pgmap v8296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:55 smithi132 ceph-mon[37580]: pgmap v8296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14872.085853 secs 2024-07-26T16:58:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14872.085853 secs 2024-07-26T16:58:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14872.085853 secs 2024-07-26T16:58:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:57 smithi112 ceph-mon[37501]: pgmap v8297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:57.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:57 smithi107 ceph-mon[32659]: pgmap v8297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:57 smithi132 ceph-mon[37580]: pgmap v8297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:58:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:58:59 smithi112 ceph-mon[37501]: pgmap v8298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:58:59 smithi107 ceph-mon[32659]: pgmap v8298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:58:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:58:59 smithi132 ceph-mon[37580]: pgmap v8298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:01 smithi112 ceph-mon[37501]: pgmap v8299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:01 smithi107 ceph-mon[32659]: pgmap v8299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:01 smithi132 ceph-mon[37580]: pgmap v8299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14877.085994 secs 2024-07-26T16:59:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14877.085994 secs 2024-07-26T16:59:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14877.085994 secs 2024-07-26T16:59:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:03 smithi112 ceph-mon[37501]: pgmap v8300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:03 smithi107 ceph-mon[32659]: pgmap v8300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:03 smithi132 ceph-mon[37580]: pgmap v8300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:03.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:03.993+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:59:03.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:03.993+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:59:03.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:03.993+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:59:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:59:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:59:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:59:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:59:05.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:59:05.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:59:05.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:59:05.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: pgmap v8301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:59:05.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:59:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:59:05.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:59:05.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:59:05.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:59:05.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: pgmap v8301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:59:05.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:59:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:59:05.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T16:59:05.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T16:59:05.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:59:05.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:59:05.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: pgmap v8301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:59:05.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T16:59:07.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14882.086121 secs 2024-07-26T16:59:07.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:07 smithi107 ceph-mon[32659]: pgmap v8302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:59:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14882.086121 secs 2024-07-26T16:59:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:07 smithi132 ceph-mon[37580]: pgmap v8302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:59:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14882.086121 secs 2024-07-26T16:59:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:07 smithi112 ceph-mon[37501]: pgmap v8302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:59:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:09 smithi107 ceph-mon[32659]: pgmap v8303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:59:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:09 smithi132 ceph-mon[37580]: pgmap v8303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:59:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:09 smithi112 ceph-mon[37501]: pgmap v8303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:59:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:11 smithi107 ceph-mon[32659]: pgmap v8304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:59:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:11 smithi132 ceph-mon[37580]: pgmap v8304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:59:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:11 smithi112 ceph-mon[37501]: pgmap v8304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:59:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14887.086138 secs 2024-07-26T16:59:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14887.086138 secs 2024-07-26T16:59:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14887.086138 secs 2024-07-26T16:59:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:13 smithi132 ceph-mon[37580]: pgmap v8305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:59:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:13 smithi112 ceph-mon[37501]: pgmap v8305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:59:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:13 smithi107 ceph-mon[32659]: pgmap v8305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:59:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:15 smithi112 ceph-mon[37501]: pgmap v8306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:59:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:15 smithi107 ceph-mon[32659]: pgmap v8306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:59:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:15 smithi132 ceph-mon[37580]: pgmap v8306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:59:17.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14892.086206 secs 2024-07-26T16:59:17.596 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:17 smithi107 ceph-mon[32659]: pgmap v8307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:17.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14892.086206 secs 2024-07-26T16:59:17.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:17 smithi132 ceph-mon[37580]: pgmap v8307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14892.086206 secs 2024-07-26T16:59:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:17 smithi112 ceph-mon[37501]: pgmap v8307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:19.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:19.009+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:59:19.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:19.009+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:59:19.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:19.009+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:59:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:19 smithi112 ceph-mon[37501]: pgmap v8308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:19 smithi107 ceph-mon[32659]: pgmap v8308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:19 smithi132 ceph-mon[37580]: pgmap v8308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:21.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:21 smithi112 ceph-mon[37501]: pgmap v8309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:21 smithi107 ceph-mon[32659]: pgmap v8309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:21 smithi132 ceph-mon[37580]: pgmap v8309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14897.086316 secs 2024-07-26T16:59:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14897.086316 secs 2024-07-26T16:59:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14897.086316 secs 2024-07-26T16:59:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:23 smithi112 ceph-mon[37501]: pgmap v8310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:23 smithi107 ceph-mon[32659]: pgmap v8310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:23 smithi132 ceph-mon[37580]: pgmap v8310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:25 smithi112 ceph-mon[37501]: pgmap v8311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:25 smithi107 ceph-mon[32659]: pgmap v8311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:25 smithi132 ceph-mon[37580]: pgmap v8311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14902.086427 secs 2024-07-26T16:59:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:27 smithi112 ceph-mon[37501]: pgmap v8312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14902.086427 secs 2024-07-26T16:59:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:27 smithi107 ceph-mon[32659]: pgmap v8312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14902.086427 secs 2024-07-26T16:59:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:27 smithi132 ceph-mon[37580]: pgmap v8312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:29 smithi112 ceph-mon[37501]: pgmap v8313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:29 smithi107 ceph-mon[32659]: pgmap v8313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:29 smithi132 ceph-mon[37580]: pgmap v8313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:31 smithi112 ceph-mon[37501]: pgmap v8314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14907.086499 secs 2024-07-26T16:59:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:31 smithi107 ceph-mon[32659]: pgmap v8314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14907.086499 secs 2024-07-26T16:59:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:31 smithi132 ceph-mon[37580]: pgmap v8314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14907.086499 secs 2024-07-26T16:59:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:33 smithi112 ceph-mon[37501]: pgmap v8315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:33 smithi107 ceph-mon[32659]: pgmap v8315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:33 smithi132 ceph-mon[37580]: pgmap v8315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:34.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:34.019+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:59:34.021 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:34.019+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:59:34.022 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:34.019+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:59:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:35 smithi112 ceph-mon[37501]: pgmap v8316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:35 smithi107 ceph-mon[32659]: pgmap v8316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:35 smithi132 ceph-mon[37580]: pgmap v8316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14912.086628 secs 2024-07-26T16:59:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14912.086628 secs 2024-07-26T16:59:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14912.086628 secs 2024-07-26T16:59:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:37 smithi112 ceph-mon[37501]: pgmap v8317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:37 smithi107 ceph-mon[32659]: pgmap v8317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:37 smithi132 ceph-mon[37580]: pgmap v8317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:39.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:39 smithi112 ceph-mon[37501]: pgmap v8318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:39 smithi107 ceph-mon[32659]: pgmap v8318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:39 smithi132 ceph-mon[37580]: pgmap v8318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:41 smithi112 ceph-mon[37501]: pgmap v8319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14917.086763 secs 2024-07-26T16:59:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:41 smithi107 ceph-mon[32659]: pgmap v8319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14917.086763 secs 2024-07-26T16:59:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:41 smithi132 ceph-mon[37580]: pgmap v8319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14917.086763 secs 2024-07-26T16:59:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:43 smithi112 ceph-mon[37501]: pgmap v8320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:43 smithi107 ceph-mon[32659]: pgmap v8320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:43 smithi132 ceph-mon[37580]: pgmap v8320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:45 smithi112 ceph-mon[37501]: pgmap v8321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:45 smithi107 ceph-mon[32659]: pgmap v8321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:45 smithi132 ceph-mon[37580]: pgmap v8321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14922.086828 secs 2024-07-26T16:59:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14922.086828 secs 2024-07-26T16:59:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14922.086828 secs 2024-07-26T16:59:47.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:47 smithi107 ceph-mon[32659]: pgmap v8322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:47 smithi112 ceph-mon[37501]: pgmap v8322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:47.712 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:47 smithi132 ceph-mon[37580]: pgmap v8322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:49.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:49.025+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T16:59:49.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:49.025+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:59:49.028 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T16:59:49.025+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T16:59:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:49 smithi112 ceph-mon[37501]: pgmap v8323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:50.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:49 smithi107 ceph-mon[32659]: pgmap v8323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:49 smithi132 ceph-mon[37580]: pgmap v8323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:59:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:51 smithi112 ceph-mon[37501]: pgmap v8324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14927.086935 secs 2024-07-26T16:59:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:51 smithi107 ceph-mon[32659]: pgmap v8324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14927.086935 secs 2024-07-26T16:59:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:51 smithi132 ceph-mon[37580]: pgmap v8324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14927.086935 secs 2024-07-26T16:59:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:53 smithi112 ceph-mon[37501]: pgmap v8325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:53 smithi107 ceph-mon[32659]: pgmap v8325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:53 smithi132 ceph-mon[37580]: pgmap v8325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:59:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:55 smithi112 ceph-mon[37501]: pgmap v8326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:55 smithi107 ceph-mon[32659]: pgmap v8326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:55 smithi132 ceph-mon[37580]: pgmap v8326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:59:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14932.087056 secs 2024-07-26T16:59:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14932.087056 secs 2024-07-26T16:59:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14932.087056 secs 2024-07-26T16:59:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:57 smithi112 ceph-mon[37501]: pgmap v8327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:57 smithi107 ceph-mon[32659]: pgmap v8327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:57 smithi132 ceph-mon[37580]: pgmap v8327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:59:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 16:59:59 smithi112 ceph-mon[37501]: pgmap v8328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:00:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 16:59:59 smithi107 ceph-mon[32659]: pgmap v8328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:00:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 16:59:59 smithi132 ceph-mon[37580]: pgmap v8328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:00:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:00:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:00:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:00:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:01 smithi112 ceph-mon[37501]: pgmap v8329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14937.087173 secs 2024-07-26T17:00:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:01 smithi107 ceph-mon[32659]: pgmap v8329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14937.087173 secs 2024-07-26T17:00:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:01 smithi132 ceph-mon[37580]: pgmap v8329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14937.087173 secs 2024-07-26T17:00:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:03 smithi112 ceph-mon[37501]: pgmap v8330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:04.032+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:00:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:04.032+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:00:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:04.032+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:00:04.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:03 smithi107 ceph-mon[32659]: pgmap v8330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:03 smithi132 ceph-mon[37580]: pgmap v8330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:00:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:00:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:00:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: pgmap v8331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:05.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:00:05.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:00:05.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:00:05.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:00:05.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:00:05.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:00:05.955 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:00:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: pgmap v8331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:06.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:00:06.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:00:06.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:00:06.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:00:06.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:00:06.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:00:06.047 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:00:06.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: pgmap v8331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:06.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:00:06.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:00:06.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:00:06.064 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:00:06.064 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:00:06.064 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:00:06.064 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:00:06.065 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:00:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:06 smithi112 ceph-mon[37501]: pgmap v8332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:00:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:06 smithi112 ceph-mon[37501]: pgmap v8333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:00:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14942.087293 secs 2024-07-26T17:00:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:06 smithi107 ceph-mon[32659]: pgmap v8332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:00:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:06 smithi107 ceph-mon[32659]: pgmap v8333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:00:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14942.087293 secs 2024-07-26T17:00:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:06 smithi132 ceph-mon[37580]: pgmap v8332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:00:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:06 smithi132 ceph-mon[37580]: pgmap v8333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:00:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14942.087293 secs 2024-07-26T17:00:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:08 smithi112 ceph-mon[37501]: pgmap v8334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T17:00:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:08 smithi107 ceph-mon[32659]: pgmap v8334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T17:00:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:08 smithi132 ceph-mon[37580]: pgmap v8334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T17:00:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:10 smithi112 ceph-mon[37501]: pgmap v8335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:00:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:10 smithi107 ceph-mon[32659]: pgmap v8335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:00:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:10 smithi132 ceph-mon[37580]: pgmap v8335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:00:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14947.087421 secs 2024-07-26T17:00:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14947.087421 secs 2024-07-26T17:00:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14947.087421 secs 2024-07-26T17:00:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:12 smithi107 ceph-mon[32659]: pgmap v8336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:00:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:12 smithi132 ceph-mon[37580]: pgmap v8336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:00:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:12 smithi112 ceph-mon[37501]: pgmap v8336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:00:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:14 smithi107 ceph-mon[32659]: pgmap v8337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:00:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:14 smithi132 ceph-mon[37580]: pgmap v8337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:00:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:14 smithi112 ceph-mon[37501]: pgmap v8337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:00:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:15.217+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:00:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:15.217+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:00:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:15.218+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:00:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:15.218+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:00:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:16 smithi107 ceph-mon[32659]: pgmap v8338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:00:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14952.087547 secs 2024-07-26T17:00:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:16 smithi132 ceph-mon[37580]: pgmap v8338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:00:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14952.087547 secs 2024-07-26T17:00:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:16 smithi112 ceph-mon[37501]: pgmap v8338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:00:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14952.087547 secs 2024-07-26T17:00:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:18 smithi107 ceph-mon[32659]: pgmap v8339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:19.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:19.047+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:00:19.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:19.047+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:00:19.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:19.047+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:00:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:18 smithi132 ceph-mon[37580]: pgmap v8339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:18 smithi112 ceph-mon[37501]: pgmap v8339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:20 smithi107 ceph-mon[32659]: pgmap v8340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T17:00:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:20 smithi132 ceph-mon[37580]: pgmap v8340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T17:00:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:20 smithi112 ceph-mon[37501]: pgmap v8340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T17:00:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14957.087676 secs 2024-07-26T17:00:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14957.087676 secs 2024-07-26T17:00:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14957.087676 secs 2024-07-26T17:00:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:22 smithi107 ceph-mon[32659]: pgmap v8341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:00:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:22 smithi132 ceph-mon[37580]: pgmap v8341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:00:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:22 smithi112 ceph-mon[37501]: pgmap v8341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:00:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:24 smithi107 ceph-mon[32659]: pgmap v8342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:24 smithi132 ceph-mon[37580]: pgmap v8342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:24 smithi112 ceph-mon[37501]: pgmap v8342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:27.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:26 smithi112 ceph-mon[37501]: pgmap v8343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:27.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14962.087795 secs 2024-07-26T17:00:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:26 smithi107 ceph-mon[32659]: pgmap v8343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14962.087795 secs 2024-07-26T17:00:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:26 smithi132 ceph-mon[37580]: pgmap v8343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14962.087795 secs 2024-07-26T17:00:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:28 smithi107 ceph-mon[32659]: pgmap v8344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:28 smithi132 ceph-mon[37580]: pgmap v8344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:28 smithi112 ceph-mon[37501]: pgmap v8344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:31.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:30 smithi107 ceph-mon[32659]: pgmap v8345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:00:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:30 smithi132 ceph-mon[37580]: pgmap v8345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:00:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:30 smithi112 ceph-mon[37501]: pgmap v8345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:00:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14967.087895 secs 2024-07-26T17:00:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14967.087895 secs 2024-07-26T17:00:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14967.087895 secs 2024-07-26T17:00:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:32 smithi107 ceph-mon[32659]: pgmap v8346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:32 smithi132 ceph-mon[37580]: pgmap v8346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:33.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:32 smithi112 ceph-mon[37501]: pgmap v8346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:34.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:34.062+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:00:34.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:34.062+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:00:34.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:34.062+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:00:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:34 smithi107 ceph-mon[32659]: pgmap v8347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:34 smithi132 ceph-mon[37580]: pgmap v8347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:34 smithi112 ceph-mon[37501]: pgmap v8347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:36 smithi107 ceph-mon[32659]: pgmap v8348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14972.088025 secs 2024-07-26T17:00:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:36 smithi132 ceph-mon[37580]: pgmap v8348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:37.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14972.088025 secs 2024-07-26T17:00:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:36 smithi112 ceph-mon[37501]: pgmap v8348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14972.088025 secs 2024-07-26T17:00:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:38 smithi107 ceph-mon[32659]: pgmap v8349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:38 smithi132 ceph-mon[37580]: pgmap v8349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:38 smithi112 ceph-mon[37501]: pgmap v8349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:40 smithi132 ceph-mon[37580]: pgmap v8350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:00:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:40 smithi112 ceph-mon[37501]: pgmap v8350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:00:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:40 smithi107 ceph-mon[32659]: pgmap v8350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:00:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14977.088153 secs 2024-07-26T17:00:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14977.088153 secs 2024-07-26T17:00:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14977.088153 secs 2024-07-26T17:00:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:42 smithi132 ceph-mon[37580]: pgmap v8351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:42 smithi112 ceph-mon[37501]: pgmap v8351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:42 smithi107 ceph-mon[32659]: pgmap v8351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:44 smithi112 ceph-mon[37501]: pgmap v8352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:44 smithi107 ceph-mon[32659]: pgmap v8352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:44 smithi132 ceph-mon[37580]: pgmap v8352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:47.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:46 smithi107 ceph-mon[32659]: pgmap v8353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:47.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14982.088278 secs 2024-07-26T17:00:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:46 smithi112 ceph-mon[37501]: pgmap v8353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14982.088278 secs 2024-07-26T17:00:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:46 smithi132 ceph-mon[37580]: pgmap v8353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:47.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14982.088278 secs 2024-07-26T17:00:49.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:49.078+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:00:49.080 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:49.078+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:00:49.081 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:00:49.078+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:00:49.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:48 smithi107 ceph-mon[32659]: pgmap v8354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:49.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:48 smithi132 ceph-mon[37580]: pgmap v8354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:48 smithi112 ceph-mon[37501]: pgmap v8354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:50 smithi112 ceph-mon[37501]: pgmap v8355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:00:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:50 smithi107 ceph-mon[32659]: pgmap v8355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:00:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:50 smithi132 ceph-mon[37580]: pgmap v8355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:00:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14987.088392 secs 2024-07-26T17:00:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14987.088392 secs 2024-07-26T17:00:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14987.088392 secs 2024-07-26T17:00:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:52 smithi112 ceph-mon[37501]: pgmap v8356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:52 smithi107 ceph-mon[32659]: pgmap v8356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:52 smithi132 ceph-mon[37580]: pgmap v8356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:00:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:54 smithi112 ceph-mon[37501]: pgmap v8357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:54 smithi107 ceph-mon[32659]: pgmap v8357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:54 smithi132 ceph-mon[37580]: pgmap v8357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:56 smithi112 ceph-mon[37501]: pgmap v8358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14992.088520 secs 2024-07-26T17:00:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:56 smithi107 ceph-mon[32659]: pgmap v8358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14992.088520 secs 2024-07-26T17:00:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:56 smithi132 ceph-mon[37580]: pgmap v8358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:00:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14992.088520 secs 2024-07-26T17:00:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:00:58 smithi112 ceph-mon[37501]: pgmap v8359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:00:58 smithi107 ceph-mon[32659]: pgmap v8359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:00:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:00:58 smithi132 ceph-mon[37580]: pgmap v8359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:00 smithi112 ceph-mon[37501]: pgmap v8360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:01.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:00 smithi107 ceph-mon[32659]: pgmap v8360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:00 smithi132 ceph-mon[37580]: pgmap v8360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 14997.088662 secs 2024-07-26T17:01:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 14997.088662 secs 2024-07-26T17:01:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 14997.088662 secs 2024-07-26T17:01:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:02 smithi112 ceph-mon[37501]: pgmap v8361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:02 smithi107 ceph-mon[32659]: pgmap v8361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:02 smithi132 ceph-mon[37580]: pgmap v8361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:04.082+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:01:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:04.082+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:01:04.086 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:04.082+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:01:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:04 smithi112 ceph-mon[37501]: pgmap v8362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:04 smithi107 ceph-mon[32659]: pgmap v8362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:04 smithi132 ceph-mon[37580]: pgmap v8362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:01:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:01:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:01:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:06 smithi112 ceph-mon[37501]: pgmap v8363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15002.088810 secs 2024-07-26T17:01:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:01:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:01:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:01:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:06 smithi107 ceph-mon[32659]: pgmap v8363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15002.088810 secs 2024-07-26T17:01:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:01:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:01:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:01:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:06 smithi132 ceph-mon[37580]: pgmap v8363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15002.088810 secs 2024-07-26T17:01:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:01:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:01:07.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:01:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:07 smithi112 ceph-mon[37501]: pgmap v8364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:01:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:07 smithi112 ceph-mon[37501]: pgmap v8365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:08.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:07 smithi107 ceph-mon[32659]: pgmap v8364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:01:08.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:07 smithi107 ceph-mon[32659]: pgmap v8365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:07 smithi132 ceph-mon[37580]: pgmap v8364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:01:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:07 smithi132 ceph-mon[37580]: pgmap v8365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:09 smithi112 ceph-mon[37501]: pgmap v8366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:01:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:09 smithi107 ceph-mon[32659]: pgmap v8366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:01:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:09 smithi132 ceph-mon[37580]: pgmap v8366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:01:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15007.088938 secs 2024-07-26T17:01:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:11 smithi112 ceph-mon[37501]: pgmap v8367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:01:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15007.088938 secs 2024-07-26T17:01:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:11 smithi107 ceph-mon[32659]: pgmap v8367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:01:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15007.088938 secs 2024-07-26T17:01:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:11 smithi132 ceph-mon[37580]: pgmap v8367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:01:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:13 smithi112 ceph-mon[37501]: pgmap v8368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:01:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:13 smithi107 ceph-mon[32659]: pgmap v8368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:01:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:13 smithi132 ceph-mon[37580]: pgmap v8368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:01:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:15 smithi112 ceph-mon[37501]: pgmap v8369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:01:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:15 smithi107 ceph-mon[32659]: pgmap v8369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:01:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:15 smithi132 ceph-mon[37580]: pgmap v8369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:01:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15012.089000 secs 2024-07-26T17:01:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15012.089000 secs 2024-07-26T17:01:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15012.089000 secs 2024-07-26T17:01:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:17 smithi107 ceph-mon[32659]: pgmap v8370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:01:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:17 smithi132 ceph-mon[37580]: pgmap v8370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:01:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:17 smithi112 ceph-mon[37501]: pgmap v8370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:01:19.097 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:19.096+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:01:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:19.096+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:01:19.098 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:19.096+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:01:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:19 smithi107 ceph-mon[32659]: pgmap v8371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:19 smithi132 ceph-mon[37580]: pgmap v8371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:19 smithi112 ceph-mon[37501]: pgmap v8371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15017.089101 secs 2024-07-26T17:01:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:21 smithi107 ceph-mon[32659]: pgmap v8372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15017.089101 secs 2024-07-26T17:01:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:21 smithi132 ceph-mon[37580]: pgmap v8372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15017.089101 secs 2024-07-26T17:01:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:21 smithi112 ceph-mon[37501]: pgmap v8372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:23 smithi107 ceph-mon[32659]: pgmap v8373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:23 smithi132 ceph-mon[37580]: pgmap v8373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:23 smithi112 ceph-mon[37501]: pgmap v8373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:25 smithi107 ceph-mon[32659]: pgmap v8374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:25 smithi132 ceph-mon[37580]: pgmap v8374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:25 smithi112 ceph-mon[37501]: pgmap v8374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15022.089221 secs 2024-07-26T17:01:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15022.089221 secs 2024-07-26T17:01:27.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15022.089221 secs 2024-07-26T17:01:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:27 smithi107 ceph-mon[32659]: pgmap v8375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:27 smithi132 ceph-mon[37580]: pgmap v8375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:27 smithi112 ceph-mon[37501]: pgmap v8375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:29 smithi107 ceph-mon[32659]: pgmap v8376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:29 smithi132 ceph-mon[37580]: pgmap v8376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:29 smithi112 ceph-mon[37501]: pgmap v8376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15027.089341 secs 2024-07-26T17:01:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:31 smithi107 ceph-mon[32659]: pgmap v8377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15027.089341 secs 2024-07-26T17:01:32.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:31 smithi132 ceph-mon[37580]: pgmap v8377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15027.089341 secs 2024-07-26T17:01:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:31 smithi112 ceph-mon[37501]: pgmap v8377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:34.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:34.097+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:01:34.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:34.097+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:01:34.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:34.097+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:01:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:34 smithi107 ceph-mon[32659]: pgmap v8378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:34 smithi132 ceph-mon[37580]: pgmap v8378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:34 smithi112 ceph-mon[37501]: pgmap v8378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:36 smithi107 ceph-mon[32659]: pgmap v8379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:36 smithi132 ceph-mon[37580]: pgmap v8379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:36 smithi112 ceph-mon[37501]: pgmap v8379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15032.089425 secs 2024-07-26T17:01:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15032.089425 secs 2024-07-26T17:01:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15032.089425 secs 2024-07-26T17:01:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:38 smithi107 ceph-mon[32659]: pgmap v8380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:38 smithi132 ceph-mon[37580]: pgmap v8380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:38 smithi112 ceph-mon[37501]: pgmap v8380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:40 smithi132 ceph-mon[37580]: pgmap v8381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:40 smithi112 ceph-mon[37501]: pgmap v8381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:40 smithi107 ceph-mon[32659]: pgmap v8381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15037.089562 secs 2024-07-26T17:01:42.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:42 smithi132 ceph-mon[37580]: pgmap v8382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15037.089562 secs 2024-07-26T17:01:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:42 smithi112 ceph-mon[37501]: pgmap v8382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15037.089562 secs 2024-07-26T17:01:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:42 smithi107 ceph-mon[32659]: pgmap v8382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:44 smithi132 ceph-mon[37580]: pgmap v8383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:44 smithi112 ceph-mon[37501]: pgmap v8383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:44 smithi107 ceph-mon[32659]: pgmap v8383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:46 smithi112 ceph-mon[37501]: pgmap v8384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:46 smithi107 ceph-mon[32659]: pgmap v8384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:46 smithi132 ceph-mon[37580]: pgmap v8384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:01:47.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15042.089677 secs 2024-07-26T17:01:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15042.089677 secs 2024-07-26T17:01:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15042.089677 secs 2024-07-26T17:01:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:48 smithi107 ceph-mon[32659]: pgmap v8385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:01:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:48 smithi112 ceph-mon[37501]: pgmap v8385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:01:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:48 smithi132 ceph-mon[37580]: pgmap v8385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:01:49.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:49.100+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:01:49.103 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:49.100+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:01:49.104 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:01:49.100+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:01:50.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:50 smithi132 ceph-mon[37580]: pgmap v8386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:50 smithi112 ceph-mon[37501]: pgmap v8386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:50 smithi107 ceph-mon[32659]: pgmap v8386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:51 smithi112 ceph-mon[37501]: pgmap v8387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:51 smithi107 ceph-mon[32659]: pgmap v8387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:51 smithi132 ceph-mon[37580]: pgmap v8387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:01:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15047.089820 secs 2024-07-26T17:01:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15047.089820 secs 2024-07-26T17:01:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15047.089820 secs 2024-07-26T17:01:53.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:53 smithi112 ceph-mon[37501]: pgmap v8388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:53.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:53 smithi107 ceph-mon[32659]: pgmap v8388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:53 smithi132 ceph-mon[37580]: pgmap v8388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:01:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:55 smithi112 ceph-mon[37501]: pgmap v8389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:55 smithi107 ceph-mon[32659]: pgmap v8389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:55 smithi132 ceph-mon[37580]: pgmap v8389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15052.089950 secs 2024-07-26T17:01:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15052.089950 secs 2024-07-26T17:01:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15052.089950 secs 2024-07-26T17:01:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:57 smithi112 ceph-mon[37501]: pgmap v8390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:57 smithi107 ceph-mon[32659]: pgmap v8390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:57 smithi132 ceph-mon[37580]: pgmap v8390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:01:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:01:59 smithi112 ceph-mon[37501]: pgmap v8391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:01:59 smithi107 ceph-mon[32659]: pgmap v8391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:01:59 smithi132 ceph-mon[37580]: pgmap v8391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15057.090082 secs 2024-07-26T17:02:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:01 smithi112 ceph-mon[37501]: pgmap v8392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15057.090082 secs 2024-07-26T17:02:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:01 smithi107 ceph-mon[32659]: pgmap v8392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15057.090082 secs 2024-07-26T17:02:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:01 smithi132 ceph-mon[37580]: pgmap v8392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:03 smithi107 ceph-mon[32659]: pgmap v8393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:03 smithi132 ceph-mon[37580]: pgmap v8393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:04.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:04.116+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:02:04.118 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:04.117+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:02:04.119 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:04.117+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:02:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:03 smithi112 ceph-mon[37501]: pgmap v8393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:05 smithi107 ceph-mon[32659]: pgmap v8394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:05 smithi132 ceph-mon[37580]: pgmap v8394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:05 smithi112 ceph-mon[37501]: pgmap v8394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15062.090214 secs 2024-07-26T17:02:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15062.090214 secs 2024-07-26T17:02:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15062.090214 secs 2024-07-26T17:02:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:07 smithi107 ceph-mon[32659]: pgmap v8395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:02:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:02:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:02:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:07 smithi132 ceph-mon[37580]: pgmap v8395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:02:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:02:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:02:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:07 smithi112 ceph-mon[37501]: pgmap v8395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:02:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:02:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:02:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:02:09.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:02:09.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:02:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:02:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:02:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:02:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:02:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:02:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:02:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:09 smithi107 ceph-mon[32659]: pgmap v8396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:09 smithi107 ceph-mon[32659]: pgmap v8397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:09 smithi132 ceph-mon[37580]: pgmap v8396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:09 smithi132 ceph-mon[37580]: pgmap v8397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:09 smithi112 ceph-mon[37501]: pgmap v8396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:09 smithi112 ceph-mon[37501]: pgmap v8397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15067.090337 secs 2024-07-26T17:02:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:11 smithi107 ceph-mon[32659]: pgmap v8398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15067.090337 secs 2024-07-26T17:02:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:11 smithi132 ceph-mon[37580]: pgmap v8398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15067.090337 secs 2024-07-26T17:02:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:11 smithi112 ceph-mon[37501]: pgmap v8398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:13 smithi107 ceph-mon[32659]: pgmap v8399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:02:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:13 smithi132 ceph-mon[37580]: pgmap v8399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:02:14.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:13 smithi112 ceph-mon[37501]: pgmap v8399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:02:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:15 smithi107 ceph-mon[32659]: pgmap v8400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:16.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:15 smithi132 ceph-mon[37580]: pgmap v8400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:15 smithi112 ceph-mon[37501]: pgmap v8400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15072.090442 secs 2024-07-26T17:02:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15072.090442 secs 2024-07-26T17:02:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15072.090442 secs 2024-07-26T17:02:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:17 smithi107 ceph-mon[32659]: pgmap v8401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:17 smithi132 ceph-mon[37580]: pgmap v8401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:17 smithi112 ceph-mon[37501]: pgmap v8401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:02:19.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:19.124+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:02:19.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:19.124+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:02:19.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:19.124+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:02:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:19 smithi107 ceph-mon[32659]: pgmap v8402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:02:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:19 smithi132 ceph-mon[37580]: pgmap v8402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:02:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:19 smithi112 ceph-mon[37501]: pgmap v8402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:02:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15077.090578 secs 2024-07-26T17:02:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:21 smithi107 ceph-mon[32659]: pgmap v8403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15077.090578 secs 2024-07-26T17:02:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:21 smithi132 ceph-mon[37580]: pgmap v8403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15077.090578 secs 2024-07-26T17:02:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:21 smithi112 ceph-mon[37501]: pgmap v8403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:23 smithi107 ceph-mon[32659]: pgmap v8404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:23 smithi132 ceph-mon[37580]: pgmap v8404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:23 smithi112 ceph-mon[37501]: pgmap v8404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:25 smithi107 ceph-mon[32659]: pgmap v8405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:25 smithi132 ceph-mon[37580]: pgmap v8405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:25 smithi112 ceph-mon[37501]: pgmap v8405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15082.090702 secs 2024-07-26T17:02:27.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15082.090702 secs 2024-07-26T17:02:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15082.090702 secs 2024-07-26T17:02:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:27 smithi132 ceph-mon[37580]: pgmap v8406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:02:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:27 smithi112 ceph-mon[37501]: pgmap v8406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:02:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:27 smithi107 ceph-mon[32659]: pgmap v8406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:02:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:29 smithi112 ceph-mon[37501]: pgmap v8407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:29 smithi107 ceph-mon[32659]: pgmap v8407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:29 smithi132 ceph-mon[37580]: pgmap v8407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:32.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15087.090847 secs 2024-07-26T17:02:32.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:31 smithi112 ceph-mon[37501]: pgmap v8408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15087.090847 secs 2024-07-26T17:02:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:31 smithi107 ceph-mon[32659]: pgmap v8408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15087.090847 secs 2024-07-26T17:02:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:31 smithi132 ceph-mon[37580]: pgmap v8408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:34.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:34.137+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:02:34.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:34.137+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:02:34.140 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:34.137+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:02:34.141 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:33 smithi107 ceph-mon[32659]: pgmap v8409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:33 smithi112 ceph-mon[37501]: pgmap v8409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:33 smithi132 ceph-mon[37580]: pgmap v8409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:35 smithi112 ceph-mon[37501]: pgmap v8410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:35 smithi107 ceph-mon[32659]: pgmap v8410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:35 smithi132 ceph-mon[37580]: pgmap v8410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15092.090946 secs 2024-07-26T17:02:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15092.090946 secs 2024-07-26T17:02:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15092.090946 secs 2024-07-26T17:02:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:37 smithi112 ceph-mon[37501]: pgmap v8411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:37 smithi107 ceph-mon[32659]: pgmap v8411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:37 smithi132 ceph-mon[37580]: pgmap v8411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:39 smithi112 ceph-mon[37501]: pgmap v8412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:39 smithi107 ceph-mon[32659]: pgmap v8412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:39 smithi132 ceph-mon[37580]: pgmap v8412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15097.091093 secs 2024-07-26T17:02:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:41 smithi112 ceph-mon[37501]: pgmap v8413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15097.091093 secs 2024-07-26T17:02:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:41 smithi107 ceph-mon[32659]: pgmap v8413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15097.091093 secs 2024-07-26T17:02:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:41 smithi132 ceph-mon[37580]: pgmap v8413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:43 smithi112 ceph-mon[37501]: pgmap v8414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:43 smithi107 ceph-mon[32659]: pgmap v8414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:43 smithi132 ceph-mon[37580]: pgmap v8414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:45 smithi112 ceph-mon[37501]: pgmap v8415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:45 smithi107 ceph-mon[32659]: pgmap v8415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:45 smithi132 ceph-mon[37580]: pgmap v8415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:47.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15102.091213 secs 2024-07-26T17:02:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15102.091213 secs 2024-07-26T17:02:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15102.091213 secs 2024-07-26T17:02:48.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:47 smithi112 ceph-mon[37501]: pgmap v8416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:02:48.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:47 smithi107 ceph-mon[32659]: pgmap v8416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:02:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:47 smithi132 ceph-mon[37580]: pgmap v8416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:02:49.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:49.153+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:02:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:49.153+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:02:49.156 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:02:49.153+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:02:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:49 smithi112 ceph-mon[37501]: pgmap v8417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:49 smithi107 ceph-mon[32659]: pgmap v8417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:49 smithi132 ceph-mon[37580]: pgmap v8417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15107.091334 secs 2024-07-26T17:02:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:51 smithi112 ceph-mon[37501]: pgmap v8418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15107.091334 secs 2024-07-26T17:02:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:51 smithi107 ceph-mon[32659]: pgmap v8418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15107.091334 secs 2024-07-26T17:02:52.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:51 smithi132 ceph-mon[37580]: pgmap v8418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:02:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:53 smithi112 ceph-mon[37501]: pgmap v8419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:53 smithi107 ceph-mon[32659]: pgmap v8419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:53 smithi132 ceph-mon[37580]: pgmap v8419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:02:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:55 smithi112 ceph-mon[37501]: pgmap v8420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:55 smithi107 ceph-mon[32659]: pgmap v8420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:55 smithi132 ceph-mon[37580]: pgmap v8420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:02:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15112.091445 secs 2024-07-26T17:02:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15112.091445 secs 2024-07-26T17:02:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15112.091445 secs 2024-07-26T17:02:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:57 smithi112 ceph-mon[37501]: pgmap v8421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:57 smithi107 ceph-mon[32659]: pgmap v8421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:02:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:57 smithi132 ceph-mon[37580]: pgmap v8421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:00.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:02:59 smithi112 ceph-mon[37501]: pgmap v8422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:02:59 smithi107 ceph-mon[32659]: pgmap v8422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:02:59 smithi132 ceph-mon[37580]: pgmap v8422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15117.091524 secs 2024-07-26T17:03:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:01 smithi112 ceph-mon[37501]: pgmap v8423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15117.091524 secs 2024-07-26T17:03:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:01 smithi107 ceph-mon[32659]: pgmap v8423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15117.091524 secs 2024-07-26T17:03:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:01 smithi132 ceph-mon[37580]: pgmap v8423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:04.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:04.169+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:03:04.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:04.169+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:03:04.172 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:04.169+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:03:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:03 smithi112 ceph-mon[37501]: pgmap v8424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:03 smithi107 ceph-mon[32659]: pgmap v8424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:03 smithi132 ceph-mon[37580]: pgmap v8424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:05 smithi112 ceph-mon[37501]: pgmap v8425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:03:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:05 smithi107 ceph-mon[32659]: pgmap v8425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:03:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:05 smithi132 ceph-mon[37580]: pgmap v8425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:03:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15122.091646 secs 2024-07-26T17:03:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15122.091646 secs 2024-07-26T17:03:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15122.091646 secs 2024-07-26T17:03:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:07 smithi107 ceph-mon[32659]: pgmap v8426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:07 smithi132 ceph-mon[37580]: pgmap v8426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:07 smithi112 ceph-mon[37501]: pgmap v8426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:03:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:03:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:03:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:10 smithi112 ceph-mon[37501]: pgmap v8427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:10 smithi107 ceph-mon[32659]: pgmap v8427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:10 smithi132 ceph-mon[37580]: pgmap v8427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:10.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15127.091688 secs 2024-07-26T17:03:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:11 smithi107 ceph-mon[32659]: pgmap v8428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15127.091688 secs 2024-07-26T17:03:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:11 smithi132 ceph-mon[37580]: pgmap v8428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:11.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15127.091688 secs 2024-07-26T17:03:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:11 smithi112 ceph-mon[37501]: pgmap v8428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:11.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:14 smithi112 ceph-mon[37501]: pgmap v8429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:03:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:03:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:14 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:14 smithi107 ceph-mon[32659]: pgmap v8429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:03:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:03:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:14 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:14 smithi132 ceph-mon[37580]: pgmap v8429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:03:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:03:14.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:14 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:03:15.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:15.246+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:03:15.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:15.247+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:03:15.248 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:15.247+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:03:15.249 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:15.247+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:03:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:15 smithi112 ceph-mon[37501]: pgmap v8430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:03:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:15 smithi112 ceph-mon[37501]: pgmap v8431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:03:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:15 smithi107 ceph-mon[32659]: pgmap v8430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:03:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:15 smithi107 ceph-mon[32659]: pgmap v8431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:03:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:15 smithi132 ceph-mon[37580]: pgmap v8430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:03:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:15 smithi132 ceph-mon[37580]: pgmap v8431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:03:17.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:17 smithi107 ceph-mon[32659]: pgmap v8432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:03:17.346 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15132.091830 secs 2024-07-26T17:03:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:17 smithi112 ceph-mon[37501]: pgmap v8432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:03:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15132.091830 secs 2024-07-26T17:03:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:17 smithi132 ceph-mon[37580]: pgmap v8432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:03:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15132.091830 secs 2024-07-26T17:03:19.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:19.185+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:03:19.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:19.185+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:03:19.188 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:19.185+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:03:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:19 smithi107 ceph-mon[32659]: pgmap v8433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:03:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:19 smithi112 ceph-mon[37501]: pgmap v8433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:03:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:19 smithi132 ceph-mon[37580]: pgmap v8433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:03:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:21 smithi112 ceph-mon[37501]: pgmap v8434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T17:03:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:21 smithi107 ceph-mon[32659]: pgmap v8434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T17:03:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:21 smithi132 ceph-mon[37580]: pgmap v8434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T17:03:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15137.091986 secs 2024-07-26T17:03:22.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15137.091986 secs 2024-07-26T17:03:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15137.091986 secs 2024-07-26T17:03:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:23 smithi112 ceph-mon[37501]: pgmap v8435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:03:23.479 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:23 smithi132 ceph-mon[37580]: pgmap v8435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:03:23.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:23 smithi107 ceph-mon[32659]: pgmap v8435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:03:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:24 smithi112 ceph-mon[37501]: pgmap v8436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:03:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:24 smithi107 ceph-mon[32659]: pgmap v8436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:03:24.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:24 smithi132 ceph-mon[37580]: pgmap v8436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:03:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:26 smithi112 ceph-mon[37501]: pgmap v8437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:26 smithi107 ceph-mon[32659]: pgmap v8437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:26 smithi132 ceph-mon[37580]: pgmap v8437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:27.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15142.092048 secs 2024-07-26T17:03:27.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15142.092048 secs 2024-07-26T17:03:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15142.092048 secs 2024-07-26T17:03:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:28 smithi112 ceph-mon[37501]: pgmap v8438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:28 smithi107 ceph-mon[32659]: pgmap v8438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:28 smithi132 ceph-mon[37580]: pgmap v8438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:30 smithi107 ceph-mon[32659]: pgmap v8439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:30 smithi112 ceph-mon[37501]: pgmap v8439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:30 smithi132 ceph-mon[37580]: pgmap v8439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:32 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15147.092167 secs 2024-07-26T17:03:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:32 smithi112 ceph-mon[37501]: pgmap v8440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:32 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15147.092167 secs 2024-07-26T17:03:32.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:32 smithi107 ceph-mon[32659]: pgmap v8440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:32 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15147.092167 secs 2024-07-26T17:03:32.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:32 smithi132 ceph-mon[37580]: pgmap v8440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:34.203 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:34.200+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:03:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:34.200+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:03:34.204 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:34.200+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:03:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:34 smithi112 ceph-mon[37501]: pgmap v8441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:34.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:34 smithi107 ceph-mon[32659]: pgmap v8441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:34 smithi132 ceph-mon[37580]: pgmap v8441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:36 smithi112 ceph-mon[37501]: pgmap v8442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:36 smithi107 ceph-mon[32659]: pgmap v8442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:36 smithi132 ceph-mon[37580]: pgmap v8442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15152.092310 secs 2024-07-26T17:03:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15152.092310 secs 2024-07-26T17:03:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15152.092310 secs 2024-07-26T17:03:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:38 smithi112 ceph-mon[37501]: pgmap v8443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:38 smithi107 ceph-mon[32659]: pgmap v8443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:38 smithi132 ceph-mon[37580]: pgmap v8443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:40 smithi112 ceph-mon[37501]: pgmap v8444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:03:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:40 smithi107 ceph-mon[32659]: pgmap v8444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:03:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:40 smithi132 ceph-mon[37580]: pgmap v8444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:03:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15157.092433 secs 2024-07-26T17:03:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:42 smithi112 ceph-mon[37501]: pgmap v8445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:03:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15157.092433 secs 2024-07-26T17:03:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:42 smithi107 ceph-mon[32659]: pgmap v8445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:03:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15157.092433 secs 2024-07-26T17:03:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:42 smithi132 ceph-mon[37580]: pgmap v8445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:03:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:44 smithi107 ceph-mon[32659]: pgmap v8446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:03:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:44 smithi132 ceph-mon[37580]: pgmap v8446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:03:44.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:44 smithi112 ceph-mon[37501]: pgmap v8446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:03:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:46 smithi107 ceph-mon[32659]: pgmap v8447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:46 smithi132 ceph-mon[37580]: pgmap v8447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:46.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:46 smithi112 ceph-mon[37501]: pgmap v8447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:47.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15162.092561 secs 2024-07-26T17:03:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15162.092561 secs 2024-07-26T17:03:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15162.092561 secs 2024-07-26T17:03:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:48 smithi107 ceph-mon[32659]: pgmap v8448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:48 smithi132 ceph-mon[37580]: pgmap v8448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:48 smithi112 ceph-mon[37501]: pgmap v8448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:49.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:49.215+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:03:49.218 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:49.215+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:03:49.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:03:49.215+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:03:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:50 smithi107 ceph-mon[32659]: pgmap v8449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:50 smithi132 ceph-mon[37580]: pgmap v8449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:50 smithi112 ceph-mon[37501]: pgmap v8449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15167.092679 secs 2024-07-26T17:03:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:52 smithi107 ceph-mon[32659]: pgmap v8450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15167.092679 secs 2024-07-26T17:03:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:52 smithi132 ceph-mon[37580]: pgmap v8450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15167.092679 secs 2024-07-26T17:03:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:52 smithi112 ceph-mon[37501]: pgmap v8450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:03:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:54 smithi107 ceph-mon[32659]: pgmap v8451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:54 smithi132 ceph-mon[37580]: pgmap v8451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:54.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:54 smithi112 ceph-mon[37501]: pgmap v8451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:03:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:56 smithi107 ceph-mon[32659]: pgmap v8452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:56 smithi132 ceph-mon[37580]: pgmap v8452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:56 smithi112 ceph-mon[37501]: pgmap v8452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15172.092800 secs 2024-07-26T17:03:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15172.092800 secs 2024-07-26T17:03:57.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15172.092800 secs 2024-07-26T17:03:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:03:58 smithi107 ceph-mon[32659]: pgmap v8453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:03:58 smithi132 ceph-mon[37580]: pgmap v8453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:03:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:03:58 smithi112 ceph-mon[37501]: pgmap v8453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:00.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:00 smithi107 ceph-mon[32659]: pgmap v8454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:00 smithi132 ceph-mon[37580]: pgmap v8454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:00 smithi112 ceph-mon[37501]: pgmap v8454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15177.092930 secs 2024-07-26T17:04:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:02 smithi107 ceph-mon[32659]: pgmap v8455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15177.092930 secs 2024-07-26T17:04:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:02 smithi132 ceph-mon[37580]: pgmap v8455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15177.092930 secs 2024-07-26T17:04:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:02 smithi112 ceph-mon[37501]: pgmap v8455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:04.232 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:04.230+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:04:04.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:04.230+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:04:04.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:04.230+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:04:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:04 smithi107 ceph-mon[32659]: pgmap v8456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:04 smithi132 ceph-mon[37580]: pgmap v8456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:04 smithi112 ceph-mon[37501]: pgmap v8456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:06 smithi107 ceph-mon[32659]: pgmap v8457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:06.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:06 smithi132 ceph-mon[37580]: pgmap v8457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:06 smithi112 ceph-mon[37501]: pgmap v8457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15182.093054 secs 2024-07-26T17:04:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15182.093054 secs 2024-07-26T17:04:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15182.093054 secs 2024-07-26T17:04:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:08 smithi107 ceph-mon[32659]: pgmap v8458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:08 smithi132 ceph-mon[37580]: pgmap v8458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:08 smithi112 ceph-mon[37501]: pgmap v8458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:10 smithi132 ceph-mon[37580]: pgmap v8459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:10 smithi112 ceph-mon[37501]: pgmap v8459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:10 smithi107 ceph-mon[32659]: pgmap v8459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15187.093173 secs 2024-07-26T17:04:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:12 smithi132 ceph-mon[37580]: pgmap v8460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15187.093173 secs 2024-07-26T17:04:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:12 smithi112 ceph-mon[37501]: pgmap v8460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15187.093173 secs 2024-07-26T17:04:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:12 smithi107 ceph-mon[32659]: pgmap v8460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:14 smithi112 ceph-mon[37501]: pgmap v8461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:04:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:14 smithi107 ceph-mon[32659]: pgmap v8461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:14.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:04:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:14 smithi132 ceph-mon[37580]: pgmap v8461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:04:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:04:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:04:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:04:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:15 smithi112 ceph-mon[37501]: pgmap v8462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:04:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:04:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:04:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:04:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:15 smithi107 ceph-mon[32659]: pgmap v8462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:04:15.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:04:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:04:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:04:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:15 smithi132 ceph-mon[37580]: pgmap v8462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:04:15.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:04:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15192.093239 secs 2024-07-26T17:04:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:17 smithi112 ceph-mon[37501]: pgmap v8463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:04:17.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15192.093239 secs 2024-07-26T17:04:17.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:17 smithi107 ceph-mon[32659]: pgmap v8463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:04:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15192.093239 secs 2024-07-26T17:04:17.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:17 smithi132 ceph-mon[37580]: pgmap v8463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:04:19.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:19.233+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:04:19.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:19.233+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:04:19.237 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:19.233+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:04:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:19 smithi107 ceph-mon[32659]: pgmap v8464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:04:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:19 smithi112 ceph-mon[37501]: pgmap v8464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:04:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:19 smithi132 ceph-mon[37580]: pgmap v8464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:04:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:21 smithi112 ceph-mon[37501]: pgmap v8465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:04:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15197.093381 secs 2024-07-26T17:04:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:21 smithi107 ceph-mon[32659]: pgmap v8465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:04:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15197.093381 secs 2024-07-26T17:04:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:21 smithi132 ceph-mon[37580]: pgmap v8465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:04:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15197.093381 secs 2024-07-26T17:04:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:23 smithi112 ceph-mon[37501]: pgmap v8466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:04:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:23 smithi107 ceph-mon[32659]: pgmap v8466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:04:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:23 smithi132 ceph-mon[37580]: pgmap v8466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:04:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:25 smithi112 ceph-mon[37501]: pgmap v8467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:04:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:25 smithi107 ceph-mon[32659]: pgmap v8467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:04:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:25 smithi132 ceph-mon[37580]: pgmap v8467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:04:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15202.093520 secs 2024-07-26T17:04:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15202.093520 secs 2024-07-26T17:04:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15202.093520 secs 2024-07-26T17:04:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:27 smithi112 ceph-mon[37501]: pgmap v8468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:04:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:27 smithi107 ceph-mon[32659]: pgmap v8468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:04:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:27 smithi132 ceph-mon[37580]: pgmap v8468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:04:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:29 smithi112 ceph-mon[37501]: pgmap v8469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:04:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:29 smithi107 ceph-mon[32659]: pgmap v8469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:04:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:29 smithi132 ceph-mon[37580]: pgmap v8469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:04:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:31 smithi112 ceph-mon[37501]: pgmap v8470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15207.093637 secs 2024-07-26T17:04:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:31 smithi107 ceph-mon[32659]: pgmap v8470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15207.093637 secs 2024-07-26T17:04:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:31 smithi132 ceph-mon[37580]: pgmap v8470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15207.093637 secs 2024-07-26T17:04:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:33 smithi112 ceph-mon[37501]: pgmap v8471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:33 smithi107 ceph-mon[32659]: pgmap v8471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:33 smithi132 ceph-mon[37580]: pgmap v8471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:34.248+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:04:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:34.248+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:04:34.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:34.248+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:04:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:35 smithi107 ceph-mon[32659]: pgmap v8472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:04:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:35 smithi132 ceph-mon[37580]: pgmap v8472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:04:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:35 smithi112 ceph-mon[37501]: pgmap v8472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:04:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15212.093735 secs 2024-07-26T17:04:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15212.093735 secs 2024-07-26T17:04:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15212.093735 secs 2024-07-26T17:04:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:37 smithi107 ceph-mon[32659]: pgmap v8473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:04:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:37 smithi132 ceph-mon[37580]: pgmap v8473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:04:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:37 smithi112 ceph-mon[37501]: pgmap v8473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:04:39.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:39 smithi107 ceph-mon[32659]: pgmap v8474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:04:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:39 smithi132 ceph-mon[37580]: pgmap v8474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:04:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:39 smithi112 ceph-mon[37501]: pgmap v8474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:04:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:41 smithi107 ceph-mon[32659]: pgmap v8475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15217.093853 secs 2024-07-26T17:04:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:41 smithi132 ceph-mon[37580]: pgmap v8475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15217.093853 secs 2024-07-26T17:04:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:41 smithi112 ceph-mon[37501]: pgmap v8475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15217.093853 secs 2024-07-26T17:04:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:43 smithi107 ceph-mon[32659]: pgmap v8476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:43 smithi132 ceph-mon[37580]: pgmap v8476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:43 smithi112 ceph-mon[37501]: pgmap v8476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:45 smithi107 ceph-mon[32659]: pgmap v8477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:04:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:45 smithi132 ceph-mon[37580]: pgmap v8477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:04:45.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:45 smithi112 ceph-mon[37501]: pgmap v8477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:04:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15222.093970 secs 2024-07-26T17:04:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15222.093970 secs 2024-07-26T17:04:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15222.093970 secs 2024-07-26T17:04:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:47 smithi107 ceph-mon[32659]: pgmap v8478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:04:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:47 smithi132 ceph-mon[37580]: pgmap v8478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:04:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:47 smithi112 ceph-mon[37501]: pgmap v8478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:04:49.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:49.264+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:04:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:49.264+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:04:49.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:04:49.264+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:04:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:49 smithi107 ceph-mon[32659]: pgmap v8479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:04:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:49 smithi132 ceph-mon[37580]: pgmap v8479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:04:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:49 smithi112 ceph-mon[37501]: pgmap v8479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:04:51.516 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:51 smithi112 ceph-mon[37501]: pgmap v8480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:51 smithi107 ceph-mon[32659]: pgmap v8480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15227.094029 secs 2024-07-26T17:04:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:51 smithi132 ceph-mon[37580]: pgmap v8480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:04:51.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15227.094029 secs 2024-07-26T17:04:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:53 smithi107 ceph-mon[32659]: pgmap v8481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:53 smithi132 ceph-mon[37580]: pgmap v8481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15227.094029 secs 2024-07-26T17:04:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:53 smithi112 ceph-mon[37501]: pgmap v8481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:55 smithi107 ceph-mon[32659]: pgmap v8482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:04:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:55 smithi132 ceph-mon[37580]: pgmap v8482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:04:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:55 smithi112 ceph-mon[37501]: pgmap v8482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:04:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15232.094159 secs 2024-07-26T17:04:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15232.094159 secs 2024-07-26T17:04:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15232.094159 secs 2024-07-26T17:04:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:57 smithi107 ceph-mon[32659]: pgmap v8483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:57 smithi132 ceph-mon[37580]: pgmap v8483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:57.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:57 smithi112 ceph-mon[37501]: pgmap v8483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:04:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:04:59 smithi132 ceph-mon[37580]: pgmap v8484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:04:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:04:59 smithi112 ceph-mon[37501]: pgmap v8484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:05:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:04:59 smithi107 ceph-mon[32659]: pgmap v8484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:05:01.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:01 smithi112 ceph-mon[37501]: pgmap v8485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:01.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15237.094279 secs 2024-07-26T17:05:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:01 smithi107 ceph-mon[32659]: pgmap v8485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15237.094279 secs 2024-07-26T17:05:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:01 smithi132 ceph-mon[37580]: pgmap v8485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15237.094279 secs 2024-07-26T17:05:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:03 smithi112 ceph-mon[37501]: pgmap v8486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:03 smithi107 ceph-mon[32659]: pgmap v8486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:03 smithi132 ceph-mon[37580]: pgmap v8486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:04.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:04.280+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:05:04.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:04.280+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:05:04.282 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:04.280+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:05:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:05 smithi112 ceph-mon[37501]: pgmap v8487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:05:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:05 smithi107 ceph-mon[32659]: pgmap v8487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:05:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:05 smithi132 ceph-mon[37580]: pgmap v8487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:05:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15242.094400 secs 2024-07-26T17:05:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15242.094400 secs 2024-07-26T17:05:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15242.094400 secs 2024-07-26T17:05:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:07 smithi112 ceph-mon[37501]: pgmap v8488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:07 smithi107 ceph-mon[32659]: pgmap v8488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:07 smithi132 ceph-mon[37580]: pgmap v8488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:09 smithi112 ceph-mon[37501]: pgmap v8489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:09 smithi107 ceph-mon[32659]: pgmap v8489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:09 smithi132 ceph-mon[37580]: pgmap v8489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:11 smithi112 ceph-mon[37501]: pgmap v8490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15247.094546 secs 2024-07-26T17:05:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:11 smithi107 ceph-mon[32659]: pgmap v8490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15247.094546 secs 2024-07-26T17:05:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:11 smithi132 ceph-mon[37580]: pgmap v8490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15247.094546 secs 2024-07-26T17:05:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:13 smithi112 ceph-mon[37501]: pgmap v8491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:14.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:13 smithi107 ceph-mon[32659]: pgmap v8491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:13 smithi132 ceph-mon[37580]: pgmap v8491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:05:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:05:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:05:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:15 smithi112 ceph-mon[37501]: pgmap v8492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:05:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:05:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:05:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:05:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:15 smithi107 ceph-mon[32659]: pgmap v8492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:05:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:05:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:05:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:05:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:15 smithi132 ceph-mon[37580]: pgmap v8492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:05:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:05:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:05:16.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:05:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15252.094671 secs 2024-07-26T17:05:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:16 smithi112 ceph-mon[37501]: pgmap v8493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:05:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:16 smithi112 ceph-mon[37501]: pgmap v8494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:05:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15252.094671 secs 2024-07-26T17:05:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:16 smithi107 ceph-mon[32659]: pgmap v8493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:05:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:16 smithi107 ceph-mon[32659]: pgmap v8494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:05:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15252.094671 secs 2024-07-26T17:05:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:16 smithi132 ceph-mon[37580]: pgmap v8493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:05:17.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:16 smithi132 ceph-mon[37580]: pgmap v8494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:05:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:18 smithi112 ceph-mon[37501]: pgmap v8495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:05:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:18 smithi107 ceph-mon[32659]: pgmap v8495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:05:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:18 smithi132 ceph-mon[37580]: pgmap v8495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:05:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:19.296+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:05:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:19.296+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:05:19.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:19.296+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:05:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:20 smithi107 ceph-mon[32659]: pgmap v8496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:05:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:20 smithi132 ceph-mon[37580]: pgmap v8496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:05:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:20 smithi112 ceph-mon[37501]: pgmap v8496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:05:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15257.094794 secs 2024-07-26T17:05:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15257.094794 secs 2024-07-26T17:05:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15257.094794 secs 2024-07-26T17:05:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:22 smithi107 ceph-mon[32659]: pgmap v8497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:05:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:22 smithi132 ceph-mon[37580]: pgmap v8497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:05:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:22 smithi112 ceph-mon[37501]: pgmap v8497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:05:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:24 smithi107 ceph-mon[32659]: pgmap v8498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:05:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:24 smithi132 ceph-mon[37580]: pgmap v8498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:05:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:24 smithi112 ceph-mon[37501]: pgmap v8498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:05:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15262.094914 secs 2024-07-26T17:05:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:26 smithi107 ceph-mon[32659]: pgmap v8499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:05:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15262.094914 secs 2024-07-26T17:05:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:26 smithi132 ceph-mon[37580]: pgmap v8499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:05:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15262.094914 secs 2024-07-26T17:05:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:26 smithi112 ceph-mon[37501]: pgmap v8499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:05:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:28 smithi107 ceph-mon[32659]: pgmap v8500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:29.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:28 smithi132 ceph-mon[37580]: pgmap v8500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:28 smithi112 ceph-mon[37501]: pgmap v8500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:30 smithi132 ceph-mon[37580]: pgmap v8501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:05:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:30 smithi112 ceph-mon[37501]: pgmap v8501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:05:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:30 smithi107 ceph-mon[32659]: pgmap v8501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:05:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15267.095085 secs 2024-07-26T17:05:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15267.095085 secs 2024-07-26T17:05:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15267.095085 secs 2024-07-26T17:05:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:32 smithi112 ceph-mon[37501]: pgmap v8502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:32 smithi107 ceph-mon[32659]: pgmap v8502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:32 smithi132 ceph-mon[37580]: pgmap v8502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:34.313 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:34.311+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:05:34.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:34.311+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:05:34.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:34.311+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:05:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:34 smithi112 ceph-mon[37501]: pgmap v8503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:34 smithi107 ceph-mon[32659]: pgmap v8503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:34 smithi132 ceph-mon[37580]: pgmap v8503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15272.095159 secs 2024-07-26T17:05:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:36 smithi112 ceph-mon[37501]: pgmap v8504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15272.095159 secs 2024-07-26T17:05:37.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:36 smithi107 ceph-mon[32659]: pgmap v8504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15272.095159 secs 2024-07-26T17:05:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:36 smithi132 ceph-mon[37580]: pgmap v8504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:39.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:38 smithi112 ceph-mon[37501]: pgmap v8505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:38 smithi107 ceph-mon[32659]: pgmap v8505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:38 smithi132 ceph-mon[37580]: pgmap v8505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:41.095 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:40 smithi107 ceph-mon[32659]: pgmap v8506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:05:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:40 smithi112 ceph-mon[37501]: pgmap v8506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:05:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:40 smithi132 ceph-mon[37580]: pgmap v8506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:05:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15277.095274 secs 2024-07-26T17:05:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15277.095274 secs 2024-07-26T17:05:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15277.095274 secs 2024-07-26T17:05:43.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:42 smithi112 ceph-mon[37501]: pgmap v8507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:43.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:42 smithi107 ceph-mon[32659]: pgmap v8507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:42 smithi132 ceph-mon[37580]: pgmap v8507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:45.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:44 smithi112 ceph-mon[37501]: pgmap v8508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:44 smithi107 ceph-mon[32659]: pgmap v8508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:44 smithi132 ceph-mon[37580]: pgmap v8508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15282.095394 secs 2024-07-26T17:05:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:46 smithi112 ceph-mon[37501]: pgmap v8509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15282.095394 secs 2024-07-26T17:05:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:46 smithi107 ceph-mon[32659]: pgmap v8509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15282.095394 secs 2024-07-26T17:05:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:46 smithi132 ceph-mon[37580]: pgmap v8509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:49.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:48 smithi112 ceph-mon[37501]: pgmap v8510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:48 smithi107 ceph-mon[32659]: pgmap v8510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:48 smithi132 ceph-mon[37580]: pgmap v8510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:49.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:49.326+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:05:49.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:49.326+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:05:49.329 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:05:49.326+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:05:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:50 smithi112 ceph-mon[37501]: pgmap v8511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:05:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:50 smithi107 ceph-mon[32659]: pgmap v8511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:05:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:50 smithi132 ceph-mon[37580]: pgmap v8511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:05:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15287.095540 secs 2024-07-26T17:05:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15287.095540 secs 2024-07-26T17:05:52.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15287.095540 secs 2024-07-26T17:05:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:52 smithi112 ceph-mon[37501]: pgmap v8512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:52 smithi107 ceph-mon[32659]: pgmap v8512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:52 smithi132 ceph-mon[37580]: pgmap v8512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:05:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:54 smithi112 ceph-mon[37501]: pgmap v8513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:54 smithi107 ceph-mon[32659]: pgmap v8513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:54 smithi132 ceph-mon[37580]: pgmap v8513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15292.095622 secs 2024-07-26T17:05:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:56 smithi112 ceph-mon[37501]: pgmap v8514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15292.095622 secs 2024-07-26T17:05:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:56 smithi107 ceph-mon[32659]: pgmap v8514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15292.095622 secs 2024-07-26T17:05:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:56 smithi132 ceph-mon[37580]: pgmap v8514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:05:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:05:58 smithi112 ceph-mon[37501]: pgmap v8515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:05:58 smithi107 ceph-mon[32659]: pgmap v8515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:05:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:05:58 smithi132 ceph-mon[37580]: pgmap v8515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:00 smithi112 ceph-mon[37501]: pgmap v8516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:06:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:00 smithi107 ceph-mon[32659]: pgmap v8516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:06:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:00 smithi132 ceph-mon[37580]: pgmap v8516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:06:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15297.095758 secs 2024-07-26T17:06:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15297.095758 secs 2024-07-26T17:06:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15297.095758 secs 2024-07-26T17:06:03.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:02 smithi112 ceph-mon[37501]: pgmap v8517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:02 smithi107 ceph-mon[32659]: pgmap v8517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:02 smithi132 ceph-mon[37580]: pgmap v8517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:04.344 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:04.343+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:06:04.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:04.343+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:06:04.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:04.343+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:06:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:04 smithi112 ceph-mon[37501]: pgmap v8518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:04 smithi107 ceph-mon[32659]: pgmap v8518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:04 smithi132 ceph-mon[37580]: pgmap v8518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15302.095877 secs 2024-07-26T17:06:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:06 smithi112 ceph-mon[37501]: pgmap v8519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15302.095877 secs 2024-07-26T17:06:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:06 smithi107 ceph-mon[32659]: pgmap v8519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15302.095877 secs 2024-07-26T17:06:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:06 smithi132 ceph-mon[37580]: pgmap v8519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:08 smithi112 ceph-mon[37501]: pgmap v8520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:08 smithi107 ceph-mon[32659]: pgmap v8520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:08 smithi132 ceph-mon[37580]: pgmap v8520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:10 smithi112 ceph-mon[37501]: pgmap v8521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:06:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:10 smithi107 ceph-mon[32659]: pgmap v8521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:06:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:10 smithi132 ceph-mon[37580]: pgmap v8521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:06:11.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:11.686+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T17:06:11.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:11.686+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T17:06:11.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:11.686+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8079040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T17:06:11.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:11.686+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8079040 comp rx=0 tx=0).stop 2024-07-26T17:06:11.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:11.686+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T17:06:11.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:11.686+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T17:06:11.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:11.686+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T17:06:11.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:11.686+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T17:06:11.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:11.687+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T17:06:11.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:11.689+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f8067040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T17:06:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15307.095943 secs 2024-07-26T17:06:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15307.095943 secs 2024-07-26T17:06:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15307.095943 secs 2024-07-26T17:06:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:12 smithi107 ceph-mon[32659]: pgmap v8522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:12 smithi132 ceph-mon[37580]: pgmap v8522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:12 smithi112 ceph-mon[37501]: pgmap v8522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:15.278 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:15.276+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:06:15.278 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:14 smithi107 ceph-mon[32659]: pgmap v8523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:15.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:15.277+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:06:15.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:15.277+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:06:15.279 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:15.278+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:06:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:14 smithi132 ceph-mon[37580]: pgmap v8523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:14 smithi112 ceph-mon[37501]: pgmap v8523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:06:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:06:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:06:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15312.096106 secs 2024-07-26T17:06:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:16 smithi107 ceph-mon[32659]: pgmap v8524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:06:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:06:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:06:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15312.096106 secs 2024-07-26T17:06:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:16 smithi132 ceph-mon[37580]: pgmap v8524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:17.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:06:17.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:06:17.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:06:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15312.096106 secs 2024-07-26T17:06:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:16 smithi112 ceph-mon[37501]: pgmap v8524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:06:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:06:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:06:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:17 smithi107 ceph-mon[32659]: pgmap v8525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:06:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:17 smithi107 ceph-mon[32659]: pgmap v8526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:17 smithi132 ceph-mon[37580]: pgmap v8525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:06:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:17 smithi132 ceph-mon[37580]: pgmap v8526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:17 smithi112 ceph-mon[37501]: pgmap v8525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:06:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:17 smithi112 ceph-mon[37501]: pgmap v8526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:19.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:19.351+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:06:19.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:19.351+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:06:19.353 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:19.351+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:06:20.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:19 smithi107 ceph-mon[32659]: pgmap v8527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:06:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:19 smithi132 ceph-mon[37580]: pgmap v8527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:06:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:19 smithi112 ceph-mon[37501]: pgmap v8527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:06:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15317.096228 secs 2024-07-26T17:06:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:21 smithi107 ceph-mon[32659]: pgmap v8528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:06:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15317.096228 secs 2024-07-26T17:06:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:21 smithi132 ceph-mon[37580]: pgmap v8528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:06:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15317.096228 secs 2024-07-26T17:06:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:21 smithi112 ceph-mon[37501]: pgmap v8528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:06:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:24 smithi107 ceph-mon[32659]: pgmap v8529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:06:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:24 smithi132 ceph-mon[37580]: pgmap v8529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:06:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:24 smithi112 ceph-mon[37501]: pgmap v8529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:06:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:26 smithi107 ceph-mon[32659]: pgmap v8530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:06:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:26 smithi132 ceph-mon[37580]: pgmap v8530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:06:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:26 smithi112 ceph-mon[37501]: pgmap v8530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:06:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15322.096311 secs 2024-07-26T17:06:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15322.096311 secs 2024-07-26T17:06:27.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15322.096311 secs 2024-07-26T17:06:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:28 smithi107 ceph-mon[32659]: pgmap v8531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:06:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:28 smithi132 ceph-mon[37580]: pgmap v8531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:06:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:28 smithi112 ceph-mon[37501]: pgmap v8531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:06:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:30 smithi107 ceph-mon[32659]: pgmap v8532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:06:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:30 smithi132 ceph-mon[37580]: pgmap v8532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:06:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:30 smithi112 ceph-mon[37501]: pgmap v8532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:06:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:32 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15327.096373 secs 2024-07-26T17:06:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:32 smithi107 ceph-mon[32659]: pgmap v8533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:06:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:32 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15327.096373 secs 2024-07-26T17:06:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:32 smithi132 ceph-mon[37580]: pgmap v8533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:06:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:32 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15327.096373 secs 2024-07-26T17:06:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:32 smithi112 ceph-mon[37501]: pgmap v8533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:06:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:34 smithi132 ceph-mon[37580]: pgmap v8534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:06:34.368 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:34.366+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:06:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:34.366+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:06:34.369 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:34.366+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:06:34.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:34 smithi107 ceph-mon[32659]: pgmap v8534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:06:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:34 smithi112 ceph-mon[37501]: pgmap v8534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:06:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:36 smithi132 ceph-mon[37580]: pgmap v8535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:36 smithi112 ceph-mon[37501]: pgmap v8535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:36 smithi107 ceph-mon[32659]: pgmap v8535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15332.096473 secs 2024-07-26T17:06:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15332.096473 secs 2024-07-26T17:06:37.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15332.096473 secs 2024-07-26T17:06:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:38 smithi112 ceph-mon[37501]: pgmap v8536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:38 smithi107 ceph-mon[32659]: pgmap v8536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:38 smithi132 ceph-mon[37580]: pgmap v8536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:40 smithi112 ceph-mon[37501]: pgmap v8537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:40 smithi107 ceph-mon[32659]: pgmap v8537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:40 smithi132 ceph-mon[37580]: pgmap v8537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15337.096593 secs 2024-07-26T17:06:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:42 smithi112 ceph-mon[37501]: pgmap v8538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15337.096593 secs 2024-07-26T17:06:42.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:42 smithi107 ceph-mon[32659]: pgmap v8538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15337.096593 secs 2024-07-26T17:06:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:42 smithi132 ceph-mon[37580]: pgmap v8538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:44 smithi112 ceph-mon[37501]: pgmap v8539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:44 smithi107 ceph-mon[32659]: pgmap v8539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:44 smithi132 ceph-mon[37580]: pgmap v8539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:46 smithi112 ceph-mon[37501]: pgmap v8540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:46 smithi107 ceph-mon[32659]: pgmap v8540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:46 smithi132 ceph-mon[37580]: pgmap v8540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15342.096734 secs 2024-07-26T17:06:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15342.096734 secs 2024-07-26T17:06:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15342.096734 secs 2024-07-26T17:06:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:48 smithi112 ceph-mon[37501]: pgmap v8541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:48 smithi107 ceph-mon[32659]: pgmap v8541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:48 smithi132 ceph-mon[37580]: pgmap v8541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:49.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:49.381+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:06:49.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:49.381+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:06:49.384 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:49.381+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:06:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:50 smithi112 ceph-mon[37501]: pgmap v8542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:50 smithi107 ceph-mon[32659]: pgmap v8542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:50 smithi132 ceph-mon[37580]: pgmap v8542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15347.096871 secs 2024-07-26T17:06:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:52 smithi112 ceph-mon[37501]: pgmap v8543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15347.096871 secs 2024-07-26T17:06:52.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:52 smithi107 ceph-mon[32659]: pgmap v8543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15347.096871 secs 2024-07-26T17:06:52.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:52 smithi132 ceph-mon[37580]: pgmap v8543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:06:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:54 smithi112 ceph-mon[37501]: pgmap v8544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:54 smithi107 ceph-mon[32659]: pgmap v8544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:54 smithi132 ceph-mon[37580]: pgmap v8544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:06:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:56 smithi112 ceph-mon[37501]: pgmap v8545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:56 smithi107 ceph-mon[32659]: pgmap v8545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:56 smithi132 ceph-mon[37580]: pgmap v8545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15352.096988 secs 2024-07-26T17:06:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15352.096988 secs 2024-07-26T17:06:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15352.096988 secs 2024-07-26T17:06:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:06:58 smithi112 ceph-mon[37501]: pgmap v8546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:06:58 smithi107 ceph-mon[32659]: pgmap v8546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:06:58 smithi132 ceph-mon[37580]: pgmap v8546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:06:59.285 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:59.284+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:06:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:59.284+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:06:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:59.284+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:06:59.286 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:06:59.284+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:07:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:00 smithi112 ceph-mon[37501]: pgmap v8547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:00 smithi107 ceph-mon[32659]: pgmap v8547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:00 smithi132 ceph-mon[37580]: pgmap v8547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15357.097108 secs 2024-07-26T17:07:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:02 smithi112 ceph-mon[37501]: pgmap v8548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15357.097108 secs 2024-07-26T17:07:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:02 smithi107 ceph-mon[32659]: pgmap v8548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15357.097108 secs 2024-07-26T17:07:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:02 smithi132 ceph-mon[37580]: pgmap v8548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:03 smithi112 ceph-mon[37501]: pgmap v8549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:03 smithi107 ceph-mon[32659]: pgmap v8549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:03 smithi132 ceph-mon[37580]: pgmap v8549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:04.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:04.397+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:07:04.399 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:04.398+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:07:04.400 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:04.398+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:07:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:05 smithi112 ceph-mon[37501]: pgmap v8550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:05 smithi107 ceph-mon[32659]: pgmap v8550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:05 smithi132 ceph-mon[37580]: pgmap v8550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15362.097200 secs 2024-07-26T17:07:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15362.097200 secs 2024-07-26T17:07:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15362.097200 secs 2024-07-26T17:07:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:07 smithi112 ceph-mon[37501]: pgmap v8551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:08.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:07 smithi107 ceph-mon[32659]: pgmap v8551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:07 smithi132 ceph-mon[37580]: pgmap v8551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:09 smithi112 ceph-mon[37501]: pgmap v8552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:09 smithi107 ceph-mon[32659]: pgmap v8552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:09 smithi132 ceph-mon[37580]: pgmap v8552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15367.097322 secs 2024-07-26T17:07:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:11 smithi112 ceph-mon[37501]: pgmap v8553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15367.097322 secs 2024-07-26T17:07:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:11 smithi107 ceph-mon[32659]: pgmap v8553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15367.097322 secs 2024-07-26T17:07:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:11 smithi132 ceph-mon[37580]: pgmap v8553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:13 smithi112 ceph-mon[37501]: pgmap v8554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:13 smithi107 ceph-mon[32659]: pgmap v8554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:13 smithi132 ceph-mon[37580]: pgmap v8554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:15 smithi112 ceph-mon[37501]: pgmap v8555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:15 smithi107 ceph-mon[32659]: pgmap v8555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:15 smithi132 ceph-mon[37580]: pgmap v8555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15372.097419 secs 2024-07-26T17:07:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15372.097419 secs 2024-07-26T17:07:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15372.097419 secs 2024-07-26T17:07:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:17 smithi112 ceph-mon[37501]: pgmap v8556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:18.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:07:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:17 smithi107 ceph-mon[32659]: pgmap v8556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:07:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:17 smithi132 ceph-mon[37580]: pgmap v8556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:07:19.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:07:19.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:07:19.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:18 smithi107 ceph-mon[32659]: pgmap v8557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:19.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:18 smithi107 ceph-mon[32659]: pgmap v8558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:07:19.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:07:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:07:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:07:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:18 smithi112 ceph-mon[37501]: pgmap v8557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:18 smithi112 ceph-mon[37501]: pgmap v8558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:07:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:07:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:07:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:07:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:18 smithi132 ceph-mon[37580]: pgmap v8557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:18 smithi132 ceph-mon[37580]: pgmap v8558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:07:19.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:07:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:19.413+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:07:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:19.413+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:07:19.415 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:19.413+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:07:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:20 smithi107 ceph-mon[32659]: pgmap v8559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:07:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:20 smithi132 ceph-mon[37580]: pgmap v8559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:07:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:20 smithi112 ceph-mon[37501]: pgmap v8559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:07:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15377.097472 secs 2024-07-26T17:07:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15377.097472 secs 2024-07-26T17:07:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15377.097472 secs 2024-07-26T17:07:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:23 smithi107 ceph-mon[32659]: pgmap v8560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:23 smithi132 ceph-mon[37580]: pgmap v8560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:23 smithi112 ceph-mon[37501]: pgmap v8560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:25 smithi107 ceph-mon[32659]: pgmap v8561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:07:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:25 smithi132 ceph-mon[37580]: pgmap v8561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:07:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:25 smithi112 ceph-mon[37501]: pgmap v8561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:07:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15382.097597 secs 2024-07-26T17:07:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:27 smithi107 ceph-mon[32659]: pgmap v8562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:07:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15382.097597 secs 2024-07-26T17:07:27.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:27 smithi132 ceph-mon[37580]: pgmap v8562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:07:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15382.097597 secs 2024-07-26T17:07:27.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:27 smithi112 ceph-mon[37501]: pgmap v8562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:07:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:29 smithi107 ceph-mon[32659]: pgmap v8563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:07:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:29 smithi132 ceph-mon[37580]: pgmap v8563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:07:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:29 smithi112 ceph-mon[37501]: pgmap v8563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:07:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:31 smithi107 ceph-mon[32659]: pgmap v8564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:31 smithi132 ceph-mon[37580]: pgmap v8564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:31 smithi112 ceph-mon[37501]: pgmap v8564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:32 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15387.097678 secs 2024-07-26T17:07:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:32 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15387.097678 secs 2024-07-26T17:07:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:32 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15387.097678 secs 2024-07-26T17:07:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:33 smithi132 ceph-mon[37580]: pgmap v8565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:33 smithi112 ceph-mon[37501]: pgmap v8565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:33 smithi107 ceph-mon[32659]: pgmap v8565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:34.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:34.427+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:07:34.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:34.427+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:07:34.430 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:34.427+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:07:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:35 smithi132 ceph-mon[37580]: pgmap v8566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:35 smithi112 ceph-mon[37501]: pgmap v8566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:35 smithi107 ceph-mon[32659]: pgmap v8566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15392.097777 secs 2024-07-26T17:07:37.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:37 smithi132 ceph-mon[37580]: pgmap v8567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15392.097777 secs 2024-07-26T17:07:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:37 smithi112 ceph-mon[37501]: pgmap v8567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:37.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15392.097777 secs 2024-07-26T17:07:37.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:37 smithi107 ceph-mon[32659]: pgmap v8567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:39 smithi112 ceph-mon[37501]: pgmap v8568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:39 smithi107 ceph-mon[32659]: pgmap v8568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:39 smithi132 ceph-mon[37580]: pgmap v8568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:41 smithi112 ceph-mon[37501]: pgmap v8569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:41 smithi107 ceph-mon[32659]: pgmap v8569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:41 smithi132 ceph-mon[37580]: pgmap v8569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15397.097877 secs 2024-07-26T17:07:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15397.097877 secs 2024-07-26T17:07:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15397.097877 secs 2024-07-26T17:07:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:43 smithi112 ceph-mon[37501]: pgmap v8570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:43 smithi107 ceph-mon[32659]: pgmap v8570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:43 smithi132 ceph-mon[37580]: pgmap v8570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:45 smithi112 ceph-mon[37501]: pgmap v8571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:45 smithi107 ceph-mon[32659]: pgmap v8571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:45 smithi132 ceph-mon[37580]: pgmap v8571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:07:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:47 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15402.097978 secs 2024-07-26T17:07:47.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:47 smithi112 ceph-mon[37501]: pgmap v8572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:47 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15402.097978 secs 2024-07-26T17:07:47.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:47 smithi107 ceph-mon[32659]: pgmap v8572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:47 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15402.097978 secs 2024-07-26T17:07:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:47 smithi132 ceph-mon[37580]: pgmap v8572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:07:49.435 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:49.433+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:07:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:49.433+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:07:49.436 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:07:49.433+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:07:49.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:49 smithi107 ceph-mon[32659]: pgmap v8573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:49 smithi112 ceph-mon[37501]: pgmap v8573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:49 smithi132 ceph-mon[37580]: pgmap v8573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:51 smithi112 ceph-mon[37501]: pgmap v8574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:51 smithi107 ceph-mon[32659]: pgmap v8574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:51 smithi132 ceph-mon[37580]: pgmap v8574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15407.098084 secs 2024-07-26T17:07:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15407.098084 secs 2024-07-26T17:07:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15407.098084 secs 2024-07-26T17:07:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:53 smithi112 ceph-mon[37501]: pgmap v8575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:53 smithi107 ceph-mon[32659]: pgmap v8575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:53 smithi132 ceph-mon[37580]: pgmap v8575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:07:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:55 smithi112 ceph-mon[37501]: pgmap v8576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:55 smithi107 ceph-mon[32659]: pgmap v8576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:55 smithi132 ceph-mon[37580]: pgmap v8576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:57 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15412.098190 secs 2024-07-26T17:07:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:57 smithi112 ceph-mon[37501]: pgmap v8577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:07:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:57 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15412.098190 secs 2024-07-26T17:07:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:57 smithi107 ceph-mon[32659]: pgmap v8577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:07:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:57 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15412.098190 secs 2024-07-26T17:07:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:57 smithi132 ceph-mon[37580]: pgmap v8577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:07:59.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:07:59 smithi112 ceph-mon[37501]: pgmap v8578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:07:59 smithi107 ceph-mon[32659]: pgmap v8578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:07:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:07:59 smithi132 ceph-mon[37580]: pgmap v8578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:08:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:01 smithi112 ceph-mon[37501]: pgmap v8579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:01 smithi107 ceph-mon[32659]: pgmap v8579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:01 smithi132 ceph-mon[37580]: pgmap v8579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:02 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15417.098258 secs 2024-07-26T17:08:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:02 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15417.098258 secs 2024-07-26T17:08:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:02 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15417.098258 secs 2024-07-26T17:08:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:03 smithi112 ceph-mon[37501]: pgmap v8580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:03 smithi107 ceph-mon[32659]: pgmap v8580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:03 smithi132 ceph-mon[37580]: pgmap v8580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:04.449+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:08:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:04.449+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:08:04.451 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:04.449+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:08:05.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:04 smithi112 ceph-mon[37501]: pgmap v8581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:04 smithi107 ceph-mon[32659]: pgmap v8581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:04 smithi132 ceph-mon[37580]: pgmap v8581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15422.098333 secs 2024-07-26T17:08:07.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:07 smithi107 ceph-mon[32659]: pgmap v8582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15422.098333 secs 2024-07-26T17:08:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:07 smithi132 ceph-mon[37580]: pgmap v8582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15422.098333 secs 2024-07-26T17:08:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:07 smithi112 ceph-mon[37501]: pgmap v8582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:09 smithi107 ceph-mon[32659]: pgmap v8583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:09 smithi132 ceph-mon[37580]: pgmap v8583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:09.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:09 smithi112 ceph-mon[37501]: pgmap v8583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:11 smithi107 ceph-mon[32659]: pgmap v8584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:11 smithi132 ceph-mon[37580]: pgmap v8584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:11 smithi112 ceph-mon[37501]: pgmap v8584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15427.098392 secs 2024-07-26T17:08:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15427.098392 secs 2024-07-26T17:08:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15427.098392 secs 2024-07-26T17:08:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:13 smithi107 ceph-mon[32659]: pgmap v8585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:13 smithi132 ceph-mon[37580]: pgmap v8585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:13 smithi112 ceph-mon[37501]: pgmap v8585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:15 smithi132 ceph-mon[37580]: pgmap v8586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:15 smithi112 ceph-mon[37501]: pgmap v8586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:15 smithi107 ceph-mon[32659]: pgmap v8586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15432.098486 secs 2024-07-26T17:08:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15432.098486 secs 2024-07-26T17:08:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15432.098486 secs 2024-07-26T17:08:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:17 smithi112 ceph-mon[37501]: pgmap v8587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:17 smithi107 ceph-mon[32659]: pgmap v8587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:17 smithi132 ceph-mon[37580]: pgmap v8587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:19 smithi112 ceph-mon[37501]: pgmap v8588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:08:19.466 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:19.465+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:08:19.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:19.465+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:08:19.467 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:19.465+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:08:19.467 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:19 smithi107 ceph-mon[32659]: pgmap v8588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:19.468 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:08:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:19 smithi132 ceph-mon[37580]: pgmap v8588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:19.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:08:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:08:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:08:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:20 smithi112 ceph-mon[37501]: pgmap v8589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:08:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:08:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:08:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:08:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:20 smithi107 ceph-mon[32659]: pgmap v8589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:08:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:08:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:08:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:08:20.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:20 smithi132 ceph-mon[37580]: pgmap v8589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:08:20.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:08:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15437.098613 secs 2024-07-26T17:08:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15437.098613 secs 2024-07-26T17:08:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15437.098613 secs 2024-07-26T17:08:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:22 smithi112 ceph-mon[37501]: pgmap v8590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:08:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:22 smithi107 ceph-mon[32659]: pgmap v8590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:08:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:22 smithi132 ceph-mon[37580]: pgmap v8590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:08:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:23 smithi107 ceph-mon[32659]: pgmap v8591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:08:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:23 smithi132 ceph-mon[37580]: pgmap v8591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:08:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:23 smithi112 ceph-mon[37501]: pgmap v8591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:08:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:25 smithi107 ceph-mon[32659]: pgmap v8592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:08:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:25 smithi132 ceph-mon[37580]: pgmap v8592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:08:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:25 smithi112 ceph-mon[37501]: pgmap v8592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:08:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15442.098739 secs 2024-07-26T17:08:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15442.098739 secs 2024-07-26T17:08:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15442.098739 secs 2024-07-26T17:08:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:27 smithi107 ceph-mon[32659]: pgmap v8593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:08:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:27 smithi132 ceph-mon[37580]: pgmap v8593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:08:27.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:27 smithi112 ceph-mon[37501]: pgmap v8593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:08:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:29 smithi107 ceph-mon[32659]: pgmap v8594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:08:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:29 smithi132 ceph-mon[37580]: pgmap v8594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:08:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:29 smithi112 ceph-mon[37501]: pgmap v8594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:08:30.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15447.098854 secs 2024-07-26T17:08:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15447.098854 secs 2024-07-26T17:08:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15447.098854 secs 2024-07-26T17:08:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:31 smithi107 ceph-mon[32659]: pgmap v8595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:31 smithi132 ceph-mon[37580]: pgmap v8595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:31 smithi112 ceph-mon[37501]: pgmap v8595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:33 smithi107 ceph-mon[32659]: pgmap v8596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:33 smithi132 ceph-mon[37580]: pgmap v8596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:33 smithi112 ceph-mon[37501]: pgmap v8596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:34.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:34.468+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:08:34.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:34.468+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:08:34.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:34.468+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:08:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:35 smithi107 ceph-mon[32659]: pgmap v8597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:35 smithi132 ceph-mon[37580]: pgmap v8597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:35 smithi112 ceph-mon[37501]: pgmap v8597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15452.098972 secs 2024-07-26T17:08:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15452.098972 secs 2024-07-26T17:08:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15452.098972 secs 2024-07-26T17:08:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:37 smithi107 ceph-mon[32659]: pgmap v8598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:37 smithi132 ceph-mon[37580]: pgmap v8598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:37 smithi112 ceph-mon[37501]: pgmap v8598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:39 smithi112 ceph-mon[37501]: pgmap v8599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:39 smithi107 ceph-mon[32659]: pgmap v8599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:40.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:39 smithi132 ceph-mon[37580]: pgmap v8599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15457.099101 secs 2024-07-26T17:08:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15457.099101 secs 2024-07-26T17:08:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15457.099101 secs 2024-07-26T17:08:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:41 smithi112 ceph-mon[37501]: pgmap v8600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:41 smithi107 ceph-mon[32659]: pgmap v8600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:41 smithi132 ceph-mon[37580]: pgmap v8600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:43 smithi107 ceph-mon[32659]: pgmap v8601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:43 smithi132 ceph-mon[37580]: pgmap v8601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:43 smithi112 ceph-mon[37501]: pgmap v8601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:45 smithi107 ceph-mon[32659]: pgmap v8602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:45 smithi132 ceph-mon[37580]: pgmap v8602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:45 smithi112 ceph-mon[37501]: pgmap v8602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15462.099225 secs 2024-07-26T17:08:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15462.099225 secs 2024-07-26T17:08:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15462.099225 secs 2024-07-26T17:08:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:47 smithi107 ceph-mon[32659]: pgmap v8603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:47 smithi132 ceph-mon[37580]: pgmap v8603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:47 smithi112 ceph-mon[37501]: pgmap v8603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:49.486 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:49.484+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:08:49.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:49.484+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:08:49.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:08:49.484+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:08:49.487 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:49 smithi107 ceph-mon[32659]: pgmap v8604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:49 smithi132 ceph-mon[37580]: pgmap v8604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:49 smithi112 ceph-mon[37501]: pgmap v8604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15467.099343 secs 2024-07-26T17:08:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15467.099343 secs 2024-07-26T17:08:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15467.099343 secs 2024-07-26T17:08:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:51 smithi107 ceph-mon[32659]: pgmap v8605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:51 smithi132 ceph-mon[37580]: pgmap v8605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:51 smithi112 ceph-mon[37501]: pgmap v8605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:08:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:53 smithi107 ceph-mon[32659]: pgmap v8606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:53 smithi132 ceph-mon[37580]: pgmap v8606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:53 smithi112 ceph-mon[37501]: pgmap v8606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:55 smithi107 ceph-mon[32659]: pgmap v8607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:55 smithi132 ceph-mon[37580]: pgmap v8607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:55 smithi112 ceph-mon[37501]: pgmap v8607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15472.099463 secs 2024-07-26T17:08:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15472.099463 secs 2024-07-26T17:08:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15472.099463 secs 2024-07-26T17:08:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:57 smithi107 ceph-mon[32659]: pgmap v8608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:57 smithi132 ceph-mon[37580]: pgmap v8608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:57.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:57 smithi112 ceph-mon[37501]: pgmap v8608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:08:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:08:59 smithi107 ceph-mon[32659]: pgmap v8609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:08:59 smithi132 ceph-mon[37580]: pgmap v8609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:08:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:08:59 smithi112 ceph-mon[37501]: pgmap v8609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15477.099530 secs 2024-07-26T17:09:00.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15477.099530 secs 2024-07-26T17:09:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15477.099530 secs 2024-07-26T17:09:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:01 smithi107 ceph-mon[32659]: pgmap v8610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:01 smithi132 ceph-mon[37580]: pgmap v8610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:01 smithi112 ceph-mon[37501]: pgmap v8610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:03 smithi107 ceph-mon[32659]: pgmap v8611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:03 smithi132 ceph-mon[37580]: pgmap v8611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:03 smithi112 ceph-mon[37501]: pgmap v8611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:04.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:04.498+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:09:04.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:04.498+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:09:04.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:04.498+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:09:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:05 smithi107 ceph-mon[32659]: pgmap v8612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:05 smithi132 ceph-mon[37580]: pgmap v8612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:05 smithi112 ceph-mon[37501]: pgmap v8612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15482.099674 secs 2024-07-26T17:09:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15482.099674 secs 2024-07-26T17:09:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15482.099674 secs 2024-07-26T17:09:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:08 smithi107 ceph-mon[32659]: pgmap v8613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:08 smithi132 ceph-mon[37580]: pgmap v8613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:08 smithi112 ceph-mon[37501]: pgmap v8613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:10 smithi107 ceph-mon[32659]: pgmap v8614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:10 smithi132 ceph-mon[37580]: pgmap v8614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:10 smithi112 ceph-mon[37501]: pgmap v8614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15487.099811 secs 2024-07-26T17:09:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15487.099811 secs 2024-07-26T17:09:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15487.099811 secs 2024-07-26T17:09:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:12 smithi107 ceph-mon[32659]: pgmap v8615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:12 smithi132 ceph-mon[37580]: pgmap v8615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:12 smithi112 ceph-mon[37501]: pgmap v8615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:14 smithi107 ceph-mon[32659]: pgmap v8616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:14.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:14 smithi132 ceph-mon[37580]: pgmap v8616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:14 smithi112 ceph-mon[37501]: pgmap v8616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:15.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:15.308+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:09:15.310 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:15.308+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:09:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:15.309+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:09:15.311 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:15.309+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:09:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:16 smithi107 ceph-mon[32659]: pgmap v8617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15492.099932 secs 2024-07-26T17:09:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:16 smithi132 ceph-mon[37580]: pgmap v8617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:16.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15492.099932 secs 2024-07-26T17:09:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:16 smithi112 ceph-mon[37501]: pgmap v8617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15492.099932 secs 2024-07-26T17:09:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:18 smithi107 ceph-mon[32659]: pgmap v8618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:18 smithi132 ceph-mon[37580]: pgmap v8618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:18 smithi112 ceph-mon[37501]: pgmap v8618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:19.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:19.513+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:09:19.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:19.513+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:09:19.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:19.513+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:09:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:09:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:09:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:09:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: pgmap v8619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:09:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:09:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:09:21.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15497.100040 secs 2024-07-26T17:09:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: pgmap v8619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:09:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:09:21.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:09:21.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15497.100040 secs 2024-07-26T17:09:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: pgmap v8619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:09:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:09:21.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:09:21.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:09:21.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:09:21.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15497.100040 secs 2024-07-26T17:09:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:21 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:09:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:21 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:09:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:21 smithi112 ceph-mon[37501]: pgmap v8620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:09:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:21 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:09:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:21 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:09:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:21 smithi107 ceph-mon[32659]: pgmap v8620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:09:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:21 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:09:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:21 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:09:22.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:21 smithi132 ceph-mon[37580]: pgmap v8620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:09:24.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:23 smithi112 ceph-mon[37501]: pgmap v8621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:09:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:23 smithi107 ceph-mon[32659]: pgmap v8621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:09:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:23 smithi132 ceph-mon[37580]: pgmap v8621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:09:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:25 smithi112 ceph-mon[37501]: pgmap v8622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:09:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15502.100186 secs 2024-07-26T17:09:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:25 smithi107 ceph-mon[32659]: pgmap v8622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:09:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15502.100186 secs 2024-07-26T17:09:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:25 smithi132 ceph-mon[37580]: pgmap v8622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:09:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15502.100186 secs 2024-07-26T17:09:28.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:27 smithi112 ceph-mon[37501]: pgmap v8623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:09:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:27 smithi107 ceph-mon[32659]: pgmap v8623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:09:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:27 smithi132 ceph-mon[37580]: pgmap v8623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:09:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:29 smithi112 ceph-mon[37501]: pgmap v8624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:09:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:29 smithi107 ceph-mon[32659]: pgmap v8624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:09:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:29 smithi132 ceph-mon[37580]: pgmap v8624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:09:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15507.100303 secs 2024-07-26T17:09:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15507.100303 secs 2024-07-26T17:09:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15507.100303 secs 2024-07-26T17:09:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:31 smithi112 ceph-mon[37501]: pgmap v8625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T17:09:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:31 smithi107 ceph-mon[32659]: pgmap v8625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T17:09:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:31 smithi132 ceph-mon[37580]: pgmap v8625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T17:09:34.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:33 smithi112 ceph-mon[37501]: pgmap v8626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:33 smithi107 ceph-mon[32659]: pgmap v8626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:33 smithi132 ceph-mon[37580]: pgmap v8626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:34.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:34.518+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:09:34.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:34.518+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:09:34.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:34.518+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:09:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:35 smithi112 ceph-mon[37501]: pgmap v8627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T17:09:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15512.100427 secs 2024-07-26T17:09:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:35 smithi107 ceph-mon[32659]: pgmap v8627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T17:09:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15512.100427 secs 2024-07-26T17:09:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:35 smithi132 ceph-mon[37580]: pgmap v8627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T17:09:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15512.100427 secs 2024-07-26T17:09:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:37 smithi107 ceph-mon[32659]: pgmap v8628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:09:38.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:37 smithi132 ceph-mon[37580]: pgmap v8628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:09:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:37 smithi112 ceph-mon[37501]: pgmap v8628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:09:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:39 smithi107 ceph-mon[32659]: pgmap v8629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:09:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:39 smithi132 ceph-mon[37580]: pgmap v8629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:09:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:39 smithi112 ceph-mon[37501]: pgmap v8629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:09:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15517.100531 secs 2024-07-26T17:09:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15517.100531 secs 2024-07-26T17:09:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15517.100531 secs 2024-07-26T17:09:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:41 smithi107 ceph-mon[32659]: pgmap v8630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:41 smithi132 ceph-mon[37580]: pgmap v8630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:41 smithi112 ceph-mon[37501]: pgmap v8630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:43 smithi107 ceph-mon[32659]: pgmap v8631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:43 smithi132 ceph-mon[37580]: pgmap v8631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:43 smithi112 ceph-mon[37501]: pgmap v8631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:45 smithi107 ceph-mon[32659]: pgmap v8632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:09:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15522.100630 secs 2024-07-26T17:09:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:45 smithi132 ceph-mon[37580]: pgmap v8632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:09:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15522.100630 secs 2024-07-26T17:09:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:45 smithi112 ceph-mon[37501]: pgmap v8632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:09:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15522.100630 secs 2024-07-26T17:09:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:47 smithi107 ceph-mon[32659]: pgmap v8633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:09:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:47 smithi132 ceph-mon[37580]: pgmap v8633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:09:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:47 smithi112 ceph-mon[37501]: pgmap v8633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:09:49.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:49.528+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:09:49.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:49.528+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:09:49.531 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:09:49.528+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:09:50.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:49 smithi107 ceph-mon[32659]: pgmap v8634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:09:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:49 smithi132 ceph-mon[37580]: pgmap v8634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:09:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:49 smithi112 ceph-mon[37501]: pgmap v8634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:09:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15527.100727 secs 2024-07-26T17:09:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15527.100727 secs 2024-07-26T17:09:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15527.100727 secs 2024-07-26T17:09:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:52 smithi107 ceph-mon[32659]: pgmap v8635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:52 smithi132 ceph-mon[37580]: pgmap v8635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:52 smithi112 ceph-mon[37501]: pgmap v8635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:09:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:54 smithi107 ceph-mon[32659]: pgmap v8636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:54 smithi132 ceph-mon[37580]: pgmap v8636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:54 smithi112 ceph-mon[37501]: pgmap v8636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:09:56.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:56 smithi107 ceph-mon[32659]: pgmap v8637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:09:56.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15532.100866 secs 2024-07-26T17:09:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:56 smithi132 ceph-mon[37580]: pgmap v8637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:09:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15532.100866 secs 2024-07-26T17:09:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:56 smithi112 ceph-mon[37501]: pgmap v8637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:09:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15532.100866 secs 2024-07-26T17:09:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:09:58 smithi107 ceph-mon[32659]: pgmap v8638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:09:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:09:58 smithi132 ceph-mon[37580]: pgmap v8638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:09:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:09:58 smithi112 ceph-mon[37501]: pgmap v8638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:10:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:00 smithi107 ceph-mon[32659]: pgmap v8639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:10:00.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:10:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:00 smithi132 ceph-mon[37580]: pgmap v8639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:10:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:10:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:00 smithi112 ceph-mon[37501]: pgmap v8639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:10:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:10:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15537.101006 secs 2024-07-26T17:10:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15537.101006 secs 2024-07-26T17:10:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15537.101006 secs 2024-07-26T17:10:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:02 smithi132 ceph-mon[37580]: pgmap v8640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:02 smithi112 ceph-mon[37501]: pgmap v8640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:02 smithi107 ceph-mon[32659]: pgmap v8640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:04 smithi132 ceph-mon[37580]: pgmap v8641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:04 smithi112 ceph-mon[37501]: pgmap v8641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:04.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:04 smithi107 ceph-mon[32659]: pgmap v8641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:04.545+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:10:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:04.545+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:10:04.546 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:04.545+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:10:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:06 smithi107 ceph-mon[32659]: pgmap v8642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:10:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15542.101144 secs 2024-07-26T17:10:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:06 smithi132 ceph-mon[37580]: pgmap v8642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:10:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15542.101144 secs 2024-07-26T17:10:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:06 smithi112 ceph-mon[37501]: pgmap v8642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:10:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15542.101144 secs 2024-07-26T17:10:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:08 smithi112 ceph-mon[37501]: pgmap v8643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:10:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:08 smithi107 ceph-mon[32659]: pgmap v8643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:10:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:08 smithi132 ceph-mon[37580]: pgmap v8643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:10:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:10 smithi132 ceph-mon[37580]: pgmap v8644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:10:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:10 smithi112 ceph-mon[37501]: pgmap v8644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:10:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:10 smithi107 ceph-mon[32659]: pgmap v8644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:10:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15547.101228 secs 2024-07-26T17:10:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15547.101228 secs 2024-07-26T17:10:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15547.101228 secs 2024-07-26T17:10:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:12 smithi112 ceph-mon[37501]: pgmap v8645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:12 smithi107 ceph-mon[32659]: pgmap v8645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:12 smithi132 ceph-mon[37580]: pgmap v8645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:14 smithi112 ceph-mon[37501]: pgmap v8646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:14 smithi107 ceph-mon[32659]: pgmap v8646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:14 smithi132 ceph-mon[37580]: pgmap v8646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:16 smithi112 ceph-mon[37501]: pgmap v8647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:10:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15552.101311 secs 2024-07-26T17:10:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:16 smithi107 ceph-mon[32659]: pgmap v8647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:10:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15552.101311 secs 2024-07-26T17:10:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:16 smithi132 ceph-mon[37580]: pgmap v8647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T17:10:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15552.101311 secs 2024-07-26T17:10:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:18 smithi112 ceph-mon[37501]: pgmap v8648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:10:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:18 smithi107 ceph-mon[32659]: pgmap v8648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:10:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:18 smithi132 ceph-mon[37580]: pgmap v8648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:10:19.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:19.551+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:10:19.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:19.551+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:10:19.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:19.551+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:10:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:20 smithi112 ceph-mon[37501]: pgmap v8649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:10:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:20 smithi107 ceph-mon[32659]: pgmap v8649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:10:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:20 smithi132 ceph-mon[37580]: pgmap v8649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:10:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15557.101432 secs 2024-07-26T17:10:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:10:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15557.101432 secs 2024-07-26T17:10:21.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:10:21.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15557.101432 secs 2024-07-26T17:10:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:10:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:21.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:10:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:22 smithi107 ceph-mon[32659]: pgmap v8650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:22 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:10:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:22 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:10:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:22 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:10:22.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:22 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:10:22.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:10:22.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:10:22.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:10:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:22 smithi112 ceph-mon[37501]: pgmap v8650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:22 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:10:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:22 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:10:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:22 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:10:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:22 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:10:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:10:22.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:10:22.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:10:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:22 smithi132 ceph-mon[37580]: pgmap v8650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:22 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:10:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:22 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:10:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:22 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:10:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:22 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:10:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:10:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:10:22.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:10:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:23 smithi112 ceph-mon[37501]: pgmap v8651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:10:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:23 smithi112 ceph-mon[37501]: pgmap v8652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:10:23.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:23 smithi107 ceph-mon[32659]: pgmap v8651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:10:23.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:23 smithi107 ceph-mon[32659]: pgmap v8652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:10:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:23 smithi132 ceph-mon[37580]: pgmap v8651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:10:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:23 smithi132 ceph-mon[37580]: pgmap v8652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:10:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:25 smithi112 ceph-mon[37501]: pgmap v8653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:10:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:25 smithi107 ceph-mon[32659]: pgmap v8653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:10:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:25 smithi132 ceph-mon[37580]: pgmap v8653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:10:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15562.101496 secs 2024-07-26T17:10:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15562.101496 secs 2024-07-26T17:10:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15562.101496 secs 2024-07-26T17:10:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:27 smithi112 ceph-mon[37501]: pgmap v8654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:10:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:27 smithi107 ceph-mon[32659]: pgmap v8654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:10:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:27 smithi132 ceph-mon[37580]: pgmap v8654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:10:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:29 smithi112 ceph-mon[37501]: pgmap v8655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:10:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:29 smithi107 ceph-mon[32659]: pgmap v8655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:10:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:29 smithi132 ceph-mon[37580]: pgmap v8655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:10:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:31 smithi112 ceph-mon[37501]: pgmap v8656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:10:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15567.101589 secs 2024-07-26T17:10:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:31 smithi107 ceph-mon[32659]: pgmap v8656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:10:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15567.101589 secs 2024-07-26T17:10:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:31 smithi132 ceph-mon[37580]: pgmap v8656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:10:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15567.101589 secs 2024-07-26T17:10:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:33 smithi112 ceph-mon[37501]: pgmap v8657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:10:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:33 smithi107 ceph-mon[32659]: pgmap v8657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:10:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:33 smithi132 ceph-mon[37580]: pgmap v8657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:10:34.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:34.557+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:10:34.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:34.557+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:10:34.559 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:34.557+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:10:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:35 smithi112 ceph-mon[37501]: pgmap v8658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:35 smithi107 ceph-mon[32659]: pgmap v8658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:35 smithi132 ceph-mon[37580]: pgmap v8658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15572.101715 secs 2024-07-26T17:10:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15572.101715 secs 2024-07-26T17:10:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15572.101715 secs 2024-07-26T17:10:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:37 smithi112 ceph-mon[37501]: pgmap v8659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:37 smithi107 ceph-mon[32659]: pgmap v8659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:37 smithi132 ceph-mon[37580]: pgmap v8659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:39 smithi112 ceph-mon[37501]: pgmap v8660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:39 smithi107 ceph-mon[32659]: pgmap v8660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:39 smithi132 ceph-mon[37580]: pgmap v8660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:41 smithi112 ceph-mon[37501]: pgmap v8661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:10:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15577.101836 secs 2024-07-26T17:10:41.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:41 smithi107 ceph-mon[32659]: pgmap v8661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:10:41.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15577.101836 secs 2024-07-26T17:10:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:41 smithi132 ceph-mon[37580]: pgmap v8661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:10:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15577.101836 secs 2024-07-26T17:10:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:42 smithi107 ceph-mon[32659]: pgmap v8662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:42 smithi132 ceph-mon[37580]: pgmap v8662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:42 smithi112 ceph-mon[37501]: pgmap v8662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:44 smithi112 ceph-mon[37501]: pgmap v8663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:44 smithi107 ceph-mon[32659]: pgmap v8663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:44 smithi132 ceph-mon[37580]: pgmap v8663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15582.101948 secs 2024-07-26T17:10:46.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15582.101948 secs 2024-07-26T17:10:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15582.101948 secs 2024-07-26T17:10:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:46 smithi112 ceph-mon[37501]: pgmap v8664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:46 smithi107 ceph-mon[32659]: pgmap v8664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:46 smithi132 ceph-mon[37580]: pgmap v8664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:48 smithi112 ceph-mon[37501]: pgmap v8665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:48 smithi107 ceph-mon[32659]: pgmap v8665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:48 smithi132 ceph-mon[37580]: pgmap v8665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:49.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:49.572+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:10:49.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:49.572+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:10:49.575 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:10:49.572+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:10:50.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:50 smithi107 ceph-mon[32659]: pgmap v8666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:10:50.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15587.102089 secs 2024-07-26T17:10:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:50 smithi112 ceph-mon[37501]: pgmap v8666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:10:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15587.102089 secs 2024-07-26T17:10:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:50 smithi132 ceph-mon[37580]: pgmap v8666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:10:51.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15587.102089 secs 2024-07-26T17:10:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:52 smithi112 ceph-mon[37501]: pgmap v8667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:52 smithi107 ceph-mon[32659]: pgmap v8667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:52 smithi132 ceph-mon[37580]: pgmap v8667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:10:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:54 smithi112 ceph-mon[37501]: pgmap v8668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:54 smithi107 ceph-mon[32659]: pgmap v8668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:54 smithi132 ceph-mon[37580]: pgmap v8668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15592.102184 secs 2024-07-26T17:10:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15592.102184 secs 2024-07-26T17:10:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15592.102184 secs 2024-07-26T17:10:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:56 smithi112 ceph-mon[37501]: pgmap v8669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:56 smithi107 ceph-mon[32659]: pgmap v8669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:56 smithi132 ceph-mon[37580]: pgmap v8669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:10:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:10:58 smithi112 ceph-mon[37501]: pgmap v8670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:10:58 smithi107 ceph-mon[32659]: pgmap v8670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:10:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:10:58 smithi132 ceph-mon[37580]: pgmap v8670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:00 smithi112 ceph-mon[37501]: pgmap v8671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:11:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15597.102301 secs 2024-07-26T17:11:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:00 smithi107 ceph-mon[32659]: pgmap v8671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:11:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15597.102301 secs 2024-07-26T17:11:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:00 smithi132 ceph-mon[37580]: pgmap v8671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:11:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15597.102301 secs 2024-07-26T17:11:02.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:02 smithi112 ceph-mon[37501]: pgmap v8672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:02 smithi107 ceph-mon[32659]: pgmap v8672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:02 smithi132 ceph-mon[37580]: pgmap v8672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:04.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:04.585+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:11:04.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:04.585+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:11:04.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:04.585+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:11:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:04 smithi112 ceph-mon[37501]: pgmap v8673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:04 smithi107 ceph-mon[32659]: pgmap v8673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:04 smithi132 ceph-mon[37580]: pgmap v8673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15602.102424 secs 2024-07-26T17:11:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15602.102424 secs 2024-07-26T17:11:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15602.102424 secs 2024-07-26T17:11:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:06 smithi112 ceph-mon[37501]: pgmap v8674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:06 smithi107 ceph-mon[32659]: pgmap v8674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:06 smithi132 ceph-mon[37580]: pgmap v8674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:09.018 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:08 smithi107 ceph-mon[32659]: pgmap v8675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:08 smithi132 ceph-mon[37580]: pgmap v8675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:08 smithi112 ceph-mon[37501]: pgmap v8675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:10 smithi107 ceph-mon[32659]: pgmap v8676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:11:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15607.102542 secs 2024-07-26T17:11:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:10 smithi132 ceph-mon[37580]: pgmap v8676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:11:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15607.102542 secs 2024-07-26T17:11:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:10 smithi112 ceph-mon[37501]: pgmap v8676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:11:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15607.102542 secs 2024-07-26T17:11:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:12 smithi107 ceph-mon[32659]: pgmap v8677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:12 smithi132 ceph-mon[37580]: pgmap v8677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:12 smithi112 ceph-mon[37501]: pgmap v8677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:14 smithi107 ceph-mon[32659]: pgmap v8678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:14 smithi132 ceph-mon[37580]: pgmap v8678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:14 smithi112 ceph-mon[37501]: pgmap v8678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15612.102661 secs 2024-07-26T17:11:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15612.102661 secs 2024-07-26T17:11:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15612.102661 secs 2024-07-26T17:11:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:16 smithi107 ceph-mon[32659]: pgmap v8679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:16 smithi132 ceph-mon[37580]: pgmap v8679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:16 smithi112 ceph-mon[37501]: pgmap v8679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:19.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:18 smithi107 ceph-mon[32659]: pgmap v8680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:18 smithi132 ceph-mon[37580]: pgmap v8680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:18 smithi112 ceph-mon[37501]: pgmap v8680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:19.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:19.587+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:11:19.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:19.587+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:11:19.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:19.588+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:11:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:20 smithi107 ceph-mon[32659]: pgmap v8681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:11:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15617.102776 secs 2024-07-26T17:11:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:20 smithi132 ceph-mon[37580]: pgmap v8681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:11:21.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15617.102776 secs 2024-07-26T17:11:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:20 smithi112 ceph-mon[37501]: pgmap v8681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:11:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15617.102776 secs 2024-07-26T17:11:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:22 smithi107 ceph-mon[32659]: pgmap v8682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:11:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:11:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:11:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:11:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:22 smithi132 ceph-mon[37580]: pgmap v8682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:11:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:11:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:11:23.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:11:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:22 smithi112 ceph-mon[37501]: pgmap v8682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:11:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:11:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:11:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:11:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:23 smithi107 ceph-mon[32659]: pgmap v8683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:11:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:23 smithi107 ceph-mon[32659]: pgmap v8684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:11:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:23 smithi132 ceph-mon[37580]: pgmap v8683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:11:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:23 smithi132 ceph-mon[37580]: pgmap v8684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:11:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:23 smithi112 ceph-mon[37501]: pgmap v8683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:11:24.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:23 smithi112 ceph-mon[37501]: pgmap v8684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:11:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:25 smithi107 ceph-mon[32659]: pgmap v8685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:11:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15622.102900 secs 2024-07-26T17:11:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:25 smithi132 ceph-mon[37580]: pgmap v8685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:11:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15622.102900 secs 2024-07-26T17:11:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:25 smithi112 ceph-mon[37501]: pgmap v8685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:11:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15622.102900 secs 2024-07-26T17:11:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:27 smithi107 ceph-mon[32659]: pgmap v8686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:27 smithi132 ceph-mon[37580]: pgmap v8686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:27 smithi112 ceph-mon[37501]: pgmap v8686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:29 smithi107 ceph-mon[32659]: pgmap v8687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:11:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:29 smithi132 ceph-mon[37580]: pgmap v8687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:11:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:29 smithi112 ceph-mon[37501]: pgmap v8687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:11:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15627.102998 secs 2024-07-26T17:11:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15627.102998 secs 2024-07-26T17:11:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15627.102998 secs 2024-07-26T17:11:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:31 smithi132 ceph-mon[37580]: pgmap v8688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:11:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:31 smithi112 ceph-mon[37501]: pgmap v8688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:11:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:31 smithi107 ceph-mon[32659]: pgmap v8688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:11:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:33 smithi132 ceph-mon[37580]: pgmap v8689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:11:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:33 smithi112 ceph-mon[37501]: pgmap v8689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:11:34.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:33 smithi107 ceph-mon[32659]: pgmap v8689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:11:34.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:34.594+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:11:34.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:34.594+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:11:34.596 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:34.594+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:11:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:35 smithi132 ceph-mon[37580]: pgmap v8690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:36.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15632.103129 secs 2024-07-26T17:11:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:35 smithi112 ceph-mon[37501]: pgmap v8690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15632.103129 secs 2024-07-26T17:11:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:35 smithi107 ceph-mon[32659]: pgmap v8690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15632.103129 secs 2024-07-26T17:11:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:37 smithi132 ceph-mon[37580]: pgmap v8691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:37 smithi112 ceph-mon[37501]: pgmap v8691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:37 smithi107 ceph-mon[32659]: pgmap v8691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:39 smithi112 ceph-mon[37501]: pgmap v8692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:39 smithi107 ceph-mon[32659]: pgmap v8692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:39 smithi132 ceph-mon[37580]: pgmap v8692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15637.103246 secs 2024-07-26T17:11:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15637.103246 secs 2024-07-26T17:11:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15637.103246 secs 2024-07-26T17:11:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:41 smithi112 ceph-mon[37501]: pgmap v8693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:41 smithi107 ceph-mon[32659]: pgmap v8693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:41 smithi132 ceph-mon[37580]: pgmap v8693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:43 smithi112 ceph-mon[37501]: pgmap v8694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:43 smithi107 ceph-mon[32659]: pgmap v8694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:43 smithi132 ceph-mon[37580]: pgmap v8694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:45 smithi112 ceph-mon[37501]: pgmap v8695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15642.103399 secs 2024-07-26T17:11:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:45 smithi107 ceph-mon[32659]: pgmap v8695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15642.103399 secs 2024-07-26T17:11:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:45 smithi132 ceph-mon[37580]: pgmap v8695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15642.103399 secs 2024-07-26T17:11:48.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:47 smithi112 ceph-mon[37501]: pgmap v8696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:47 smithi107 ceph-mon[32659]: pgmap v8696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:47 smithi132 ceph-mon[37580]: pgmap v8696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:49.609+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:11:49.611 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:49.609+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:11:49.612 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:11:49.609+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:11:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:49 smithi107 ceph-mon[32659]: pgmap v8697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:49 smithi112 ceph-mon[37501]: pgmap v8697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:49 smithi132 ceph-mon[37580]: pgmap v8697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:51.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15647.103491 secs 2024-07-26T17:11:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15647.103491 secs 2024-07-26T17:11:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15647.103491 secs 2024-07-26T17:11:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:51 smithi112 ceph-mon[37501]: pgmap v8698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:51 smithi107 ceph-mon[32659]: pgmap v8698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:51 smithi132 ceph-mon[37580]: pgmap v8698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:11:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:53 smithi112 ceph-mon[37501]: pgmap v8699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:53 smithi107 ceph-mon[32659]: pgmap v8699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:53 smithi132 ceph-mon[37580]: pgmap v8699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:11:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:55 smithi112 ceph-mon[37501]: pgmap v8700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15652.103608 secs 2024-07-26T17:11:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:55 smithi107 ceph-mon[32659]: pgmap v8700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15652.103608 secs 2024-07-26T17:11:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:55 smithi132 ceph-mon[37580]: pgmap v8700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15652.103608 secs 2024-07-26T17:11:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:57 smithi112 ceph-mon[37501]: pgmap v8701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:57 smithi107 ceph-mon[32659]: pgmap v8701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:11:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:57 smithi132 ceph-mon[37580]: pgmap v8701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:11:59 smithi112 ceph-mon[37501]: pgmap v8702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:11:59 smithi107 ceph-mon[32659]: pgmap v8702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:11:59 smithi132 ceph-mon[37580]: pgmap v8702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15657.103730 secs 2024-07-26T17:12:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15657.103730 secs 2024-07-26T17:12:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15657.103730 secs 2024-07-26T17:12:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:01 smithi112 ceph-mon[37501]: pgmap v8703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:01 smithi107 ceph-mon[32659]: pgmap v8703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:01 smithi132 ceph-mon[37580]: pgmap v8703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:03 smithi112 ceph-mon[37501]: pgmap v8704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:03 smithi107 ceph-mon[32659]: pgmap v8704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:03 smithi132 ceph-mon[37580]: pgmap v8704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:04.625+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:12:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:04.625+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:12:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:04.625+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:12:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:05 smithi112 ceph-mon[37501]: pgmap v8705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15662.103793 secs 2024-07-26T17:12:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:05 smithi107 ceph-mon[32659]: pgmap v8705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15662.103793 secs 2024-07-26T17:12:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:05 smithi132 ceph-mon[37580]: pgmap v8705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15662.103793 secs 2024-07-26T17:12:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:07 smithi112 ceph-mon[37501]: pgmap v8706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:07 smithi107 ceph-mon[32659]: pgmap v8706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:07 smithi132 ceph-mon[37580]: pgmap v8706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:09 smithi112 ceph-mon[37501]: pgmap v8707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:09 smithi107 ceph-mon[32659]: pgmap v8707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:09 smithi132 ceph-mon[37580]: pgmap v8707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15667.103903 secs 2024-07-26T17:12:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15667.103903 secs 2024-07-26T17:12:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15667.103903 secs 2024-07-26T17:12:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:11 smithi107 ceph-mon[32659]: pgmap v8708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:11 smithi132 ceph-mon[37580]: pgmap v8708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:11 smithi112 ceph-mon[37501]: pgmap v8708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:13 smithi107 ceph-mon[32659]: pgmap v8709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:13 smithi132 ceph-mon[37580]: pgmap v8709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:13 smithi112 ceph-mon[37501]: pgmap v8709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:15.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:15.340+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:12:15.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:15.340+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:12:15.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:15.341+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:12:15.343 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:15.341+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:12:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:15 smithi107 ceph-mon[32659]: pgmap v8710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15672.104042 secs 2024-07-26T17:12:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:15 smithi132 ceph-mon[37580]: pgmap v8710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15672.104042 secs 2024-07-26T17:12:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:15 smithi112 ceph-mon[37501]: pgmap v8710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15672.104042 secs 2024-07-26T17:12:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:17 smithi107 ceph-mon[32659]: pgmap v8711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:17 smithi132 ceph-mon[37580]: pgmap v8711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:17 smithi112 ceph-mon[37501]: pgmap v8711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:19.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:19.641+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:12:19.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:19.641+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:12:19.643 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:19.641+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:12:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:19 smithi107 ceph-mon[32659]: pgmap v8712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:19 smithi132 ceph-mon[37580]: pgmap v8712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:19 smithi112 ceph-mon[37501]: pgmap v8712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15677.104166 secs 2024-07-26T17:12:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15677.104166 secs 2024-07-26T17:12:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15677.104166 secs 2024-07-26T17:12:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:21 smithi107 ceph-mon[32659]: pgmap v8713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:21 smithi132 ceph-mon[37580]: pgmap v8713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:21 smithi112 ceph-mon[37501]: pgmap v8713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:12:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:12:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:12:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:23 smithi107 ceph-mon[32659]: pgmap v8714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:12:24.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:12:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:23 smithi132 ceph-mon[37580]: pgmap v8714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:12:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:12:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:23 smithi112 ceph-mon[37501]: pgmap v8714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:12:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:12:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:12:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:12:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:12:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:12:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:12:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:12:25.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:12:25.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:12:25.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:12:26.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:26 smithi107 ceph-mon[32659]: pgmap v8715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:12:26.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:26 smithi107 ceph-mon[32659]: pgmap v8716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:26.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15682.104258 secs 2024-07-26T17:12:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:26 smithi132 ceph-mon[37580]: pgmap v8715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:12:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:26 smithi132 ceph-mon[37580]: pgmap v8716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:26.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15682.104258 secs 2024-07-26T17:12:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:26 smithi112 ceph-mon[37501]: pgmap v8715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:12:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:26 smithi112 ceph-mon[37501]: pgmap v8716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15682.104258 secs 2024-07-26T17:12:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:28 smithi107 ceph-mon[32659]: pgmap v8717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:28 smithi132 ceph-mon[37580]: pgmap v8717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:28 smithi112 ceph-mon[37501]: pgmap v8717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:30 smithi107 ceph-mon[32659]: pgmap v8718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:30 smithi132 ceph-mon[37580]: pgmap v8718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:30 smithi112 ceph-mon[37501]: pgmap v8718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15687.104374 secs 2024-07-26T17:12:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15687.104374 secs 2024-07-26T17:12:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15687.104374 secs 2024-07-26T17:12:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:32 smithi107 ceph-mon[32659]: pgmap v8719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:32 smithi132 ceph-mon[37580]: pgmap v8719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:32 smithi112 ceph-mon[37501]: pgmap v8719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:12:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:34 smithi132 ceph-mon[37580]: pgmap v8720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:12:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:34 smithi112 ceph-mon[37501]: pgmap v8720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:12:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:34 smithi107 ceph-mon[32659]: pgmap v8720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:12:34.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:34.653+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:12:34.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:34.654+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:12:34.656 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:34.654+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:12:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:36 smithi132 ceph-mon[37580]: pgmap v8721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:12:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15692.104491 secs 2024-07-26T17:12:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:36 smithi112 ceph-mon[37501]: pgmap v8721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:12:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15692.104491 secs 2024-07-26T17:12:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:36 smithi107 ceph-mon[32659]: pgmap v8721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:12:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15692.104491 secs 2024-07-26T17:12:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:38 smithi132 ceph-mon[37580]: pgmap v8722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:12:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:38 smithi112 ceph-mon[37501]: pgmap v8722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:12:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:38 smithi107 ceph-mon[32659]: pgmap v8722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:12:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:40 smithi132 ceph-mon[37580]: pgmap v8723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:40 smithi112 ceph-mon[37501]: pgmap v8723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:40 smithi107 ceph-mon[32659]: pgmap v8723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15697.104601 secs 2024-07-26T17:12:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15697.104601 secs 2024-07-26T17:12:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15697.104601 secs 2024-07-26T17:12:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:42 smithi112 ceph-mon[37501]: pgmap v8724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:42 smithi107 ceph-mon[32659]: pgmap v8724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:42 smithi132 ceph-mon[37580]: pgmap v8724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:44 smithi112 ceph-mon[37501]: pgmap v8725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:44 smithi107 ceph-mon[32659]: pgmap v8725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:44 smithi132 ceph-mon[37580]: pgmap v8725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:46 smithi112 ceph-mon[37501]: pgmap v8726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:12:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15702.104740 secs 2024-07-26T17:12:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:46 smithi107 ceph-mon[32659]: pgmap v8726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:12:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15702.104740 secs 2024-07-26T17:12:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:46 smithi132 ceph-mon[37580]: pgmap v8726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:12:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15702.104740 secs 2024-07-26T17:12:48.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:48 smithi112 ceph-mon[37501]: pgmap v8727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:48 smithi107 ceph-mon[32659]: pgmap v8727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:48 smithi132 ceph-mon[37580]: pgmap v8727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:49.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:49.668+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:12:49.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:49.668+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:12:49.671 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:12:49.668+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:12:50.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:50 smithi112 ceph-mon[37501]: pgmap v8728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:50 smithi107 ceph-mon[32659]: pgmap v8728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:50 smithi132 ceph-mon[37580]: pgmap v8728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:51.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15707.104877 secs 2024-07-26T17:12:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15707.104877 secs 2024-07-26T17:12:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15707.104877 secs 2024-07-26T17:12:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:52 smithi112 ceph-mon[37501]: pgmap v8729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:52 smithi107 ceph-mon[32659]: pgmap v8729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:52 smithi132 ceph-mon[37580]: pgmap v8729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:12:54.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:54 smithi112 ceph-mon[37501]: pgmap v8730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:54 smithi107 ceph-mon[32659]: pgmap v8730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:54 smithi132 ceph-mon[37580]: pgmap v8730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:12:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:56 smithi112 ceph-mon[37501]: pgmap v8731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:12:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15712.105005 secs 2024-07-26T17:12:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:56 smithi107 ceph-mon[32659]: pgmap v8731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:12:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15712.105005 secs 2024-07-26T17:12:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:56 smithi132 ceph-mon[37580]: pgmap v8731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:12:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15712.105005 secs 2024-07-26T17:12:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:12:58 smithi112 ceph-mon[37501]: pgmap v8732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:12:58 smithi107 ceph-mon[32659]: pgmap v8732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:12:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:12:58 smithi132 ceph-mon[37580]: pgmap v8732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:00.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:00 smithi112 ceph-mon[37501]: pgmap v8733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:00 smithi107 ceph-mon[32659]: pgmap v8733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:00 smithi132 ceph-mon[37580]: pgmap v8733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15717.105127 secs 2024-07-26T17:13:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15717.105127 secs 2024-07-26T17:13:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15717.105127 secs 2024-07-26T17:13:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:02 smithi112 ceph-mon[37501]: pgmap v8734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:02 smithi107 ceph-mon[32659]: pgmap v8734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:02 smithi132 ceph-mon[37580]: pgmap v8734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:04.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:04 smithi112 ceph-mon[37501]: pgmap v8735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:04 smithi107 ceph-mon[32659]: pgmap v8735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:04 smithi132 ceph-mon[37580]: pgmap v8735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:04.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:04.676+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:13:04.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:04.676+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:13:04.678 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:04.676+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:13:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:06 smithi112 ceph-mon[37501]: pgmap v8736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15722.105251 secs 2024-07-26T17:13:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:06 smithi107 ceph-mon[32659]: pgmap v8736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15722.105251 secs 2024-07-26T17:13:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:06 smithi132 ceph-mon[37580]: pgmap v8736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15722.105251 secs 2024-07-26T17:13:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:08 smithi112 ceph-mon[37501]: pgmap v8737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:08 smithi107 ceph-mon[32659]: pgmap v8737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:08 smithi132 ceph-mon[37580]: pgmap v8737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:10 smithi112 ceph-mon[37501]: pgmap v8738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:10 smithi107 ceph-mon[32659]: pgmap v8738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:10 smithi132 ceph-mon[37580]: pgmap v8738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15727.105317 secs 2024-07-26T17:13:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15727.105317 secs 2024-07-26T17:13:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15727.105317 secs 2024-07-26T17:13:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:12 smithi112 ceph-mon[37501]: pgmap v8739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:12 smithi107 ceph-mon[32659]: pgmap v8739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:12 smithi132 ceph-mon[37580]: pgmap v8739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:14 smithi112 ceph-mon[37501]: pgmap v8740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:14 smithi107 ceph-mon[32659]: pgmap v8740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:14 smithi132 ceph-mon[37580]: pgmap v8740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:16 smithi112 ceph-mon[37501]: pgmap v8741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15732.105451 secs 2024-07-26T17:13:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:16 smithi107 ceph-mon[32659]: pgmap v8741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15732.105451 secs 2024-07-26T17:13:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:16 smithi132 ceph-mon[37580]: pgmap v8741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15732.105451 secs 2024-07-26T17:13:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:18 smithi107 ceph-mon[32659]: pgmap v8742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:13:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:18 smithi132 ceph-mon[37580]: pgmap v8742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:13:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:18 smithi112 ceph-mon[37501]: pgmap v8742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:13:19.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:19.686+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:13:19.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:19.686+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:13:19.690 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:19.686+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:13:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:20 smithi107 ceph-mon[32659]: pgmap v8743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:20 smithi132 ceph-mon[37580]: pgmap v8743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:20 smithi112 ceph-mon[37501]: pgmap v8743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15737.105554 secs 2024-07-26T17:13:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15737.105554 secs 2024-07-26T17:13:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15737.105554 secs 2024-07-26T17:13:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:22 smithi107 ceph-mon[32659]: pgmap v8744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:22 smithi132 ceph-mon[37580]: pgmap v8744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:22 smithi112 ceph-mon[37501]: pgmap v8744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:24 smithi107 ceph-mon[32659]: pgmap v8745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:24 smithi132 ceph-mon[37580]: pgmap v8745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:24 smithi112 ceph-mon[37501]: pgmap v8745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:25 smithi107 ceph-mon[32659]: pgmap v8746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:13:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:25 smithi132 ceph-mon[37580]: pgmap v8746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:13:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:25 smithi112 ceph-mon[37501]: pgmap v8746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:13:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15742.105626 secs 2024-07-26T17:13:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15742.105626 secs 2024-07-26T17:13:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15742.105626 secs 2024-07-26T17:13:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:27 smithi112 ceph-mon[37501]: pgmap v8747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:27.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:27 smithi107 ceph-mon[32659]: pgmap v8747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:27 smithi132 ceph-mon[37580]: pgmap v8747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.905 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.905 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.905 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.905 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:30 smithi112 ceph-mon[37501]: pgmap v8748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:13:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:13:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:30 smithi107 ceph-mon[32659]: pgmap v8748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:13:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:13:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:30 smithi132 ceph-mon[37580]: pgmap v8748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:30.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:30.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:13:30.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:13:30.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:13:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:31 smithi112 ceph-mon[37501]: pgmap v8749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:13:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:31 smithi112 ceph-mon[37501]: pgmap v8750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:13:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15747.105726 secs 2024-07-26T17:13:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:31 smithi107 ceph-mon[32659]: pgmap v8749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:13:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:31 smithi107 ceph-mon[32659]: pgmap v8750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:13:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15747.105726 secs 2024-07-26T17:13:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:31 smithi132 ceph-mon[37580]: pgmap v8749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:13:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:31 smithi132 ceph-mon[37580]: pgmap v8750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:13:31.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15747.105726 secs 2024-07-26T17:13:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:33 smithi112 ceph-mon[37501]: pgmap v8751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:33 smithi107 ceph-mon[32659]: pgmap v8751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:33 smithi132 ceph-mon[37580]: pgmap v8751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:34.705 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:34.702+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:13:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:34.702+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:13:34.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:34.702+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:13:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:35 smithi112 ceph-mon[37501]: pgmap v8752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T17:13:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:35 smithi107 ceph-mon[32659]: pgmap v8752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T17:13:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:35 smithi132 ceph-mon[37580]: pgmap v8752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T17:13:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15752.105857 secs 2024-07-26T17:13:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15752.105857 secs 2024-07-26T17:13:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15752.105857 secs 2024-07-26T17:13:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:37 smithi112 ceph-mon[37501]: pgmap v8753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T17:13:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:37 smithi107 ceph-mon[32659]: pgmap v8753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T17:13:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:37 smithi132 ceph-mon[37580]: pgmap v8753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T17:13:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:39 smithi112 ceph-mon[37501]: pgmap v8754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:13:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:39 smithi107 ceph-mon[32659]: pgmap v8754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:13:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:39 smithi132 ceph-mon[37580]: pgmap v8754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:13:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:41 smithi112 ceph-mon[37501]: pgmap v8755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:13:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15757.105981 secs 2024-07-26T17:13:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:41 smithi107 ceph-mon[32659]: pgmap v8755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:13:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15757.105981 secs 2024-07-26T17:13:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:41 smithi132 ceph-mon[37580]: pgmap v8755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:13:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15757.105981 secs 2024-07-26T17:13:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:43 smithi112 ceph-mon[37501]: pgmap v8756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:43 smithi107 ceph-mon[32659]: pgmap v8756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:43 smithi132 ceph-mon[37580]: pgmap v8756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:45 smithi112 ceph-mon[37501]: pgmap v8757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:45 smithi107 ceph-mon[32659]: pgmap v8757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:45 smithi132 ceph-mon[37580]: pgmap v8757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15762.106113 secs 2024-07-26T17:13:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15762.106113 secs 2024-07-26T17:13:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15762.106113 secs 2024-07-26T17:13:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:47 smithi112 ceph-mon[37501]: pgmap v8758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:47 smithi107 ceph-mon[32659]: pgmap v8758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:47.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:47 smithi132 ceph-mon[37580]: pgmap v8758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:49 smithi112 ceph-mon[37501]: pgmap v8759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:49.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:49 smithi107 ceph-mon[32659]: pgmap v8759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:49 smithi132 ceph-mon[37580]: pgmap v8759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:49.704+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:13:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:49.704+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:13:49.707 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:13:49.704+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:13:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:51 smithi107 ceph-mon[32659]: pgmap v8760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15767.106240 secs 2024-07-26T17:13:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:51 smithi132 ceph-mon[37580]: pgmap v8760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15767.106240 secs 2024-07-26T17:13:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:51 smithi112 ceph-mon[37501]: pgmap v8760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:13:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15767.106240 secs 2024-07-26T17:13:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:53 smithi107 ceph-mon[32659]: pgmap v8761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:53 smithi132 ceph-mon[37580]: pgmap v8761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:53 smithi112 ceph-mon[37501]: pgmap v8761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:13:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:55 smithi107 ceph-mon[32659]: pgmap v8762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:55 smithi132 ceph-mon[37580]: pgmap v8762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:55 smithi112 ceph-mon[37501]: pgmap v8762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15772.106372 secs 2024-07-26T17:13:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15772.106372 secs 2024-07-26T17:13:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15772.106372 secs 2024-07-26T17:13:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:57 smithi107 ceph-mon[32659]: pgmap v8763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:57 smithi132 ceph-mon[37580]: pgmap v8763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:57 smithi112 ceph-mon[37501]: pgmap v8763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:13:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:13:59 smithi107 ceph-mon[32659]: pgmap v8764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:13:59 smithi132 ceph-mon[37580]: pgmap v8764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:13:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:13:59 smithi112 ceph-mon[37501]: pgmap v8764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:00 smithi107 ceph-mon[32659]: pgmap v8765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:00 smithi132 ceph-mon[37580]: pgmap v8765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:00.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:00 smithi112 ceph-mon[37501]: pgmap v8765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15777.106512 secs 2024-07-26T17:14:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15777.106512 secs 2024-07-26T17:14:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15777.106512 secs 2024-07-26T17:14:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:02 smithi107 ceph-mon[32659]: pgmap v8766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:14:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:02 smithi132 ceph-mon[37580]: pgmap v8766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:14:02.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:02 smithi112 ceph-mon[37501]: pgmap v8766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:14:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:04.720+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:14:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:04.720+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:14:04.723 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:04.720+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:14:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:04 smithi112 ceph-mon[37501]: pgmap v8767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:14:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:04 smithi107 ceph-mon[32659]: pgmap v8767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:14:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:04 smithi132 ceph-mon[37580]: pgmap v8767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:14:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15782.106586 secs 2024-07-26T17:14:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15782.106586 secs 2024-07-26T17:14:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15782.106586 secs 2024-07-26T17:14:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:06 smithi112 ceph-mon[37501]: pgmap v8768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:14:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:06 smithi107 ceph-mon[32659]: pgmap v8768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:14:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:06 smithi132 ceph-mon[37580]: pgmap v8768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:14:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:08 smithi112 ceph-mon[37501]: pgmap v8769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:14:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:08 smithi107 ceph-mon[32659]: pgmap v8769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:14:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:08 smithi132 ceph-mon[37580]: pgmap v8769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:14:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:10 smithi112 ceph-mon[37501]: pgmap v8770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15787.106712 secs 2024-07-26T17:14:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:10 smithi107 ceph-mon[32659]: pgmap v8770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15787.106712 secs 2024-07-26T17:14:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:10 smithi132 ceph-mon[37580]: pgmap v8770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15787.106712 secs 2024-07-26T17:14:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:12 smithi112 ceph-mon[37501]: pgmap v8771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:14:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:12 smithi107 ceph-mon[32659]: pgmap v8771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:14:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:12 smithi132 ceph-mon[37580]: pgmap v8771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:14:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:14 smithi112 ceph-mon[37501]: pgmap v8772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:14 smithi107 ceph-mon[32659]: pgmap v8772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:14 smithi132 ceph-mon[37580]: pgmap v8772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15792.106849 secs 2024-07-26T17:14:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15792.106849 secs 2024-07-26T17:14:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15792.106849 secs 2024-07-26T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:16 smithi112 ceph-mon[37501]: pgmap v8773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:14:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:16 smithi107 ceph-mon[32659]: pgmap v8773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:14:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:16 smithi132 ceph-mon[37580]: pgmap v8773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:14:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:18 smithi112 ceph-mon[37501]: pgmap v8774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:18 smithi107 ceph-mon[32659]: pgmap v8774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:18 smithi132 ceph-mon[37580]: pgmap v8774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:19.738 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:19.737+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:14:19.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:19.737+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:14:19.739 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:19.737+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:14:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:20 smithi112 ceph-mon[37501]: pgmap v8775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15797.106975 secs 2024-07-26T17:14:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:20 smithi107 ceph-mon[32659]: pgmap v8775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15797.106975 secs 2024-07-26T17:14:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:20 smithi132 ceph-mon[37580]: pgmap v8775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15797.106975 secs 2024-07-26T17:14:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:22 smithi112 ceph-mon[37501]: pgmap v8776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:22 smithi107 ceph-mon[32659]: pgmap v8776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:22 smithi132 ceph-mon[37580]: pgmap v8776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:24 smithi112 ceph-mon[37501]: pgmap v8777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:24 smithi107 ceph-mon[32659]: pgmap v8777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:24 smithi132 ceph-mon[37580]: pgmap v8777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15802.107101 secs 2024-07-26T17:14:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15802.107101 secs 2024-07-26T17:14:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15802.107101 secs 2024-07-26T17:14:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:26 smithi112 ceph-mon[37501]: pgmap v8778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:14:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:26 smithi107 ceph-mon[32659]: pgmap v8778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:14:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:26 smithi132 ceph-mon[37580]: pgmap v8778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:14:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:28 smithi112 ceph-mon[37501]: pgmap v8779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:28 smithi107 ceph-mon[32659]: pgmap v8779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:28 smithi132 ceph-mon[37580]: pgmap v8779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:30 smithi112 ceph-mon[37501]: pgmap v8780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:14:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15807.107192 secs 2024-07-26T17:14:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:30 smithi107 ceph-mon[32659]: pgmap v8780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:14:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15807.107192 secs 2024-07-26T17:14:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:30 smithi132 ceph-mon[37580]: pgmap v8780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:14:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:14:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15807.107192 secs 2024-07-26T17:14:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:14:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:14:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:31 smithi107 ceph-mon[32659]: pgmap v8781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:14:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:31 smithi107 ceph-mon[32659]: pgmap v8782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:14:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:14:32.055 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:14:32.055 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:14:32.055 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:31 smithi132 ceph-mon[37580]: pgmap v8781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:14:32.055 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:31 smithi132 ceph-mon[37580]: pgmap v8782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:14:32.055 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:14:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:14:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:14:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:31 smithi112 ceph-mon[37501]: pgmap v8781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:14:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:31 smithi112 ceph-mon[37501]: pgmap v8782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:14:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:14:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:33 smithi107 ceph-mon[32659]: pgmap v8783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:14:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:33 smithi132 ceph-mon[37580]: pgmap v8783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:14:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:33 smithi112 ceph-mon[37501]: pgmap v8783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:14:34.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:34.740+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:14:34.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:34.740+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:14:34.742 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:34.740+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:14:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:35 smithi107 ceph-mon[32659]: pgmap v8784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:14:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15812.107314 secs 2024-07-26T17:14:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:35 smithi132 ceph-mon[37580]: pgmap v8784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:14:36.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15812.107314 secs 2024-07-26T17:14:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:35 smithi112 ceph-mon[37501]: pgmap v8784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:14:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15812.107314 secs 2024-07-26T17:14:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:37 smithi107 ceph-mon[32659]: pgmap v8785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T17:14:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:37 smithi132 ceph-mon[37580]: pgmap v8785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T17:14:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:37 smithi112 ceph-mon[37501]: pgmap v8785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T17:14:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:39 smithi107 ceph-mon[32659]: pgmap v8786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:14:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:39 smithi132 ceph-mon[37580]: pgmap v8786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:14:40.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:39 smithi112 ceph-mon[37501]: pgmap v8786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:14:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15817.107426 secs 2024-07-26T17:14:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15817.107426 secs 2024-07-26T17:14:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15817.107426 secs 2024-07-26T17:14:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:41 smithi132 ceph-mon[37580]: pgmap v8787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:14:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:41 smithi112 ceph-mon[37501]: pgmap v8787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:14:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:41 smithi107 ceph-mon[32659]: pgmap v8787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:14:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:43 smithi132 ceph-mon[37580]: pgmap v8788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:14:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:43 smithi112 ceph-mon[37501]: pgmap v8788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:14:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:43 smithi107 ceph-mon[32659]: pgmap v8788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:14:46.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:45 smithi107 ceph-mon[32659]: pgmap v8789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:46.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15822.107561 secs 2024-07-26T17:14:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:45 smithi112 ceph-mon[37501]: pgmap v8789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15822.107561 secs 2024-07-26T17:14:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:45 smithi132 ceph-mon[37580]: pgmap v8789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15822.107561 secs 2024-07-26T17:14:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:47 smithi112 ceph-mon[37501]: pgmap v8790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:47 smithi107 ceph-mon[32659]: pgmap v8790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:47 smithi132 ceph-mon[37580]: pgmap v8790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:49.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:49.743+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:14:49.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:49.743+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:14:49.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:14:49.743+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:14:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:49 smithi107 ceph-mon[32659]: pgmap v8791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:49 smithi112 ceph-mon[37501]: pgmap v8791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:49 smithi132 ceph-mon[37580]: pgmap v8791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:51.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15827.107693 secs 2024-07-26T17:14:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15827.107693 secs 2024-07-26T17:14:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15827.107693 secs 2024-07-26T17:14:52.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:51 smithi107 ceph-mon[32659]: pgmap v8792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:14:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:51 smithi112 ceph-mon[37501]: pgmap v8792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:14:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:51 smithi132 ceph-mon[37580]: pgmap v8792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:14:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:53 smithi112 ceph-mon[37501]: pgmap v8793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:53 smithi107 ceph-mon[32659]: pgmap v8793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:53 smithi132 ceph-mon[37580]: pgmap v8793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:14:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:55 smithi112 ceph-mon[37501]: pgmap v8794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15832.107829 secs 2024-07-26T17:14:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:55 smithi107 ceph-mon[32659]: pgmap v8794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15832.107829 secs 2024-07-26T17:14:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:55 smithi132 ceph-mon[37580]: pgmap v8794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15832.107829 secs 2024-07-26T17:14:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:57 smithi112 ceph-mon[37501]: pgmap v8795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:57 smithi107 ceph-mon[32659]: pgmap v8795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:14:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:57 smithi132 ceph-mon[37580]: pgmap v8795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:00.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:14:59 smithi132 ceph-mon[37580]: pgmap v8796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:14:59 smithi112 ceph-mon[37501]: pgmap v8796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:14:59 smithi107 ceph-mon[32659]: pgmap v8796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15837.107969 secs 2024-07-26T17:15:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15837.107969 secs 2024-07-26T17:15:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15837.107969 secs 2024-07-26T17:15:02.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:01 smithi107 ceph-mon[32659]: pgmap v8797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:01 smithi112 ceph-mon[37501]: pgmap v8797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:01 smithi132 ceph-mon[37580]: pgmap v8797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:04.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:03 smithi112 ceph-mon[37501]: pgmap v8798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:03 smithi107 ceph-mon[32659]: pgmap v8798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:03 smithi132 ceph-mon[37580]: pgmap v8798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:04.749+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:15:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:04.749+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:15:04.752 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:04.749+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:15:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:05 smithi112 ceph-mon[37501]: pgmap v8799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15842.108093 secs 2024-07-26T17:15:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:05 smithi107 ceph-mon[32659]: pgmap v8799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15842.108093 secs 2024-07-26T17:15:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:05 smithi132 ceph-mon[37580]: pgmap v8799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15842.108093 secs 2024-07-26T17:15:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:07 smithi112 ceph-mon[37501]: pgmap v8800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:07 smithi107 ceph-mon[32659]: pgmap v8800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:07 smithi132 ceph-mon[37580]: pgmap v8800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:09 smithi112 ceph-mon[37501]: pgmap v8801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:09 smithi107 ceph-mon[32659]: pgmap v8801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:09 smithi132 ceph-mon[37580]: pgmap v8801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15847.108182 secs 2024-07-26T17:15:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15847.108182 secs 2024-07-26T17:15:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15847.108182 secs 2024-07-26T17:15:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:11 smithi112 ceph-mon[37501]: pgmap v8802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:11 smithi107 ceph-mon[32659]: pgmap v8802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:11 smithi132 ceph-mon[37580]: pgmap v8802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:13 smithi112 ceph-mon[37501]: pgmap v8803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:13 smithi107 ceph-mon[32659]: pgmap v8803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:13 smithi132 ceph-mon[37580]: pgmap v8803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:14.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:14.372+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:15:14.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:14.372+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:15:14.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:14.373+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:15:14.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:14.373+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:15:15.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:15.372+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:15:15.374 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:15.372+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:15:15.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:15.373+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:15:15.375 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:15.373+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:15:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:15 smithi112 ceph-mon[37501]: pgmap v8804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15852.108298 secs 2024-07-26T17:15:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:15 smithi107 ceph-mon[32659]: pgmap v8804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15852.108298 secs 2024-07-26T17:15:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:15 smithi132 ceph-mon[37580]: pgmap v8804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15852.108298 secs 2024-07-26T17:15:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:17 smithi112 ceph-mon[37501]: pgmap v8805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:17 smithi107 ceph-mon[32659]: pgmap v8805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:17 smithi132 ceph-mon[37580]: pgmap v8805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:19.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:19.759+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:15:19.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:19.759+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:15:19.761 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:19.759+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:15:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:19 smithi107 ceph-mon[32659]: pgmap v8806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:19 smithi112 ceph-mon[37501]: pgmap v8806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:19 smithi132 ceph-mon[37580]: pgmap v8806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15857.108409 secs 2024-07-26T17:15:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15857.108409 secs 2024-07-26T17:15:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15857.108409 secs 2024-07-26T17:15:22.196 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:21 smithi107 ceph-mon[32659]: pgmap v8807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:21 smithi112 ceph-mon[37501]: pgmap v8807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:21 smithi132 ceph-mon[37580]: pgmap v8807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:23 smithi107 ceph-mon[32659]: pgmap v8808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:23 smithi132 ceph-mon[37580]: pgmap v8808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:23 smithi112 ceph-mon[37501]: pgmap v8808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:26 smithi132 ceph-mon[37580]: pgmap v8809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15862.108477 secs 2024-07-26T17:15:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:26 smithi112 ceph-mon[37501]: pgmap v8809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15862.108477 secs 2024-07-26T17:15:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:26 smithi107 ceph-mon[32659]: pgmap v8809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15862.108477 secs 2024-07-26T17:15:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:28 smithi132 ceph-mon[37580]: pgmap v8810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:15:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:28 smithi112 ceph-mon[37501]: pgmap v8810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:15:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:28 smithi107 ceph-mon[32659]: pgmap v8810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:15:30.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:30 smithi132 ceph-mon[37580]: pgmap v8811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:15:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:30 smithi112 ceph-mon[37501]: pgmap v8811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:15:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:30 smithi107 ceph-mon[32659]: pgmap v8811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:15:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15867.108603 secs 2024-07-26T17:15:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:15:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15867.108603 secs 2024-07-26T17:15:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:15:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15867.108603 secs 2024-07-26T17:15:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:15:32.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:32 smithi107 ceph-mon[32659]: pgmap v8812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:15:32.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:15:32.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:15:32.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:15:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:32 smithi112 ceph-mon[37501]: pgmap v8812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:15:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:15:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:15:32.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:15:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:32 smithi132 ceph-mon[37580]: pgmap v8812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:15:32.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:15:32.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:15:32.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:15:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:33 smithi112 ceph-mon[37501]: pgmap v8813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:15:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:33 smithi112 ceph-mon[37501]: pgmap v8814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:15:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:33 smithi107 ceph-mon[32659]: pgmap v8813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:15:33.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:33 smithi107 ceph-mon[32659]: pgmap v8814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:15:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:33 smithi132 ceph-mon[37580]: pgmap v8813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:15:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:33 smithi132 ceph-mon[37580]: pgmap v8814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:15:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:34.768+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:15:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:34.769+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:15:34.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:34.769+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:15:35.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:35 smithi112 ceph-mon[37501]: pgmap v8815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:15:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:35 smithi107 ceph-mon[32659]: pgmap v8815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:15:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:35 smithi132 ceph-mon[37580]: pgmap v8815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:15:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15872.108752 secs 2024-07-26T17:15:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15872.108752 secs 2024-07-26T17:15:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15872.108752 secs 2024-07-26T17:15:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:37 smithi112 ceph-mon[37501]: pgmap v8816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:15:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:37 smithi107 ceph-mon[32659]: pgmap v8816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:15:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:37 smithi132 ceph-mon[37580]: pgmap v8816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:15:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:39 smithi112 ceph-mon[37501]: pgmap v8817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:15:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:39 smithi107 ceph-mon[32659]: pgmap v8817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:15:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:39 smithi132 ceph-mon[37580]: pgmap v8817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:15:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:41 smithi112 ceph-mon[37501]: pgmap v8818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:15:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15877.108897 secs 2024-07-26T17:15:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:41 smithi107 ceph-mon[32659]: pgmap v8818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:15:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15877.108897 secs 2024-07-26T17:15:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:41 smithi132 ceph-mon[37580]: pgmap v8818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:15:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15877.108897 secs 2024-07-26T17:15:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:43 smithi112 ceph-mon[37501]: pgmap v8819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:15:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:43 smithi107 ceph-mon[32659]: pgmap v8819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:15:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:43 smithi132 ceph-mon[37580]: pgmap v8819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:15:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:45 smithi112 ceph-mon[37501]: pgmap v8820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:45 smithi107 ceph-mon[32659]: pgmap v8820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:45 smithi132 ceph-mon[37580]: pgmap v8820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15882.109030 secs 2024-07-26T17:15:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15882.109030 secs 2024-07-26T17:15:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15882.109030 secs 2024-07-26T17:15:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:47 smithi112 ceph-mon[37501]: pgmap v8821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:47 smithi107 ceph-mon[32659]: pgmap v8821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:47 smithi132 ceph-mon[37580]: pgmap v8821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:49 smithi112 ceph-mon[37501]: pgmap v8822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:49 smithi107 ceph-mon[32659]: pgmap v8822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:49 smithi132 ceph-mon[37580]: pgmap v8822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:49.783+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:15:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:49.783+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:15:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:15:49.783+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:15:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:51 smithi112 ceph-mon[37501]: pgmap v8823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15887.109158 secs 2024-07-26T17:15:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:51 smithi107 ceph-mon[32659]: pgmap v8823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15887.109158 secs 2024-07-26T17:15:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:51 smithi132 ceph-mon[37580]: pgmap v8823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15887.109158 secs 2024-07-26T17:15:53.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:53 smithi112 ceph-mon[37501]: pgmap v8824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:53 smithi107 ceph-mon[32659]: pgmap v8824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:53 smithi132 ceph-mon[37580]: pgmap v8824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:15:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:55 smithi112 ceph-mon[37501]: pgmap v8825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:55 smithi107 ceph-mon[32659]: pgmap v8825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:55 smithi132 ceph-mon[37580]: pgmap v8825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15892.109296 secs 2024-07-26T17:15:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15892.109296 secs 2024-07-26T17:15:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15892.109296 secs 2024-07-26T17:15:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:57 smithi112 ceph-mon[37501]: pgmap v8826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:57 smithi107 ceph-mon[32659]: pgmap v8826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:57 smithi132 ceph-mon[37580]: pgmap v8826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:15:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:15:59 smithi112 ceph-mon[37501]: pgmap v8827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:15:59 smithi107 ceph-mon[32659]: pgmap v8827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:15:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:15:59 smithi132 ceph-mon[37580]: pgmap v8827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:01 smithi112 ceph-mon[37501]: pgmap v8828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15897.109366 secs 2024-07-26T17:16:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:01 smithi107 ceph-mon[32659]: pgmap v8828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15897.109366 secs 2024-07-26T17:16:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:01 smithi132 ceph-mon[37580]: pgmap v8828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15897.109366 secs 2024-07-26T17:16:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:03 smithi112 ceph-mon[37501]: pgmap v8829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:03 smithi107 ceph-mon[32659]: pgmap v8829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:03 smithi132 ceph-mon[37580]: pgmap v8829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:04.801 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:04.799+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:16:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:04.799+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:16:04.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:04.799+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:16:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:05 smithi112 ceph-mon[37501]: pgmap v8830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:05 smithi107 ceph-mon[32659]: pgmap v8830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:05 smithi132 ceph-mon[37580]: pgmap v8830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15902.109483 secs 2024-07-26T17:16:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15902.109483 secs 2024-07-26T17:16:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15902.109483 secs 2024-07-26T17:16:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:07 smithi107 ceph-mon[32659]: pgmap v8831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:07 smithi132 ceph-mon[37580]: pgmap v8831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:07.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:07 smithi112 ceph-mon[37501]: pgmap v8831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:09.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:09 smithi107 ceph-mon[32659]: pgmap v8832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:09 smithi132 ceph-mon[37580]: pgmap v8832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:09 smithi112 ceph-mon[37501]: pgmap v8832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:11 smithi107 ceph-mon[32659]: pgmap v8833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15907.109606 secs 2024-07-26T17:16:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:11 smithi132 ceph-mon[37580]: pgmap v8833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15907.109606 secs 2024-07-26T17:16:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:11 smithi112 ceph-mon[37501]: pgmap v8833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15907.109606 secs 2024-07-26T17:16:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:13 smithi107 ceph-mon[32659]: pgmap v8834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:13 smithi132 ceph-mon[37580]: pgmap v8834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:13 smithi112 ceph-mon[37501]: pgmap v8834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:15 smithi107 ceph-mon[32659]: pgmap v8835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:15 smithi132 ceph-mon[37580]: pgmap v8835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:15 smithi112 ceph-mon[37501]: pgmap v8835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15912.109710 secs 2024-07-26T17:16:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15912.109710 secs 2024-07-26T17:16:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15912.109710 secs 2024-07-26T17:16:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:17 smithi107 ceph-mon[32659]: pgmap v8836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:17 smithi132 ceph-mon[37580]: pgmap v8836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:17 smithi112 ceph-mon[37501]: pgmap v8836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:19 smithi107 ceph-mon[32659]: pgmap v8837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:19 smithi132 ceph-mon[37580]: pgmap v8837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:19 smithi112 ceph-mon[37501]: pgmap v8837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:19.817 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:19.815+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:16:19.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:19.815+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:16:19.818 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:19.815+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:16:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:21 smithi107 ceph-mon[32659]: pgmap v8838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15917.109831 secs 2024-07-26T17:16:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:21 smithi132 ceph-mon[37580]: pgmap v8838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15917.109831 secs 2024-07-26T17:16:21.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:21 smithi112 ceph-mon[37501]: pgmap v8838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:21.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15917.109831 secs 2024-07-26T17:16:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:23 smithi107 ceph-mon[32659]: pgmap v8839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:23 smithi132 ceph-mon[37580]: pgmap v8839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:23 smithi112 ceph-mon[37501]: pgmap v8839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:25 smithi107 ceph-mon[32659]: pgmap v8840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:25 smithi132 ceph-mon[37580]: pgmap v8840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:25 smithi112 ceph-mon[37501]: pgmap v8840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15922.109955 secs 2024-07-26T17:16:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15922.109955 secs 2024-07-26T17:16:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15922.109955 secs 2024-07-26T17:16:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:27 smithi107 ceph-mon[32659]: pgmap v8841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:27 smithi132 ceph-mon[37580]: pgmap v8841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:27 smithi112 ceph-mon[37501]: pgmap v8841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:28 smithi107 ceph-mon[32659]: pgmap v8842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:28 smithi132 ceph-mon[37580]: pgmap v8842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:28 smithi112 ceph-mon[37501]: pgmap v8842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:31 smithi107 ceph-mon[32659]: pgmap v8843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15927.110049 secs 2024-07-26T17:16:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:31 smithi132 ceph-mon[37580]: pgmap v8843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15927.110049 secs 2024-07-26T17:16:31.403 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:31 smithi112 ceph-mon[37501]: pgmap v8843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:31.403 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15927.110049 secs 2024-07-26T17:16:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:33 smithi107 ceph-mon[32659]: pgmap v8844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:16:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:33 smithi132 ceph-mon[37580]: pgmap v8844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:16:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:33 smithi112 ceph-mon[37501]: pgmap v8844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:16:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:16:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:16:34.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:34 smithi132 ceph-mon[37580]: pgmap v8845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:16:34.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:34 smithi132 ceph-mon[37580]: pgmap v8846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:16:34.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:34 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:16:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:16:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:16:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:34 smithi112 ceph-mon[37501]: pgmap v8845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:16:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:34 smithi112 ceph-mon[37501]: pgmap v8846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:16:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:34 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:16:34.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:16:34.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:16:34.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:34 smithi107 ceph-mon[32659]: pgmap v8845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:16:34.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:34 smithi107 ceph-mon[32659]: pgmap v8846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:16:34.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:34 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:16:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:34.831+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:16:34.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:34.831+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:16:34.833 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:34.831+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:16:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:36 smithi112 ceph-mon[37501]: pgmap v8847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:16:36.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15932.110175 secs 2024-07-26T17:16:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:36 smithi107 ceph-mon[32659]: pgmap v8847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:16:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15932.110175 secs 2024-07-26T17:16:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:36 smithi132 ceph-mon[37580]: pgmap v8847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:16:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15932.110175 secs 2024-07-26T17:16:38.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:38 smithi112 ceph-mon[37501]: pgmap v8848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:38.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:38 smithi107 ceph-mon[32659]: pgmap v8848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:38 smithi132 ceph-mon[37580]: pgmap v8848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:40 smithi112 ceph-mon[37501]: pgmap v8849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:16:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:40 smithi107 ceph-mon[32659]: pgmap v8849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:16:40.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:40 smithi132 ceph-mon[37580]: pgmap v8849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:16:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15937.110299 secs 2024-07-26T17:16:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15937.110299 secs 2024-07-26T17:16:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15937.110299 secs 2024-07-26T17:16:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:42 smithi107 ceph-mon[32659]: pgmap v8850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:16:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:42 smithi132 ceph-mon[37580]: pgmap v8850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:16:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:42 smithi112 ceph-mon[37501]: pgmap v8850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:16:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:44 smithi107 ceph-mon[32659]: pgmap v8851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:16:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:44 smithi132 ceph-mon[37580]: pgmap v8851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:16:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:44 smithi112 ceph-mon[37501]: pgmap v8851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:16:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:46 smithi107 ceph-mon[32659]: pgmap v8852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15942.110427 secs 2024-07-26T17:16:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:46 smithi132 ceph-mon[37580]: pgmap v8852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15942.110427 secs 2024-07-26T17:16:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:46 smithi112 ceph-mon[37501]: pgmap v8852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15942.110427 secs 2024-07-26T17:16:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:48 smithi107 ceph-mon[32659]: pgmap v8853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:48 smithi132 ceph-mon[37580]: pgmap v8853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:48 smithi112 ceph-mon[37501]: pgmap v8853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:49.848 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:49.846+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:16:49.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:49.846+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:16:49.849 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:16:49.846+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:16:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:50 smithi107 ceph-mon[32659]: pgmap v8854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:50 smithi132 ceph-mon[37580]: pgmap v8854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:50 smithi112 ceph-mon[37501]: pgmap v8854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15947.110540 secs 2024-07-26T17:16:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15947.110540 secs 2024-07-26T17:16:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15947.110540 secs 2024-07-26T17:16:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:52 smithi107 ceph-mon[32659]: pgmap v8855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:52 smithi132 ceph-mon[37580]: pgmap v8855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:52 smithi112 ceph-mon[37501]: pgmap v8855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:16:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:54 smithi107 ceph-mon[32659]: pgmap v8856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:54 smithi132 ceph-mon[37580]: pgmap v8856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:54 smithi112 ceph-mon[37501]: pgmap v8856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:16:56.254 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:56 smithi132 ceph-mon[37580]: pgmap v8857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:56 smithi107 ceph-mon[32659]: pgmap v8857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15952.110638 secs 2024-07-26T17:16:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15952.110638 secs 2024-07-26T17:16:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:56 smithi112 ceph-mon[37501]: pgmap v8857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15952.110638 secs 2024-07-26T17:16:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:16:58 smithi107 ceph-mon[32659]: pgmap v8858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:16:58 smithi132 ceph-mon[37580]: pgmap v8858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:16:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:16:58 smithi112 ceph-mon[37501]: pgmap v8858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:00 smithi107 ceph-mon[32659]: pgmap v8859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:00 smithi132 ceph-mon[37580]: pgmap v8859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:00 smithi112 ceph-mon[37501]: pgmap v8859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15957.110768 secs 2024-07-26T17:17:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15957.110768 secs 2024-07-26T17:17:01.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15957.110768 secs 2024-07-26T17:17:02.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:02 smithi107 ceph-mon[32659]: pgmap v8860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:02 smithi132 ceph-mon[37580]: pgmap v8860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:02 smithi112 ceph-mon[37501]: pgmap v8860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:03 smithi107 ceph-mon[32659]: pgmap v8861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:03 smithi132 ceph-mon[37580]: pgmap v8861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:03 smithi112 ceph-mon[37501]: pgmap v8861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:04.848+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:17:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:04.848+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:17:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:04.848+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:17:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:06 smithi112 ceph-mon[37501]: pgmap v8862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15962.110909 secs 2024-07-26T17:17:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:06 smithi107 ceph-mon[32659]: pgmap v8862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15962.110909 secs 2024-07-26T17:17:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:06 smithi132 ceph-mon[37580]: pgmap v8862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15962.110909 secs 2024-07-26T17:17:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:08 smithi107 ceph-mon[32659]: pgmap v8863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:08 smithi132 ceph-mon[37580]: pgmap v8863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:08 smithi112 ceph-mon[37501]: pgmap v8863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:10 smithi107 ceph-mon[32659]: pgmap v8864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:10 smithi132 ceph-mon[37580]: pgmap v8864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:10 smithi112 ceph-mon[37501]: pgmap v8864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15967.111034 secs 2024-07-26T17:17:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15967.111034 secs 2024-07-26T17:17:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15967.111034 secs 2024-07-26T17:17:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:12 smithi107 ceph-mon[32659]: pgmap v8865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:12 smithi132 ceph-mon[37580]: pgmap v8865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:12 smithi112 ceph-mon[37501]: pgmap v8865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:14 smithi107 ceph-mon[32659]: pgmap v8866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:14 smithi132 ceph-mon[37580]: pgmap v8866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:14 smithi112 ceph-mon[37501]: pgmap v8866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:16 smithi107 ceph-mon[32659]: pgmap v8867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15972.111168 secs 2024-07-26T17:17:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:16 smithi132 ceph-mon[37580]: pgmap v8867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15972.111168 secs 2024-07-26T17:17:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:16 smithi112 ceph-mon[37501]: pgmap v8867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15972.111168 secs 2024-07-26T17:17:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:18 smithi107 ceph-mon[32659]: pgmap v8868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:18 smithi132 ceph-mon[37580]: pgmap v8868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:18 smithi112 ceph-mon[37501]: pgmap v8868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:19.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:19.854+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:17:19.856 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:19.854+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:17:19.857 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:19.854+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:17:20.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:20 smithi112 ceph-mon[37501]: pgmap v8869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:20 smithi107 ceph-mon[32659]: pgmap v8869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:20 smithi132 ceph-mon[37580]: pgmap v8869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15977.111304 secs 2024-07-26T17:17:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15977.111304 secs 2024-07-26T17:17:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15977.111304 secs 2024-07-26T17:17:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:22 smithi112 ceph-mon[37501]: pgmap v8870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:22 smithi107 ceph-mon[32659]: pgmap v8870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:22 smithi132 ceph-mon[37580]: pgmap v8870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:24 smithi107 ceph-mon[32659]: pgmap v8871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:24 smithi132 ceph-mon[37580]: pgmap v8871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:24 smithi112 ceph-mon[37501]: pgmap v8871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:26 smithi107 ceph-mon[32659]: pgmap v8872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15982.111439 secs 2024-07-26T17:17:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:26 smithi132 ceph-mon[37580]: pgmap v8872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15982.111439 secs 2024-07-26T17:17:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:26 smithi112 ceph-mon[37501]: pgmap v8872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15982.111439 secs 2024-07-26T17:17:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:28 smithi107 ceph-mon[32659]: pgmap v8873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:28 smithi132 ceph-mon[37580]: pgmap v8873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:28 smithi112 ceph-mon[37501]: pgmap v8873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:30 smithi107 ceph-mon[32659]: pgmap v8874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:30 smithi132 ceph-mon[37580]: pgmap v8874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:30 smithi112 ceph-mon[37501]: pgmap v8874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15987.111567 secs 2024-07-26T17:17:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15987.111567 secs 2024-07-26T17:17:31.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15987.111567 secs 2024-07-26T17:17:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:32 smithi107 ceph-mon[32659]: pgmap v8875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:32 smithi132 ceph-mon[37580]: pgmap v8875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:32.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:32 smithi112 ceph-mon[37501]: pgmap v8875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:34 smithi107 ceph-mon[32659]: pgmap v8876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:34.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:17:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:34 smithi132 ceph-mon[37580]: pgmap v8876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:17:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:34 smithi112 ceph-mon[37501]: pgmap v8876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:17:34.871 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:34.870+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:17:34.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:34.870+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:17:34.872 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:34.870+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:17:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:17:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:17:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:17:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:17:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:17:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:17:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:17:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:17:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:17:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:36 smithi107 ceph-mon[32659]: pgmap v8877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:17:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15992.111702 secs 2024-07-26T17:17:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:36 smithi132 ceph-mon[37580]: pgmap v8877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:17:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15992.111702 secs 2024-07-26T17:17:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:36 smithi112 ceph-mon[37501]: pgmap v8877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:17:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15992.111702 secs 2024-07-26T17:17:38.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:38 smithi107 ceph-mon[32659]: pgmap v8878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:17:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:38 smithi132 ceph-mon[37580]: pgmap v8878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:17:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:38 smithi112 ceph-mon[37501]: pgmap v8878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:17:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:40 smithi107 ceph-mon[32659]: pgmap v8879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:17:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:40 smithi132 ceph-mon[37580]: pgmap v8879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:17:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:40 smithi112 ceph-mon[37501]: pgmap v8879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:17:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 15997.111821 secs 2024-07-26T17:17:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 15997.111821 secs 2024-07-26T17:17:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 15997.111821 secs 2024-07-26T17:17:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:42 smithi107 ceph-mon[32659]: pgmap v8880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:17:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:42 smithi132 ceph-mon[37580]: pgmap v8880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:17:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:42 smithi112 ceph-mon[37501]: pgmap v8880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:17:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:43 smithi132 ceph-mon[37580]: pgmap v8881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:17:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:43 smithi112 ceph-mon[37501]: pgmap v8881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:17:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:43 smithi107 ceph-mon[32659]: pgmap v8881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:17:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:45 smithi112 ceph-mon[37501]: pgmap v8882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:17:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:45 smithi107 ceph-mon[32659]: pgmap v8882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:17:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:45 smithi132 ceph-mon[37580]: pgmap v8882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:17:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16002.111967 secs 2024-07-26T17:17:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16002.111967 secs 2024-07-26T17:17:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16002.111967 secs 2024-07-26T17:17:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:47 smithi112 ceph-mon[37501]: pgmap v8883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:47 smithi107 ceph-mon[32659]: pgmap v8883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:47 smithi132 ceph-mon[37580]: pgmap v8883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:49 smithi112 ceph-mon[37501]: pgmap v8884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:49 smithi107 ceph-mon[32659]: pgmap v8884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:49 smithi132 ceph-mon[37580]: pgmap v8884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:49.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:49.884+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:17:49.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:49.884+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:17:49.887 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:17:49.884+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:17:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:51 smithi112 ceph-mon[37501]: pgmap v8885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16007.112028 secs 2024-07-26T17:17:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:51 smithi107 ceph-mon[32659]: pgmap v8885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16007.112028 secs 2024-07-26T17:17:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:51 smithi132 ceph-mon[37580]: pgmap v8885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:17:51.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16007.112028 secs 2024-07-26T17:17:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:53 smithi112 ceph-mon[37501]: pgmap v8886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:53 smithi107 ceph-mon[32659]: pgmap v8886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:53 smithi132 ceph-mon[37580]: pgmap v8886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:55 smithi107 ceph-mon[32659]: pgmap v8887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:55 smithi132 ceph-mon[37580]: pgmap v8887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:55 smithi112 ceph-mon[37501]: pgmap v8887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16012.112136 secs 2024-07-26T17:17:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16012.112136 secs 2024-07-26T17:17:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16012.112136 secs 2024-07-26T17:17:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:57 smithi107 ceph-mon[32659]: pgmap v8888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:57 smithi132 ceph-mon[37580]: pgmap v8888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:57 smithi112 ceph-mon[37501]: pgmap v8888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:17:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:17:59 smithi107 ceph-mon[32659]: pgmap v8889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:17:59 smithi132 ceph-mon[37580]: pgmap v8889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:17:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:17:59 smithi112 ceph-mon[37501]: pgmap v8889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:01.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:01 smithi112 ceph-mon[37501]: pgmap v8890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:01.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16017.112253 secs 2024-07-26T17:18:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:01 smithi107 ceph-mon[32659]: pgmap v8890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16017.112253 secs 2024-07-26T17:18:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:01 smithi132 ceph-mon[37580]: pgmap v8890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16017.112253 secs 2024-07-26T17:18:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:03 smithi107 ceph-mon[32659]: pgmap v8891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:03 smithi132 ceph-mon[37580]: pgmap v8891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:03 smithi112 ceph-mon[37501]: pgmap v8891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:04.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:04.892+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:18:04.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:04.892+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:18:04.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:04.892+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:18:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:05 smithi107 ceph-mon[32659]: pgmap v8892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:05 smithi132 ceph-mon[37580]: pgmap v8892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:05 smithi112 ceph-mon[37501]: pgmap v8892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16022.112370 secs 2024-07-26T17:18:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16022.112370 secs 2024-07-26T17:18:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16022.112370 secs 2024-07-26T17:18:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:07 smithi107 ceph-mon[32659]: pgmap v8893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:07 smithi132 ceph-mon[37580]: pgmap v8893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:07 smithi112 ceph-mon[37501]: pgmap v8893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:09 smithi107 ceph-mon[32659]: pgmap v8894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:09 smithi132 ceph-mon[37580]: pgmap v8894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:09 smithi112 ceph-mon[37501]: pgmap v8894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:11 smithi107 ceph-mon[32659]: pgmap v8895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16027.112465 secs 2024-07-26T17:18:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:11 smithi132 ceph-mon[37580]: pgmap v8895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16027.112465 secs 2024-07-26T17:18:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:11 smithi112 ceph-mon[37501]: pgmap v8895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16027.112465 secs 2024-07-26T17:18:13.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:13 smithi107 ceph-mon[32659]: pgmap v8896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:13 smithi132 ceph-mon[37580]: pgmap v8896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:13 smithi112 ceph-mon[37501]: pgmap v8896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:15.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:15.404+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:18:15.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:15.404+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:18:15.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:15.405+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:18:15.407 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:15.405+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:18:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:15 smithi107 ceph-mon[32659]: pgmap v8897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:15.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16032.112606 secs 2024-07-26T17:18:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:15 smithi132 ceph-mon[37580]: pgmap v8897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16032.112606 secs 2024-07-26T17:18:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:15 smithi112 ceph-mon[37501]: pgmap v8897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16032.112606 secs 2024-07-26T17:18:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:17 smithi107 ceph-mon[32659]: pgmap v8898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:17 smithi132 ceph-mon[37580]: pgmap v8898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:17 smithi112 ceph-mon[37501]: pgmap v8898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:19 smithi107 ceph-mon[32659]: pgmap v8899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:19 smithi132 ceph-mon[37580]: pgmap v8899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:19.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:19.896+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:18:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:19.896+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:18:19.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:19.896+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:18:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:19 smithi112 ceph-mon[37501]: pgmap v8899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16037.112767 secs 2024-07-26T17:18:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16037.112767 secs 2024-07-26T17:18:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16037.112767 secs 2024-07-26T17:18:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:21 smithi132 ceph-mon[37580]: pgmap v8900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:21 smithi112 ceph-mon[37501]: pgmap v8900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:21 smithi107 ceph-mon[32659]: pgmap v8900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:23 smithi132 ceph-mon[37580]: pgmap v8901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:23 smithi112 ceph-mon[37501]: pgmap v8901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:23 smithi107 ceph-mon[32659]: pgmap v8901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:25 smithi112 ceph-mon[37501]: pgmap v8902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16042.112887 secs 2024-07-26T17:18:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:25 smithi107 ceph-mon[32659]: pgmap v8902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16042.112887 secs 2024-07-26T17:18:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:25 smithi132 ceph-mon[37580]: pgmap v8902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16042.112887 secs 2024-07-26T17:18:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:27 smithi112 ceph-mon[37501]: pgmap v8903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:27 smithi107 ceph-mon[32659]: pgmap v8903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:27 smithi132 ceph-mon[37580]: pgmap v8903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:29 smithi112 ceph-mon[37501]: pgmap v8904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:29 smithi107 ceph-mon[32659]: pgmap v8904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:29 smithi132 ceph-mon[37580]: pgmap v8904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16047.112996 secs 2024-07-26T17:18:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16047.112996 secs 2024-07-26T17:18:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16047.112996 secs 2024-07-26T17:18:31.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:31 smithi112 ceph-mon[37501]: pgmap v8905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:31.962 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:31 smithi132 ceph-mon[37580]: pgmap v8905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:31 smithi107 ceph-mon[32659]: pgmap v8905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:33 smithi112 ceph-mon[37501]: pgmap v8906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:33 smithi107 ceph-mon[32659]: pgmap v8906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:33 smithi132 ceph-mon[37580]: pgmap v8906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:34.911+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:18:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:34.911+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:18:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:34.911+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:18:34.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:34 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:18:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:34 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:18:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:34 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:18:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:35 smithi112 ceph-mon[37501]: pgmap v8907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16052.113149 secs 2024-07-26T17:18:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:18:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:35 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:18:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:35 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:18:36.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:35 smithi107 ceph-mon[32659]: pgmap v8907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:36.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16052.113149 secs 2024-07-26T17:18:36.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:18:36.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:35 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:18:36.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:35 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:18:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:35 smithi132 ceph-mon[37580]: pgmap v8907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16052.113149 secs 2024-07-26T17:18:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:18:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:35 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:18:36.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:35 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:18:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:36 smithi112 ceph-mon[37501]: pgmap v8908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:18:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:36 smithi107 ceph-mon[32659]: pgmap v8908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:18:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:36 smithi132 ceph-mon[37580]: pgmap v8908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:18:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:38 smithi112 ceph-mon[37501]: pgmap v8909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:18:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:38 smithi107 ceph-mon[32659]: pgmap v8909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:18:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:38 smithi132 ceph-mon[37580]: pgmap v8909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:18:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:40 smithi112 ceph-mon[37501]: pgmap v8910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:18:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16057.113272 secs 2024-07-26T17:18:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:40 smithi107 ceph-mon[32659]: pgmap v8910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:18:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16057.113272 secs 2024-07-26T17:18:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:40 smithi132 ceph-mon[37580]: pgmap v8910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:18:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16057.113272 secs 2024-07-26T17:18:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:42 smithi112 ceph-mon[37501]: pgmap v8911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:18:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:42 smithi107 ceph-mon[32659]: pgmap v8911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:18:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:42 smithi132 ceph-mon[37580]: pgmap v8911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:18:44.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:44 smithi112 ceph-mon[37501]: pgmap v8912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:18:45.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:44 smithi107 ceph-mon[32659]: pgmap v8912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:18:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:44 smithi132 ceph-mon[37580]: pgmap v8912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:18:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16062.113353 secs 2024-07-26T17:18:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16062.113353 secs 2024-07-26T17:18:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16062.113353 secs 2024-07-26T17:18:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:46 smithi112 ceph-mon[37501]: pgmap v8913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:18:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:46 smithi107 ceph-mon[32659]: pgmap v8913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:18:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:46 smithi132 ceph-mon[37580]: pgmap v8913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:18:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:48 smithi112 ceph-mon[37501]: pgmap v8914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:48 smithi107 ceph-mon[32659]: pgmap v8914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:48 smithi132 ceph-mon[37580]: pgmap v8914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:49.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:49.926+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:18:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:49.926+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:18:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:18:49.926+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:18:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:50 smithi112 ceph-mon[37501]: pgmap v8915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:18:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16067.113479 secs 2024-07-26T17:18:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:50 smithi107 ceph-mon[32659]: pgmap v8915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:18:51.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16067.113479 secs 2024-07-26T17:18:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:50 smithi132 ceph-mon[37580]: pgmap v8915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:18:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16067.113479 secs 2024-07-26T17:18:52.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:52 smithi112 ceph-mon[37501]: pgmap v8916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:52 smithi107 ceph-mon[32659]: pgmap v8916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:52 smithi132 ceph-mon[37580]: pgmap v8916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:18:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:54 smithi112 ceph-mon[37501]: pgmap v8917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:54 smithi107 ceph-mon[32659]: pgmap v8917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:54 smithi132 ceph-mon[37580]: pgmap v8917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16072.113582 secs 2024-07-26T17:18:55.962 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16072.113582 secs 2024-07-26T17:18:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16072.113582 secs 2024-07-26T17:18:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:56 smithi112 ceph-mon[37501]: pgmap v8918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:56 smithi107 ceph-mon[32659]: pgmap v8918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:56 smithi132 ceph-mon[37580]: pgmap v8918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:18:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:18:58 smithi112 ceph-mon[37501]: pgmap v8919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:18:58 smithi107 ceph-mon[32659]: pgmap v8919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:18:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:18:58 smithi132 ceph-mon[37580]: pgmap v8919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:00 smithi107 ceph-mon[32659]: pgmap v8920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16077.113701 secs 2024-07-26T17:19:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:00 smithi132 ceph-mon[37580]: pgmap v8920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16077.113701 secs 2024-07-26T17:19:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:00 smithi112 ceph-mon[37501]: pgmap v8920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16077.113701 secs 2024-07-26T17:19:03.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:02 smithi112 ceph-mon[37501]: pgmap v8921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:02 smithi107 ceph-mon[32659]: pgmap v8921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:02 smithi132 ceph-mon[37580]: pgmap v8921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:04.944 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:04.941+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:19:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:04.941+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:19:04.945 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:04.941+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:19:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:04 smithi107 ceph-mon[32659]: pgmap v8922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:04 smithi132 ceph-mon[37580]: pgmap v8922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:04 smithi112 ceph-mon[37501]: pgmap v8922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16082.113837 secs 2024-07-26T17:19:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16082.113837 secs 2024-07-26T17:19:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16082.113837 secs 2024-07-26T17:19:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:06 smithi107 ceph-mon[32659]: pgmap v8923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:06 smithi132 ceph-mon[37580]: pgmap v8923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:06 smithi112 ceph-mon[37501]: pgmap v8923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:08 smithi107 ceph-mon[32659]: pgmap v8924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:08 smithi132 ceph-mon[37580]: pgmap v8924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:08 smithi112 ceph-mon[37501]: pgmap v8924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:10 smithi107 ceph-mon[32659]: pgmap v8925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:11.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16087.113955 secs 2024-07-26T17:19:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:10 smithi132 ceph-mon[37580]: pgmap v8925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16087.113955 secs 2024-07-26T17:19:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:10 smithi112 ceph-mon[37501]: pgmap v8925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16087.113955 secs 2024-07-26T17:19:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:12 smithi107 ceph-mon[32659]: pgmap v8926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:12 smithi132 ceph-mon[37580]: pgmap v8926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:12 smithi112 ceph-mon[37501]: pgmap v8926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:14 smithi107 ceph-mon[32659]: pgmap v8927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:14 smithi132 ceph-mon[37580]: pgmap v8927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:14 smithi112 ceph-mon[37501]: pgmap v8927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16092.114099 secs 2024-07-26T17:19:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16092.114099 secs 2024-07-26T17:19:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16092.114099 secs 2024-07-26T17:19:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:16 smithi107 ceph-mon[32659]: pgmap v8928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:16 smithi132 ceph-mon[37580]: pgmap v8928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:16 smithi112 ceph-mon[37501]: pgmap v8928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:18 smithi107 ceph-mon[32659]: pgmap v8929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:19.063 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:18 smithi132 ceph-mon[37580]: pgmap v8929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:18 smithi112 ceph-mon[37501]: pgmap v8929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:19.959 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:19.958+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:19:19.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:19.958+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:19:19.960 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:19.958+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:19:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:21 smithi112 ceph-mon[37501]: pgmap v8930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16097.114211 secs 2024-07-26T17:19:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:21 smithi107 ceph-mon[32659]: pgmap v8930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16097.114211 secs 2024-07-26T17:19:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:21 smithi132 ceph-mon[37580]: pgmap v8930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16097.114211 secs 2024-07-26T17:19:23.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:23 smithi112 ceph-mon[37501]: pgmap v8931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:23 smithi107 ceph-mon[32659]: pgmap v8931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:23 smithi132 ceph-mon[37580]: pgmap v8931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:25 smithi112 ceph-mon[37501]: pgmap v8932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:25 smithi107 ceph-mon[32659]: pgmap v8932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:25 smithi132 ceph-mon[37580]: pgmap v8932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16102.114329 secs 2024-07-26T17:19:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16102.114329 secs 2024-07-26T17:19:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16102.114329 secs 2024-07-26T17:19:27.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:27 smithi107 ceph-mon[32659]: pgmap v8933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:27 smithi112 ceph-mon[37501]: pgmap v8933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:27 smithi132 ceph-mon[37580]: pgmap v8933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:29 smithi112 ceph-mon[37501]: pgmap v8934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:29 smithi107 ceph-mon[32659]: pgmap v8934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:29 smithi132 ceph-mon[37580]: pgmap v8934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:31 smithi112 ceph-mon[37501]: pgmap v8935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16107.114451 secs 2024-07-26T17:19:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:31 smithi107 ceph-mon[32659]: pgmap v8935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16107.114451 secs 2024-07-26T17:19:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:31 smithi132 ceph-mon[37580]: pgmap v8935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16107.114451 secs 2024-07-26T17:19:33.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:33 smithi112 ceph-mon[37501]: pgmap v8936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:33 smithi107 ceph-mon[32659]: pgmap v8936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:33 smithi132 ceph-mon[37580]: pgmap v8936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:34.968+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:19:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:34.968+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:19:34.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:34.968+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:19:35.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:35 smithi107 ceph-mon[32659]: pgmap v8937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:35 smithi132 ceph-mon[37580]: pgmap v8937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:35 smithi112 ceph-mon[37501]: pgmap v8937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16112.114533 secs 2024-07-26T17:19:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:36 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:19:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16112.114533 secs 2024-07-26T17:19:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:36 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:19:36.616 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16112.114533 secs 2024-07-26T17:19:36.616 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:36 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:19:37.475 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:37 smithi107 ceph-mon[32659]: pgmap v8938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:37 smithi132 ceph-mon[37580]: pgmap v8938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:37.632 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:37 smithi112 ceph-mon[37501]: pgmap v8938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:39.379 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:39 smithi112 ceph-mon[37501]: pgmap v8939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:39.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:39 smithi107 ceph-mon[32659]: pgmap v8939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:39 smithi132 ceph-mon[37580]: pgmap v8939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:40.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:40 smithi107 ceph-mon[32659]: pgmap v8940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:40 smithi132 ceph-mon[37580]: pgmap v8940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:40 smithi112 ceph-mon[37501]: pgmap v8940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:19:41.697 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16117.114651 secs 2024-07-26T17:19:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16117.114651 secs 2024-07-26T17:19:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16117.114651 secs 2024-07-26T17:19:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: pgmap v8941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:19:43.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:19:43.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:19:43.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:19:43.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: pgmap v8942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:19:43.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: pgmap v8943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:19:43.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: pgmap v8941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:19:43.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:19:43.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:19:43.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:19:43.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: pgmap v8942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:19:43.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: pgmap v8943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:19:43.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: pgmap v8941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:19:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:19:43.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:19:43.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:19:43.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:19:43.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:19:43.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: pgmap v8942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:19:43.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: pgmap v8943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:19:43.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:19:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:45 smithi112 ceph-mon[37501]: pgmap v8944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:45.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:45 smithi107 ceph-mon[32659]: pgmap v8944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:45 smithi132 ceph-mon[37580]: pgmap v8944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16122.114773 secs 2024-07-26T17:19:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16122.114773 secs 2024-07-26T17:19:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16122.114773 secs 2024-07-26T17:19:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:47 smithi112 ceph-mon[37501]: pgmap v8945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:19:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:47 smithi107 ceph-mon[32659]: pgmap v8945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:19:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:47 smithi132 ceph-mon[37580]: pgmap v8945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-07-26T17:19:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:49 smithi107 ceph-mon[32659]: pgmap v8946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:19:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:49 smithi132 ceph-mon[37580]: pgmap v8946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:19:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:49 smithi112 ceph-mon[37501]: pgmap v8946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:19:49.985 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:49.982+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:19:49.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:49.982+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:19:49.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:19:49.982+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:19:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:51 smithi107 ceph-mon[32659]: pgmap v8947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:19:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16127.114892 secs 2024-07-26T17:19:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:51 smithi132 ceph-mon[37580]: pgmap v8947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:19:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16127.114892 secs 2024-07-26T17:19:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:51 smithi112 ceph-mon[37501]: pgmap v8947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:19:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16127.114892 secs 2024-07-26T17:19:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:53 smithi107 ceph-mon[32659]: pgmap v8948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:19:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:53 smithi132 ceph-mon[37580]: pgmap v8948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:19:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:53 smithi112 ceph-mon[37501]: pgmap v8948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:19:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:55 smithi107 ceph-mon[32659]: pgmap v8949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:55 smithi132 ceph-mon[37580]: pgmap v8949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:55 smithi112 ceph-mon[37501]: pgmap v8949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:56.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16132.115040 secs 2024-07-26T17:19:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16132.115040 secs 2024-07-26T17:19:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16132.115040 secs 2024-07-26T17:19:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:57 smithi107 ceph-mon[32659]: pgmap v8950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:57 smithi132 ceph-mon[37580]: pgmap v8950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:57 smithi112 ceph-mon[37501]: pgmap v8950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:19:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:19:59 smithi107 ceph-mon[32659]: pgmap v8951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:19:59 smithi132 ceph-mon[37580]: pgmap v8951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:19:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:19:59 smithi112 ceph-mon[37501]: pgmap v8951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:20:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:20:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:20:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:01 smithi132 ceph-mon[37580]: pgmap v8952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:01.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16137.115154 secs 2024-07-26T17:20:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:01 smithi112 ceph-mon[37501]: pgmap v8952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16137.115154 secs 2024-07-26T17:20:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:01 smithi107 ceph-mon[32659]: pgmap v8952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16137.115154 secs 2024-07-26T17:20:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:03 smithi112 ceph-mon[37501]: pgmap v8953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:03 smithi107 ceph-mon[32659]: pgmap v8953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:03 smithi132 ceph-mon[37580]: pgmap v8953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:04 smithi112 ceph-mon[37501]: pgmap v8954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:04 smithi107 ceph-mon[32659]: pgmap v8954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:04 smithi132 ceph-mon[37580]: pgmap v8954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:04.993+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:20:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:04.993+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:20:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:04.993+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:20:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16142.115282 secs 2024-07-26T17:20:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16142.115282 secs 2024-07-26T17:20:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16142.115282 secs 2024-07-26T17:20:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:06 smithi112 ceph-mon[37501]: pgmap v8955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:06 smithi107 ceph-mon[32659]: pgmap v8955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:06 smithi132 ceph-mon[37580]: pgmap v8955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:09 smithi107 ceph-mon[32659]: pgmap v8956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:09 smithi132 ceph-mon[37580]: pgmap v8956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:09 smithi112 ceph-mon[37501]: pgmap v8956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:11 smithi107 ceph-mon[32659]: pgmap v8957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16147.115403 secs 2024-07-26T17:20:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:11 smithi132 ceph-mon[37580]: pgmap v8957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16147.115403 secs 2024-07-26T17:20:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:11 smithi112 ceph-mon[37501]: pgmap v8957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16147.115403 secs 2024-07-26T17:20:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:13 smithi107 ceph-mon[32659]: pgmap v8958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:13 smithi132 ceph-mon[37580]: pgmap v8958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:13 smithi112 ceph-mon[37501]: pgmap v8958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:15 smithi107 ceph-mon[32659]: pgmap v8959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:15 smithi132 ceph-mon[37580]: pgmap v8959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:15 smithi112 ceph-mon[37501]: pgmap v8959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16152.115492 secs 2024-07-26T17:20:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16152.115492 secs 2024-07-26T17:20:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16152.115492 secs 2024-07-26T17:20:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:17 smithi132 ceph-mon[37580]: pgmap v8960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:17.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:17 smithi112 ceph-mon[37501]: pgmap v8960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:17 smithi107 ceph-mon[32659]: pgmap v8960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:19.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:19 smithi112 ceph-mon[37501]: pgmap v8961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:19 smithi107 ceph-mon[32659]: pgmap v8961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:19 smithi132 ceph-mon[37580]: pgmap v8961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:20.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:20.002+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:20:20.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:20.002+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:20:20.004 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:20.002+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:20:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:21 smithi112 ceph-mon[37501]: pgmap v8962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16157.115631 secs 2024-07-26T17:20:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:21 smithi107 ceph-mon[32659]: pgmap v8962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16157.115631 secs 2024-07-26T17:20:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:21 smithi132 ceph-mon[37580]: pgmap v8962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16157.115631 secs 2024-07-26T17:20:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:22 smithi112 ceph-mon[37501]: pgmap v8963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:22 smithi107 ceph-mon[32659]: pgmap v8963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:22 smithi132 ceph-mon[37580]: pgmap v8963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:25.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:25 smithi107 ceph-mon[32659]: pgmap v8964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:25 smithi132 ceph-mon[37580]: pgmap v8964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:25 smithi112 ceph-mon[37501]: pgmap v8964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16162.115769 secs 2024-07-26T17:20:26.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16162.115769 secs 2024-07-26T17:20:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16162.115769 secs 2024-07-26T17:20:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:27 smithi132 ceph-mon[37580]: pgmap v8965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:27 smithi112 ceph-mon[37501]: pgmap v8965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:27 smithi107 ceph-mon[32659]: pgmap v8965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:29 smithi132 ceph-mon[37580]: pgmap v8966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:29 smithi112 ceph-mon[37501]: pgmap v8966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:29 smithi107 ceph-mon[32659]: pgmap v8966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:31.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:31 smithi132 ceph-mon[37580]: pgmap v8967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:31.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16167.115909 secs 2024-07-26T17:20:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:31 smithi112 ceph-mon[37501]: pgmap v8967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16167.115909 secs 2024-07-26T17:20:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:31 smithi107 ceph-mon[32659]: pgmap v8967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16167.115909 secs 2024-07-26T17:20:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:33 smithi112 ceph-mon[37501]: pgmap v8968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:33 smithi107 ceph-mon[32659]: pgmap v8968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:33 smithi132 ceph-mon[37580]: pgmap v8968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:35.019 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:35.018+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:20:35.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:35.018+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:20:35.020 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:35.018+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:20:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:35 smithi112 ceph-mon[37501]: pgmap v8969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:35 smithi107 ceph-mon[32659]: pgmap v8969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:35 smithi132 ceph-mon[37580]: pgmap v8969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16172.116047 secs 2024-07-26T17:20:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16172.116047 secs 2024-07-26T17:20:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16172.116047 secs 2024-07-26T17:20:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:37 smithi112 ceph-mon[37501]: pgmap v8970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:37 smithi107 ceph-mon[32659]: pgmap v8970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:37 smithi132 ceph-mon[37580]: pgmap v8970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:20:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:38 smithi112 ceph-mon[37501]: pgmap v8971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:38 smithi107 ceph-mon[32659]: pgmap v8971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:38 smithi132 ceph-mon[37580]: pgmap v8971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:41 smithi107 ceph-mon[32659]: pgmap v8972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16177.116161 secs 2024-07-26T17:20:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:41 smithi132 ceph-mon[37580]: pgmap v8972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16177.116161 secs 2024-07-26T17:20:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:41 smithi112 ceph-mon[37501]: pgmap v8972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16177.116161 secs 2024-07-26T17:20:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:43 smithi132 ceph-mon[37580]: pgmap v8973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:20:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:43 smithi112 ceph-mon[37501]: pgmap v8973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:20:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:43 smithi107 ceph-mon[32659]: pgmap v8973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:20:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:20:43.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:43.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:44 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:20:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:44 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:20:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:44 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:20:44.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:44 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:20:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:44 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:20:44.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:44 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:20:45.690 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:45 smithi132 ceph-mon[37580]: pgmap v8974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:45 smithi112 ceph-mon[37501]: pgmap v8974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:45 smithi107 ceph-mon[32659]: pgmap v8974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:46.627 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16182.116247 secs 2024-07-26T17:20:46.628 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:20:46.628 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:20:46.628 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.628 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.628 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.628 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.629 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.629 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.629 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.629 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.629 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:20:46.629 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:20:46.629 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:20:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16182.116247 secs 2024-07-26T17:20:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:20:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:20:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:20:46.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:20:46.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:20:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16182.116247 secs 2024-07-26T17:20:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:20:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:20:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:20:46.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:20:46.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:20:46.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:20:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:47 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:20:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:47 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:20:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:47 smithi112 ceph-mon[37501]: pgmap v8975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T17:20:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:47 smithi112 ceph-mon[37501]: pgmap v8976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:47 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:20:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:47 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:20:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:47 smithi107 ceph-mon[32659]: pgmap v8975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T17:20:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:47 smithi107 ceph-mon[32659]: pgmap v8976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:47 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:20:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:47 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:20:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:47 smithi132 ceph-mon[37580]: pgmap v8975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T17:20:47.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:47 smithi132 ceph-mon[37580]: pgmap v8976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:49 smithi112 ceph-mon[37501]: pgmap v8977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:49 smithi107 ceph-mon[32659]: pgmap v8977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:49 smithi132 ceph-mon[37580]: pgmap v8977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:50.032+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:50.032+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:20:50.036 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:20:50.032+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:20:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:50 smithi112 ceph-mon[37501]: pgmap v8978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:50 smithi107 ceph-mon[32659]: pgmap v8978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:50 smithi132 ceph-mon[37580]: pgmap v8978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16187.116370 secs 2024-07-26T17:20:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16187.116370 secs 2024-07-26T17:20:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16187.116370 secs 2024-07-26T17:20:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:52 smithi112 ceph-mon[37501]: pgmap v8979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:52 smithi107 ceph-mon[32659]: pgmap v8979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:52 smithi132 ceph-mon[37580]: pgmap v8979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:20:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:55 smithi107 ceph-mon[32659]: pgmap v8980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:20:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:55 smithi132 ceph-mon[37580]: pgmap v8980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:20:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:55 smithi112 ceph-mon[37501]: pgmap v8980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:20:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16192.116495 secs 2024-07-26T17:20:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16192.116495 secs 2024-07-26T17:20:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16192.116495 secs 2024-07-26T17:20:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:57 smithi107 ceph-mon[32659]: pgmap v8981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:20:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:57 smithi132 ceph-mon[37580]: pgmap v8981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:20:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:57 smithi112 ceph-mon[37501]: pgmap v8981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:20:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:20:59 smithi107 ceph-mon[32659]: pgmap v8982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:20:59 smithi132 ceph-mon[37580]: pgmap v8982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:20:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:20:59 smithi112 ceph-mon[37501]: pgmap v8982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:01 smithi107 ceph-mon[32659]: pgmap v8983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16197.116590 secs 2024-07-26T17:21:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:01 smithi132 ceph-mon[37580]: pgmap v8983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16197.116590 secs 2024-07-26T17:21:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:01 smithi112 ceph-mon[37501]: pgmap v8983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16197.116590 secs 2024-07-26T17:21:03.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:03 smithi112 ceph-mon[37501]: pgmap v8984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:03 smithi107 ceph-mon[32659]: pgmap v8984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:03 smithi132 ceph-mon[37580]: pgmap v8984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:05.048 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:05.045+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:21:05.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:05.045+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:21:05.049 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:05.045+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:21:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:05 smithi107 ceph-mon[32659]: pgmap v8985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:05 smithi132 ceph-mon[37580]: pgmap v8985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:05 smithi112 ceph-mon[37501]: pgmap v8985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16202.116731 secs 2024-07-26T17:21:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16202.116731 secs 2024-07-26T17:21:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16202.116731 secs 2024-07-26T17:21:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:07 smithi107 ceph-mon[32659]: pgmap v8986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:07 smithi132 ceph-mon[37580]: pgmap v8986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:07 smithi112 ceph-mon[37501]: pgmap v8986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:09 smithi107 ceph-mon[32659]: pgmap v8987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:09 smithi132 ceph-mon[37580]: pgmap v8987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:09 smithi112 ceph-mon[37501]: pgmap v8987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:11 smithi107 ceph-mon[32659]: pgmap v8988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16207.116866 secs 2024-07-26T17:21:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:11 smithi132 ceph-mon[37580]: pgmap v8988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16207.116866 secs 2024-07-26T17:21:11.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:11.690+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T17:21:11.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:11.690+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T17:21:11.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:11.690+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f8067040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T17:21:11.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:11.690+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f8067040 comp rx=0 tx=0).stop 2024-07-26T17:21:11.693 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:11.690+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T17:21:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:11.690+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T17:21:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:11.690+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T17:21:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:11.690+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T17:21:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:11.691+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T17:21:11.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:11.693+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8082040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T17:21:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:11 smithi112 ceph-mon[37501]: pgmap v8988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16207.116866 secs 2024-07-26T17:21:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:13 smithi107 ceph-mon[32659]: pgmap v8989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:13 smithi132 ceph-mon[37580]: pgmap v8989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:13 smithi112 ceph-mon[37501]: pgmap v8989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:15.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:15.436+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:21:15.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:15.436+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:21:15.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:15.437+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:21:15.438 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:15.437+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:21:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:15 smithi107 ceph-mon[32659]: pgmap v8990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:15 smithi132 ceph-mon[37580]: pgmap v8990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:15 smithi112 ceph-mon[37501]: pgmap v8990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16212.116987 secs 2024-07-26T17:21:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16212.116987 secs 2024-07-26T17:21:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16212.116987 secs 2024-07-26T17:21:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:17 smithi132 ceph-mon[37580]: pgmap v8991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:17 smithi112 ceph-mon[37501]: pgmap v8991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:17 smithi107 ceph-mon[32659]: pgmap v8991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:19 smithi132 ceph-mon[37580]: pgmap v8992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:19 smithi112 ceph-mon[37501]: pgmap v8992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:19 smithi107 ceph-mon[32659]: pgmap v8992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:20.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:20.055+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:21:20.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:20.055+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:21:20.057 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:20.055+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:21:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:21 smithi112 ceph-mon[37501]: pgmap v8993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16217.117108 secs 2024-07-26T17:21:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:21 smithi107 ceph-mon[32659]: pgmap v8993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16217.117108 secs 2024-07-26T17:21:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:21 smithi132 ceph-mon[37580]: pgmap v8993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16217.117108 secs 2024-07-26T17:21:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:23 smithi112 ceph-mon[37501]: pgmap v8994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:23 smithi107 ceph-mon[32659]: pgmap v8994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:23 smithi132 ceph-mon[37580]: pgmap v8994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:25 smithi112 ceph-mon[37501]: pgmap v8995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:21:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:25 smithi107 ceph-mon[32659]: pgmap v8995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:21:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:25 smithi132 ceph-mon[37580]: pgmap v8995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:21:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16222.117204 secs 2024-07-26T17:21:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16222.117204 secs 2024-07-26T17:21:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16222.117204 secs 2024-07-26T17:21:27.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:27 smithi107 ceph-mon[32659]: pgmap v8996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:21:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:27 smithi112 ceph-mon[37501]: pgmap v8996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:21:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:27 smithi132 ceph-mon[37580]: pgmap v8996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:21:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:28 smithi112 ceph-mon[37501]: pgmap v8997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:21:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:28 smithi107 ceph-mon[32659]: pgmap v8997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:21:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:28 smithi132 ceph-mon[37580]: pgmap v8997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:21:31.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:31 smithi107 ceph-mon[32659]: pgmap v8998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:31.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16227.117328 secs 2024-07-26T17:21:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:31 smithi132 ceph-mon[37580]: pgmap v8998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16227.117328 secs 2024-07-26T17:21:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:31 smithi112 ceph-mon[37501]: pgmap v8998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16227.117328 secs 2024-07-26T17:21:33.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:33 smithi112 ceph-mon[37501]: pgmap v8999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:33 smithi107 ceph-mon[32659]: pgmap v8999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:33 smithi132 ceph-mon[37580]: pgmap v8999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:35.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:35.058+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:21:35.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:35.058+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:21:35.060 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:35.058+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:21:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:35 smithi107 ceph-mon[32659]: pgmap v9000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:35 smithi132 ceph-mon[37580]: pgmap v9000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:35 smithi112 ceph-mon[37501]: pgmap v9000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16232.117449 secs 2024-07-26T17:21:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16232.117449 secs 2024-07-26T17:21:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16232.117449 secs 2024-07-26T17:21:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:37 smithi107 ceph-mon[32659]: pgmap v9001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:37 smithi132 ceph-mon[37580]: pgmap v9001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:37 smithi112 ceph-mon[37501]: pgmap v9001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:39 smithi107 ceph-mon[32659]: pgmap v9002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:39 smithi132 ceph-mon[37580]: pgmap v9002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:39 smithi112 ceph-mon[37501]: pgmap v9002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:41 smithi107 ceph-mon[32659]: pgmap v9003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16237.117567 secs 2024-07-26T17:21:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:41 smithi132 ceph-mon[37580]: pgmap v9003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16237.117567 secs 2024-07-26T17:21:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:41 smithi112 ceph-mon[37501]: pgmap v9003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16237.117567 secs 2024-07-26T17:21:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:43 smithi107 ceph-mon[32659]: pgmap v9004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:43 smithi132 ceph-mon[37580]: pgmap v9004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:43 smithi112 ceph-mon[37501]: pgmap v9004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:21:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:45 smithi107 ceph-mon[32659]: pgmap v9005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:45 smithi132 ceph-mon[37580]: pgmap v9005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:45 smithi112 ceph-mon[37501]: pgmap v9005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:21:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16242.117642 secs 2024-07-26T17:21:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16242.117642 secs 2024-07-26T17:21:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16242.117642 secs 2024-07-26T17:21:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:47 smithi132 ceph-mon[37580]: pgmap v9006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:47 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:21:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:47 smithi112 ceph-mon[37501]: pgmap v9006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:47 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:21:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:47 smithi107 ceph-mon[32659]: pgmap v9006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:21:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:47 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:21:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:48 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:21:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:48 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:21:48.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:48 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:21:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:48 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:21:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:48 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:21:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:48 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:21:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:48 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:21:48.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:48 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:21:48.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:48 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:21:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:49 smithi132 ceph-mon[37580]: pgmap v9007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:21:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:49 smithi132 ceph-mon[37580]: pgmap v9008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:21:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:49 smithi112 ceph-mon[37501]: pgmap v9007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:21:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:49 smithi112 ceph-mon[37501]: pgmap v9008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:21:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:49 smithi107 ceph-mon[32659]: pgmap v9007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:21:49.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:49 smithi107 ceph-mon[32659]: pgmap v9008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:21:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:50.072+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:21:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:50.072+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:21:50.075 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:21:50.072+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:21:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:51 smithi112 ceph-mon[37501]: pgmap v9009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:21:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16247.117784 secs 2024-07-26T17:21:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:51 smithi107 ceph-mon[32659]: pgmap v9009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:21:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16247.117784 secs 2024-07-26T17:21:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:51 smithi132 ceph-mon[37580]: pgmap v9009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:21:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16247.117784 secs 2024-07-26T17:21:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:53 smithi112 ceph-mon[37501]: pgmap v9010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:21:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:53 smithi107 ceph-mon[32659]: pgmap v9010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:21:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:53 smithi132 ceph-mon[37580]: pgmap v9010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:21:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:55 smithi112 ceph-mon[37501]: pgmap v9011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:21:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:55 smithi107 ceph-mon[32659]: pgmap v9011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:21:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:55 smithi132 ceph-mon[37580]: pgmap v9011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:21:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16252.117907 secs 2024-07-26T17:21:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16252.117907 secs 2024-07-26T17:21:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16252.117907 secs 2024-07-26T17:21:57.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:57 smithi107 ceph-mon[32659]: pgmap v9012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:21:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:57 smithi112 ceph-mon[37501]: pgmap v9012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:21:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:57 smithi132 ceph-mon[37580]: pgmap v9012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:21:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:21:58 smithi112 ceph-mon[37501]: pgmap v9013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:21:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:21:58 smithi107 ceph-mon[32659]: pgmap v9013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:21:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:21:58 smithi132 ceph-mon[37580]: pgmap v9013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:22:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:00 smithi112 ceph-mon[37501]: pgmap v9014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:00 smithi107 ceph-mon[32659]: pgmap v9014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:00 smithi132 ceph-mon[37580]: pgmap v9014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16257.118031 secs 2024-07-26T17:22:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16257.118031 secs 2024-07-26T17:22:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16257.118031 secs 2024-07-26T17:22:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:02 smithi112 ceph-mon[37501]: pgmap v9015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:22:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:02 smithi107 ceph-mon[32659]: pgmap v9015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:22:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:02 smithi132 ceph-mon[37580]: pgmap v9015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:22:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:04 smithi112 ceph-mon[37501]: pgmap v9016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:22:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:04 smithi107 ceph-mon[32659]: pgmap v9016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:22:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:04 smithi132 ceph-mon[37580]: pgmap v9016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:22:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:05.081+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:22:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:05.081+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:22:05.084 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:05.081+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:22:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:06 smithi112 ceph-mon[37501]: pgmap v9017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16262.118111 secs 2024-07-26T17:22:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:06 smithi107 ceph-mon[32659]: pgmap v9017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16262.118111 secs 2024-07-26T17:22:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:06 smithi132 ceph-mon[37580]: pgmap v9017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16262.118111 secs 2024-07-26T17:22:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:08 smithi107 ceph-mon[32659]: pgmap v9018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:22:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:08 smithi132 ceph-mon[37580]: pgmap v9018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:22:08.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:08 smithi112 ceph-mon[37501]: pgmap v9018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:22:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:11 smithi107 ceph-mon[32659]: pgmap v9019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16267.118244 secs 2024-07-26T17:22:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:11 smithi132 ceph-mon[37580]: pgmap v9019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16267.118244 secs 2024-07-26T17:22:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:11 smithi112 ceph-mon[37501]: pgmap v9019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16267.118244 secs 2024-07-26T17:22:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:13 smithi107 ceph-mon[32659]: pgmap v9020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:13 smithi132 ceph-mon[37580]: pgmap v9020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:13 smithi112 ceph-mon[37501]: pgmap v9020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:15 smithi107 ceph-mon[32659]: pgmap v9021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:15 smithi132 ceph-mon[37580]: pgmap v9021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:15 smithi112 ceph-mon[37501]: pgmap v9021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16272.118337 secs 2024-07-26T17:22:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16272.118337 secs 2024-07-26T17:22:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16272.118337 secs 2024-07-26T17:22:17.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:17 smithi107 ceph-mon[32659]: pgmap v9022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:17 smithi132 ceph-mon[37580]: pgmap v9022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:17 smithi112 ceph-mon[37501]: pgmap v9022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:19 smithi107 ceph-mon[32659]: pgmap v9023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:19.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:19 smithi132 ceph-mon[37580]: pgmap v9023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:19 smithi112 ceph-mon[37501]: pgmap v9023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:20.099 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:20.098+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:22:20.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:20.098+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:22:20.100 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:20.098+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:22:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:21 smithi107 ceph-mon[32659]: pgmap v9024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16277.118402 secs 2024-07-26T17:22:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:21 smithi132 ceph-mon[37580]: pgmap v9024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16277.118402 secs 2024-07-26T17:22:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:21 smithi112 ceph-mon[37501]: pgmap v9024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16277.118402 secs 2024-07-26T17:22:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:23 smithi107 ceph-mon[32659]: pgmap v9025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:23 smithi132 ceph-mon[37580]: pgmap v9025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:23 smithi112 ceph-mon[37501]: pgmap v9025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:25 smithi107 ceph-mon[32659]: pgmap v9026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:25 smithi132 ceph-mon[37580]: pgmap v9026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:25 smithi112 ceph-mon[37501]: pgmap v9026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16282.118536 secs 2024-07-26T17:22:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16282.118536 secs 2024-07-26T17:22:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16282.118536 secs 2024-07-26T17:22:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:27 smithi132 ceph-mon[37580]: pgmap v9027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:27.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:27 smithi107 ceph-mon[32659]: pgmap v9027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:27 smithi112 ceph-mon[37501]: pgmap v9027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:29 smithi132 ceph-mon[37580]: pgmap v9028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:29 smithi112 ceph-mon[37501]: pgmap v9028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:29 smithi107 ceph-mon[32659]: pgmap v9028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:31 smithi112 ceph-mon[37501]: pgmap v9029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16287.118658 secs 2024-07-26T17:22:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:31 smithi107 ceph-mon[32659]: pgmap v9029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16287.118658 secs 2024-07-26T17:22:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:31 smithi132 ceph-mon[37580]: pgmap v9029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16287.118658 secs 2024-07-26T17:22:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:33 smithi112 ceph-mon[37501]: pgmap v9030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:33 smithi107 ceph-mon[32659]: pgmap v9030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:33 smithi132 ceph-mon[37580]: pgmap v9030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:35.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:35.109+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:22:35.111 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:35.110+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:22:35.112 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:35.110+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:22:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:35 smithi107 ceph-mon[32659]: pgmap v9031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:35.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:35 smithi132 ceph-mon[37580]: pgmap v9031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:35 smithi112 ceph-mon[37501]: pgmap v9031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16292.118796 secs 2024-07-26T17:22:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16292.118796 secs 2024-07-26T17:22:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16292.118796 secs 2024-07-26T17:22:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:37 smithi112 ceph-mon[37501]: pgmap v9032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:37 smithi107 ceph-mon[32659]: pgmap v9032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:37 smithi132 ceph-mon[37580]: pgmap v9032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:39.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:39 smithi112 ceph-mon[37501]: pgmap v9033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:39 smithi107 ceph-mon[32659]: pgmap v9033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:39 smithi132 ceph-mon[37580]: pgmap v9033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:40 smithi112 ceph-mon[37501]: pgmap v9034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:40 smithi107 ceph-mon[32659]: pgmap v9034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:40 smithi132 ceph-mon[37580]: pgmap v9034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16297.118943 secs 2024-07-26T17:22:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16297.118943 secs 2024-07-26T17:22:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16297.118943 secs 2024-07-26T17:22:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:42 smithi112 ceph-mon[37501]: pgmap v9035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:42 smithi107 ceph-mon[32659]: pgmap v9035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:42 smithi132 ceph-mon[37580]: pgmap v9035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:22:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:44 smithi112 ceph-mon[37501]: pgmap v9036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:44 smithi107 ceph-mon[32659]: pgmap v9036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:44 smithi132 ceph-mon[37580]: pgmap v9036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:46 smithi112 ceph-mon[37501]: pgmap v9037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16302.119004 secs 2024-07-26T17:22:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:46 smithi107 ceph-mon[32659]: pgmap v9037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16302.119004 secs 2024-07-26T17:22:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:46 smithi132 ceph-mon[37580]: pgmap v9037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:22:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16302.119004 secs 2024-07-26T17:22:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:48 smithi112 ceph-mon[37501]: pgmap v9038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:48 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:22:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:48 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:22:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:48 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:22:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:48 smithi107 ceph-mon[32659]: pgmap v9038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:48 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:22:48.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:48 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:22:48.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:48 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:22:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:48 smithi132 ceph-mon[37580]: pgmap v9038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:22:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:48 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:22:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:48 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:22:48.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:48 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:22:49.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:22:49.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:49 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:22:49.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:49 smithi107 ceph-mon[32659]: pgmap v9039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:22:49.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:49 smithi107 ceph-mon[32659]: pgmap v9040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:22:49.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:22:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:22:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:49 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:22:49.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:49 smithi132 ceph-mon[37580]: pgmap v9039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:22:49.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:49 smithi132 ceph-mon[37580]: pgmap v9040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:22:49.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:22:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:22:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:49 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:22:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:49 smithi112 ceph-mon[37501]: pgmap v9039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:22:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:49 smithi112 ceph-mon[37501]: pgmap v9040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:22:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:22:50.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:50.123+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:22:50.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:50.123+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:22:50.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:22:50.123+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:22:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16307.119144 secs 2024-07-26T17:22:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16307.119144 secs 2024-07-26T17:22:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16307.119144 secs 2024-07-26T17:22:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:51 smithi112 ceph-mon[37501]: pgmap v9041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:22:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:51 smithi107 ceph-mon[32659]: pgmap v9041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:22:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:51 smithi132 ceph-mon[37580]: pgmap v9041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:22:54.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:54 smithi107 ceph-mon[32659]: pgmap v9042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:22:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:54 smithi132 ceph-mon[37580]: pgmap v9042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:22:54.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:54 smithi112 ceph-mon[37501]: pgmap v9042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:22:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:56 smithi107 ceph-mon[32659]: pgmap v9043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:22:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16312.119248 secs 2024-07-26T17:22:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:56 smithi132 ceph-mon[37580]: pgmap v9043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:22:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16312.119248 secs 2024-07-26T17:22:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:56 smithi112 ceph-mon[37501]: pgmap v9043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:22:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16312.119248 secs 2024-07-26T17:22:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:22:58 smithi107 ceph-mon[32659]: pgmap v9044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:22:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:22:58 smithi132 ceph-mon[37580]: pgmap v9044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:22:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:22:58 smithi112 ceph-mon[37501]: pgmap v9044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:23:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:00 smithi107 ceph-mon[32659]: pgmap v9045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:23:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:00 smithi132 ceph-mon[37580]: pgmap v9045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:23:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:00 smithi112 ceph-mon[37501]: pgmap v9045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:23:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16317.119392 secs 2024-07-26T17:23:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16317.119392 secs 2024-07-26T17:23:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16317.119392 secs 2024-07-26T17:23:02.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:02 smithi107 ceph-mon[32659]: pgmap v9046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:02 smithi132 ceph-mon[37580]: pgmap v9046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:02 smithi112 ceph-mon[37501]: pgmap v9046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:04 smithi107 ceph-mon[32659]: pgmap v9047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:04 smithi132 ceph-mon[37580]: pgmap v9047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:04 smithi112 ceph-mon[37501]: pgmap v9047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:05.133+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:23:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:05.133+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:23:05.135 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:05.133+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:23:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:06 smithi107 ceph-mon[32659]: pgmap v9048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16322.119495 secs 2024-07-26T17:23:06.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:06 smithi132 ceph-mon[37580]: pgmap v9048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:06.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16322.119495 secs 2024-07-26T17:23:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:06 smithi112 ceph-mon[37501]: pgmap v9048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16322.119495 secs 2024-07-26T17:23:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:08 smithi107 ceph-mon[32659]: pgmap v9049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:08 smithi132 ceph-mon[37580]: pgmap v9049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:08 smithi112 ceph-mon[37501]: pgmap v9049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:10 smithi107 ceph-mon[32659]: pgmap v9050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:10 smithi132 ceph-mon[37580]: pgmap v9050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:10 smithi112 ceph-mon[37501]: pgmap v9050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16327.119621 secs 2024-07-26T17:23:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16327.119621 secs 2024-07-26T17:23:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16327.119621 secs 2024-07-26T17:23:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:12 smithi107 ceph-mon[32659]: pgmap v9051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:12 smithi132 ceph-mon[37580]: pgmap v9051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:12.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:12 smithi112 ceph-mon[37501]: pgmap v9051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:14.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:14 smithi107 ceph-mon[32659]: pgmap v9052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:14 smithi132 ceph-mon[37580]: pgmap v9052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:14 smithi112 ceph-mon[37501]: pgmap v9052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:16 smithi107 ceph-mon[32659]: pgmap v9053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16332.119768 secs 2024-07-26T17:23:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:16 smithi132 ceph-mon[37580]: pgmap v9053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16332.119768 secs 2024-07-26T17:23:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:16 smithi112 ceph-mon[37501]: pgmap v9053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16332.119768 secs 2024-07-26T17:23:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:18 smithi132 ceph-mon[37580]: pgmap v9054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:18 smithi112 ceph-mon[37501]: pgmap v9054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:18 smithi107 ceph-mon[32659]: pgmap v9054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:20.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:20.149+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:23:20.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:20.149+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:23:20.151 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:20.149+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:23:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:20 smithi107 ceph-mon[32659]: pgmap v9055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:20 smithi132 ceph-mon[37580]: pgmap v9055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:20.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:20 smithi112 ceph-mon[37501]: pgmap v9055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16337.119882 secs 2024-07-26T17:23:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16337.119882 secs 2024-07-26T17:23:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16337.119882 secs 2024-07-26T17:23:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:22 smithi112 ceph-mon[37501]: pgmap v9056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:22.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:22 smithi107 ceph-mon[32659]: pgmap v9056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:22 smithi132 ceph-mon[37580]: pgmap v9056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:24 smithi112 ceph-mon[37501]: pgmap v9057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:24 smithi107 ceph-mon[32659]: pgmap v9057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:24 smithi132 ceph-mon[37580]: pgmap v9057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:26 smithi112 ceph-mon[37501]: pgmap v9058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16342.120001 secs 2024-07-26T17:23:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:26 smithi107 ceph-mon[32659]: pgmap v9058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16342.120001 secs 2024-07-26T17:23:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:26 smithi132 ceph-mon[37580]: pgmap v9058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16342.120001 secs 2024-07-26T17:23:28.596 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:28 smithi107 ceph-mon[32659]: pgmap v9059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:28 smithi112 ceph-mon[37501]: pgmap v9059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:28 smithi132 ceph-mon[37580]: pgmap v9059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:29.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:29.459+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:23:29.461 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:29.459+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:23:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:29.460+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:23:29.462 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:29.460+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:23:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:30 smithi112 ceph-mon[37501]: pgmap v9060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:30 smithi107 ceph-mon[32659]: pgmap v9060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:30 smithi132 ceph-mon[37580]: pgmap v9060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16347.120110 secs 2024-07-26T17:23:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16347.120110 secs 2024-07-26T17:23:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16347.120110 secs 2024-07-26T17:23:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:32 smithi112 ceph-mon[37501]: pgmap v9061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:32 smithi107 ceph-mon[32659]: pgmap v9061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:32 smithi132 ceph-mon[37580]: pgmap v9061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:34 smithi112 ceph-mon[37501]: pgmap v9062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:34 smithi107 ceph-mon[32659]: pgmap v9062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:34 smithi132 ceph-mon[37580]: pgmap v9062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:35.166 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:35.165+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:23:35.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:35.165+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:23:35.167 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:35.165+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:23:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:35 smithi107 ceph-mon[32659]: pgmap v9063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:35 smithi112 ceph-mon[37501]: pgmap v9063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:35 smithi132 ceph-mon[37580]: pgmap v9063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16352.120225 secs 2024-07-26T17:23:36.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16352.120225 secs 2024-07-26T17:23:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16352.120225 secs 2024-07-26T17:23:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:37 smithi112 ceph-mon[37501]: pgmap v9064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:37 smithi107 ceph-mon[32659]: pgmap v9064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:37 smithi132 ceph-mon[37580]: pgmap v9064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:40.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:40 smithi107 ceph-mon[32659]: pgmap v9065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:40 smithi132 ceph-mon[37580]: pgmap v9065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:40 smithi112 ceph-mon[37501]: pgmap v9065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16357.120346 secs 2024-07-26T17:23:41.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16357.120346 secs 2024-07-26T17:23:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16357.120346 secs 2024-07-26T17:23:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:42 smithi107 ceph-mon[32659]: pgmap v9066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:42 smithi132 ceph-mon[37580]: pgmap v9066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:42 smithi112 ceph-mon[37501]: pgmap v9066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:44 smithi107 ceph-mon[32659]: pgmap v9067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:44 smithi132 ceph-mon[37580]: pgmap v9067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:44 smithi112 ceph-mon[37501]: pgmap v9067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:46 smithi107 ceph-mon[32659]: pgmap v9068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16362.120485 secs 2024-07-26T17:23:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:46 smithi132 ceph-mon[37580]: pgmap v9068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16362.120485 secs 2024-07-26T17:23:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:46 smithi112 ceph-mon[37501]: pgmap v9068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16362.120485 secs 2024-07-26T17:23:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:48 smithi107 ceph-mon[32659]: pgmap v9069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:48 smithi132 ceph-mon[37580]: pgmap v9069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:48 smithi112 ceph-mon[37501]: pgmap v9069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:23:50.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:50.179+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:23:50.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:50.179+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:23:50.182 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:23:50.179+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:23:50.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:50 smithi107 ceph-mon[32659]: pgmap v9070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:50.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:50 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:23:50.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:50 smithi132 ceph-mon[37580]: pgmap v9070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:50 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:23:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.620 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:50 smithi112 ceph-mon[37501]: pgmap v9070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:50.621 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:50 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:23:50.621 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.621 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.621 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:50.621 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16367.120569 secs 2024-07-26T17:23:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16367.120569 secs 2024-07-26T17:23:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16367.120569 secs 2024-07-26T17:23:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:52 smithi107 ceph-mon[32659]: pgmap v9071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:52 smithi132 ceph-mon[37580]: pgmap v9071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:52 smithi112 ceph-mon[37501]: pgmap v9071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:23:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:52.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:54.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:53 smithi107 ceph-mon[32659]: pgmap v9072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:54.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:54.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:53 smithi132 ceph-mon[37580]: pgmap v9072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:54.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:53 smithi112 ceph-mon[37501]: pgmap v9072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:23:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:23:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:23:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:54 smithi107 ceph-mon[32659]: pgmap v9073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:23:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:23:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:23:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:54 smithi132 ceph-mon[37580]: pgmap v9073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:23:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:23:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:23:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:54 smithi112 ceph-mon[37501]: pgmap v9073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:23:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:23:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16372.120681 secs 2024-07-26T17:23:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16372.120681 secs 2024-07-26T17:23:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16372.120681 secs 2024-07-26T17:23:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:56 smithi107 ceph-mon[32659]: pgmap v9074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:23:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:56 smithi132 ceph-mon[37580]: pgmap v9074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:23:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:56 smithi112 ceph-mon[37501]: pgmap v9074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:23:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:23:58 smithi107 ceph-mon[32659]: pgmap v9075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T17:23:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:23:58 smithi132 ceph-mon[37580]: pgmap v9075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T17:23:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:23:58 smithi112 ceph-mon[37501]: pgmap v9075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T17:24:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:00 smithi107 ceph-mon[32659]: pgmap v9076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:24:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16377.120832 secs 2024-07-26T17:24:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:00 smithi132 ceph-mon[37580]: pgmap v9076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:24:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16377.120832 secs 2024-07-26T17:24:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:00 smithi112 ceph-mon[37501]: pgmap v9076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:24:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16377.120832 secs 2024-07-26T17:24:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:02 smithi107 ceph-mon[32659]: pgmap v9077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:24:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:02 smithi132 ceph-mon[37580]: pgmap v9077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:24:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:02 smithi112 ceph-mon[37501]: pgmap v9077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:24:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:04 smithi107 ceph-mon[32659]: pgmap v9078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:24:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:04 smithi132 ceph-mon[37580]: pgmap v9078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:24:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:05.195+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:24:05.197 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:05.195+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:24:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:05.195+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:24:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:04 smithi112 ceph-mon[37501]: pgmap v9078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:24:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16382.120961 secs 2024-07-26T17:24:06.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16382.120961 secs 2024-07-26T17:24:06.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16382.120961 secs 2024-07-26T17:24:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:06 smithi132 ceph-mon[37580]: pgmap v9079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:06 smithi112 ceph-mon[37501]: pgmap v9079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:06 smithi107 ceph-mon[32659]: pgmap v9079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:08 smithi132 ceph-mon[37580]: pgmap v9080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:08 smithi112 ceph-mon[37501]: pgmap v9080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:08 smithi107 ceph-mon[32659]: pgmap v9080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:10 smithi112 ceph-mon[37501]: pgmap v9081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16387.121097 secs 2024-07-26T17:24:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:10 smithi107 ceph-mon[32659]: pgmap v9081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16387.121097 secs 2024-07-26T17:24:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:10 smithi132 ceph-mon[37580]: pgmap v9081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16387.121097 secs 2024-07-26T17:24:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:12 smithi112 ceph-mon[37501]: pgmap v9082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:12 smithi107 ceph-mon[32659]: pgmap v9082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:12 smithi132 ceph-mon[37580]: pgmap v9082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:14 smithi112 ceph-mon[37501]: pgmap v9083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:14 smithi107 ceph-mon[32659]: pgmap v9083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:14 smithi132 ceph-mon[37580]: pgmap v9083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:15.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:15.468+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:24:15.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:15.468+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:24:15.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:15.468+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:24:15.470 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:15.468+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:24:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16392.121212 secs 2024-07-26T17:24:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16392.121212 secs 2024-07-26T17:24:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16392.121212 secs 2024-07-26T17:24:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:16 smithi112 ceph-mon[37501]: pgmap v9084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:16 smithi107 ceph-mon[32659]: pgmap v9084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:16 smithi132 ceph-mon[37580]: pgmap v9084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:18 smithi112 ceph-mon[37501]: pgmap v9085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:18 smithi107 ceph-mon[32659]: pgmap v9085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:18 smithi132 ceph-mon[37580]: pgmap v9085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:20.211+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:24:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:20.211+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:24:20.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:20.211+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:24:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:20 smithi112 ceph-mon[37501]: pgmap v9086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16397.121329 secs 2024-07-26T17:24:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:20 smithi107 ceph-mon[32659]: pgmap v9086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16397.121329 secs 2024-07-26T17:24:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:20 smithi132 ceph-mon[37580]: pgmap v9086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16397.121329 secs 2024-07-26T17:24:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:22 smithi112 ceph-mon[37501]: pgmap v9087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:22 smithi107 ceph-mon[32659]: pgmap v9087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:22 smithi132 ceph-mon[37580]: pgmap v9087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:24 smithi112 ceph-mon[37501]: pgmap v9088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:24 smithi107 ceph-mon[32659]: pgmap v9088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:24 smithi132 ceph-mon[37580]: pgmap v9088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16402.121460 secs 2024-07-26T17:24:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16402.121460 secs 2024-07-26T17:24:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16402.121460 secs 2024-07-26T17:24:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:26 smithi112 ceph-mon[37501]: pgmap v9089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:26 smithi107 ceph-mon[32659]: pgmap v9089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:26 smithi132 ceph-mon[37580]: pgmap v9089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:28 smithi112 ceph-mon[37501]: pgmap v9090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:28 smithi107 ceph-mon[32659]: pgmap v9090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:28 smithi132 ceph-mon[37580]: pgmap v9090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:30 smithi112 ceph-mon[37501]: pgmap v9091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16407.121580 secs 2024-07-26T17:24:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:30 smithi107 ceph-mon[32659]: pgmap v9091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16407.121580 secs 2024-07-26T17:24:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:30 smithi132 ceph-mon[37580]: pgmap v9091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16407.121580 secs 2024-07-26T17:24:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:32 smithi112 ceph-mon[37501]: pgmap v9092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:32 smithi107 ceph-mon[32659]: pgmap v9092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:32 smithi132 ceph-mon[37580]: pgmap v9092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:35.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:35.227+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:24:35.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:35.227+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:24:35.229 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:35.227+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:24:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:35 smithi107 ceph-mon[32659]: pgmap v9093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:35 smithi112 ceph-mon[37501]: pgmap v9093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:35 smithi132 ceph-mon[37580]: pgmap v9093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:36.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16412.121661 secs 2024-07-26T17:24:36.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16412.121661 secs 2024-07-26T17:24:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16412.121661 secs 2024-07-26T17:24:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:37 smithi112 ceph-mon[37501]: pgmap v9094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:37 smithi107 ceph-mon[32659]: pgmap v9094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:37 smithi132 ceph-mon[37580]: pgmap v9094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:39 smithi112 ceph-mon[37501]: pgmap v9095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:39 smithi107 ceph-mon[32659]: pgmap v9095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:39 smithi132 ceph-mon[37580]: pgmap v9095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:24:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:41 smithi112 ceph-mon[37501]: pgmap v9096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16417.121777 secs 2024-07-26T17:24:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:41 smithi107 ceph-mon[32659]: pgmap v9096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16417.121777 secs 2024-07-26T17:24:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:41 smithi132 ceph-mon[37580]: pgmap v9096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:24:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16417.121777 secs 2024-07-26T17:24:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:43 smithi112 ceph-mon[37501]: pgmap v9097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:43 smithi107 ceph-mon[32659]: pgmap v9097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:43 smithi132 ceph-mon[37580]: pgmap v9097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:24:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:45 smithi112 ceph-mon[37501]: pgmap v9098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:45 smithi107 ceph-mon[32659]: pgmap v9098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:45 smithi132 ceph-mon[37580]: pgmap v9098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16422.121903 secs 2024-07-26T17:24:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16422.121903 secs 2024-07-26T17:24:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16422.121903 secs 2024-07-26T17:24:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:47 smithi112 ceph-mon[37501]: pgmap v9099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:24:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:47 smithi107 ceph-mon[32659]: pgmap v9099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:24:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:47 smithi132 ceph-mon[37580]: pgmap v9099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:24:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:49 smithi112 ceph-mon[37501]: pgmap v9100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:49.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:49 smithi107 ceph-mon[32659]: pgmap v9100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:49 smithi132 ceph-mon[37580]: pgmap v9100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:50.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:50.234+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:24:50.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:50.234+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:24:50.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:24:50.234+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:24:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:51 smithi112 ceph-mon[37501]: pgmap v9101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:24:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16427.122026 secs 2024-07-26T17:24:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:51 smithi107 ceph-mon[32659]: pgmap v9101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:24:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16427.122026 secs 2024-07-26T17:24:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:51 smithi132 ceph-mon[37580]: pgmap v9101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:24:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16427.122026 secs 2024-07-26T17:24:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:52 smithi112 ceph-mon[37501]: pgmap v9102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T17:24:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:52 smithi107 ceph-mon[32659]: pgmap v9102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T17:24:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:52 smithi132 ceph-mon[37580]: pgmap v9102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T17:24:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:54 smithi112 ceph-mon[37501]: pgmap v9103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:55.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:24:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:54 smithi107 ceph-mon[32659]: pgmap v9103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:55.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:24:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:54 smithi132 ceph-mon[37580]: pgmap v9103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:24:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:24:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:24:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:55 smithi112 ceph-mon[37501]: pgmap v9104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:24:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:55 smithi112 ceph-mon[37501]: pgmap v9105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:55 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:24:56.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16432.122149 secs 2024-07-26T17:24:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:24:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:24:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:55 smithi107 ceph-mon[32659]: pgmap v9104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:24:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:55 smithi107 ceph-mon[32659]: pgmap v9105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:55 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:24:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16432.122149 secs 2024-07-26T17:24:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:24:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:24:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:55 smithi132 ceph-mon[37580]: pgmap v9104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:24:56.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:55 smithi132 ceph-mon[37580]: pgmap v9105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:24:56.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:55 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:24:56.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16432.122149 secs 2024-07-26T17:24:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:24:58 smithi107 ceph-mon[32659]: pgmap v9106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:24:58 smithi132 ceph-mon[37580]: pgmap v9106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:24:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:24:58 smithi112 ceph-mon[37501]: pgmap v9106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:00 smithi107 ceph-mon[32659]: pgmap v9107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:25:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:00 smithi132 ceph-mon[37580]: pgmap v9107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:25:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:00 smithi112 ceph-mon[37501]: pgmap v9107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:25:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16437.122272 secs 2024-07-26T17:25:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16437.122272 secs 2024-07-26T17:25:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16437.122272 secs 2024-07-26T17:25:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:02 smithi107 ceph-mon[32659]: pgmap v9108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:25:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:02 smithi132 ceph-mon[37580]: pgmap v9108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:25:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:02 smithi112 ceph-mon[37501]: pgmap v9108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:25:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:04 smithi107 ceph-mon[32659]: pgmap v9109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:25:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:04 smithi132 ceph-mon[37580]: pgmap v9109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:25:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:04 smithi112 ceph-mon[37501]: pgmap v9109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:25:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:05.249+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:25:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:05.249+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:25:05.252 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:05.249+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:25:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:06 smithi107 ceph-mon[32659]: pgmap v9110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:25:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16442.122391 secs 2024-07-26T17:25:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:06 smithi132 ceph-mon[37580]: pgmap v9110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:25:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16442.122391 secs 2024-07-26T17:25:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:06 smithi112 ceph-mon[37501]: pgmap v9110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:25:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16442.122391 secs 2024-07-26T17:25:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:08 smithi107 ceph-mon[32659]: pgmap v9111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:08 smithi132 ceph-mon[37580]: pgmap v9111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:08 smithi112 ceph-mon[37501]: pgmap v9111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:10 smithi132 ceph-mon[37580]: pgmap v9112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:25:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:10 smithi112 ceph-mon[37501]: pgmap v9112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:25:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:10 smithi107 ceph-mon[32659]: pgmap v9112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:25:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16447.122515 secs 2024-07-26T17:25:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16447.122515 secs 2024-07-26T17:25:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16447.122515 secs 2024-07-26T17:25:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:12 smithi132 ceph-mon[37580]: pgmap v9113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:25:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:12 smithi112 ceph-mon[37501]: pgmap v9113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:25:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:12 smithi107 ceph-mon[32659]: pgmap v9113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:25:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:14 smithi112 ceph-mon[37501]: pgmap v9114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:25:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:14 smithi107 ceph-mon[32659]: pgmap v9114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:25:14.566 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:14 smithi132 ceph-mon[37580]: pgmap v9114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:25:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:16 smithi112 ceph-mon[37501]: pgmap v9115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16452.122652 secs 2024-07-26T17:25:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:16 smithi107 ceph-mon[32659]: pgmap v9115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16452.122652 secs 2024-07-26T17:25:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:16 smithi132 ceph-mon[37580]: pgmap v9115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16452.122652 secs 2024-07-26T17:25:18.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:18 smithi112 ceph-mon[37501]: pgmap v9116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:18 smithi107 ceph-mon[32659]: pgmap v9116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:18 smithi132 ceph-mon[37580]: pgmap v9116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:20.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:20.265+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:25:20.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:20.266+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:25:20.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:20.266+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:25:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:20 smithi107 ceph-mon[32659]: pgmap v9117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:20 smithi112 ceph-mon[37501]: pgmap v9117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:20 smithi132 ceph-mon[37580]: pgmap v9117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16457.122776 secs 2024-07-26T17:25:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16457.122776 secs 2024-07-26T17:25:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16457.122776 secs 2024-07-26T17:25:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:22 smithi112 ceph-mon[37501]: pgmap v9118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:22 smithi107 ceph-mon[32659]: pgmap v9118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:22 smithi132 ceph-mon[37580]: pgmap v9118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:24 smithi112 ceph-mon[37501]: pgmap v9119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:24.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:24 smithi107 ceph-mon[32659]: pgmap v9119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:24 smithi132 ceph-mon[37580]: pgmap v9119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:26 smithi112 ceph-mon[37501]: pgmap v9120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16462.122901 secs 2024-07-26T17:25:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:26 smithi107 ceph-mon[32659]: pgmap v9120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16462.122901 secs 2024-07-26T17:25:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:26 smithi132 ceph-mon[37580]: pgmap v9120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16462.122901 secs 2024-07-26T17:25:28.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:28 smithi112 ceph-mon[37501]: pgmap v9121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:28 smithi107 ceph-mon[32659]: pgmap v9121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:28 smithi132 ceph-mon[37580]: pgmap v9121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:30 smithi112 ceph-mon[37501]: pgmap v9122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:30 smithi107 ceph-mon[32659]: pgmap v9122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:30 smithi132 ceph-mon[37580]: pgmap v9122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16467.123023 secs 2024-07-26T17:25:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16467.123023 secs 2024-07-26T17:25:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16467.123023 secs 2024-07-26T17:25:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:32 smithi112 ceph-mon[37501]: pgmap v9123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:32 smithi107 ceph-mon[32659]: pgmap v9123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:32 smithi132 ceph-mon[37580]: pgmap v9123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:33 smithi112 ceph-mon[37501]: pgmap v9124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:33 smithi107 ceph-mon[32659]: pgmap v9124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:33 smithi132 ceph-mon[37580]: pgmap v9124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:35.281+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:25:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:35.281+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:25:35.283 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:35.281+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:25:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:36 smithi107 ceph-mon[32659]: pgmap v9125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16472.123080 secs 2024-07-26T17:25:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:36 smithi132 ceph-mon[37580]: pgmap v9125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16472.123080 secs 2024-07-26T17:25:36.404 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:36 smithi112 ceph-mon[37501]: pgmap v9125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:36.404 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16472.123080 secs 2024-07-26T17:25:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:38 smithi107 ceph-mon[32659]: pgmap v9126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:38 smithi132 ceph-mon[37580]: pgmap v9126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:38 smithi112 ceph-mon[37501]: pgmap v9126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:40 smithi107 ceph-mon[32659]: pgmap v9127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:40 smithi132 ceph-mon[37580]: pgmap v9127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:40 smithi112 ceph-mon[37501]: pgmap v9127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16477.123198 secs 2024-07-26T17:25:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16477.123198 secs 2024-07-26T17:25:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16477.123198 secs 2024-07-26T17:25:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:42 smithi132 ceph-mon[37580]: pgmap v9128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:42 smithi112 ceph-mon[37501]: pgmap v9128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:42 smithi107 ceph-mon[32659]: pgmap v9128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:44 smithi132 ceph-mon[37580]: pgmap v9129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:44 smithi112 ceph-mon[37501]: pgmap v9129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:44 smithi107 ceph-mon[32659]: pgmap v9129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:46 smithi112 ceph-mon[37501]: pgmap v9130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16482.123266 secs 2024-07-26T17:25:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:46 smithi107 ceph-mon[32659]: pgmap v9130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16482.123266 secs 2024-07-26T17:25:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:46 smithi132 ceph-mon[37580]: pgmap v9130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16482.123266 secs 2024-07-26T17:25:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:48 smithi112 ceph-mon[37501]: pgmap v9131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:48 smithi107 ceph-mon[32659]: pgmap v9131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:48 smithi132 ceph-mon[37580]: pgmap v9131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:50.298 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:50.297+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:25:50.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:50.297+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:25:50.299 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:25:50.297+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:25:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:50 smithi112 ceph-mon[37501]: pgmap v9132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:50 smithi107 ceph-mon[32659]: pgmap v9132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:50 smithi132 ceph-mon[37580]: pgmap v9132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16487.123350 secs 2024-07-26T17:25:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16487.123350 secs 2024-07-26T17:25:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16487.123350 secs 2024-07-26T17:25:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:52 smithi112 ceph-mon[37501]: pgmap v9133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:52 smithi107 ceph-mon[32659]: pgmap v9133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:52 smithi132 ceph-mon[37580]: pgmap v9133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:25:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:54 smithi112 ceph-mon[37501]: pgmap v9134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:54 smithi107 ceph-mon[32659]: pgmap v9134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:54 smithi132 ceph-mon[37580]: pgmap v9134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:25:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:56 smithi112 ceph-mon[37501]: pgmap v9135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:25:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16492.123442 secs 2024-07-26T17:25:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:56 smithi107 ceph-mon[32659]: pgmap v9135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:25:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16492.123442 secs 2024-07-26T17:25:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:56 smithi132 ceph-mon[37580]: pgmap v9135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:25:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:25:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16492.123442 secs 2024-07-26T17:25:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:25:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:25:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:57 smithi112 ceph-mon[37501]: pgmap v9136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:25:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:57 smithi112 ceph-mon[37501]: pgmap v9137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:25:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:25:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:25:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:25:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:57 smithi107 ceph-mon[32659]: pgmap v9136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:25:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:57 smithi107 ceph-mon[32659]: pgmap v9137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:25:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:25:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:25:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:25:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:57 smithi132 ceph-mon[37580]: pgmap v9136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:25:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:57 smithi132 ceph-mon[37580]: pgmap v9137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:25:57.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:25:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:25:59 smithi112 ceph-mon[37501]: pgmap v9138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:25:59.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:25:59 smithi107 ceph-mon[32659]: pgmap v9138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:25:59.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:25:59 smithi132 ceph-mon[37580]: pgmap v9138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:26:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:01 smithi112 ceph-mon[37501]: pgmap v9139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:26:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16497.123563 secs 2024-07-26T17:26:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:01 smithi107 ceph-mon[32659]: pgmap v9139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:26:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16497.123563 secs 2024-07-26T17:26:01.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:01 smithi132 ceph-mon[37580]: pgmap v9139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:26:01.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16497.123563 secs 2024-07-26T17:26:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:03 smithi112 ceph-mon[37501]: pgmap v9140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:03 smithi107 ceph-mon[32659]: pgmap v9140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:03 smithi132 ceph-mon[37580]: pgmap v9140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:05.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:05.311+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:26:05.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:05.311+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:26:05.314 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:05.311+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:26:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:05 smithi112 ceph-mon[37501]: pgmap v9141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:26:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:05 smithi107 ceph-mon[32659]: pgmap v9141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:26:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:05 smithi132 ceph-mon[37580]: pgmap v9141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:26:06.462 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16502.123699 secs 2024-07-26T17:26:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16502.123699 secs 2024-07-26T17:26:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16502.123699 secs 2024-07-26T17:26:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:07 smithi107 ceph-mon[32659]: pgmap v9142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:26:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:07 smithi132 ceph-mon[37580]: pgmap v9142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:26:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:07 smithi112 ceph-mon[37501]: pgmap v9142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:26:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:09 smithi107 ceph-mon[32659]: pgmap v9143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:09 smithi132 ceph-mon[37580]: pgmap v9143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:09 smithi112 ceph-mon[37501]: pgmap v9143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:11 smithi107 ceph-mon[32659]: pgmap v9144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16507.123817 secs 2024-07-26T17:26:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:11 smithi132 ceph-mon[37580]: pgmap v9144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16507.123817 secs 2024-07-26T17:26:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:11 smithi112 ceph-mon[37501]: pgmap v9144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16507.123817 secs 2024-07-26T17:26:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:13 smithi107 ceph-mon[32659]: pgmap v9145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:13 smithi132 ceph-mon[37580]: pgmap v9145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:13 smithi112 ceph-mon[37501]: pgmap v9145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:15 smithi107 ceph-mon[32659]: pgmap v9146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:15 smithi132 ceph-mon[37580]: pgmap v9146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:15 smithi112 ceph-mon[37501]: pgmap v9146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16512.123936 secs 2024-07-26T17:26:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16512.123936 secs 2024-07-26T17:26:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16512.123936 secs 2024-07-26T17:26:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:17 smithi107 ceph-mon[32659]: pgmap v9147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:17 smithi132 ceph-mon[37580]: pgmap v9147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:17 smithi112 ceph-mon[37501]: pgmap v9147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:19 smithi107 ceph-mon[32659]: pgmap v9148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:19 smithi132 ceph-mon[37580]: pgmap v9148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:19 smithi112 ceph-mon[37501]: pgmap v9148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:20.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:20.327+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:26:20.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:20.327+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:26:20.330 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:20.327+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:26:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:21 smithi107 ceph-mon[32659]: pgmap v9149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16517.123999 secs 2024-07-26T17:26:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:21 smithi132 ceph-mon[37580]: pgmap v9149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16517.123999 secs 2024-07-26T17:26:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:21 smithi112 ceph-mon[37501]: pgmap v9149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16517.123999 secs 2024-07-26T17:26:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:23 smithi107 ceph-mon[32659]: pgmap v9150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:23 smithi132 ceph-mon[37580]: pgmap v9150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:23 smithi112 ceph-mon[37501]: pgmap v9150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:25 smithi107 ceph-mon[32659]: pgmap v9151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:25 smithi132 ceph-mon[37580]: pgmap v9151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:25 smithi112 ceph-mon[37501]: pgmap v9151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16522.124123 secs 2024-07-26T17:26:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16522.124123 secs 2024-07-26T17:26:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16522.124123 secs 2024-07-26T17:26:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:27 smithi132 ceph-mon[37580]: pgmap v9152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:27 smithi112 ceph-mon[37501]: pgmap v9152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:27 smithi107 ceph-mon[32659]: pgmap v9152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:29 smithi132 ceph-mon[37580]: pgmap v9153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:29 smithi112 ceph-mon[37501]: pgmap v9153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:29 smithi107 ceph-mon[32659]: pgmap v9153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:31 smithi132 ceph-mon[37580]: pgmap v9154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16527.124257 secs 2024-07-26T17:26:31.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:31 smithi107 ceph-mon[32659]: pgmap v9154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:31.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16527.124257 secs 2024-07-26T17:26:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:31 smithi112 ceph-mon[37501]: pgmap v9154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16527.124257 secs 2024-07-26T17:26:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:33 smithi112 ceph-mon[37501]: pgmap v9155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:33 smithi107 ceph-mon[32659]: pgmap v9155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:33 smithi132 ceph-mon[37580]: pgmap v9155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:35.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:35.343+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:26:35.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:35.344+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:26:35.345 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:35.344+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:26:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:35 smithi112 ceph-mon[37501]: pgmap v9156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:35 smithi107 ceph-mon[32659]: pgmap v9156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:35 smithi132 ceph-mon[37580]: pgmap v9156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:36.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16532.124369 secs 2024-07-26T17:26:36.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16532.124369 secs 2024-07-26T17:26:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16532.124369 secs 2024-07-26T17:26:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:37 smithi112 ceph-mon[37501]: pgmap v9157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:37 smithi107 ceph-mon[32659]: pgmap v9157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:37 smithi132 ceph-mon[37580]: pgmap v9157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:39 smithi112 ceph-mon[37501]: pgmap v9158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:39 smithi107 ceph-mon[32659]: pgmap v9158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:39 smithi132 ceph-mon[37580]: pgmap v9158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:41 smithi112 ceph-mon[37501]: pgmap v9159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16537.124506 secs 2024-07-26T17:26:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:41 smithi107 ceph-mon[32659]: pgmap v9159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16537.124506 secs 2024-07-26T17:26:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:41 smithi132 ceph-mon[37580]: pgmap v9159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16537.124506 secs 2024-07-26T17:26:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:43 smithi112 ceph-mon[37501]: pgmap v9160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:43 smithi107 ceph-mon[32659]: pgmap v9160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:43 smithi132 ceph-mon[37580]: pgmap v9160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:45 smithi112 ceph-mon[37501]: pgmap v9161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:45 smithi107 ceph-mon[32659]: pgmap v9161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:45 smithi132 ceph-mon[37580]: pgmap v9161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16542.124632 secs 2024-07-26T17:26:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16542.124632 secs 2024-07-26T17:26:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16542.124632 secs 2024-07-26T17:26:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:47 smithi112 ceph-mon[37501]: pgmap v9162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:47 smithi107 ceph-mon[32659]: pgmap v9162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:47 smithi132 ceph-mon[37580]: pgmap v9162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:49 smithi112 ceph-mon[37501]: pgmap v9163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:49 smithi107 ceph-mon[32659]: pgmap v9163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:49 smithi132 ceph-mon[37580]: pgmap v9163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:26:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:50.359+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:26:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:50.359+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:26:50.361 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:26:50.359+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:26:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:51 smithi112 ceph-mon[37501]: pgmap v9164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16547.124744 secs 2024-07-26T17:26:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:51 smithi107 ceph-mon[32659]: pgmap v9164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16547.124744 secs 2024-07-26T17:26:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:51 smithi132 ceph-mon[37580]: pgmap v9164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16547.124744 secs 2024-07-26T17:26:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:53 smithi112 ceph-mon[37501]: pgmap v9165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:53 smithi107 ceph-mon[32659]: pgmap v9165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:53 smithi132 ceph-mon[37580]: pgmap v9165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:26:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:55 smithi112 ceph-mon[37501]: pgmap v9166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:55 smithi107 ceph-mon[32659]: pgmap v9166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:55 smithi132 ceph-mon[37580]: pgmap v9166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:26:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16552.124863 secs 2024-07-26T17:26:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:26:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16552.124863 secs 2024-07-26T17:26:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:26:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16552.124863 secs 2024-07-26T17:26:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:26:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:57 smithi107 ceph-mon[32659]: pgmap v9167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:26:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:26:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:57 smithi107 ceph-mon[32659]: pgmap v9168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:26:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:57 smithi107 ceph-mon[32659]: pgmap v9169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:26:57.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:57 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:26:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:57 smithi112 ceph-mon[37501]: pgmap v9167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:26:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:26:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:57 smithi112 ceph-mon[37501]: pgmap v9168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:26:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:57 smithi112 ceph-mon[37501]: pgmap v9169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:26:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:57 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:26:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:57 smithi132 ceph-mon[37580]: pgmap v9167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:26:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:26:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:26:57.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:57 smithi132 ceph-mon[37580]: pgmap v9168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:26:57.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:57 smithi132 ceph-mon[37580]: pgmap v9169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:26:57.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:57 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:26:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:26:59 smithi107 ceph-mon[32659]: pgmap v9170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:26:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:26:59 smithi132 ceph-mon[37580]: pgmap v9170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:26:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:26:59 smithi112 ceph-mon[37501]: pgmap v9170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:27:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16557.124985 secs 2024-07-26T17:27:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16557.124985 secs 2024-07-26T17:27:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16557.124985 secs 2024-07-26T17:27:01.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:01 smithi107 ceph-mon[32659]: pgmap v9171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:27:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:01 smithi112 ceph-mon[37501]: pgmap v9171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:27:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:01 smithi132 ceph-mon[37580]: pgmap v9171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:27:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:03 smithi107 ceph-mon[32659]: pgmap v9172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:27:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:03 smithi132 ceph-mon[37580]: pgmap v9172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:27:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:03 smithi112 ceph-mon[37501]: pgmap v9172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:27:05.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:05.361+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:27:05.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:05.362+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:27:05.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:05.362+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:27:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:05 smithi107 ceph-mon[32659]: pgmap v9173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:27:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:05 smithi132 ceph-mon[37580]: pgmap v9173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:27:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:05 smithi112 ceph-mon[37501]: pgmap v9173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:27:06.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16562.125053 secs 2024-07-26T17:27:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16562.125053 secs 2024-07-26T17:27:06.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16562.125053 secs 2024-07-26T17:27:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:07 smithi107 ceph-mon[32659]: pgmap v9174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:27:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:07 smithi132 ceph-mon[37580]: pgmap v9174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:27:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:07 smithi112 ceph-mon[37501]: pgmap v9174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:27:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:09 smithi107 ceph-mon[32659]: pgmap v9175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:09 smithi132 ceph-mon[37580]: pgmap v9175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:09 smithi112 ceph-mon[37501]: pgmap v9175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16567.125169 secs 2024-07-26T17:27:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16567.125169 secs 2024-07-26T17:27:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16567.125169 secs 2024-07-26T17:27:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:11 smithi112 ceph-mon[37501]: pgmap v9176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:11 smithi107 ceph-mon[32659]: pgmap v9176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:11 smithi132 ceph-mon[37580]: pgmap v9176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:13 smithi107 ceph-mon[32659]: pgmap v9177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:13 smithi132 ceph-mon[37580]: pgmap v9177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:13 smithi112 ceph-mon[37501]: pgmap v9177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:15.499+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:27:15.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:15.499+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:27:15.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:15.499+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:27:15.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:15.499+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:27:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:15 smithi107 ceph-mon[32659]: pgmap v9178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:15 smithi132 ceph-mon[37580]: pgmap v9178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:15 smithi112 ceph-mon[37501]: pgmap v9178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16572.125238 secs 2024-07-26T17:27:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16572.125238 secs 2024-07-26T17:27:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16572.125238 secs 2024-07-26T17:27:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:17 smithi107 ceph-mon[32659]: pgmap v9179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:17 smithi132 ceph-mon[37580]: pgmap v9179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:17 smithi112 ceph-mon[37501]: pgmap v9179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:19 smithi107 ceph-mon[32659]: pgmap v9180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:19 smithi132 ceph-mon[37580]: pgmap v9180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:19 smithi112 ceph-mon[37501]: pgmap v9180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:20.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:20.377+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:27:20.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:20.377+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:27:20.380 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:20.377+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:27:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16577.125358 secs 2024-07-26T17:27:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16577.125358 secs 2024-07-26T17:27:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16577.125358 secs 2024-07-26T17:27:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:21 smithi112 ceph-mon[37501]: pgmap v9181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:21 smithi107 ceph-mon[32659]: pgmap v9181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:21 smithi132 ceph-mon[37580]: pgmap v9181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:23 smithi107 ceph-mon[32659]: pgmap v9182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:23 smithi132 ceph-mon[37580]: pgmap v9182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:23 smithi112 ceph-mon[37501]: pgmap v9182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:25 smithi107 ceph-mon[32659]: pgmap v9183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:25 smithi132 ceph-mon[37580]: pgmap v9183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:25 smithi112 ceph-mon[37501]: pgmap v9183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16582.125428 secs 2024-07-26T17:27:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16582.125428 secs 2024-07-26T17:27:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16582.125428 secs 2024-07-26T17:27:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:27 smithi107 ceph-mon[32659]: pgmap v9184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:27 smithi132 ceph-mon[37580]: pgmap v9184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:27 smithi112 ceph-mon[37501]: pgmap v9184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:29 smithi107 ceph-mon[32659]: pgmap v9185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:29 smithi132 ceph-mon[37580]: pgmap v9185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:29 smithi112 ceph-mon[37501]: pgmap v9185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16587.125544 secs 2024-07-26T17:27:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16587.125544 secs 2024-07-26T17:27:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16587.125544 secs 2024-07-26T17:27:31.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:31 smithi107 ceph-mon[32659]: pgmap v9186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:31 smithi112 ceph-mon[37501]: pgmap v9186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:31 smithi132 ceph-mon[37580]: pgmap v9186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:33 smithi107 ceph-mon[32659]: pgmap v9187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:33 smithi132 ceph-mon[37580]: pgmap v9187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:33 smithi112 ceph-mon[37501]: pgmap v9187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:35.384+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:27:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:35.384+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:27:35.386 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:35.384+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:27:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:35 smithi107 ceph-mon[32659]: pgmap v9188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:35 smithi132 ceph-mon[37580]: pgmap v9188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:35 smithi112 ceph-mon[37501]: pgmap v9188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16592.125611 secs 2024-07-26T17:27:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16592.125611 secs 2024-07-26T17:27:36.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16592.125611 secs 2024-07-26T17:27:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:37 smithi107 ceph-mon[32659]: pgmap v9189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:37 smithi132 ceph-mon[37580]: pgmap v9189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:37 smithi112 ceph-mon[37501]: pgmap v9189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:39 smithi107 ceph-mon[32659]: pgmap v9190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:39 smithi132 ceph-mon[37580]: pgmap v9190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:39 smithi112 ceph-mon[37501]: pgmap v9190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16597.125727 secs 2024-07-26T17:27:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16597.125727 secs 2024-07-26T17:27:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16597.125727 secs 2024-07-26T17:27:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:41 smithi112 ceph-mon[37501]: pgmap v9191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:41 smithi107 ceph-mon[32659]: pgmap v9191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:41 smithi132 ceph-mon[37580]: pgmap v9191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:43 smithi107 ceph-mon[32659]: pgmap v9192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:43 smithi132 ceph-mon[37580]: pgmap v9192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:43 smithi112 ceph-mon[37501]: pgmap v9192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:45 smithi107 ceph-mon[32659]: pgmap v9193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:45 smithi132 ceph-mon[37580]: pgmap v9193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:45 smithi112 ceph-mon[37501]: pgmap v9193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16602.125852 secs 2024-07-26T17:27:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16602.125852 secs 2024-07-26T17:27:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16602.125852 secs 2024-07-26T17:27:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:47 smithi107 ceph-mon[32659]: pgmap v9194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:47 smithi132 ceph-mon[37580]: pgmap v9194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:47 smithi112 ceph-mon[37501]: pgmap v9194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:49 smithi107 ceph-mon[32659]: pgmap v9195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:49 smithi132 ceph-mon[37580]: pgmap v9195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:49 smithi112 ceph-mon[37501]: pgmap v9195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:27:50.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:50.388+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:27:50.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:50.388+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:27:50.391 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:27:50.389+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:27:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16607.125967 secs 2024-07-26T17:27:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16607.125967 secs 2024-07-26T17:27:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16607.125967 secs 2024-07-26T17:27:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:51 smithi112 ceph-mon[37501]: pgmap v9196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:51 smithi107 ceph-mon[32659]: pgmap v9196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:51 smithi132 ceph-mon[37580]: pgmap v9196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:27:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:53 smithi107 ceph-mon[32659]: pgmap v9197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:53 smithi132 ceph-mon[37580]: pgmap v9197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:53 smithi112 ceph-mon[37501]: pgmap v9197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:27:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:55 smithi107 ceph-mon[32659]: pgmap v9198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:55 smithi132 ceph-mon[37580]: pgmap v9198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:55 smithi112 ceph-mon[37501]: pgmap v9198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:27:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16612.126088 secs 2024-07-26T17:27:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16612.126088 secs 2024-07-26T17:27:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16612.126088 secs 2024-07-26T17:27:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:57 smithi107 ceph-mon[32659]: pgmap v9199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:57 smithi132 ceph-mon[37580]: pgmap v9199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:57 smithi112 ceph-mon[37501]: pgmap v9199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:27:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:27:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:27:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:27:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:27:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:27:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:59 smithi107 ceph-mon[32659]: pgmap v9200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:27:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:59 smithi107 ceph-mon[32659]: pgmap v9201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:27:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:27:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:27:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:27:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:27:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:59 smithi132 ceph-mon[37580]: pgmap v9200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:27:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:59 smithi132 ceph-mon[37580]: pgmap v9201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:27:59.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:27:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:27:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:27:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:27:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:59 smithi112 ceph-mon[37501]: pgmap v9200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:27:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:59 smithi112 ceph-mon[37501]: pgmap v9201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:27:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:27:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:28:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16617.126209 secs 2024-07-26T17:28:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:01 smithi112 ceph-mon[37501]: pgmap v9202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:28:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16617.126209 secs 2024-07-26T17:28:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:01 smithi107 ceph-mon[32659]: pgmap v9202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:28:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16617.126209 secs 2024-07-26T17:28:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:01 smithi132 ceph-mon[37580]: pgmap v9202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:28:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:03 smithi112 ceph-mon[37501]: pgmap v9203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:28:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:03 smithi107 ceph-mon[32659]: pgmap v9203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:28:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:03 smithi132 ceph-mon[37580]: pgmap v9203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:28:05.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:05.403+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:28:05.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:05.403+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:28:05.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:05.403+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:28:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:05 smithi107 ceph-mon[32659]: pgmap v9204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:28:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16622.126328 secs 2024-07-26T17:28:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:05 smithi112 ceph-mon[37501]: pgmap v9204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:28:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16622.126328 secs 2024-07-26T17:28:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:05 smithi132 ceph-mon[37580]: pgmap v9204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:28:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16622.126328 secs 2024-07-26T17:28:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:07 smithi112 ceph-mon[37501]: pgmap v9205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:07 smithi107 ceph-mon[32659]: pgmap v9205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:07 smithi132 ceph-mon[37580]: pgmap v9205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:09 smithi112 ceph-mon[37501]: pgmap v9206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:28:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:09 smithi107 ceph-mon[32659]: pgmap v9206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:28:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:09 smithi132 ceph-mon[37580]: pgmap v9206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:28:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16627.126450 secs 2024-07-26T17:28:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16627.126450 secs 2024-07-26T17:28:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16627.126450 secs 2024-07-26T17:28:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:11 smithi112 ceph-mon[37501]: pgmap v9207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:11 smithi107 ceph-mon[32659]: pgmap v9207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:11 smithi132 ceph-mon[37580]: pgmap v9207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:13 smithi112 ceph-mon[37501]: pgmap v9208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:13 smithi107 ceph-mon[32659]: pgmap v9208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:13 smithi132 ceph-mon[37580]: pgmap v9208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:15 smithi112 ceph-mon[37501]: pgmap v9209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16632.126554 secs 2024-07-26T17:28:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:15 smithi107 ceph-mon[32659]: pgmap v9209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16632.126554 secs 2024-07-26T17:28:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:15 smithi132 ceph-mon[37580]: pgmap v9209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16632.126554 secs 2024-07-26T17:28:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:17 smithi112 ceph-mon[37501]: pgmap v9210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:17 smithi107 ceph-mon[32659]: pgmap v9210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:17 smithi132 ceph-mon[37580]: pgmap v9210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:19 smithi112 ceph-mon[37501]: pgmap v9211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:19 smithi107 ceph-mon[32659]: pgmap v9211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:19 smithi132 ceph-mon[37580]: pgmap v9211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:20.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:20.419+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:28:20.421 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:20.419+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:28:20.422 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:20.419+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:28:20.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16637.126694 secs 2024-07-26T17:28:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16637.126694 secs 2024-07-26T17:28:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16637.126694 secs 2024-07-26T17:28:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:21 smithi112 ceph-mon[37501]: pgmap v9212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:21 smithi107 ceph-mon[32659]: pgmap v9212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:21 smithi132 ceph-mon[37580]: pgmap v9212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:23 smithi112 ceph-mon[37501]: pgmap v9213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:23 smithi107 ceph-mon[32659]: pgmap v9213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:23 smithi132 ceph-mon[37580]: pgmap v9213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:25 smithi112 ceph-mon[37501]: pgmap v9214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:25.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16642.126847 secs 2024-07-26T17:28:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:25 smithi107 ceph-mon[32659]: pgmap v9214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16642.126847 secs 2024-07-26T17:28:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:25 smithi132 ceph-mon[37580]: pgmap v9214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16642.126847 secs 2024-07-26T17:28:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:27 smithi112 ceph-mon[37501]: pgmap v9215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:27 smithi107 ceph-mon[32659]: pgmap v9215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:27 smithi132 ceph-mon[37580]: pgmap v9215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:29 smithi107 ceph-mon[32659]: pgmap v9216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:29 smithi132 ceph-mon[37580]: pgmap v9216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:29 smithi112 ceph-mon[37501]: pgmap v9216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:31.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16647.126985 secs 2024-07-26T17:28:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16647.126985 secs 2024-07-26T17:28:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16647.126985 secs 2024-07-26T17:28:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:31 smithi107 ceph-mon[32659]: pgmap v9217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:31 smithi132 ceph-mon[37580]: pgmap v9217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:31 smithi112 ceph-mon[37501]: pgmap v9217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:33 smithi107 ceph-mon[32659]: pgmap v9218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:33 smithi132 ceph-mon[37580]: pgmap v9218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:33 smithi112 ceph-mon[37501]: pgmap v9218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:35.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:35.435+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:28:35.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:35.435+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:28:35.437 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:35.435+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:28:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:35 smithi107 ceph-mon[32659]: pgmap v9219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16652.127131 secs 2024-07-26T17:28:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:35 smithi132 ceph-mon[37580]: pgmap v9219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16652.127131 secs 2024-07-26T17:28:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:35 smithi112 ceph-mon[37501]: pgmap v9219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16652.127131 secs 2024-07-26T17:28:38.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:37 smithi112 ceph-mon[37501]: pgmap v9220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:37 smithi107 ceph-mon[32659]: pgmap v9220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:37 smithi132 ceph-mon[37580]: pgmap v9220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:39 smithi107 ceph-mon[32659]: pgmap v9221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:39 smithi132 ceph-mon[37580]: pgmap v9221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:39 smithi112 ceph-mon[37501]: pgmap v9221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16657.127255 secs 2024-07-26T17:28:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16657.127255 secs 2024-07-26T17:28:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16657.127255 secs 2024-07-26T17:28:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:41 smithi107 ceph-mon[32659]: pgmap v9222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:41 smithi132 ceph-mon[37580]: pgmap v9222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:41 smithi112 ceph-mon[37501]: pgmap v9222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:43 smithi107 ceph-mon[32659]: pgmap v9223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:43 smithi132 ceph-mon[37580]: pgmap v9223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:43 smithi112 ceph-mon[37501]: pgmap v9223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:45 smithi107 ceph-mon[32659]: pgmap v9224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:46.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16662.127370 secs 2024-07-26T17:28:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:45 smithi132 ceph-mon[37580]: pgmap v9224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16662.127370 secs 2024-07-26T17:28:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:45 smithi112 ceph-mon[37501]: pgmap v9224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:46.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16662.127370 secs 2024-07-26T17:28:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:47 smithi107 ceph-mon[32659]: pgmap v9225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:47 smithi132 ceph-mon[37580]: pgmap v9225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:47 smithi112 ceph-mon[37501]: pgmap v9225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:49 smithi107 ceph-mon[32659]: pgmap v9226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:49 smithi132 ceph-mon[37580]: pgmap v9226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:50.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:49 smithi112 ceph-mon[37501]: pgmap v9226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:50.451+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:28:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:50.451+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:28:50.453 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:28:50.451+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:28:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16667.127502 secs 2024-07-26T17:28:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16667.127502 secs 2024-07-26T17:28:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16667.127502 secs 2024-07-26T17:28:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:51 smithi107 ceph-mon[32659]: pgmap v9227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:51 smithi132 ceph-mon[37580]: pgmap v9227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:51 smithi112 ceph-mon[37501]: pgmap v9227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:28:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:53 smithi132 ceph-mon[37580]: pgmap v9228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:53 smithi112 ceph-mon[37501]: pgmap v9228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:53 smithi107 ceph-mon[32659]: pgmap v9228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:55 smithi112 ceph-mon[37501]: pgmap v9229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:55 smithi107 ceph-mon[32659]: pgmap v9229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:55 smithi132 ceph-mon[37580]: pgmap v9229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:28:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16672.127633 secs 2024-07-26T17:28:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16672.127633 secs 2024-07-26T17:28:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16672.127633 secs 2024-07-26T17:28:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:57 smithi107 ceph-mon[32659]: pgmap v9230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:57 smithi132 ceph-mon[37580]: pgmap v9230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:57 smithi112 ceph-mon[37501]: pgmap v9230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:28:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:59 smithi112 ceph-mon[37501]: pgmap v9231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:28:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:28:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:29:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:59 smithi107 ceph-mon[32659]: pgmap v9231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:00.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:28:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:29:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:59 smithi132 ceph-mon[37580]: pgmap v9231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:28:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:29:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:29:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:29:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:00 smithi112 ceph-mon[37501]: pgmap v9232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:29:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:00 smithi112 ceph-mon[37501]: pgmap v9233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:29:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:29:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16677.127752 secs 2024-07-26T17:29:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:29:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:29:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:00 smithi107 ceph-mon[32659]: pgmap v9232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:29:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:00 smithi107 ceph-mon[32659]: pgmap v9233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:29:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:29:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16677.127752 secs 2024-07-26T17:29:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:29:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:29:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:00 smithi132 ceph-mon[37580]: pgmap v9232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:29:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:00 smithi132 ceph-mon[37580]: pgmap v9233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:29:01.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:29:01.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16677.127752 secs 2024-07-26T17:29:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:02 smithi112 ceph-mon[37501]: pgmap v9234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:02 smithi107 ceph-mon[32659]: pgmap v9234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:02 smithi132 ceph-mon[37580]: pgmap v9234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:05.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:04 smithi107 ceph-mon[32659]: pgmap v9235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:29:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:04 smithi112 ceph-mon[37501]: pgmap v9235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:29:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:04 smithi132 ceph-mon[37580]: pgmap v9235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:29:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:05.466+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:29:05.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:05.466+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:29:05.469 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:05.466+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:29:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16682.127873 secs 2024-07-26T17:29:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16682.127873 secs 2024-07-26T17:29:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16682.127873 secs 2024-07-26T17:29:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:06 smithi112 ceph-mon[37501]: pgmap v9236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:29:07.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:06 smithi132 ceph-mon[37580]: pgmap v9236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:29:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:06 smithi107 ceph-mon[32659]: pgmap v9236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:29:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:08 smithi112 ceph-mon[37501]: pgmap v9237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:29:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:08 smithi107 ceph-mon[32659]: pgmap v9237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:29:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:08 smithi132 ceph-mon[37580]: pgmap v9237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:29:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:10 smithi112 ceph-mon[37501]: pgmap v9238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:29:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16687.127996 secs 2024-07-26T17:29:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:10 smithi107 ceph-mon[32659]: pgmap v9238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:29:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16687.127996 secs 2024-07-26T17:29:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:10 smithi132 ceph-mon[37580]: pgmap v9238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:29:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16687.127996 secs 2024-07-26T17:29:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:12 smithi112 ceph-mon[37501]: pgmap v9239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:12 smithi107 ceph-mon[32659]: pgmap v9239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:12 smithi132 ceph-mon[37580]: pgmap v9239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:15.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:14 smithi112 ceph-mon[37501]: pgmap v9240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:14 smithi107 ceph-mon[32659]: pgmap v9240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:14 smithi132 ceph-mon[37580]: pgmap v9240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16692.128132 secs 2024-07-26T17:29:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16692.128132 secs 2024-07-26T17:29:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16692.128132 secs 2024-07-26T17:29:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:16 smithi112 ceph-mon[37501]: pgmap v9241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:16 smithi107 ceph-mon[32659]: pgmap v9241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:16 smithi132 ceph-mon[37580]: pgmap v9241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:18 smithi112 ceph-mon[37501]: pgmap v9242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:18 smithi107 ceph-mon[32659]: pgmap v9242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:18 smithi132 ceph-mon[37580]: pgmap v9242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:20.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:20.482+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:29:20.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:20.482+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:29:20.484 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:20.482+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:29:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:20 smithi107 ceph-mon[32659]: pgmap v9243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:20 smithi132 ceph-mon[37580]: pgmap v9243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:20 smithi112 ceph-mon[37501]: pgmap v9243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16697.128209 secs 2024-07-26T17:29:21.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16697.128209 secs 2024-07-26T17:29:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16697.128209 secs 2024-07-26T17:29:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:22 smithi107 ceph-mon[32659]: pgmap v9244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:22 smithi132 ceph-mon[37580]: pgmap v9244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:22 smithi112 ceph-mon[37501]: pgmap v9244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:24 smithi112 ceph-mon[37501]: pgmap v9245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:24 smithi107 ceph-mon[32659]: pgmap v9245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:24 smithi132 ceph-mon[37580]: pgmap v9245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16702.128350 secs 2024-07-26T17:29:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16702.128350 secs 2024-07-26T17:29:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16702.128350 secs 2024-07-26T17:29:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:26 smithi112 ceph-mon[37501]: pgmap v9246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:26 smithi107 ceph-mon[32659]: pgmap v9246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:26 smithi132 ceph-mon[37580]: pgmap v9246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:28 smithi112 ceph-mon[37501]: pgmap v9247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:28 smithi107 ceph-mon[32659]: pgmap v9247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:28 smithi132 ceph-mon[37580]: pgmap v9247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:30 smithi112 ceph-mon[37501]: pgmap v9248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16707.128433 secs 2024-07-26T17:29:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:30 smithi107 ceph-mon[32659]: pgmap v9248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16707.128433 secs 2024-07-26T17:29:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:30 smithi132 ceph-mon[37580]: pgmap v9248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16707.128433 secs 2024-07-26T17:29:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:32 smithi112 ceph-mon[37501]: pgmap v9249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:29:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:32 smithi107 ceph-mon[32659]: pgmap v9249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:29:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:32 smithi132 ceph-mon[37580]: pgmap v9249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:29:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:34 smithi112 ceph-mon[37501]: pgmap v9250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:29:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:34 smithi107 ceph-mon[32659]: pgmap v9250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:29:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:34 smithi132 ceph-mon[37580]: pgmap v9250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:29:35.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:35.489+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:29:35.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:35.489+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:29:35.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:35.489+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:29:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16712.128507 secs 2024-07-26T17:29:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16712.128507 secs 2024-07-26T17:29:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16712.128507 secs 2024-07-26T17:29:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:36 smithi112 ceph-mon[37501]: pgmap v9251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:29:37.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:36 smithi132 ceph-mon[37580]: pgmap v9251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:29:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:36 smithi107 ceph-mon[32659]: pgmap v9251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:29:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:38 smithi112 ceph-mon[37501]: pgmap v9252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:29:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:38 smithi107 ceph-mon[32659]: pgmap v9252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:29:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:38 smithi132 ceph-mon[37580]: pgmap v9252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:29:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:40 smithi112 ceph-mon[37501]: pgmap v9253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16717.128623 secs 2024-07-26T17:29:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:40 smithi107 ceph-mon[32659]: pgmap v9253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16717.128623 secs 2024-07-26T17:29:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:40 smithi132 ceph-mon[37580]: pgmap v9253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16717.128623 secs 2024-07-26T17:29:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:42 smithi112 ceph-mon[37501]: pgmap v9254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:29:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:42 smithi107 ceph-mon[32659]: pgmap v9254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:29:43.333 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:42 smithi132 ceph-mon[37580]: pgmap v9254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:29:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:44 smithi112 ceph-mon[37501]: pgmap v9255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:44 smithi107 ceph-mon[32659]: pgmap v9255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:44 smithi132 ceph-mon[37580]: pgmap v9255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16722.128746 secs 2024-07-26T17:29:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16722.128746 secs 2024-07-26T17:29:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16722.128746 secs 2024-07-26T17:29:47.237 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:46 smithi107 ceph-mon[32659]: pgmap v9256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:46 smithi132 ceph-mon[37580]: pgmap v9256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:46 smithi112 ceph-mon[37501]: pgmap v9256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:48 smithi107 ceph-mon[32659]: pgmap v9257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:48 smithi132 ceph-mon[37580]: pgmap v9257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:48 smithi112 ceph-mon[37501]: pgmap v9257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:50.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:50.490+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:29:50.492 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:50.490+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:29:50.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:29:50.490+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:29:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:50 smithi107 ceph-mon[32659]: pgmap v9258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16727.128865 secs 2024-07-26T17:29:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:50 smithi132 ceph-mon[37580]: pgmap v9258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16727.128865 secs 2024-07-26T17:29:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:50 smithi112 ceph-mon[37501]: pgmap v9258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16727.128865 secs 2024-07-26T17:29:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:52 smithi107 ceph-mon[32659]: pgmap v9259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:52 smithi132 ceph-mon[37580]: pgmap v9259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:52 smithi112 ceph-mon[37501]: pgmap v9259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:29:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:54 smithi107 ceph-mon[32659]: pgmap v9260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:54 smithi132 ceph-mon[37580]: pgmap v9260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:54 smithi112 ceph-mon[37501]: pgmap v9260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16732.128998 secs 2024-07-26T17:29:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16732.128998 secs 2024-07-26T17:29:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16732.128998 secs 2024-07-26T17:29:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:56 smithi107 ceph-mon[32659]: pgmap v9261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:56 smithi132 ceph-mon[37580]: pgmap v9261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:56 smithi112 ceph-mon[37501]: pgmap v9261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:29:59.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:29:58 smithi107 ceph-mon[32659]: pgmap v9262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:29:58 smithi132 ceph-mon[37580]: pgmap v9262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:29:59.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:29:58 smithi112 ceph-mon[37501]: pgmap v9262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:00 smithi107 ceph-mon[32659]: pgmap v9263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:30:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:30:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16737.129113 secs 2024-07-26T17:30:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:01.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:00 smithi132 ceph-mon[37580]: pgmap v9263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:30:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:30:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16737.129113 secs 2024-07-26T17:30:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:01.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:00 smithi112 ceph-mon[37501]: pgmap v9263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:30:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:30:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16737.129113 secs 2024-07-26T17:30:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:30:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:30:02.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:30:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:30:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: pgmap v9264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:30:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: pgmap v9265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:30:02.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:30:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:30:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:30:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:30:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:30:02.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: pgmap v9264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:30:02.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: pgmap v9265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:30:02.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:30:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:30:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:30:02.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:30:02.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:30:02.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:30:02.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: pgmap v9264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:30:02.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: pgmap v9265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:30:02.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:30:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:04 smithi112 ceph-mon[37501]: pgmap v9266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:30:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:04 smithi107 ceph-mon[32659]: pgmap v9266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:30:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:04 smithi132 ceph-mon[37580]: pgmap v9266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:30:05.507 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:05.505+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:30:05.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:05.505+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:30:05.508 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:05.505+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:30:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:06 smithi112 ceph-mon[37501]: pgmap v9267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:30:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16742.129235 secs 2024-07-26T17:30:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:06 smithi107 ceph-mon[32659]: pgmap v9267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:30:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16742.129235 secs 2024-07-26T17:30:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:06 smithi132 ceph-mon[37580]: pgmap v9267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:30:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16742.129235 secs 2024-07-26T17:30:08.362 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:08 smithi132 ceph-mon[37580]: pgmap v9268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:08 smithi112 ceph-mon[37501]: pgmap v9268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:08 smithi107 ceph-mon[32659]: pgmap v9268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:10 smithi132 ceph-mon[37580]: pgmap v9269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:30:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:10 smithi112 ceph-mon[37501]: pgmap v9269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:30:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:10 smithi107 ceph-mon[32659]: pgmap v9269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T17:30:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16747.129341 secs 2024-07-26T17:30:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16747.129341 secs 2024-07-26T17:30:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16747.129341 secs 2024-07-26T17:30:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:12 smithi112 ceph-mon[37501]: pgmap v9270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:30:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:12 smithi107 ceph-mon[32659]: pgmap v9270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:30:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:12 smithi132 ceph-mon[37580]: pgmap v9270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:30:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:14 smithi112 ceph-mon[37501]: pgmap v9271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:14 smithi107 ceph-mon[32659]: pgmap v9271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:14 smithi132 ceph-mon[37580]: pgmap v9271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:15.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:15.531+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:30:15.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:15.531+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:30:15.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:15.531+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:30:15.535 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:15.531+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:30:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:16 smithi112 ceph-mon[37501]: pgmap v9272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16752.129459 secs 2024-07-26T17:30:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:16 smithi107 ceph-mon[32659]: pgmap v9272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16752.129459 secs 2024-07-26T17:30:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:16 smithi132 ceph-mon[37580]: pgmap v9272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16752.129459 secs 2024-07-26T17:30:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:18 smithi112 ceph-mon[37501]: pgmap v9273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:18.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:18 smithi107 ceph-mon[32659]: pgmap v9273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:18 smithi132 ceph-mon[37580]: pgmap v9273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:20 smithi112 ceph-mon[37501]: pgmap v9274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:20.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:20.515+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:30:20.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:20.515+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:30:20.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:20.515+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:30:20.519 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:20 smithi107 ceph-mon[32659]: pgmap v9274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:20 smithi132 ceph-mon[37580]: pgmap v9274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16757.129564 secs 2024-07-26T17:30:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16757.129564 secs 2024-07-26T17:30:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16757.129564 secs 2024-07-26T17:30:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:22 smithi112 ceph-mon[37501]: pgmap v9275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:22 smithi107 ceph-mon[32659]: pgmap v9275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:22 smithi132 ceph-mon[37580]: pgmap v9275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:24 smithi112 ceph-mon[37501]: pgmap v9276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:24 smithi107 ceph-mon[32659]: pgmap v9276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:24 smithi132 ceph-mon[37580]: pgmap v9276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:26.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:26 smithi112 ceph-mon[37501]: pgmap v9277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:26.455 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16762.129705 secs 2024-07-26T17:30:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:26 smithi107 ceph-mon[32659]: pgmap v9277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16762.129705 secs 2024-07-26T17:30:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:26 smithi132 ceph-mon[37580]: pgmap v9277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16762.129705 secs 2024-07-26T17:30:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:28 smithi112 ceph-mon[37501]: pgmap v9278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:28 smithi107 ceph-mon[32659]: pgmap v9278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:28 smithi132 ceph-mon[37580]: pgmap v9278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:30 smithi112 ceph-mon[37501]: pgmap v9279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:30 smithi107 ceph-mon[32659]: pgmap v9279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:30 smithi132 ceph-mon[37580]: pgmap v9279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16767.129802 secs 2024-07-26T17:30:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16767.129802 secs 2024-07-26T17:30:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16767.129802 secs 2024-07-26T17:30:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:32 smithi112 ceph-mon[37501]: pgmap v9280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:32 smithi107 ceph-mon[32659]: pgmap v9280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:32 smithi132 ceph-mon[37580]: pgmap v9280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:34 smithi112 ceph-mon[37501]: pgmap v9281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:34 smithi107 ceph-mon[32659]: pgmap v9281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:34 smithi132 ceph-mon[37580]: pgmap v9281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:35.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:35.532+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:30:35.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:35.532+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:30:35.534 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:35.532+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:30:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:36 smithi112 ceph-mon[37501]: pgmap v9282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16772.129945 secs 2024-07-26T17:30:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:36 smithi107 ceph-mon[32659]: pgmap v9282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16772.129945 secs 2024-07-26T17:30:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:36 smithi132 ceph-mon[37580]: pgmap v9282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16772.129945 secs 2024-07-26T17:30:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:38 smithi107 ceph-mon[32659]: pgmap v9283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:38 smithi132 ceph-mon[37580]: pgmap v9283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:38 smithi112 ceph-mon[37501]: pgmap v9283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:40.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:40 smithi112 ceph-mon[37501]: pgmap v9284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:40 smithi107 ceph-mon[32659]: pgmap v9284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:40 smithi132 ceph-mon[37580]: pgmap v9284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16777.130027 secs 2024-07-26T17:30:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16777.130027 secs 2024-07-26T17:30:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16777.130027 secs 2024-07-26T17:30:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:42 smithi107 ceph-mon[32659]: pgmap v9285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:42 smithi132 ceph-mon[37580]: pgmap v9285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:42 smithi112 ceph-mon[37501]: pgmap v9285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:44 smithi107 ceph-mon[32659]: pgmap v9286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:44 smithi132 ceph-mon[37580]: pgmap v9286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:44 smithi112 ceph-mon[37501]: pgmap v9286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:46 smithi107 ceph-mon[32659]: pgmap v9287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16782.130153 secs 2024-07-26T17:30:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:46 smithi132 ceph-mon[37580]: pgmap v9287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16782.130153 secs 2024-07-26T17:30:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:46 smithi112 ceph-mon[37501]: pgmap v9287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16782.130153 secs 2024-07-26T17:30:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:48 smithi107 ceph-mon[32659]: pgmap v9288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:48 smithi132 ceph-mon[37580]: pgmap v9288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:48 smithi112 ceph-mon[37501]: pgmap v9288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:50.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:50.538+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:30:50.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:50.538+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:30:50.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:30:50.538+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:30:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:50 smithi107 ceph-mon[32659]: pgmap v9289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:50 smithi132 ceph-mon[37580]: pgmap v9289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:50 smithi112 ceph-mon[37501]: pgmap v9289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16787.130285 secs 2024-07-26T17:30:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16787.130285 secs 2024-07-26T17:30:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16787.130285 secs 2024-07-26T17:30:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:52 smithi107 ceph-mon[32659]: pgmap v9290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:52 smithi132 ceph-mon[37580]: pgmap v9290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:52 smithi112 ceph-mon[37501]: pgmap v9290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:30:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:54 smithi107 ceph-mon[32659]: pgmap v9291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:54 smithi132 ceph-mon[37580]: pgmap v9291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:54 smithi112 ceph-mon[37501]: pgmap v9291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:30:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:56 smithi107 ceph-mon[32659]: pgmap v9292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16792.130409 secs 2024-07-26T17:30:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:56 smithi132 ceph-mon[37580]: pgmap v9292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16792.130409 secs 2024-07-26T17:30:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:56 smithi112 ceph-mon[37501]: pgmap v9292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16792.130409 secs 2024-07-26T17:30:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:30:58 smithi107 ceph-mon[32659]: pgmap v9293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:30:58 smithi132 ceph-mon[37580]: pgmap v9293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:30:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:30:58 smithi112 ceph-mon[37501]: pgmap v9293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:00 smithi107 ceph-mon[32659]: pgmap v9294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:31:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:00 smithi132 ceph-mon[37580]: pgmap v9294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:31:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:00 smithi112 ceph-mon[37501]: pgmap v9294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:31:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16797.130506 secs 2024-07-26T17:31:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:31:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16797.130506 secs 2024-07-26T17:31:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:31:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16797.130506 secs 2024-07-26T17:31:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:31:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: pgmap v9295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:31:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:31:02.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:31:02.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:31:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: pgmap v9295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:31:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:31:02.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:31:02.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:31:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: pgmap v9295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:31:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:31:02.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:31:02.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:31:02.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:31:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:03 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:31:03.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:03 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:31:03.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:03 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:31:03.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:03 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:31:03.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:03 smithi132 ceph-mon[37580]: pgmap v9296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:31:03.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:03 smithi132 ceph-mon[37580]: pgmap v9297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:31:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:03 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:31:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:03 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:31:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:03 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:31:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:03 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:31:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:03 smithi112 ceph-mon[37501]: pgmap v9296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:31:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:03 smithi112 ceph-mon[37501]: pgmap v9297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:31:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:03 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:31:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:03 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:31:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:03 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:31:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:03 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:31:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:03 smithi107 ceph-mon[32659]: pgmap v9296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:31:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:03 smithi107 ceph-mon[32659]: pgmap v9297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:31:05.556 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:05 smithi107 ceph-mon[32659]: pgmap v9298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:31:05.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:05.553+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:31:05.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:05.553+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:31:05.557 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:05.553+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:31:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:05 smithi132 ceph-mon[37580]: pgmap v9298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:31:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:05 smithi112 ceph-mon[37501]: pgmap v9298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:31:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16802.130589 secs 2024-07-26T17:31:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16802.130589 secs 2024-07-26T17:31:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16802.130589 secs 2024-07-26T17:31:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:07 smithi112 ceph-mon[37501]: pgmap v9299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:07 smithi107 ceph-mon[32659]: pgmap v9299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:07 smithi132 ceph-mon[37580]: pgmap v9299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:09.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:09 smithi132 ceph-mon[37580]: pgmap v9300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:31:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:09 smithi112 ceph-mon[37501]: pgmap v9300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:31:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:09 smithi107 ceph-mon[32659]: pgmap v9300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:31:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:11 smithi112 ceph-mon[37501]: pgmap v9301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:31:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16807.130701 secs 2024-07-26T17:31:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:11 smithi107 ceph-mon[32659]: pgmap v9301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:31:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16807.130701 secs 2024-07-26T17:31:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:11 smithi132 ceph-mon[37580]: pgmap v9301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:31:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16807.130701 secs 2024-07-26T17:31:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:13 smithi112 ceph-mon[37501]: pgmap v9302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:31:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:13 smithi107 ceph-mon[32659]: pgmap v9302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:31:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:13 smithi132 ceph-mon[37580]: pgmap v9302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:31:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:15 smithi112 ceph-mon[37501]: pgmap v9303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:15 smithi107 ceph-mon[32659]: pgmap v9303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:15 smithi132 ceph-mon[37580]: pgmap v9303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16812.130766 secs 2024-07-26T17:31:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16812.130766 secs 2024-07-26T17:31:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16812.130766 secs 2024-07-26T17:31:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:17 smithi112 ceph-mon[37501]: pgmap v9304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:17 smithi107 ceph-mon[32659]: pgmap v9304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:17.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:17 smithi132 ceph-mon[37580]: pgmap v9304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:19 smithi112 ceph-mon[37501]: pgmap v9305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:19 smithi107 ceph-mon[32659]: pgmap v9305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:19 smithi132 ceph-mon[37580]: pgmap v9305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:20.567+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:31:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:20.567+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:31:20.570 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:20.567+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:31:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:21 smithi112 ceph-mon[37501]: pgmap v9306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16817.130833 secs 2024-07-26T17:31:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:21 smithi107 ceph-mon[32659]: pgmap v9306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16817.130833 secs 2024-07-26T17:31:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:21 smithi132 ceph-mon[37580]: pgmap v9306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16817.130833 secs 2024-07-26T17:31:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:23 smithi112 ceph-mon[37501]: pgmap v9307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:23 smithi107 ceph-mon[32659]: pgmap v9307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:23 smithi132 ceph-mon[37580]: pgmap v9307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:25 smithi112 ceph-mon[37501]: pgmap v9308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:25 smithi107 ceph-mon[32659]: pgmap v9308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:25 smithi132 ceph-mon[37580]: pgmap v9308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16822.130931 secs 2024-07-26T17:31:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16822.130931 secs 2024-07-26T17:31:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16822.130931 secs 2024-07-26T17:31:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:27 smithi112 ceph-mon[37501]: pgmap v9309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:31:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:27 smithi107 ceph-mon[32659]: pgmap v9309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:31:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:27 smithi132 ceph-mon[37580]: pgmap v9309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:31:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:29 smithi112 ceph-mon[37501]: pgmap v9310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:29 smithi107 ceph-mon[32659]: pgmap v9310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:29 smithi132 ceph-mon[37580]: pgmap v9310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:31 smithi112 ceph-mon[37501]: pgmap v9311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16827.131053 secs 2024-07-26T17:31:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:31 smithi107 ceph-mon[32659]: pgmap v9311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16827.131053 secs 2024-07-26T17:31:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:31 smithi132 ceph-mon[37580]: pgmap v9311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16827.131053 secs 2024-07-26T17:31:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:33 smithi112 ceph-mon[37501]: pgmap v9312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:33 smithi107 ceph-mon[32659]: pgmap v9312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:33 smithi132 ceph-mon[37580]: pgmap v9312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:35.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:35.584+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:31:35.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:35.584+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:31:35.586 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:35.584+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:31:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:35 smithi112 ceph-mon[37501]: pgmap v9313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:35 smithi107 ceph-mon[32659]: pgmap v9313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:35 smithi132 ceph-mon[37580]: pgmap v9313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16832.131148 secs 2024-07-26T17:31:36.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16832.131148 secs 2024-07-26T17:31:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16832.131148 secs 2024-07-26T17:31:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:37 smithi112 ceph-mon[37501]: pgmap v9314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:37 smithi107 ceph-mon[32659]: pgmap v9314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:37 smithi132 ceph-mon[37580]: pgmap v9314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:39 smithi112 ceph-mon[37501]: pgmap v9315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:39 smithi107 ceph-mon[32659]: pgmap v9315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:39 smithi132 ceph-mon[37580]: pgmap v9315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:41 smithi112 ceph-mon[37501]: pgmap v9316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16837.131250 secs 2024-07-26T17:31:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:41 smithi107 ceph-mon[32659]: pgmap v9316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16837.131250 secs 2024-07-26T17:31:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:41 smithi132 ceph-mon[37580]: pgmap v9316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16837.131250 secs 2024-07-26T17:31:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:43 smithi112 ceph-mon[37501]: pgmap v9317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:43 smithi107 ceph-mon[32659]: pgmap v9317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:43 smithi132 ceph-mon[37580]: pgmap v9317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:44.548+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:31:44.549 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:44.548+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:31:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:44.548+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:31:44.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:44.548+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:31:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:45 smithi107 ceph-mon[32659]: pgmap v9318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:45 smithi132 ceph-mon[37580]: pgmap v9318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:45 smithi112 ceph-mon[37501]: pgmap v9318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16842.131368 secs 2024-07-26T17:31:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16842.131368 secs 2024-07-26T17:31:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16842.131368 secs 2024-07-26T17:31:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:47 smithi107 ceph-mon[32659]: pgmap v9319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:47 smithi132 ceph-mon[37580]: pgmap v9319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:47 smithi112 ceph-mon[37501]: pgmap v9319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:48 smithi107 ceph-mon[32659]: pgmap v9320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:48 smithi132 ceph-mon[37580]: pgmap v9320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:48 smithi112 ceph-mon[37501]: pgmap v9320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:50.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:50.586+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:31:50.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:50.586+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:31:50.588 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:31:50.586+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:31:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:51 smithi132 ceph-mon[37580]: pgmap v9321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16847.131451 secs 2024-07-26T17:31:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:51 smithi112 ceph-mon[37501]: pgmap v9321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16847.131451 secs 2024-07-26T17:31:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:51 smithi107 ceph-mon[32659]: pgmap v9321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16847.131451 secs 2024-07-26T17:31:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:53 smithi132 ceph-mon[37580]: pgmap v9322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:53 smithi112 ceph-mon[37501]: pgmap v9322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:53 smithi107 ceph-mon[32659]: pgmap v9322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:31:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:55 smithi112 ceph-mon[37501]: pgmap v9323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:55 smithi107 ceph-mon[32659]: pgmap v9323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:55 smithi132 ceph-mon[37580]: pgmap v9323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:31:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16852.131567 secs 2024-07-26T17:31:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16852.131567 secs 2024-07-26T17:31:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16852.131567 secs 2024-07-26T17:31:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:57 smithi112 ceph-mon[37501]: pgmap v9324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:57 smithi107 ceph-mon[32659]: pgmap v9324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:57 smithi132 ceph-mon[37580]: pgmap v9324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:31:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:31:59 smithi112 ceph-mon[37501]: pgmap v9325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:31:59 smithi107 ceph-mon[32659]: pgmap v9325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:31:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:31:59 smithi132 ceph-mon[37580]: pgmap v9325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:01 smithi112 ceph-mon[37501]: pgmap v9326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16857.131690 secs 2024-07-26T17:32:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:01 smithi107 ceph-mon[32659]: pgmap v9326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16857.131690 secs 2024-07-26T17:32:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:01 smithi132 ceph-mon[37580]: pgmap v9326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16857.131690 secs 2024-07-26T17:32:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:03 smithi112 ceph-mon[37501]: pgmap v9327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:32:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:03 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:32:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:03 smithi107 ceph-mon[32659]: pgmap v9327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:32:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:03 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:32:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:03 smithi132 ceph-mon[37580]: pgmap v9327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:32:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:03 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:32:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:32:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:32:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:04 smithi112 ceph-mon[37501]: pgmap v9328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:32:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:04 smithi112 ceph-mon[37501]: pgmap v9329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:32:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:32:04.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:32:04.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:32:04.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:04 smithi107 ceph-mon[32659]: pgmap v9328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:32:04.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:04 smithi107 ceph-mon[32659]: pgmap v9329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:32:04.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:32:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:32:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:32:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:04 smithi132 ceph-mon[37580]: pgmap v9328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:32:04.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:04 smithi132 ceph-mon[37580]: pgmap v9329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:32:04.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:32:05.603 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:05.601+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:32:05.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:05.601+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:32:05.604 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:05.601+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:32:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:06 smithi112 ceph-mon[37501]: pgmap v9330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:32:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16862.131815 secs 2024-07-26T17:32:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:06 smithi107 ceph-mon[32659]: pgmap v9330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:32:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16862.131815 secs 2024-07-26T17:32:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:06 smithi132 ceph-mon[37580]: pgmap v9330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:32:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16862.131815 secs 2024-07-26T17:32:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:08 smithi107 ceph-mon[32659]: pgmap v9331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:32:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:08 smithi132 ceph-mon[37580]: pgmap v9331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:32:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:08 smithi112 ceph-mon[37501]: pgmap v9331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:32:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:10 smithi107 ceph-mon[32659]: pgmap v9332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:32:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:10 smithi132 ceph-mon[37580]: pgmap v9332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:32:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:10 smithi112 ceph-mon[37501]: pgmap v9332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:32:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16867.131878 secs 2024-07-26T17:32:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16867.131878 secs 2024-07-26T17:32:11.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16867.131878 secs 2024-07-26T17:32:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:12 smithi107 ceph-mon[32659]: pgmap v9333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:32:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:12 smithi132 ceph-mon[37580]: pgmap v9333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:32:12.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:12 smithi112 ceph-mon[37501]: pgmap v9333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:32:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:13 smithi107 ceph-mon[32659]: pgmap v9334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:32:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:13 smithi132 ceph-mon[37580]: pgmap v9334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:32:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:13 smithi112 ceph-mon[37501]: pgmap v9334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:32:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:16 smithi107 ceph-mon[32659]: pgmap v9335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16872.131946 secs 2024-07-26T17:32:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:16 smithi132 ceph-mon[37580]: pgmap v9335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16872.131946 secs 2024-07-26T17:32:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:16 smithi112 ceph-mon[37501]: pgmap v9335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16872.131946 secs 2024-07-26T17:32:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:18 smithi107 ceph-mon[32659]: pgmap v9336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:18 smithi132 ceph-mon[37580]: pgmap v9336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:18 smithi112 ceph-mon[37501]: pgmap v9336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:20.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:20.617+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:32:20.619 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:20.617+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:32:20.620 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:20.617+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:32:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:20 smithi107 ceph-mon[32659]: pgmap v9337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:20 smithi132 ceph-mon[37580]: pgmap v9337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:20 smithi112 ceph-mon[37501]: pgmap v9337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16877.132044 secs 2024-07-26T17:32:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16877.132044 secs 2024-07-26T17:32:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16877.132044 secs 2024-07-26T17:32:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:22 smithi107 ceph-mon[32659]: pgmap v9338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:22 smithi132 ceph-mon[37580]: pgmap v9338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:22 smithi112 ceph-mon[37501]: pgmap v9338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:23 smithi107 ceph-mon[32659]: pgmap v9339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:23 smithi132 ceph-mon[37580]: pgmap v9339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:23 smithi112 ceph-mon[37501]: pgmap v9339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:26 smithi107 ceph-mon[32659]: pgmap v9340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16882.132180 secs 2024-07-26T17:32:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:26 smithi132 ceph-mon[37580]: pgmap v9340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:26.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16882.132180 secs 2024-07-26T17:32:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:26 smithi112 ceph-mon[37501]: pgmap v9340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16882.132180 secs 2024-07-26T17:32:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:28 smithi107 ceph-mon[32659]: pgmap v9341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:28 smithi132 ceph-mon[37580]: pgmap v9341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:28 smithi112 ceph-mon[37501]: pgmap v9341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:30 smithi107 ceph-mon[32659]: pgmap v9342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:30 smithi132 ceph-mon[37580]: pgmap v9342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:30 smithi112 ceph-mon[37501]: pgmap v9342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16887.132313 secs 2024-07-26T17:32:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:31 smithi107 ceph-mon[32659]: pgmap v9343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16887.132313 secs 2024-07-26T17:32:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:31 smithi132 ceph-mon[37580]: pgmap v9343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16887.132313 secs 2024-07-26T17:32:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:31 smithi112 ceph-mon[37501]: pgmap v9343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:34 smithi107 ceph-mon[32659]: pgmap v9344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:34 smithi132 ceph-mon[37580]: pgmap v9344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:34 smithi112 ceph-mon[37501]: pgmap v9344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:35.623+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:32:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:35.623+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:32:35.625 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:35.623+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:32:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:36 smithi107 ceph-mon[32659]: pgmap v9345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16892.132410 secs 2024-07-26T17:32:36.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:36 smithi132 ceph-mon[37580]: pgmap v9345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:36.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16892.132410 secs 2024-07-26T17:32:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:36 smithi112 ceph-mon[37501]: pgmap v9345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16892.132410 secs 2024-07-26T17:32:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:37 smithi107 ceph-mon[32659]: pgmap v9346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:37 smithi132 ceph-mon[37580]: pgmap v9346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:37 smithi112 ceph-mon[37501]: pgmap v9346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:40 smithi107 ceph-mon[32659]: pgmap v9347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:40 smithi132 ceph-mon[37580]: pgmap v9347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:40 smithi112 ceph-mon[37501]: pgmap v9347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16897.132476 secs 2024-07-26T17:32:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16897.132476 secs 2024-07-26T17:32:41.851 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16897.132476 secs 2024-07-26T17:32:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:42 smithi107 ceph-mon[32659]: pgmap v9348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:42 smithi132 ceph-mon[37580]: pgmap v9348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:42 smithi112 ceph-mon[37501]: pgmap v9348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:43 smithi107 ceph-mon[32659]: pgmap v9349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:43 smithi132 ceph-mon[37580]: pgmap v9349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:43 smithi112 ceph-mon[37501]: pgmap v9349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:46 smithi107 ceph-mon[32659]: pgmap v9350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16902.132555 secs 2024-07-26T17:32:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:46 smithi132 ceph-mon[37580]: pgmap v9350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16902.132555 secs 2024-07-26T17:32:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:46 smithi112 ceph-mon[37501]: pgmap v9350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16902.132555 secs 2024-07-26T17:32:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:47 smithi107 ceph-mon[32659]: pgmap v9351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:47 smithi132 ceph-mon[37580]: pgmap v9351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:47 smithi112 ceph-mon[37501]: pgmap v9351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:50.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:50.626+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:32:50.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:50.626+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:32:50.628 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:32:50.626+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:32:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:50 smithi107 ceph-mon[32659]: pgmap v9352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:50 smithi132 ceph-mon[37580]: pgmap v9352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:50 smithi112 ceph-mon[37501]: pgmap v9352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16907.132697 secs 2024-07-26T17:32:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:51 smithi107 ceph-mon[32659]: pgmap v9353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16907.132697 secs 2024-07-26T17:32:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:51 smithi132 ceph-mon[37580]: pgmap v9353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16907.132697 secs 2024-07-26T17:32:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:51 smithi112 ceph-mon[37501]: pgmap v9353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:32:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:53 smithi107 ceph-mon[32659]: pgmap v9354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:53 smithi132 ceph-mon[37580]: pgmap v9354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:53 smithi112 ceph-mon[37501]: pgmap v9354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:32:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:55 smithi107 ceph-mon[32659]: pgmap v9355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:55 smithi132 ceph-mon[37580]: pgmap v9355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:55 smithi112 ceph-mon[37501]: pgmap v9355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:32:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16912.132834 secs 2024-07-26T17:32:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16912.132834 secs 2024-07-26T17:32:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16912.132834 secs 2024-07-26T17:32:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:57 smithi132 ceph-mon[37580]: pgmap v9356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:57 smithi112 ceph-mon[37501]: pgmap v9356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:57 smithi107 ceph-mon[32659]: pgmap v9356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:32:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:32:59 smithi107 ceph-mon[32659]: pgmap v9357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:32:59 smithi132 ceph-mon[37580]: pgmap v9357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:32:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:32:59 smithi112 ceph-mon[37501]: pgmap v9357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:33:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16917.132915 secs 2024-07-26T17:33:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16917.132915 secs 2024-07-26T17:33:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16917.132915 secs 2024-07-26T17:33:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:01 smithi107 ceph-mon[32659]: pgmap v9358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:33:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:01 smithi132 ceph-mon[37580]: pgmap v9358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:33:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:01 smithi112 ceph-mon[37501]: pgmap v9358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:33:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:03 smithi107 ceph-mon[32659]: pgmap v9359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:03 smithi132 ceph-mon[37580]: pgmap v9359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:03 smithi112 ceph-mon[37501]: pgmap v9359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:04 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:33:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:33:04.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:04 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:33:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:04 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:33:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:33:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:04 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:33:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:04 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:33:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:33:04.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:04 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:33:05.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:05.633+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:33:05.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:05.633+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:33:05.636 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:05.633+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:33:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:33:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:05 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:33:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:05 smithi132 ceph-mon[37580]: pgmap v9360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T17:33:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:05 smithi132 ceph-mon[37580]: pgmap v9361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:33:05.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:33:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:33:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:05 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:33:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:05 smithi112 ceph-mon[37501]: pgmap v9360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T17:33:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:05 smithi112 ceph-mon[37501]: pgmap v9361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:33:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:33:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:33:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:05 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:33:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:05 smithi107 ceph-mon[32659]: pgmap v9360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T17:33:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:05 smithi107 ceph-mon[32659]: pgmap v9361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:33:06.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:33:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16922.133061 secs 2024-07-26T17:33:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16922.133061 secs 2024-07-26T17:33:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16922.133061 secs 2024-07-26T17:33:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:07 smithi132 ceph-mon[37580]: pgmap v9362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:33:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:07 smithi112 ceph-mon[37501]: pgmap v9362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:33:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:07 smithi107 ceph-mon[32659]: pgmap v9362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:33:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:10 smithi132 ceph-mon[37580]: pgmap v9363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:33:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:10 smithi112 ceph-mon[37501]: pgmap v9363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:33:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:10 smithi107 ceph-mon[32659]: pgmap v9363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:33:11.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16927.133203 secs 2024-07-26T17:33:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16927.133203 secs 2024-07-26T17:33:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16927.133203 secs 2024-07-26T17:33:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:12 smithi112 ceph-mon[37501]: pgmap v9364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:33:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:12 smithi107 ceph-mon[32659]: pgmap v9364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:33:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:12 smithi132 ceph-mon[37580]: pgmap v9364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:33:14.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:14 smithi112 ceph-mon[37501]: pgmap v9365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:33:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:14 smithi107 ceph-mon[32659]: pgmap v9365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:33:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:14 smithi132 ceph-mon[37580]: pgmap v9365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:33:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:15.564+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:33:15.566 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:15.564+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:33:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:15.564+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:33:15.567 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:15.565+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:33:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:16 smithi112 ceph-mon[37501]: pgmap v9366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:33:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16932.133328 secs 2024-07-26T17:33:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:16 smithi107 ceph-mon[32659]: pgmap v9366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:33:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16932.133328 secs 2024-07-26T17:33:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:16 smithi132 ceph-mon[37580]: pgmap v9366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:33:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16932.133328 secs 2024-07-26T17:33:18.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:18 smithi112 ceph-mon[37501]: pgmap v9367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:18 smithi107 ceph-mon[32659]: pgmap v9367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:18 smithi132 ceph-mon[37580]: pgmap v9367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:20.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:20.649+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:33:20.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:20.649+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:33:20.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:20.649+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:33:20.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:20 smithi107 ceph-mon[32659]: pgmap v9368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:20 smithi112 ceph-mon[37501]: pgmap v9368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:20 smithi132 ceph-mon[37580]: pgmap v9368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16937.133412 secs 2024-07-26T17:33:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16937.133412 secs 2024-07-26T17:33:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16937.133412 secs 2024-07-26T17:33:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:22 smithi112 ceph-mon[37501]: pgmap v9369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:22 smithi107 ceph-mon[32659]: pgmap v9369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:22 smithi132 ceph-mon[37580]: pgmap v9369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:24 smithi112 ceph-mon[37501]: pgmap v9370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:24 smithi107 ceph-mon[32659]: pgmap v9370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:24 smithi132 ceph-mon[37580]: pgmap v9370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:26 smithi112 ceph-mon[37501]: pgmap v9371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16942.133532 secs 2024-07-26T17:33:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:26 smithi107 ceph-mon[32659]: pgmap v9371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16942.133532 secs 2024-07-26T17:33:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:26 smithi132 ceph-mon[37580]: pgmap v9371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:26.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16942.133532 secs 2024-07-26T17:33:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:28 smithi112 ceph-mon[37501]: pgmap v9372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:28 smithi107 ceph-mon[32659]: pgmap v9372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:28 smithi132 ceph-mon[37580]: pgmap v9372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:30 smithi112 ceph-mon[37501]: pgmap v9373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:30 smithi107 ceph-mon[32659]: pgmap v9373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:30 smithi132 ceph-mon[37580]: pgmap v9373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16947.133618 secs 2024-07-26T17:33:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16947.133618 secs 2024-07-26T17:33:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16947.133618 secs 2024-07-26T17:33:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:32 smithi112 ceph-mon[37501]: pgmap v9374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:32 smithi107 ceph-mon[32659]: pgmap v9374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:32 smithi132 ceph-mon[37580]: pgmap v9374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:34 smithi112 ceph-mon[37501]: pgmap v9375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:34 smithi107 ceph-mon[32659]: pgmap v9375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:34 smithi132 ceph-mon[37580]: pgmap v9375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:35.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:35.653+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:33:35.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:35.653+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:33:35.655 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:35.653+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:33:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:36 smithi112 ceph-mon[37501]: pgmap v9376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16952.133772 secs 2024-07-26T17:33:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:36 smithi107 ceph-mon[32659]: pgmap v9376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16952.133772 secs 2024-07-26T17:33:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:36 smithi132 ceph-mon[37580]: pgmap v9376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16952.133772 secs 2024-07-26T17:33:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:38 smithi112 ceph-mon[37501]: pgmap v9377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:38 smithi107 ceph-mon[32659]: pgmap v9377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:38 smithi132 ceph-mon[37580]: pgmap v9377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:40 smithi107 ceph-mon[32659]: pgmap v9378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:40 smithi112 ceph-mon[37501]: pgmap v9378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:40 smithi132 ceph-mon[37580]: pgmap v9378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16957.133890 secs 2024-07-26T17:33:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16957.133890 secs 2024-07-26T17:33:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16957.133890 secs 2024-07-26T17:33:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:42 smithi112 ceph-mon[37501]: pgmap v9379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:42 smithi107 ceph-mon[32659]: pgmap v9379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:42 smithi132 ceph-mon[37580]: pgmap v9379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:44 smithi112 ceph-mon[37501]: pgmap v9380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:44 smithi107 ceph-mon[32659]: pgmap v9380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:44 smithi132 ceph-mon[37580]: pgmap v9380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:33:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:46 smithi112 ceph-mon[37501]: pgmap v9381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16962.134011 secs 2024-07-26T17:33:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:46 smithi107 ceph-mon[32659]: pgmap v9381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:46.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16962.134011 secs 2024-07-26T17:33:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:46 smithi132 ceph-mon[37580]: pgmap v9381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:33:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16962.134011 secs 2024-07-26T17:33:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:48 smithi107 ceph-mon[32659]: pgmap v9382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:48 smithi132 ceph-mon[37580]: pgmap v9382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:48 smithi112 ceph-mon[37501]: pgmap v9382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:33:50.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:50.668+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:33:50.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:50.668+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:33:50.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:33:50.668+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:33:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:50 smithi107 ceph-mon[32659]: pgmap v9383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:50 smithi132 ceph-mon[37580]: pgmap v9383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:50 smithi112 ceph-mon[37501]: pgmap v9383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:33:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16967.134098 secs 2024-07-26T17:33:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16967.134098 secs 2024-07-26T17:33:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16967.134098 secs 2024-07-26T17:33:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:52 smithi107 ceph-mon[32659]: pgmap v9384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:33:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:52 smithi132 ceph-mon[37580]: pgmap v9384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:33:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:52 smithi112 ceph-mon[37501]: pgmap v9384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:33:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:54 smithi107 ceph-mon[32659]: pgmap v9385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:33:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:54 smithi132 ceph-mon[37580]: pgmap v9385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:33:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:54 smithi112 ceph-mon[37501]: pgmap v9385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:33:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:56 smithi107 ceph-mon[32659]: pgmap v9386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:33:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16972.134165 secs 2024-07-26T17:33:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:56 smithi132 ceph-mon[37580]: pgmap v9386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:33:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16972.134165 secs 2024-07-26T17:33:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:56 smithi112 ceph-mon[37501]: pgmap v9386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:33:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16972.134165 secs 2024-07-26T17:33:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:33:57 smithi107 ceph-mon[32659]: pgmap v9387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:33:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:33:57 smithi132 ceph-mon[37580]: pgmap v9387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:33:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:33:57 smithi112 ceph-mon[37501]: pgmap v9387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:34:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:00 smithi112 ceph-mon[37501]: pgmap v9388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:34:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:00 smithi107 ceph-mon[32659]: pgmap v9388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:34:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:00 smithi132 ceph-mon[37580]: pgmap v9388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:34:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16977.134232 secs 2024-07-26T17:34:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16977.134232 secs 2024-07-26T17:34:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16977.134232 secs 2024-07-26T17:34:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:02 smithi112 ceph-mon[37501]: pgmap v9389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:34:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:02 smithi107 ceph-mon[32659]: pgmap v9389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:34:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:02 smithi132 ceph-mon[37580]: pgmap v9389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:34:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:04 smithi112 ceph-mon[37501]: pgmap v9390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:34:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:04 smithi107 ceph-mon[32659]: pgmap v9390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:34:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:04 smithi132 ceph-mon[37580]: pgmap v9390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:34:05.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:05.682+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:34:05.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:05.682+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:34:05.684 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:05.682+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:34:06.661 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:06 smithi132 ceph-mon[37580]: pgmap v9391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:06.662 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16982.134352 secs 2024-07-26T17:34:06.662 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:06 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:34:06.662 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.663 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.663 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.663 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:06 smithi107 ceph-mon[32659]: pgmap v9391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:06.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16982.134352 secs 2024-07-26T17:34:06.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:06 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:34:06.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.672 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.694 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:06 smithi112 ceph-mon[37501]: pgmap v9391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:06.695 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16982.134352 secs 2024-07-26T17:34:06.695 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:06 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:34:06.695 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.695 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.695 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:06.695 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:08 smithi107 ceph-mon[32659]: pgmap v9392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:08 smithi132 ceph-mon[37580]: pgmap v9392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:08 smithi112 ceph-mon[37501]: pgmap v9392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:08.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:10.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:09 smithi107 ceph-mon[32659]: pgmap v9393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:09 smithi132 ceph-mon[37580]: pgmap v9393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:09 smithi112 ceph-mon[37501]: pgmap v9393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:34:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:34:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:10 smithi132 ceph-mon[37580]: pgmap v9394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:34:11.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:11.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16987.134471 secs 2024-07-26T17:34:11.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:34:11.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:34:11.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:10 smithi112 ceph-mon[37501]: pgmap v9394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:34:11.103 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:11.103 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16987.134471 secs 2024-07-26T17:34:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:34:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:34:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:10 smithi107 ceph-mon[32659]: pgmap v9394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:34:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:34:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16987.134471 secs 2024-07-26T17:34:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:12 smithi112 ceph-mon[37501]: pgmap v9395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:34:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:12 smithi107 ceph-mon[32659]: pgmap v9395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:34:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:12 smithi132 ceph-mon[37580]: pgmap v9395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:34:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:14 smithi112 ceph-mon[37501]: pgmap v9396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:34:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:14 smithi107 ceph-mon[32659]: pgmap v9396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:34:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:14 smithi132 ceph-mon[37580]: pgmap v9396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:34:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16992.134595 secs 2024-07-26T17:34:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16992.134595 secs 2024-07-26T17:34:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16992.134595 secs 2024-07-26T17:34:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:16 smithi112 ceph-mon[37501]: pgmap v9397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:34:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:16 smithi107 ceph-mon[32659]: pgmap v9397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:34:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:16 smithi132 ceph-mon[37580]: pgmap v9397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:34:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:18 smithi112 ceph-mon[37501]: pgmap v9398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T17:34:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:18 smithi107 ceph-mon[32659]: pgmap v9398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T17:34:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:18 smithi132 ceph-mon[37580]: pgmap v9398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T17:34:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:20.697+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:34:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:20.697+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:34:20.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:20.697+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:34:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:20 smithi107 ceph-mon[32659]: pgmap v9399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:34:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 16997.134707 secs 2024-07-26T17:34:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:20 smithi112 ceph-mon[37501]: pgmap v9399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:34:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 16997.134707 secs 2024-07-26T17:34:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:20 smithi132 ceph-mon[37580]: pgmap v9399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:34:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 16997.134707 secs 2024-07-26T17:34:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:22 smithi112 ceph-mon[37501]: pgmap v9400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:22 smithi107 ceph-mon[32659]: pgmap v9400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:22 smithi132 ceph-mon[37580]: pgmap v9400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:24 smithi112 ceph-mon[37501]: pgmap v9401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:24 smithi107 ceph-mon[32659]: pgmap v9401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:24 smithi132 ceph-mon[37580]: pgmap v9401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17002.134799 secs 2024-07-26T17:34:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17002.134799 secs 2024-07-26T17:34:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17002.134799 secs 2024-07-26T17:34:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:26 smithi112 ceph-mon[37501]: pgmap v9402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:27.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:26 smithi107 ceph-mon[32659]: pgmap v9402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:26 smithi132 ceph-mon[37580]: pgmap v9402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:28 smithi112 ceph-mon[37501]: pgmap v9403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:28 smithi107 ceph-mon[32659]: pgmap v9403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:28 smithi132 ceph-mon[37580]: pgmap v9403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:30 smithi112 ceph-mon[37501]: pgmap v9404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17007.134915 secs 2024-07-26T17:34:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:30 smithi107 ceph-mon[32659]: pgmap v9404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17007.134915 secs 2024-07-26T17:34:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:30 smithi132 ceph-mon[37580]: pgmap v9404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17007.134915 secs 2024-07-26T17:34:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:32 smithi112 ceph-mon[37501]: pgmap v9405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:33.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:32 smithi107 ceph-mon[32659]: pgmap v9405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:32 smithi132 ceph-mon[37580]: pgmap v9405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:34 smithi112 ceph-mon[37501]: pgmap v9406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:34 smithi107 ceph-mon[32659]: pgmap v9406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:34 smithi132 ceph-mon[37580]: pgmap v9406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:35.710 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:35.709+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:34:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:35.709+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:34:35.711 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:35.709+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:34:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17012.135036 secs 2024-07-26T17:34:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17012.135036 secs 2024-07-26T17:34:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17012.135036 secs 2024-07-26T17:34:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:36 smithi107 ceph-mon[32659]: pgmap v9407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:36 smithi132 ceph-mon[37580]: pgmap v9407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:36 smithi112 ceph-mon[37501]: pgmap v9407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:38 smithi107 ceph-mon[32659]: pgmap v9408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:38 smithi132 ceph-mon[37580]: pgmap v9408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:38 smithi112 ceph-mon[37501]: pgmap v9408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:41.252 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:40 smithi112 ceph-mon[37501]: pgmap v9409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:41.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17017.135148 secs 2024-07-26T17:34:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:40 smithi107 ceph-mon[32659]: pgmap v9409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:41.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17017.135148 secs 2024-07-26T17:34:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:40 smithi132 ceph-mon[37580]: pgmap v9409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17017.135148 secs 2024-07-26T17:34:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:42 smithi107 ceph-mon[32659]: pgmap v9410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:42 smithi132 ceph-mon[37580]: pgmap v9410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:42 smithi112 ceph-mon[37501]: pgmap v9410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:44 smithi107 ceph-mon[32659]: pgmap v9411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:44 smithi132 ceph-mon[37580]: pgmap v9411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:44 smithi112 ceph-mon[37501]: pgmap v9411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17022.135273 secs 2024-07-26T17:34:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17022.135273 secs 2024-07-26T17:34:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17022.135273 secs 2024-07-26T17:34:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:46 smithi107 ceph-mon[32659]: pgmap v9412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:46 smithi132 ceph-mon[37580]: pgmap v9412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:46 smithi112 ceph-mon[37501]: pgmap v9412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:49.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:49 smithi107 ceph-mon[32659]: pgmap v9413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:49 smithi132 ceph-mon[37580]: pgmap v9413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:49 smithi112 ceph-mon[37501]: pgmap v9413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:50.719+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:34:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:50.719+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:34:50.721 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:34:50.719+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:34:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:51 smithi107 ceph-mon[32659]: pgmap v9414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17027.135382 secs 2024-07-26T17:34:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:51 smithi132 ceph-mon[37580]: pgmap v9414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17027.135382 secs 2024-07-26T17:34:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:51 smithi112 ceph-mon[37501]: pgmap v9414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17027.135382 secs 2024-07-26T17:34:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:53 smithi107 ceph-mon[32659]: pgmap v9415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:53 smithi132 ceph-mon[37580]: pgmap v9415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:53 smithi112 ceph-mon[37501]: pgmap v9415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:34:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:55 smithi107 ceph-mon[32659]: pgmap v9416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:55 smithi132 ceph-mon[37580]: pgmap v9416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:55 smithi112 ceph-mon[37501]: pgmap v9416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17032.135506 secs 2024-07-26T17:34:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17032.135506 secs 2024-07-26T17:34:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17032.135506 secs 2024-07-26T17:34:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:57 smithi107 ceph-mon[32659]: pgmap v9417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:57 smithi132 ceph-mon[37580]: pgmap v9417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:57 smithi112 ceph-mon[37501]: pgmap v9417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:34:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:34:59 smithi107 ceph-mon[32659]: pgmap v9418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:34:59 smithi132 ceph-mon[37580]: pgmap v9418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:34:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:34:59 smithi112 ceph-mon[37501]: pgmap v9418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:01 smithi132 ceph-mon[37580]: pgmap v9419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17037.135642 secs 2024-07-26T17:35:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:01 smithi112 ceph-mon[37501]: pgmap v9419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17037.135642 secs 2024-07-26T17:35:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:01 smithi107 ceph-mon[32659]: pgmap v9419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17037.135642 secs 2024-07-26T17:35:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:03 smithi132 ceph-mon[37580]: pgmap v9420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:03 smithi112 ceph-mon[37501]: pgmap v9420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:03 smithi107 ceph-mon[32659]: pgmap v9420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:05 smithi132 ceph-mon[37580]: pgmap v9421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:05.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:05 smithi107 ceph-mon[32659]: pgmap v9421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:05 smithi112 ceph-mon[37501]: pgmap v9421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:05.721+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:35:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:05.722+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:35:05.724 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:05.722+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:35:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17042.135785 secs 2024-07-26T17:35:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17042.135785 secs 2024-07-26T17:35:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17042.135785 secs 2024-07-26T17:35:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:07 smithi112 ceph-mon[37501]: pgmap v9422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:07 smithi107 ceph-mon[32659]: pgmap v9422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:07 smithi132 ceph-mon[37580]: pgmap v9422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:09 smithi112 ceph-mon[37501]: pgmap v9423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:09 smithi107 ceph-mon[32659]: pgmap v9423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:09 smithi132 ceph-mon[37580]: pgmap v9423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:35:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:35:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:35:11.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:11 smithi112 ceph-mon[37501]: pgmap v9424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:11.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17047.135928 secs 2024-07-26T17:35:11.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:35:11.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:35:11.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:35:11.460 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:11 smithi132 ceph-mon[37580]: pgmap v9424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:11.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17047.135928 secs 2024-07-26T17:35:11.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:35:11.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:35:11.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:35:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:11 smithi107 ceph-mon[32659]: pgmap v9424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17047.135928 secs 2024-07-26T17:35:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:35:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:35:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:35:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:12 smithi112 ceph-mon[37501]: pgmap v9425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:35:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:12 smithi112 ceph-mon[37501]: pgmap v9426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:35:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:12 smithi107 ceph-mon[32659]: pgmap v9425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:35:12.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:12 smithi107 ceph-mon[32659]: pgmap v9426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:35:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:12 smithi132 ceph-mon[37580]: pgmap v9425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:35:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:12 smithi132 ceph-mon[37580]: pgmap v9426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:35:14.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:14 smithi112 ceph-mon[37501]: pgmap v9427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:35:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:14 smithi107 ceph-mon[32659]: pgmap v9427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:35:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:14 smithi132 ceph-mon[37580]: pgmap v9427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:35:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:16 smithi112 ceph-mon[37501]: pgmap v9428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:35:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17052.136094 secs 2024-07-26T17:35:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:16 smithi107 ceph-mon[32659]: pgmap v9428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:35:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17052.136094 secs 2024-07-26T17:35:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:16 smithi132 ceph-mon[37580]: pgmap v9428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:35:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17052.136094 secs 2024-07-26T17:35:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:18 smithi112 ceph-mon[37501]: pgmap v9429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:18 smithi107 ceph-mon[32659]: pgmap v9429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:18 smithi132 ceph-mon[37580]: pgmap v9429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:20 smithi112 ceph-mon[37501]: pgmap v9430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:35:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:20 smithi107 ceph-mon[32659]: pgmap v9430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:35:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:20 smithi132 ceph-mon[37580]: pgmap v9430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:35:20.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:20.737+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:35:20.740 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:20.737+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:35:20.741 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:20.737+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:35:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17057.136210 secs 2024-07-26T17:35:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17057.136210 secs 2024-07-26T17:35:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17057.136210 secs 2024-07-26T17:35:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:22 smithi112 ceph-mon[37501]: pgmap v9431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:35:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:22 smithi107 ceph-mon[32659]: pgmap v9431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:35:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:22 smithi132 ceph-mon[37580]: pgmap v9431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:35:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:24 smithi112 ceph-mon[37501]: pgmap v9432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:24 smithi107 ceph-mon[32659]: pgmap v9432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:24 smithi132 ceph-mon[37580]: pgmap v9432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:26 smithi112 ceph-mon[37501]: pgmap v9433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17062.136312 secs 2024-07-26T17:35:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:26 smithi107 ceph-mon[32659]: pgmap v9433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17062.136312 secs 2024-07-26T17:35:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:26 smithi132 ceph-mon[37580]: pgmap v9433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17062.136312 secs 2024-07-26T17:35:28.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:28 smithi112 ceph-mon[37501]: pgmap v9434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:28 smithi107 ceph-mon[32659]: pgmap v9434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:28 smithi132 ceph-mon[37580]: pgmap v9434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:30 smithi112 ceph-mon[37501]: pgmap v9435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:30 smithi107 ceph-mon[32659]: pgmap v9435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:30 smithi132 ceph-mon[37580]: pgmap v9435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17067.136437 secs 2024-07-26T17:35:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17067.136437 secs 2024-07-26T17:35:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17067.136437 secs 2024-07-26T17:35:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:32 smithi112 ceph-mon[37501]: pgmap v9436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:32 smithi107 ceph-mon[32659]: pgmap v9436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:32 smithi132 ceph-mon[37580]: pgmap v9436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:34 smithi112 ceph-mon[37501]: pgmap v9437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:34 smithi107 ceph-mon[32659]: pgmap v9437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:34 smithi132 ceph-mon[37580]: pgmap v9437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:35.755 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:35.754+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:35:35.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:35.754+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:35:35.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:35.754+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:35:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:36 smithi112 ceph-mon[37501]: pgmap v9438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17072.136501 secs 2024-07-26T17:35:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:36 smithi107 ceph-mon[32659]: pgmap v9438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17072.136501 secs 2024-07-26T17:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:36 smithi132 ceph-mon[37580]: pgmap v9438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17072.136501 secs 2024-07-26T17:35:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:38 smithi112 ceph-mon[37501]: pgmap v9439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:38 smithi107 ceph-mon[32659]: pgmap v9439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:38 smithi132 ceph-mon[37580]: pgmap v9439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:40 smithi107 ceph-mon[32659]: pgmap v9440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:40 smithi132 ceph-mon[37580]: pgmap v9440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:40 smithi112 ceph-mon[37501]: pgmap v9440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17077.136623 secs 2024-07-26T17:35:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17077.136623 secs 2024-07-26T17:35:41.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17077.136623 secs 2024-07-26T17:35:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:42 smithi107 ceph-mon[32659]: pgmap v9441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:42.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:42 smithi132 ceph-mon[37580]: pgmap v9441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:42 smithi112 ceph-mon[37501]: pgmap v9441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:44.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:44 smithi107 ceph-mon[32659]: pgmap v9442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:44 smithi132 ceph-mon[37580]: pgmap v9442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:44.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:44 smithi112 ceph-mon[37501]: pgmap v9442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:45 smithi112 ceph-mon[37501]: pgmap v9443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:45 smithi107 ceph-mon[32659]: pgmap v9443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:45 smithi132 ceph-mon[37580]: pgmap v9443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:46.955 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17082.136763 secs 2024-07-26T17:35:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17082.136763 secs 2024-07-26T17:35:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17082.136763 secs 2024-07-26T17:35:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:47 smithi112 ceph-mon[37501]: pgmap v9444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:47 smithi107 ceph-mon[32659]: pgmap v9444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:47 smithi132 ceph-mon[37580]: pgmap v9444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:50 smithi112 ceph-mon[37501]: pgmap v9445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:50 smithi107 ceph-mon[32659]: pgmap v9445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:50 smithi132 ceph-mon[37580]: pgmap v9445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:50.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:50.769+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:35:50.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:50.769+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:35:50.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:35:50.769+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:35:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17087.136902 secs 2024-07-26T17:35:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17087.136902 secs 2024-07-26T17:35:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17087.136902 secs 2024-07-26T17:35:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:52 smithi112 ceph-mon[37501]: pgmap v9446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:52 smithi107 ceph-mon[32659]: pgmap v9446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:52 smithi132 ceph-mon[37580]: pgmap v9446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:35:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:54 smithi112 ceph-mon[37501]: pgmap v9447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:54 smithi107 ceph-mon[32659]: pgmap v9447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:54 smithi132 ceph-mon[37580]: pgmap v9447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:35:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:56 smithi112 ceph-mon[37501]: pgmap v9448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17092.137019 secs 2024-07-26T17:35:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:56 smithi107 ceph-mon[32659]: pgmap v9448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:56.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17092.137019 secs 2024-07-26T17:35:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:56 smithi132 ceph-mon[37580]: pgmap v9448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17092.137019 secs 2024-07-26T17:35:58.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:35:58 smithi112 ceph-mon[37501]: pgmap v9449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:58.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:35:58 smithi107 ceph-mon[32659]: pgmap v9449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:35:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:35:58 smithi132 ceph-mon[37580]: pgmap v9449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:00 smithi112 ceph-mon[37501]: pgmap v9450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:00 smithi107 ceph-mon[32659]: pgmap v9450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:00 smithi132 ceph-mon[37580]: pgmap v9450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17097.137168 secs 2024-07-26T17:36:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17097.137168 secs 2024-07-26T17:36:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17097.137168 secs 2024-07-26T17:36:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:02 smithi107 ceph-mon[32659]: pgmap v9451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:02 smithi112 ceph-mon[37501]: pgmap v9451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:02.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:02 smithi132 ceph-mon[37580]: pgmap v9451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:04 smithi112 ceph-mon[37501]: pgmap v9452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:04.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:04 smithi107 ceph-mon[32659]: pgmap v9452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:04.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:04 smithi132 ceph-mon[37580]: pgmap v9452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:05.778+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:36:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:05.778+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:36:05.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:05.778+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:36:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:06 smithi112 ceph-mon[37501]: pgmap v9453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17102.137302 secs 2024-07-26T17:36:06.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:06 smithi107 ceph-mon[32659]: pgmap v9453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:06.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17102.137302 secs 2024-07-26T17:36:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:06 smithi132 ceph-mon[37580]: pgmap v9453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17102.137302 secs 2024-07-26T17:36:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:08 smithi112 ceph-mon[37501]: pgmap v9454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:08 smithi107 ceph-mon[32659]: pgmap v9454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:08 smithi132 ceph-mon[37580]: pgmap v9454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:10 smithi112 ceph-mon[37501]: pgmap v9455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:10 smithi107 ceph-mon[32659]: pgmap v9455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:10 smithi132 ceph-mon[37580]: pgmap v9455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17107.137431 secs 2024-07-26T17:36:11.460 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17107.137431 secs 2024-07-26T17:36:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17107.137431 secs 2024-07-26T17:36:11.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:11.693+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T17:36:11.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:11.693+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T17:36:11.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:11.693+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8082040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T17:36:11.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:11.693+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8082040 comp rx=0 tx=0).stop 2024-07-26T17:36:11.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:11.693+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T17:36:11.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:11.693+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T17:36:11.696 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:11.693+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T17:36:11.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:11.693+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T17:36:11.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:11.694+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T17:36:11.697 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:11.695+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f8067040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T17:36:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:12 smithi112 ceph-mon[37501]: pgmap v9456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:36:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:12 smithi107 ceph-mon[32659]: pgmap v9456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:12.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:36:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:12 smithi132 ceph-mon[37580]: pgmap v9456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:36:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:36:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:36:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:13 smithi112 ceph-mon[37501]: pgmap v9457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:36:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:13 smithi112 ceph-mon[37501]: pgmap v9458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:13.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:36:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:36:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:36:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:13 smithi107 ceph-mon[32659]: pgmap v9457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:36:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:13 smithi107 ceph-mon[32659]: pgmap v9458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:13.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:36:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:36:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:36:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:13 smithi132 ceph-mon[37580]: pgmap v9457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:36:13.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:13 smithi132 ceph-mon[37580]: pgmap v9458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:13.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:36:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:15 smithi107 ceph-mon[32659]: pgmap v9459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:36:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:15 smithi132 ceph-mon[37580]: pgmap v9459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:36:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:15.597+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:36:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:15.597+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:36:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:15.597+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:36:15.599 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:15.597+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:36:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:15 smithi112 ceph-mon[37501]: pgmap v9459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:36:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17112.137512 secs 2024-07-26T17:36:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17112.137512 secs 2024-07-26T17:36:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17112.137512 secs 2024-07-26T17:36:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:17 smithi107 ceph-mon[32659]: pgmap v9460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:36:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:17 smithi132 ceph-mon[37580]: pgmap v9460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:36:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:17 smithi112 ceph-mon[37501]: pgmap v9460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:36:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:19 smithi107 ceph-mon[32659]: pgmap v9461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:36:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:19 smithi132 ceph-mon[37580]: pgmap v9461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:36:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:19 smithi112 ceph-mon[37501]: pgmap v9461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:36:20.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:20.787+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:36:20.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:20.787+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:36:20.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:20.787+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:36:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:21 smithi132 ceph-mon[37580]: pgmap v9462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:36:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17117.137577 secs 2024-07-26T17:36:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:21 smithi112 ceph-mon[37501]: pgmap v9462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:36:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17117.137577 secs 2024-07-26T17:36:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:21 smithi107 ceph-mon[32659]: pgmap v9462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:36:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17117.137577 secs 2024-07-26T17:36:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:23 smithi132 ceph-mon[37580]: pgmap v9463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:36:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:23 smithi112 ceph-mon[37501]: pgmap v9463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:36:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:23 smithi107 ceph-mon[32659]: pgmap v9463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:36:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:25 smithi132 ceph-mon[37580]: pgmap v9464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:25 smithi112 ceph-mon[37501]: pgmap v9464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:25 smithi107 ceph-mon[32659]: pgmap v9464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17122.137722 secs 2024-07-26T17:36:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17122.137722 secs 2024-07-26T17:36:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17122.137722 secs 2024-07-26T17:36:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:27 smithi112 ceph-mon[37501]: pgmap v9465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:27 smithi107 ceph-mon[32659]: pgmap v9465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:27 smithi132 ceph-mon[37580]: pgmap v9465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:29 smithi112 ceph-mon[37501]: pgmap v9466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:36:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:29 smithi107 ceph-mon[32659]: pgmap v9466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:36:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:29 smithi132 ceph-mon[37580]: pgmap v9466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:36:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:31 smithi112 ceph-mon[37501]: pgmap v9467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17127.137845 secs 2024-07-26T17:36:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:31 smithi107 ceph-mon[32659]: pgmap v9467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17127.137845 secs 2024-07-26T17:36:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:31 smithi132 ceph-mon[37580]: pgmap v9467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17127.137845 secs 2024-07-26T17:36:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:33 smithi112 ceph-mon[37501]: pgmap v9468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:33 smithi107 ceph-mon[32659]: pgmap v9468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:33 smithi132 ceph-mon[37580]: pgmap v9468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:35.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:35 smithi112 ceph-mon[37501]: pgmap v9469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:35 smithi107 ceph-mon[32659]: pgmap v9469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:35.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:35.803+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:36:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:35.803+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:36:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:35.803+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:36:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:35 smithi132 ceph-mon[37580]: pgmap v9469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17132.137970 secs 2024-07-26T17:36:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17132.137970 secs 2024-07-26T17:36:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17132.137970 secs 2024-07-26T17:36:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:37 smithi112 ceph-mon[37501]: pgmap v9470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:37 smithi107 ceph-mon[32659]: pgmap v9470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:37 smithi132 ceph-mon[37580]: pgmap v9470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:39 smithi112 ceph-mon[37501]: pgmap v9471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:36:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:39 smithi107 ceph-mon[32659]: pgmap v9471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:36:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:39 smithi132 ceph-mon[37580]: pgmap v9471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:36:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:41 smithi112 ceph-mon[37501]: pgmap v9472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17137.138093 secs 2024-07-26T17:36:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:41 smithi107 ceph-mon[32659]: pgmap v9472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17137.138093 secs 2024-07-26T17:36:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:41 smithi132 ceph-mon[37580]: pgmap v9472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17137.138093 secs 2024-07-26T17:36:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:43 smithi112 ceph-mon[37501]: pgmap v9473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:43 smithi107 ceph-mon[32659]: pgmap v9473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:43 smithi132 ceph-mon[37580]: pgmap v9473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:45 smithi112 ceph-mon[37501]: pgmap v9474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:45 smithi107 ceph-mon[32659]: pgmap v9474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:45 smithi132 ceph-mon[37580]: pgmap v9474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:36:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17142.138194 secs 2024-07-26T17:36:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17142.138194 secs 2024-07-26T17:36:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17142.138194 secs 2024-07-26T17:36:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:47 smithi112 ceph-mon[37501]: pgmap v9475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:47 smithi107 ceph-mon[32659]: pgmap v9475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:47 smithi132 ceph-mon[37580]: pgmap v9475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:49 smithi112 ceph-mon[37501]: pgmap v9476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:36:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:49 smithi107 ceph-mon[32659]: pgmap v9476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:36:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:49 smithi132 ceph-mon[37580]: pgmap v9476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:36:50.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:50.809+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:36:50.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:50.809+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:36:50.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:36:50.809+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:36:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:51 smithi112 ceph-mon[37501]: pgmap v9477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17147.138260 secs 2024-07-26T17:36:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:51 smithi107 ceph-mon[32659]: pgmap v9477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17147.138260 secs 2024-07-26T17:36:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:51 smithi132 ceph-mon[37580]: pgmap v9477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:36:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17147.138260 secs 2024-07-26T17:36:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:53 smithi112 ceph-mon[37501]: pgmap v9478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:53 smithi107 ceph-mon[32659]: pgmap v9478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:53 smithi132 ceph-mon[37580]: pgmap v9478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:36:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:55 smithi112 ceph-mon[37501]: pgmap v9479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:36:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:55 smithi107 ceph-mon[32659]: pgmap v9479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:36:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:55 smithi132 ceph-mon[37580]: pgmap v9479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:36:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17152.138395 secs 2024-07-26T17:36:56.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17152.138395 secs 2024-07-26T17:36:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17152.138395 secs 2024-07-26T17:36:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:57 smithi112 ceph-mon[37501]: pgmap v9480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:36:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:57 smithi107 ceph-mon[32659]: pgmap v9480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:36:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:57 smithi132 ceph-mon[37580]: pgmap v9480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:36:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:36:59 smithi112 ceph-mon[37501]: pgmap v9481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:36:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:36:59 smithi107 ceph-mon[32659]: pgmap v9481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:36:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:36:59 smithi132 ceph-mon[37580]: pgmap v9481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:37:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:01 smithi112 ceph-mon[37501]: pgmap v9482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:37:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17157.138529 secs 2024-07-26T17:37:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:01 smithi107 ceph-mon[32659]: pgmap v9482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:37:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17157.138529 secs 2024-07-26T17:37:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:01 smithi132 ceph-mon[37580]: pgmap v9482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:37:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17157.138529 secs 2024-07-26T17:37:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:03 smithi107 ceph-mon[32659]: pgmap v9483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:37:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:03 smithi132 ceph-mon[37580]: pgmap v9483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:37:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:03 smithi112 ceph-mon[37501]: pgmap v9483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:37:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:05 smithi107 ceph-mon[32659]: pgmap v9484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:05 smithi132 ceph-mon[37580]: pgmap v9484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:05.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:05.825+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:37:05.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:05.825+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:37:05.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:05.825+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:37:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:05 smithi112 ceph-mon[37501]: pgmap v9484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17162.138657 secs 2024-07-26T17:37:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17162.138657 secs 2024-07-26T17:37:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17162.138657 secs 2024-07-26T17:37:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:07 smithi107 ceph-mon[32659]: pgmap v9485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:37:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:07 smithi132 ceph-mon[37580]: pgmap v9485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:37:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:07 smithi112 ceph-mon[37501]: pgmap v9485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:37:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:09 smithi107 ceph-mon[32659]: pgmap v9486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:37:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:09 smithi132 ceph-mon[37580]: pgmap v9486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:37:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:09 smithi112 ceph-mon[37501]: pgmap v9486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:37:11.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:11 smithi112 ceph-mon[37501]: pgmap v9487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:37:11.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17167.138776 secs 2024-07-26T17:37:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:11 smithi107 ceph-mon[32659]: pgmap v9487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:37:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17167.138776 secs 2024-07-26T17:37:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:11 smithi132 ceph-mon[37580]: pgmap v9487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:37:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17167.138776 secs 2024-07-26T17:37:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:37:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:37:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:37:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:13 smithi107 ceph-mon[32659]: pgmap v9488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:37:13.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:37:13.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:37:13.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:37:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:13 smithi132 ceph-mon[37580]: pgmap v9488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:37:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:37:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:37:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:37:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:13 smithi112 ceph-mon[37501]: pgmap v9488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:37:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:37:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:37:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:37:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:14 smithi107 ceph-mon[32659]: pgmap v9489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:37:14.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:14 smithi107 ceph-mon[32659]: pgmap v9490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:14 smithi132 ceph-mon[37580]: pgmap v9489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:37:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:14 smithi132 ceph-mon[37580]: pgmap v9490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:14 smithi112 ceph-mon[37501]: pgmap v9489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:37:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:14 smithi112 ceph-mon[37501]: pgmap v9490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:16 smithi107 ceph-mon[32659]: pgmap v9491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17172.138896 secs 2024-07-26T17:37:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:16 smithi132 ceph-mon[37580]: pgmap v9491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17172.138896 secs 2024-07-26T17:37:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:16 smithi112 ceph-mon[37501]: pgmap v9491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17172.138896 secs 2024-07-26T17:37:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:18 smithi107 ceph-mon[32659]: pgmap v9492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:37:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:18 smithi132 ceph-mon[37580]: pgmap v9492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:37:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:18 smithi112 ceph-mon[37501]: pgmap v9492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:37:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:20 smithi107 ceph-mon[32659]: pgmap v9493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:20 smithi132 ceph-mon[37580]: pgmap v9493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:20.842 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:20.840+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:37:20.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:20.840+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:37:20.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:20.840+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:37:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:20 smithi112 ceph-mon[37501]: pgmap v9493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:37:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17177.139021 secs 2024-07-26T17:37:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17177.139021 secs 2024-07-26T17:37:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17177.139021 secs 2024-07-26T17:37:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:22 smithi107 ceph-mon[32659]: pgmap v9494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:37:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:22 smithi132 ceph-mon[37580]: pgmap v9494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:37:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:22 smithi112 ceph-mon[37501]: pgmap v9494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:37:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:24 smithi107 ceph-mon[32659]: pgmap v9495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:37:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:24 smithi132 ceph-mon[37580]: pgmap v9495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:37:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:24 smithi112 ceph-mon[37501]: pgmap v9495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:37:25.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:25 smithi112 ceph-mon[37501]: pgmap v9496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:25 smithi107 ceph-mon[32659]: pgmap v9496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:25 smithi132 ceph-mon[37580]: pgmap v9496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17182.139143 secs 2024-07-26T17:37:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17182.139143 secs 2024-07-26T17:37:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17182.139143 secs 2024-07-26T17:37:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:27 smithi112 ceph-mon[37501]: pgmap v9497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:27 smithi107 ceph-mon[32659]: pgmap v9497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:27 smithi132 ceph-mon[37580]: pgmap v9497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:30 smithi107 ceph-mon[32659]: pgmap v9498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:37:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:30 smithi132 ceph-mon[37580]: pgmap v9498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:37:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:30 smithi112 ceph-mon[37501]: pgmap v9498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:37:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17187.139262 secs 2024-07-26T17:37:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17187.139262 secs 2024-07-26T17:37:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17187.139262 secs 2024-07-26T17:37:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:32 smithi107 ceph-mon[32659]: pgmap v9499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:37:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:32 smithi132 ceph-mon[37580]: pgmap v9499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:37:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:32 smithi112 ceph-mon[37501]: pgmap v9499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:37:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:34 smithi107 ceph-mon[32659]: pgmap v9500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:37:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:34 smithi132 ceph-mon[37580]: pgmap v9500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:37:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:34 smithi112 ceph-mon[37501]: pgmap v9500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:37:35.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:35.850+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:37:35.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:35.850+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:37:35.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:35.850+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:37:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:36 smithi107 ceph-mon[32659]: pgmap v9501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17192.139345 secs 2024-07-26T17:37:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:36 smithi132 ceph-mon[37580]: pgmap v9501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17192.139345 secs 2024-07-26T17:37:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:36 smithi112 ceph-mon[37501]: pgmap v9501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17192.139345 secs 2024-07-26T17:37:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:38 smithi107 ceph-mon[32659]: pgmap v9502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:38 smithi132 ceph-mon[37580]: pgmap v9502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:38 smithi112 ceph-mon[37501]: pgmap v9502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:40 smithi107 ceph-mon[32659]: pgmap v9503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:37:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:40 smithi132 ceph-mon[37580]: pgmap v9503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:37:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:40 smithi112 ceph-mon[37501]: pgmap v9503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:37:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17197.139445 secs 2024-07-26T17:37:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17197.139445 secs 2024-07-26T17:37:41.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17197.139445 secs 2024-07-26T17:37:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:42 smithi107 ceph-mon[32659]: pgmap v9504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:37:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:42 smithi132 ceph-mon[37580]: pgmap v9504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:37:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:42 smithi112 ceph-mon[37501]: pgmap v9504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:37:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:44 smithi107 ceph-mon[32659]: pgmap v9505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:37:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:44 smithi132 ceph-mon[37580]: pgmap v9505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:37:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:44 smithi112 ceph-mon[37501]: pgmap v9505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:37:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:45 smithi132 ceph-mon[37580]: pgmap v9506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:45 smithi112 ceph-mon[37501]: pgmap v9506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:45 smithi107 ceph-mon[32659]: pgmap v9506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17202.139542 secs 2024-07-26T17:37:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17202.139542 secs 2024-07-26T17:37:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17202.139542 secs 2024-07-26T17:37:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:47 smithi112 ceph-mon[37501]: pgmap v9507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:47 smithi107 ceph-mon[32659]: pgmap v9507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:47 smithi132 ceph-mon[37580]: pgmap v9507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:50 smithi107 ceph-mon[32659]: pgmap v9508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:37:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:50 smithi132 ceph-mon[37580]: pgmap v9508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:37:50.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:50.866+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:37:50.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:50.866+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:37:50.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:37:50.866+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:37:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:50 smithi112 ceph-mon[37501]: pgmap v9508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:37:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17207.139676 secs 2024-07-26T17:37:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17207.139676 secs 2024-07-26T17:37:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17207.139676 secs 2024-07-26T17:37:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:52 smithi107 ceph-mon[32659]: pgmap v9509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:37:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:52 smithi132 ceph-mon[37580]: pgmap v9509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:37:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:52 smithi112 ceph-mon[37501]: pgmap v9509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:37:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:54 smithi107 ceph-mon[32659]: pgmap v9510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:37:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:54 smithi132 ceph-mon[37580]: pgmap v9510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:37:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:54 smithi112 ceph-mon[37501]: pgmap v9510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:37:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:56 smithi107 ceph-mon[32659]: pgmap v9511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17212.139797 secs 2024-07-26T17:37:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:56 smithi132 ceph-mon[37580]: pgmap v9511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17212.139797 secs 2024-07-26T17:37:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:56 smithi112 ceph-mon[37501]: pgmap v9511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:37:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17212.139797 secs 2024-07-26T17:37:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:37:58 smithi107 ceph-mon[32659]: pgmap v9512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:37:58 smithi132 ceph-mon[37580]: pgmap v9512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:37:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:37:58 smithi112 ceph-mon[37501]: pgmap v9512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:00 smithi107 ceph-mon[32659]: pgmap v9513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:00 smithi132 ceph-mon[37580]: pgmap v9513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:00 smithi112 ceph-mon[37501]: pgmap v9513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17217.139919 secs 2024-07-26T17:38:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17217.139919 secs 2024-07-26T17:38:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17217.139919 secs 2024-07-26T17:38:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:02 smithi132 ceph-mon[37580]: pgmap v9514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:38:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:02 smithi112 ceph-mon[37501]: pgmap v9514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:38:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:02 smithi107 ceph-mon[32659]: pgmap v9514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:38:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:03 smithi132 ceph-mon[37580]: pgmap v9515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:38:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:03 smithi112 ceph-mon[37501]: pgmap v9515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:38:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:03 smithi107 ceph-mon[32659]: pgmap v9515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:38:05.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:05.880+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:38:05.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:05.880+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:38:05.882 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:05.880+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:38:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:06 smithi107 ceph-mon[32659]: pgmap v9516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17222.140038 secs 2024-07-26T17:38:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:06 smithi132 ceph-mon[37580]: pgmap v9516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17222.140038 secs 2024-07-26T17:38:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:06 smithi112 ceph-mon[37501]: pgmap v9516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17222.140038 secs 2024-07-26T17:38:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:08 smithi107 ceph-mon[32659]: pgmap v9517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:08 smithi132 ceph-mon[37580]: pgmap v9517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:08 smithi112 ceph-mon[37501]: pgmap v9517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:10 smithi107 ceph-mon[32659]: pgmap v9518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:10 smithi132 ceph-mon[37580]: pgmap v9518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:10 smithi112 ceph-mon[37501]: pgmap v9518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17227.140139 secs 2024-07-26T17:38:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17227.140139 secs 2024-07-26T17:38:11.857 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17227.140139 secs 2024-07-26T17:38:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:12 smithi107 ceph-mon[32659]: pgmap v9519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:38:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:12 smithi132 ceph-mon[37580]: pgmap v9519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:38:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:12 smithi112 ceph-mon[37501]: pgmap v9519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:38:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:14 smithi107 ceph-mon[32659]: pgmap v9520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:38:14.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:38:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:14 smithi132 ceph-mon[37580]: pgmap v9520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:38:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:38:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:14 smithi112 ceph-mon[37501]: pgmap v9520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:38:14.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:38:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:38:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:38:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:38:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:38:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:38:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:38:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:38:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:38:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:38:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:16 smithi132 ceph-mon[37580]: pgmap v9521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:38:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:16 smithi132 ceph-mon[37580]: pgmap v9522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:38:16.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17232.140274 secs 2024-07-26T17:38:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:16 smithi112 ceph-mon[37501]: pgmap v9521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:38:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:16 smithi112 ceph-mon[37501]: pgmap v9522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:38:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17232.140274 secs 2024-07-26T17:38:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:16 smithi107 ceph-mon[32659]: pgmap v9521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:38:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:16 smithi107 ceph-mon[32659]: pgmap v9522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:38:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17232.140274 secs 2024-07-26T17:38:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:17 smithi112 ceph-mon[37501]: pgmap v9523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:17 smithi107 ceph-mon[32659]: pgmap v9523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:17 smithi132 ceph-mon[37580]: pgmap v9523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:19 smithi112 ceph-mon[37501]: pgmap v9524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:38:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:19 smithi107 ceph-mon[32659]: pgmap v9524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:38:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:19 smithi132 ceph-mon[37580]: pgmap v9524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:38:20.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:20.892+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:38:20.894 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:20.892+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:38:20.895 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:20.892+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:38:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17237.140362 secs 2024-07-26T17:38:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:21 smithi112 ceph-mon[37501]: pgmap v9525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:38:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17237.140362 secs 2024-07-26T17:38:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:21 smithi107 ceph-mon[32659]: pgmap v9525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:38:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17237.140362 secs 2024-07-26T17:38:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:21 smithi132 ceph-mon[37580]: pgmap v9525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:38:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:23 smithi112 ceph-mon[37501]: pgmap v9526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:38:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:23 smithi107 ceph-mon[32659]: pgmap v9526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:38:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:23 smithi132 ceph-mon[37580]: pgmap v9526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:38:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:25 smithi107 ceph-mon[32659]: pgmap v9527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:38:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17242.140485 secs 2024-07-26T17:38:26.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:25 smithi132 ceph-mon[37580]: pgmap v9527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:38:26.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17242.140485 secs 2024-07-26T17:38:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:25 smithi112 ceph-mon[37501]: pgmap v9527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:38:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17242.140485 secs 2024-07-26T17:38:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:27 smithi107 ceph-mon[32659]: pgmap v9528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:27 smithi132 ceph-mon[37580]: pgmap v9528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:27 smithi112 ceph-mon[37501]: pgmap v9528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:29 smithi107 ceph-mon[32659]: pgmap v9529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:29 smithi132 ceph-mon[37580]: pgmap v9529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:29 smithi112 ceph-mon[37501]: pgmap v9529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17247.140579 secs 2024-07-26T17:38:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17247.140579 secs 2024-07-26T17:38:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17247.140579 secs 2024-07-26T17:38:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:31 smithi107 ceph-mon[32659]: pgmap v9530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:31 smithi132 ceph-mon[37580]: pgmap v9530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:31 smithi112 ceph-mon[37501]: pgmap v9530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:33 smithi107 ceph-mon[32659]: pgmap v9531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:33 smithi132 ceph-mon[37580]: pgmap v9531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:33 smithi112 ceph-mon[37501]: pgmap v9531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:35.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:35.907+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:38:35.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:35.907+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:38:35.910 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:35.907+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:38:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:35 smithi107 ceph-mon[32659]: pgmap v9532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:36.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17252.140706 secs 2024-07-26T17:38:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:35 smithi132 ceph-mon[37580]: pgmap v9532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17252.140706 secs 2024-07-26T17:38:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:35 smithi112 ceph-mon[37501]: pgmap v9532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17252.140706 secs 2024-07-26T17:38:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:37 smithi107 ceph-mon[32659]: pgmap v9533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:37 smithi132 ceph-mon[37580]: pgmap v9533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:37 smithi112 ceph-mon[37501]: pgmap v9533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:39 smithi107 ceph-mon[32659]: pgmap v9534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:39 smithi132 ceph-mon[37580]: pgmap v9534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:39 smithi112 ceph-mon[37501]: pgmap v9534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17257.140827 secs 2024-07-26T17:38:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17257.140827 secs 2024-07-26T17:38:41.101 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17257.140827 secs 2024-07-26T17:38:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:41 smithi107 ceph-mon[32659]: pgmap v9535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:41 smithi132 ceph-mon[37580]: pgmap v9535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:41 smithi112 ceph-mon[37501]: pgmap v9535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:43 smithi107 ceph-mon[32659]: pgmap v9536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:43 smithi132 ceph-mon[37580]: pgmap v9536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:43 smithi112 ceph-mon[37501]: pgmap v9536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:45 smithi107 ceph-mon[32659]: pgmap v9537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:46.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17262.140948 secs 2024-07-26T17:38:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:45 smithi132 ceph-mon[37580]: pgmap v9537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17262.140948 secs 2024-07-26T17:38:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:45 smithi112 ceph-mon[37501]: pgmap v9537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17262.140948 secs 2024-07-26T17:38:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:47 smithi107 ceph-mon[32659]: pgmap v9538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:47 smithi132 ceph-mon[37580]: pgmap v9538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:47 smithi112 ceph-mon[37501]: pgmap v9538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:49 smithi107 ceph-mon[32659]: pgmap v9539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:49 smithi132 ceph-mon[37580]: pgmap v9539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:49 smithi112 ceph-mon[37501]: pgmap v9539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:50.925 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:50.924+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:38:50.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:50.924+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:38:50.926 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:38:50.924+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:38:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17267.141055 secs 2024-07-26T17:38:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17267.141055 secs 2024-07-26T17:38:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17267.141055 secs 2024-07-26T17:38:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:51 smithi132 ceph-mon[37580]: pgmap v9540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:51 smithi112 ceph-mon[37501]: pgmap v9540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:51 smithi107 ceph-mon[32659]: pgmap v9540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:38:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:53 smithi132 ceph-mon[37580]: pgmap v9541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:53 smithi112 ceph-mon[37501]: pgmap v9541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:53 smithi107 ceph-mon[32659]: pgmap v9541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:38:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:55 smithi112 ceph-mon[37501]: pgmap v9542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17272.141176 secs 2024-07-26T17:38:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:55 smithi107 ceph-mon[32659]: pgmap v9542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17272.141176 secs 2024-07-26T17:38:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:55 smithi132 ceph-mon[37580]: pgmap v9542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:38:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17272.141176 secs 2024-07-26T17:38:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:57 smithi112 ceph-mon[37501]: pgmap v9543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:57 smithi107 ceph-mon[32659]: pgmap v9543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:38:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:57 smithi132 ceph-mon[37580]: pgmap v9543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:38:59 smithi112 ceph-mon[37501]: pgmap v9544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:38:59 smithi107 ceph-mon[32659]: pgmap v9544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:38:59 smithi132 ceph-mon[37580]: pgmap v9544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17277.141297 secs 2024-07-26T17:39:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17277.141297 secs 2024-07-26T17:39:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17277.141297 secs 2024-07-26T17:39:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:01 smithi112 ceph-mon[37501]: pgmap v9545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:01 smithi107 ceph-mon[32659]: pgmap v9545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:01 smithi132 ceph-mon[37580]: pgmap v9545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:03 smithi112 ceph-mon[37501]: pgmap v9546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:03 smithi107 ceph-mon[32659]: pgmap v9546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:03 smithi132 ceph-mon[37580]: pgmap v9546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:05.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:05.929+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:39:05.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:05.929+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:39:05.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:05.929+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:39:06.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:05 smithi107 ceph-mon[32659]: pgmap v9547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:39:06.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17282.141382 secs 2024-07-26T17:39:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:05 smithi112 ceph-mon[37501]: pgmap v9547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:39:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17282.141382 secs 2024-07-26T17:39:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:05 smithi132 ceph-mon[37580]: pgmap v9547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:39:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17282.141382 secs 2024-07-26T17:39:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:07 smithi112 ceph-mon[37501]: pgmap v9548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:07 smithi107 ceph-mon[32659]: pgmap v9548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:07 smithi132 ceph-mon[37580]: pgmap v9548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:09 smithi112 ceph-mon[37501]: pgmap v9549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:09 smithi107 ceph-mon[32659]: pgmap v9549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:09 smithi132 ceph-mon[37580]: pgmap v9549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17287.141528 secs 2024-07-26T17:39:11.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17287.141528 secs 2024-07-26T17:39:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17287.141528 secs 2024-07-26T17:39:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:11 smithi112 ceph-mon[37501]: pgmap v9550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:11 smithi107 ceph-mon[32659]: pgmap v9550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:11 smithi132 ceph-mon[37580]: pgmap v9550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:13 smithi112 ceph-mon[37501]: pgmap v9551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:13 smithi107 ceph-mon[32659]: pgmap v9551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:13 smithi132 ceph-mon[37580]: pgmap v9551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:14 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:39:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:14 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:39:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:14 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:39:15.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:15.628+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:39:15.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:15.628+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:39:15.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:15.629+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:39:15.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:15.629+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:39:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:15 smithi112 ceph-mon[37501]: pgmap v9552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:39:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17292.141627 secs 2024-07-26T17:39:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:39:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:39:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:39:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:15 smithi107 ceph-mon[32659]: pgmap v9552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:39:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17292.141627 secs 2024-07-26T17:39:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:39:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:39:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:39:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:15 smithi132 ceph-mon[37580]: pgmap v9552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:39:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17292.141627 secs 2024-07-26T17:39:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:39:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:39:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:39:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:16 smithi112 ceph-mon[37501]: pgmap v9553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:39:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:16 smithi112 ceph-mon[37501]: pgmap v9554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:39:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:16 smithi107 ceph-mon[32659]: pgmap v9553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:39:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:16 smithi107 ceph-mon[32659]: pgmap v9554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:39:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:16 smithi132 ceph-mon[37580]: pgmap v9553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:39:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:16 smithi132 ceph-mon[37580]: pgmap v9554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:39:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:18 smithi112 ceph-mon[37501]: pgmap v9555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:39:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:18 smithi107 ceph-mon[32659]: pgmap v9555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:39:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:18 smithi132 ceph-mon[37580]: pgmap v9555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:39:20.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:20.931+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:39:20.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:20.931+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:39:20.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:20.931+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:39:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:20 smithi112 ceph-mon[37501]: pgmap v9556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:39:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17297.141746 secs 2024-07-26T17:39:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:20 smithi107 ceph-mon[32659]: pgmap v9556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:39:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17297.141746 secs 2024-07-26T17:39:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:20 smithi132 ceph-mon[37580]: pgmap v9556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:39:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17297.141746 secs 2024-07-26T17:39:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:22 smithi112 ceph-mon[37501]: pgmap v9557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:23.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:22 smithi107 ceph-mon[32659]: pgmap v9557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:22 smithi132 ceph-mon[37580]: pgmap v9557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:24 smithi112 ceph-mon[37501]: pgmap v9558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:39:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:24 smithi107 ceph-mon[32659]: pgmap v9558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:39:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:24 smithi132 ceph-mon[37580]: pgmap v9558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:39:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17302.141871 secs 2024-07-26T17:39:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17302.141871 secs 2024-07-26T17:39:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17302.141871 secs 2024-07-26T17:39:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:26 smithi112 ceph-mon[37501]: pgmap v9559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:39:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:26 smithi107 ceph-mon[32659]: pgmap v9559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:39:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:26 smithi132 ceph-mon[37580]: pgmap v9559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:39:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:28 smithi112 ceph-mon[37501]: pgmap v9560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:28 smithi107 ceph-mon[32659]: pgmap v9560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:28 smithi132 ceph-mon[37580]: pgmap v9560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:30 smithi112 ceph-mon[37501]: pgmap v9561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:31.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17307.142017 secs 2024-07-26T17:39:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:30 smithi107 ceph-mon[32659]: pgmap v9561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:31.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17307.142017 secs 2024-07-26T17:39:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:30 smithi132 ceph-mon[37580]: pgmap v9561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17307.142017 secs 2024-07-26T17:39:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:32 smithi107 ceph-mon[32659]: pgmap v9562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:32 smithi132 ceph-mon[37580]: pgmap v9562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:32 smithi112 ceph-mon[37501]: pgmap v9562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:34 smithi107 ceph-mon[32659]: pgmap v9563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:34 smithi132 ceph-mon[37580]: pgmap v9563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:34 smithi112 ceph-mon[37501]: pgmap v9563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:35.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:35.933+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:39:35.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:35.933+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:39:35.936 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:35.933+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:39:36.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17312.142157 secs 2024-07-26T17:39:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17312.142157 secs 2024-07-26T17:39:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17312.142157 secs 2024-07-26T17:39:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:36 smithi107 ceph-mon[32659]: pgmap v9564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:36 smithi132 ceph-mon[37580]: pgmap v9564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:36 smithi112 ceph-mon[37501]: pgmap v9564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:38 smithi107 ceph-mon[32659]: pgmap v9565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:38 smithi132 ceph-mon[37580]: pgmap v9565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:38 smithi112 ceph-mon[37501]: pgmap v9565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:41.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:40 smithi112 ceph-mon[37501]: pgmap v9566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:41.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17317.142273 secs 2024-07-26T17:39:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:40 smithi107 ceph-mon[32659]: pgmap v9566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:41.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17317.142273 secs 2024-07-26T17:39:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:40 smithi132 ceph-mon[37580]: pgmap v9566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17317.142273 secs 2024-07-26T17:39:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:42 smithi107 ceph-mon[32659]: pgmap v9567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:42 smithi132 ceph-mon[37580]: pgmap v9567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:43.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:42 smithi112 ceph-mon[37501]: pgmap v9567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:44 smithi107 ceph-mon[32659]: pgmap v9568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:44 smithi132 ceph-mon[37580]: pgmap v9568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:44 smithi112 ceph-mon[37501]: pgmap v9568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17322.142362 secs 2024-07-26T17:39:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17322.142362 secs 2024-07-26T17:39:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17322.142362 secs 2024-07-26T17:39:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:47 smithi107 ceph-mon[32659]: pgmap v9569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:47 smithi132 ceph-mon[37580]: pgmap v9569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:47 smithi112 ceph-mon[37501]: pgmap v9569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:49 smithi107 ceph-mon[32659]: pgmap v9570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:49 smithi132 ceph-mon[37580]: pgmap v9570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:49 smithi112 ceph-mon[37501]: pgmap v9570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:50.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:50.947+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:39:50.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:50.947+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:39:50.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:50.947+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:39:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:51 smithi107 ceph-mon[32659]: pgmap v9571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:51.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17327.142457 secs 2024-07-26T17:39:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:51 smithi132 ceph-mon[37580]: pgmap v9571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17327.142457 secs 2024-07-26T17:39:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:51 smithi112 ceph-mon[37501]: pgmap v9571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17327.142457 secs 2024-07-26T17:39:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:53 smithi107 ceph-mon[32659]: pgmap v9572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:53 smithi132 ceph-mon[37580]: pgmap v9572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:53 smithi112 ceph-mon[37501]: pgmap v9572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:39:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:55 smithi107 ceph-mon[32659]: pgmap v9573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:55 smithi132 ceph-mon[37580]: pgmap v9573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:55 smithi112 ceph-mon[37501]: pgmap v9573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17332.142603 secs 2024-07-26T17:39:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17332.142603 secs 2024-07-26T17:39:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17332.142603 secs 2024-07-26T17:39:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:57 smithi132 ceph-mon[37580]: pgmap v9574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:57 smithi112 ceph-mon[37501]: pgmap v9574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:57 smithi107 ceph-mon[32659]: pgmap v9574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:39:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:39:59 smithi132 ceph-mon[37580]: pgmap v9575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:39:59 smithi112 ceph-mon[37501]: pgmap v9575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:39:59 smithi107 ceph-mon[32659]: pgmap v9575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:39:59.638 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:59.637+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:39:59.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:59.637+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:39:59.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:59.638+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:39:59.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:39:59.638+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:40:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:40:00.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:40:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:40:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:01 smithi112 ceph-mon[37501]: pgmap v9576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17337.142700 secs 2024-07-26T17:40:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:01 smithi107 ceph-mon[32659]: pgmap v9576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17337.142700 secs 2024-07-26T17:40:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:01 smithi132 ceph-mon[37580]: pgmap v9576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17337.142700 secs 2024-07-26T17:40:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:03 smithi112 ceph-mon[37501]: pgmap v9577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:03 smithi107 ceph-mon[32659]: pgmap v9577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:03 smithi132 ceph-mon[37580]: pgmap v9577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:05.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:05 smithi112 ceph-mon[37501]: pgmap v9578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:05 smithi107 ceph-mon[32659]: pgmap v9578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:05 smithi132 ceph-mon[37580]: pgmap v9578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:05.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:05.963+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:40:05.964 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:05.963+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:40:05.965 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:05.963+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:40:06.218 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17342.142842 secs 2024-07-26T17:40:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17342.142842 secs 2024-07-26T17:40:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17342.142842 secs 2024-07-26T17:40:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:07 smithi112 ceph-mon[37501]: pgmap v9579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:07.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:07 smithi107 ceph-mon[32659]: pgmap v9579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:07.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:07 smithi132 ceph-mon[37580]: pgmap v9579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:09 smithi112 ceph-mon[37501]: pgmap v9580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:09 smithi107 ceph-mon[32659]: pgmap v9580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:09 smithi132 ceph-mon[37580]: pgmap v9580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:11 smithi112 ceph-mon[37501]: pgmap v9581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17347.142987 secs 2024-07-26T17:40:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:11 smithi107 ceph-mon[32659]: pgmap v9581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17347.142987 secs 2024-07-26T17:40:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:11 smithi132 ceph-mon[37580]: pgmap v9581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17347.142987 secs 2024-07-26T17:40:13.415 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:13 smithi132 ceph-mon[37580]: pgmap v9582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:13 smithi112 ceph-mon[37501]: pgmap v9582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:13 smithi107 ceph-mon[32659]: pgmap v9582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:15 smithi112 ceph-mon[37501]: pgmap v9583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:15 smithi107 ceph-mon[32659]: pgmap v9583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:15 smithi132 ceph-mon[37580]: pgmap v9583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17352.143115 secs 2024-07-26T17:40:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:40:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17352.143115 secs 2024-07-26T17:40:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:40:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17352.143115 secs 2024-07-26T17:40:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:40:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: pgmap v9584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:40:17.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:40:17.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:40:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: pgmap v9584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:40:17.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:40:17.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:40:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: pgmap v9584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:40:17.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:40:17.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:40:17.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:40:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:18 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:40:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:18 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:40:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:18 smithi112 ceph-mon[37501]: pgmap v9585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:40:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:18 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:40:18.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:18 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:40:18.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:18 smithi107 ceph-mon[32659]: pgmap v9585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:40:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:18 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:40:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:18 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:40:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:18 smithi132 ceph-mon[37580]: pgmap v9585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:40:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:20 smithi112 ceph-mon[37501]: pgmap v9586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:40:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:20 smithi107 ceph-mon[32659]: pgmap v9586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:40:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:20 smithi132 ceph-mon[37580]: pgmap v9586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:40:20.971 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:20.968+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:40:20.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:20.968+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:40:20.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:20.968+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:40:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17357.143273 secs 2024-07-26T17:40:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17357.143273 secs 2024-07-26T17:40:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17357.143273 secs 2024-07-26T17:40:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:22 smithi112 ceph-mon[37501]: pgmap v9587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:40:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:22 smithi107 ceph-mon[32659]: pgmap v9587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:40:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:22 smithi132 ceph-mon[37580]: pgmap v9587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:40:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:24 smithi112 ceph-mon[37501]: pgmap v9588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:40:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:24 smithi107 ceph-mon[32659]: pgmap v9588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:40:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:24 smithi132 ceph-mon[37580]: pgmap v9588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:40:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:26 smithi112 ceph-mon[37501]: pgmap v9589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:40:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17362.143389 secs 2024-07-26T17:40:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:26 smithi107 ceph-mon[32659]: pgmap v9589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:40:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17362.143389 secs 2024-07-26T17:40:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:26 smithi132 ceph-mon[37580]: pgmap v9589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:40:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17362.143389 secs 2024-07-26T17:40:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:28 smithi112 ceph-mon[37501]: pgmap v9590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:40:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:28 smithi107 ceph-mon[32659]: pgmap v9590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:40:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:28 smithi132 ceph-mon[37580]: pgmap v9590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:40:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:30 smithi112 ceph-mon[37501]: pgmap v9591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:30 smithi107 ceph-mon[32659]: pgmap v9591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:30 smithi132 ceph-mon[37580]: pgmap v9591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17367.143526 secs 2024-07-26T17:40:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17367.143526 secs 2024-07-26T17:40:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17367.143526 secs 2024-07-26T17:40:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:32 smithi107 ceph-mon[32659]: pgmap v9592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:32 smithi132 ceph-mon[37580]: pgmap v9592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:32 smithi112 ceph-mon[37501]: pgmap v9592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:34 smithi107 ceph-mon[32659]: pgmap v9593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:34 smithi132 ceph-mon[37580]: pgmap v9593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:34.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:34 smithi112 ceph-mon[37501]: pgmap v9593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:35.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:35.984+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:40:35.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:35.984+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:40:35.988 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:35.984+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:40:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:36 smithi107 ceph-mon[32659]: pgmap v9594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17372.143641 secs 2024-07-26T17:40:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:36 smithi132 ceph-mon[37580]: pgmap v9594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17372.143641 secs 2024-07-26T17:40:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:36 smithi112 ceph-mon[37501]: pgmap v9594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17372.143641 secs 2024-07-26T17:40:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:38 smithi107 ceph-mon[32659]: pgmap v9595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:38 smithi132 ceph-mon[37580]: pgmap v9595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:38 smithi112 ceph-mon[37501]: pgmap v9595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:40 smithi107 ceph-mon[32659]: pgmap v9596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:40 smithi132 ceph-mon[37580]: pgmap v9596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:40 smithi112 ceph-mon[37501]: pgmap v9596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17377.143788 secs 2024-07-26T17:40:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17377.143788 secs 2024-07-26T17:40:41.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17377.143788 secs 2024-07-26T17:40:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:42 smithi107 ceph-mon[32659]: pgmap v9597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:42 smithi132 ceph-mon[37580]: pgmap v9597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:42 smithi112 ceph-mon[37501]: pgmap v9597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:44 smithi107 ceph-mon[32659]: pgmap v9598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:44 smithi132 ceph-mon[37580]: pgmap v9598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:44 smithi112 ceph-mon[37501]: pgmap v9598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:46 smithi107 ceph-mon[32659]: pgmap v9599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17382.143884 secs 2024-07-26T17:40:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:46 smithi132 ceph-mon[37580]: pgmap v9599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17382.143884 secs 2024-07-26T17:40:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:46 smithi112 ceph-mon[37501]: pgmap v9599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17382.143884 secs 2024-07-26T17:40:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:48 smithi107 ceph-mon[32659]: pgmap v9600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:48 smithi132 ceph-mon[37580]: pgmap v9600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:48.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:48 smithi112 ceph-mon[37501]: pgmap v9600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:50 smithi107 ceph-mon[32659]: pgmap v9601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:50 smithi132 ceph-mon[37580]: pgmap v9601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:50 smithi112 ceph-mon[37501]: pgmap v9601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:51.000+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:40:51.002 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:51.000+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:40:51.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:40:51.000+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:40:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17387.143984 secs 2024-07-26T17:40:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17387.143984 secs 2024-07-26T17:40:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17387.143984 secs 2024-07-26T17:40:52.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:52 smithi107 ceph-mon[32659]: pgmap v9602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:52 smithi132 ceph-mon[37580]: pgmap v9602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:52 smithi112 ceph-mon[37501]: pgmap v9602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:40:54.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:54 smithi107 ceph-mon[32659]: pgmap v9603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:54 smithi132 ceph-mon[37580]: pgmap v9603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:54 smithi112 ceph-mon[37501]: pgmap v9603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:40:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:56 smithi132 ceph-mon[37580]: pgmap v9604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17392.144138 secs 2024-07-26T17:40:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:56 smithi112 ceph-mon[37501]: pgmap v9604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17392.144138 secs 2024-07-26T17:40:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:56 smithi107 ceph-mon[32659]: pgmap v9604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17392.144138 secs 2024-07-26T17:40:58.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:40:58 smithi132 ceph-mon[37580]: pgmap v9605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:58.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:40:58 smithi112 ceph-mon[37501]: pgmap v9605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:40:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:40:58 smithi107 ceph-mon[32659]: pgmap v9605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:00.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:00 smithi132 ceph-mon[37580]: pgmap v9606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:00 smithi112 ceph-mon[37501]: pgmap v9606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:00 smithi107 ceph-mon[32659]: pgmap v9606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17397.144262 secs 2024-07-26T17:41:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17397.144262 secs 2024-07-26T17:41:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17397.144262 secs 2024-07-26T17:41:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:02 smithi112 ceph-mon[37501]: pgmap v9607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:02 smithi107 ceph-mon[32659]: pgmap v9607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:02 smithi132 ceph-mon[37580]: pgmap v9607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:04 smithi112 ceph-mon[37501]: pgmap v9608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:04 smithi107 ceph-mon[32659]: pgmap v9608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:04 smithi132 ceph-mon[37580]: pgmap v9608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:06.016+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:41:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:06.016+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:41:06.018 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:06.016+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:41:06.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:05 smithi107 ceph-mon[32659]: pgmap v9609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:06.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17402.144337 secs 2024-07-26T17:41:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:05 smithi132 ceph-mon[37580]: pgmap v9609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17402.144337 secs 2024-07-26T17:41:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:05 smithi112 ceph-mon[37501]: pgmap v9609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17402.144337 secs 2024-07-26T17:41:08.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:08 smithi112 ceph-mon[37501]: pgmap v9610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:08 smithi107 ceph-mon[32659]: pgmap v9610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:08 smithi132 ceph-mon[37580]: pgmap v9610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:10 smithi112 ceph-mon[37501]: pgmap v9611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:10 smithi107 ceph-mon[32659]: pgmap v9611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:10 smithi132 ceph-mon[37580]: pgmap v9611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17407.144451 secs 2024-07-26T17:41:11.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17407.144451 secs 2024-07-26T17:41:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17407.144451 secs 2024-07-26T17:41:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:12 smithi112 ceph-mon[37501]: pgmap v9612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:12 smithi107 ceph-mon[32659]: pgmap v9612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:12 smithi132 ceph-mon[37580]: pgmap v9612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:14 smithi112 ceph-mon[37501]: pgmap v9613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:14 smithi107 ceph-mon[32659]: pgmap v9613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:14 smithi132 ceph-mon[37580]: pgmap v9613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:16 smithi112 ceph-mon[37501]: pgmap v9614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17412.144566 secs 2024-07-26T17:41:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:16 smithi107 ceph-mon[32659]: pgmap v9614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17412.144566 secs 2024-07-26T17:41:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:16 smithi132 ceph-mon[37580]: pgmap v9614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17412.144566 secs 2024-07-26T17:41:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: pgmap v9615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:41:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: pgmap v9615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:18.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:41:18.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: pgmap v9615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:41:18.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:18.563 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:41:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:19 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:41:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:19 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:41:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:19 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:41:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:19 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:41:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:41:19.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:41:19.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:19 smithi112 ceph-mon[37501]: pgmap v9616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:41:19.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:19 smithi112 ceph-mon[37501]: pgmap v9617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:41:19.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:41:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:19 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:41:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:19 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:41:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:19 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:41:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:19 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:41:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:41:19.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:41:19.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:19 smithi107 ceph-mon[32659]: pgmap v9616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:41:19.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:19 smithi107 ceph-mon[32659]: pgmap v9617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:41:19.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:41:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:19 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:41:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:19 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:41:19.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:19 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:41:19.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:19 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:41:19.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:41:19.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:41:19.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:19 smithi132 ceph-mon[37580]: pgmap v9616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:41:19.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:19 smithi132 ceph-mon[37580]: pgmap v9617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:41:19.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:41:21.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:21.026+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:41:21.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:21.026+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:41:21.029 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:21.026+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:41:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:20 smithi107 ceph-mon[32659]: pgmap v9618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:41:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17417.144710 secs 2024-07-26T17:41:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:20 smithi132 ceph-mon[37580]: pgmap v9618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:41:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17417.144710 secs 2024-07-26T17:41:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:20 smithi112 ceph-mon[37501]: pgmap v9618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:41:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17417.144710 secs 2024-07-26T17:41:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:23 smithi107 ceph-mon[32659]: pgmap v9619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:23 smithi132 ceph-mon[37580]: pgmap v9619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:23 smithi112 ceph-mon[37501]: pgmap v9619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:25 smithi107 ceph-mon[32659]: pgmap v9620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:41:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:25 smithi132 ceph-mon[37580]: pgmap v9620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:41:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:25 smithi112 ceph-mon[37501]: pgmap v9620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:41:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17422.144809 secs 2024-07-26T17:41:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17422.144809 secs 2024-07-26T17:41:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17422.144809 secs 2024-07-26T17:41:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:27 smithi132 ceph-mon[37580]: pgmap v9621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:41:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:27 smithi112 ceph-mon[37501]: pgmap v9621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:41:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:27 smithi107 ceph-mon[32659]: pgmap v9621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:41:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:29 smithi132 ceph-mon[37580]: pgmap v9622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:41:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:29 smithi112 ceph-mon[37501]: pgmap v9622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:41:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:29 smithi107 ceph-mon[32659]: pgmap v9622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:41:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:31 smithi132 ceph-mon[37580]: pgmap v9623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17427.144950 secs 2024-07-26T17:41:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:31 smithi112 ceph-mon[37501]: pgmap v9623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17427.144950 secs 2024-07-26T17:41:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:31 smithi107 ceph-mon[32659]: pgmap v9623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17427.144950 secs 2024-07-26T17:41:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:33 smithi132 ceph-mon[37580]: pgmap v9624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:33 smithi112 ceph-mon[37501]: pgmap v9624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:33 smithi107 ceph-mon[32659]: pgmap v9624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:35 smithi112 ceph-mon[37501]: pgmap v9625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:35 smithi107 ceph-mon[32659]: pgmap v9625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:35 smithi132 ceph-mon[37580]: pgmap v9625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:41:36.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:36.041+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:41:36.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:36.041+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:41:36.044 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:36.041+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:41:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17432.145052 secs 2024-07-26T17:41:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17432.145052 secs 2024-07-26T17:41:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17432.145052 secs 2024-07-26T17:41:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:37 smithi112 ceph-mon[37501]: pgmap v9626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:37 smithi107 ceph-mon[32659]: pgmap v9626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:37 smithi132 ceph-mon[37580]: pgmap v9626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:39 smithi112 ceph-mon[37501]: pgmap v9627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:41:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:39 smithi107 ceph-mon[32659]: pgmap v9627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:41:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:39 smithi132 ceph-mon[37580]: pgmap v9627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:41:41.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:41 smithi132 ceph-mon[37580]: pgmap v9628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:41.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17437.145179 secs 2024-07-26T17:41:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:41 smithi112 ceph-mon[37501]: pgmap v9628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17437.145179 secs 2024-07-26T17:41:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:41 smithi107 ceph-mon[32659]: pgmap v9628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17437.145179 secs 2024-07-26T17:41:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:43 smithi112 ceph-mon[37501]: pgmap v9629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:43 smithi107 ceph-mon[32659]: pgmap v9629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:43 smithi132 ceph-mon[37580]: pgmap v9629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:45 smithi112 ceph-mon[37501]: pgmap v9630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:41:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:45 smithi107 ceph-mon[32659]: pgmap v9630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:41:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:45 smithi132 ceph-mon[37580]: pgmap v9630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:41:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17442.145303 secs 2024-07-26T17:41:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17442.145303 secs 2024-07-26T17:41:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17442.145303 secs 2024-07-26T17:41:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:47 smithi112 ceph-mon[37501]: pgmap v9631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:41:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:47 smithi107 ceph-mon[32659]: pgmap v9631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:41:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:47 smithi132 ceph-mon[37580]: pgmap v9631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:41:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:49 smithi112 ceph-mon[37501]: pgmap v9632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:41:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:49 smithi107 ceph-mon[32659]: pgmap v9632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:41:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:49 smithi132 ceph-mon[37580]: pgmap v9632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:41:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:51.049+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:41:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:51.049+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:41:51.051 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:41:51.049+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:41:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:51 smithi112 ceph-mon[37501]: pgmap v9633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:41:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17447.145386 secs 2024-07-26T17:41:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:51 smithi107 ceph-mon[32659]: pgmap v9633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:41:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17447.145386 secs 2024-07-26T17:41:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:51 smithi132 ceph-mon[37580]: pgmap v9633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:41:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17447.145386 secs 2024-07-26T17:41:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:53 smithi112 ceph-mon[37501]: pgmap v9634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:53 smithi107 ceph-mon[32659]: pgmap v9634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:53 smithi132 ceph-mon[37580]: pgmap v9634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:41:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:55 smithi112 ceph-mon[37501]: pgmap v9635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:41:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:55 smithi107 ceph-mon[32659]: pgmap v9635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:41:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:55 smithi132 ceph-mon[37580]: pgmap v9635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:41:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17452.145510 secs 2024-07-26T17:41:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17452.145510 secs 2024-07-26T17:41:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17452.145510 secs 2024-07-26T17:41:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:57 smithi112 ceph-mon[37501]: pgmap v9636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:57 smithi107 ceph-mon[32659]: pgmap v9636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:57 smithi132 ceph-mon[37580]: pgmap v9636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:41:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:41:59 smithi112 ceph-mon[37501]: pgmap v9637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:41:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:41:59 smithi107 ceph-mon[32659]: pgmap v9637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:41:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:41:59 smithi132 ceph-mon[37580]: pgmap v9637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:42:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:01 smithi112 ceph-mon[37501]: pgmap v9638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17457.145663 secs 2024-07-26T17:42:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:01 smithi107 ceph-mon[32659]: pgmap v9638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17457.145663 secs 2024-07-26T17:42:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:01 smithi132 ceph-mon[37580]: pgmap v9638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17457.145663 secs 2024-07-26T17:42:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:03 smithi112 ceph-mon[37501]: pgmap v9639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:03 smithi107 ceph-mon[32659]: pgmap v9639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:03 smithi132 ceph-mon[37580]: pgmap v9639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:05 smithi112 ceph-mon[37501]: pgmap v9640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:05 smithi107 ceph-mon[32659]: pgmap v9640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:05 smithi132 ceph-mon[37580]: pgmap v9640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:06.066 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:06.065+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:42:06.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:06.065+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:42:06.067 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:06.065+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:42:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17462.145791 secs 2024-07-26T17:42:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17462.145791 secs 2024-07-26T17:42:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17462.145791 secs 2024-07-26T17:42:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:07 smithi112 ceph-mon[37501]: pgmap v9641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:07.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:07 smithi107 ceph-mon[32659]: pgmap v9641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:07 smithi132 ceph-mon[37580]: pgmap v9641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:09 smithi112 ceph-mon[37501]: pgmap v9642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:42:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:09 smithi107 ceph-mon[32659]: pgmap v9642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:42:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:09 smithi132 ceph-mon[37580]: pgmap v9642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:42:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:11 smithi107 ceph-mon[32659]: pgmap v9643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:42:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17467.145917 secs 2024-07-26T17:42:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:11 smithi132 ceph-mon[37580]: pgmap v9643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:42:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17467.145917 secs 2024-07-26T17:42:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:11 smithi112 ceph-mon[37501]: pgmap v9643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:42:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17467.145917 secs 2024-07-26T17:42:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:13 smithi107 ceph-mon[32659]: pgmap v9644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:13 smithi132 ceph-mon[37580]: pgmap v9644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:13 smithi112 ceph-mon[37501]: pgmap v9644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:15.662 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:15.660+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:42:15.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:15.660+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:42:15.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:15.661+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:42:15.663 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:15.661+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:42:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:15 smithi107 ceph-mon[32659]: pgmap v9645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:42:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:15 smithi132 ceph-mon[37580]: pgmap v9645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:42:15.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:15 smithi112 ceph-mon[37501]: pgmap v9645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:42:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17472.146092 secs 2024-07-26T17:42:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17472.146092 secs 2024-07-26T17:42:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17472.146092 secs 2024-07-26T17:42:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:17 smithi107 ceph-mon[32659]: pgmap v9646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:42:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:17 smithi132 ceph-mon[37580]: pgmap v9646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:42:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:17 smithi112 ceph-mon[37501]: pgmap v9646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:42:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:42:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:42:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:42:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:19 smithi107 ceph-mon[32659]: pgmap v9647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:42:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:42:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:42:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:42:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:19 smithi132 ceph-mon[37580]: pgmap v9647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:42:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:42:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:42:19.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:42:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:19 smithi112 ceph-mon[37501]: pgmap v9647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T17:42:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:42:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:42:19.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:42:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:20 smithi107 ceph-mon[32659]: pgmap v9648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:42:20.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:20 smithi107 ceph-mon[32659]: pgmap v9649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:20 smithi132 ceph-mon[37580]: pgmap v9648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:42:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:20 smithi132 ceph-mon[37580]: pgmap v9649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:20 smithi112 ceph-mon[37501]: pgmap v9648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:42:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:20 smithi112 ceph-mon[37501]: pgmap v9649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:21.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:21.079+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:42:21.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:21.079+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:42:21.082 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:21.079+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:42:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17477.146185 secs 2024-07-26T17:42:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17477.146185 secs 2024-07-26T17:42:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17477.146185 secs 2024-07-26T17:42:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:22 smithi107 ceph-mon[32659]: pgmap v9650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:22 smithi132 ceph-mon[37580]: pgmap v9650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:22 smithi112 ceph-mon[37501]: pgmap v9650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:24 smithi107 ceph-mon[32659]: pgmap v9651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:24 smithi132 ceph-mon[37580]: pgmap v9651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:24 smithi112 ceph-mon[37501]: pgmap v9651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:25 smithi112 ceph-mon[37501]: pgmap v9652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17482.146314 secs 2024-07-26T17:42:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:25 smithi107 ceph-mon[32659]: pgmap v9652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17482.146314 secs 2024-07-26T17:42:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:25 smithi132 ceph-mon[37580]: pgmap v9652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:42:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17482.146314 secs 2024-07-26T17:42:28.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:28 smithi107 ceph-mon[32659]: pgmap v9653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:42:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:28 smithi132 ceph-mon[37580]: pgmap v9653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:42:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:28 smithi112 ceph-mon[37501]: pgmap v9653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:42:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:30 smithi107 ceph-mon[32659]: pgmap v9654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:42:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:30 smithi132 ceph-mon[37580]: pgmap v9654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:42:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:30 smithi112 ceph-mon[37501]: pgmap v9654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:42:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17487.146466 secs 2024-07-26T17:42:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17487.146466 secs 2024-07-26T17:42:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17487.146466 secs 2024-07-26T17:42:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:32 smithi107 ceph-mon[32659]: pgmap v9655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:32 smithi132 ceph-mon[37580]: pgmap v9655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:32 smithi112 ceph-mon[37501]: pgmap v9655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:34 smithi107 ceph-mon[32659]: pgmap v9656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:34 smithi132 ceph-mon[37580]: pgmap v9656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:34 smithi112 ceph-mon[37501]: pgmap v9656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:36.088 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:36.086+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:42:36.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:36.086+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:42:36.089 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:36.086+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:42:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:36 smithi107 ceph-mon[32659]: pgmap v9657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17492.146586 secs 2024-07-26T17:42:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:36 smithi132 ceph-mon[37580]: pgmap v9657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17492.146586 secs 2024-07-26T17:42:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:36 smithi112 ceph-mon[37501]: pgmap v9657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17492.146586 secs 2024-07-26T17:42:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:38 smithi107 ceph-mon[32659]: pgmap v9658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:38 smithi132 ceph-mon[37580]: pgmap v9658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:38 smithi112 ceph-mon[37501]: pgmap v9658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:40 smithi107 ceph-mon[32659]: pgmap v9659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:42:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:40 smithi132 ceph-mon[37580]: pgmap v9659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:42:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:40 smithi112 ceph-mon[37501]: pgmap v9659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:42:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17497.146718 secs 2024-07-26T17:42:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17497.146718 secs 2024-07-26T17:42:41.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17497.146718 secs 2024-07-26T17:42:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:42 smithi107 ceph-mon[32659]: pgmap v9660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:42 smithi132 ceph-mon[37580]: pgmap v9660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:42 smithi112 ceph-mon[37501]: pgmap v9660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:44.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:44 smithi132 ceph-mon[37580]: pgmap v9661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:44 smithi112 ceph-mon[37501]: pgmap v9661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:44 smithi107 ceph-mon[32659]: pgmap v9661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:46 smithi132 ceph-mon[37580]: pgmap v9662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17502.146839 secs 2024-07-26T17:42:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:46 smithi112 ceph-mon[37501]: pgmap v9662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17502.146839 secs 2024-07-26T17:42:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:46 smithi107 ceph-mon[32659]: pgmap v9662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17502.146839 secs 2024-07-26T17:42:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:48 smithi132 ceph-mon[37580]: pgmap v9663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:48 smithi112 ceph-mon[37501]: pgmap v9663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:48 smithi107 ceph-mon[32659]: pgmap v9663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:50 smithi132 ceph-mon[37580]: pgmap v9664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:42:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:50 smithi112 ceph-mon[37501]: pgmap v9664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:42:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:50 smithi107 ceph-mon[32659]: pgmap v9664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:42:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:51.090+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:42:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:51.090+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:42:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:42:51.090+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:42:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17507.146987 secs 2024-07-26T17:42:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17507.146987 secs 2024-07-26T17:42:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17507.146987 secs 2024-07-26T17:42:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:52 smithi112 ceph-mon[37501]: pgmap v9665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:52 smithi107 ceph-mon[32659]: pgmap v9665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:52 smithi132 ceph-mon[37580]: pgmap v9665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:42:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:54 smithi112 ceph-mon[37501]: pgmap v9666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:54 smithi107 ceph-mon[32659]: pgmap v9666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:55.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:54 smithi132 ceph-mon[37580]: pgmap v9666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:56 smithi112 ceph-mon[37501]: pgmap v9667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17512.147105 secs 2024-07-26T17:42:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:56 smithi107 ceph-mon[32659]: pgmap v9667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17512.147105 secs 2024-07-26T17:42:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:56 smithi132 ceph-mon[37580]: pgmap v9667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:42:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17512.147105 secs 2024-07-26T17:42:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:42:58 smithi112 ceph-mon[37501]: pgmap v9668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:42:58 smithi107 ceph-mon[32659]: pgmap v9668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:42:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:42:58 smithi132 ceph-mon[37580]: pgmap v9668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:00.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:00 smithi107 ceph-mon[32659]: pgmap v9669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:43:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:00 smithi112 ceph-mon[37501]: pgmap v9669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:43:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:00 smithi132 ceph-mon[37580]: pgmap v9669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:43:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17517.147212 secs 2024-07-26T17:43:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17517.147212 secs 2024-07-26T17:43:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17517.147212 secs 2024-07-26T17:43:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:02 smithi112 ceph-mon[37501]: pgmap v9670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:02 smithi107 ceph-mon[32659]: pgmap v9670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:02 smithi132 ceph-mon[37580]: pgmap v9670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:03 smithi112 ceph-mon[37501]: pgmap v9671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:03 smithi107 ceph-mon[32659]: pgmap v9671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:03 smithi132 ceph-mon[37580]: pgmap v9671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:06.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:06.106+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:43:06.107 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:06.106+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:43:06.108 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:06.106+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:43:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:06 smithi107 ceph-mon[32659]: pgmap v9672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17522.147277 secs 2024-07-26T17:43:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:06 smithi132 ceph-mon[37580]: pgmap v9672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17522.147277 secs 2024-07-26T17:43:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:06 smithi112 ceph-mon[37501]: pgmap v9672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17522.147277 secs 2024-07-26T17:43:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:08 smithi107 ceph-mon[32659]: pgmap v9673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:08 smithi132 ceph-mon[37580]: pgmap v9673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:08 smithi112 ceph-mon[37501]: pgmap v9673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:10.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:10 smithi107 ceph-mon[32659]: pgmap v9674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:43:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:10 smithi132 ceph-mon[37580]: pgmap v9674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:43:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:10 smithi112 ceph-mon[37501]: pgmap v9674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:43:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17527.147387 secs 2024-07-26T17:43:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17527.147387 secs 2024-07-26T17:43:11.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17527.147387 secs 2024-07-26T17:43:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:12 smithi107 ceph-mon[32659]: pgmap v9675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:12 smithi132 ceph-mon[37580]: pgmap v9675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:12 smithi112 ceph-mon[37501]: pgmap v9675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:14 smithi107 ceph-mon[32659]: pgmap v9676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:14 smithi132 ceph-mon[37580]: pgmap v9676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:14 smithi112 ceph-mon[37501]: pgmap v9676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:16 smithi132 ceph-mon[37580]: pgmap v9677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17532.147509 secs 2024-07-26T17:43:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:16 smithi112 ceph-mon[37501]: pgmap v9677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17532.147509 secs 2024-07-26T17:43:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:16 smithi107 ceph-mon[32659]: pgmap v9677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17532.147509 secs 2024-07-26T17:43:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:18 smithi132 ceph-mon[37580]: pgmap v9678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:18 smithi112 ceph-mon[37501]: pgmap v9678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:18 smithi107 ceph-mon[32659]: pgmap v9678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:20 smithi132 ceph-mon[37580]: pgmap v9679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:43:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:20 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:43:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:43:20.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:20 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:43:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:20 smithi112 ceph-mon[37501]: pgmap v9679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:43:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:20 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:43:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:43:20.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:20 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:43:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:20 smithi107 ceph-mon[32659]: pgmap v9679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:43:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:20 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:43:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:43:21.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:20 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:43:21.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:21.120+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:43:21.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:21.120+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:43:21.124 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:21.120+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:43:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17537.147592 secs 2024-07-26T17:43:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:43:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:21 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:43:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:43:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17537.147592 secs 2024-07-26T17:43:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:43:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:21 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:43:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:43:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17537.147592 secs 2024-07-26T17:43:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:43:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:21 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:43:22.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:43:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:22 smithi112 ceph-mon[37501]: pgmap v9680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T17:43:22.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:22 smithi112 ceph-mon[37501]: pgmap v9681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:43:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:22 smithi107 ceph-mon[32659]: pgmap v9680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T17:43:23.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:22 smithi107 ceph-mon[32659]: pgmap v9681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:43:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:22 smithi132 ceph-mon[37580]: pgmap v9680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T17:43:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:22 smithi132 ceph-mon[37580]: pgmap v9681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:43:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:24 smithi112 ceph-mon[37501]: pgmap v9682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:43:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:24 smithi107 ceph-mon[32659]: pgmap v9682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:43:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:24 smithi132 ceph-mon[37580]: pgmap v9682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:43:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:26 smithi112 ceph-mon[37501]: pgmap v9683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:43:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17542.147736 secs 2024-07-26T17:43:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:26 smithi107 ceph-mon[32659]: pgmap v9683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:43:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17542.147736 secs 2024-07-26T17:43:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:26 smithi132 ceph-mon[37580]: pgmap v9683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:43:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17542.147736 secs 2024-07-26T17:43:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:28 smithi112 ceph-mon[37501]: pgmap v9684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:43:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:28 smithi107 ceph-mon[32659]: pgmap v9684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:43:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:28 smithi132 ceph-mon[37580]: pgmap v9684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:43:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:30 smithi112 ceph-mon[37501]: pgmap v9685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:43:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:30 smithi107 ceph-mon[32659]: pgmap v9685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:43:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:30 smithi132 ceph-mon[37580]: pgmap v9685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T17:43:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17547.147867 secs 2024-07-26T17:43:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17547.147867 secs 2024-07-26T17:43:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17547.147867 secs 2024-07-26T17:43:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:32 smithi112 ceph-mon[37501]: pgmap v9686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:43:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:32 smithi107 ceph-mon[32659]: pgmap v9686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:43:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:32 smithi132 ceph-mon[37580]: pgmap v9686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:43:34.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:34 smithi112 ceph-mon[37501]: pgmap v9687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:43:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:34 smithi107 ceph-mon[32659]: pgmap v9687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:43:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:34 smithi132 ceph-mon[37580]: pgmap v9687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:43:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:35 smithi112 ceph-mon[37501]: pgmap v9688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:35 smithi107 ceph-mon[32659]: pgmap v9688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:35 smithi132 ceph-mon[37580]: pgmap v9688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:36.138 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:36.136+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:36.136+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:43:36.139 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:36.136+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:43:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17552.148008 secs 2024-07-26T17:43:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17552.148008 secs 2024-07-26T17:43:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17552.148008 secs 2024-07-26T17:43:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:37 smithi112 ceph-mon[37501]: pgmap v9689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:37 smithi107 ceph-mon[32659]: pgmap v9689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:37 smithi132 ceph-mon[37580]: pgmap v9689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:40 smithi107 ceph-mon[32659]: pgmap v9690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:40 smithi132 ceph-mon[37580]: pgmap v9690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:40 smithi112 ceph-mon[37501]: pgmap v9690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17557.148114 secs 2024-07-26T17:43:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17557.148114 secs 2024-07-26T17:43:41.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17557.148114 secs 2024-07-26T17:43:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:42 smithi107 ceph-mon[32659]: pgmap v9691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:42 smithi132 ceph-mon[37580]: pgmap v9691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:42 smithi112 ceph-mon[37501]: pgmap v9691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:44 smithi107 ceph-mon[32659]: pgmap v9692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:43:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:44 smithi132 ceph-mon[37580]: pgmap v9692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:43:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:44 smithi112 ceph-mon[37501]: pgmap v9692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:43:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:46 smithi107 ceph-mon[32659]: pgmap v9693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17562.148244 secs 2024-07-26T17:43:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:46 smithi132 ceph-mon[37580]: pgmap v9693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17562.148244 secs 2024-07-26T17:43:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:46 smithi112 ceph-mon[37501]: pgmap v9693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17562.148244 secs 2024-07-26T17:43:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:48 smithi107 ceph-mon[32659]: pgmap v9694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:48 smithi132 ceph-mon[37580]: pgmap v9694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:48 smithi112 ceph-mon[37501]: pgmap v9694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:50 smithi107 ceph-mon[32659]: pgmap v9695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:50 smithi132 ceph-mon[37580]: pgmap v9695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:50 smithi112 ceph-mon[37501]: pgmap v9695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:43:51.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:51.153+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:43:51.154 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:51.153+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:43:51.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:43:51.153+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:43:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17567.148357 secs 2024-07-26T17:43:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17567.148357 secs 2024-07-26T17:43:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17567.148357 secs 2024-07-26T17:43:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:52 smithi132 ceph-mon[37580]: pgmap v9696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:52 smithi112 ceph-mon[37501]: pgmap v9696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:52.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:52 smithi107 ceph-mon[32659]: pgmap v9696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:54 smithi132 ceph-mon[37580]: pgmap v9697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:43:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:54 smithi112 ceph-mon[37501]: pgmap v9697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:43:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:54 smithi107 ceph-mon[32659]: pgmap v9697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:43:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:56 smithi112 ceph-mon[37501]: pgmap v9698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17572.148477 secs 2024-07-26T17:43:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:56 smithi107 ceph-mon[32659]: pgmap v9698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17572.148477 secs 2024-07-26T17:43:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:56 smithi132 ceph-mon[37580]: pgmap v9698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:43:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17572.148477 secs 2024-07-26T17:43:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:43:58 smithi112 ceph-mon[37501]: pgmap v9699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:43:58 smithi107 ceph-mon[32659]: pgmap v9699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:43:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:43:58 smithi132 ceph-mon[37580]: pgmap v9699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:44:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:00 smithi112 ceph-mon[37501]: pgmap v9700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:00 smithi107 ceph-mon[32659]: pgmap v9700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:00 smithi132 ceph-mon[37580]: pgmap v9700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17577.148598 secs 2024-07-26T17:44:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17577.148598 secs 2024-07-26T17:44:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17577.148598 secs 2024-07-26T17:44:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:02 smithi112 ceph-mon[37501]: pgmap v9701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:02.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:02 smithi107 ceph-mon[32659]: pgmap v9701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:02 smithi132 ceph-mon[37580]: pgmap v9701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:04 smithi112 ceph-mon[37501]: pgmap v9702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:44:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:04 smithi107 ceph-mon[32659]: pgmap v9702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:44:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:04 smithi132 ceph-mon[37580]: pgmap v9702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:44:06.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:06.168+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:44:06.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:06.168+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:44:06.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:06.168+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:44:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:06 smithi107 ceph-mon[32659]: pgmap v9703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:06.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17582.148723 secs 2024-07-26T17:44:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:06 smithi112 ceph-mon[37501]: pgmap v9703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17582.148723 secs 2024-07-26T17:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:06 smithi132 ceph-mon[37580]: pgmap v9703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17582.148723 secs 2024-07-26T17:44:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:08 smithi112 ceph-mon[37501]: pgmap v9704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:44:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:08 smithi107 ceph-mon[32659]: pgmap v9704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:44:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:08 smithi132 ceph-mon[37580]: pgmap v9704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:44:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:10 smithi112 ceph-mon[37501]: pgmap v9705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:44:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:10 smithi107 ceph-mon[32659]: pgmap v9705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:44:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:10 smithi132 ceph-mon[37580]: pgmap v9705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:44:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17587.148850 secs 2024-07-26T17:44:11.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17587.148850 secs 2024-07-26T17:44:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17587.148850 secs 2024-07-26T17:44:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:12 smithi112 ceph-mon[37501]: pgmap v9706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:44:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:12 smithi107 ceph-mon[32659]: pgmap v9706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:44:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:12 smithi132 ceph-mon[37580]: pgmap v9706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T17:44:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:14 smithi112 ceph-mon[37501]: pgmap v9707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:44:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:14 smithi107 ceph-mon[32659]: pgmap v9707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:44:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:14 smithi132 ceph-mon[37580]: pgmap v9707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:44:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:16 smithi112 ceph-mon[37501]: pgmap v9708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:44:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17592.148966 secs 2024-07-26T17:44:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:16 smithi107 ceph-mon[32659]: pgmap v9708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:44:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17592.148966 secs 2024-07-26T17:44:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:16 smithi132 ceph-mon[37580]: pgmap v9708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:44:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17592.148966 secs 2024-07-26T17:44:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:18 smithi112 ceph-mon[37501]: pgmap v9709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:44:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:18 smithi107 ceph-mon[32659]: pgmap v9709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:44:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:18 smithi132 ceph-mon[37580]: pgmap v9709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:44:20.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:20 smithi112 ceph-mon[37501]: pgmap v9710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:44:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:20 smithi107 ceph-mon[32659]: pgmap v9710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:44:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:20 smithi132 ceph-mon[37580]: pgmap v9710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:44:21.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:21.171+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:44:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:21.171+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:44:21.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:21.171+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:44:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17597.149103 secs 2024-07-26T17:44:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17597.149103 secs 2024-07-26T17:44:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17597.149103 secs 2024-07-26T17:44:22.676 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:22 smithi132 ceph-mon[37580]: pgmap v9711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:44:22.676 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:22 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:44:22.676 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.676 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.677 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.677 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.696 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:22 smithi112 ceph-mon[37501]: pgmap v9711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:44:22.697 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:22 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:44:22.697 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.697 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.697 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.697 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:22 smithi107 ceph-mon[32659]: pgmap v9711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:44:22.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:22 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:44:22.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:22.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:24 smithi107 ceph-mon[32659]: pgmap v9712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:44:24.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:24 smithi132 ceph-mon[37580]: pgmap v9712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:44:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:24 smithi112 ceph-mon[37501]: pgmap v9712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:44:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:24.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:26 smithi112 ceph-mon[37501]: pgmap v9713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17602.149229 secs 2024-07-26T17:44:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:26 smithi107 ceph-mon[32659]: pgmap v9713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17602.149229 secs 2024-07-26T17:44:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:26 smithi132 ceph-mon[37580]: pgmap v9713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17602.149229 secs 2024-07-26T17:44:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:44:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:44:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:27 smithi112 ceph-mon[37501]: pgmap v9714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:44:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:27 smithi112 ceph-mon[37501]: pgmap v9715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:27.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:27.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:27.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:44:27.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:44:27.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:27 smithi107 ceph-mon[32659]: pgmap v9714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:44:27.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:27 smithi107 ceph-mon[32659]: pgmap v9715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:27.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:27.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:27.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:44:27.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:44:27.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:27 smithi132 ceph-mon[37580]: pgmap v9714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:44:27.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:27 smithi132 ceph-mon[37580]: pgmap v9715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:27.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:44:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:29 smithi107 ceph-mon[32659]: pgmap v9716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:29 smithi132 ceph-mon[37580]: pgmap v9716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:29 smithi112 ceph-mon[37501]: pgmap v9716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:31 smithi107 ceph-mon[32659]: pgmap v9717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17607.149306 secs 2024-07-26T17:44:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:31 smithi132 ceph-mon[37580]: pgmap v9717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17607.149306 secs 2024-07-26T17:44:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:31 smithi112 ceph-mon[37501]: pgmap v9717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17607.149306 secs 2024-07-26T17:44:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:33 smithi107 ceph-mon[32659]: pgmap v9718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:44:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:33 smithi132 ceph-mon[37580]: pgmap v9718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:44:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:33 smithi112 ceph-mon[37501]: pgmap v9718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:44:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:35 smithi107 ceph-mon[32659]: pgmap v9719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:35 smithi132 ceph-mon[37580]: pgmap v9719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:35 smithi112 ceph-mon[37501]: pgmap v9719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:44:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:36.172+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:44:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:36.172+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:44:36.175 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:36.172+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:44:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17612.149418 secs 2024-07-26T17:44:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17612.149418 secs 2024-07-26T17:44:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17612.149418 secs 2024-07-26T17:44:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:37 smithi107 ceph-mon[32659]: pgmap v9720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:44:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:37 smithi132 ceph-mon[37580]: pgmap v9720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:44:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:37 smithi112 ceph-mon[37501]: pgmap v9720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:44:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:39 smithi132 ceph-mon[37580]: pgmap v9721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:39 smithi112 ceph-mon[37501]: pgmap v9721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:39 smithi107 ceph-mon[32659]: pgmap v9721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:41 smithi132 ceph-mon[37580]: pgmap v9722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17617.149541 secs 2024-07-26T17:44:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:41 smithi112 ceph-mon[37501]: pgmap v9722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:41.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17617.149541 secs 2024-07-26T17:44:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:41 smithi107 ceph-mon[32659]: pgmap v9722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17617.149541 secs 2024-07-26T17:44:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:43 smithi132 ceph-mon[37580]: pgmap v9723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:43 smithi112 ceph-mon[37501]: pgmap v9723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:43 smithi107 ceph-mon[32659]: pgmap v9723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:45 smithi112 ceph-mon[37501]: pgmap v9724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:44:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:45 smithi107 ceph-mon[32659]: pgmap v9724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:44:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:45 smithi132 ceph-mon[37580]: pgmap v9724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:44:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17622.149623 secs 2024-07-26T17:44:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17622.149623 secs 2024-07-26T17:44:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17622.149623 secs 2024-07-26T17:44:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:47 smithi112 ceph-mon[37501]: pgmap v9725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:44:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:47 smithi107 ceph-mon[32659]: pgmap v9725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:44:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:47 smithi132 ceph-mon[37580]: pgmap v9725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:44:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:49 smithi112 ceph-mon[37501]: pgmap v9726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:49 smithi107 ceph-mon[32659]: pgmap v9726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:49 smithi132 ceph-mon[37580]: pgmap v9726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:51.189+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:44:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:51.189+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:44:51.191 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:44:51.189+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:44:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:51 smithi112 ceph-mon[37501]: pgmap v9727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17627.149734 secs 2024-07-26T17:44:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:51 smithi107 ceph-mon[32659]: pgmap v9727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17627.149734 secs 2024-07-26T17:44:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:51 smithi132 ceph-mon[37580]: pgmap v9727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:44:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17627.149734 secs 2024-07-26T17:44:53.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:53 smithi112 ceph-mon[37501]: pgmap v9728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:53 smithi107 ceph-mon[32659]: pgmap v9728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:53 smithi132 ceph-mon[37580]: pgmap v9728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:44:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:55 smithi112 ceph-mon[37501]: pgmap v9729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:44:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:55 smithi107 ceph-mon[32659]: pgmap v9729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:44:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:55 smithi132 ceph-mon[37580]: pgmap v9729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:44:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17632.149853 secs 2024-07-26T17:44:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17632.149853 secs 2024-07-26T17:44:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17632.149853 secs 2024-07-26T17:44:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:57 smithi112 ceph-mon[37501]: pgmap v9730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:44:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:57 smithi107 ceph-mon[32659]: pgmap v9730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:44:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:57 smithi132 ceph-mon[37580]: pgmap v9730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:44:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:44:59 smithi112 ceph-mon[37501]: pgmap v9731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:44:59 smithi107 ceph-mon[32659]: pgmap v9731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:44:59 smithi132 ceph-mon[37580]: pgmap v9731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:01 smithi112 ceph-mon[37501]: pgmap v9732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17637.149989 secs 2024-07-26T17:45:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:01 smithi107 ceph-mon[32659]: pgmap v9732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17637.149989 secs 2024-07-26T17:45:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:01 smithi132 ceph-mon[37580]: pgmap v9732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17637.149989 secs 2024-07-26T17:45:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:03 smithi112 ceph-mon[37501]: pgmap v9733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:03 smithi107 ceph-mon[32659]: pgmap v9733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:03 smithi132 ceph-mon[37580]: pgmap v9733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:04 smithi112 ceph-mon[37501]: pgmap v9734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:04 smithi107 ceph-mon[32659]: pgmap v9734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:04 smithi132 ceph-mon[37580]: pgmap v9734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17642.150046 secs 2024-07-26T17:45:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17642.150046 secs 2024-07-26T17:45:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:06.203+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:45:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:06.203+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:45:06.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:06.203+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:45:06.205 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17642.150046 secs 2024-07-26T17:45:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:06 smithi112 ceph-mon[37501]: pgmap v9735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:45:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:06 smithi107 ceph-mon[32659]: pgmap v9735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:45:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:06 smithi132 ceph-mon[37580]: pgmap v9735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:45:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:09 smithi107 ceph-mon[32659]: pgmap v9736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:09 smithi132 ceph-mon[37580]: pgmap v9736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:09 smithi112 ceph-mon[37501]: pgmap v9736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:11 smithi107 ceph-mon[32659]: pgmap v9737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17647.150173 secs 2024-07-26T17:45:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:11 smithi132 ceph-mon[37580]: pgmap v9737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17647.150173 secs 2024-07-26T17:45:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:11 smithi112 ceph-mon[37501]: pgmap v9737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17647.150173 secs 2024-07-26T17:45:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:13 smithi132 ceph-mon[37580]: pgmap v9738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:13 smithi112 ceph-mon[37501]: pgmap v9738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:13 smithi107 ceph-mon[32659]: pgmap v9738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:15.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:15.693+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:45:15.694 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:15.693+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:45:15.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:15.693+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:45:15.695 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:15.693+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:45:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:15 smithi132 ceph-mon[37580]: pgmap v9739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:15 smithi112 ceph-mon[37501]: pgmap v9739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:15 smithi107 ceph-mon[32659]: pgmap v9739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17652.150289 secs 2024-07-26T17:45:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17652.150289 secs 2024-07-26T17:45:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17652.150289 secs 2024-07-26T17:45:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:17 smithi112 ceph-mon[37501]: pgmap v9740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:45:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:17 smithi107 ceph-mon[32659]: pgmap v9740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:45:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:17 smithi132 ceph-mon[37580]: pgmap v9740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:45:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:19 smithi112 ceph-mon[37501]: pgmap v9741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:19 smithi107 ceph-mon[32659]: pgmap v9741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:19 smithi132 ceph-mon[37580]: pgmap v9741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:20 smithi112 ceph-mon[37501]: pgmap v9742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17657.150351 secs 2024-07-26T17:45:21.220 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:21.217+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:45:21.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:21.218+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:45:21.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:21.218+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:45:21.221 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:20 smithi107 ceph-mon[32659]: pgmap v9742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:21.221 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17657.150351 secs 2024-07-26T17:45:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:20 smithi132 ceph-mon[37580]: pgmap v9742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17657.150351 secs 2024-07-26T17:45:23.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:23 smithi107 ceph-mon[32659]: pgmap v9743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:23 smithi132 ceph-mon[37580]: pgmap v9743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:23 smithi112 ceph-mon[37501]: pgmap v9743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:25 smithi132 ceph-mon[37580]: pgmap v9744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:25 smithi112 ceph-mon[37501]: pgmap v9744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:25 smithi107 ceph-mon[32659]: pgmap v9744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17662.150475 secs 2024-07-26T17:45:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17662.150475 secs 2024-07-26T17:45:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17662.150475 secs 2024-07-26T17:45:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:27 smithi112 ceph-mon[37501]: pgmap v9745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:45:27.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:45:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:27 smithi107 ceph-mon[32659]: pgmap v9745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:45:28.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:45:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:27 smithi132 ceph-mon[37580]: pgmap v9745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:45:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:45:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:45:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:45:28.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:45:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:45:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:45:29.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:45:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:45:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:45:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:45:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:29 smithi112 ceph-mon[37501]: pgmap v9746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:45:29.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:29 smithi112 ceph-mon[37501]: pgmap v9747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:45:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:29 smithi107 ceph-mon[32659]: pgmap v9746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:45:30.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:29 smithi107 ceph-mon[32659]: pgmap v9747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:45:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:29 smithi132 ceph-mon[37580]: pgmap v9746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:45:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:29 smithi132 ceph-mon[37580]: pgmap v9747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:45:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:31 smithi112 ceph-mon[37501]: pgmap v9748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:45:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17667.150598 secs 2024-07-26T17:45:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:31 smithi107 ceph-mon[32659]: pgmap v9748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:45:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17667.150598 secs 2024-07-26T17:45:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:31 smithi132 ceph-mon[37580]: pgmap v9748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:45:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17667.150598 secs 2024-07-26T17:45:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:33 smithi112 ceph-mon[37501]: pgmap v9749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:33 smithi107 ceph-mon[32659]: pgmap v9749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:33 smithi132 ceph-mon[37580]: pgmap v9749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:35.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:35 smithi112 ceph-mon[37501]: pgmap v9750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:45:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:35 smithi107 ceph-mon[32659]: pgmap v9750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:45:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:35 smithi132 ceph-mon[37580]: pgmap v9750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:45:36.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:36.225+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:45:36.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:36.225+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:45:36.228 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:36.225+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:45:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17672.150715 secs 2024-07-26T17:45:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17672.150715 secs 2024-07-26T17:45:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17672.150715 secs 2024-07-26T17:45:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:37 smithi112 ceph-mon[37501]: pgmap v9751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:45:38.010 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:37 smithi107 ceph-mon[32659]: pgmap v9751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:45:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:37 smithi132 ceph-mon[37580]: pgmap v9751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:45:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:39 smithi112 ceph-mon[37501]: pgmap v9752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:45:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:39 smithi107 ceph-mon[32659]: pgmap v9752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:45:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:39 smithi132 ceph-mon[37580]: pgmap v9752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:45:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:41 smithi112 ceph-mon[37501]: pgmap v9753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17677.150854 secs 2024-07-26T17:45:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:41 smithi107 ceph-mon[32659]: pgmap v9753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17677.150854 secs 2024-07-26T17:45:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:41 smithi132 ceph-mon[37580]: pgmap v9753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17677.150854 secs 2024-07-26T17:45:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:43 smithi112 ceph-mon[37501]: pgmap v9754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:45:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:43 smithi107 ceph-mon[32659]: pgmap v9754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:45:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:43 smithi132 ceph-mon[37580]: pgmap v9754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:45:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:45 smithi112 ceph-mon[37501]: pgmap v9755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:45 smithi107 ceph-mon[32659]: pgmap v9755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:45 smithi132 ceph-mon[37580]: pgmap v9755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17682.151007 secs 2024-07-26T17:45:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17682.151007 secs 2024-07-26T17:45:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17682.151007 secs 2024-07-26T17:45:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:47 smithi112 ceph-mon[37501]: pgmap v9756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:47 smithi107 ceph-mon[32659]: pgmap v9756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:47 smithi132 ceph-mon[37580]: pgmap v9756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:48 smithi112 ceph-mon[37501]: pgmap v9757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:48 smithi107 ceph-mon[32659]: pgmap v9757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:48 smithi132 ceph-mon[37580]: pgmap v9757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:50 smithi107 ceph-mon[32659]: pgmap v9758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:50 smithi132 ceph-mon[37580]: pgmap v9758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:50 smithi112 ceph-mon[37501]: pgmap v9758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:51.242+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:51.242+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:45:51.243 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:45:51.242+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:45:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17687.151142 secs 2024-07-26T17:45:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17687.151142 secs 2024-07-26T17:45:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17687.151142 secs 2024-07-26T17:45:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:52 smithi107 ceph-mon[32659]: pgmap v9759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:45:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:52 smithi132 ceph-mon[37580]: pgmap v9759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:45:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:52 smithi112 ceph-mon[37501]: pgmap v9759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:45:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:54 smithi107 ceph-mon[32659]: pgmap v9760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:54 smithi132 ceph-mon[37580]: pgmap v9760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:54 smithi112 ceph-mon[37501]: pgmap v9760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17692.151274 secs 2024-07-26T17:45:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:56 smithi107 ceph-mon[32659]: pgmap v9761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17692.151274 secs 2024-07-26T17:45:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:56 smithi132 ceph-mon[37580]: pgmap v9761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17692.151274 secs 2024-07-26T17:45:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:56 smithi112 ceph-mon[37501]: pgmap v9761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:45:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:45:58 smithi107 ceph-mon[32659]: pgmap v9762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:45:58 smithi132 ceph-mon[37580]: pgmap v9762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:45:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:45:58 smithi112 ceph-mon[37501]: pgmap v9762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:00 smithi107 ceph-mon[32659]: pgmap v9763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:46:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:00 smithi132 ceph-mon[37580]: pgmap v9763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:46:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:00 smithi112 ceph-mon[37501]: pgmap v9763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:46:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17697.151376 secs 2024-07-26T17:46:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17697.151376 secs 2024-07-26T17:46:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17697.151376 secs 2024-07-26T17:46:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:02 smithi107 ceph-mon[32659]: pgmap v9764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:02 smithi132 ceph-mon[37580]: pgmap v9764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:02 smithi112 ceph-mon[37501]: pgmap v9764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:04 smithi107 ceph-mon[32659]: pgmap v9765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:04 smithi132 ceph-mon[37580]: pgmap v9765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:04 smithi112 ceph-mon[37501]: pgmap v9765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17702.151526 secs 2024-07-26T17:46:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17702.151526 secs 2024-07-26T17:46:06.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:06.257+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:46:06.259 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:06.257+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:46:06.260 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:06.257+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:46:06.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17702.151526 secs 2024-07-26T17:46:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:06 smithi132 ceph-mon[37580]: pgmap v9766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:06 smithi112 ceph-mon[37501]: pgmap v9766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:06 smithi107 ceph-mon[32659]: pgmap v9766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:09.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:08 smithi132 ceph-mon[37580]: pgmap v9767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:08 smithi112 ceph-mon[37501]: pgmap v9767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:08 smithi107 ceph-mon[32659]: pgmap v9767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:10 smithi132 ceph-mon[37580]: pgmap v9768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:46:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17707.151649 secs 2024-07-26T17:46:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:10 smithi112 ceph-mon[37501]: pgmap v9768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:46:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17707.151649 secs 2024-07-26T17:46:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:10 smithi107 ceph-mon[32659]: pgmap v9768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:46:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17707.151649 secs 2024-07-26T17:46:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:12 smithi112 ceph-mon[37501]: pgmap v9769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:12 smithi107 ceph-mon[32659]: pgmap v9769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:12 smithi132 ceph-mon[37580]: pgmap v9769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:14 smithi112 ceph-mon[37501]: pgmap v9770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:14 smithi107 ceph-mon[32659]: pgmap v9770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:14 smithi132 ceph-mon[37580]: pgmap v9770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17712.151775 secs 2024-07-26T17:46:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17712.151775 secs 2024-07-26T17:46:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17712.151775 secs 2024-07-26T17:46:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:16 smithi112 ceph-mon[37501]: pgmap v9771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:16 smithi107 ceph-mon[32659]: pgmap v9771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:16 smithi132 ceph-mon[37580]: pgmap v9771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:18 smithi112 ceph-mon[37501]: pgmap v9772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:18 smithi107 ceph-mon[32659]: pgmap v9772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:18 smithi132 ceph-mon[37580]: pgmap v9772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:20 smithi112 ceph-mon[37501]: pgmap v9773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:46:21.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17717.151866 secs 2024-07-26T17:46:21.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:21.272+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:46:21.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:21.272+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:46:21.274 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:21.272+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:46:21.275 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:20 smithi107 ceph-mon[32659]: pgmap v9773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:46:21.275 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17717.151866 secs 2024-07-26T17:46:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:20 smithi132 ceph-mon[37580]: pgmap v9773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:46:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17717.151866 secs 2024-07-26T17:46:23.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:22 smithi112 ceph-mon[37501]: pgmap v9774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:22 smithi107 ceph-mon[32659]: pgmap v9774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:22 smithi132 ceph-mon[37580]: pgmap v9774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:24 smithi112 ceph-mon[37501]: pgmap v9775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:24 smithi107 ceph-mon[32659]: pgmap v9775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:24 smithi132 ceph-mon[37580]: pgmap v9775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17722.151986 secs 2024-07-26T17:46:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17722.151986 secs 2024-07-26T17:46:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17722.151986 secs 2024-07-26T17:46:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:26 smithi112 ceph-mon[37501]: pgmap v9776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:26 smithi107 ceph-mon[32659]: pgmap v9776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:26 smithi132 ceph-mon[37580]: pgmap v9776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:46:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:46:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:46:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:28 smithi112 ceph-mon[37501]: pgmap v9777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:46:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:46:29.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:46:29.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:28 smithi107 ceph-mon[32659]: pgmap v9777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:29.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:46:29.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:46:29.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:46:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:28 smithi132 ceph-mon[37580]: pgmap v9777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:46:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:46:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:46:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:29 smithi112 ceph-mon[37501]: pgmap v9778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:46:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:29 smithi112 ceph-mon[37501]: pgmap v9779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:46:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:29 smithi107 ceph-mon[32659]: pgmap v9778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:46:30.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:29 smithi107 ceph-mon[32659]: pgmap v9779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:46:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:29 smithi132 ceph-mon[37580]: pgmap v9778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:46:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:29 smithi132 ceph-mon[37580]: pgmap v9779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:46:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:30 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17727.152102 secs 2024-07-26T17:46:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:30 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17727.152102 secs 2024-07-26T17:46:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:30 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17727.152102 secs 2024-07-26T17:46:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:31 smithi112 ceph-mon[37501]: pgmap v9780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:46:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:31 smithi107 ceph-mon[32659]: pgmap v9780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:46:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:31 smithi132 ceph-mon[37580]: pgmap v9780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:46:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:33 smithi112 ceph-mon[37501]: pgmap v9781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:46:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:33 smithi107 ceph-mon[32659]: pgmap v9781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:46:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:33 smithi132 ceph-mon[37580]: pgmap v9781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:46:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:35 smithi112 ceph-mon[37501]: pgmap v9782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:46:36.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17732.152215 secs 2024-07-26T17:46:36.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:36.273+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:46:36.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:36.273+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:46:36.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:36.273+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:46:36.276 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:35 smithi107 ceph-mon[32659]: pgmap v9782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:46:36.277 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17732.152215 secs 2024-07-26T17:46:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:35 smithi132 ceph-mon[37580]: pgmap v9782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:46:36.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17732.152215 secs 2024-07-26T17:46:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:37 smithi112 ceph-mon[37501]: pgmap v9783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:37 smithi107 ceph-mon[32659]: pgmap v9783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:37 smithi132 ceph-mon[37580]: pgmap v9783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:39 smithi112 ceph-mon[37501]: pgmap v9784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:46:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:39 smithi107 ceph-mon[32659]: pgmap v9784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:46:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:39 smithi132 ceph-mon[37580]: pgmap v9784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:46:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:40 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17737.152301 secs 2024-07-26T17:46:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:40 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17737.152301 secs 2024-07-26T17:46:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:40 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17737.152301 secs 2024-07-26T17:46:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:41 smithi112 ceph-mon[37501]: pgmap v9785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:42.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:41 smithi132 ceph-mon[37580]: pgmap v9785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:41 smithi107 ceph-mon[32659]: pgmap v9785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:43 smithi107 ceph-mon[32659]: pgmap v9786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:43 smithi132 ceph-mon[37580]: pgmap v9786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:43 smithi112 ceph-mon[37501]: pgmap v9786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:45 smithi107 ceph-mon[32659]: pgmap v9787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17742.152421 secs 2024-07-26T17:46:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:45 smithi132 ceph-mon[37580]: pgmap v9787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17742.152421 secs 2024-07-26T17:46:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:45 smithi112 ceph-mon[37501]: pgmap v9787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17742.152421 secs 2024-07-26T17:46:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:47 smithi107 ceph-mon[32659]: pgmap v9788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:47 smithi132 ceph-mon[37580]: pgmap v9788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:47 smithi112 ceph-mon[37501]: pgmap v9788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:49 smithi107 ceph-mon[32659]: pgmap v9789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:49 smithi132 ceph-mon[37580]: pgmap v9789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:49 smithi112 ceph-mon[37501]: pgmap v9789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:51.289 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:51.287+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:46:51.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:51.288+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:46:51.290 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:46:51.288+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:46:51.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17747.152544 secs 2024-07-26T17:46:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17747.152544 secs 2024-07-26T17:46:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17747.152544 secs 2024-07-26T17:46:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:51 smithi107 ceph-mon[32659]: pgmap v9790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:51 smithi132 ceph-mon[37580]: pgmap v9790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:51 smithi112 ceph-mon[37501]: pgmap v9790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:46:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:53 smithi107 ceph-mon[32659]: pgmap v9791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:53 smithi132 ceph-mon[37580]: pgmap v9791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:53 smithi112 ceph-mon[37501]: pgmap v9791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:46:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:55 smithi107 ceph-mon[32659]: pgmap v9792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17752.152665 secs 2024-07-26T17:46:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:55 smithi132 ceph-mon[37580]: pgmap v9792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17752.152665 secs 2024-07-26T17:46:56.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:55 smithi112 ceph-mon[37501]: pgmap v9792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:56.454 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17752.152665 secs 2024-07-26T17:46:58.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:46:57 smithi107 ceph-mon[32659]: pgmap v9793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:46:57 smithi132 ceph-mon[37580]: pgmap v9793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:46:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:46:57 smithi112 ceph-mon[37501]: pgmap v9793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:00 smithi107 ceph-mon[32659]: pgmap v9794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:00 smithi132 ceph-mon[37580]: pgmap v9794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:00 smithi112 ceph-mon[37501]: pgmap v9794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17757.152801 secs 2024-07-26T17:47:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17757.152801 secs 2024-07-26T17:47:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17757.152801 secs 2024-07-26T17:47:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:02 smithi107 ceph-mon[32659]: pgmap v9795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:02 smithi132 ceph-mon[37580]: pgmap v9795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:02 smithi112 ceph-mon[37501]: pgmap v9795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:04 smithi107 ceph-mon[32659]: pgmap v9796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:04 smithi132 ceph-mon[37580]: pgmap v9796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:04 smithi112 ceph-mon[37501]: pgmap v9796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:06 smithi107 ceph-mon[32659]: pgmap v9797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17762.152937 secs 2024-07-26T17:47:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:06.303+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:47:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:06.303+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:47:06.305 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:06.303+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:47:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:06 smithi132 ceph-mon[37580]: pgmap v9797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17762.152937 secs 2024-07-26T17:47:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:06 smithi112 ceph-mon[37501]: pgmap v9797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17762.152937 secs 2024-07-26T17:47:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:08 smithi107 ceph-mon[32659]: pgmap v9798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:08 smithi132 ceph-mon[37580]: pgmap v9798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:08 smithi112 ceph-mon[37501]: pgmap v9798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:10 smithi107 ceph-mon[32659]: pgmap v9799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:10 smithi132 ceph-mon[37580]: pgmap v9799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:10.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:10 smithi112 ceph-mon[37501]: pgmap v9799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17767.153078 secs 2024-07-26T17:47:11.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17767.153078 secs 2024-07-26T17:47:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17767.153078 secs 2024-07-26T17:47:12.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:12 smithi132 ceph-mon[37580]: pgmap v9800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:12 smithi112 ceph-mon[37501]: pgmap v9800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:12 smithi107 ceph-mon[32659]: pgmap v9800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:14 smithi112 ceph-mon[37501]: pgmap v9801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:14 smithi107 ceph-mon[32659]: pgmap v9801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:14 smithi132 ceph-mon[37580]: pgmap v9801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:16 smithi112 ceph-mon[37501]: pgmap v9802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17772.153204 secs 2024-07-26T17:47:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:16 smithi107 ceph-mon[32659]: pgmap v9802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:16.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17772.153204 secs 2024-07-26T17:47:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:16 smithi132 ceph-mon[37580]: pgmap v9802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17772.153204 secs 2024-07-26T17:47:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:18 smithi112 ceph-mon[37501]: pgmap v9803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:18 smithi107 ceph-mon[32659]: pgmap v9803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:18 smithi132 ceph-mon[37580]: pgmap v9803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:20 smithi112 ceph-mon[37501]: pgmap v9804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:20 smithi107 ceph-mon[32659]: pgmap v9804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:20 smithi132 ceph-mon[37580]: pgmap v9804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:21.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:21.306+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:47:21.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:21.306+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:47:21.308 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:21.306+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:47:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17777.153284 secs 2024-07-26T17:47:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17777.153284 secs 2024-07-26T17:47:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17777.153284 secs 2024-07-26T17:47:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:22 smithi112 ceph-mon[37501]: pgmap v9805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:22 smithi107 ceph-mon[32659]: pgmap v9805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:22 smithi132 ceph-mon[37580]: pgmap v9805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:24.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:24 smithi112 ceph-mon[37501]: pgmap v9806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:24 smithi107 ceph-mon[32659]: pgmap v9806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:24 smithi132 ceph-mon[37580]: pgmap v9806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:26 smithi107 ceph-mon[32659]: pgmap v9807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17782.153388 secs 2024-07-26T17:47:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:26 smithi112 ceph-mon[37501]: pgmap v9807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17782.153388 secs 2024-07-26T17:47:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:26 smithi132 ceph-mon[37580]: pgmap v9807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17782.153388 secs 2024-07-26T17:47:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:28 smithi112 ceph-mon[37501]: pgmap v9808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:28 smithi107 ceph-mon[32659]: pgmap v9808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:28 smithi132 ceph-mon[37580]: pgmap v9808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:29 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:47:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:29 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:47:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:29 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:47:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:30 smithi112 ceph-mon[37501]: pgmap v9809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:47:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:47:30.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:30 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:47:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:30 smithi107 ceph-mon[32659]: pgmap v9809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:47:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:47:30.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:30 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:47:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:30 smithi132 ceph-mon[37580]: pgmap v9809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:47:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:47:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:30 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:47:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:31 smithi112 ceph-mon[37501]: pgmap v9810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:47:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:31 smithi112 ceph-mon[37501]: pgmap v9811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:47:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17787.153513 secs 2024-07-26T17:47:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:31 smithi107 ceph-mon[32659]: pgmap v9810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:47:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:31 smithi107 ceph-mon[32659]: pgmap v9811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:47:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17787.153513 secs 2024-07-26T17:47:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:31 smithi132 ceph-mon[37580]: pgmap v9810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:47:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:31 smithi132 ceph-mon[37580]: pgmap v9811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:47:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17787.153513 secs 2024-07-26T17:47:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:33 smithi112 ceph-mon[37501]: pgmap v9812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:33 smithi107 ceph-mon[32659]: pgmap v9812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:33 smithi132 ceph-mon[37580]: pgmap v9812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:35 smithi112 ceph-mon[37501]: pgmap v9813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:47:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:35 smithi107 ceph-mon[32659]: pgmap v9813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:47:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:35 smithi132 ceph-mon[37580]: pgmap v9813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:47:36.317 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:36.315+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:47:36.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:36.315+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:47:36.318 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:36.315+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:47:36.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17792.153659 secs 2024-07-26T17:47:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17792.153659 secs 2024-07-26T17:47:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17792.153659 secs 2024-07-26T17:47:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:37 smithi112 ceph-mon[37501]: pgmap v9814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:47:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:37 smithi107 ceph-mon[32659]: pgmap v9814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:47:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:37 smithi132 ceph-mon[37580]: pgmap v9814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:47:39.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:39 smithi112 ceph-mon[37501]: pgmap v9815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:47:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:39 smithi107 ceph-mon[32659]: pgmap v9815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:47:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:39 smithi132 ceph-mon[37580]: pgmap v9815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:47:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:41 smithi112 ceph-mon[37501]: pgmap v9816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:47:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17797.153777 secs 2024-07-26T17:47:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:41 smithi107 ceph-mon[32659]: pgmap v9816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:47:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17797.153777 secs 2024-07-26T17:47:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:41 smithi132 ceph-mon[37580]: pgmap v9816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:47:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17797.153777 secs 2024-07-26T17:47:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:43 smithi112 ceph-mon[37501]: pgmap v9817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:43 smithi107 ceph-mon[32659]: pgmap v9817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:43 smithi132 ceph-mon[37580]: pgmap v9817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:45 smithi112 ceph-mon[37501]: pgmap v9818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:47:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:45 smithi107 ceph-mon[32659]: pgmap v9818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:47:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:45 smithi132 ceph-mon[37580]: pgmap v9818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:47:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17802.153877 secs 2024-07-26T17:47:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17802.153877 secs 2024-07-26T17:47:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17802.153877 secs 2024-07-26T17:47:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:47 smithi112 ceph-mon[37501]: pgmap v9819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:47:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:47 smithi107 ceph-mon[32659]: pgmap v9819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:47:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:47 smithi132 ceph-mon[37580]: pgmap v9819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:47:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:49 smithi107 ceph-mon[32659]: pgmap v9820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:47:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:49 smithi132 ceph-mon[37580]: pgmap v9820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:47:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:49 smithi112 ceph-mon[37501]: pgmap v9820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:47:51.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:51.330+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:47:51.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:51.330+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:47:51.332 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:47:51.330+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:47:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:51 smithi107 ceph-mon[32659]: pgmap v9821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17807.153979 secs 2024-07-26T17:47:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:51 smithi132 ceph-mon[37580]: pgmap v9821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17807.153979 secs 2024-07-26T17:47:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:51 smithi112 ceph-mon[37501]: pgmap v9821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17807.153979 secs 2024-07-26T17:47:53.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:53 smithi107 ceph-mon[32659]: pgmap v9822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:53 smithi132 ceph-mon[37580]: pgmap v9822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:53 smithi112 ceph-mon[37501]: pgmap v9822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:47:55.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:55 smithi107 ceph-mon[32659]: pgmap v9823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:55 smithi132 ceph-mon[37580]: pgmap v9823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:55 smithi112 ceph-mon[37501]: pgmap v9823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17812.154103 secs 2024-07-26T17:47:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17812.154103 secs 2024-07-26T17:47:56.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17812.154103 secs 2024-07-26T17:47:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:57 smithi107 ceph-mon[32659]: pgmap v9824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:57 smithi132 ceph-mon[37580]: pgmap v9824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:57 smithi112 ceph-mon[37501]: pgmap v9824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:47:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:47:59 smithi107 ceph-mon[32659]: pgmap v9825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:47:59 smithi132 ceph-mon[37580]: pgmap v9825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:47:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:47:59 smithi112 ceph-mon[37501]: pgmap v9825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:01 smithi107 ceph-mon[32659]: pgmap v9826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17817.154179 secs 2024-07-26T17:48:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:01 smithi132 ceph-mon[37580]: pgmap v9826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17817.154179 secs 2024-07-26T17:48:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:01 smithi112 ceph-mon[37501]: pgmap v9826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17817.154179 secs 2024-07-26T17:48:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:03 smithi107 ceph-mon[32659]: pgmap v9827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:03 smithi132 ceph-mon[37580]: pgmap v9827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:03 smithi112 ceph-mon[37501]: pgmap v9827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:05 smithi107 ceph-mon[32659]: pgmap v9828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:05 smithi132 ceph-mon[37580]: pgmap v9828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:05.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:05 smithi112 ceph-mon[37501]: pgmap v9828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:06.335 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:06.334+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:48:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:06.334+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:48:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:06.334+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:48:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17822.154305 secs 2024-07-26T17:48:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17822.154305 secs 2024-07-26T17:48:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17822.154305 secs 2024-07-26T17:48:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:07 smithi107 ceph-mon[32659]: pgmap v9829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:07 smithi132 ceph-mon[37580]: pgmap v9829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:07 smithi112 ceph-mon[37501]: pgmap v9829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:09 smithi107 ceph-mon[32659]: pgmap v9830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:09 smithi132 ceph-mon[37580]: pgmap v9830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:09 smithi112 ceph-mon[37501]: pgmap v9830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:11 smithi107 ceph-mon[32659]: pgmap v9831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17827.154419 secs 2024-07-26T17:48:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:11 smithi132 ceph-mon[37580]: pgmap v9831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:11.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17827.154419 secs 2024-07-26T17:48:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:11 smithi112 ceph-mon[37501]: pgmap v9831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17827.154419 secs 2024-07-26T17:48:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:13 smithi107 ceph-mon[32659]: pgmap v9832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:13 smithi132 ceph-mon[37580]: pgmap v9832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:13 smithi112 ceph-mon[37501]: pgmap v9832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:14.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:14.725+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:48:14.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:14.725+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:48:14.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:14.726+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:48:14.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:14.726+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:48:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:15 smithi132 ceph-mon[37580]: pgmap v9833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:15 smithi112 ceph-mon[37501]: pgmap v9833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:15.726 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:15.725+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:48:15.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:15.725+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:48:15.727 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:15 smithi107 ceph-mon[32659]: pgmap v9833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:15.727 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:15.726+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:48:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:15.726+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:48:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17832.154516 secs 2024-07-26T17:48:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17832.154516 secs 2024-07-26T17:48:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17832.154516 secs 2024-07-26T17:48:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:17 smithi112 ceph-mon[37501]: pgmap v9834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:17 smithi107 ceph-mon[32659]: pgmap v9834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:17 smithi132 ceph-mon[37580]: pgmap v9834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:19 smithi112 ceph-mon[37501]: pgmap v9835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:19 smithi107 ceph-mon[32659]: pgmap v9835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:19 smithi132 ceph-mon[37580]: pgmap v9835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:21.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:21.349+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:48:21.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:21.349+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:48:21.351 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:21.349+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:48:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:21 smithi112 ceph-mon[37501]: pgmap v9836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17837.154659 secs 2024-07-26T17:48:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:21 smithi107 ceph-mon[32659]: pgmap v9836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17837.154659 secs 2024-07-26T17:48:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:21 smithi132 ceph-mon[37580]: pgmap v9836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17837.154659 secs 2024-07-26T17:48:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:23 smithi112 ceph-mon[37501]: pgmap v9837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:23 smithi107 ceph-mon[32659]: pgmap v9837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:23.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:23 smithi132 ceph-mon[37580]: pgmap v9837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:25 smithi112 ceph-mon[37501]: pgmap v9838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:25 smithi107 ceph-mon[32659]: pgmap v9838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:25 smithi132 ceph-mon[37580]: pgmap v9838: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17842.154798 secs 2024-07-26T17:48:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17842.154798 secs 2024-07-26T17:48:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17842.154798 secs 2024-07-26T17:48:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:27 smithi112 ceph-mon[37501]: pgmap v9839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:27 smithi107 ceph-mon[32659]: pgmap v9839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:27 smithi132 ceph-mon[37580]: pgmap v9839: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:29 smithi112 ceph-mon[37501]: pgmap v9840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:29 smithi107 ceph-mon[32659]: pgmap v9840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:29 smithi132 ceph-mon[37580]: pgmap v9840: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:30 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:48:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:30 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:48:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:30 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:48:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:31 smithi112 ceph-mon[37501]: pgmap v9841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17847.154886 secs 2024-07-26T17:48:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:48:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:48:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:31 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:48:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:31 smithi107 ceph-mon[32659]: pgmap v9841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17847.154886 secs 2024-07-26T17:48:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:48:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:48:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:31 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:48:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:31 smithi132 ceph-mon[37580]: pgmap v9841: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17847.154886 secs 2024-07-26T17:48:31.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:48:31.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:48:31.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:31 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:48:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:32 smithi112 ceph-mon[37501]: pgmap v9842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:48:32.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:32 smithi112 ceph-mon[37501]: pgmap v9843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:48:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:32 smithi107 ceph-mon[32659]: pgmap v9842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:48:32.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:32 smithi107 ceph-mon[32659]: pgmap v9843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:48:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:32 smithi132 ceph-mon[37580]: pgmap v9842: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:48:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:32 smithi132 ceph-mon[37580]: pgmap v9843: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:48:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:34 smithi112 ceph-mon[37501]: pgmap v9844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:48:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:34 smithi107 ceph-mon[32659]: pgmap v9844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:48:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:34 smithi132 ceph-mon[37580]: pgmap v9844: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:48:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:36.363+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:48:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:36.363+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:48:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:36.363+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:48:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:36 smithi112 ceph-mon[37501]: pgmap v9845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:48:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17852.155013 secs 2024-07-26T17:48:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:36 smithi107 ceph-mon[32659]: pgmap v9845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:48:36.910 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17852.155013 secs 2024-07-26T17:48:36.910 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:36 smithi132 ceph-mon[37580]: pgmap v9845: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:48:36.911 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17852.155013 secs 2024-07-26T17:48:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:38 smithi112 ceph-mon[37501]: pgmap v9846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:38 smithi107 ceph-mon[32659]: pgmap v9846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:38 smithi132 ceph-mon[37580]: pgmap v9846: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:40 smithi112 ceph-mon[37501]: pgmap v9847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:48:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:40 smithi107 ceph-mon[32659]: pgmap v9847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:48:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:40 smithi132 ceph-mon[37580]: pgmap v9847: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:48:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17857.155142 secs 2024-07-26T17:48:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17857.155142 secs 2024-07-26T17:48:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17857.155142 secs 2024-07-26T17:48:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:42 smithi112 ceph-mon[37501]: pgmap v9848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:48:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:42 smithi107 ceph-mon[32659]: pgmap v9848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:48:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:42 smithi132 ceph-mon[37580]: pgmap v9848: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:48:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:44 smithi112 ceph-mon[37501]: pgmap v9849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:48:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:44 smithi107 ceph-mon[32659]: pgmap v9849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:48:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:44 smithi132 ceph-mon[37580]: pgmap v9849: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:48:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:46 smithi112 ceph-mon[37501]: pgmap v9850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17862.155287 secs 2024-07-26T17:48:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:46 smithi107 ceph-mon[32659]: pgmap v9850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17862.155287 secs 2024-07-26T17:48:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:46 smithi132 ceph-mon[37580]: pgmap v9850: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17862.155287 secs 2024-07-26T17:48:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:48 smithi112 ceph-mon[37501]: pgmap v9851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:48 smithi107 ceph-mon[32659]: pgmap v9851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:48 smithi132 ceph-mon[37580]: pgmap v9851: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:50 smithi112 ceph-mon[37501]: pgmap v9852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:50 smithi107 ceph-mon[32659]: pgmap v9852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:50 smithi132 ceph-mon[37580]: pgmap v9852: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:48:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:51.370+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:48:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:51.370+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:48:51.373 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:48:51.370+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:48:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17867.155378 secs 2024-07-26T17:48:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17867.155378 secs 2024-07-26T17:48:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17867.155378 secs 2024-07-26T17:48:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:52 smithi107 ceph-mon[32659]: pgmap v9853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:52 smithi132 ceph-mon[37580]: pgmap v9853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:52 smithi112 ceph-mon[37501]: pgmap v9853: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:54.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:54 smithi107 ceph-mon[32659]: pgmap v9854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:48:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:54 smithi132 ceph-mon[37580]: pgmap v9854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:48:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:54 smithi112 ceph-mon[37501]: pgmap v9854: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:48:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:56 smithi107 ceph-mon[32659]: pgmap v9855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17872.155484 secs 2024-07-26T17:48:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:56 smithi132 ceph-mon[37580]: pgmap v9855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17872.155484 secs 2024-07-26T17:48:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:56 smithi112 ceph-mon[37501]: pgmap v9855: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:48:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17872.155484 secs 2024-07-26T17:48:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:48:58 smithi107 ceph-mon[32659]: pgmap v9856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:48:58 smithi132 ceph-mon[37580]: pgmap v9856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:48:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:48:58 smithi112 ceph-mon[37501]: pgmap v9856: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:00 smithi107 ceph-mon[32659]: pgmap v9857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:00 smithi132 ceph-mon[37580]: pgmap v9857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:00 smithi112 ceph-mon[37501]: pgmap v9857: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17877.155612 secs 2024-07-26T17:49:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17877.155612 secs 2024-07-26T17:49:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17877.155612 secs 2024-07-26T17:49:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:02 smithi107 ceph-mon[32659]: pgmap v9858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:02 smithi132 ceph-mon[37580]: pgmap v9858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:02 smithi112 ceph-mon[37501]: pgmap v9858: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:04 smithi107 ceph-mon[32659]: pgmap v9859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:49:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:04 smithi132 ceph-mon[37580]: pgmap v9859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:49:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:04 smithi112 ceph-mon[37501]: pgmap v9859: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:49:06.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:06.386+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:49:06.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:06.386+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:49:06.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:06.386+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:49:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:06 smithi107 ceph-mon[32659]: pgmap v9860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17882.155754 secs 2024-07-26T17:49:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:06 smithi132 ceph-mon[37580]: pgmap v9860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17882.155754 secs 2024-07-26T17:49:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:06 smithi112 ceph-mon[37501]: pgmap v9860: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17882.155754 secs 2024-07-26T17:49:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:08 smithi107 ceph-mon[32659]: pgmap v9861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:08 smithi132 ceph-mon[37580]: pgmap v9861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:08 smithi112 ceph-mon[37501]: pgmap v9861: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:10.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:10 smithi107 ceph-mon[32659]: pgmap v9862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:10 smithi132 ceph-mon[37580]: pgmap v9862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:10.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:10 smithi112 ceph-mon[37501]: pgmap v9862: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17887.155874 secs 2024-07-26T17:49:11.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17887.155874 secs 2024-07-26T17:49:11.852 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17887.155874 secs 2024-07-26T17:49:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:12 smithi107 ceph-mon[32659]: pgmap v9863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:12 smithi132 ceph-mon[37580]: pgmap v9863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:12 smithi112 ceph-mon[37501]: pgmap v9863: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:14 smithi107 ceph-mon[32659]: pgmap v9864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:49:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:14 smithi132 ceph-mon[37580]: pgmap v9864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:49:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:14 smithi112 ceph-mon[37501]: pgmap v9864: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:49:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:16 smithi107 ceph-mon[32659]: pgmap v9865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17892.156013 secs 2024-07-26T17:49:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:16 smithi132 ceph-mon[37580]: pgmap v9865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17892.156013 secs 2024-07-26T17:49:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:16 smithi112 ceph-mon[37501]: pgmap v9865: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17892.156013 secs 2024-07-26T17:49:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:18 smithi107 ceph-mon[32659]: pgmap v9866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:18 smithi132 ceph-mon[37580]: pgmap v9866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:18.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:18 smithi112 ceph-mon[37501]: pgmap v9866: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:20 smithi132 ceph-mon[37580]: pgmap v9867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:20 smithi112 ceph-mon[37501]: pgmap v9867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:20 smithi107 ceph-mon[32659]: pgmap v9867: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:21.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:21.388+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:49:21.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:21.388+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:49:21.390 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:21.388+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:49:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17897.156142 secs 2024-07-26T17:49:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17897.156142 secs 2024-07-26T17:49:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17897.156142 secs 2024-07-26T17:49:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:22 smithi132 ceph-mon[37580]: pgmap v9868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:22 smithi112 ceph-mon[37501]: pgmap v9868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:22 smithi107 ceph-mon[32659]: pgmap v9868: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:24 smithi112 ceph-mon[37501]: pgmap v9869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:49:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:24 smithi107 ceph-mon[32659]: pgmap v9869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:49:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:24 smithi132 ceph-mon[37580]: pgmap v9869: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T17:49:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:26 smithi112 ceph-mon[37501]: pgmap v9870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17902.156210 secs 2024-07-26T17:49:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:26 smithi107 ceph-mon[32659]: pgmap v9870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17902.156210 secs 2024-07-26T17:49:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:26 smithi132 ceph-mon[37580]: pgmap v9870: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17902.156210 secs 2024-07-26T17:49:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:28 smithi112 ceph-mon[37501]: pgmap v9871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:28 smithi107 ceph-mon[32659]: pgmap v9871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:28 smithi132 ceph-mon[37580]: pgmap v9871: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:30.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:30 smithi112 ceph-mon[37501]: pgmap v9872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:30.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:30 smithi107 ceph-mon[32659]: pgmap v9872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:30 smithi132 ceph-mon[37580]: pgmap v9872: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17907.156338 secs 2024-07-26T17:49:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:31 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:49:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17907.156338 secs 2024-07-26T17:49:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:31 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:49:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17907.156338 secs 2024-07-26T17:49:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:31 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:49:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:32 smithi112 ceph-mon[37501]: pgmap v9873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:49:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:32 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:49:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:32 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:49:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:32 smithi107 ceph-mon[32659]: pgmap v9873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:49:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:32 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:49:33.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:32 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:49:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:32 smithi132 ceph-mon[37580]: pgmap v9873: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:49:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:32 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:49:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:32 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:49:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:33 smithi112 ceph-mon[37501]: pgmap v9874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:49:33.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:33 smithi112 ceph-mon[37501]: pgmap v9875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:49:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:33 smithi107 ceph-mon[32659]: pgmap v9874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:49:34.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:33 smithi107 ceph-mon[32659]: pgmap v9875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:49:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:33 smithi132 ceph-mon[37580]: pgmap v9874: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T17:49:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:33 smithi132 ceph-mon[37580]: pgmap v9875: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:49:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:35 smithi112 ceph-mon[37501]: pgmap v9876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:35 smithi107 ceph-mon[32659]: pgmap v9876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:35 smithi132 ceph-mon[37580]: pgmap v9876: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:36.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:36.403+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:49:36.405 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:36.403+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:49:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:36.403+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:49:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17912.156459 secs 2024-07-26T17:49:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17912.156459 secs 2024-07-26T17:49:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17912.156459 secs 2024-07-26T17:49:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:37 smithi112 ceph-mon[37501]: pgmap v9877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:37 smithi107 ceph-mon[32659]: pgmap v9877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:37 smithi132 ceph-mon[37580]: pgmap v9877: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:39 smithi112 ceph-mon[37501]: pgmap v9878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:39 smithi107 ceph-mon[32659]: pgmap v9878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:39 smithi132 ceph-mon[37580]: pgmap v9878: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:41 smithi112 ceph-mon[37501]: pgmap v9879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17917.156610 secs 2024-07-26T17:49:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:41 smithi107 ceph-mon[32659]: pgmap v9879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17917.156610 secs 2024-07-26T17:49:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:41 smithi132 ceph-mon[37580]: pgmap v9879: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:49:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17917.156610 secs 2024-07-26T17:49:43.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:43 smithi112 ceph-mon[37501]: pgmap v9880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:49:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:43 smithi107 ceph-mon[32659]: pgmap v9880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:49:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:43 smithi132 ceph-mon[37580]: pgmap v9880: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:49:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:45 smithi112 ceph-mon[37501]: pgmap v9881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:49:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:45 smithi107 ceph-mon[32659]: pgmap v9881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:49:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:45 smithi132 ceph-mon[37580]: pgmap v9881: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:49:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17922.156748 secs 2024-07-26T17:49:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17922.156748 secs 2024-07-26T17:49:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17922.156748 secs 2024-07-26T17:49:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:47 smithi112 ceph-mon[37501]: pgmap v9882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:47 smithi107 ceph-mon[32659]: pgmap v9882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:47 smithi132 ceph-mon[37580]: pgmap v9882: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:49:49.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:49 smithi112 ceph-mon[37501]: pgmap v9883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:49 smithi107 ceph-mon[32659]: pgmap v9883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:49 smithi132 ceph-mon[37580]: pgmap v9883: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:49:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:51.406+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:49:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:51.406+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:49:51.409 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:49:51.406+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:49:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:51 smithi112 ceph-mon[37501]: pgmap v9884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17927.156867 secs 2024-07-26T17:49:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:51 smithi107 ceph-mon[32659]: pgmap v9884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17927.156867 secs 2024-07-26T17:49:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:51 smithi132 ceph-mon[37580]: pgmap v9884: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:49:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17927.156867 secs 2024-07-26T17:49:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:53 smithi112 ceph-mon[37501]: pgmap v9885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:49:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:53 smithi107 ceph-mon[32659]: pgmap v9885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:49:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:53 smithi132 ceph-mon[37580]: pgmap v9885: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:49:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:55 smithi112 ceph-mon[37501]: pgmap v9886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:49:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:55 smithi107 ceph-mon[32659]: pgmap v9886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:49:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:55 smithi132 ceph-mon[37580]: pgmap v9886: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:49:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17932.156988 secs 2024-07-26T17:49:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:56 smithi107 ceph-mon[32659]: pgmap v9887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:49:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17932.156988 secs 2024-07-26T17:49:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:56 smithi132 ceph-mon[37580]: pgmap v9887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:49:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17932.156988 secs 2024-07-26T17:49:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:56 smithi112 ceph-mon[37501]: pgmap v9887: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:49:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:49:59 smithi107 ceph-mon[32659]: pgmap v9888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:49:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:49:59 smithi132 ceph-mon[37580]: pgmap v9888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:49:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:49:59 smithi112 ceph-mon[37501]: pgmap v9888: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:50:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:50:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:50:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T17:50:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:01 smithi107 ceph-mon[32659]: pgmap v9889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:50:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17937.157131 secs 2024-07-26T17:50:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:01 smithi132 ceph-mon[37580]: pgmap v9889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:50:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17937.157131 secs 2024-07-26T17:50:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:01 smithi112 ceph-mon[37501]: pgmap v9889: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:50:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17937.157131 secs 2024-07-26T17:50:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:03 smithi107 ceph-mon[32659]: pgmap v9890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:50:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:03 smithi132 ceph-mon[37580]: pgmap v9890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:50:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:03 smithi112 ceph-mon[37501]: pgmap v9890: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:50:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:05 smithi107 ceph-mon[32659]: pgmap v9891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:05 smithi132 ceph-mon[37580]: pgmap v9891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:05 smithi112 ceph-mon[37501]: pgmap v9891: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:06.422+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:50:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:06.423+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:50:06.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:06.423+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:50:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17942.157248 secs 2024-07-26T17:50:06.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17942.157248 secs 2024-07-26T17:50:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17942.157248 secs 2024-07-26T17:50:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:07 smithi132 ceph-mon[37580]: pgmap v9892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:07 smithi112 ceph-mon[37501]: pgmap v9892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:07 smithi107 ceph-mon[32659]: pgmap v9892: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:09 smithi112 ceph-mon[37501]: pgmap v9893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:10.017 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:09 smithi132 ceph-mon[37580]: pgmap v9893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:09 smithi107 ceph-mon[32659]: pgmap v9893: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:11 smithi112 ceph-mon[37501]: pgmap v9894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17947.157312 secs 2024-07-26T17:50:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:11 smithi107 ceph-mon[32659]: pgmap v9894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17947.157312 secs 2024-07-26T17:50:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:11 smithi132 ceph-mon[37580]: pgmap v9894: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17947.157312 secs 2024-07-26T17:50:13.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:13 smithi112 ceph-mon[37501]: pgmap v9895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:13 smithi107 ceph-mon[32659]: pgmap v9895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:14.173 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:13 smithi132 ceph-mon[37580]: pgmap v9895: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:15.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:15 smithi132 ceph-mon[37580]: pgmap v9896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:15 smithi112 ceph-mon[37501]: pgmap v9896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:15 smithi107 ceph-mon[32659]: pgmap v9896: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:16.878 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17952.157407 secs 2024-07-26T17:50:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17952.157407 secs 2024-07-26T17:50:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17952.157407 secs 2024-07-26T17:50:17.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:17 smithi112 ceph-mon[37501]: pgmap v9897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:17 smithi107 ceph-mon[32659]: pgmap v9897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:17 smithi132 ceph-mon[37580]: pgmap v9897: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:19 smithi112 ceph-mon[37501]: pgmap v9898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:19 smithi107 ceph-mon[32659]: pgmap v9898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:19 smithi132 ceph-mon[37580]: pgmap v9898: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:21.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:21.438+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:50:21.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:21.438+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:50:21.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:21.438+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:50:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:21 smithi107 ceph-mon[32659]: pgmap v9899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17957.157524 secs 2024-07-26T17:50:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:21 smithi112 ceph-mon[37501]: pgmap v9899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17957.157524 secs 2024-07-26T17:50:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:21 smithi132 ceph-mon[37580]: pgmap v9899: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17957.157524 secs 2024-07-26T17:50:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:23 smithi112 ceph-mon[37501]: pgmap v9900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:23 smithi107 ceph-mon[32659]: pgmap v9900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:23 smithi132 ceph-mon[37580]: pgmap v9900: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:25 smithi112 ceph-mon[37501]: pgmap v9901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:25 smithi107 ceph-mon[32659]: pgmap v9901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:25 smithi132 ceph-mon[37580]: pgmap v9901: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17962.157669 secs 2024-07-26T17:50:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17962.157669 secs 2024-07-26T17:50:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17962.157669 secs 2024-07-26T17:50:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:27 smithi112 ceph-mon[37501]: pgmap v9902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:27 smithi107 ceph-mon[32659]: pgmap v9902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:27 smithi132 ceph-mon[37580]: pgmap v9902: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:29 smithi112 ceph-mon[37501]: pgmap v9903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:29 smithi107 ceph-mon[32659]: pgmap v9903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:29 smithi132 ceph-mon[37580]: pgmap v9903: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:31 smithi112 ceph-mon[37501]: pgmap v9904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17967.157763 secs 2024-07-26T17:50:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:31 smithi107 ceph-mon[32659]: pgmap v9904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17967.157763 secs 2024-07-26T17:50:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:31 smithi132 ceph-mon[37580]: pgmap v9904: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17967.157763 secs 2024-07-26T17:50:33.758 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:33 smithi112 ceph-mon[37501]: pgmap v9905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:33.759 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:33 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:50:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:33 smithi107 ceph-mon[32659]: pgmap v9905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:33 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:50:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:33 smithi132 ceph-mon[37580]: pgmap v9905: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:33 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:50:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:35 smithi107 ceph-mon[32659]: pgmap v9906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:35.910 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:35 smithi112 ceph-mon[37501]: pgmap v9906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:35 smithi132 ceph-mon[37580]: pgmap v9906: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:50:36.455 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:36.453+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:50:36.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:36.453+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:50:36.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:36.453+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:50:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17972.157887 secs 2024-07-26T17:50:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:36 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:36.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17972.157887 secs 2024-07-26T17:50:36.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:36.972 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:36 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17972.157887 secs 2024-07-26T17:50:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:36 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:37.911 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:37 smithi107 ceph-mon[32659]: pgmap v9907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:37 smithi112 ceph-mon[37501]: pgmap v9907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:37 smithi132 ceph-mon[37580]: pgmap v9907: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: pgmap v9908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:39.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:39.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:39.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:39.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:39.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:39.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:39.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:39.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:50:39.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:50:39.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:39 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: pgmap v9908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:50:40.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:50:40.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:39 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: pgmap v9908: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:40.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:40.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:50:40.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:50:40.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:50:40.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:39 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:50:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:40 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:50:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:40 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:50:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:40 smithi112 ceph-mon[37501]: pgmap v9909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:50:40.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:40 smithi112 ceph-mon[37501]: pgmap v9910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:50:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:40 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:50:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:40 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:50:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:40 smithi107 ceph-mon[32659]: pgmap v9909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:50:41.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:40 smithi107 ceph-mon[32659]: pgmap v9910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:50:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:40 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T17:50:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:40 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T17:50:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:40 smithi132 ceph-mon[37580]: pgmap v9909: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T17:50:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:40 smithi132 ceph-mon[37580]: pgmap v9910: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T17:50:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17977.158013 secs 2024-07-26T17:50:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17977.158013 secs 2024-07-26T17:50:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17977.158013 secs 2024-07-26T17:50:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:42 smithi112 ceph-mon[37501]: pgmap v9911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T17:50:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:42 smithi107 ceph-mon[32659]: pgmap v9911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T17:50:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:42 smithi132 ceph-mon[37580]: pgmap v9911: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2024-07-26T17:50:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:44 smithi112 ceph-mon[37501]: pgmap v9912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T17:50:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:44 smithi107 ceph-mon[32659]: pgmap v9912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T17:50:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:44 smithi132 ceph-mon[37580]: pgmap v9912: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T17:50:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:46 smithi112 ceph-mon[37501]: pgmap v9913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T17:50:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17982.158137 secs 2024-07-26T17:50:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:46 smithi107 ceph-mon[32659]: pgmap v9913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T17:50:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17982.158137 secs 2024-07-26T17:50:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:46 smithi132 ceph-mon[37580]: pgmap v9913: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T17:50:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17982.158137 secs 2024-07-26T17:50:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:48 smithi112 ceph-mon[37501]: pgmap v9914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T17:50:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:48 smithi107 ceph-mon[32659]: pgmap v9914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T17:50:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:48 smithi132 ceph-mon[37580]: pgmap v9914: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T17:50:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:50 smithi112 ceph-mon[37501]: pgmap v9915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:50:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:50 smithi107 ceph-mon[32659]: pgmap v9915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:50:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:50 smithi132 ceph-mon[37580]: pgmap v9915: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:50:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:51.468+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:50:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:51.469+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:50:51.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:50:51.469+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:50:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17987.158198 secs 2024-07-26T17:50:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17987.158198 secs 2024-07-26T17:50:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17987.158198 secs 2024-07-26T17:50:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:52 smithi107 ceph-mon[32659]: pgmap v9916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:52 smithi132 ceph-mon[37580]: pgmap v9916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:52 smithi112 ceph-mon[37501]: pgmap v9916: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:50:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:53 smithi107 ceph-mon[32659]: pgmap v9917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:53 smithi132 ceph-mon[37580]: pgmap v9917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:53 smithi112 ceph-mon[37501]: pgmap v9917: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:50:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:55 smithi112 ceph-mon[37501]: pgmap v9918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:56.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17992.158316 secs 2024-07-26T17:50:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:55 smithi107 ceph-mon[32659]: pgmap v9918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17992.158316 secs 2024-07-26T17:50:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:55 smithi132 ceph-mon[37580]: pgmap v9918: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17992.158316 secs 2024-07-26T17:50:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:57 smithi112 ceph-mon[37501]: pgmap v9919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:57 smithi107 ceph-mon[32659]: pgmap v9919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:50:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:57 smithi132 ceph-mon[37580]: pgmap v9919: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:50:59 smithi112 ceph-mon[37501]: pgmap v9920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:50:59 smithi107 ceph-mon[32659]: pgmap v9920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:50:59 smithi132 ceph-mon[37580]: pgmap v9920: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 17997.158415 secs 2024-07-26T17:51:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 17997.158415 secs 2024-07-26T17:51:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 17997.158415 secs 2024-07-26T17:51:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:01 smithi112 ceph-mon[37501]: pgmap v9921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:01 smithi107 ceph-mon[32659]: pgmap v9921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:01 smithi132 ceph-mon[37580]: pgmap v9921: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:04.154 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:03 smithi107 ceph-mon[32659]: pgmap v9922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:03 smithi112 ceph-mon[37501]: pgmap v9922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:03 smithi132 ceph-mon[37580]: pgmap v9922: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:05 smithi112 ceph-mon[37501]: pgmap v9923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:05 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18002.158535 secs 2024-07-26T17:51:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:05 smithi107 ceph-mon[32659]: pgmap v9923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:05 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18002.158535 secs 2024-07-26T17:51:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:05 smithi132 ceph-mon[37580]: pgmap v9923: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:05 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18002.158535 secs 2024-07-26T17:51:06.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:06.485+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:51:06.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:06.485+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:51:06.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:06.485+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:51:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:07 smithi112 ceph-mon[37501]: pgmap v9924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:07 smithi107 ceph-mon[32659]: pgmap v9924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:07 smithi132 ceph-mon[37580]: pgmap v9924: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:09 smithi112 ceph-mon[37501]: pgmap v9925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:09 smithi107 ceph-mon[32659]: pgmap v9925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:09 smithi132 ceph-mon[37580]: pgmap v9925: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:11.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18007.158617 secs 2024-07-26T17:51:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18007.158617 secs 2024-07-26T17:51:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18007.158617 secs 2024-07-26T17:51:11.698 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:11.696+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T17:51:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:11.696+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T17:51:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:11.696+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f8067040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T17:51:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:11.696+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f8067040 comp rx=0 tx=0).stop 2024-07-26T17:51:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:11.696+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.107:6800/3577384363, fault. 2024-07-26T17:51:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:11.696+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 msgr2=0x7ff0ec0c4360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T17:51:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:11.696+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T17:51:11.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:11.696+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T17:51:11.700 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:11.698+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T17:51:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:11.699+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8014040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T17:51:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:11 smithi107 ceph-mon[32659]: pgmap v9926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:11 smithi112 ceph-mon[37501]: pgmap v9926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:11 smithi132 ceph-mon[37580]: pgmap v9926: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:13 smithi112 ceph-mon[37501]: pgmap v9927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:13 smithi107 ceph-mon[32659]: pgmap v9927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:13 smithi132 ceph-mon[37580]: pgmap v9927: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:15.757+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:51:15.758 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:15.757+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:51:15.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:15.757+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:51:15.759 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:15.757+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:51:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:15 smithi107 ceph-mon[32659]: pgmap v9928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:16.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18012.158754 secs 2024-07-26T17:51:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:15 smithi112 ceph-mon[37501]: pgmap v9928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:16.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18012.158754 secs 2024-07-26T17:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:15 smithi132 ceph-mon[37580]: pgmap v9928: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18012.158754 secs 2024-07-26T17:51:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:17 smithi112 ceph-mon[37501]: pgmap v9929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:17 smithi107 ceph-mon[32659]: pgmap v9929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:17 smithi132 ceph-mon[37580]: pgmap v9929: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:19 smithi107 ceph-mon[32659]: pgmap v9930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:19 smithi132 ceph-mon[37580]: pgmap v9930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:19 smithi112 ceph-mon[37501]: pgmap v9930: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:20 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18017.158876 secs 2024-07-26T17:51:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:20 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18017.158876 secs 2024-07-26T17:51:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:20 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18017.158876 secs 2024-07-26T17:51:21.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:21.492+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:51:21.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:21.493+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:51:21.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:21.493+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:51:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:21 smithi107 ceph-mon[32659]: pgmap v9931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:21 smithi132 ceph-mon[37580]: pgmap v9931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:21 smithi112 ceph-mon[37501]: pgmap v9931: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:24.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:23 smithi107 ceph-mon[32659]: pgmap v9932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:23 smithi132 ceph-mon[37580]: pgmap v9932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:23 smithi112 ceph-mon[37501]: pgmap v9932: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:25 smithi107 ceph-mon[32659]: pgmap v9933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:25 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18022.158997 secs 2024-07-26T17:51:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:25 smithi132 ceph-mon[37580]: pgmap v9933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:25 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18022.158997 secs 2024-07-26T17:51:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:25 smithi112 ceph-mon[37501]: pgmap v9933: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:25 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18022.158997 secs 2024-07-26T17:51:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:27 smithi107 ceph-mon[32659]: pgmap v9934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:27 smithi132 ceph-mon[37580]: pgmap v9934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:27 smithi112 ceph-mon[37501]: pgmap v9934: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:29 smithi107 ceph-mon[32659]: pgmap v9935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:29 smithi132 ceph-mon[37580]: pgmap v9935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:29 smithi112 ceph-mon[37501]: pgmap v9935: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18027.159116 secs 2024-07-26T17:51:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18027.159116 secs 2024-07-26T17:51:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18027.159116 secs 2024-07-26T17:51:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:32 smithi107 ceph-mon[32659]: pgmap v9936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:32 smithi132 ceph-mon[37580]: pgmap v9936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:32 smithi112 ceph-mon[37501]: pgmap v9936: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:34 smithi107 ceph-mon[32659]: pgmap v9937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:34 smithi132 ceph-mon[37580]: pgmap v9937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:34.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:34 smithi112 ceph-mon[37501]: pgmap v9937: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:36 smithi107 ceph-mon[32659]: pgmap v9938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:36.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18032.159136 secs 2024-07-26T17:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:36 smithi132 ceph-mon[37580]: pgmap v9938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18032.159136 secs 2024-07-26T17:51:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:36 smithi112 ceph-mon[37501]: pgmap v9938: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18032.159136 secs 2024-07-26T17:51:36.509 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:36.507+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:51:36.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:36.507+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:51:36.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:36.507+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:51:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:38 smithi107 ceph-mon[32659]: pgmap v9939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:51:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:38 smithi132 ceph-mon[37580]: pgmap v9939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:51:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:38 smithi112 ceph-mon[37501]: pgmap v9939: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:51:40.232 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: pgmap v9940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:40.233 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:51:40.233 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.233 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.233 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.233 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.233 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.234 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.234 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.234 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.234 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:51:40.234 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:40 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:51:40.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: pgmap v9940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:40.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:51:40.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:51:40.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:40 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:51:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: pgmap v9940: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:51:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:40.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T17:51:40.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:40 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T17:51:41.279 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18037.159272 secs 2024-07-26T17:51:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18037.159272 secs 2024-07-26T17:51:41.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18037.159272 secs 2024-07-26T17:51:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:42 smithi107 ceph-mon[32659]: pgmap v9941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:42 smithi132 ceph-mon[37580]: pgmap v9941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:42.366 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:42 smithi112 ceph-mon[37501]: pgmap v9941: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:51:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: pgmap v9942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:51:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:51:44.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:51:44.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:51:44.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:51:44.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:51:44.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: pgmap v9943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:51:44.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: pgmap v9944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:51:44.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:43 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:51:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: pgmap v9942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:44.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:51:44.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:51:44.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:51:44.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:51:44.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:51:44.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:51:44.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: pgmap v9943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:51:44.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: pgmap v9944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:51:44.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:43 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:51:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: pgmap v9942: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:51:44.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:51:44.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:51:44.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T17:51:44.314 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T17:51:44.314 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T17:51:44.314 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:51:44.315 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:51:44.315 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: pgmap v9943: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:51:44.315 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: pgmap v9944: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:51:44.315 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:43 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:51:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:45 smithi107 ceph-mon[32659]: pgmap v9945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T17:51:46.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:45 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18042.159401 secs 2024-07-26T17:51:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:45 smithi132 ceph-mon[37580]: pgmap v9945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T17:51:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:45 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18042.159401 secs 2024-07-26T17:51:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:45 smithi112 ceph-mon[37501]: pgmap v9945: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T17:51:46.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:45 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18042.159401 secs 2024-07-26T17:51:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:47 smithi107 ceph-mon[32659]: pgmap v9946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:51:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:47 smithi132 ceph-mon[37580]: pgmap v9946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:51:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:47 smithi112 ceph-mon[37501]: pgmap v9946: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T17:51:50.234 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:49 smithi107 ceph-mon[32659]: pgmap v9947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T17:51:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:49 smithi132 ceph-mon[37580]: pgmap v9947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T17:51:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:49 smithi112 ceph-mon[37501]: pgmap v9947: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T17:51:51.269 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:50 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18047.159523 secs 2024-07-26T17:51:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:50 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18047.159523 secs 2024-07-26T17:51:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:50 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18047.159523 secs 2024-07-26T17:51:51.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:51.523+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:51:51.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:51.523+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:51:51.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:51:51.523+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:51:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:51 smithi107 ceph-mon[32659]: pgmap v9948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T17:51:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:51 smithi132 ceph-mon[37580]: pgmap v9948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T17:51:52.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:51 smithi112 ceph-mon[37501]: pgmap v9948: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T17:51:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:53 smithi107 ceph-mon[32659]: pgmap v9949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:51:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:53 smithi132 ceph-mon[37580]: pgmap v9949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:51:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:53 smithi112 ceph-mon[37501]: pgmap v9949: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:51:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:55 smithi107 ceph-mon[32659]: pgmap v9950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:56.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:55 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18052.159603 secs 2024-07-26T17:51:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:55 smithi132 ceph-mon[37580]: pgmap v9950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:55 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18052.159603 secs 2024-07-26T17:51:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:55 smithi112 ceph-mon[37501]: pgmap v9950: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:56.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:55 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18052.159603 secs 2024-07-26T17:51:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:57 smithi107 ceph-mon[32659]: pgmap v9951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:57 smithi132 ceph-mon[37580]: pgmap v9951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:51:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:57 smithi112 ceph-mon[37501]: pgmap v9951: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:51:59 smithi107 ceph-mon[32659]: pgmap v9952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:51:59 smithi132 ceph-mon[37580]: pgmap v9952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:51:59 smithi112 ceph-mon[37501]: pgmap v9952: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18057.159729 secs 2024-07-26T17:52:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18057.159729 secs 2024-07-26T17:52:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18057.159729 secs 2024-07-26T17:52:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:02 smithi107 ceph-mon[32659]: pgmap v9953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:02 smithi132 ceph-mon[37580]: pgmap v9953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:02 smithi112 ceph-mon[37501]: pgmap v9953: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:04 smithi107 ceph-mon[32659]: pgmap v9954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:04 smithi132 ceph-mon[37580]: pgmap v9954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:04 smithi112 ceph-mon[37501]: pgmap v9954: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:06 smithi107 ceph-mon[32659]: pgmap v9955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18062.159867 secs 2024-07-26T17:52:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:06 smithi132 ceph-mon[37580]: pgmap v9955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:06.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18062.159867 secs 2024-07-26T17:52:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:06 smithi112 ceph-mon[37501]: pgmap v9955: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:06.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18062.159867 secs 2024-07-26T17:52:06.541 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:06.539+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:52:06.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:06.539+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:52:06.542 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:06.539+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:52:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:08 smithi107 ceph-mon[32659]: pgmap v9956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:08 smithi132 ceph-mon[37580]: pgmap v9956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:08 smithi112 ceph-mon[37501]: pgmap v9956: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:10 smithi107 ceph-mon[32659]: pgmap v9957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:10 smithi132 ceph-mon[37580]: pgmap v9957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:10 smithi112 ceph-mon[37501]: pgmap v9957: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18067.159995 secs 2024-07-26T17:52:11.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18067.159995 secs 2024-07-26T17:52:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18067.159995 secs 2024-07-26T17:52:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:12 smithi132 ceph-mon[37580]: pgmap v9958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:12 smithi112 ceph-mon[37501]: pgmap v9958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:12 smithi107 ceph-mon[32659]: pgmap v9958: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:14 smithi132 ceph-mon[37580]: pgmap v9959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:14 smithi112 ceph-mon[37501]: pgmap v9959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:14 smithi107 ceph-mon[32659]: pgmap v9959: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:16 smithi132 ceph-mon[37580]: pgmap v9960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18072.160092 secs 2024-07-26T17:52:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:16 smithi112 ceph-mon[37501]: pgmap v9960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18072.160092 secs 2024-07-26T17:52:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:16 smithi107 ceph-mon[32659]: pgmap v9960: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18072.160092 secs 2024-07-26T17:52:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:18 smithi112 ceph-mon[37501]: pgmap v9961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:18 smithi107 ceph-mon[32659]: pgmap v9961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:18 smithi132 ceph-mon[37580]: pgmap v9961: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:20 smithi112 ceph-mon[37501]: pgmap v9962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:20 smithi107 ceph-mon[32659]: pgmap v9962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:20 smithi132 ceph-mon[37580]: pgmap v9962: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18077.160162 secs 2024-07-26T17:52:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18077.160162 secs 2024-07-26T17:52:21.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:21.552+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:52:21.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:21.552+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:52:21.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:21.552+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:52:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18077.160162 secs 2024-07-26T17:52:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:22 smithi107 ceph-mon[32659]: pgmap v9963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:22 smithi112 ceph-mon[37501]: pgmap v9963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:22 smithi132 ceph-mon[37580]: pgmap v9963: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:24 smithi112 ceph-mon[37501]: pgmap v9964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:24 smithi107 ceph-mon[32659]: pgmap v9964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:24 smithi132 ceph-mon[37580]: pgmap v9964: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:26 smithi112 ceph-mon[37501]: pgmap v9965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18082.160259 secs 2024-07-26T17:52:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:26 smithi107 ceph-mon[32659]: pgmap v9965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18082.160259 secs 2024-07-26T17:52:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:26 smithi132 ceph-mon[37580]: pgmap v9965: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18082.160259 secs 2024-07-26T17:52:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:28 smithi112 ceph-mon[37501]: pgmap v9966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:28 smithi107 ceph-mon[32659]: pgmap v9966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:28 smithi132 ceph-mon[37580]: pgmap v9966: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:30 smithi112 ceph-mon[37501]: pgmap v9967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:30 smithi107 ceph-mon[32659]: pgmap v9967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:30.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:30 smithi132 ceph-mon[37580]: pgmap v9967: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18087.160387 secs 2024-07-26T17:52:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18087.160387 secs 2024-07-26T17:52:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18087.160387 secs 2024-07-26T17:52:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:32 smithi112 ceph-mon[37501]: pgmap v9968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:32 smithi107 ceph-mon[32659]: pgmap v9968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:32 smithi132 ceph-mon[37580]: pgmap v9968: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:34 smithi112 ceph-mon[37501]: pgmap v9969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:34 smithi107 ceph-mon[32659]: pgmap v9969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:34 smithi132 ceph-mon[37580]: pgmap v9969: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:36.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:36 smithi112 ceph-mon[37501]: pgmap v9970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18092.160510 secs 2024-07-26T17:52:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:36 smithi107 ceph-mon[32659]: pgmap v9970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18092.160510 secs 2024-07-26T17:52:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:36 smithi132 ceph-mon[37580]: pgmap v9970: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18092.160510 secs 2024-07-26T17:52:36.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:36.567+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:52:36.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:36.567+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:52:36.569 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:36.567+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:52:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:38 smithi112 ceph-mon[37501]: pgmap v9971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:38 smithi107 ceph-mon[32659]: pgmap v9971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:38.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:38 smithi132 ceph-mon[37580]: pgmap v9971: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:52:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:40 smithi112 ceph-mon[37501]: pgmap v9972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:40 smithi107 ceph-mon[32659]: pgmap v9972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:40 smithi132 ceph-mon[37580]: pgmap v9972: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18097.160637 secs 2024-07-26T17:52:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18097.160637 secs 2024-07-26T17:52:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18097.160637 secs 2024-07-26T17:52:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:42 smithi112 ceph-mon[37501]: pgmap v9973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:42 smithi107 ceph-mon[32659]: pgmap v9973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:42 smithi132 ceph-mon[37580]: pgmap v9973: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:43 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:52:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:43 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:52:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:43 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:52:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:44 smithi112 ceph-mon[37501]: pgmap v9974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:52:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:44 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:52:44.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:44 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:52:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:44 smithi107 ceph-mon[32659]: pgmap v9974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:44.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:52:44.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:44 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:52:44.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:44 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:52:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:44 smithi132 ceph-mon[37580]: pgmap v9974: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:52:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:44 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:52:44.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:44 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:52:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:45 smithi112 ceph-mon[37501]: pgmap v9975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:52:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:45 smithi112 ceph-mon[37501]: pgmap v9976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:52:45.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:45 smithi107 ceph-mon[32659]: pgmap v9975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:52:45.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:45 smithi107 ceph-mon[32659]: pgmap v9976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:52:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:45 smithi132 ceph-mon[37580]: pgmap v9975: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:52:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:45 smithi132 ceph-mon[37580]: pgmap v9976: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T17:52:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18102.160761 secs 2024-07-26T17:52:46.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18102.160761 secs 2024-07-26T17:52:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18102.160761 secs 2024-07-26T17:52:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:47 smithi112 ceph-mon[37501]: pgmap v9977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:52:47.474 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:47 smithi132 ceph-mon[37580]: pgmap v9977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:52:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:47 smithi107 ceph-mon[32659]: pgmap v9977: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:52:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:49 smithi107 ceph-mon[32659]: pgmap v9978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:52:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:49 smithi132 ceph-mon[37580]: pgmap v9978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:52:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:49 smithi112 ceph-mon[37501]: pgmap v9978: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:52:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:51 smithi107 ceph-mon[32659]: pgmap v9979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:52:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18107.160903 secs 2024-07-26T17:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:51 smithi132 ceph-mon[37580]: pgmap v9979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:52:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18107.160903 secs 2024-07-26T17:52:51.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:51.580+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:52:51.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:51.580+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:52:51.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:52:51.580+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:52:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:51 smithi112 ceph-mon[37501]: pgmap v9979: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:52:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18107.160903 secs 2024-07-26T17:52:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:53 smithi107 ceph-mon[32659]: pgmap v9980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:53 smithi132 ceph-mon[37580]: pgmap v9980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:53 smithi112 ceph-mon[37501]: pgmap v9980: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:55 smithi107 ceph-mon[32659]: pgmap v9981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:52:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:55 smithi132 ceph-mon[37580]: pgmap v9981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:52:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:55 smithi112 ceph-mon[37501]: pgmap v9981: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:52:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18112.161026 secs 2024-07-26T17:52:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18112.161026 secs 2024-07-26T17:52:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18112.161026 secs 2024-07-26T17:52:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:57 smithi107 ceph-mon[32659]: pgmap v9982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:57 smithi132 ceph-mon[37580]: pgmap v9982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:57.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:57 smithi112 ceph-mon[37501]: pgmap v9982: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:52:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:52:59 smithi107 ceph-mon[32659]: pgmap v9983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:52:59 smithi132 ceph-mon[37580]: pgmap v9983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:52:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:52:59 smithi112 ceph-mon[37501]: pgmap v9983: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:01 smithi107 ceph-mon[32659]: pgmap v9984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18117.161143 secs 2024-07-26T17:53:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:01 smithi132 ceph-mon[37580]: pgmap v9984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18117.161143 secs 2024-07-26T17:53:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:01 smithi112 ceph-mon[37501]: pgmap v9984: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18117.161143 secs 2024-07-26T17:53:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:03 smithi107 ceph-mon[32659]: pgmap v9985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:03 smithi132 ceph-mon[37580]: pgmap v9985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:03 smithi112 ceph-mon[37501]: pgmap v9985: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:05 smithi107 ceph-mon[32659]: pgmap v9986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:05 smithi132 ceph-mon[37580]: pgmap v9986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:05 smithi112 ceph-mon[37501]: pgmap v9986: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18122.161264 secs 2024-07-26T17:53:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18122.161264 secs 2024-07-26T17:53:06.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:06.589+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:53:06.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:06.589+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:53:06.591 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:06.589+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:53:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18122.161264 secs 2024-07-26T17:53:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:07 smithi107 ceph-mon[32659]: pgmap v9987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:07 smithi132 ceph-mon[37580]: pgmap v9987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:07 smithi112 ceph-mon[37501]: pgmap v9987: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:09 smithi107 ceph-mon[32659]: pgmap v9988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:09 smithi132 ceph-mon[37580]: pgmap v9988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:09 smithi112 ceph-mon[37501]: pgmap v9988: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:11 smithi107 ceph-mon[32659]: pgmap v9989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18127.161389 secs 2024-07-26T17:53:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:11 smithi132 ceph-mon[37580]: pgmap v9989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18127.161389 secs 2024-07-26T17:53:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:11 smithi112 ceph-mon[37501]: pgmap v9989: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18127.161389 secs 2024-07-26T17:53:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:13 smithi107 ceph-mon[32659]: pgmap v9990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:13 smithi132 ceph-mon[37580]: pgmap v9990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:13 smithi112 ceph-mon[37501]: pgmap v9990: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:15 smithi132 ceph-mon[37580]: pgmap v9991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:15 smithi112 ceph-mon[37501]: pgmap v9991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:15 smithi107 ceph-mon[32659]: pgmap v9991: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18132.161509 secs 2024-07-26T17:53:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18132.161509 secs 2024-07-26T17:53:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18132.161509 secs 2024-07-26T17:53:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:17 smithi112 ceph-mon[37501]: pgmap v9992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:17 smithi107 ceph-mon[32659]: pgmap v9992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:17 smithi132 ceph-mon[37580]: pgmap v9992: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:19 smithi112 ceph-mon[37501]: pgmap v9993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:19 smithi107 ceph-mon[32659]: pgmap v9993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:19 smithi132 ceph-mon[37580]: pgmap v9993: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:21 smithi112 ceph-mon[37501]: pgmap v9994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18137.161622 secs 2024-07-26T17:53:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:21 smithi107 ceph-mon[32659]: pgmap v9994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18137.161622 secs 2024-07-26T17:53:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:21 smithi132 ceph-mon[37580]: pgmap v9994: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18137.161622 secs 2024-07-26T17:53:21.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:21.596+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:53:21.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:21.597+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:53:21.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:21.597+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:53:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:23 smithi112 ceph-mon[37501]: pgmap v9995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:23 smithi107 ceph-mon[32659]: pgmap v9995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:23 smithi132 ceph-mon[37580]: pgmap v9995: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:25 smithi112 ceph-mon[37501]: pgmap v9996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:25 smithi107 ceph-mon[32659]: pgmap v9996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:25 smithi132 ceph-mon[37580]: pgmap v9996: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18142.161739 secs 2024-07-26T17:53:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18142.161739 secs 2024-07-26T17:53:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18142.161739 secs 2024-07-26T17:53:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:27 smithi112 ceph-mon[37501]: pgmap v9997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:27 smithi107 ceph-mon[32659]: pgmap v9997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:27 smithi132 ceph-mon[37580]: pgmap v9997: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:29 smithi112 ceph-mon[37501]: pgmap v9998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:29 smithi107 ceph-mon[32659]: pgmap v9998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:29 smithi132 ceph-mon[37580]: pgmap v9998: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:31 smithi112 ceph-mon[37501]: pgmap v9999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18147.161865 secs 2024-07-26T17:53:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:31 smithi107 ceph-mon[32659]: pgmap v9999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18147.161865 secs 2024-07-26T17:53:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:31 smithi132 ceph-mon[37580]: pgmap v9999: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18147.161865 secs 2024-07-26T17:53:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:33 smithi107 ceph-mon[32659]: pgmap v10000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:33 smithi132 ceph-mon[37580]: pgmap v10000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:33 smithi112 ceph-mon[37501]: pgmap v10000: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:35.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:35 smithi107 ceph-mon[32659]: pgmap v10001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:35 smithi132 ceph-mon[37580]: pgmap v10001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:35 smithi112 ceph-mon[37501]: pgmap v10001: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18152.161993 secs 2024-07-26T17:53:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18152.161993 secs 2024-07-26T17:53:36.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:36.602+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:53:36.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:36.602+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:53:36.605 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:36.602+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:53:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18152.161993 secs 2024-07-26T17:53:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:37 smithi107 ceph-mon[32659]: pgmap v10002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:37 smithi132 ceph-mon[37580]: pgmap v10002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:37.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:37 smithi112 ceph-mon[37501]: pgmap v10002: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:39 smithi107 ceph-mon[32659]: pgmap v10003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:39 smithi132 ceph-mon[37580]: pgmap v10003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:39 smithi112 ceph-mon[37501]: pgmap v10003: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:41.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:41 smithi112 ceph-mon[37501]: pgmap v10004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:41.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18157.162136 secs 2024-07-26T17:53:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:41 smithi107 ceph-mon[32659]: pgmap v10004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18157.162136 secs 2024-07-26T17:53:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:41 smithi132 ceph-mon[37580]: pgmap v10004: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18157.162136 secs 2024-07-26T17:53:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:43 smithi107 ceph-mon[32659]: pgmap v10005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:43 smithi132 ceph-mon[37580]: pgmap v10005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:43 smithi112 ceph-mon[37501]: pgmap v10005: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:53:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:45 smithi107 ceph-mon[32659]: pgmap v10006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:53:45.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:45 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:53:45.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:53:45.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:45 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:53:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:45 smithi132 ceph-mon[37580]: pgmap v10006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:53:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:45 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:53:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:53:45.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:45 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:53:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:45 smithi112 ceph-mon[37501]: pgmap v10006: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:53:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:45 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:53:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:53:45.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:45 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:53:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18162.162226 secs 2024-07-26T17:53:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:53:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:53:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:46 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:53:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18162.162226 secs 2024-07-26T17:53:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:53:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:53:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:46 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:53:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18162.162226 secs 2024-07-26T17:53:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:53:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:53:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:46 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:53:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:47 smithi107 ceph-mon[32659]: pgmap v10007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:47.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:47 smithi107 ceph-mon[32659]: pgmap v10008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:53:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:47 smithi132 ceph-mon[37580]: pgmap v10007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:47 smithi132 ceph-mon[37580]: pgmap v10008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:53:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:47 smithi112 ceph-mon[37501]: pgmap v10007: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:53:47.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:47 smithi112 ceph-mon[37501]: pgmap v10008: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:53:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:49 smithi107 ceph-mon[32659]: pgmap v10009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:53:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:49 smithi132 ceph-mon[37580]: pgmap v10009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:53:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:49 smithi112 ceph-mon[37501]: pgmap v10009: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:53:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:51 smithi107 ceph-mon[32659]: pgmap v10010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:53:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18167.162331 secs 2024-07-26T17:53:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:51 smithi132 ceph-mon[37580]: pgmap v10010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:53:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18167.162331 secs 2024-07-26T17:53:51.621 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:51.618+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:53:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:51.618+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:53:51.622 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:53:51.618+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:53:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:51 smithi112 ceph-mon[37501]: pgmap v10010: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:53:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18167.162331 secs 2024-07-26T17:53:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:53 smithi107 ceph-mon[32659]: pgmap v10011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:53:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:53 smithi132 ceph-mon[37580]: pgmap v10011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:53:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:53 smithi112 ceph-mon[37501]: pgmap v10011: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:53:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:55 smithi107 ceph-mon[32659]: pgmap v10012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:53:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:55 smithi132 ceph-mon[37580]: pgmap v10012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:53:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:55 smithi112 ceph-mon[37501]: pgmap v10012: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T17:53:56.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18172.162415 secs 2024-07-26T17:53:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18172.162415 secs 2024-07-26T17:53:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18172.162415 secs 2024-07-26T17:53:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:57 smithi132 ceph-mon[37580]: pgmap v10013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:53:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:57 smithi112 ceph-mon[37501]: pgmap v10013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:53:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:57 smithi107 ceph-mon[32659]: pgmap v10013: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:53:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:53:59 smithi132 ceph-mon[37580]: pgmap v10014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:59.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:53:59 smithi112 ceph-mon[37501]: pgmap v10014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:53:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:53:59 smithi107 ceph-mon[32659]: pgmap v10014: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:01 smithi112 ceph-mon[37501]: pgmap v10015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18177.162532 secs 2024-07-26T17:54:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:01 smithi107 ceph-mon[32659]: pgmap v10015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18177.162532 secs 2024-07-26T17:54:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:01 smithi132 ceph-mon[37580]: pgmap v10015: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18177.162532 secs 2024-07-26T17:54:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:03 smithi112 ceph-mon[37501]: pgmap v10016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:03 smithi107 ceph-mon[32659]: pgmap v10016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:03 smithi132 ceph-mon[37580]: pgmap v10016: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:05 smithi112 ceph-mon[37501]: pgmap v10017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:05 smithi107 ceph-mon[32659]: pgmap v10017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:05 smithi132 ceph-mon[37580]: pgmap v10017: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:06.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:06.628+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:54:06.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:06.628+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:54:06.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:06.628+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:54:06.630 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18182.162685 secs 2024-07-26T17:54:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18182.162685 secs 2024-07-26T17:54:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18182.162685 secs 2024-07-26T17:54:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:07 smithi112 ceph-mon[37501]: pgmap v10018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:07 smithi107 ceph-mon[32659]: pgmap v10018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:07 smithi132 ceph-mon[37580]: pgmap v10018: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:09 smithi112 ceph-mon[37501]: pgmap v10019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:09 smithi107 ceph-mon[32659]: pgmap v10019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:09 smithi132 ceph-mon[37580]: pgmap v10019: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:11 smithi112 ceph-mon[37501]: pgmap v10020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18187.162805 secs 2024-07-26T17:54:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:11 smithi107 ceph-mon[32659]: pgmap v10020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:11.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18187.162805 secs 2024-07-26T17:54:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:11 smithi132 ceph-mon[37580]: pgmap v10020: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18187.162805 secs 2024-07-26T17:54:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:13 smithi112 ceph-mon[37501]: pgmap v10021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:13 smithi107 ceph-mon[32659]: pgmap v10021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:13 smithi132 ceph-mon[37580]: pgmap v10021: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:15.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:15 smithi112 ceph-mon[37501]: pgmap v10022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:15.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:15.788+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:54:15.789 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:15.788+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:54:15.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:15 smithi107 ceph-mon[32659]: pgmap v10022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:15.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:15.789+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:54:15.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:15.789+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:54:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:15 smithi132 ceph-mon[37580]: pgmap v10022: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18192.162922 secs 2024-07-26T17:54:16.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18192.162922 secs 2024-07-26T17:54:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18192.162922 secs 2024-07-26T17:54:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:17 smithi112 ceph-mon[37501]: pgmap v10023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:17 smithi107 ceph-mon[32659]: pgmap v10023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:17 smithi132 ceph-mon[37580]: pgmap v10023: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:19 smithi112 ceph-mon[37501]: pgmap v10024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:19 smithi107 ceph-mon[32659]: pgmap v10024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:19 smithi132 ceph-mon[37580]: pgmap v10024: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:21.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:21.637+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:54:21.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:21.638+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:54:21.639 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:21.638+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:54:21.640 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:21 smithi107 ceph-mon[32659]: pgmap v10025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:21.641 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18197.163060 secs 2024-07-26T17:54:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:21 smithi112 ceph-mon[37501]: pgmap v10025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18197.163060 secs 2024-07-26T17:54:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:21 smithi132 ceph-mon[37580]: pgmap v10025: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18197.163060 secs 2024-07-26T17:54:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:23 smithi112 ceph-mon[37501]: pgmap v10026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:23 smithi107 ceph-mon[32659]: pgmap v10026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:23 smithi132 ceph-mon[37580]: pgmap v10026: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:25 smithi112 ceph-mon[37501]: pgmap v10027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:54:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:25 smithi107 ceph-mon[32659]: pgmap v10027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:54:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:25 smithi132 ceph-mon[37580]: pgmap v10027: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:54:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18202.163198 secs 2024-07-26T17:54:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18202.163198 secs 2024-07-26T17:54:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18202.163198 secs 2024-07-26T17:54:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:27 smithi112 ceph-mon[37501]: pgmap v10028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:54:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:27 smithi107 ceph-mon[32659]: pgmap v10028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:54:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:27 smithi132 ceph-mon[37580]: pgmap v10028: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:54:29.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:29 smithi112 ceph-mon[37501]: pgmap v10029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:54:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:29 smithi107 ceph-mon[32659]: pgmap v10029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:54:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:29 smithi132 ceph-mon[37580]: pgmap v10029: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:54:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:31 smithi112 ceph-mon[37501]: pgmap v10030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:54:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18207.163277 secs 2024-07-26T17:54:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:31 smithi107 ceph-mon[32659]: pgmap v10030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:54:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18207.163277 secs 2024-07-26T17:54:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:31 smithi132 ceph-mon[37580]: pgmap v10030: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:54:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18207.163277 secs 2024-07-26T17:54:33.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:33 smithi112 ceph-mon[37501]: pgmap v10031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:54:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:33 smithi107 ceph-mon[32659]: pgmap v10031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:54:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:33 smithi132 ceph-mon[37580]: pgmap v10031: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:54:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:35 smithi112 ceph-mon[37501]: pgmap v10032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:54:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:35 smithi107 ceph-mon[32659]: pgmap v10032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:54:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:35 smithi132 ceph-mon[37580]: pgmap v10032: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:54:36.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:36.651+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:54:36.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:36.651+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:54:36.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:36.651+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:54:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18212.163343 secs 2024-07-26T17:54:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18212.163343 secs 2024-07-26T17:54:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18212.163343 secs 2024-07-26T17:54:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:37 smithi107 ceph-mon[32659]: pgmap v10033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:37 smithi132 ceph-mon[37580]: pgmap v10033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:37 smithi112 ceph-mon[37501]: pgmap v10033: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:39 smithi107 ceph-mon[32659]: pgmap v10034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:39 smithi132 ceph-mon[37580]: pgmap v10034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:39 smithi112 ceph-mon[37501]: pgmap v10034: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:41 smithi107 ceph-mon[32659]: pgmap v10035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18217.163420 secs 2024-07-26T17:54:41.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:41 smithi112 ceph-mon[37501]: pgmap v10035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:41.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18217.163420 secs 2024-07-26T17:54:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:41 smithi132 ceph-mon[37580]: pgmap v10035: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18217.163420 secs 2024-07-26T17:54:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:43 smithi107 ceph-mon[32659]: pgmap v10036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:43 smithi132 ceph-mon[37580]: pgmap v10036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:43 smithi112 ceph-mon[37501]: pgmap v10036: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:45 smithi107 ceph-mon[32659]: pgmap v10037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:45 smithi132 ceph-mon[37580]: pgmap v10037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:45 smithi112 ceph-mon[37501]: pgmap v10037: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18222.163491 secs 2024-07-26T17:54:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:46 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:54:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18222.163491 secs 2024-07-26T17:54:46.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:46 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:54:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18222.163491 secs 2024-07-26T17:54:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:46 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:54:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:47 smithi107 ceph-mon[32659]: pgmap v10038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:47 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:47 smithi132 ceph-mon[37580]: pgmap v10038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:47 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:47 smithi112 ceph-mon[37501]: pgmap v10038: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:54:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:47.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:47 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:49 smithi107 ceph-mon[32659]: pgmap v10039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:49 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:49 smithi132 ceph-mon[37580]: pgmap v10039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:49 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:49 smithi112 ceph-mon[37501]: pgmap v10039: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:54:49.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:49 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:50.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:50 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:50 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:50.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:50 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:51.669 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:51.667+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:54:51.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:51.667+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:54:51.670 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:54:51.667+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:54:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:51 smithi112 ceph-mon[37501]: pgmap v10040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18227.163604 secs 2024-07-26T17:54:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:54:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:54:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:51 smithi112 ceph-mon[37501]: pgmap v10041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:54:51.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:51 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:51 smithi107 ceph-mon[32659]: pgmap v10040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18227.163604 secs 2024-07-26T17:54:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:54:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:54:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:51 smithi107 ceph-mon[32659]: pgmap v10041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:54:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:51 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:51 smithi132 ceph-mon[37580]: pgmap v10040: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:54:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18227.163604 secs 2024-07-26T17:54:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:54:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:54:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:51 smithi132 ceph-mon[37580]: pgmap v10041: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:54:52.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:51 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:54:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:53 smithi112 ceph-mon[37501]: pgmap v10042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T17:54:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:53 smithi107 ceph-mon[32659]: pgmap v10042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T17:54:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:53 smithi132 ceph-mon[37580]: pgmap v10042: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T17:54:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:55 smithi112 ceph-mon[37501]: pgmap v10043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:54:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:55 smithi107 ceph-mon[32659]: pgmap v10043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:54:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:55 smithi132 ceph-mon[37580]: pgmap v10043: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:54:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18232.163719 secs 2024-07-26T17:54:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18232.163719 secs 2024-07-26T17:54:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18232.163719 secs 2024-07-26T17:54:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:57 smithi112 ceph-mon[37501]: pgmap v10044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T17:54:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:57 smithi107 ceph-mon[32659]: pgmap v10044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T17:54:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:57 smithi132 ceph-mon[37580]: pgmap v10044: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T17:54:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:54:59 smithi112 ceph-mon[37501]: pgmap v10045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T17:55:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:54:59 smithi107 ceph-mon[32659]: pgmap v10045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T17:55:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:54:59 smithi132 ceph-mon[37580]: pgmap v10045: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T17:55:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18237.163846 secs 2024-07-26T17:55:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:01 smithi112 ceph-mon[37501]: pgmap v10046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:55:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18237.163846 secs 2024-07-26T17:55:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:01 smithi107 ceph-mon[32659]: pgmap v10046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:55:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18237.163846 secs 2024-07-26T17:55:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:01 smithi132 ceph-mon[37580]: pgmap v10046: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:55:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:03 smithi112 ceph-mon[37501]: pgmap v10047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:03 smithi107 ceph-mon[32659]: pgmap v10047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:03 smithi132 ceph-mon[37580]: pgmap v10047: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:05 smithi112 ceph-mon[37501]: pgmap v10048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:55:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:05 smithi107 ceph-mon[32659]: pgmap v10048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:55:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:05 smithi132 ceph-mon[37580]: pgmap v10048: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:55:06.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:06.683+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:55:06.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:06.683+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:55:06.685 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:06.683+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:55:06.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18242.163931 secs 2024-07-26T17:55:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18242.163931 secs 2024-07-26T17:55:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18242.163931 secs 2024-07-26T17:55:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:07 smithi112 ceph-mon[37501]: pgmap v10049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:07 smithi107 ceph-mon[32659]: pgmap v10049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:07 smithi132 ceph-mon[37580]: pgmap v10049: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:09 smithi107 ceph-mon[32659]: pgmap v10050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:09 smithi132 ceph-mon[37580]: pgmap v10050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:09 smithi112 ceph-mon[37501]: pgmap v10050: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:12.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18247.164090 secs 2024-07-26T17:55:12.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:11 smithi112 ceph-mon[37501]: pgmap v10051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18247.164090 secs 2024-07-26T17:55:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:11 smithi107 ceph-mon[32659]: pgmap v10051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18247.164090 secs 2024-07-26T17:55:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:11 smithi132 ceph-mon[37580]: pgmap v10051: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:13 smithi107 ceph-mon[32659]: pgmap v10052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:13 smithi132 ceph-mon[37580]: pgmap v10052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:13 smithi112 ceph-mon[37501]: pgmap v10052: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:15 smithi107 ceph-mon[32659]: pgmap v10053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:55:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:15 smithi132 ceph-mon[37580]: pgmap v10053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:55:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:15 smithi112 ceph-mon[37501]: pgmap v10053: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:55:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18252.164184 secs 2024-07-26T17:55:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18252.164184 secs 2024-07-26T17:55:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18252.164184 secs 2024-07-26T17:55:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:17 smithi107 ceph-mon[32659]: pgmap v10054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:17 smithi132 ceph-mon[37580]: pgmap v10054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:17 smithi112 ceph-mon[37501]: pgmap v10054: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:19 smithi107 ceph-mon[32659]: pgmap v10055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:19 smithi132 ceph-mon[37580]: pgmap v10055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:19 smithi112 ceph-mon[37501]: pgmap v10055: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:21.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:21.697+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:55:21.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:21.697+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:55:21.699 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:21.697+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:55:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18257.164309 secs 2024-07-26T17:55:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:21 smithi107 ceph-mon[32659]: pgmap v10056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18257.164309 secs 2024-07-26T17:55:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:21 smithi132 ceph-mon[37580]: pgmap v10056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18257.164309 secs 2024-07-26T17:55:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:21 smithi112 ceph-mon[37501]: pgmap v10056: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:23 smithi107 ceph-mon[32659]: pgmap v10057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:23 smithi132 ceph-mon[37580]: pgmap v10057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:23 smithi112 ceph-mon[37501]: pgmap v10057: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:25 smithi107 ceph-mon[32659]: pgmap v10058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:55:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:25 smithi132 ceph-mon[37580]: pgmap v10058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:55:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:25 smithi112 ceph-mon[37501]: pgmap v10058: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:55:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18262.164426 secs 2024-07-26T17:55:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18262.164426 secs 2024-07-26T17:55:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18262.164426 secs 2024-07-26T17:55:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:27 smithi107 ceph-mon[32659]: pgmap v10059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:27 smithi132 ceph-mon[37580]: pgmap v10059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:27 smithi112 ceph-mon[37501]: pgmap v10059: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:28 smithi107 ceph-mon[32659]: pgmap v10060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:28 smithi132 ceph-mon[37580]: pgmap v10060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:28 smithi112 ceph-mon[37501]: pgmap v10060: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18267.164537 secs 2024-07-26T17:55:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:31 smithi112 ceph-mon[37501]: pgmap v10061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18267.164537 secs 2024-07-26T17:55:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:31 smithi107 ceph-mon[32659]: pgmap v10061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18267.164537 secs 2024-07-26T17:55:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:31 smithi132 ceph-mon[37580]: pgmap v10061: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:33 smithi112 ceph-mon[37501]: pgmap v10062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:33 smithi107 ceph-mon[32659]: pgmap v10062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:33 smithi132 ceph-mon[37580]: pgmap v10062: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:35 smithi112 ceph-mon[37501]: pgmap v10063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:35 smithi107 ceph-mon[32659]: pgmap v10063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:35 smithi132 ceph-mon[37580]: pgmap v10063: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:36.714 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:36.713+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:55:36.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:36.713+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:55:36.715 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:36.713+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:55:36.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18272.164655 secs 2024-07-26T17:55:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18272.164655 secs 2024-07-26T17:55:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18272.164655 secs 2024-07-26T17:55:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:37 smithi112 ceph-mon[37501]: pgmap v10064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:37 smithi107 ceph-mon[32659]: pgmap v10064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:37 smithi132 ceph-mon[37580]: pgmap v10064: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:39 smithi112 ceph-mon[37501]: pgmap v10065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:39 smithi107 ceph-mon[32659]: pgmap v10065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:39 smithi132 ceph-mon[37580]: pgmap v10065: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18277.164800 secs 2024-07-26T17:55:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:41 smithi107 ceph-mon[32659]: pgmap v10066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18277.164800 secs 2024-07-26T17:55:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:41 smithi132 ceph-mon[37580]: pgmap v10066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18277.164800 secs 2024-07-26T17:55:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:41 smithi112 ceph-mon[37501]: pgmap v10066: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:44.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:43 smithi112 ceph-mon[37501]: pgmap v10067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:43 smithi107 ceph-mon[32659]: pgmap v10067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:43 smithi132 ceph-mon[37580]: pgmap v10067: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:45 smithi107 ceph-mon[32659]: pgmap v10068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:45 smithi132 ceph-mon[37580]: pgmap v10068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:45 smithi112 ceph-mon[37501]: pgmap v10068: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18282.164946 secs 2024-07-26T17:55:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18282.164946 secs 2024-07-26T17:55:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18282.164946 secs 2024-07-26T17:55:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:47 smithi107 ceph-mon[32659]: pgmap v10069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:47 smithi132 ceph-mon[37580]: pgmap v10069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:47 smithi112 ceph-mon[37501]: pgmap v10069: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:55:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:49 smithi107 ceph-mon[32659]: pgmap v10070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:49 smithi132 ceph-mon[37580]: pgmap v10070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:49 smithi112 ceph-mon[37501]: pgmap v10070: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:55:51.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:51.728+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:55:51.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:51.728+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:55:51.731 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:55:51.728+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:55:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18287.165018 secs 2024-07-26T17:55:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:51 smithi107 ceph-mon[32659]: pgmap v10071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:51 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:55:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18287.165018 secs 2024-07-26T17:55:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:51 smithi132 ceph-mon[37580]: pgmap v10071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:51 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:55:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18287.165018 secs 2024-07-26T17:55:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:51 smithi112 ceph-mon[37501]: pgmap v10071: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:55:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:51 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:55:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:55:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:55:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:52 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:55:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:55:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:55:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:52 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:55:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:55:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:55:53.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:52 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:55:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:53 smithi107 ceph-mon[32659]: pgmap v10072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:55:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:53 smithi132 ceph-mon[37580]: pgmap v10072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:55:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:53 smithi112 ceph-mon[37501]: pgmap v10072: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:55:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:55 smithi107 ceph-mon[32659]: pgmap v10073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:55:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:55 smithi132 ceph-mon[37580]: pgmap v10073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:55:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:55 smithi112 ceph-mon[37501]: pgmap v10073: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:55:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18292.165168 secs 2024-07-26T17:55:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18292.165168 secs 2024-07-26T17:55:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18292.165168 secs 2024-07-26T17:55:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:57 smithi107 ceph-mon[32659]: pgmap v10074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:55:58.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:57 smithi132 ceph-mon[37580]: pgmap v10074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:55:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:57 smithi112 ceph-mon[37501]: pgmap v10074: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:55:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:55:58 smithi107 ceph-mon[32659]: pgmap v10075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:55:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:55:58 smithi132 ceph-mon[37580]: pgmap v10075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:55:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:55:58 smithi112 ceph-mon[37501]: pgmap v10075: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:01 smithi112 ceph-mon[37501]: pgmap v10076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:56:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18297.165271 secs 2024-07-26T17:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:01 smithi107 ceph-mon[32659]: pgmap v10076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:56:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18297.165271 secs 2024-07-26T17:56:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:01 smithi132 ceph-mon[37580]: pgmap v10076: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:56:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18297.165271 secs 2024-07-26T17:56:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:03 smithi112 ceph-mon[37501]: pgmap v10077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:56:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:03 smithi107 ceph-mon[32659]: pgmap v10077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:56:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:03 smithi132 ceph-mon[37580]: pgmap v10077: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:56:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:05 smithi112 ceph-mon[37501]: pgmap v10078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:05 smithi107 ceph-mon[32659]: pgmap v10078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:05 smithi132 ceph-mon[37580]: pgmap v10078: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18302.165393 secs 2024-07-26T17:56:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18302.165393 secs 2024-07-26T17:56:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18302.165393 secs 2024-07-26T17:56:06.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:06.744+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:56:06.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:06.744+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:56:06.746 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:06.744+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:56:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:07 smithi107 ceph-mon[32659]: pgmap v10079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:07 smithi132 ceph-mon[37580]: pgmap v10079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:07 smithi112 ceph-mon[37501]: pgmap v10079: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:09 smithi107 ceph-mon[32659]: pgmap v10080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:09 smithi132 ceph-mon[37580]: pgmap v10080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:09 smithi112 ceph-mon[37501]: pgmap v10080: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:11 smithi107 ceph-mon[32659]: pgmap v10081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18307.165535 secs 2024-07-26T17:56:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:11 smithi132 ceph-mon[37580]: pgmap v10081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18307.165535 secs 2024-07-26T17:56:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:11 smithi112 ceph-mon[37501]: pgmap v10081: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18307.165535 secs 2024-07-26T17:56:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:13 smithi107 ceph-mon[32659]: pgmap v10082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:13.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:13 smithi132 ceph-mon[37580]: pgmap v10082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:13 smithi112 ceph-mon[37501]: pgmap v10082: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:15.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:15 smithi112 ceph-mon[37501]: pgmap v10083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:15 smithi107 ceph-mon[32659]: pgmap v10083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:15 smithi132 ceph-mon[37580]: pgmap v10083: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18312.165676 secs 2024-07-26T17:56:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18312.165676 secs 2024-07-26T17:56:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18312.165676 secs 2024-07-26T17:56:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:17 smithi107 ceph-mon[32659]: pgmap v10084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:17 smithi132 ceph-mon[37580]: pgmap v10084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:17 smithi112 ceph-mon[37501]: pgmap v10084: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:19.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:19 smithi107 ceph-mon[32659]: pgmap v10085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:19.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:19 smithi132 ceph-mon[37580]: pgmap v10085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:19 smithi112 ceph-mon[37501]: pgmap v10085: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:21 smithi107 ceph-mon[32659]: pgmap v10086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18317.165808 secs 2024-07-26T17:56:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:21 smithi132 ceph-mon[37580]: pgmap v10086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18317.165808 secs 2024-07-26T17:56:21.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:21 smithi112 ceph-mon[37501]: pgmap v10086: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18317.165808 secs 2024-07-26T17:56:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:21.755+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:56:21.756 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:21.755+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:56:21.757 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:21.755+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:56:23.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:23 smithi107 ceph-mon[32659]: pgmap v10087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:56:23.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:23 smithi132 ceph-mon[37580]: pgmap v10087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:56:23.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:23 smithi112 ceph-mon[37501]: pgmap v10087: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:56:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:25 smithi107 ceph-mon[32659]: pgmap v10088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:25.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:25 smithi132 ceph-mon[37580]: pgmap v10088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:25 smithi112 ceph-mon[37501]: pgmap v10088: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18322.165952 secs 2024-07-26T17:56:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18322.165952 secs 2024-07-26T17:56:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18322.165952 secs 2024-07-26T17:56:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:27 smithi107 ceph-mon[32659]: pgmap v10089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:56:27.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:27 smithi132 ceph-mon[37580]: pgmap v10089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:56:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:27 smithi112 ceph-mon[37501]: pgmap v10089: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:56:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:29 smithi107 ceph-mon[32659]: pgmap v10090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:29.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:29 smithi132 ceph-mon[37580]: pgmap v10090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:29 smithi112 ceph-mon[37501]: pgmap v10090: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:29.811+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:56:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:29.811+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:56:29.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:29.811+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T17:56:29.813 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:29.811+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T17:56:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:31 smithi107 ceph-mon[32659]: pgmap v10091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18327.166104 secs 2024-07-26T17:56:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:31 smithi132 ceph-mon[37580]: pgmap v10091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18327.166104 secs 2024-07-26T17:56:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:31 smithi112 ceph-mon[37501]: pgmap v10091: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18327.166104 secs 2024-07-26T17:56:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:33 smithi107 ceph-mon[32659]: pgmap v10092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:33 smithi132 ceph-mon[37580]: pgmap v10092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:33 smithi112 ceph-mon[37501]: pgmap v10092: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:35 smithi132 ceph-mon[37580]: pgmap v10093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:35.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:35 smithi112 ceph-mon[37501]: pgmap v10093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:35 smithi107 ceph-mon[32659]: pgmap v10093: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18332.166228 secs 2024-07-26T17:56:36.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18332.166228 secs 2024-07-26T17:56:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18332.166228 secs 2024-07-26T17:56:36.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:36.770+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:56:36.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:36.770+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:56:36.772 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:36.770+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:56:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:37 smithi112 ceph-mon[37501]: pgmap v10094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:37.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:37 smithi107 ceph-mon[32659]: pgmap v10094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:37 smithi132 ceph-mon[37580]: pgmap v10094: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:39 smithi112 ceph-mon[37501]: pgmap v10095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:39 smithi107 ceph-mon[32659]: pgmap v10095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:39 smithi132 ceph-mon[37580]: pgmap v10095: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:41 smithi112 ceph-mon[37501]: pgmap v10096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18337.166345 secs 2024-07-26T17:56:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:41 smithi107 ceph-mon[32659]: pgmap v10096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18337.166345 secs 2024-07-26T17:56:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:41 smithi132 ceph-mon[37580]: pgmap v10096: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18337.166345 secs 2024-07-26T17:56:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:43 smithi112 ceph-mon[37501]: pgmap v10097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:43 smithi107 ceph-mon[32659]: pgmap v10097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:43 smithi132 ceph-mon[37580]: pgmap v10097: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:45 smithi112 ceph-mon[37501]: pgmap v10098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:45 smithi107 ceph-mon[32659]: pgmap v10098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:45 smithi132 ceph-mon[37580]: pgmap v10098: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18342.166462 secs 2024-07-26T17:56:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18342.166462 secs 2024-07-26T17:56:46.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18342.166462 secs 2024-07-26T17:56:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:47 smithi112 ceph-mon[37501]: pgmap v10099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:47 smithi107 ceph-mon[32659]: pgmap v10099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:47 smithi132 ceph-mon[37580]: pgmap v10099: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:56:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:49 smithi112 ceph-mon[37501]: pgmap v10100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:49 smithi107 ceph-mon[32659]: pgmap v10100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:49 smithi132 ceph-mon[37580]: pgmap v10100: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:56:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:51 smithi112 ceph-mon[37501]: pgmap v10101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18347.166611 secs 2024-07-26T17:56:51.787 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:51.785+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:56:51.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:51.785+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:56:51.788 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:56:51.785+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:56:51.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:51 smithi107 ceph-mon[32659]: pgmap v10101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:51.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18347.166611 secs 2024-07-26T17:56:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:51 smithi132 ceph-mon[37580]: pgmap v10101: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18347.166611 secs 2024-07-26T17:56:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:52 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:56:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:52 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:56:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:52 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:56:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:53 smithi112 ceph-mon[37501]: pgmap v10102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:56:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:56:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:53 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:56:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:53 smithi107 ceph-mon[32659]: pgmap v10102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:56:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:56:53.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:53 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:56:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:53 smithi132 ceph-mon[37580]: pgmap v10102: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:56:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:56:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:56:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:53 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:56:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:54 smithi112 ceph-mon[37501]: pgmap v10103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:54 smithi107 ceph-mon[32659]: pgmap v10103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:54 smithi132 ceph-mon[37580]: pgmap v10103: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:56:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:56 smithi112 ceph-mon[37501]: pgmap v10104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:56:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18352.166727 secs 2024-07-26T17:56:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:56 smithi107 ceph-mon[32659]: pgmap v10104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:56:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18352.166727 secs 2024-07-26T17:56:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:56 smithi132 ceph-mon[37580]: pgmap v10104: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T17:56:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18352.166727 secs 2024-07-26T17:56:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:56:58 smithi112 ceph-mon[37501]: pgmap v10105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:56:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:56:58 smithi107 ceph-mon[32659]: pgmap v10105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:56:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:56:58 smithi132 ceph-mon[37580]: pgmap v10105: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:57:00.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:00 smithi112 ceph-mon[37501]: pgmap v10106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:57:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:00 smithi107 ceph-mon[32659]: pgmap v10106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:57:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:00 smithi132 ceph-mon[37580]: pgmap v10106: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:57:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18357.166845 secs 2024-07-26T17:57:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18357.166845 secs 2024-07-26T17:57:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18357.166845 secs 2024-07-26T17:57:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:02 smithi112 ceph-mon[37501]: pgmap v10107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:57:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:02 smithi107 ceph-mon[32659]: pgmap v10107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:57:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:02 smithi132 ceph-mon[37580]: pgmap v10107: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:57:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:04 smithi112 ceph-mon[37501]: pgmap v10108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:57:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:04 smithi107 ceph-mon[32659]: pgmap v10108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:57:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:04 smithi132 ceph-mon[37580]: pgmap v10108: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:57:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:06 smithi112 ceph-mon[37501]: pgmap v10109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18362.166994 secs 2024-07-26T17:57:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:06 smithi107 ceph-mon[32659]: pgmap v10109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18362.166994 secs 2024-07-26T17:57:06.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:06.801+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:57:06.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:06.801+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:57:06.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:06.801+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:57:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:06 smithi132 ceph-mon[37580]: pgmap v10109: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18362.166994 secs 2024-07-26T17:57:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:08 smithi112 ceph-mon[37501]: pgmap v10110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:08 smithi107 ceph-mon[32659]: pgmap v10110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:08 smithi132 ceph-mon[37580]: pgmap v10110: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:10 smithi112 ceph-mon[37501]: pgmap v10111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:10 smithi107 ceph-mon[32659]: pgmap v10111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:10 smithi132 ceph-mon[37580]: pgmap v10111: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18367.167128 secs 2024-07-26T17:57:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18367.167128 secs 2024-07-26T17:57:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18367.167128 secs 2024-07-26T17:57:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:12 smithi107 ceph-mon[32659]: pgmap v10112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:12 smithi132 ceph-mon[37580]: pgmap v10112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:12 smithi112 ceph-mon[37501]: pgmap v10112: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:14 smithi107 ceph-mon[32659]: pgmap v10113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:14 smithi132 ceph-mon[37580]: pgmap v10113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:14 smithi112 ceph-mon[37501]: pgmap v10113: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:15.819+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:57:15.820 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:15.819+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:57:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:15.820+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T17:57:15.821 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:15.820+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T17:57:16.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:16 smithi112 ceph-mon[37501]: pgmap v10114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:16.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18372.167239 secs 2024-07-26T17:57:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:16 smithi107 ceph-mon[32659]: pgmap v10114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18372.167239 secs 2024-07-26T17:57:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:16 smithi132 ceph-mon[37580]: pgmap v10114: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18372.167239 secs 2024-07-26T17:57:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:18 smithi107 ceph-mon[32659]: pgmap v10115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:18.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:18 smithi132 ceph-mon[37580]: pgmap v10115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:18 smithi112 ceph-mon[37501]: pgmap v10115: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:20 smithi107 ceph-mon[32659]: pgmap v10116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:20 smithi132 ceph-mon[37580]: pgmap v10116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:20 smithi112 ceph-mon[37501]: pgmap v10116: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18377.167361 secs 2024-07-26T17:57:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18377.167361 secs 2024-07-26T17:57:21.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:21.810+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:57:21.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:21.810+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:57:21.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:21.810+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:57:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18377.167361 secs 2024-07-26T17:57:22.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:22 smithi107 ceph-mon[32659]: pgmap v10117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:22 smithi132 ceph-mon[37580]: pgmap v10117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:22 smithi112 ceph-mon[37501]: pgmap v10117: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:24.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:24 smithi107 ceph-mon[32659]: pgmap v10118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:24 smithi132 ceph-mon[37580]: pgmap v10118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:24 smithi112 ceph-mon[37501]: pgmap v10118: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:26 smithi107 ceph-mon[32659]: pgmap v10119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:26.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18382.167424 secs 2024-07-26T17:57:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:26 smithi132 ceph-mon[37580]: pgmap v10119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18382.167424 secs 2024-07-26T17:57:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:26 smithi112 ceph-mon[37501]: pgmap v10119: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18382.167424 secs 2024-07-26T17:57:28.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:28 smithi107 ceph-mon[32659]: pgmap v10120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:28 smithi132 ceph-mon[37580]: pgmap v10120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:28 smithi112 ceph-mon[37501]: pgmap v10120: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:30.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:30 smithi107 ceph-mon[32659]: pgmap v10121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:30.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:30 smithi132 ceph-mon[37580]: pgmap v10121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:30 smithi112 ceph-mon[37501]: pgmap v10121: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18387.167538 secs 2024-07-26T17:57:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18387.167538 secs 2024-07-26T17:57:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18387.167538 secs 2024-07-26T17:57:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:32 smithi107 ceph-mon[32659]: pgmap v10122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:32.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:32 smithi132 ceph-mon[37580]: pgmap v10122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:32 smithi112 ceph-mon[37501]: pgmap v10122: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:34 smithi107 ceph-mon[32659]: pgmap v10123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:34 smithi132 ceph-mon[37580]: pgmap v10123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:34 smithi112 ceph-mon[37501]: pgmap v10123: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:36 smithi107 ceph-mon[32659]: pgmap v10124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18392.167651 secs 2024-07-26T17:57:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:36 smithi132 ceph-mon[37580]: pgmap v10124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18392.167651 secs 2024-07-26T17:57:36.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:36.826+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:57:36.827 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:36.826+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:57:36.828 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:36.826+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:57:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:36 smithi112 ceph-mon[37501]: pgmap v10124: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18392.167651 secs 2024-07-26T17:57:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:38 smithi132 ceph-mon[37580]: pgmap v10125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:38 smithi112 ceph-mon[37501]: pgmap v10125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:38 smithi107 ceph-mon[32659]: pgmap v10125: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:40 smithi132 ceph-mon[37580]: pgmap v10126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:40.956 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:40 smithi112 ceph-mon[37501]: pgmap v10126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:40 smithi107 ceph-mon[32659]: pgmap v10126: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18397.167776 secs 2024-07-26T17:57:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18397.167776 secs 2024-07-26T17:57:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18397.167776 secs 2024-07-26T17:57:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:42 smithi112 ceph-mon[37501]: pgmap v10127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:42 smithi107 ceph-mon[32659]: pgmap v10127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:42 smithi132 ceph-mon[37580]: pgmap v10127: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:44 smithi112 ceph-mon[37501]: pgmap v10128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:44 smithi107 ceph-mon[32659]: pgmap v10128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:44 smithi132 ceph-mon[37580]: pgmap v10128: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:46.862 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:46 smithi132 ceph-mon[37580]: pgmap v10129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:46.863 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18402.167932 secs 2024-07-26T17:57:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:46 smithi112 ceph-mon[37501]: pgmap v10129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18402.167932 secs 2024-07-26T17:57:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:46 smithi107 ceph-mon[32659]: pgmap v10129: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18402.167932 secs 2024-07-26T17:57:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:48 smithi112 ceph-mon[37501]: pgmap v10130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:48 smithi107 ceph-mon[32659]: pgmap v10130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:48 smithi132 ceph-mon[37580]: pgmap v10130: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:57:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:50 smithi112 ceph-mon[37501]: pgmap v10131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:50 smithi107 ceph-mon[32659]: pgmap v10131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:50 smithi132 ceph-mon[37580]: pgmap v10131: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:51.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:51.841+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:57:51.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:51.841+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:57:51.843 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:57:51.841+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:57:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18407.168049 secs 2024-07-26T17:57:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18407.168049 secs 2024-07-26T17:57:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18407.168049 secs 2024-07-26T17:57:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:52 smithi112 ceph-mon[37501]: pgmap v10132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:52 smithi107 ceph-mon[32659]: pgmap v10132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:52 smithi132 ceph-mon[37580]: pgmap v10132: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:57:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:53 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:57:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:53 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:57:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:53 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:57:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:54 smithi112 ceph-mon[37501]: pgmap v10133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:57:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:54 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:57:54.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:54 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:57:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:54 smithi107 ceph-mon[32659]: pgmap v10133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:57:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:54 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:57:55.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:54 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:57:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:54 smithi132 ceph-mon[37580]: pgmap v10133: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:57:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:57:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:54 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:57:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:54 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:57:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:55 smithi112 ceph-mon[37501]: pgmap v10134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:57:55.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:55 smithi112 ceph-mon[37501]: pgmap v10135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:57:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:55 smithi107 ceph-mon[32659]: pgmap v10134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:57:56.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:55 smithi107 ceph-mon[32659]: pgmap v10135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:57:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:55 smithi132 ceph-mon[37580]: pgmap v10134: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T17:57:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:55 smithi132 ceph-mon[37580]: pgmap v10135: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:57:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18412.168211 secs 2024-07-26T17:57:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18412.168211 secs 2024-07-26T17:57:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18412.168211 secs 2024-07-26T17:57:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:57 smithi112 ceph-mon[37501]: pgmap v10136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:57:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:57 smithi107 ceph-mon[32659]: pgmap v10136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:57:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:57 smithi132 ceph-mon[37580]: pgmap v10136: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T17:57:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:57:59 smithi112 ceph-mon[37501]: pgmap v10137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:58:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:57:59 smithi107 ceph-mon[32659]: pgmap v10137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:58:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:57:59 smithi132 ceph-mon[37580]: pgmap v10137: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T17:58:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:01 smithi112 ceph-mon[37501]: pgmap v10138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:58:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18417.168349 secs 2024-07-26T17:58:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:01 smithi107 ceph-mon[32659]: pgmap v10138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:58:02.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18417.168349 secs 2024-07-26T17:58:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:01 smithi132 ceph-mon[37580]: pgmap v10138: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T17:58:02.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18417.168349 secs 2024-07-26T17:58:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:03 smithi112 ceph-mon[37501]: pgmap v10139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:03 smithi107 ceph-mon[32659]: pgmap v10139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:03 smithi132 ceph-mon[37580]: pgmap v10139: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:05.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:05 smithi112 ceph-mon[37501]: pgmap v10140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:58:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:05 smithi107 ceph-mon[32659]: pgmap v10140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:58:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:05 smithi132 ceph-mon[37580]: pgmap v10140: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:58:06.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:06.857+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:58:06.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:06.857+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:58:06.859 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:06.857+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:58:06.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18422.168479 secs 2024-07-26T17:58:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18422.168479 secs 2024-07-26T17:58:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18422.168479 secs 2024-07-26T17:58:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:07 smithi112 ceph-mon[37501]: pgmap v10141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:07 smithi107 ceph-mon[32659]: pgmap v10141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:07 smithi132 ceph-mon[37580]: pgmap v10141: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:09 smithi112 ceph-mon[37501]: pgmap v10142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:09 smithi107 ceph-mon[32659]: pgmap v10142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:09 smithi132 ceph-mon[37580]: pgmap v10142: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:11 smithi107 ceph-mon[32659]: pgmap v10143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18427.168598 secs 2024-07-26T17:58:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:11 smithi132 ceph-mon[37580]: pgmap v10143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18427.168598 secs 2024-07-26T17:58:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:11 smithi112 ceph-mon[37501]: pgmap v10143: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18427.168598 secs 2024-07-26T17:58:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:13 smithi107 ceph-mon[32659]: pgmap v10144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:13 smithi132 ceph-mon[37580]: pgmap v10144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:13 smithi112 ceph-mon[37501]: pgmap v10144: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:16.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:15 smithi112 ceph-mon[37501]: pgmap v10145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:15 smithi107 ceph-mon[32659]: pgmap v10145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:15 smithi132 ceph-mon[37580]: pgmap v10145: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18432.168660 secs 2024-07-26T17:58:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18432.168660 secs 2024-07-26T17:58:17.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18432.168660 secs 2024-07-26T17:58:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:17 smithi107 ceph-mon[32659]: pgmap v10146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:17 smithi132 ceph-mon[37580]: pgmap v10146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:17 smithi112 ceph-mon[37501]: pgmap v10146: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:19 smithi107 ceph-mon[32659]: pgmap v10147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:19 smithi132 ceph-mon[37580]: pgmap v10147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:19 smithi112 ceph-mon[37501]: pgmap v10147: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:21.866+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:58:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:21.867+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:58:21.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:21.867+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:58:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:21 smithi107 ceph-mon[32659]: pgmap v10148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18437.168776 secs 2024-07-26T17:58:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:21 smithi132 ceph-mon[37580]: pgmap v10148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18437.168776 secs 2024-07-26T17:58:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:21 smithi112 ceph-mon[37501]: pgmap v10148: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18437.168776 secs 2024-07-26T17:58:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:23 smithi107 ceph-mon[32659]: pgmap v10149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:23 smithi132 ceph-mon[37580]: pgmap v10149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:23 smithi112 ceph-mon[37501]: pgmap v10149: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:26.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:25 smithi107 ceph-mon[32659]: pgmap v10150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:26.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:25 smithi132 ceph-mon[37580]: pgmap v10150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:25 smithi112 ceph-mon[37501]: pgmap v10150: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18442.168913 secs 2024-07-26T17:58:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18442.168913 secs 2024-07-26T17:58:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18442.168913 secs 2024-07-26T17:58:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:27 smithi107 ceph-mon[32659]: pgmap v10151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:27 smithi132 ceph-mon[37580]: pgmap v10151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:27 smithi112 ceph-mon[37501]: pgmap v10151: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:29 smithi107 ceph-mon[32659]: pgmap v10152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:29 smithi132 ceph-mon[37580]: pgmap v10152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:29 smithi112 ceph-mon[37501]: pgmap v10152: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:31 smithi107 ceph-mon[32659]: pgmap v10153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18447.169024 secs 2024-07-26T17:58:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:31 smithi132 ceph-mon[37580]: pgmap v10153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18447.169024 secs 2024-07-26T17:58:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:31 smithi112 ceph-mon[37501]: pgmap v10153: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18447.169024 secs 2024-07-26T17:58:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:33 smithi107 ceph-mon[32659]: pgmap v10154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:33 smithi132 ceph-mon[37580]: pgmap v10154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:33 smithi112 ceph-mon[37501]: pgmap v10154: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:35 smithi107 ceph-mon[32659]: pgmap v10155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:35 smithi132 ceph-mon[37580]: pgmap v10155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:35 smithi112 ceph-mon[37501]: pgmap v10155: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:36.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:36.882+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:58:36.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:36.882+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:58:36.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:36.882+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:58:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18452.169138 secs 2024-07-26T17:58:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18452.169138 secs 2024-07-26T17:58:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18452.169138 secs 2024-07-26T17:58:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:37 smithi132 ceph-mon[37580]: pgmap v10156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:58:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:37 smithi112 ceph-mon[37501]: pgmap v10156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:58:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:37 smithi107 ceph-mon[32659]: pgmap v10156: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:58:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:38 smithi112 ceph-mon[37501]: pgmap v10157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:38 smithi107 ceph-mon[32659]: pgmap v10157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:38 smithi132 ceph-mon[37580]: pgmap v10157: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:41 smithi107 ceph-mon[32659]: pgmap v10158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18457.169253 secs 2024-07-26T17:58:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:41 smithi132 ceph-mon[37580]: pgmap v10158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18457.169253 secs 2024-07-26T17:58:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:41 smithi112 ceph-mon[37501]: pgmap v10158: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18457.169253 secs 2024-07-26T17:58:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:43 smithi107 ceph-mon[32659]: pgmap v10159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:43 smithi132 ceph-mon[37580]: pgmap v10159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:43 smithi112 ceph-mon[37501]: pgmap v10159: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:45 smithi107 ceph-mon[32659]: pgmap v10160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:45.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:45 smithi132 ceph-mon[37580]: pgmap v10160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:45 smithi112 ceph-mon[37501]: pgmap v10160: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18462.169373 secs 2024-07-26T17:58:46.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18462.169373 secs 2024-07-26T17:58:46.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18462.169373 secs 2024-07-26T17:58:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:47 smithi107 ceph-mon[32659]: pgmap v10161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:47 smithi132 ceph-mon[37580]: pgmap v10161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:47 smithi112 ceph-mon[37501]: pgmap v10161: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:58:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:49 smithi107 ceph-mon[32659]: pgmap v10162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:49 smithi132 ceph-mon[37580]: pgmap v10162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:49 smithi112 ceph-mon[37501]: pgmap v10162: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:51 smithi107 ceph-mon[32659]: pgmap v10163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18467.169490 secs 2024-07-26T17:58:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:51 smithi132 ceph-mon[37580]: pgmap v10163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18467.169490 secs 2024-07-26T17:58:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:51 smithi112 ceph-mon[37501]: pgmap v10163: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18467.169490 secs 2024-07-26T17:58:51.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:51.897+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:58:51.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:51.897+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:58:51.900 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:58:51.897+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:58:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:53 smithi132 ceph-mon[37580]: pgmap v10164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:53 smithi112 ceph-mon[37501]: pgmap v10164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:53 smithi107 ceph-mon[32659]: pgmap v10164: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:58:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:55 smithi132 ceph-mon[37580]: pgmap v10165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:58:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:55 smithi112 ceph-mon[37501]: pgmap v10165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:55.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:58:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:55 smithi107 ceph-mon[32659]: pgmap v10165: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:58:55.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:58:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:58:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:58:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:56 smithi112 ceph-mon[37501]: pgmap v10166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:58:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:56 smithi112 ceph-mon[37501]: pgmap v10167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:58:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:58:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18472.169608 secs 2024-07-26T17:58:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:58:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:58:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:56 smithi107 ceph-mon[32659]: pgmap v10166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:58:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:56 smithi107 ceph-mon[32659]: pgmap v10167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:58:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:58:56.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18472.169608 secs 2024-07-26T17:58:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:58:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:58:56.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:56 smithi132 ceph-mon[37580]: pgmap v10166: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T17:58:56.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:56 smithi132 ceph-mon[37580]: pgmap v10167: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:58:56.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:58:56.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18472.169608 secs 2024-07-26T17:58:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:58:58 smithi112 ceph-mon[37501]: pgmap v10168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:58:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:58:58 smithi107 ceph-mon[32659]: pgmap v10168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:58:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:58:58 smithi132 ceph-mon[37580]: pgmap v10168: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:59:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:00 smithi107 ceph-mon[32659]: pgmap v10169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:59:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:00 smithi132 ceph-mon[37580]: pgmap v10169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:59:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:00 smithi112 ceph-mon[37501]: pgmap v10169: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:59:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18477.169729 secs 2024-07-26T17:59:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18477.169729 secs 2024-07-26T17:59:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18477.169729 secs 2024-07-26T17:59:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:02 smithi107 ceph-mon[32659]: pgmap v10170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:59:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:02 smithi132 ceph-mon[37580]: pgmap v10170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:59:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:02 smithi112 ceph-mon[37501]: pgmap v10170: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:59:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:04 smithi107 ceph-mon[32659]: pgmap v10171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:59:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:04 smithi132 ceph-mon[37580]: pgmap v10171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:59:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:04 smithi112 ceph-mon[37501]: pgmap v10171: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T17:59:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:06 smithi107 ceph-mon[32659]: pgmap v10172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:59:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18482.169806 secs 2024-07-26T17:59:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:06 smithi132 ceph-mon[37580]: pgmap v10172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:59:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18482.169806 secs 2024-07-26T17:59:06.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:06.913+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:59:06.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:06.913+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:59:06.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:06.913+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:59:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:06 smithi112 ceph-mon[37501]: pgmap v10172: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:59:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18482.169806 secs 2024-07-26T17:59:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:08 smithi107 ceph-mon[32659]: pgmap v10173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:08 smithi132 ceph-mon[37580]: pgmap v10173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:08 smithi112 ceph-mon[37501]: pgmap v10173: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:10 smithi107 ceph-mon[32659]: pgmap v10174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:10 smithi132 ceph-mon[37580]: pgmap v10174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:10 smithi112 ceph-mon[37501]: pgmap v10174: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18487.169935 secs 2024-07-26T17:59:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18487.169935 secs 2024-07-26T17:59:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18487.169935 secs 2024-07-26T17:59:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:12 smithi107 ceph-mon[32659]: pgmap v10175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:12 smithi132 ceph-mon[37580]: pgmap v10175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:12 smithi112 ceph-mon[37501]: pgmap v10175: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:14 smithi107 ceph-mon[32659]: pgmap v10176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:14 smithi132 ceph-mon[37580]: pgmap v10176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:14 smithi112 ceph-mon[37501]: pgmap v10176: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:16 smithi107 ceph-mon[32659]: pgmap v10177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18492.170025 secs 2024-07-26T17:59:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:16 smithi132 ceph-mon[37580]: pgmap v10177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18492.170025 secs 2024-07-26T17:59:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:16 smithi112 ceph-mon[37501]: pgmap v10177: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18492.170025 secs 2024-07-26T17:59:18.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:18 smithi107 ceph-mon[32659]: pgmap v10178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:18.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:18 smithi132 ceph-mon[37580]: pgmap v10178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:18 smithi112 ceph-mon[37501]: pgmap v10178: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:20.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:20 smithi107 ceph-mon[32659]: pgmap v10179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:20.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:20 smithi132 ceph-mon[37580]: pgmap v10179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:20 smithi112 ceph-mon[37501]: pgmap v10179: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18497.170159 secs 2024-07-26T17:59:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18497.170159 secs 2024-07-26T17:59:21.918 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:21.917+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:59:21.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:21.917+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:59:21.919 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:21.917+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:59:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18497.170159 secs 2024-07-26T17:59:22.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:22 smithi132 ceph-mon[37580]: pgmap v10180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:22 smithi112 ceph-mon[37501]: pgmap v10180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:22 smithi107 ceph-mon[32659]: pgmap v10180: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:24.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:24 smithi132 ceph-mon[37580]: pgmap v10181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:24 smithi112 ceph-mon[37501]: pgmap v10181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:24 smithi107 ceph-mon[32659]: pgmap v10181: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:26 smithi132 ceph-mon[37580]: pgmap v10182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18502.170282 secs 2024-07-26T17:59:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:26 smithi112 ceph-mon[37501]: pgmap v10182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18502.170282 secs 2024-07-26T17:59:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:26 smithi107 ceph-mon[32659]: pgmap v10182: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18502.170282 secs 2024-07-26T17:59:28.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:28 smithi132 ceph-mon[37580]: pgmap v10183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:28 smithi112 ceph-mon[37501]: pgmap v10183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:29.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:28 smithi107 ceph-mon[32659]: pgmap v10183: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:30 smithi112 ceph-mon[37501]: pgmap v10184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:30 smithi107 ceph-mon[32659]: pgmap v10184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:30 smithi132 ceph-mon[37580]: pgmap v10184: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18507.170373 secs 2024-07-26T17:59:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18507.170373 secs 2024-07-26T17:59:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18507.170373 secs 2024-07-26T17:59:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:32 smithi112 ceph-mon[37501]: pgmap v10185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:32 smithi107 ceph-mon[32659]: pgmap v10185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:32 smithi132 ceph-mon[37580]: pgmap v10185: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:34 smithi112 ceph-mon[37501]: pgmap v10186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:59:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:34 smithi107 ceph-mon[32659]: pgmap v10186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:59:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:34 smithi132 ceph-mon[37580]: pgmap v10186: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:59:36.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:36 smithi107 ceph-mon[32659]: pgmap v10187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:59:36.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18512.170464 secs 2024-07-26T17:59:36.934 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:36.933+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:59:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:36.933+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:59:36.935 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:36.933+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:59:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:36 smithi112 ceph-mon[37501]: pgmap v10187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:59:36.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18512.170464 secs 2024-07-26T17:59:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:36 smithi132 ceph-mon[37580]: pgmap v10187: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T17:59:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18512.170464 secs 2024-07-26T17:59:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:38 smithi112 ceph-mon[37501]: pgmap v10188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:59:39.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:38 smithi107 ceph-mon[32659]: pgmap v10188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:59:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:38 smithi132 ceph-mon[37580]: pgmap v10188: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:59:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:40 smithi112 ceph-mon[37501]: pgmap v10189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:40 smithi107 ceph-mon[32659]: pgmap v10189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:41.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:40 smithi132 ceph-mon[37580]: pgmap v10189: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18517.170549 secs 2024-07-26T17:59:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18517.170549 secs 2024-07-26T17:59:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18517.170549 secs 2024-07-26T17:59:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:42 smithi112 ceph-mon[37501]: pgmap v10190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:59:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:42 smithi107 ceph-mon[32659]: pgmap v10190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:59:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:42 smithi132 ceph-mon[37580]: pgmap v10190: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T17:59:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:44 smithi112 ceph-mon[37501]: pgmap v10191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:59:45.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:44 smithi107 ceph-mon[32659]: pgmap v10191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:59:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:44 smithi132 ceph-mon[37580]: pgmap v10191: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T17:59:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:46 smithi112 ceph-mon[37501]: pgmap v10192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18522.170610 secs 2024-07-26T17:59:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:46 smithi107 ceph-mon[32659]: pgmap v10192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18522.170610 secs 2024-07-26T17:59:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:46 smithi132 ceph-mon[37580]: pgmap v10192: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18522.170610 secs 2024-07-26T17:59:48.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:48 smithi112 ceph-mon[37501]: pgmap v10193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:48 smithi107 ceph-mon[32659]: pgmap v10193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:48 smithi132 ceph-mon[37580]: pgmap v10193: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:50 smithi112 ceph-mon[37501]: pgmap v10194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:50 smithi107 ceph-mon[32659]: pgmap v10194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:50 smithi132 ceph-mon[37580]: pgmap v10194: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T17:59:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:51.937+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T17:59:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:51.937+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T17:59:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T17:59:51.937+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T17:59:51.940 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18527.170732 secs 2024-07-26T17:59:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18527.170732 secs 2024-07-26T17:59:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18527.170732 secs 2024-07-26T17:59:52.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:52 smithi112 ceph-mon[37501]: pgmap v10195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:52 smithi107 ceph-mon[32659]: pgmap v10195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:52 smithi132 ceph-mon[37580]: pgmap v10195: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T17:59:54.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:54 smithi112 ceph-mon[37501]: pgmap v10196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:54 smithi107 ceph-mon[32659]: pgmap v10196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:54 smithi132 ceph-mon[37580]: pgmap v10196: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T17:59:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:55 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:59:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:55 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:59:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:55 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T17:59:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:56 smithi112 ceph-mon[37501]: pgmap v10197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18532.170812 secs 2024-07-26T17:59:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:59:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:56 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:59:56.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:56 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:59:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:56 smithi107 ceph-mon[32659]: pgmap v10197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18532.170812 secs 2024-07-26T17:59:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:59:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:56 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:59:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:56 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:59:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:56 smithi132 ceph-mon[37580]: pgmap v10197: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T17:59:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18532.170812 secs 2024-07-26T17:59:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T17:59:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:56 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T17:59:57.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:56 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T17:59:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:57 smithi112 ceph-mon[37501]: pgmap v10198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:59:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:57 smithi112 ceph-mon[37501]: pgmap v10199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:59:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:57 smithi107 ceph-mon[32659]: pgmap v10198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:59:58.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:57 smithi107 ceph-mon[32659]: pgmap v10199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:59:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:57 smithi132 ceph-mon[37580]: pgmap v10198: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T17:59:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:57 smithi132 ceph-mon[37580]: pgmap v10199: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T17:59:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 17:59:59 smithi112 ceph-mon[37501]: pgmap v10200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:00:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 17:59:59 smithi107 ceph-mon[32659]: pgmap v10200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:00:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 17:59:59 smithi132 ceph-mon[37580]: pgmap v10200: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:00:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T18:00:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T18:00:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T18:00:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18537.170954 secs 2024-07-26T18:00:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:01 smithi107 ceph-mon[32659]: pgmap v10201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:00:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18537.170954 secs 2024-07-26T18:00:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:01 smithi132 ceph-mon[37580]: pgmap v10201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:00:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18537.170954 secs 2024-07-26T18:00:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:01 smithi112 ceph-mon[37501]: pgmap v10201: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:00:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:03 smithi107 ceph-mon[32659]: pgmap v10202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:00:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:03 smithi132 ceph-mon[37580]: pgmap v10202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:00:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:03 smithi112 ceph-mon[37501]: pgmap v10202: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:00:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:05 smithi107 ceph-mon[32659]: pgmap v10203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:00:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:05 smithi132 ceph-mon[37580]: pgmap v10203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:00:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:05 smithi112 ceph-mon[37501]: pgmap v10203: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:00:06.956 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18542.171100 secs 2024-07-26T18:00:06.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:06.953+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:00:06.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:06.953+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:00:06.957 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:06.953+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:00:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18542.171100 secs 2024-07-26T18:00:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18542.171100 secs 2024-07-26T18:00:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:07 smithi107 ceph-mon[32659]: pgmap v10204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:00:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:07 smithi132 ceph-mon[37580]: pgmap v10204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:00:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:07 smithi112 ceph-mon[37501]: pgmap v10204: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:00:10.018 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:09 smithi132 ceph-mon[37580]: pgmap v10205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:09 smithi107 ceph-mon[32659]: pgmap v10205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:09 smithi112 ceph-mon[37501]: pgmap v10205: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18547.171220 secs 2024-07-26T18:00:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:11 smithi107 ceph-mon[32659]: pgmap v10206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18547.171220 secs 2024-07-26T18:00:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:11 smithi132 ceph-mon[37580]: pgmap v10206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18547.171220 secs 2024-07-26T18:00:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:11 smithi112 ceph-mon[37501]: pgmap v10206: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:14.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:13 smithi107 ceph-mon[32659]: pgmap v10207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:13 smithi132 ceph-mon[37580]: pgmap v10207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:13 smithi112 ceph-mon[37501]: pgmap v10207: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:15.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:15.850+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:00:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:15.850+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:00:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:15.851+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T18:00:15.852 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:15.851+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T18:00:16.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:15 smithi107 ceph-mon[32659]: pgmap v10208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:00:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:15 smithi132 ceph-mon[37580]: pgmap v10208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:00:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:15 smithi112 ceph-mon[37501]: pgmap v10208: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:00:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18552.171342 secs 2024-07-26T18:00:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18552.171342 secs 2024-07-26T18:00:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18552.171342 secs 2024-07-26T18:00:18.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:17 smithi107 ceph-mon[32659]: pgmap v10209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:18.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:17 smithi132 ceph-mon[37580]: pgmap v10209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:17 smithi112 ceph-mon[37501]: pgmap v10209: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:20.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:19 smithi107 ceph-mon[32659]: pgmap v10210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:19 smithi132 ceph-mon[37580]: pgmap v10210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:19 smithi112 ceph-mon[37501]: pgmap v10210: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:21.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:21.961+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:00:21.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:21.962+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:00:21.963 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:21.962+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:00:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18557.171464 secs 2024-07-26T18:00:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:21 smithi107 ceph-mon[32659]: pgmap v10211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18557.171464 secs 2024-07-26T18:00:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:21 smithi132 ceph-mon[37580]: pgmap v10211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18557.171464 secs 2024-07-26T18:00:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:21 smithi112 ceph-mon[37501]: pgmap v10211: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:23 smithi107 ceph-mon[32659]: pgmap v10212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:23 smithi132 ceph-mon[37580]: pgmap v10212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:23 smithi112 ceph-mon[37501]: pgmap v10212: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:25 smithi107 ceph-mon[32659]: pgmap v10213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:25 smithi132 ceph-mon[37580]: pgmap v10213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:25 smithi112 ceph-mon[37501]: pgmap v10213: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18562.171590 secs 2024-07-26T18:00:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18562.171590 secs 2024-07-26T18:00:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18562.171590 secs 2024-07-26T18:00:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:27 smithi132 ceph-mon[37580]: pgmap v10214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:27 smithi112 ceph-mon[37501]: pgmap v10214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:27 smithi107 ceph-mon[32659]: pgmap v10214: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:29 smithi112 ceph-mon[37501]: pgmap v10215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:29 smithi107 ceph-mon[32659]: pgmap v10215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:29 smithi132 ceph-mon[37580]: pgmap v10215: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18567.171727 secs 2024-07-26T18:00:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:31 smithi112 ceph-mon[37501]: pgmap v10216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18567.171727 secs 2024-07-26T18:00:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:31 smithi107 ceph-mon[32659]: pgmap v10216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18567.171727 secs 2024-07-26T18:00:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:31 smithi132 ceph-mon[37580]: pgmap v10216: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:33 smithi112 ceph-mon[37501]: pgmap v10217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:33 smithi107 ceph-mon[32659]: pgmap v10217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:33 smithi132 ceph-mon[37580]: pgmap v10217: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:34 smithi112 ceph-mon[37501]: pgmap v10218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:34 smithi107 ceph-mon[32659]: pgmap v10218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:34 smithi132 ceph-mon[37580]: pgmap v10218: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:36.197 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:35 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18572.171866 secs 2024-07-26T18:00:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:35 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18572.171866 secs 2024-07-26T18:00:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:35 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18572.171866 secs 2024-07-26T18:00:36.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:36.977+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:00:36.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:36.977+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:00:36.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:36.977+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:00:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:36 smithi107 ceph-mon[32659]: pgmap v10219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:36 smithi132 ceph-mon[37580]: pgmap v10219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:36 smithi112 ceph-mon[37501]: pgmap v10219: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:39 smithi112 ceph-mon[37501]: pgmap v10220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:39 smithi107 ceph-mon[32659]: pgmap v10220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:39 smithi132 ceph-mon[37580]: pgmap v10220: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18577.172003 secs 2024-07-26T18:00:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:41 smithi112 ceph-mon[37501]: pgmap v10221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18577.172003 secs 2024-07-26T18:00:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:41 smithi107 ceph-mon[32659]: pgmap v10221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18577.172003 secs 2024-07-26T18:00:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:41 smithi132 ceph-mon[37580]: pgmap v10221: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:43.956 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:43 smithi112 ceph-mon[37501]: pgmap v10222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:43 smithi107 ceph-mon[32659]: pgmap v10222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:43 smithi132 ceph-mon[37580]: pgmap v10222: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:45 smithi107 ceph-mon[32659]: pgmap v10223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:45 smithi132 ceph-mon[37580]: pgmap v10223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:45 smithi112 ceph-mon[37501]: pgmap v10223: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18582.172131 secs 2024-07-26T18:00:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18582.172131 secs 2024-07-26T18:00:47.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18582.172131 secs 2024-07-26T18:00:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:47 smithi107 ceph-mon[32659]: pgmap v10224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:47 smithi132 ceph-mon[37580]: pgmap v10224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:47 smithi112 ceph-mon[37501]: pgmap v10224: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:49 smithi107 ceph-mon[32659]: pgmap v10225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:49 smithi132 ceph-mon[37580]: pgmap v10225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:49 smithi112 ceph-mon[37501]: pgmap v10225: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:51.994 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:51.992+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:00:51.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:51.992+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:00:51.995 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:00:51.992+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:00:51.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18587.172249 secs 2024-07-26T18:00:51.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:51 smithi107 ceph-mon[32659]: pgmap v10226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18587.172249 secs 2024-07-26T18:00:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:51 smithi132 ceph-mon[37580]: pgmap v10226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18587.172249 secs 2024-07-26T18:00:52.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:51 smithi112 ceph-mon[37501]: pgmap v10226: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:00:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:53 smithi107 ceph-mon[32659]: pgmap v10227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:53 smithi132 ceph-mon[37580]: pgmap v10227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:53 smithi112 ceph-mon[37501]: pgmap v10227: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:00:56.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:55 smithi107 ceph-mon[32659]: pgmap v10228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:55 smithi132 ceph-mon[37580]: pgmap v10228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:55 smithi112 ceph-mon[37501]: pgmap v10228: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18592.172368 secs 2024-07-26T18:00:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18592.172368 secs 2024-07-26T18:00:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18592.172368 secs 2024-07-26T18:00:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:57 smithi107 ceph-mon[32659]: pgmap v10229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:58.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:57 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:00:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:57 smithi132 ceph-mon[37580]: pgmap v10229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:57 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:00:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:57 smithi112 ceph-mon[37501]: pgmap v10229: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:00:58.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:57 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:00:59.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T18:00:59.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T18:00:59.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:00:59.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:00:59.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: pgmap v10230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:00:59.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: pgmap v10231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:00:59.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:58 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:00:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T18:00:59.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T18:00:59.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:00:59.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:00:59.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: pgmap v10230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:00:59.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: pgmap v10231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:00:59.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:58 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:00:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:00:59.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T18:00:59.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T18:00:59.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:00:59.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:00:59.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: pgmap v10230: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:00:59.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: pgmap v10231: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:00:59.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:58 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:01:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:00:59 smithi112 ceph-mon[37501]: pgmap v10232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:01:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:00:59 smithi107 ceph-mon[32659]: pgmap v10232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:01:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:00:59 smithi132 ceph-mon[37580]: pgmap v10232: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:01:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:00 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18597.172468 secs 2024-07-26T18:01:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:00 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18597.172468 secs 2024-07-26T18:01:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:00 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18597.172468 secs 2024-07-26T18:01:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:01 smithi112 ceph-mon[37501]: pgmap v10233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:01:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:01 smithi107 ceph-mon[32659]: pgmap v10233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:01:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:01 smithi132 ceph-mon[37580]: pgmap v10233: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:01:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:03 smithi112 ceph-mon[37501]: pgmap v10234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:01:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:03 smithi107 ceph-mon[32659]: pgmap v10234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:01:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:03 smithi132 ceph-mon[37580]: pgmap v10234: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:01:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:05 smithi112 ceph-mon[37501]: pgmap v10235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:01:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:05 smithi107 ceph-mon[32659]: pgmap v10235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:01:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:05 smithi132 ceph-mon[37580]: pgmap v10235: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:01:07.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:07.008+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:01:07.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:07.008+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:01:07.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:07.008+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:01:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18602.172587 secs 2024-07-26T18:01:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18602.172587 secs 2024-07-26T18:01:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18602.172587 secs 2024-07-26T18:01:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:07 smithi112 ceph-mon[37501]: pgmap v10236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:01:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:07 smithi107 ceph-mon[32659]: pgmap v10236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:01:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:07 smithi132 ceph-mon[37580]: pgmap v10236: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:01:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:10 smithi112 ceph-mon[37501]: pgmap v10237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:10 smithi107 ceph-mon[32659]: pgmap v10237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:10 smithi132 ceph-mon[37580]: pgmap v10237: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18607.172723 secs 2024-07-26T18:01:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18607.172723 secs 2024-07-26T18:01:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18607.172723 secs 2024-07-26T18:01:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:12 smithi112 ceph-mon[37501]: pgmap v10238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:12 smithi107 ceph-mon[32659]: pgmap v10238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:12 smithi132 ceph-mon[37580]: pgmap v10238: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:14 smithi112 ceph-mon[37501]: pgmap v10239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:14 smithi107 ceph-mon[32659]: pgmap v10239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:14 smithi132 ceph-mon[37580]: pgmap v10239: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:15 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18612.172841 secs 2024-07-26T18:01:16.212 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18612.172841 secs 2024-07-26T18:01:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18612.172841 secs 2024-07-26T18:01:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:16 smithi112 ceph-mon[37501]: pgmap v10240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:16 smithi107 ceph-mon[32659]: pgmap v10240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:16 smithi132 ceph-mon[37580]: pgmap v10240: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:18 smithi112 ceph-mon[37501]: pgmap v10241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:18 smithi107 ceph-mon[32659]: pgmap v10241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:18 smithi132 ceph-mon[37580]: pgmap v10241: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:21 smithi112 ceph-mon[37501]: pgmap v10242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:21.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18617.172944 secs 2024-07-26T18:01:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:21 smithi107 ceph-mon[32659]: pgmap v10242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18617.172944 secs 2024-07-26T18:01:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:21 smithi132 ceph-mon[37580]: pgmap v10242: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18617.172944 secs 2024-07-26T18:01:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:22.024+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:01:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:22.024+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:01:22.026 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:22.024+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:01:23.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:23 smithi112 ceph-mon[37501]: pgmap v10243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:23 smithi107 ceph-mon[32659]: pgmap v10243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:23 smithi132 ceph-mon[37580]: pgmap v10243: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:25.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:25 smithi112 ceph-mon[37501]: pgmap v10244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:25 smithi107 ceph-mon[32659]: pgmap v10244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:25 smithi132 ceph-mon[37580]: pgmap v10244: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18622.173098 secs 2024-07-26T18:01:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18622.173098 secs 2024-07-26T18:01:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18622.173098 secs 2024-07-26T18:01:27.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:27 smithi112 ceph-mon[37501]: pgmap v10245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:27 smithi107 ceph-mon[32659]: pgmap v10245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:27 smithi132 ceph-mon[37580]: pgmap v10245: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:29.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:29 smithi112 ceph-mon[37501]: pgmap v10246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:29.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:29 smithi107 ceph-mon[32659]: pgmap v10246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:29 smithi132 ceph-mon[37580]: pgmap v10246: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:31 smithi112 ceph-mon[37501]: pgmap v10247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:31.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18627.173222 secs 2024-07-26T18:01:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:31 smithi107 ceph-mon[32659]: pgmap v10247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18627.173222 secs 2024-07-26T18:01:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:31 smithi132 ceph-mon[37580]: pgmap v10247: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18627.173222 secs 2024-07-26T18:01:33.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:33 smithi112 ceph-mon[37501]: pgmap v10248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:33 smithi107 ceph-mon[32659]: pgmap v10248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:33 smithi132 ceph-mon[37580]: pgmap v10248: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:35 smithi112 ceph-mon[37501]: pgmap v10249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:35 smithi107 ceph-mon[32659]: pgmap v10249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:35 smithi132 ceph-mon[37580]: pgmap v10249: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:36.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18632.173343 secs 2024-07-26T18:01:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18632.173343 secs 2024-07-26T18:01:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18632.173343 secs 2024-07-26T18:01:37.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:37.025+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:01:37.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:37.025+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:01:37.027 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:37.025+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:01:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:37 smithi112 ceph-mon[37501]: pgmap v10250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:37 smithi107 ceph-mon[32659]: pgmap v10250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:37 smithi132 ceph-mon[37580]: pgmap v10250: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:39 smithi112 ceph-mon[37501]: pgmap v10251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:39 smithi107 ceph-mon[32659]: pgmap v10251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:39 smithi132 ceph-mon[37580]: pgmap v10251: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:41 smithi112 ceph-mon[37501]: pgmap v10252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:41.702 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18637.173475 secs 2024-07-26T18:01:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:41 smithi107 ceph-mon[32659]: pgmap v10252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18637.173475 secs 2024-07-26T18:01:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:41 smithi132 ceph-mon[37580]: pgmap v10252: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18637.173475 secs 2024-07-26T18:01:43.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:43 smithi112 ceph-mon[37501]: pgmap v10253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:43 smithi107 ceph-mon[32659]: pgmap v10253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:43 smithi132 ceph-mon[37580]: pgmap v10253: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:45 smithi112 ceph-mon[37501]: pgmap v10254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:45.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:45 smithi107 ceph-mon[32659]: pgmap v10254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:45 smithi132 ceph-mon[37580]: pgmap v10254: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18642.173562 secs 2024-07-26T18:01:46.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18642.173562 secs 2024-07-26T18:01:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18642.173562 secs 2024-07-26T18:01:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:47 smithi112 ceph-mon[37501]: pgmap v10255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:47 smithi107 ceph-mon[32659]: pgmap v10255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:47 smithi132 ceph-mon[37580]: pgmap v10255: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:49.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:49 smithi112 ceph-mon[37501]: pgmap v10256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:49 smithi107 ceph-mon[32659]: pgmap v10256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:49 smithi132 ceph-mon[37580]: pgmap v10256: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:51 smithi112 ceph-mon[37501]: pgmap v10257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18647.173710 secs 2024-07-26T18:01:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:51 smithi107 ceph-mon[32659]: pgmap v10257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18647.173710 secs 2024-07-26T18:01:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:51 smithi132 ceph-mon[37580]: pgmap v10257: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18647.173710 secs 2024-07-26T18:01:52.030 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:52.028+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:01:52.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:52.028+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:01:52.031 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:01:52.028+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:01:53.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:53 smithi112 ceph-mon[37501]: pgmap v10258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:53 smithi107 ceph-mon[32659]: pgmap v10258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:53 smithi132 ceph-mon[37580]: pgmap v10258: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:01:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:55 smithi107 ceph-mon[32659]: pgmap v10259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:55 smithi132 ceph-mon[37580]: pgmap v10259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:55 smithi112 ceph-mon[37501]: pgmap v10259: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18652.173843 secs 2024-07-26T18:01:56.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18652.173843 secs 2024-07-26T18:01:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18652.173843 secs 2024-07-26T18:01:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:57 smithi107 ceph-mon[32659]: pgmap v10260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:57 smithi132 ceph-mon[37580]: pgmap v10260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:57.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:57 smithi112 ceph-mon[37501]: pgmap v10260: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:01:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:58 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:01:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:58 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:01:58.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:58 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:01:59.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: pgmap v10261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:59.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T18:01:59.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T18:01:59.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T18:01:59.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T18:01:59.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:01:59.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:01:59.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:01:59 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:01:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: pgmap v10261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:59.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.812 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T18:01:59.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T18:01:59.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T18:01:59.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T18:01:59.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:01:59.813 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:01:59.814 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:01:59 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:01:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: pgmap v10261: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:01:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.952 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:01:59.953 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T18:01:59.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T18:01:59.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T18:01:59.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T18:01:59.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:01:59.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:01:59.954 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:01:59 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:02:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:00 smithi107 ceph-mon[32659]: pgmap v10262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:02:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:00 smithi107 ceph-mon[32659]: pgmap v10263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:02:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:00 smithi132 ceph-mon[37580]: pgmap v10262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:02:00.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:00 smithi132 ceph-mon[37580]: pgmap v10263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:02:00.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:00 smithi112 ceph-mon[37501]: pgmap v10262: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:02:00.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:00 smithi112 ceph-mon[37501]: pgmap v10263: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:02:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18657.173957 secs 2024-07-26T18:02:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18657.173957 secs 2024-07-26T18:02:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18657.173957 secs 2024-07-26T18:02:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:02 smithi107 ceph-mon[32659]: pgmap v10264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:02:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:02 smithi132 ceph-mon[37580]: pgmap v10264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:02:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:02 smithi112 ceph-mon[37501]: pgmap v10264: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:02:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:04 smithi107 ceph-mon[32659]: pgmap v10265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:02:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:04 smithi132 ceph-mon[37580]: pgmap v10265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:02:04.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:04 smithi112 ceph-mon[37501]: pgmap v10265: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:02:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:06 smithi107 ceph-mon[32659]: pgmap v10266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18662.174111 secs 2024-07-26T18:02:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:06 smithi132 ceph-mon[37580]: pgmap v10266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18662.174111 secs 2024-07-26T18:02:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:06 smithi112 ceph-mon[37501]: pgmap v10266: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:06.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18662.174111 secs 2024-07-26T18:02:07.046 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:07.044+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:02:07.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:07.044+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:02:07.047 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:07.044+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:02:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:08 smithi107 ceph-mon[32659]: pgmap v10267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:08 smithi132 ceph-mon[37580]: pgmap v10267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:08.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:08 smithi112 ceph-mon[37501]: pgmap v10267: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:10 smithi107 ceph-mon[32659]: pgmap v10268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:02:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:10 smithi132 ceph-mon[37580]: pgmap v10268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:02:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:10 smithi112 ceph-mon[37501]: pgmap v10268: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:02:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18667.174226 secs 2024-07-26T18:02:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18667.174226 secs 2024-07-26T18:02:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18667.174226 secs 2024-07-26T18:02:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:12 smithi132 ceph-mon[37580]: pgmap v10269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:12.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:12 smithi107 ceph-mon[32659]: pgmap v10269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:12.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:12 smithi112 ceph-mon[37501]: pgmap v10269: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:14.814 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:14 smithi132 ceph-mon[37580]: pgmap v10270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:14.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:14 smithi112 ceph-mon[37501]: pgmap v10270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:15.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:14 smithi107 ceph-mon[32659]: pgmap v10270: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:16 smithi112 ceph-mon[37501]: pgmap v10271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:16.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18672.174348 secs 2024-07-26T18:02:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:16 smithi107 ceph-mon[32659]: pgmap v10271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18672.174348 secs 2024-07-26T18:02:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:16 smithi132 ceph-mon[37580]: pgmap v10271: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18672.174348 secs 2024-07-26T18:02:18.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:18 smithi112 ceph-mon[37501]: pgmap v10272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:18 smithi107 ceph-mon[32659]: pgmap v10272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:18 smithi132 ceph-mon[37580]: pgmap v10272: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:20.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:20 smithi112 ceph-mon[37501]: pgmap v10273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:20 smithi107 ceph-mon[32659]: pgmap v10273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:20 smithi132 ceph-mon[37580]: pgmap v10273: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18677.174435 secs 2024-07-26T18:02:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18677.174435 secs 2024-07-26T18:02:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18677.174435 secs 2024-07-26T18:02:22.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:22.060+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:02:22.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:22.060+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:02:22.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:22.060+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:02:22.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:22 smithi112 ceph-mon[37501]: pgmap v10274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:22 smithi107 ceph-mon[32659]: pgmap v10274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:22 smithi132 ceph-mon[37580]: pgmap v10274: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:24 smithi112 ceph-mon[37501]: pgmap v10275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:24 smithi107 ceph-mon[32659]: pgmap v10275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:24 smithi132 ceph-mon[37580]: pgmap v10275: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:26 smithi112 ceph-mon[37501]: pgmap v10276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:26.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18682.174557 secs 2024-07-26T18:02:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:26 smithi107 ceph-mon[32659]: pgmap v10276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18682.174557 secs 2024-07-26T18:02:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:26 smithi132 ceph-mon[37580]: pgmap v10276: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18682.174557 secs 2024-07-26T18:02:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:28 smithi112 ceph-mon[37501]: pgmap v10277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:28 smithi107 ceph-mon[32659]: pgmap v10277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:28 smithi132 ceph-mon[37580]: pgmap v10277: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:30 smithi112 ceph-mon[37501]: pgmap v10278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:30 smithi107 ceph-mon[32659]: pgmap v10278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:30 smithi132 ceph-mon[37580]: pgmap v10278: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18687.174677 secs 2024-07-26T18:02:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18687.174677 secs 2024-07-26T18:02:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18687.174677 secs 2024-07-26T18:02:32.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:32 smithi112 ceph-mon[37501]: pgmap v10279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:02:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:32 smithi107 ceph-mon[32659]: pgmap v10279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:02:33.065 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:32 smithi132 ceph-mon[37580]: pgmap v10279: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:02:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:34 smithi112 ceph-mon[37501]: pgmap v10280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:34 smithi107 ceph-mon[32659]: pgmap v10280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:34 smithi132 ceph-mon[37580]: pgmap v10280: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:36.956 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:36 smithi112 ceph-mon[37501]: pgmap v10281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:02:36.957 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18692.174818 secs 2024-07-26T18:02:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:36 smithi107 ceph-mon[32659]: pgmap v10281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:02:37.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18692.174818 secs 2024-07-26T18:02:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:36 smithi132 ceph-mon[37580]: pgmap v10281: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:02:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18692.174818 secs 2024-07-26T18:02:37.077 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:37.076+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:02:37.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:37.076+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:02:37.078 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:37.076+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:02:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:38 smithi112 ceph-mon[37501]: pgmap v10282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:02:39.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:38 smithi107 ceph-mon[32659]: pgmap v10282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:02:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:38 smithi132 ceph-mon[37580]: pgmap v10282: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:02:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:40 smithi112 ceph-mon[37501]: pgmap v10283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:40 smithi107 ceph-mon[32659]: pgmap v10283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:40 smithi132 ceph-mon[37580]: pgmap v10283: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:02:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18697.174948 secs 2024-07-26T18:02:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18697.174948 secs 2024-07-26T18:02:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18697.174948 secs 2024-07-26T18:02:42.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:42 smithi112 ceph-mon[37501]: pgmap v10284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:42 smithi107 ceph-mon[32659]: pgmap v10284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:42 smithi132 ceph-mon[37580]: pgmap v10284: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:44 smithi112 ceph-mon[37501]: pgmap v10285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:45.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:44 smithi107 ceph-mon[32659]: pgmap v10285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:44 smithi132 ceph-mon[37580]: pgmap v10285: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:46 smithi112 ceph-mon[37501]: pgmap v10286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18702.175102 secs 2024-07-26T18:02:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:46 smithi107 ceph-mon[32659]: pgmap v10286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:47.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18702.175102 secs 2024-07-26T18:02:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:46 smithi132 ceph-mon[37580]: pgmap v10286: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18702.175102 secs 2024-07-26T18:02:48.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:48 smithi112 ceph-mon[37501]: pgmap v10287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:48 smithi107 ceph-mon[32659]: pgmap v10287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:48 smithi132 ceph-mon[37580]: pgmap v10287: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:50.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:50 smithi112 ceph-mon[37501]: pgmap v10288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:51.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:50 smithi107 ceph-mon[32659]: pgmap v10288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:50 smithi132 ceph-mon[37580]: pgmap v10288: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18707.175213 secs 2024-07-26T18:02:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18707.175213 secs 2024-07-26T18:02:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18707.175213 secs 2024-07-26T18:02:52.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:52.091+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:02:52.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:52.091+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:02:52.093 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:02:52.091+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:02:53.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:52 smithi107 ceph-mon[32659]: pgmap v10289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:52 smithi132 ceph-mon[37580]: pgmap v10289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:52 smithi112 ceph-mon[37501]: pgmap v10289: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:02:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:54 smithi107 ceph-mon[32659]: pgmap v10290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:54 smithi132 ceph-mon[37580]: pgmap v10290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:54 smithi112 ceph-mon[37501]: pgmap v10290: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:02:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:56 smithi107 ceph-mon[32659]: pgmap v10291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:57.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18712.175332 secs 2024-07-26T18:02:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:56 smithi132 ceph-mon[37580]: pgmap v10291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18712.175332 secs 2024-07-26T18:02:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:56 smithi112 ceph-mon[37501]: pgmap v10291: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18712.175332 secs 2024-07-26T18:02:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:58 smithi107 ceph-mon[32659]: pgmap v10292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:58 smithi132 ceph-mon[37580]: pgmap v10292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:02:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:58 smithi112 ceph-mon[37501]: pgmap v10292: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:02:59 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:03:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:02:59 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:03:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:02:59 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:03:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:00 smithi107 ceph-mon[32659]: pgmap v10293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:03:01.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:00 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:03:01.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:00 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:03:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:00 smithi132 ceph-mon[37580]: pgmap v10293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:03:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:00 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:03:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:00 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:03:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:00 smithi112 ceph-mon[37501]: pgmap v10293: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:03:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:00 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:03:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:00 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:03:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:01 smithi107 ceph-mon[32659]: pgmap v10294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T18:03:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:01 smithi107 ceph-mon[32659]: pgmap v10295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:03:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18717.175408 secs 2024-07-26T18:03:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:01 smithi132 ceph-mon[37580]: pgmap v10294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T18:03:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:01 smithi132 ceph-mon[37580]: pgmap v10295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:03:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18717.175408 secs 2024-07-26T18:03:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:01 smithi112 ceph-mon[37501]: pgmap v10294: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T18:03:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:01 smithi112 ceph-mon[37501]: pgmap v10295: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:03:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18717.175408 secs 2024-07-26T18:03:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:03 smithi107 ceph-mon[32659]: pgmap v10296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:03:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:03 smithi132 ceph-mon[37580]: pgmap v10296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:03:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:03 smithi112 ceph-mon[37501]: pgmap v10296: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:03:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:05 smithi107 ceph-mon[32659]: pgmap v10297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:03:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:05 smithi132 ceph-mon[37580]: pgmap v10297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:03:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:05 smithi112 ceph-mon[37501]: pgmap v10297: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:03:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18722.175531 secs 2024-07-26T18:03:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18722.175531 secs 2024-07-26T18:03:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:07.107+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:03:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:07.107+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:03:07.109 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:07.107+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:03:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18722.175531 secs 2024-07-26T18:03:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:07 smithi107 ceph-mon[32659]: pgmap v10298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:03:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:07 smithi132 ceph-mon[37580]: pgmap v10298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:03:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:07 smithi112 ceph-mon[37501]: pgmap v10298: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:03:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:09 smithi107 ceph-mon[32659]: pgmap v10299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:03:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:09 smithi132 ceph-mon[37580]: pgmap v10299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:03:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:09 smithi112 ceph-mon[37501]: pgmap v10299: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:03:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:11 smithi107 ceph-mon[32659]: pgmap v10300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:03:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18727.175589 secs 2024-07-26T18:03:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:11 smithi132 ceph-mon[37580]: pgmap v10300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:03:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18727.175589 secs 2024-07-26T18:03:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:11 smithi112 ceph-mon[37501]: pgmap v10300: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:03:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18727.175589 secs 2024-07-26T18:03:14.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:13 smithi132 ceph-mon[37580]: pgmap v10301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:13 smithi112 ceph-mon[37501]: pgmap v10301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:13 smithi107 ceph-mon[32659]: pgmap v10301: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:15.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:15.881+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:03:15.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:15.881+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:03:15.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:15.881+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T18:03:15.884 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:15.881+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T18:03:16.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:15 smithi132 ceph-mon[37580]: pgmap v10302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:15 smithi112 ceph-mon[37501]: pgmap v10302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:15 smithi107 ceph-mon[32659]: pgmap v10302: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18732.175716 secs 2024-07-26T18:03:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18732.175716 secs 2024-07-26T18:03:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18732.175716 secs 2024-07-26T18:03:18.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:17 smithi112 ceph-mon[37501]: pgmap v10303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:17 smithi107 ceph-mon[32659]: pgmap v10303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:17 smithi132 ceph-mon[37580]: pgmap v10303: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:19 smithi112 ceph-mon[37501]: pgmap v10304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:19 smithi107 ceph-mon[32659]: pgmap v10304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:19 smithi132 ceph-mon[37580]: pgmap v10304: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:22.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:22.123+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:03:22.125 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:22.123+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:03:22.126 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:22.123+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:03:22.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:21 smithi107 ceph-mon[32659]: pgmap v10305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:22.126 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18737.175840 secs 2024-07-26T18:03:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:21 smithi112 ceph-mon[37501]: pgmap v10305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18737.175840 secs 2024-07-26T18:03:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:21 smithi132 ceph-mon[37580]: pgmap v10305: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18737.175840 secs 2024-07-26T18:03:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:23 smithi112 ceph-mon[37501]: pgmap v10306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:23 smithi107 ceph-mon[32659]: pgmap v10306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:23 smithi132 ceph-mon[37580]: pgmap v10306: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:25 smithi112 ceph-mon[37501]: pgmap v10307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:25 smithi107 ceph-mon[32659]: pgmap v10307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:25 smithi132 ceph-mon[37580]: pgmap v10307: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18742.175945 secs 2024-07-26T18:03:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18742.175945 secs 2024-07-26T18:03:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18742.175945 secs 2024-07-26T18:03:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:27 smithi112 ceph-mon[37501]: pgmap v10308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:27 smithi107 ceph-mon[32659]: pgmap v10308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:27 smithi132 ceph-mon[37580]: pgmap v10308: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:29 smithi112 ceph-mon[37501]: pgmap v10309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:29 smithi107 ceph-mon[32659]: pgmap v10309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:29 smithi132 ceph-mon[37580]: pgmap v10309: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:31 smithi112 ceph-mon[37501]: pgmap v10310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18747.176106 secs 2024-07-26T18:03:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:31 smithi107 ceph-mon[32659]: pgmap v10310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18747.176106 secs 2024-07-26T18:03:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:31 smithi132 ceph-mon[37580]: pgmap v10310: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18747.176106 secs 2024-07-26T18:03:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:32 smithi112 ceph-mon[37501]: pgmap v10311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:32 smithi107 ceph-mon[32659]: pgmap v10311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:32 smithi132 ceph-mon[37580]: pgmap v10311: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:35 smithi132 ceph-mon[37580]: pgmap v10312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:35.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:35 smithi112 ceph-mon[37501]: pgmap v10312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:35 smithi107 ceph-mon[32659]: pgmap v10312: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18752.176226 secs 2024-07-26T18:03:36.693 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18752.176226 secs 2024-07-26T18:03:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18752.176226 secs 2024-07-26T18:03:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:37.132+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:03:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:37.132+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:03:37.134 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:37.132+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:03:37.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:37 smithi107 ceph-mon[32659]: pgmap v10313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:37.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:37 smithi112 ceph-mon[37501]: pgmap v10313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:37 smithi132 ceph-mon[37580]: pgmap v10313: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:39.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:39 smithi112 ceph-mon[37501]: pgmap v10314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:39.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:39 smithi107 ceph-mon[32659]: pgmap v10314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:39 smithi132 ceph-mon[37580]: pgmap v10314: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:41 smithi112 ceph-mon[37501]: pgmap v10315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:41.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18757.176292 secs 2024-07-26T18:03:41.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:41 smithi107 ceph-mon[32659]: pgmap v10315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:41.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18757.176292 secs 2024-07-26T18:03:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:41 smithi132 ceph-mon[37580]: pgmap v10315: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18757.176292 secs 2024-07-26T18:03:43.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:43 smithi112 ceph-mon[37501]: pgmap v10316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:43.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:43 smithi107 ceph-mon[32659]: pgmap v10316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:43 smithi132 ceph-mon[37580]: pgmap v10316: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:45.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:45 smithi112 ceph-mon[37501]: pgmap v10317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:03:45.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:45 smithi107 ceph-mon[32659]: pgmap v10317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:03:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:45 smithi132 ceph-mon[37580]: pgmap v10317: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:03:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18762.176383 secs 2024-07-26T18:03:46.711 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18762.176383 secs 2024-07-26T18:03:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18762.176383 secs 2024-07-26T18:03:47.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:47 smithi112 ceph-mon[37501]: pgmap v10318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T18:03:47.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:47 smithi107 ceph-mon[32659]: pgmap v10318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T18:03:47.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:47 smithi132 ceph-mon[37580]: pgmap v10318: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T18:03:49.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:49 smithi112 ceph-mon[37501]: pgmap v10319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:03:49.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:49 smithi107 ceph-mon[32659]: pgmap v10319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:03:49.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:49 smithi132 ceph-mon[37580]: pgmap v10319: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:03:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:51 smithi112 ceph-mon[37501]: pgmap v10320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:51.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18767.176449 secs 2024-07-26T18:03:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:51 smithi107 ceph-mon[32659]: pgmap v10320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18767.176449 secs 2024-07-26T18:03:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:51 smithi132 ceph-mon[37580]: pgmap v10320: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:51.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18767.176449 secs 2024-07-26T18:03:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:52.146+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:03:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:52.146+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:03:52.149 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:03:52.146+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:03:53.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:53 smithi112 ceph-mon[37501]: pgmap v10321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:53.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:53 smithi107 ceph-mon[32659]: pgmap v10321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:53.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:53 smithi132 ceph-mon[37580]: pgmap v10321: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:03:55.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:55 smithi112 ceph-mon[37501]: pgmap v10322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:55.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:55 smithi107 ceph-mon[32659]: pgmap v10322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:55.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:55 smithi132 ceph-mon[37580]: pgmap v10322: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18772.176591 secs 2024-07-26T18:03:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18772.176591 secs 2024-07-26T18:03:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18772.176591 secs 2024-07-26T18:03:57.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:57 smithi112 ceph-mon[37501]: pgmap v10323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:57.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:57 smithi107 ceph-mon[32659]: pgmap v10323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:57.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:57 smithi132 ceph-mon[37580]: pgmap v10323: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:03:59.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:03:59 smithi112 ceph-mon[37501]: pgmap v10324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:59.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:03:59 smithi107 ceph-mon[32659]: pgmap v10324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:03:59.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:03:59 smithi132 ceph-mon[37580]: pgmap v10324: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:01 smithi112 ceph-mon[37501]: pgmap v10325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:01 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:04:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18777.176728 secs 2024-07-26T18:04:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:04:01.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:01 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:04:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:01 smithi107 ceph-mon[32659]: pgmap v10325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:01 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:04:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18777.176728 secs 2024-07-26T18:04:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:04:01.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:01 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:04:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:01 smithi132 ceph-mon[37580]: pgmap v10325: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:01 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:04:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18777.176728 secs 2024-07-26T18:04:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:04:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:01 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:04:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:04:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:04:02.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:02 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:04:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:04:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:04:02.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:02 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:04:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:04:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:04:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:02 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:04:03.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:03 smithi112 ceph-mon[37501]: pgmap v10326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:04:03.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:03 smithi112 ceph-mon[37501]: pgmap v10327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:03.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:03 smithi107 ceph-mon[32659]: pgmap v10326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:04:03.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:03 smithi107 ceph-mon[32659]: pgmap v10327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:03 smithi132 ceph-mon[37580]: pgmap v10326: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:04:03.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:03 smithi132 ceph-mon[37580]: pgmap v10327: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:05 smithi112 ceph-mon[37501]: pgmap v10328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:05.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:05 smithi107 ceph-mon[32659]: pgmap v10328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:05.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:05 smithi132 ceph-mon[37580]: pgmap v10328: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:06.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18782.176854 secs 2024-07-26T18:04:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18782.176854 secs 2024-07-26T18:04:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18782.176854 secs 2024-07-26T18:04:07.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:07.160+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:04:07.163 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:07.160+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:04:07.164 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:07.160+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:04:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:07 smithi112 ceph-mon[37501]: pgmap v10329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:07 smithi107 ceph-mon[32659]: pgmap v10329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:07.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:07 smithi132 ceph-mon[37580]: pgmap v10329: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:09 smithi112 ceph-mon[37501]: pgmap v10330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:09 smithi107 ceph-mon[32659]: pgmap v10330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:09 smithi132 ceph-mon[37580]: pgmap v10330: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:11 smithi112 ceph-mon[37501]: pgmap v10331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18787.176975 secs 2024-07-26T18:04:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:11 smithi107 ceph-mon[32659]: pgmap v10331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18787.176975 secs 2024-07-26T18:04:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:11 smithi132 ceph-mon[37580]: pgmap v10331: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:04:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18787.176975 secs 2024-07-26T18:04:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:13 smithi112 ceph-mon[37501]: pgmap v10332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:04:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:13 smithi107 ceph-mon[32659]: pgmap v10332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:04:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:13 smithi132 ceph-mon[37580]: pgmap v10332: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:04:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:15 smithi112 ceph-mon[37501]: pgmap v10333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:15 smithi107 ceph-mon[32659]: pgmap v10333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:15 smithi132 ceph-mon[37580]: pgmap v10333: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18792.177095 secs 2024-07-26T18:04:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18792.177095 secs 2024-07-26T18:04:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18792.177095 secs 2024-07-26T18:04:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:17 smithi107 ceph-mon[32659]: pgmap v10334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:17 smithi132 ceph-mon[37580]: pgmap v10334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:17 smithi112 ceph-mon[37501]: pgmap v10334: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:19.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:19 smithi112 ceph-mon[37501]: pgmap v10335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:19 smithi107 ceph-mon[32659]: pgmap v10335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:19 smithi132 ceph-mon[37580]: pgmap v10335: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:21 smithi107 ceph-mon[32659]: pgmap v10336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18797.177187 secs 2024-07-26T18:04:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:21 smithi132 ceph-mon[37580]: pgmap v10336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18797.177187 secs 2024-07-26T18:04:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:21 smithi112 ceph-mon[37501]: pgmap v10336: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18797.177187 secs 2024-07-26T18:04:22.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:22.175+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:04:22.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:22.175+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:04:22.177 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:22.175+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:04:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:23 smithi107 ceph-mon[32659]: pgmap v10337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:23.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:23 smithi132 ceph-mon[37580]: pgmap v10337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:23 smithi112 ceph-mon[37501]: pgmap v10337: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:25 smithi107 ceph-mon[32659]: pgmap v10338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:25 smithi132 ceph-mon[37580]: pgmap v10338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:25 smithi112 ceph-mon[37501]: pgmap v10338: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18802.177280 secs 2024-07-26T18:04:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18802.177280 secs 2024-07-26T18:04:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18802.177280 secs 2024-07-26T18:04:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:27 smithi107 ceph-mon[32659]: pgmap v10339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:27 smithi132 ceph-mon[37580]: pgmap v10339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:27 smithi112 ceph-mon[37501]: pgmap v10339: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:29 smithi107 ceph-mon[32659]: pgmap v10340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:29 smithi132 ceph-mon[37580]: pgmap v10340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:29 smithi112 ceph-mon[37501]: pgmap v10340: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:31 smithi107 ceph-mon[32659]: pgmap v10341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18807.177347 secs 2024-07-26T18:04:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:31 smithi132 ceph-mon[37580]: pgmap v10341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18807.177347 secs 2024-07-26T18:04:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:31 smithi112 ceph-mon[37501]: pgmap v10341: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18807.177347 secs 2024-07-26T18:04:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:33 smithi107 ceph-mon[32659]: pgmap v10342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:33 smithi132 ceph-mon[37580]: pgmap v10342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:33 smithi112 ceph-mon[37501]: pgmap v10342: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:35 smithi107 ceph-mon[32659]: pgmap v10343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:35 smithi132 ceph-mon[37580]: pgmap v10343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:35 smithi112 ceph-mon[37501]: pgmap v10343: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18812.177460 secs 2024-07-26T18:04:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18812.177460 secs 2024-07-26T18:04:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18812.177460 secs 2024-07-26T18:04:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:37.191+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:04:37.192 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:37.191+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:04:37.193 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:37.191+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:04:37.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:37 smithi107 ceph-mon[32659]: pgmap v10344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:37 smithi132 ceph-mon[37580]: pgmap v10344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:37.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:37 smithi112 ceph-mon[37501]: pgmap v10344: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:39 smithi132 ceph-mon[37580]: pgmap v10345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:39 smithi112 ceph-mon[37501]: pgmap v10345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:40.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:39 smithi107 ceph-mon[32659]: pgmap v10345: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:41 smithi132 ceph-mon[37580]: pgmap v10346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:41.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18817.177578 secs 2024-07-26T18:04:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:41 smithi112 ceph-mon[37501]: pgmap v10346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18817.177578 secs 2024-07-26T18:04:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:41 smithi107 ceph-mon[32659]: pgmap v10346: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:42.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18817.177578 secs 2024-07-26T18:04:43.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:43 smithi132 ceph-mon[37580]: pgmap v10347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:43 smithi112 ceph-mon[37501]: pgmap v10347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:43 smithi107 ceph-mon[32659]: pgmap v10347: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:44.897+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:04:44.898 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:44.897+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:04:44.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:44.897+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T18:04:44.899 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:44.897+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T18:04:45.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:45 smithi132 ceph-mon[37580]: pgmap v10348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:45 smithi112 ceph-mon[37501]: pgmap v10348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:45 smithi107 ceph-mon[32659]: pgmap v10348: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18822.177701 secs 2024-07-26T18:04:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18822.177701 secs 2024-07-26T18:04:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18822.177701 secs 2024-07-26T18:04:47.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:47 smithi112 ceph-mon[37501]: pgmap v10349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:47 smithi107 ceph-mon[32659]: pgmap v10349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:47 smithi132 ceph-mon[37580]: pgmap v10349: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:49 smithi112 ceph-mon[37501]: pgmap v10350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:49 smithi107 ceph-mon[32659]: pgmap v10350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:49 smithi132 ceph-mon[37580]: pgmap v10350: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:51 smithi112 ceph-mon[37501]: pgmap v10351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18827.177792 secs 2024-07-26T18:04:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:51 smithi107 ceph-mon[32659]: pgmap v10351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18827.177792 secs 2024-07-26T18:04:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:51 smithi132 ceph-mon[37580]: pgmap v10351: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18827.177792 secs 2024-07-26T18:04:52.205 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:52.203+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:04:52.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:52.203+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:04:52.206 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:04:52.203+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:04:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:53 smithi112 ceph-mon[37501]: pgmap v10352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:53 smithi107 ceph-mon[32659]: pgmap v10352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:53 smithi132 ceph-mon[37580]: pgmap v10352: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:04:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:55 smithi112 ceph-mon[37501]: pgmap v10353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:55 smithi107 ceph-mon[32659]: pgmap v10353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:55 smithi132 ceph-mon[37580]: pgmap v10353: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:04:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18832.177922 secs 2024-07-26T18:04:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18832.177922 secs 2024-07-26T18:04:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18832.177922 secs 2024-07-26T18:04:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:57 smithi112 ceph-mon[37501]: pgmap v10354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:58.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:57 smithi107 ceph-mon[32659]: pgmap v10354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:57 smithi132 ceph-mon[37580]: pgmap v10354: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:04:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:04:59 smithi112 ceph-mon[37501]: pgmap v10355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:04:59 smithi107 ceph-mon[32659]: pgmap v10355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:04:59 smithi132 ceph-mon[37580]: pgmap v10355: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:01 smithi112 ceph-mon[37501]: pgmap v10356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18837.178039 secs 2024-07-26T18:05:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:01 smithi107 ceph-mon[32659]: pgmap v10356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18837.178039 secs 2024-07-26T18:05:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:01 smithi132 ceph-mon[37580]: pgmap v10356: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18837.178039 secs 2024-07-26T18:05:02.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:02 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:05:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:02 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:05:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:02 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:05:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:03 smithi112 ceph-mon[37501]: pgmap v10357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:03.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:03 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:03 smithi107 ceph-mon[32659]: pgmap v10357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:04.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:03 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:03 smithi132 ceph-mon[37580]: pgmap v10357: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:04.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:04.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:04.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:04.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:03 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:05 smithi107 ceph-mon[32659]: pgmap v10358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:05 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:05 smithi132 ceph-mon[37580]: pgmap v10358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:05 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:05 smithi112 ceph-mon[37501]: pgmap v10358: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:05.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:05 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18842.178171 secs 2024-07-26T18:05:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:07.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:06 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18842.178171 secs 2024-07-26T18:05:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:06 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18842.178171 secs 2024-07-26T18:05:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:06 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:07.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:07.217+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:05:07.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:07.217+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:05:07.221 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:07.217+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:05:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:07 smithi107 ceph-mon[32659]: pgmap v10359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:05:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:05:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:07 smithi107 ceph-mon[32659]: pgmap v10360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T18:05:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:07 smithi107 ceph-mon[32659]: pgmap v10361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T18:05:08.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:07 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:07 smithi132 ceph-mon[37580]: pgmap v10359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:05:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:05:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:07 smithi132 ceph-mon[37580]: pgmap v10360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T18:05:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:07 smithi132 ceph-mon[37580]: pgmap v10361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T18:05:08.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:07 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:07 smithi112 ceph-mon[37501]: pgmap v10359: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:05:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:05:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:07 smithi112 ceph-mon[37501]: pgmap v10360: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T18:05:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:07 smithi112 ceph-mon[37501]: pgmap v10361: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T18:05:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:07 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:05:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:09 smithi132 ceph-mon[37580]: pgmap v10362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T18:05:10.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:09 smithi112 ceph-mon[37501]: pgmap v10362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T18:05:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:09 smithi107 ceph-mon[32659]: pgmap v10362: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T18:05:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18847.178289 secs 2024-07-26T18:05:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:11 smithi112 ceph-mon[37501]: pgmap v10363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:05:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18847.178289 secs 2024-07-26T18:05:12.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:11 smithi107 ceph-mon[32659]: pgmap v10363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:05:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18847.178289 secs 2024-07-26T18:05:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:11 smithi132 ceph-mon[37580]: pgmap v10363: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:05:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:13 smithi112 ceph-mon[37501]: pgmap v10364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:05:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:13 smithi107 ceph-mon[32659]: pgmap v10364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:05:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:13 smithi132 ceph-mon[37580]: pgmap v10364: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:05:16.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:15 smithi132 ceph-mon[37580]: pgmap v10365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T18:05:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:15 smithi112 ceph-mon[37501]: pgmap v10365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T18:05:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:15 smithi107 ceph-mon[32659]: pgmap v10365: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T18:05:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18852.178419 secs 2024-07-26T18:05:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18852.178419 secs 2024-07-26T18:05:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18852.178419 secs 2024-07-26T18:05:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:17 smithi112 ceph-mon[37501]: pgmap v10366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:05:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:17 smithi107 ceph-mon[32659]: pgmap v10366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:05:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:17 smithi132 ceph-mon[37580]: pgmap v10366: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:05:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:19 smithi112 ceph-mon[37501]: pgmap v10367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:19 smithi107 ceph-mon[32659]: pgmap v10367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:19 smithi132 ceph-mon[37580]: pgmap v10367: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18857.178539 secs 2024-07-26T18:05:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:21 smithi107 ceph-mon[32659]: pgmap v10368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18857.178539 secs 2024-07-26T18:05:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:21 smithi132 ceph-mon[37580]: pgmap v10368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18857.178539 secs 2024-07-26T18:05:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:21 smithi112 ceph-mon[37501]: pgmap v10368: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:22.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:22.234+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:05:22.235 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:22.234+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:05:22.236 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:22.234+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:05:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:23 smithi112 ceph-mon[37501]: pgmap v10369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:23 smithi107 ceph-mon[32659]: pgmap v10369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:23 smithi132 ceph-mon[37580]: pgmap v10369: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:25 smithi112 ceph-mon[37501]: pgmap v10370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:25 smithi107 ceph-mon[32659]: pgmap v10370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:25 smithi132 ceph-mon[37580]: pgmap v10370: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18862.178655 secs 2024-07-26T18:05:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18862.178655 secs 2024-07-26T18:05:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18862.178655 secs 2024-07-26T18:05:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:27 smithi112 ceph-mon[37501]: pgmap v10371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:27 smithi107 ceph-mon[32659]: pgmap v10371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:27 smithi132 ceph-mon[37580]: pgmap v10371: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:29 smithi112 ceph-mon[37501]: pgmap v10372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:05:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:29 smithi107 ceph-mon[32659]: pgmap v10372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:05:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:29 smithi132 ceph-mon[37580]: pgmap v10372: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:05:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18867.178791 secs 2024-07-26T18:05:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:31 smithi112 ceph-mon[37501]: pgmap v10373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:05:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18867.178791 secs 2024-07-26T18:05:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:31 smithi107 ceph-mon[32659]: pgmap v10373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:05:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18867.178791 secs 2024-07-26T18:05:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:31 smithi132 ceph-mon[37580]: pgmap v10373: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:05:34.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:33 smithi112 ceph-mon[37501]: pgmap v10374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:05:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:33 smithi107 ceph-mon[32659]: pgmap v10374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:05:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:33 smithi132 ceph-mon[37580]: pgmap v10374: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:05:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:35 smithi112 ceph-mon[37501]: pgmap v10375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:35 smithi107 ceph-mon[32659]: pgmap v10375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:35 smithi132 ceph-mon[37580]: pgmap v10375: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:37.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18872.178889 secs 2024-07-26T18:05:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18872.178889 secs 2024-07-26T18:05:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:37.249+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:05:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:37.249+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:05:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:37.249+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:05:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18872.178889 secs 2024-07-26T18:05:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:37 smithi112 ceph-mon[37501]: pgmap v10376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:05:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:37 smithi107 ceph-mon[32659]: pgmap v10376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:05:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:37 smithi132 ceph-mon[37580]: pgmap v10376: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:05:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:39 smithi112 ceph-mon[37501]: pgmap v10377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:05:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:39 smithi107 ceph-mon[32659]: pgmap v10377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:05:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:39 smithi132 ceph-mon[37580]: pgmap v10377: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:05:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18877.179011 secs 2024-07-26T18:05:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:41 smithi112 ceph-mon[37501]: pgmap v10378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18877.179011 secs 2024-07-26T18:05:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:41 smithi107 ceph-mon[32659]: pgmap v10378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18877.179011 secs 2024-07-26T18:05:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:41 smithi132 ceph-mon[37580]: pgmap v10378: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:42 smithi112 ceph-mon[37501]: pgmap v10379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:42 smithi107 ceph-mon[32659]: pgmap v10379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:42 smithi132 ceph-mon[37580]: pgmap v10379: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:45 smithi112 ceph-mon[37501]: pgmap v10380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:45 smithi107 ceph-mon[32659]: pgmap v10380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:45 smithi132 ceph-mon[37580]: pgmap v10380: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18882.179104 secs 2024-07-26T18:05:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18882.179104 secs 2024-07-26T18:05:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18882.179104 secs 2024-07-26T18:05:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:47 smithi112 ceph-mon[37501]: pgmap v10381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:47 smithi107 ceph-mon[32659]: pgmap v10381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:47 smithi132 ceph-mon[37580]: pgmap v10381: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:49 smithi112 ceph-mon[37501]: pgmap v10382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:49 smithi107 ceph-mon[32659]: pgmap v10382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:49 smithi132 ceph-mon[37580]: pgmap v10382: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18887.179219 secs 2024-07-26T18:05:51.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:51 smithi107 ceph-mon[32659]: pgmap v10383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18887.179219 secs 2024-07-26T18:05:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:51 smithi132 ceph-mon[37580]: pgmap v10383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:51.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18887.179219 secs 2024-07-26T18:05:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:51 smithi112 ceph-mon[37501]: pgmap v10383: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:05:52.266 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:52.265+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:05:52.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:52.265+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:05:52.267 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:05:52.265+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:05:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:53 smithi112 ceph-mon[37501]: pgmap v10384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:53 smithi107 ceph-mon[32659]: pgmap v10384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:53 smithi132 ceph-mon[37580]: pgmap v10384: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:05:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:55 smithi112 ceph-mon[37501]: pgmap v10385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:55 smithi107 ceph-mon[32659]: pgmap v10385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:55 smithi132 ceph-mon[37580]: pgmap v10385: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:05:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18892.179302 secs 2024-07-26T18:05:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18892.179302 secs 2024-07-26T18:05:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18892.179302 secs 2024-07-26T18:05:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:57 smithi112 ceph-mon[37501]: pgmap v10386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:05:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:57 smithi107 ceph-mon[32659]: pgmap v10386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:05:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:57 smithi132 ceph-mon[37580]: pgmap v10386: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:06:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:05:59 smithi112 ceph-mon[37501]: pgmap v10387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:05:59 smithi107 ceph-mon[32659]: pgmap v10387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:05:59 smithi132 ceph-mon[37580]: pgmap v10387: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18897.179371 secs 2024-07-26T18:06:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:01 smithi112 ceph-mon[37501]: pgmap v10388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18897.179371 secs 2024-07-26T18:06:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:01 smithi107 ceph-mon[32659]: pgmap v10388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18897.179371 secs 2024-07-26T18:06:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:01 smithi132 ceph-mon[37580]: pgmap v10388: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:03 smithi112 ceph-mon[37501]: pgmap v10389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:03 smithi107 ceph-mon[32659]: pgmap v10389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:03 smithi132 ceph-mon[37580]: pgmap v10389: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:05 smithi112 ceph-mon[37501]: pgmap v10390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:05 smithi107 ceph-mon[32659]: pgmap v10390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:05 smithi132 ceph-mon[37580]: pgmap v10390: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18902.179455 secs 2024-07-26T18:06:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:07.273+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:06:07.276 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:07.274+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:06:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:07.274+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:06:07.277 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18902.179455 secs 2024-07-26T18:06:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18902.179455 secs 2024-07-26T18:06:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:07 smithi112 ceph-mon[37501]: pgmap v10391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:08.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:07 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:06:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:07 smithi107 ceph-mon[32659]: pgmap v10391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:08.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:07 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:06:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:07 smithi132 ceph-mon[37580]: pgmap v10391: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:07 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:06:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:06:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:06:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:08 smithi112 ceph-mon[37501]: pgmap v10392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:06:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:08 smithi112 ceph-mon[37501]: pgmap v10393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:06:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:08 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:06:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:06:09.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:06:09.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:08 smithi107 ceph-mon[32659]: pgmap v10392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:06:09.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:08 smithi107 ceph-mon[32659]: pgmap v10393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:06:09.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:08 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:06:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:06:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:06:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:08 smithi132 ceph-mon[37580]: pgmap v10392: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:06:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:08 smithi132 ceph-mon[37580]: pgmap v10393: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:06:09.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:08 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:06:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:10 smithi107 ceph-mon[32659]: pgmap v10394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:06:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:10 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18907.179608 secs 2024-07-26T18:06:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:10 smithi132 ceph-mon[37580]: pgmap v10394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:06:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:10 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18907.179608 secs 2024-07-26T18:06:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:10 smithi112 ceph-mon[37501]: pgmap v10394: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:06:11.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:10 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18907.179608 secs 2024-07-26T18:06:11.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:11.700+0000 7ff108e00640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900001378) for more than 900000000 us, fault. 2024-07-26T18:06:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:11.700+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8000c00 tx=0x7ff0f8014040 comp rx=0 tx=0).stop 2024-07-26T18:06:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:11.700+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 msgr2=0x7ff0ec0c3e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T18:06:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:11.700+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c1500 0x7ff0ec0c3e20 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T18:06:11.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:11.700+0000 7ff101600640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T18:06:11.703 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:11.700+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T18:06:11.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:11.702+0000 7ff108e00640 1 --2- 172.21.15.107:0/1770124043 >> [v2:172.21.15.107:6800/3577384363,v1:172.21.15.107:6801/3577384363] conn(0x7ff0ec0c0ee0 0x7ff0ec0c4360 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8072000 tx=0x7ff0f8047040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T18:06:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:12 smithi107 ceph-mon[32659]: pgmap v10395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:06:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:12 smithi132 ceph-mon[37580]: pgmap v10395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:06:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:12 smithi112 ceph-mon[37501]: pgmap v10395: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:06:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:15 smithi107 ceph-mon[32659]: pgmap v10396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:06:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:15 smithi132 ceph-mon[37580]: pgmap v10396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:06:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:15 smithi112 ceph-mon[37501]: pgmap v10396: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:06:15.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:15.912+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:06:15.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:15.912+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:06:15.915 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:15.913+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T18:06:15.916 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:15.913+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T18:06:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18912.179746 secs 2024-07-26T18:06:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18912.179746 secs 2024-07-26T18:06:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18912.179746 secs 2024-07-26T18:06:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:17 smithi107 ceph-mon[32659]: pgmap v10397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:06:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:17 smithi132 ceph-mon[37580]: pgmap v10397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:06:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:17 smithi112 ceph-mon[37501]: pgmap v10397: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:06:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:19 smithi107 ceph-mon[32659]: pgmap v10398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:06:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:19 smithi132 ceph-mon[37580]: pgmap v10398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:06:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:19 smithi112 ceph-mon[37501]: pgmap v10398: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:06:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:21 smithi107 ceph-mon[32659]: pgmap v10399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:21.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18917.179861 secs 2024-07-26T18:06:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:21 smithi132 ceph-mon[37580]: pgmap v10399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18917.179861 secs 2024-07-26T18:06:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:21 smithi112 ceph-mon[37501]: pgmap v10399: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18917.179861 secs 2024-07-26T18:06:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:22.282+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:06:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:22.282+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:06:22.284 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:22.282+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:06:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:23 smithi107 ceph-mon[32659]: pgmap v10400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:23 smithi132 ceph-mon[37580]: pgmap v10400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:23 smithi112 ceph-mon[37501]: pgmap v10400: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:25 smithi107 ceph-mon[32659]: pgmap v10401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:25 smithi132 ceph-mon[37580]: pgmap v10401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:25 smithi112 ceph-mon[37501]: pgmap v10401: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18922.179998 secs 2024-07-26T18:06:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18922.179998 secs 2024-07-26T18:06:26.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18922.179998 secs 2024-07-26T18:06:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:27 smithi132 ceph-mon[37580]: pgmap v10402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:27.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:27 smithi112 ceph-mon[37501]: pgmap v10402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:27.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:27 smithi107 ceph-mon[32659]: pgmap v10402: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:29 smithi132 ceph-mon[37580]: pgmap v10403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:29.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:29 smithi112 ceph-mon[37501]: pgmap v10403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:29.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:29 smithi107 ceph-mon[32659]: pgmap v10403: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:31 smithi112 ceph-mon[37501]: pgmap v10404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18927.180135 secs 2024-07-26T18:06:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:31 smithi107 ceph-mon[32659]: pgmap v10404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:31.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18927.180135 secs 2024-07-26T18:06:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:31 smithi132 ceph-mon[37580]: pgmap v10404: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18927.180135 secs 2024-07-26T18:06:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:33 smithi112 ceph-mon[37501]: pgmap v10405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:33.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:33 smithi107 ceph-mon[32659]: pgmap v10405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:33.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:33 smithi132 ceph-mon[37580]: pgmap v10405: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:35 smithi112 ceph-mon[37501]: pgmap v10406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:35.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:35 smithi107 ceph-mon[32659]: pgmap v10406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:35.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:35 smithi132 ceph-mon[37580]: pgmap v10406: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:36.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18932.180270 secs 2024-07-26T18:06:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18932.180270 secs 2024-07-26T18:06:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18932.180270 secs 2024-07-26T18:06:37.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:37.290+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:06:37.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:37.290+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:06:37.292 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:37.290+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:06:37.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:37 smithi107 ceph-mon[32659]: pgmap v10407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:37 smithi112 ceph-mon[37501]: pgmap v10407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:37.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:37 smithi132 ceph-mon[37580]: pgmap v10407: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:39 smithi112 ceph-mon[37501]: pgmap v10408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:39 smithi107 ceph-mon[32659]: pgmap v10408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:39.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:39 smithi132 ceph-mon[37580]: pgmap v10408: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:41 smithi112 ceph-mon[37501]: pgmap v10409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:41.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18937.180367 secs 2024-07-26T18:06:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:41 smithi107 ceph-mon[32659]: pgmap v10409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:41.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18937.180367 secs 2024-07-26T18:06:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:41 smithi132 ceph-mon[37580]: pgmap v10409: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18937.180367 secs 2024-07-26T18:06:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:43 smithi112 ceph-mon[37501]: pgmap v10410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:43 smithi107 ceph-mon[32659]: pgmap v10410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:43.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:43 smithi132 ceph-mon[37580]: pgmap v10410: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:45.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:45 smithi112 ceph-mon[37501]: pgmap v10411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:45.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:45 smithi107 ceph-mon[32659]: pgmap v10411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:45.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:45 smithi132 ceph-mon[37580]: pgmap v10411: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18942.180464 secs 2024-07-26T18:06:46.461 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18942.180464 secs 2024-07-26T18:06:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18942.180464 secs 2024-07-26T18:06:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:47 smithi112 ceph-mon[37501]: pgmap v10412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:47.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:47 smithi107 ceph-mon[32659]: pgmap v10412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:47.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:47 smithi132 ceph-mon[37580]: pgmap v10412: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:49.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:49 smithi112 ceph-mon[37501]: pgmap v10413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:49.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:49 smithi107 ceph-mon[32659]: pgmap v10413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:49.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:49 smithi132 ceph-mon[37580]: pgmap v10413: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:51 smithi112 ceph-mon[37501]: pgmap v10414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:51.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18947.180584 secs 2024-07-26T18:06:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:51 smithi107 ceph-mon[32659]: pgmap v10414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:51.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18947.180584 secs 2024-07-26T18:06:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:51 smithi132 ceph-mon[37580]: pgmap v10414: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18947.180584 secs 2024-07-26T18:06:52.306 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:52.305+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:06:52.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:52.305+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:06:52.307 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:06:52.305+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:06:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:53 smithi112 ceph-mon[37501]: pgmap v10415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:53 smithi107 ceph-mon[32659]: pgmap v10415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:53.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:53 smithi132 ceph-mon[37580]: pgmap v10415: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:06:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:55 smithi112 ceph-mon[37501]: pgmap v10416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:55 smithi107 ceph-mon[32659]: pgmap v10416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:55 smithi132 ceph-mon[37580]: pgmap v10416: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18952.180723 secs 2024-07-26T18:06:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18952.180723 secs 2024-07-26T18:06:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18952.180723 secs 2024-07-26T18:06:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:57 smithi112 ceph-mon[37501]: pgmap v10417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:57 smithi107 ceph-mon[32659]: pgmap v10417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:57 smithi132 ceph-mon[37580]: pgmap v10417: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:06:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:06:59 smithi112 ceph-mon[37501]: pgmap v10418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:06:59 smithi107 ceph-mon[32659]: pgmap v10418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:06:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:06:59 smithi132 ceph-mon[37580]: pgmap v10418: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:01 smithi112 ceph-mon[37501]: pgmap v10419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18957.180859 secs 2024-07-26T18:07:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:01 smithi107 ceph-mon[32659]: pgmap v10419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:01.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18957.180859 secs 2024-07-26T18:07:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:01 smithi132 ceph-mon[37580]: pgmap v10419: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18957.180859 secs 2024-07-26T18:07:03.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:03 smithi112 ceph-mon[37501]: pgmap v10420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:03 smithi107 ceph-mon[32659]: pgmap v10420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:03 smithi132 ceph-mon[37580]: pgmap v10420: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:05 smithi112 ceph-mon[37501]: pgmap v10421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:05 smithi107 ceph-mon[32659]: pgmap v10421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:05 smithi132 ceph-mon[37580]: pgmap v10421: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18962.180958 secs 2024-07-26T18:07:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18962.180958 secs 2024-07-26T18:07:06.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18962.180958 secs 2024-07-26T18:07:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:07.319+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:07:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:07.319+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:07:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:07.319+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:07:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:07 smithi107 ceph-mon[32659]: pgmap v10422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:07.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:07 smithi132 ceph-mon[37580]: pgmap v10422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:07 smithi112 ceph-mon[37501]: pgmap v10422: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:08.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:08 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:07:08.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:08 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:07:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:08 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:07:09.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:09 smithi107 ceph-mon[32659]: pgmap v10423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:09.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:07:09.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:07:09.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:09 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:07:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:09 smithi132 ceph-mon[37580]: pgmap v10423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:07:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:07:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:09 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:07:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:09 smithi112 ceph-mon[37501]: pgmap v10423: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:07:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:07:09.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:09 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:07:10.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:10 smithi107 ceph-mon[32659]: pgmap v10424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:07:10.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:10 smithi107 ceph-mon[32659]: pgmap v10425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:07:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:10 smithi132 ceph-mon[37580]: pgmap v10424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:07:10.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:10 smithi132 ceph-mon[37580]: pgmap v10425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:07:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:10 smithi112 ceph-mon[37501]: pgmap v10424: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:07:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:10 smithi112 ceph-mon[37501]: pgmap v10425: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:07:11.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18967.181100 secs 2024-07-26T18:07:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18967.181100 secs 2024-07-26T18:07:11.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18967.181100 secs 2024-07-26T18:07:12.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:12 smithi107 ceph-mon[32659]: pgmap v10426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:07:12.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:12 smithi132 ceph-mon[37580]: pgmap v10426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:07:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:12 smithi112 ceph-mon[37501]: pgmap v10426: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:07:14.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:14 smithi107 ceph-mon[32659]: pgmap v10427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:07:14.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:14 smithi132 ceph-mon[37580]: pgmap v10427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:07:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:14 smithi112 ceph-mon[37501]: pgmap v10427: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:07:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:16 smithi107 ceph-mon[32659]: pgmap v10428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:07:16.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18972.181146 secs 2024-07-26T18:07:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:16 smithi132 ceph-mon[37580]: pgmap v10428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:07:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18972.181146 secs 2024-07-26T18:07:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:16 smithi112 ceph-mon[37501]: pgmap v10428: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:07:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18972.181146 secs 2024-07-26T18:07:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:18 smithi107 ceph-mon[32659]: pgmap v10429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:18 smithi132 ceph-mon[37580]: pgmap v10429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:18.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:18 smithi112 ceph-mon[37501]: pgmap v10429: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:20.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:20 smithi107 ceph-mon[32659]: pgmap v10430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:07:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:20 smithi132 ceph-mon[37580]: pgmap v10430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:07:20.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:20 smithi112 ceph-mon[37501]: pgmap v10430: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:07:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18977.181242 secs 2024-07-26T18:07:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18977.181242 secs 2024-07-26T18:07:21.602 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18977.181242 secs 2024-07-26T18:07:22.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:22.324+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:07:22.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:22.324+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:07:22.326 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:22.324+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:07:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:22 smithi107 ceph-mon[32659]: pgmap v10431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:22 smithi132 ceph-mon[37580]: pgmap v10431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:22.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:22 smithi112 ceph-mon[37501]: pgmap v10431: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:24.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:24 smithi107 ceph-mon[32659]: pgmap v10432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:24 smithi132 ceph-mon[37580]: pgmap v10432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:24.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:24 smithi112 ceph-mon[37501]: pgmap v10432: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:26 smithi107 ceph-mon[32659]: pgmap v10433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18982.181357 secs 2024-07-26T18:07:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:26 smithi132 ceph-mon[37580]: pgmap v10433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18982.181357 secs 2024-07-26T18:07:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:26 smithi112 ceph-mon[37501]: pgmap v10433: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18982.181357 secs 2024-07-26T18:07:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:28 smithi107 ceph-mon[32659]: pgmap v10434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:28 smithi132 ceph-mon[37580]: pgmap v10434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:28 smithi112 ceph-mon[37501]: pgmap v10434: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:30 smithi107 ceph-mon[32659]: pgmap v10435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:30 smithi132 ceph-mon[37580]: pgmap v10435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:30 smithi112 ceph-mon[37501]: pgmap v10435: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18987.181467 secs 2024-07-26T18:07:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18987.181467 secs 2024-07-26T18:07:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18987.181467 secs 2024-07-26T18:07:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:32 smithi132 ceph-mon[37580]: pgmap v10436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:32 smithi112 ceph-mon[37501]: pgmap v10436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:32.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:32 smithi107 ceph-mon[32659]: pgmap v10436: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:34 smithi112 ceph-mon[37501]: pgmap v10437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:34.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:34 smithi107 ceph-mon[32659]: pgmap v10437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:34.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:34 smithi132 ceph-mon[37580]: pgmap v10437: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:36 smithi112 ceph-mon[37501]: pgmap v10438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:36.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18992.181590 secs 2024-07-26T18:07:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:36 smithi107 ceph-mon[32659]: pgmap v10438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:36.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18992.181590 secs 2024-07-26T18:07:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:36 smithi132 ceph-mon[37580]: pgmap v10438: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18992.181590 secs 2024-07-26T18:07:37.341 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:37.340+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:07:37.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:37.340+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:07:37.342 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:37.340+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:07:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:38 smithi112 ceph-mon[37501]: pgmap v10439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:38.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:38 smithi107 ceph-mon[32659]: pgmap v10439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:38.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:38 smithi132 ceph-mon[37580]: pgmap v10439: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:40.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:40 smithi112 ceph-mon[37501]: pgmap v10440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:40.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:40 smithi107 ceph-mon[32659]: pgmap v10440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:40.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:40 smithi132 ceph-mon[37580]: pgmap v10440: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 18997.181707 secs 2024-07-26T18:07:41.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 18997.181707 secs 2024-07-26T18:07:41.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 18997.181707 secs 2024-07-26T18:07:42.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:42 smithi112 ceph-mon[37501]: pgmap v10441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:42.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:42 smithi107 ceph-mon[32659]: pgmap v10441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:42.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:42 smithi132 ceph-mon[37580]: pgmap v10441: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:44 smithi112 ceph-mon[37501]: pgmap v10442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:44.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:44 smithi107 ceph-mon[32659]: pgmap v10442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:44.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:44 smithi132 ceph-mon[37580]: pgmap v10442: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:46.612 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:46 smithi132 ceph-mon[37580]: pgmap v10443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:46.613 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19002.181812 secs 2024-07-26T18:07:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:46 smithi112 ceph-mon[37501]: pgmap v10443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19002.181812 secs 2024-07-26T18:07:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:46 smithi107 ceph-mon[32659]: pgmap v10443: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:46.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19002.181812 secs 2024-07-26T18:07:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:48 smithi112 ceph-mon[37501]: pgmap v10444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:48.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:48 smithi107 ceph-mon[32659]: pgmap v10444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:48.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:48 smithi132 ceph-mon[37580]: pgmap v10444: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:50.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:50 smithi112 ceph-mon[37501]: pgmap v10445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:50.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:50 smithi107 ceph-mon[32659]: pgmap v10445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:50.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:50 smithi132 ceph-mon[37580]: pgmap v10445: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19007.181933 secs 2024-07-26T18:07:51.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19007.181933 secs 2024-07-26T18:07:51.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19007.181933 secs 2024-07-26T18:07:52.355 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:52.354+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:07:52.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:52.354+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:07:52.356 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:07:52.354+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:07:52.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:52 smithi112 ceph-mon[37501]: pgmap v10446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:52.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:52 smithi107 ceph-mon[32659]: pgmap v10446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:52.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:52 smithi132 ceph-mon[37580]: pgmap v10446: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:07:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:54 smithi112 ceph-mon[37501]: pgmap v10447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:54 smithi107 ceph-mon[32659]: pgmap v10447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:54.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:54 smithi132 ceph-mon[37580]: pgmap v10447: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:07:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:56 smithi112 ceph-mon[37501]: pgmap v10448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19012.182052 secs 2024-07-26T18:07:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:56 smithi107 ceph-mon[32659]: pgmap v10448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:56.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19012.182052 secs 2024-07-26T18:07:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:56 smithi132 ceph-mon[37580]: pgmap v10448: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19012.182052 secs 2024-07-26T18:07:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:07:58 smithi112 ceph-mon[37501]: pgmap v10449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:07:58 smithi107 ceph-mon[32659]: pgmap v10449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:07:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:07:58 smithi132 ceph-mon[37580]: pgmap v10449: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:00 smithi112 ceph-mon[37501]: pgmap v10450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:00 smithi107 ceph-mon[32659]: pgmap v10450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:00 smithi132 ceph-mon[37580]: pgmap v10450: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19017.182143 secs 2024-07-26T18:08:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19017.182143 secs 2024-07-26T18:08:01.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19017.182143 secs 2024-07-26T18:08:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:02 smithi112 ceph-mon[37501]: pgmap v10451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:08:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:02 smithi107 ceph-mon[32659]: pgmap v10451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:08:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:02 smithi132 ceph-mon[37580]: pgmap v10451: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:08:04.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:04 smithi112 ceph-mon[37501]: pgmap v10452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:04 smithi107 ceph-mon[32659]: pgmap v10452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:04 smithi132 ceph-mon[37580]: pgmap v10452: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:06.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:06 smithi112 ceph-mon[37501]: pgmap v10453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19022.182265 secs 2024-07-26T18:08:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:06 smithi107 ceph-mon[32659]: pgmap v10453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:06.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19022.182265 secs 2024-07-26T18:08:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:06 smithi132 ceph-mon[37580]: pgmap v10453: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19022.182265 secs 2024-07-26T18:08:07.371 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:07.369+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:08:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:07.369+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:08:07.372 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:07.369+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:08:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:08 smithi112 ceph-mon[37501]: pgmap v10454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:08 smithi107 ceph-mon[32659]: pgmap v10454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:08 smithi132 ceph-mon[37580]: pgmap v10454: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:09 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:08:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:09 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:08:09.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:09 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:08:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:10 smithi112 ceph-mon[37501]: pgmap v10455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:08:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:08:10.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:10 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:08:10.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:10 smithi107 ceph-mon[32659]: pgmap v10455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:10.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:08:10.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:08:10.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:10 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:08:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:10 smithi132 ceph-mon[37580]: pgmap v10455: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:08:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:08:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:10 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:08:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:11 smithi107 ceph-mon[32659]: pgmap v10456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:08:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:11 smithi107 ceph-mon[32659]: pgmap v10457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:08:11.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19027.182334 secs 2024-07-26T18:08:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:11 smithi132 ceph-mon[37580]: pgmap v10456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:08:11.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:11 smithi132 ceph-mon[37580]: pgmap v10457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:08:11.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19027.182334 secs 2024-07-26T18:08:11.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:11 smithi112 ceph-mon[37501]: pgmap v10456: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:08:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:11 smithi112 ceph-mon[37501]: pgmap v10457: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:08:11.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19027.182334 secs 2024-07-26T18:08:13.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:13 smithi107 ceph-mon[32659]: pgmap v10458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:08:13.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:13 smithi132 ceph-mon[37580]: pgmap v10458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:08:13.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:13 smithi112 ceph-mon[37501]: pgmap v10458: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:08:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:15 smithi107 ceph-mon[32659]: pgmap v10459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:08:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:15 smithi132 ceph-mon[37580]: pgmap v10459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:08:15.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:15 smithi112 ceph-mon[37501]: pgmap v10459: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:08:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19032.182452 secs 2024-07-26T18:08:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19032.182452 secs 2024-07-26T18:08:16.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19032.182452 secs 2024-07-26T18:08:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:17 smithi107 ceph-mon[32659]: pgmap v10460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:08:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:17 smithi132 ceph-mon[37580]: pgmap v10460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:08:17.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:17 smithi112 ceph-mon[37501]: pgmap v10460: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:08:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:19 smithi107 ceph-mon[32659]: pgmap v10461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:08:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:19 smithi132 ceph-mon[37580]: pgmap v10461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:08:19.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:19 smithi112 ceph-mon[37501]: pgmap v10461: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:08:21.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:21 smithi112 ceph-mon[37501]: pgmap v10462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:08:21.753 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19037.182572 secs 2024-07-26T18:08:21.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:21 smithi107 ceph-mon[32659]: pgmap v10462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:08:21.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19037.182572 secs 2024-07-26T18:08:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:21 smithi132 ceph-mon[37580]: pgmap v10462: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:08:21.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19037.182572 secs 2024-07-26T18:08:22.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:22.384+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:08:22.387 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:22.384+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:08:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:22.384+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:08:23.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:23 smithi107 ceph-mon[32659]: pgmap v10463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:23.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:23 smithi132 ceph-mon[37580]: pgmap v10463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:23 smithi112 ceph-mon[37501]: pgmap v10463: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:25.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:25 smithi107 ceph-mon[32659]: pgmap v10464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:25.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:25 smithi132 ceph-mon[37580]: pgmap v10464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:25 smithi112 ceph-mon[37501]: pgmap v10464: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:26.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19042.182691 secs 2024-07-26T18:08:26.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19042.182691 secs 2024-07-26T18:08:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19042.182691 secs 2024-07-26T18:08:27.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:27 smithi107 ceph-mon[32659]: pgmap v10465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:27.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:27 smithi132 ceph-mon[37580]: pgmap v10465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:27 smithi112 ceph-mon[37501]: pgmap v10465: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:29.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:29 smithi107 ceph-mon[32659]: pgmap v10466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:29.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:29 smithi132 ceph-mon[37580]: pgmap v10466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:29 smithi112 ceph-mon[37501]: pgmap v10466: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:31.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:31 smithi107 ceph-mon[32659]: pgmap v10467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:31.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19047.182808 secs 2024-07-26T18:08:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:31 smithi132 ceph-mon[37580]: pgmap v10467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:31.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19047.182808 secs 2024-07-26T18:08:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:31 smithi112 ceph-mon[37501]: pgmap v10467: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19047.182808 secs 2024-07-26T18:08:33.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:33 smithi107 ceph-mon[32659]: pgmap v10468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:33.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:33 smithi132 ceph-mon[37580]: pgmap v10468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:33.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:33 smithi112 ceph-mon[37501]: pgmap v10468: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:35.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:35 smithi107 ceph-mon[32659]: pgmap v10469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:35.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:35 smithi132 ceph-mon[37580]: pgmap v10469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:35.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:35 smithi112 ceph-mon[37501]: pgmap v10469: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:36.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19052.182923 secs 2024-07-26T18:08:36.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19052.182923 secs 2024-07-26T18:08:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19052.182923 secs 2024-07-26T18:08:37.395 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:37.394+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:08:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:37.394+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:08:37.396 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:37.394+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:08:37.694 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:37 smithi107 ceph-mon[32659]: pgmap v10470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:37.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:37 smithi132 ceph-mon[37580]: pgmap v10470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:37.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:37 smithi112 ceph-mon[37501]: pgmap v10470: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:39.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:39 smithi132 ceph-mon[37580]: pgmap v10471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:39.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:39 smithi112 ceph-mon[37501]: pgmap v10471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:39 smithi107 ceph-mon[32659]: pgmap v10471: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:41 smithi112 ceph-mon[37501]: pgmap v10472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:41.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19057.183037 secs 2024-07-26T18:08:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:41 smithi107 ceph-mon[32659]: pgmap v10472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19057.183037 secs 2024-07-26T18:08:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:41 smithi132 ceph-mon[37580]: pgmap v10472: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19057.183037 secs 2024-07-26T18:08:43.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:43 smithi112 ceph-mon[37501]: pgmap v10473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:43 smithi107 ceph-mon[32659]: pgmap v10473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:43 smithi132 ceph-mon[37580]: pgmap v10473: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:45.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:45 smithi112 ceph-mon[37501]: pgmap v10474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:45 smithi107 ceph-mon[32659]: pgmap v10474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:45 smithi132 ceph-mon[37580]: pgmap v10474: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19062.183152 secs 2024-07-26T18:08:46.961 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19062.183152 secs 2024-07-26T18:08:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19062.183152 secs 2024-07-26T18:08:47.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:47 smithi112 ceph-mon[37501]: pgmap v10475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:47 smithi107 ceph-mon[32659]: pgmap v10475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:47 smithi132 ceph-mon[37580]: pgmap v10475: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:49.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:49 smithi112 ceph-mon[37501]: pgmap v10476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:49 smithi107 ceph-mon[32659]: pgmap v10476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:49 smithi132 ceph-mon[37580]: pgmap v10476: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:51 smithi112 ceph-mon[37501]: pgmap v10477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:51.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19067.183214 secs 2024-07-26T18:08:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:51 smithi107 ceph-mon[32659]: pgmap v10477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:52.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19067.183214 secs 2024-07-26T18:08:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:51 smithi132 ceph-mon[37580]: pgmap v10477: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:08:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19067.183214 secs 2024-07-26T18:08:52.410 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:52.409+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:08:52.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:52.409+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:08:52.411 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:08:52.409+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:08:53.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:53 smithi112 ceph-mon[37501]: pgmap v10478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:53 smithi107 ceph-mon[32659]: pgmap v10478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:53 smithi132 ceph-mon[37580]: pgmap v10478: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:08:55.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:55 smithi112 ceph-mon[37501]: pgmap v10479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:55 smithi107 ceph-mon[32659]: pgmap v10479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:55 smithi132 ceph-mon[37580]: pgmap v10479: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:08:56.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19072.183335 secs 2024-07-26T18:08:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19072.183335 secs 2024-07-26T18:08:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19072.183335 secs 2024-07-26T18:08:57.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:57 smithi107 ceph-mon[32659]: pgmap v10480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:57.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:57 smithi112 ceph-mon[37501]: pgmap v10480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:57 smithi132 ceph-mon[37580]: pgmap v10480: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:08:59.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:08:59 smithi112 ceph-mon[37501]: pgmap v10481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:00.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:08:59 smithi107 ceph-mon[32659]: pgmap v10481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:08:59 smithi132 ceph-mon[37580]: pgmap v10481: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:01.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:01 smithi112 ceph-mon[37501]: pgmap v10482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:01.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19077.183427 secs 2024-07-26T18:09:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:01 smithi107 ceph-mon[32659]: pgmap v10482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:02.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19077.183427 secs 2024-07-26T18:09:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:01 smithi132 ceph-mon[37580]: pgmap v10482: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19077.183427 secs 2024-07-26T18:09:03.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:03 smithi112 ceph-mon[37501]: pgmap v10483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:04.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:03 smithi107 ceph-mon[32659]: pgmap v10483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:04.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:03 smithi132 ceph-mon[37580]: pgmap v10483: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:05.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:05 smithi112 ceph-mon[37501]: pgmap v10484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:06.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:05 smithi107 ceph-mon[32659]: pgmap v10484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:06.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:05 smithi132 ceph-mon[37580]: pgmap v10484: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:06.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19082.183535 secs 2024-07-26T18:09:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19082.183535 secs 2024-07-26T18:09:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19082.183535 secs 2024-07-26T18:09:07.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:07.423+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:09:07.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:07.423+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:09:07.425 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:07.423+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:09:07.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:07 smithi107 ceph-mon[32659]: pgmap v10485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:09:07.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:07 smithi112 ceph-mon[37501]: pgmap v10485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:09:08.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:07 smithi132 ceph-mon[37580]: pgmap v10485: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:09:09.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:09 smithi112 ceph-mon[37501]: pgmap v10486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:10.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:09 smithi107 ceph-mon[32659]: pgmap v10486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:10.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:09 smithi132 ceph-mon[37580]: pgmap v10486: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:10.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:10 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:09:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:10 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:09:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:10 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:09:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:11 smithi107 ceph-mon[32659]: pgmap v10487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19087.183659 secs 2024-07-26T18:09:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:09:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:09:12.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:11 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:09:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:11 smithi132 ceph-mon[37580]: pgmap v10487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19087.183659 secs 2024-07-26T18:09:12.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:09:12.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:09:12.061 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:11 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:09:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:11 smithi112 ceph-mon[37501]: pgmap v10487: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19087.183659 secs 2024-07-26T18:09:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:09:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:09:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:11 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:09:13.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:12 smithi107 ceph-mon[32659]: pgmap v10488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:09:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:12 smithi132 ceph-mon[37580]: pgmap v10488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:09:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:12 smithi112 ceph-mon[37501]: pgmap v10488: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:09:15.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:14 smithi107 ceph-mon[32659]: pgmap v10489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:09:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:14 smithi132 ceph-mon[37580]: pgmap v10489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:09:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:14 smithi112 ceph-mon[37501]: pgmap v10489: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:09:15.948 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:15.946+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:09:15.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:15.946+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:09:15.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:15.946+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T18:09:15.949 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:15.946+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T18:09:17.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19092.183775 secs 2024-07-26T18:09:17.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:16 smithi107 ceph-mon[32659]: pgmap v10490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:09:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19092.183775 secs 2024-07-26T18:09:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:16 smithi132 ceph-mon[37580]: pgmap v10490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:09:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19092.183775 secs 2024-07-26T18:09:17.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:16 smithi112 ceph-mon[37501]: pgmap v10490: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:09:19.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:18 smithi107 ceph-mon[32659]: pgmap v10491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T18:09:19.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:18 smithi132 ceph-mon[37580]: pgmap v10491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T18:09:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:18 smithi112 ceph-mon[37501]: pgmap v10491: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T18:09:21.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:20 smithi112 ceph-mon[37501]: pgmap v10492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:09:21.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:20 smithi107 ceph-mon[32659]: pgmap v10492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:09:21.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:20 smithi132 ceph-mon[37580]: pgmap v10492: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:09:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19097.183872 secs 2024-07-26T18:09:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19097.183872 secs 2024-07-26T18:09:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19097.183872 secs 2024-07-26T18:09:22.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:22.439+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:09:22.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:22.439+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:09:22.441 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:22.439+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:09:23.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:22 smithi107 ceph-mon[32659]: pgmap v10493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:09:23.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:22 smithi132 ceph-mon[37580]: pgmap v10493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:09:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:22 smithi112 ceph-mon[37501]: pgmap v10493: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:09:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:24 smithi107 ceph-mon[32659]: pgmap v10494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:24 smithi132 ceph-mon[37580]: pgmap v10494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:24 smithi112 ceph-mon[37501]: pgmap v10494: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19102.183993 secs 2024-07-26T18:09:27.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:26 smithi107 ceph-mon[32659]: pgmap v10495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:09:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19102.183993 secs 2024-07-26T18:09:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:26 smithi132 ceph-mon[37580]: pgmap v10495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:09:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19102.183993 secs 2024-07-26T18:09:27.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:26 smithi112 ceph-mon[37501]: pgmap v10495: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:09:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:28 smithi107 ceph-mon[32659]: pgmap v10496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:28 smithi132 ceph-mon[37580]: pgmap v10496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:28 smithi112 ceph-mon[37501]: pgmap v10496: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:30 smithi107 ceph-mon[32659]: pgmap v10497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:31.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:30 smithi132 ceph-mon[37580]: pgmap v10497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:30 smithi112 ceph-mon[37501]: pgmap v10497: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19107.184152 secs 2024-07-26T18:09:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19107.184152 secs 2024-07-26T18:09:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19107.184152 secs 2024-07-26T18:09:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:32 smithi107 ceph-mon[32659]: pgmap v10498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:33.064 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:32 smithi132 ceph-mon[37580]: pgmap v10498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:32 smithi112 ceph-mon[37501]: pgmap v10498: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:35.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:34 smithi107 ceph-mon[32659]: pgmap v10499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:34 smithi132 ceph-mon[37580]: pgmap v10499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:35.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:34 smithi112 ceph-mon[37501]: pgmap v10499: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19112.184271 secs 2024-07-26T18:09:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:36 smithi132 ceph-mon[37580]: pgmap v10500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:09:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19112.184271 secs 2024-07-26T18:09:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:36 smithi112 ceph-mon[37501]: pgmap v10500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:09:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19112.184271 secs 2024-07-26T18:09:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:36 smithi107 ceph-mon[32659]: pgmap v10500: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:09:37.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:37.454+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:09:37.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:37.454+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:09:37.456 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:37.454+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:09:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:38 smithi132 ceph-mon[37580]: pgmap v10501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:38 smithi112 ceph-mon[37501]: pgmap v10501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:38 smithi107 ceph-mon[32659]: pgmap v10501: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:40 smithi132 ceph-mon[37580]: pgmap v10502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:41.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:40 smithi112 ceph-mon[37501]: pgmap v10502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:40 smithi107 ceph-mon[32659]: pgmap v10502: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19117.184374 secs 2024-07-26T18:09:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19117.184374 secs 2024-07-26T18:09:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19117.184374 secs 2024-07-26T18:09:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:42 smithi112 ceph-mon[37501]: pgmap v10503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:42 smithi107 ceph-mon[32659]: pgmap v10503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:42 smithi132 ceph-mon[37580]: pgmap v10503: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:44 smithi112 ceph-mon[37501]: pgmap v10504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:44 smithi107 ceph-mon[32659]: pgmap v10504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:44 smithi132 ceph-mon[37580]: pgmap v10504: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:47.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19122.184504 secs 2024-07-26T18:09:47.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:46 smithi132 ceph-mon[37580]: pgmap v10505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:09:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19122.184504 secs 2024-07-26T18:09:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:46 smithi112 ceph-mon[37501]: pgmap v10505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:09:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19122.184504 secs 2024-07-26T18:09:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:46 smithi107 ceph-mon[32659]: pgmap v10505: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:09:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:48 smithi112 ceph-mon[37501]: pgmap v10506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:48 smithi107 ceph-mon[32659]: pgmap v10506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:49.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:48 smithi132 ceph-mon[37580]: pgmap v10506: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:09:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:50 smithi112 ceph-mon[37501]: pgmap v10507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:50 smithi107 ceph-mon[32659]: pgmap v10507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:51.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:50 smithi132 ceph-mon[37580]: pgmap v10507: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:09:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19127.184630 secs 2024-07-26T18:09:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19127.184630 secs 2024-07-26T18:09:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19127.184630 secs 2024-07-26T18:09:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:52.470+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:09:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:52.470+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:09:52.472 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:09:52.470+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:09:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:52 smithi112 ceph-mon[37501]: pgmap v10508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:52 smithi107 ceph-mon[32659]: pgmap v10508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:52 smithi132 ceph-mon[37580]: pgmap v10508: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:09:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:54 smithi112 ceph-mon[37501]: pgmap v10509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:09:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:54 smithi107 ceph-mon[32659]: pgmap v10509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:09:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:54 smithi132 ceph-mon[37580]: pgmap v10509: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:09:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19132.184758 secs 2024-07-26T18:09:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:56 smithi112 ceph-mon[37501]: pgmap v10510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:09:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19132.184758 secs 2024-07-26T18:09:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:56 smithi107 ceph-mon[32659]: pgmap v10510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:09:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19132.184758 secs 2024-07-26T18:09:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:56 smithi132 ceph-mon[37580]: pgmap v10510: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:09:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:09:58 smithi112 ceph-mon[37501]: pgmap v10511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:09:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:09:58 smithi107 ceph-mon[32659]: pgmap v10511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:09:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:09:58 smithi132 ceph-mon[37580]: pgmap v10511: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:10:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:00 smithi112 ceph-mon[37501]: pgmap v10512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:10:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T18:10:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:00 smithi107 ceph-mon[32659]: pgmap v10512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:10:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T18:10:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:00 smithi132 ceph-mon[37580]: pgmap v10512: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:10:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T18:10:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19137.184880 secs 2024-07-26T18:10:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19137.184880 secs 2024-07-26T18:10:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19137.184880 secs 2024-07-26T18:10:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:02 smithi112 ceph-mon[37501]: pgmap v10513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:10:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:02 smithi107 ceph-mon[32659]: pgmap v10513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:10:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:02 smithi132 ceph-mon[37580]: pgmap v10513: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:10:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:04 smithi112 ceph-mon[37501]: pgmap v10514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:10:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:04 smithi107 ceph-mon[32659]: pgmap v10514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:10:05.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:04 smithi132 ceph-mon[37580]: pgmap v10514: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:10:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19142.185007 secs 2024-07-26T18:10:07.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:06 smithi112 ceph-mon[37501]: pgmap v10515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19142.185007 secs 2024-07-26T18:10:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:06 smithi107 ceph-mon[32659]: pgmap v10515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19142.185007 secs 2024-07-26T18:10:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:06 smithi132 ceph-mon[37580]: pgmap v10515: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:07.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:07.484+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:10:07.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:07.484+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:10:07.487 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:07.484+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:10:09.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:08 smithi112 ceph-mon[37501]: pgmap v10516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:08 smithi107 ceph-mon[32659]: pgmap v10516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:08 smithi132 ceph-mon[37580]: pgmap v10516: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:10 smithi112 ceph-mon[37501]: pgmap v10517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:10:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:10 smithi107 ceph-mon[32659]: pgmap v10517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:10:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:10 smithi132 ceph-mon[37580]: pgmap v10517: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:10:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19147.185138 secs 2024-07-26T18:10:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:11 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:10:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19147.185138 secs 2024-07-26T18:10:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:11 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:10:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19147.185138 secs 2024-07-26T18:10:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:11 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:10:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:12 smithi112 ceph-mon[37501]: pgmap v10518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:10:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:10:13.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:12 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:10:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:12 smithi107 ceph-mon[32659]: pgmap v10518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:10:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:10:13.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:12 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:10:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:12 smithi132 ceph-mon[37580]: pgmap v10518: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:10:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:10:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:12 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:10:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:13 smithi112 ceph-mon[37501]: pgmap v10519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:10:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:13 smithi107 ceph-mon[32659]: pgmap v10519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:10:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:13 smithi132 ceph-mon[37580]: pgmap v10519: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:10:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:14 smithi107 ceph-mon[32659]: pgmap v10520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T18:10:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:14 smithi132 ceph-mon[37580]: pgmap v10520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T18:10:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:14 smithi112 ceph-mon[37501]: pgmap v10520: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T18:10:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:15 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19152.185256 secs 2024-07-26T18:10:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:15 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19152.185256 secs 2024-07-26T18:10:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19152.185256 secs 2024-07-26T18:10:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:17 smithi107 ceph-mon[32659]: pgmap v10521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:10:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:17 smithi132 ceph-mon[37580]: pgmap v10521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:10:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:17 smithi112 ceph-mon[37501]: pgmap v10521: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:10:20.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:19 smithi107 ceph-mon[32659]: pgmap v10522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:10:20.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:19 smithi132 ceph-mon[37580]: pgmap v10522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:10:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:19 smithi112 ceph-mon[37501]: pgmap v10522: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T18:10:22.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19157.185378 secs 2024-07-26T18:10:22.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:21 smithi112 ceph-mon[37501]: pgmap v10523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:10:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19157.185378 secs 2024-07-26T18:10:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:21 smithi107 ceph-mon[32659]: pgmap v10523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:10:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19157.185378 secs 2024-07-26T18:10:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:21 smithi132 ceph-mon[37580]: pgmap v10523: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:10:22.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:22.500+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:10:22.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:22.500+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:10:22.503 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:22.500+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:10:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:23 smithi107 ceph-mon[32659]: pgmap v10524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:10:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:23 smithi132 ceph-mon[37580]: pgmap v10524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:10:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:23 smithi112 ceph-mon[37501]: pgmap v10524: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:10:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:25 smithi107 ceph-mon[32659]: pgmap v10525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:10:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:25 smithi132 ceph-mon[37580]: pgmap v10525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:10:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:25 smithi112 ceph-mon[37501]: pgmap v10525: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:10:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19162.185483 secs 2024-07-26T18:10:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19162.185483 secs 2024-07-26T18:10:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19162.185483 secs 2024-07-26T18:10:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:27 smithi107 ceph-mon[32659]: pgmap v10526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:27 smithi132 ceph-mon[37580]: pgmap v10526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:27 smithi112 ceph-mon[37501]: pgmap v10526: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:29 smithi107 ceph-mon[32659]: pgmap v10527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:29 smithi132 ceph-mon[37580]: pgmap v10527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:30.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:29 smithi112 ceph-mon[37501]: pgmap v10527: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19167.185570 secs 2024-07-26T18:10:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:31 smithi107 ceph-mon[32659]: pgmap v10528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19167.185570 secs 2024-07-26T18:10:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:31 smithi132 ceph-mon[37580]: pgmap v10528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:32.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19167.185570 secs 2024-07-26T18:10:32.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:31 smithi112 ceph-mon[37501]: pgmap v10528: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:33 smithi107 ceph-mon[32659]: pgmap v10529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:33 smithi132 ceph-mon[37580]: pgmap v10529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:33 smithi112 ceph-mon[37501]: pgmap v10529: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:35 smithi132 ceph-mon[37580]: pgmap v10530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:35 smithi112 ceph-mon[37501]: pgmap v10530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:35 smithi107 ceph-mon[32659]: pgmap v10530: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19172.185681 secs 2024-07-26T18:10:37.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19172.185681 secs 2024-07-26T18:10:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19172.185681 secs 2024-07-26T18:10:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:37.508+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:10:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:37.508+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:10:37.510 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:37.508+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:10:38.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:37 smithi112 ceph-mon[37501]: pgmap v10531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:38.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:37 smithi107 ceph-mon[32659]: pgmap v10531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:37 smithi132 ceph-mon[37580]: pgmap v10531: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:39 smithi112 ceph-mon[37501]: pgmap v10532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:39 smithi107 ceph-mon[32659]: pgmap v10532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:39 smithi132 ceph-mon[37580]: pgmap v10532: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19177.185810 secs 2024-07-26T18:10:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:41 smithi112 ceph-mon[37501]: pgmap v10533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19177.185810 secs 2024-07-26T18:10:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:41 smithi107 ceph-mon[32659]: pgmap v10533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19177.185810 secs 2024-07-26T18:10:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:41 smithi132 ceph-mon[37580]: pgmap v10533: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:43 smithi112 ceph-mon[37501]: pgmap v10534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:43 smithi107 ceph-mon[32659]: pgmap v10534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:43 smithi132 ceph-mon[37580]: pgmap v10534: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:45 smithi112 ceph-mon[37501]: pgmap v10535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:45 smithi107 ceph-mon[32659]: pgmap v10535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:45 smithi132 ceph-mon[37580]: pgmap v10535: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19182.185947 secs 2024-07-26T18:10:47.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19182.185947 secs 2024-07-26T18:10:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19182.185947 secs 2024-07-26T18:10:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:47 smithi112 ceph-mon[37501]: pgmap v10536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:47 smithi107 ceph-mon[32659]: pgmap v10536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:47 smithi132 ceph-mon[37580]: pgmap v10536: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:49 smithi112 ceph-mon[37501]: pgmap v10537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:49 smithi107 ceph-mon[32659]: pgmap v10537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:49 smithi132 ceph-mon[37580]: pgmap v10537: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19187.186083 secs 2024-07-26T18:10:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:51 smithi112 ceph-mon[37501]: pgmap v10538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19187.186083 secs 2024-07-26T18:10:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:51 smithi107 ceph-mon[32659]: pgmap v10538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19187.186083 secs 2024-07-26T18:10:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:51 smithi132 ceph-mon[37580]: pgmap v10538: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:10:52.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:52.516+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:10:52.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:52.516+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:10:52.518 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:10:52.516+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:10:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:53 smithi112 ceph-mon[37501]: pgmap v10539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:53 smithi107 ceph-mon[32659]: pgmap v10539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:53 smithi132 ceph-mon[37580]: pgmap v10539: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:10:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:55 smithi112 ceph-mon[37501]: pgmap v10540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:55 smithi107 ceph-mon[32659]: pgmap v10540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:55 smithi132 ceph-mon[37580]: pgmap v10540: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19192.186211 secs 2024-07-26T18:10:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19192.186211 secs 2024-07-26T18:10:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19192.186211 secs 2024-07-26T18:10:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:57 smithi112 ceph-mon[37501]: pgmap v10541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:57 smithi107 ceph-mon[32659]: pgmap v10541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:10:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:57 smithi132 ceph-mon[37580]: pgmap v10541: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:10:59 smithi112 ceph-mon[37501]: pgmap v10542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:10:59 smithi107 ceph-mon[32659]: pgmap v10542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:10:59 smithi132 ceph-mon[37580]: pgmap v10542: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19197.186361 secs 2024-07-26T18:11:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:01 smithi112 ceph-mon[37501]: pgmap v10543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:02.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19197.186361 secs 2024-07-26T18:11:02.236 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:01 smithi107 ceph-mon[32659]: pgmap v10543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19197.186361 secs 2024-07-26T18:11:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:01 smithi132 ceph-mon[37580]: pgmap v10543: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:03 smithi112 ceph-mon[37501]: pgmap v10544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:03 smithi107 ceph-mon[32659]: pgmap v10544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:03 smithi132 ceph-mon[37580]: pgmap v10544: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:06.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:05 smithi112 ceph-mon[37501]: pgmap v10545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:05 smithi107 ceph-mon[32659]: pgmap v10545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:05 smithi132 ceph-mon[37580]: pgmap v10545: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:07.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19202.186420 secs 2024-07-26T18:11:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19202.186420 secs 2024-07-26T18:11:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19202.186420 secs 2024-07-26T18:11:07.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:07.530+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:11:07.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:07.530+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:11:07.533 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:07.530+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:11:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:07 smithi112 ceph-mon[37501]: pgmap v10546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:07 smithi107 ceph-mon[32659]: pgmap v10546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:07 smithi132 ceph-mon[37580]: pgmap v10546: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:09 smithi112 ceph-mon[37501]: pgmap v10547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:09 smithi107 ceph-mon[32659]: pgmap v10547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:09 smithi132 ceph-mon[37580]: pgmap v10547: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19207.186541 secs 2024-07-26T18:11:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:11 smithi112 ceph-mon[37501]: pgmap v10548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19207.186541 secs 2024-07-26T18:11:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:11 smithi107 ceph-mon[32659]: pgmap v10548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19207.186541 secs 2024-07-26T18:11:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:11 smithi132 ceph-mon[37580]: pgmap v10548: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:12 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:11:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:12 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:11:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:12 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:11:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: pgmap v10549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:11:14.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:11:14.202 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:13 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:11:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: pgmap v10549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:11:14.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:11:14.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:13 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:11:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: pgmap v10549: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:11:14.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:11:14.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:11:14.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:13 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:11:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:14 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T18:11:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:14 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T18:11:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:14 smithi107 ceph-mon[32659]: pgmap v10550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:11:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:14 smithi107 ceph-mon[32659]: pgmap v10551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:11:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:14 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T18:11:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:14 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T18:11:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:14 smithi132 ceph-mon[37580]: pgmap v10550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:11:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:14 smithi132 ceph-mon[37580]: pgmap v10551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:11:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:14 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi107 to 176.4M 2024-07-26T18:11:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:14 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi107 to 184985292: error parsing value: Value '184985292' is below minimum 939524096 2024-07-26T18:11:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:14 smithi112 ceph-mon[37501]: pgmap v10550: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:11:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:14 smithi112 ceph-mon[37501]: pgmap v10551: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:11:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19212.186698 secs 2024-07-26T18:11:17.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:16 smithi107 ceph-mon[32659]: pgmap v10552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:11:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19212.186698 secs 2024-07-26T18:11:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:16 smithi132 ceph-mon[37580]: pgmap v10552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:11:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19212.186698 secs 2024-07-26T18:11:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:16 smithi112 ceph-mon[37501]: pgmap v10552: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:11:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:18 smithi107 ceph-mon[32659]: pgmap v10553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:11:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:18 smithi132 ceph-mon[37580]: pgmap v10553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:11:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:18 smithi112 ceph-mon[37501]: pgmap v10553: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:11:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:20 smithi112 ceph-mon[37501]: pgmap v10554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:11:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:20 smithi107 ceph-mon[32659]: pgmap v10554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:11:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:20 smithi132 ceph-mon[37580]: pgmap v10554: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:11:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19217.186822 secs 2024-07-26T18:11:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19217.186822 secs 2024-07-26T18:11:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19217.186822 secs 2024-07-26T18:11:22.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:22.546+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:11:22.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:22.546+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:11:22.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:22.546+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:11:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:22 smithi107 ceph-mon[32659]: pgmap v10555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:22 smithi132 ceph-mon[37580]: pgmap v10555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:22 smithi112 ceph-mon[37501]: pgmap v10555: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:23 smithi107 ceph-mon[32659]: pgmap v10556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:11:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:23 smithi132 ceph-mon[37580]: pgmap v10556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:11:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:23 smithi112 ceph-mon[37501]: pgmap v10556: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:11:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19222.186899 secs 2024-07-26T18:11:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:26 smithi107 ceph-mon[32659]: pgmap v10557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19222.186899 secs 2024-07-26T18:11:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:26 smithi132 ceph-mon[37580]: pgmap v10557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:27.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19222.186899 secs 2024-07-26T18:11:27.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:26 smithi112 ceph-mon[37501]: pgmap v10557: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:28 smithi107 ceph-mon[32659]: pgmap v10558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:28 smithi132 ceph-mon[37580]: pgmap v10558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:28 smithi112 ceph-mon[37501]: pgmap v10558: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:30 smithi107 ceph-mon[32659]: pgmap v10559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:31.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:30 smithi132 ceph-mon[37580]: pgmap v10559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:30 smithi112 ceph-mon[37501]: pgmap v10559: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19227.187025 secs 2024-07-26T18:11:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19227.187025 secs 2024-07-26T18:11:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19227.187025 secs 2024-07-26T18:11:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:32 smithi107 ceph-mon[32659]: pgmap v10560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:32 smithi132 ceph-mon[37580]: pgmap v10560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:32 smithi112 ceph-mon[37501]: pgmap v10560: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:34.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:33 smithi107 ceph-mon[32659]: pgmap v10561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:33 smithi132 ceph-mon[37580]: pgmap v10561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:34.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:33 smithi112 ceph-mon[37501]: pgmap v10561: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19232.187149 secs 2024-07-26T18:11:37.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:36 smithi107 ceph-mon[32659]: pgmap v10562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19232.187149 secs 2024-07-26T18:11:37.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:36 smithi132 ceph-mon[37580]: pgmap v10562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19232.187149 secs 2024-07-26T18:11:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:36 smithi112 ceph-mon[37501]: pgmap v10562: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:37.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:37.554+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:11:37.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:37.554+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:11:37.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:37.554+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:11:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:38 smithi107 ceph-mon[32659]: pgmap v10563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:11:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:38 smithi132 ceph-mon[37580]: pgmap v10563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:11:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:38 smithi112 ceph-mon[37501]: pgmap v10563: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:11:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:40 smithi107 ceph-mon[32659]: pgmap v10564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:40 smithi132 ceph-mon[37580]: pgmap v10564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:40 smithi112 ceph-mon[37501]: pgmap v10564: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:41.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19237.187245 secs 2024-07-26T18:11:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19237.187245 secs 2024-07-26T18:11:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19237.187245 secs 2024-07-26T18:11:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:42 smithi107 ceph-mon[32659]: pgmap v10565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:42 smithi132 ceph-mon[37580]: pgmap v10565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:42 smithi112 ceph-mon[37501]: pgmap v10565: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:44 smithi107 ceph-mon[32659]: pgmap v10566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:11:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:44 smithi132 ceph-mon[37580]: pgmap v10566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:11:45.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:44 smithi112 ceph-mon[37501]: pgmap v10566: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:11:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19242.187337 secs 2024-07-26T18:11:47.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:46 smithi107 ceph-mon[32659]: pgmap v10567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:11:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19242.187337 secs 2024-07-26T18:11:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:46 smithi132 ceph-mon[37580]: pgmap v10567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:11:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19242.187337 secs 2024-07-26T18:11:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:46 smithi112 ceph-mon[37501]: pgmap v10567: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:11:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:47 smithi107 ceph-mon[32659]: pgmap v10568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:11:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:47 smithi132 ceph-mon[37580]: pgmap v10568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:11:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:47 smithi112 ceph-mon[37501]: pgmap v10568: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:11:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:50 smithi107 ceph-mon[32659]: pgmap v10569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:50 smithi132 ceph-mon[37580]: pgmap v10569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:51.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:50 smithi112 ceph-mon[37501]: pgmap v10569: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19247.187407 secs 2024-07-26T18:11:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19247.187407 secs 2024-07-26T18:11:52.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19247.187407 secs 2024-07-26T18:11:52.571 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:52.570+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:11:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:52.570+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:11:52.572 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:11:52.570+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:11:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:52 smithi107 ceph-mon[32659]: pgmap v10570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:52 smithi132 ceph-mon[37580]: pgmap v10570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:52 smithi112 ceph-mon[37501]: pgmap v10570: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:11:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:53 smithi107 ceph-mon[32659]: pgmap v10571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:53 smithi132 ceph-mon[37580]: pgmap v10571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:53 smithi112 ceph-mon[37501]: pgmap v10571: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19252.187543 secs 2024-07-26T18:11:57.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:56 smithi107 ceph-mon[32659]: pgmap v10572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19252.187543 secs 2024-07-26T18:11:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:56 smithi132 ceph-mon[37580]: pgmap v10572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19252.187543 secs 2024-07-26T18:11:57.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:56 smithi112 ceph-mon[37501]: pgmap v10572: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:11:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:11:57 smithi107 ceph-mon[32659]: pgmap v10573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:11:57 smithi132 ceph-mon[37580]: pgmap v10573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:11:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:11:57 smithi112 ceph-mon[37501]: pgmap v10573: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:00 smithi107 ceph-mon[32659]: pgmap v10574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:00 smithi132 ceph-mon[37580]: pgmap v10574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:00 smithi112 ceph-mon[37501]: pgmap v10574: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19257.187686 secs 2024-07-26T18:12:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19257.187686 secs 2024-07-26T18:12:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19257.187686 secs 2024-07-26T18:12:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:02 smithi107 ceph-mon[32659]: pgmap v10575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:02.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:02 smithi132 ceph-mon[37580]: pgmap v10575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:02 smithi112 ceph-mon[37501]: pgmap v10575: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:04 smithi107 ceph-mon[32659]: pgmap v10576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:04 smithi132 ceph-mon[37580]: pgmap v10576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:04 smithi112 ceph-mon[37501]: pgmap v10576: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19262.187807 secs 2024-07-26T18:12:06.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:06 smithi107 ceph-mon[32659]: pgmap v10577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19262.187807 secs 2024-07-26T18:12:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:06 smithi132 ceph-mon[37580]: pgmap v10577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19262.187807 secs 2024-07-26T18:12:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:06 smithi112 ceph-mon[37501]: pgmap v10577: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:07.573 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:07.571+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:12:07.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:07.571+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:12:07.574 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:07.571+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:12:08.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:08 smithi107 ceph-mon[32659]: pgmap v10578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:08 smithi132 ceph-mon[37580]: pgmap v10578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:08 smithi112 ceph-mon[37501]: pgmap v10578: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:10 smithi107 ceph-mon[32659]: pgmap v10579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:10 smithi132 ceph-mon[37580]: pgmap v10579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:10 smithi112 ceph-mon[37501]: pgmap v10579: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19267.187920 secs 2024-07-26T18:12:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19267.187920 secs 2024-07-26T18:12:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19267.187920 secs 2024-07-26T18:12:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:12 smithi107 ceph-mon[32659]: pgmap v10580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:12 smithi132 ceph-mon[37580]: pgmap v10580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:12 smithi112 ceph-mon[37501]: pgmap v10580: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:14 smithi107 ceph-mon[32659]: pgmap v10581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:14 smithi132 ceph-mon[37580]: pgmap v10581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:14 smithi112 ceph-mon[37501]: pgmap v10581: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:12:15.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T18:12:15.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T18:12:15.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T18:12:15.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:15 smithi107 ceph-mon[32659]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T18:12:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:12:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.312 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T18:12:15.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T18:12:15.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T18:12:15.313 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:15 smithi132 ceph-mon[37580]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T18:12:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:12:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.452 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T18:12:15.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi112 to 176.2M 2024-07-26T18:12:15.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: Adjusting osd_memory_target on smithi132 to 176.2M 2024-07-26T18:12:15.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi112 to 184761651: error parsing value: Value '184761651' is below minimum 939524096 2024-07-26T18:12:15.453 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:15 smithi112 ceph-mon[37501]: Unable to set osd_memory_target on smithi132 to 184791756: error parsing value: Value '184791756' is below minimum 939524096 2024-07-26T18:12:15.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:15.977+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:12:15.979 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:15.977+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:12:15.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:15.982+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:12:15.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:15.983+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:12:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:12:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:12:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:16 smithi107 ceph-mon[32659]: pgmap v10582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T18:12:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:16 smithi107 ceph-mon[32659]: pgmap v10583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:12:16.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19272.188085 secs 2024-07-26T18:12:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:12:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:12:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:16 smithi132 ceph-mon[37580]: pgmap v10582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T18:12:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:16 smithi132 ceph-mon[37580]: pgmap v10583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:12:16.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19272.188085 secs 2024-07-26T18:12:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:12:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:12:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:16 smithi112 ceph-mon[37501]: pgmap v10582: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T18:12:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:16 smithi112 ceph-mon[37501]: pgmap v10583: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:12:16.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19272.188085 secs 2024-07-26T18:12:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:18 smithi112 ceph-mon[37501]: pgmap v10584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:18.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:18 smithi107 ceph-mon[32659]: pgmap v10584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:18.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:18 smithi132 ceph-mon[37580]: pgmap v10584: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:20 smithi112 ceph-mon[37501]: pgmap v10585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:20.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:20 smithi107 ceph-mon[32659]: pgmap v10585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:20.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:20 smithi132 ceph-mon[37580]: pgmap v10585: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19277.188212 secs 2024-07-26T18:12:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19277.188212 secs 2024-07-26T18:12:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19277.188212 secs 2024-07-26T18:12:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:22 smithi112 ceph-mon[37501]: pgmap v10586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:12:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:22 smithi107 ceph-mon[32659]: pgmap v10586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:12:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:22 smithi132 ceph-mon[37580]: pgmap v10586: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:12:22.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:22.580+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:12:22.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:22.580+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:12:22.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:22.580+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:12:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:24 smithi112 ceph-mon[37501]: pgmap v10587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:24.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:24 smithi107 ceph-mon[32659]: pgmap v10587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:24.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:24 smithi132 ceph-mon[37580]: pgmap v10587: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:26.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:26 smithi107 ceph-mon[32659]: pgmap v10588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:12:26.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19282.188361 secs 2024-07-26T18:12:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:26 smithi132 ceph-mon[37580]: pgmap v10588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:12:26.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19282.188361 secs 2024-07-26T18:12:26.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:26 smithi112 ceph-mon[37501]: pgmap v10588: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:12:26.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19282.188361 secs 2024-07-26T18:12:28.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:28 smithi107 ceph-mon[32659]: pgmap v10589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:28.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:28 smithi132 ceph-mon[37580]: pgmap v10589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:28.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:28 smithi112 ceph-mon[37501]: pgmap v10589: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:30 smithi107 ceph-mon[32659]: pgmap v10590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:30.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:30 smithi132 ceph-mon[37580]: pgmap v10590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:30.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:30 smithi112 ceph-mon[37501]: pgmap v10590: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19287.188444 secs 2024-07-26T18:12:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19287.188444 secs 2024-07-26T18:12:31.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19287.188444 secs 2024-07-26T18:12:32.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:32 smithi107 ceph-mon[32659]: pgmap v10591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:32.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:32 smithi132 ceph-mon[37580]: pgmap v10591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:32.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:32 smithi112 ceph-mon[37501]: pgmap v10591: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:34.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:34 smithi107 ceph-mon[32659]: pgmap v10592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:12:34.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:34 smithi132 ceph-mon[37580]: pgmap v10592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:12:34.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:34 smithi112 ceph-mon[37501]: pgmap v10592: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:12:36.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:36 smithi107 ceph-mon[32659]: pgmap v10593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:36.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19292.188579 secs 2024-07-26T18:12:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:36 smithi132 ceph-mon[37580]: pgmap v10593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:36.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19292.188579 secs 2024-07-26T18:12:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:36 smithi112 ceph-mon[37501]: pgmap v10593: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:36.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19292.188579 secs 2024-07-26T18:12:37.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:37.596+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:12:37.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:37.596+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:12:37.598 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:37.596+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:12:38.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:38 smithi107 ceph-mon[32659]: pgmap v10594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:38.562 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:38 smithi132 ceph-mon[37580]: pgmap v10594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:38.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:38 smithi112 ceph-mon[37501]: pgmap v10594: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:40.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:40 smithi107 ceph-mon[32659]: pgmap v10595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:40.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:40 smithi132 ceph-mon[37580]: pgmap v10595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:40.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:40 smithi112 ceph-mon[37501]: pgmap v10595: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:12:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19297.188709 secs 2024-07-26T18:12:41.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19297.188709 secs 2024-07-26T18:12:41.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19297.188709 secs 2024-07-26T18:12:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:42 smithi107 ceph-mon[32659]: pgmap v10596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:42 smithi132 ceph-mon[37580]: pgmap v10596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:42.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:42 smithi112 ceph-mon[37501]: pgmap v10596: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:44 smithi107 ceph-mon[32659]: pgmap v10597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T18:12:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:44 smithi132 ceph-mon[37580]: pgmap v10597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T18:12:44.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:44 smithi112 ceph-mon[37501]: pgmap v10597: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T18:12:46.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:46 smithi107 ceph-mon[32659]: pgmap v10598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:12:46.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19302.188852 secs 2024-07-26T18:12:46.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:46 smithi132 ceph-mon[37580]: pgmap v10598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:12:46.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19302.188852 secs 2024-07-26T18:12:46.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:46 smithi112 ceph-mon[37501]: pgmap v10598: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:12:46.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19302.188852 secs 2024-07-26T18:12:48.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:48 smithi107 ceph-mon[32659]: pgmap v10599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:12:48.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:48 smithi132 ceph-mon[37580]: pgmap v10599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:12:48.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:48 smithi112 ceph-mon[37501]: pgmap v10599: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:12:50.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:50 smithi107 ceph-mon[32659]: pgmap v10600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:50.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:50 smithi132 ceph-mon[37580]: pgmap v10600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:50.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:50 smithi112 ceph-mon[37501]: pgmap v10600: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:12:51.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19307.188975 secs 2024-07-26T18:12:51.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19307.188975 secs 2024-07-26T18:12:51.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19307.188975 secs 2024-07-26T18:12:52.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:52 smithi132 ceph-mon[37580]: pgmap v10601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T18:12:52.601 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:52 smithi112 ceph-mon[37501]: pgmap v10601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T18:12:52.613 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:52.612+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:12:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:52.612+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:12:52.614 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:52.612+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:12:52.614 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:52 smithi107 ceph-mon[32659]: pgmap v10601: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T18:12:54.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:54 smithi132 ceph-mon[37580]: pgmap v10602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T18:12:54.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:54 smithi112 ceph-mon[37501]: pgmap v10602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T18:12:54.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:54 smithi107 ceph-mon[32659]: pgmap v10602: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T18:12:56.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:56 smithi112 ceph-mon[37501]: pgmap v10603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:56.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19312.189102 secs 2024-07-26T18:12:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:56 smithi107 ceph-mon[32659]: pgmap v10603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19312.189102 secs 2024-07-26T18:12:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:56 smithi132 ceph-mon[37580]: pgmap v10603: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:12:56.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19312.189102 secs 2024-07-26T18:12:58.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:12:58 smithi112 ceph-mon[37501]: pgmap v10604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:58.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:12:58 smithi107 ceph-mon[32659]: pgmap v10604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:58.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:12:58 smithi132 ceph-mon[37580]: pgmap v10604: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:12:59.986 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:59.985+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:12:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:59.986+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:12:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:59.990+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:12:59.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:12:59.990+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:13:00.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:00 smithi112 ceph-mon[37501]: pgmap v10605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:00.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:00 smithi107 ceph-mon[32659]: pgmap v10605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:00.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:00 smithi132 ceph-mon[37580]: pgmap v10605: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:01.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19317.189224 secs 2024-07-26T18:13:01.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19317.189224 secs 2024-07-26T18:13:01.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19317.189224 secs 2024-07-26T18:13:02.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:02 smithi112 ceph-mon[37501]: pgmap v10606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:02.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:02 smithi107 ceph-mon[32659]: pgmap v10606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:02.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:02 smithi132 ceph-mon[37580]: pgmap v10606: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:04.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:04 smithi112 ceph-mon[37501]: pgmap v10607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:04.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:04 smithi107 ceph-mon[32659]: pgmap v10607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:04.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:04 smithi132 ceph-mon[37580]: pgmap v10607: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:06 smithi112 ceph-mon[37501]: pgmap v10608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:06.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19322.189355 secs 2024-07-26T18:13:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:06 smithi107 ceph-mon[32659]: pgmap v10608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:06.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19322.189355 secs 2024-07-26T18:13:06.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:06 smithi132 ceph-mon[37580]: pgmap v10608: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:06.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19322.189355 secs 2024-07-26T18:13:07.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:07.627+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:13:07.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:07.627+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:13:07.629 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:07.627+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:13:08.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:08 smithi112 ceph-mon[37501]: pgmap v10609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:08.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:08 smithi107 ceph-mon[32659]: pgmap v10609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:08.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:08 smithi132 ceph-mon[37580]: pgmap v10609: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:10.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:10 smithi112 ceph-mon[37501]: pgmap v10610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:10.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:10 smithi107 ceph-mon[32659]: pgmap v10610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:10.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:10 smithi132 ceph-mon[37580]: pgmap v10610: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19327.189485 secs 2024-07-26T18:13:11.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19327.189485 secs 2024-07-26T18:13:11.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19327.189485 secs 2024-07-26T18:13:12.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:12 smithi112 ceph-mon[37501]: pgmap v10611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:12.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:12 smithi107 ceph-mon[32659]: pgmap v10611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:12.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:12 smithi132 ceph-mon[37580]: pgmap v10611: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:14.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:14 smithi112 ceph-mon[37501]: pgmap v10612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:14.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:14 smithi107 ceph-mon[32659]: pgmap v10612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:14.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:14 smithi132 ceph-mon[37580]: pgmap v10612: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:15.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:15 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:13:15.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:15 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:13:15.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:15 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:13:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:16 smithi112 ceph-mon[37501]: pgmap v10613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19332.189557 secs 2024-07-26T18:13:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:13:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:13:16.701 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:16 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:13:16.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:16 smithi107 ceph-mon[32659]: pgmap v10613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19332.189557 secs 2024-07-26T18:13:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:13:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:13:16.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:16 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:13:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:16 smithi132 ceph-mon[37580]: pgmap v10613: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19332.189557 secs 2024-07-26T18:13:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:13:16.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:13:16.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:16 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:13:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:17 smithi112 ceph-mon[37501]: pgmap v10614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:13:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:17 smithi112 ceph-mon[37501]: pgmap v10615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:13:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:17 smithi107 ceph-mon[32659]: pgmap v10614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:13:17.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:17 smithi107 ceph-mon[32659]: pgmap v10615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:13:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:17 smithi132 ceph-mon[37580]: pgmap v10614: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:13:17.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:17 smithi132 ceph-mon[37580]: pgmap v10615: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:13:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:19 smithi112 ceph-mon[37501]: pgmap v10616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:13:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:19 smithi107 ceph-mon[32659]: pgmap v10616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:13:19.811 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:19 smithi132 ceph-mon[37580]: pgmap v10616: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:13:21.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:21 smithi112 ceph-mon[37501]: pgmap v10617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:13:21.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19337.189671 secs 2024-07-26T18:13:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:21 smithi107 ceph-mon[32659]: pgmap v10617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:13:22.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19337.189671 secs 2024-07-26T18:13:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:21 smithi132 ceph-mon[37580]: pgmap v10617: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:13:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19337.189671 secs 2024-07-26T18:13:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:22.629+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:13:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:22.629+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:13:22.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:22.629+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:13:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:23 smithi112 ceph-mon[37501]: pgmap v10618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T18:13:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:23 smithi107 ceph-mon[32659]: pgmap v10618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T18:13:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:23 smithi132 ceph-mon[37580]: pgmap v10618: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T18:13:25.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:25 smithi112 ceph-mon[37501]: pgmap v10619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:13:26.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:25 smithi107 ceph-mon[32659]: pgmap v10619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:13:26.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:25 smithi132 ceph-mon[37580]: pgmap v10619: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:13:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19342.189788 secs 2024-07-26T18:13:27.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19342.189788 secs 2024-07-26T18:13:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19342.189788 secs 2024-07-26T18:13:27.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:27 smithi112 ceph-mon[37501]: pgmap v10620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:13:28.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:27 smithi107 ceph-mon[32659]: pgmap v10620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:13:28.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:27 smithi132 ceph-mon[37580]: pgmap v10620: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:13:29.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:29 smithi112 ceph-mon[37501]: pgmap v10621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T18:13:30.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:29 smithi107 ceph-mon[32659]: pgmap v10621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T18:13:30.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:29 smithi132 ceph-mon[37580]: pgmap v10621: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T18:13:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:31 smithi112 ceph-mon[37501]: pgmap v10622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:31.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19347.189909 secs 2024-07-26T18:13:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:31 smithi107 ceph-mon[32659]: pgmap v10622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:32.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19347.189909 secs 2024-07-26T18:13:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:31 smithi132 ceph-mon[37580]: pgmap v10622: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:32.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19347.189909 secs 2024-07-26T18:13:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:33 smithi107 ceph-mon[32659]: pgmap v10623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:34.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:33 smithi132 ceph-mon[37580]: pgmap v10623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:33 smithi112 ceph-mon[37501]: pgmap v10623: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:35 smithi107 ceph-mon[32659]: pgmap v10624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:36.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:35 smithi132 ceph-mon[37580]: pgmap v10624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:35 smithi112 ceph-mon[37501]: pgmap v10624: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19352.190008 secs 2024-07-26T18:13:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19352.190008 secs 2024-07-26T18:13:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19352.190008 secs 2024-07-26T18:13:37.641 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:37.640+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:13:37.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:37.640+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:13:37.642 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:37.640+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:13:37.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:37 smithi107 ceph-mon[32659]: pgmap v10625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:37 smithi132 ceph-mon[37580]: pgmap v10625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:37 smithi112 ceph-mon[37501]: pgmap v10625: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:40.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:39 smithi107 ceph-mon[32659]: pgmap v10626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:39 smithi132 ceph-mon[37580]: pgmap v10626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:39 smithi112 ceph-mon[37501]: pgmap v10626: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:41 smithi107 ceph-mon[32659]: pgmap v10627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19357.190136 secs 2024-07-26T18:13:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:41 smithi132 ceph-mon[37580]: pgmap v10627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:42.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19357.190136 secs 2024-07-26T18:13:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:41 smithi112 ceph-mon[37501]: pgmap v10627: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19357.190136 secs 2024-07-26T18:13:44.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:43 smithi107 ceph-mon[32659]: pgmap v10628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:44.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:43 smithi132 ceph-mon[37580]: pgmap v10628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:43 smithi112 ceph-mon[37501]: pgmap v10628: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:46.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:45 smithi107 ceph-mon[32659]: pgmap v10629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:46.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:45 smithi132 ceph-mon[37580]: pgmap v10629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:45 smithi112 ceph-mon[37501]: pgmap v10629: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19362.190275 secs 2024-07-26T18:13:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19362.190275 secs 2024-07-26T18:13:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19362.190275 secs 2024-07-26T18:13:48.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:47 smithi107 ceph-mon[32659]: pgmap v10630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:48.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:47 smithi132 ceph-mon[37580]: pgmap v10630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:47 smithi112 ceph-mon[37501]: pgmap v10630: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:50.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:49 smithi107 ceph-mon[32659]: pgmap v10631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:50.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:49 smithi132 ceph-mon[37580]: pgmap v10631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:49 smithi112 ceph-mon[37501]: pgmap v10631: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:13:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:51 smithi107 ceph-mon[32659]: pgmap v10632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19367.190417 secs 2024-07-26T18:13:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:51 smithi132 ceph-mon[37580]: pgmap v10632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:52.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19367.190417 secs 2024-07-26T18:13:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:51 smithi112 ceph-mon[37501]: pgmap v10632: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:52.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19367.190417 secs 2024-07-26T18:13:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:52.655+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:13:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:52.655+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:13:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:13:52.655+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:13:54.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:53 smithi107 ceph-mon[32659]: pgmap v10633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:54.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:53 smithi132 ceph-mon[37580]: pgmap v10633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:54.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:53 smithi112 ceph-mon[37501]: pgmap v10633: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:13:56.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:55 smithi107 ceph-mon[32659]: pgmap v10634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:56.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:55 smithi132 ceph-mon[37580]: pgmap v10634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:55 smithi112 ceph-mon[37501]: pgmap v10634: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:13:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19372.190479 secs 2024-07-26T18:13:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19372.190479 secs 2024-07-26T18:13:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19372.190479 secs 2024-07-26T18:13:58.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:57 smithi132 ceph-mon[37580]: pgmap v10635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:57 smithi112 ceph-mon[37501]: pgmap v10635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:13:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:57 smithi107 ceph-mon[32659]: pgmap v10635: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:00.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:13:59 smithi132 ceph-mon[37580]: pgmap v10636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:14:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:13:59 smithi112 ceph-mon[37501]: pgmap v10636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:14:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:13:59 smithi107 ceph-mon[32659]: pgmap v10636: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:14:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:01 smithi132 ceph-mon[37580]: pgmap v10637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19377.190618 secs 2024-07-26T18:14:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:01 smithi112 ceph-mon[37501]: pgmap v10637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19377.190618 secs 2024-07-26T18:14:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:01 smithi107 ceph-mon[32659]: pgmap v10637: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:02.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19377.190618 secs 2024-07-26T18:14:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:03 smithi112 ceph-mon[37501]: pgmap v10638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:03 smithi107 ceph-mon[32659]: pgmap v10638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:03 smithi132 ceph-mon[37580]: pgmap v10638: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:05 smithi112 ceph-mon[37501]: pgmap v10639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:05 smithi107 ceph-mon[32659]: pgmap v10639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:05 smithi132 ceph-mon[37580]: pgmap v10639: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19382.190734 secs 2024-07-26T18:14:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19382.190734 secs 2024-07-26T18:14:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19382.190734 secs 2024-07-26T18:14:07.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:07.670+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:14:07.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:07.670+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:14:07.673 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:07.670+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:14:08.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:07 smithi107 ceph-mon[32659]: pgmap v10640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:07 smithi112 ceph-mon[37501]: pgmap v10640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:07 smithi132 ceph-mon[37580]: pgmap v10640: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:09 smithi112 ceph-mon[37501]: pgmap v10641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:14:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:09 smithi107 ceph-mon[32659]: pgmap v10641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:14:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:09 smithi132 ceph-mon[37580]: pgmap v10641: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:14:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:11 smithi112 ceph-mon[37501]: pgmap v10642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19387.190869 secs 2024-07-26T18:14:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:11 smithi107 ceph-mon[32659]: pgmap v10642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19387.190869 secs 2024-07-26T18:14:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:11 smithi132 ceph-mon[37580]: pgmap v10642: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19387.190869 secs 2024-07-26T18:14:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:13 smithi112 ceph-mon[37501]: pgmap v10643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:13 smithi107 ceph-mon[32659]: pgmap v10643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:13 smithi132 ceph-mon[37580]: pgmap v10643: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:15 smithi112 ceph-mon[37501]: pgmap v10644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:15 smithi107 ceph-mon[32659]: pgmap v10644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:15 smithi132 ceph-mon[37580]: pgmap v10644: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19392.190987 secs 2024-07-26T18:14:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:16 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:14:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19392.190987 secs 2024-07-26T18:14:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:16 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:14:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19392.190987 secs 2024-07-26T18:14:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:16 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:14:18.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:17 smithi112 ceph-mon[37501]: pgmap v10645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:18.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:14:18.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:17 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:14:18.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:17 smithi107 ceph-mon[32659]: pgmap v10645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:14:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:17 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:14:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:17 smithi132 ceph-mon[37580]: pgmap v10645: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:14:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:17 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:14:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:14:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:18 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:14:19.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:18 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:14:19.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:14:19.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:18 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:14:19.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:18 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:14:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:14:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:18 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:14:19.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:18 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:14:20.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:19 smithi112 ceph-mon[37501]: pgmap v10646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T18:14:20.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:19 smithi112 ceph-mon[37501]: pgmap v10647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:14:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:19 smithi107 ceph-mon[32659]: pgmap v10646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T18:14:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:19 smithi107 ceph-mon[32659]: pgmap v10647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:14:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:19 smithi132 ceph-mon[37580]: pgmap v10646: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T18:14:20.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:19 smithi132 ceph-mon[37580]: pgmap v10647: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:14:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:21 smithi112 ceph-mon[37501]: pgmap v10648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19397.191101 secs 2024-07-26T18:14:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:21 smithi107 ceph-mon[32659]: pgmap v10648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:22.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19397.191101 secs 2024-07-26T18:14:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:21 smithi132 ceph-mon[37580]: pgmap v10648: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19397.191101 secs 2024-07-26T18:14:22.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:22.686+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:14:22.688 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:22.686+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:14:22.689 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:22.686+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:14:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:23 smithi112 ceph-mon[37501]: pgmap v10649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:23 smithi107 ceph-mon[32659]: pgmap v10649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:23 smithi132 ceph-mon[37580]: pgmap v10649: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:25 smithi112 ceph-mon[37501]: pgmap v10650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:25 smithi107 ceph-mon[32659]: pgmap v10650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:25 smithi132 ceph-mon[37580]: pgmap v10650: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19402.191215 secs 2024-07-26T18:14:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19402.191215 secs 2024-07-26T18:14:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19402.191215 secs 2024-07-26T18:14:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:27 smithi112 ceph-mon[37501]: pgmap v10651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:27 smithi107 ceph-mon[32659]: pgmap v10651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:27 smithi132 ceph-mon[37580]: pgmap v10651: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T18:14:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:29 smithi112 ceph-mon[37501]: pgmap v10652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:14:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:29 smithi107 ceph-mon[32659]: pgmap v10652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:14:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:29 smithi132 ceph-mon[37580]: pgmap v10652: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:14:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:31 smithi112 ceph-mon[37501]: pgmap v10653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19407.191331 secs 2024-07-26T18:14:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:31 smithi107 ceph-mon[32659]: pgmap v10653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19407.191331 secs 2024-07-26T18:14:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:31 smithi132 ceph-mon[37580]: pgmap v10653: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19407.191331 secs 2024-07-26T18:14:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:33 smithi112 ceph-mon[37501]: pgmap v10654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:33 smithi107 ceph-mon[32659]: pgmap v10654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:33 smithi132 ceph-mon[37580]: pgmap v10654: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:35 smithi112 ceph-mon[37501]: pgmap v10655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:35 smithi107 ceph-mon[32659]: pgmap v10655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:35 smithi132 ceph-mon[37580]: pgmap v10655: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19412.191466 secs 2024-07-26T18:14:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19412.191466 secs 2024-07-26T18:14:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19412.191466 secs 2024-07-26T18:14:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:37.702+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:14:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:37.702+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:14:37.704 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:37.702+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:14:38.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:37 smithi107 ceph-mon[32659]: pgmap v10656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:37 smithi132 ceph-mon[37580]: pgmap v10656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:38.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:37 smithi112 ceph-mon[37501]: pgmap v10656: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:39 smithi107 ceph-mon[32659]: pgmap v10657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:39 smithi132 ceph-mon[37580]: pgmap v10657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:40.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:39 smithi112 ceph-mon[37501]: pgmap v10657: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:41 smithi107 ceph-mon[32659]: pgmap v10658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19417.191580 secs 2024-07-26T18:14:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:41 smithi132 ceph-mon[37580]: pgmap v10658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19417.191580 secs 2024-07-26T18:14:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:41 smithi112 ceph-mon[37501]: pgmap v10658: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:42.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19417.191580 secs 2024-07-26T18:14:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:43 smithi107 ceph-mon[32659]: pgmap v10659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:43 smithi132 ceph-mon[37580]: pgmap v10659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:43 smithi112 ceph-mon[37501]: pgmap v10659: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:45 smithi107 ceph-mon[32659]: pgmap v10660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:45 smithi132 ceph-mon[37580]: pgmap v10660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:46.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:45 smithi112 ceph-mon[37501]: pgmap v10660: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19422.191700 secs 2024-07-26T18:14:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19422.191700 secs 2024-07-26T18:14:47.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19422.191700 secs 2024-07-26T18:14:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:47 smithi107 ceph-mon[32659]: pgmap v10661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:47 smithi132 ceph-mon[37580]: pgmap v10661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:48.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:47 smithi112 ceph-mon[37501]: pgmap v10661: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:49 smithi107 ceph-mon[32659]: pgmap v10662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:49 smithi132 ceph-mon[37580]: pgmap v10662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:50.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:49 smithi112 ceph-mon[37501]: pgmap v10662: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:52 smithi107 ceph-mon[32659]: pgmap v10663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:52.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:52 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19427.191816 secs 2024-07-26T18:14:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:52 smithi132 ceph-mon[37580]: pgmap v10663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:52 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19427.191816 secs 2024-07-26T18:14:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:52 smithi112 ceph-mon[37501]: pgmap v10663: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:52 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19427.191816 secs 2024-07-26T18:14:52.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:52.718+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:14:52.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:52.718+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:14:52.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:14:52.718+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:14:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:53 smithi132 ceph-mon[37580]: pgmap v10664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:53.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:53 smithi112 ceph-mon[37501]: pgmap v10664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:53.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:53 smithi107 ceph-mon[32659]: pgmap v10664: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:14:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:55 smithi112 ceph-mon[37501]: pgmap v10665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:55.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:55 smithi107 ceph-mon[32659]: pgmap v10665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:55.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:55 smithi132 ceph-mon[37580]: pgmap v10665: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19432.191929 secs 2024-07-26T18:14:56.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19432.191929 secs 2024-07-26T18:14:56.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19432.191929 secs 2024-07-26T18:14:57.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:57 smithi112 ceph-mon[37501]: pgmap v10666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:57.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:57 smithi107 ceph-mon[32659]: pgmap v10666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:57.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:57 smithi132 ceph-mon[37580]: pgmap v10666: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:14:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:14:59 smithi112 ceph-mon[37501]: pgmap v10667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:59.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:14:59 smithi107 ceph-mon[32659]: pgmap v10667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:14:59.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:14:59 smithi132 ceph-mon[37580]: pgmap v10667: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:01 smithi112 ceph-mon[37501]: pgmap v10668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:01.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19437.192049 secs 2024-07-26T18:15:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:01 smithi107 ceph-mon[32659]: pgmap v10668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:01.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19437.192049 secs 2024-07-26T18:15:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:01 smithi132 ceph-mon[37580]: pgmap v10668: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:01.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19437.192049 secs 2024-07-26T18:15:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:03 smithi112 ceph-mon[37501]: pgmap v10669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:03.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:03 smithi107 ceph-mon[32659]: pgmap v10669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:03.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:03 smithi132 ceph-mon[37580]: pgmap v10669: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:05 smithi112 ceph-mon[37501]: pgmap v10670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:05.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:05 smithi107 ceph-mon[32659]: pgmap v10670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:05.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:05 smithi132 ceph-mon[37580]: pgmap v10670: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19442.192179 secs 2024-07-26T18:15:06.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19442.192179 secs 2024-07-26T18:15:06.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19442.192179 secs 2024-07-26T18:15:07.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:07 smithi107 ceph-mon[32659]: pgmap v10671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:07.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:07 smithi132 ceph-mon[37580]: pgmap v10671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:07.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:07 smithi112 ceph-mon[37501]: pgmap v10671: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:07.733+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:15:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:07.733+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:15:07.735 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:07.733+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:15:09.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:09 smithi107 ceph-mon[32659]: pgmap v10672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:09.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:09 smithi132 ceph-mon[37580]: pgmap v10672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:09.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:09 smithi112 ceph-mon[37501]: pgmap v10672: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:11.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:11 smithi107 ceph-mon[32659]: pgmap v10673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:11.345 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19447.192294 secs 2024-07-26T18:15:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:11 smithi132 ceph-mon[37580]: pgmap v10673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:11.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19447.192294 secs 2024-07-26T18:15:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:11 smithi112 ceph-mon[37501]: pgmap v10673: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:11.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19447.192294 secs 2024-07-26T18:15:13.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:13 smithi107 ceph-mon[32659]: pgmap v10674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:13.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:13 smithi132 ceph-mon[37580]: pgmap v10674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:13.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:13 smithi112 ceph-mon[37501]: pgmap v10674: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:15.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:15 smithi107 ceph-mon[32659]: pgmap v10675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:15:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:15 smithi132 ceph-mon[37580]: pgmap v10675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:15:15.703 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:15 smithi112 ceph-mon[37501]: pgmap v10675: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:15:16.010 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:16.007+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:15:16.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:16.008+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:15:16.011 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:16.009+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:15:16.012 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:16.010+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:15:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19452.192417 secs 2024-07-26T18:15:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19452.192417 secs 2024-07-26T18:15:16.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19452.192417 secs 2024-07-26T18:15:17.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:17 smithi107 ceph-mon[32659]: pgmap v10676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T18:15:17.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:17 smithi132 ceph-mon[37580]: pgmap v10676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T18:15:17.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:17 smithi112 ceph-mon[37501]: pgmap v10676: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T18:15:19.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:19 smithi107 ceph-mon[32659]: pgmap v10677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:15:19.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:19 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:15:19.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:19 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:19 smithi112 ceph-mon[37501]: pgmap v10677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:15:19.503 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:19 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:15:19.504 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.504 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.504 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.504 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:19 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.525 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:19 smithi132 ceph-mon[37580]: pgmap v10677: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:15:19.526 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:19 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:15:19.526 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.526 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.526 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:19.526 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:19 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:21 smithi112 ceph-mon[37501]: pgmap v10678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19457.192489 secs 2024-07-26T18:15:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:21 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:21 smithi107 ceph-mon[32659]: pgmap v10678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19457.192489 secs 2024-07-26T18:15:21.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:21 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:21 smithi132 ceph-mon[37580]: pgmap v10678: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19457.192489 secs 2024-07-26T18:15:21.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:21.561 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:21 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:22 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:22 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:22 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:22.750 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:22.748+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:15:22.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:22.748+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:15:22.751 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:22.748+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:15:23.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:23 smithi112 ceph-mon[37501]: pgmap v10679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:15:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:15:23.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:23 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:23 smithi107 ceph-mon[32659]: pgmap v10679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:15:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:15:24.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:23 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:23 smithi132 ceph-mon[37580]: pgmap v10679: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:15:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:15:24.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:23 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:15:24.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:24 smithi112 ceph-mon[37501]: pgmap v10680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T18:15:24.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:24 smithi112 ceph-mon[37501]: pgmap v10681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:15:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:24 smithi107 ceph-mon[32659]: pgmap v10680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T18:15:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:24 smithi107 ceph-mon[32659]: pgmap v10681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:15:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:24 smithi132 ceph-mon[37580]: pgmap v10680: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T18:15:25.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:24 smithi132 ceph-mon[37580]: pgmap v10681: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:15:26.938 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:26 smithi107 ceph-mon[32659]: pgmap v10682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:15:26.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19462.192583 secs 2024-07-26T18:15:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:26 smithi112 ceph-mon[37501]: pgmap v10682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:15:26.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19462.192583 secs 2024-07-26T18:15:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:26 smithi132 ceph-mon[37580]: pgmap v10682: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:15:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19462.192583 secs 2024-07-26T18:15:28.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:28 smithi112 ceph-mon[37501]: pgmap v10683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:29.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:28 smithi107 ceph-mon[32659]: pgmap v10683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:28 smithi132 ceph-mon[37580]: pgmap v10683: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:30.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:30 smithi112 ceph-mon[37501]: pgmap v10684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:15:31.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:30 smithi107 ceph-mon[32659]: pgmap v10684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:15:31.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:30 smithi132 ceph-mon[37580]: pgmap v10684: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:15:31.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19467.192678 secs 2024-07-26T18:15:32.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19467.192678 secs 2024-07-26T18:15:32.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19467.192678 secs 2024-07-26T18:15:32.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:32 smithi112 ceph-mon[37501]: pgmap v10685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T18:15:33.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:32 smithi107 ceph-mon[32659]: pgmap v10685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T18:15:33.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:32 smithi132 ceph-mon[37580]: pgmap v10685: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T18:15:34.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:34 smithi112 ceph-mon[37501]: pgmap v10686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:15:35.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:34 smithi107 ceph-mon[32659]: pgmap v10686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:15:35.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:34 smithi132 ceph-mon[37580]: pgmap v10686: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:15:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:36 smithi112 ceph-mon[37501]: pgmap v10687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:36.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19472.192818 secs 2024-07-26T18:15:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:36 smithi107 ceph-mon[32659]: pgmap v10687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:37.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19472.192818 secs 2024-07-26T18:15:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:36 smithi132 ceph-mon[37580]: pgmap v10687: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19472.192818 secs 2024-07-26T18:15:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:37.762+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:15:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:37.762+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:15:37.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:37.762+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:15:38.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:38 smithi107 ceph-mon[32659]: pgmap v10688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:38.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:38 smithi112 ceph-mon[37501]: pgmap v10688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:39.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:38 smithi132 ceph-mon[37580]: pgmap v10688: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:40.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:40 smithi112 ceph-mon[37501]: pgmap v10689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:41.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:40 smithi107 ceph-mon[32659]: pgmap v10689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:41.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:40 smithi132 ceph-mon[37580]: pgmap v10689: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:41.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19477.192929 secs 2024-07-26T18:15:42.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19477.192929 secs 2024-07-26T18:15:42.062 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19477.192929 secs 2024-07-26T18:15:42.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:42 smithi112 ceph-mon[37501]: pgmap v10690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:43.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:42 smithi107 ceph-mon[32659]: pgmap v10690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:43.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:42 smithi132 ceph-mon[37580]: pgmap v10690: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:44.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:44 smithi112 ceph-mon[37501]: pgmap v10691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:15:45.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:44 smithi107 ceph-mon[32659]: pgmap v10691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:15:45.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:44 smithi132 ceph-mon[37580]: pgmap v10691: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:15:46.950 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:46 smithi112 ceph-mon[37501]: pgmap v10692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:46.951 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19482.193050 secs 2024-07-26T18:15:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:46 smithi107 ceph-mon[32659]: pgmap v10692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:47.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19482.193050 secs 2024-07-26T18:15:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:46 smithi132 ceph-mon[37580]: pgmap v10692: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:47.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19482.193050 secs 2024-07-26T18:15:49.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:48 smithi107 ceph-mon[32659]: pgmap v10693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:49.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:48 smithi132 ceph-mon[37580]: pgmap v10693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:48 smithi112 ceph-mon[37501]: pgmap v10693: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:51.017 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:50 smithi107 ceph-mon[32659]: pgmap v10694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:51.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:50 smithi132 ceph-mon[37580]: pgmap v10694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:50 smithi112 ceph-mon[37501]: pgmap v10694: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:15:52.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19487.193165 secs 2024-07-26T18:15:52.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19487.193165 secs 2024-07-26T18:15:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19487.193165 secs 2024-07-26T18:15:52.770 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:52.766+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:15:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:52.766+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:15:52.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:15:52.766+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:15:53.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:52 smithi107 ceph-mon[32659]: pgmap v10695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:53.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:52 smithi132 ceph-mon[37580]: pgmap v10695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:52 smithi112 ceph-mon[37501]: pgmap v10695: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:55.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:54 smithi107 ceph-mon[32659]: pgmap v10696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:15:55.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:54 smithi132 ceph-mon[37580]: pgmap v10696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:15:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:54 smithi112 ceph-mon[37501]: pgmap v10696: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:15:57.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:56 smithi112 ceph-mon[37501]: pgmap v10697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:57.003 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19492.193280 secs 2024-07-26T18:15:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:56 smithi107 ceph-mon[32659]: pgmap v10697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:57.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19492.193280 secs 2024-07-26T18:15:57.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:56 smithi132 ceph-mon[37580]: pgmap v10697: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:15:57.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19492.193280 secs 2024-07-26T18:15:59.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:15:58 smithi107 ceph-mon[32659]: pgmap v10698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:59.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:15:58 smithi132 ceph-mon[37580]: pgmap v10698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:15:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:15:58 smithi112 ceph-mon[37501]: pgmap v10698: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:01.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:00 smithi107 ceph-mon[32659]: pgmap v10699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:01.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:00 smithi132 ceph-mon[37580]: pgmap v10699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:01.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:00 smithi112 ceph-mon[37501]: pgmap v10699: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:02.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19497.193399 secs 2024-07-26T18:16:02.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19497.193399 secs 2024-07-26T18:16:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19497.193399 secs 2024-07-26T18:16:03.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:02 smithi107 ceph-mon[32659]: pgmap v10700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:03.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:02 smithi132 ceph-mon[37580]: pgmap v10700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:02 smithi112 ceph-mon[37501]: pgmap v10700: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:05.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:04 smithi107 ceph-mon[32659]: pgmap v10701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:05.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:04 smithi132 ceph-mon[37580]: pgmap v10701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:04 smithi112 ceph-mon[37501]: pgmap v10701: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:06 smithi107 ceph-mon[32659]: pgmap v10702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:07.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19502.193517 secs 2024-07-26T18:16:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:06 smithi132 ceph-mon[37580]: pgmap v10702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:07.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19502.193517 secs 2024-07-26T18:16:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:06 smithi112 ceph-mon[37501]: pgmap v10702: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19502.193517 secs 2024-07-26T18:16:07.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:07.781+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:16:07.783 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:07.781+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:16:07.784 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:07.781+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:16:09.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:08 smithi107 ceph-mon[32659]: pgmap v10703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:09.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:08 smithi132 ceph-mon[37580]: pgmap v10703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:09.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:08 smithi112 ceph-mon[37501]: pgmap v10703: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:11.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:10 smithi107 ceph-mon[32659]: pgmap v10704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:11.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:10 smithi132 ceph-mon[37580]: pgmap v10704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:11.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:10 smithi112 ceph-mon[37501]: pgmap v10704: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:12.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19507.193637 secs 2024-07-26T18:16:12.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19507.193637 secs 2024-07-26T18:16:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19507.193637 secs 2024-07-26T18:16:13.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:12 smithi132 ceph-mon[37580]: pgmap v10705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:13.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:12 smithi112 ceph-mon[37501]: pgmap v10705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:13.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:12 smithi107 ceph-mon[32659]: pgmap v10705: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:15.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:14 smithi132 ceph-mon[37580]: pgmap v10706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:15.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:14 smithi112 ceph-mon[37501]: pgmap v10706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:14 smithi107 ceph-mon[32659]: pgmap v10706: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:16 smithi132 ceph-mon[37580]: pgmap v10707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:17.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19512.193754 secs 2024-07-26T18:16:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:16 smithi112 ceph-mon[37501]: pgmap v10707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19512.193754 secs 2024-07-26T18:16:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:16 smithi107 ceph-mon[32659]: pgmap v10707: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19512.193754 secs 2024-07-26T18:16:19.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:18 smithi112 ceph-mon[37501]: pgmap v10708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:18 smithi107 ceph-mon[32659]: pgmap v10708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:18 smithi132 ceph-mon[37580]: pgmap v10708: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:21.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:20 smithi132 ceph-mon[37580]: pgmap v10709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:21.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:20 smithi112 ceph-mon[37501]: pgmap v10709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:20 smithi107 ceph-mon[32659]: pgmap v10709: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:22.128 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19517.193892 secs 2024-07-26T18:16:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19517.193892 secs 2024-07-26T18:16:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19517.193892 secs 2024-07-26T18:16:22.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:22.796+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:16:22.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:22.796+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:16:22.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:22.796+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:16:23.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:22 smithi112 ceph-mon[37501]: pgmap v10710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:22 smithi107 ceph-mon[32659]: pgmap v10710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:22 smithi132 ceph-mon[37580]: pgmap v10710: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:23 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:16:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:23 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:16:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:23 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:16:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:24 smithi112 ceph-mon[37501]: pgmap v10711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:16:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:16:25.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:24 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:16:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:24 smithi107 ceph-mon[32659]: pgmap v10711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:16:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:16:25.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:24 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:16:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:24 smithi132 ceph-mon[37580]: pgmap v10711: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:16:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:16:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:24 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:16:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:25 smithi112 ceph-mon[37501]: pgmap v10712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:16:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:25 smithi112 ceph-mon[37501]: pgmap v10713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:16:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:25 smithi107 ceph-mon[32659]: pgmap v10712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:16:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:25 smithi107 ceph-mon[32659]: pgmap v10713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:16:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:25 smithi132 ceph-mon[37580]: pgmap v10712: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:16:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:25 smithi132 ceph-mon[37580]: pgmap v10713: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:16:27.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19522.193990 secs 2024-07-26T18:16:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19522.193990 secs 2024-07-26T18:16:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19522.193990 secs 2024-07-26T18:16:28.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:27 smithi112 ceph-mon[37501]: pgmap v10714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:16:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:27 smithi107 ceph-mon[32659]: pgmap v10714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:16:28.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:27 smithi132 ceph-mon[37580]: pgmap v10714: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:16:30.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:29 smithi112 ceph-mon[37501]: pgmap v10715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:16:30.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:29 smithi107 ceph-mon[32659]: pgmap v10715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:16:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:29 smithi132 ceph-mon[37580]: pgmap v10715: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:16:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:31 smithi112 ceph-mon[37501]: pgmap v10716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:16:32.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19527.194113 secs 2024-07-26T18:16:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:31 smithi107 ceph-mon[32659]: pgmap v10716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:16:32.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19527.194113 secs 2024-07-26T18:16:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:31 smithi132 ceph-mon[37580]: pgmap v10716: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:16:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19527.194113 secs 2024-07-26T18:16:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:33 smithi112 ceph-mon[37501]: pgmap v10717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T18:16:34.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:33 smithi107 ceph-mon[32659]: pgmap v10717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T18:16:34.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:33 smithi132 ceph-mon[37580]: pgmap v10717: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T18:16:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:35 smithi112 ceph-mon[37501]: pgmap v10718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:16:36.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:35 smithi107 ceph-mon[32659]: pgmap v10718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:16:36.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:35 smithi132 ceph-mon[37580]: pgmap v10718: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:16:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19532.194232 secs 2024-07-26T18:16:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19532.194232 secs 2024-07-26T18:16:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19532.194232 secs 2024-07-26T18:16:37.804 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:37.803+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:16:37.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:37.803+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:16:37.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:37.803+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:16:38.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:37 smithi107 ceph-mon[32659]: pgmap v10719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:37 smithi112 ceph-mon[37501]: pgmap v10719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:38.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:37 smithi132 ceph-mon[37580]: pgmap v10719: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:39 smithi112 ceph-mon[37501]: pgmap v10720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:39 smithi107 ceph-mon[32659]: pgmap v10720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:40.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:39 smithi132 ceph-mon[37580]: pgmap v10720: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:41 smithi112 ceph-mon[37501]: pgmap v10721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:42.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19537.194355 secs 2024-07-26T18:16:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:41 smithi107 ceph-mon[32659]: pgmap v10721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:42.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19537.194355 secs 2024-07-26T18:16:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:41 smithi132 ceph-mon[37580]: pgmap v10721: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19537.194355 secs 2024-07-26T18:16:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:43 smithi112 ceph-mon[37501]: pgmap v10722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:43 smithi107 ceph-mon[32659]: pgmap v10722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:43 smithi132 ceph-mon[37580]: pgmap v10722: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:45 smithi112 ceph-mon[37501]: pgmap v10723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:45 smithi107 ceph-mon[32659]: pgmap v10723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:46.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:45 smithi132 ceph-mon[37580]: pgmap v10723: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19542.194476 secs 2024-07-26T18:16:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19542.194476 secs 2024-07-26T18:16:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19542.194476 secs 2024-07-26T18:16:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:47 smithi112 ceph-mon[37501]: pgmap v10724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:47 smithi107 ceph-mon[32659]: pgmap v10724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:47 smithi132 ceph-mon[37580]: pgmap v10724: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:49 smithi112 ceph-mon[37501]: pgmap v10725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:49 smithi107 ceph-mon[32659]: pgmap v10725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:49 smithi132 ceph-mon[37580]: pgmap v10725: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:16:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:51 smithi107 ceph-mon[32659]: pgmap v10726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19547.194595 secs 2024-07-26T18:16:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:51 smithi132 ceph-mon[37580]: pgmap v10726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:52.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19547.194595 secs 2024-07-26T18:16:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:51 smithi112 ceph-mon[37501]: pgmap v10726: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:52.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19547.194595 secs 2024-07-26T18:16:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:52.804+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:16:52.805 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:52.804+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:16:52.806 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:16:52.804+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:16:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:53 smithi107 ceph-mon[32659]: pgmap v10727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:53 smithi132 ceph-mon[37580]: pgmap v10727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:54.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:53 smithi112 ceph-mon[37501]: pgmap v10727: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:16:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:55 smithi107 ceph-mon[32659]: pgmap v10728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:55 smithi132 ceph-mon[37580]: pgmap v10728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:56.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:55 smithi112 ceph-mon[37501]: pgmap v10728: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:16:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19552.194659 secs 2024-07-26T18:16:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19552.194659 secs 2024-07-26T18:16:57.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19552.194659 secs 2024-07-26T18:16:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:57 smithi107 ceph-mon[32659]: pgmap v10729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:57 smithi132 ceph-mon[37580]: pgmap v10729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:16:58.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:57 smithi112 ceph-mon[37501]: pgmap v10729: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:16:59 smithi107 ceph-mon[32659]: pgmap v10730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:16:59 smithi132 ceph-mon[37580]: pgmap v10730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:00.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:16:59 smithi112 ceph-mon[37501]: pgmap v10730: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:01 smithi107 ceph-mon[32659]: pgmap v10731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19557.194771 secs 2024-07-26T18:17:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:01 smithi132 ceph-mon[37580]: pgmap v10731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19557.194771 secs 2024-07-26T18:17:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:01 smithi112 ceph-mon[37501]: pgmap v10731: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:02.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19557.194771 secs 2024-07-26T18:17:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:03 smithi107 ceph-mon[32659]: pgmap v10732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:17:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:03 smithi132 ceph-mon[37580]: pgmap v10732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:17:04.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:03 smithi112 ceph-mon[37501]: pgmap v10732: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:17:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:06 smithi107 ceph-mon[32659]: pgmap v10733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:06.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:06 smithi132 ceph-mon[37580]: pgmap v10733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:06.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:06 smithi112 ceph-mon[37501]: pgmap v10733: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:07 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19562.194925 secs 2024-07-26T18:17:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:07 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19562.194925 secs 2024-07-26T18:17:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:07 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19562.194925 secs 2024-07-26T18:17:07.811 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:07.810+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:17:07.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:07.810+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:17:07.812 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:07.810+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:17:08.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:08 smithi107 ceph-mon[32659]: pgmap v10734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:08 smithi132 ceph-mon[37580]: pgmap v10734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:08.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:08 smithi112 ceph-mon[37501]: pgmap v10734: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:10 smithi107 ceph-mon[32659]: pgmap v10735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:10.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:10 smithi132 ceph-mon[37580]: pgmap v10735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:10.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:10 smithi112 ceph-mon[37501]: pgmap v10735: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:12 smithi107 ceph-mon[32659]: pgmap v10736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19567.195051 secs 2024-07-26T18:17:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:12 smithi132 ceph-mon[37580]: pgmap v10736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19567.195051 secs 2024-07-26T18:17:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:12 smithi112 ceph-mon[37501]: pgmap v10736: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:12.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19567.195051 secs 2024-07-26T18:17:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:14 smithi107 ceph-mon[32659]: pgmap v10737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:17:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:14 smithi132 ceph-mon[37580]: pgmap v10737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:17:14.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:14 smithi112 ceph-mon[37501]: pgmap v10737: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:17:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:15 smithi112 ceph-mon[37501]: pgmap v10738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:15.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:15 smithi107 ceph-mon[32659]: pgmap v10738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:15.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:15 smithi132 ceph-mon[37580]: pgmap v10738: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:16.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19572.195167 secs 2024-07-26T18:17:16.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19572.195167 secs 2024-07-26T18:17:16.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19572.195167 secs 2024-07-26T18:17:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:17 smithi107 ceph-mon[32659]: pgmap v10739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:17 smithi112 ceph-mon[37501]: pgmap v10739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:17.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:17 smithi132 ceph-mon[37580]: pgmap v10739: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:19.700 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:19 smithi112 ceph-mon[37501]: pgmap v10740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:19 smithi107 ceph-mon[32659]: pgmap v10740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:19.810 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:19 smithi132 ceph-mon[37580]: pgmap v10740: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:21 smithi107 ceph-mon[32659]: pgmap v10741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:22.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19577.195289 secs 2024-07-26T18:17:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:21 smithi132 ceph-mon[37580]: pgmap v10741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:22.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19577.195289 secs 2024-07-26T18:17:22.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:21 smithi112 ceph-mon[37501]: pgmap v10741: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:22.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19577.195289 secs 2024-07-26T18:17:22.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:22.823+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:17:22.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:22.823+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:17:22.826 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:22.823+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:17:24.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:23 smithi107 ceph-mon[32659]: pgmap v10742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:17:24.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:23 smithi132 ceph-mon[37580]: pgmap v10742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:17:24.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:23 smithi112 ceph-mon[37501]: pgmap v10742: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:17:25.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:24 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:17:25.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:24 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:17:25.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:24 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:17:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:25 smithi132 ceph-mon[37580]: pgmap v10743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:17:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:25 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:17:26.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:25 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:17:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:25 smithi112 ceph-mon[37501]: pgmap v10743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:17:26.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:25 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:17:26.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:25 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:17:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:25 smithi107 ceph-mon[32659]: pgmap v10743: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:17:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:25 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:17:26.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:25 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:17:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:26 smithi132 ceph-mon[37580]: pgmap v10744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:17:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:26 smithi132 ceph-mon[37580]: pgmap v10745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:17:27.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19582.195385 secs 2024-07-26T18:17:27.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:26 smithi112 ceph-mon[37501]: pgmap v10744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:17:27.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:26 smithi112 ceph-mon[37501]: pgmap v10745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:17:27.102 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19582.195385 secs 2024-07-26T18:17:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:26 smithi107 ceph-mon[32659]: pgmap v10744: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T18:17:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:26 smithi107 ceph-mon[32659]: pgmap v10745: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:17:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19582.195385 secs 2024-07-26T18:17:29.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:28 smithi132 ceph-mon[37580]: pgmap v10746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:17:29.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:28 smithi112 ceph-mon[37501]: pgmap v10746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:17:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:28 smithi107 ceph-mon[32659]: pgmap v10746: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:17:31.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:30 smithi112 ceph-mon[37501]: pgmap v10747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:17:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:30 smithi107 ceph-mon[32659]: pgmap v10747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:17:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:30 smithi132 ceph-mon[37580]: pgmap v10747: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:17:32.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19587.195500 secs 2024-07-26T18:17:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19587.195500 secs 2024-07-26T18:17:32.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19587.195500 secs 2024-07-26T18:17:33.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:32 smithi112 ceph-mon[37501]: pgmap v10748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:17:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:32 smithi107 ceph-mon[32659]: pgmap v10748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:17:33.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:32 smithi132 ceph-mon[37580]: pgmap v10748: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:17:35.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:34 smithi112 ceph-mon[37501]: pgmap v10749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:17:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:34 smithi107 ceph-mon[32659]: pgmap v10749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:17:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:34 smithi132 ceph-mon[37580]: pgmap v10749: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:17:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:36 smithi112 ceph-mon[37501]: pgmap v10750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:17:37.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19592.195638 secs 2024-07-26T18:17:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:36 smithi107 ceph-mon[32659]: pgmap v10750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:17:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19592.195638 secs 2024-07-26T18:17:37.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:36 smithi132 ceph-mon[37580]: pgmap v10750: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:17:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19592.195638 secs 2024-07-26T18:17:37.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:37.838+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:17:37.840 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:37.838+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:17:37.841 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:37.838+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:17:39.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:38 smithi112 ceph-mon[37501]: pgmap v10751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:39.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:38 smithi107 ceph-mon[32659]: pgmap v10751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:39.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:38 smithi132 ceph-mon[37580]: pgmap v10751: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:41.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:40 smithi112 ceph-mon[37501]: pgmap v10752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:17:41.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:40 smithi107 ceph-mon[32659]: pgmap v10752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:17:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:40 smithi132 ceph-mon[37580]: pgmap v10752: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:17:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19597.195755 secs 2024-07-26T18:17:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19597.195755 secs 2024-07-26T18:17:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19597.195755 secs 2024-07-26T18:17:43.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:42 smithi112 ceph-mon[37501]: pgmap v10753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:17:43.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:42 smithi107 ceph-mon[32659]: pgmap v10753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:17:43.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:42 smithi132 ceph-mon[37580]: pgmap v10753: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:17:45.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:44 smithi112 ceph-mon[37501]: pgmap v10754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:45.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:44 smithi107 ceph-mon[32659]: pgmap v10754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:45.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:44 smithi132 ceph-mon[37580]: pgmap v10754: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:46 smithi112 ceph-mon[37501]: pgmap v10755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19602.195877 secs 2024-07-26T18:17:47.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:46 smithi107 ceph-mon[32659]: pgmap v10755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:47.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19602.195877 secs 2024-07-26T18:17:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:46 smithi132 ceph-mon[37580]: pgmap v10755: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19602.195877 secs 2024-07-26T18:17:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:48 smithi112 ceph-mon[37501]: pgmap v10756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:48 smithi107 ceph-mon[32659]: pgmap v10756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:48 smithi132 ceph-mon[37580]: pgmap v10756: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:50 smithi112 ceph-mon[37501]: pgmap v10757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:17:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:50 smithi107 ceph-mon[32659]: pgmap v10757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:17:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:50 smithi132 ceph-mon[37580]: pgmap v10757: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:17:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19607.195966 secs 2024-07-26T18:17:52.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19607.195966 secs 2024-07-26T18:17:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19607.195966 secs 2024-07-26T18:17:52.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:52.849+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:17:52.850 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:52.849+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:17:52.851 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:17:52.849+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:17:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:52 smithi112 ceph-mon[37501]: pgmap v10758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:17:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:52 smithi107 ceph-mon[32659]: pgmap v10758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:17:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:52 smithi132 ceph-mon[37580]: pgmap v10758: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:17:55.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:54 smithi112 ceph-mon[37501]: pgmap v10759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:54 smithi107 ceph-mon[32659]: pgmap v10759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:54 smithi132 ceph-mon[37580]: pgmap v10759: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:17:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:56 smithi112 ceph-mon[37501]: pgmap v10760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:57.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19612.196100 secs 2024-07-26T18:17:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:56 smithi107 ceph-mon[32659]: pgmap v10760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19612.196100 secs 2024-07-26T18:17:57.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:56 smithi132 ceph-mon[37580]: pgmap v10760: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:17:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19612.196100 secs 2024-07-26T18:17:59.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:17:58 smithi112 ceph-mon[37501]: pgmap v10761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:17:58 smithi107 ceph-mon[32659]: pgmap v10761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:17:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:17:58 smithi132 ceph-mon[37580]: pgmap v10761: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:01.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:00 smithi112 ceph-mon[37501]: pgmap v10762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:00 smithi107 ceph-mon[32659]: pgmap v10762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:00 smithi132 ceph-mon[37580]: pgmap v10762: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19617.196190 secs 2024-07-26T18:18:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19617.196190 secs 2024-07-26T18:18:02.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19617.196190 secs 2024-07-26T18:18:03.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:02 smithi112 ceph-mon[37501]: pgmap v10763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:18:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:02 smithi107 ceph-mon[32659]: pgmap v10763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:18:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:02 smithi132 ceph-mon[37580]: pgmap v10763: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:18:05.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:04 smithi112 ceph-mon[37501]: pgmap v10764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:18:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:04 smithi107 ceph-mon[32659]: pgmap v10764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:18:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:04 smithi132 ceph-mon[37580]: pgmap v10764: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:18:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:06 smithi112 ceph-mon[37501]: pgmap v10765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19622.196315 secs 2024-07-26T18:18:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:06 smithi107 ceph-mon[32659]: pgmap v10765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19622.196315 secs 2024-07-26T18:18:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:06 smithi132 ceph-mon[37580]: pgmap v10765: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19622.196315 secs 2024-07-26T18:18:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:07.859+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:18:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:07.859+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:18:07.861 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:07.859+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:18:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:08 smithi107 ceph-mon[32659]: pgmap v10766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:09.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:08 smithi132 ceph-mon[37580]: pgmap v10766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:08 smithi112 ceph-mon[37501]: pgmap v10766: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:10 smithi107 ceph-mon[32659]: pgmap v10767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:11.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:10 smithi132 ceph-mon[37580]: pgmap v10767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:10 smithi112 ceph-mon[37501]: pgmap v10767: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19627.196430 secs 2024-07-26T18:18:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19627.196430 secs 2024-07-26T18:18:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19627.196430 secs 2024-07-26T18:18:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:12 smithi107 ceph-mon[32659]: pgmap v10768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:18:13.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:12 smithi132 ceph-mon[37580]: pgmap v10768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:18:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:12 smithi112 ceph-mon[37501]: pgmap v10768: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:18:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:14 smithi107 ceph-mon[32659]: pgmap v10769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:18:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:14 smithi132 ceph-mon[37580]: pgmap v10769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:18:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:14 smithi112 ceph-mon[37501]: pgmap v10769: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:18:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:16.038+0000 7ff0f7e00640 1 -- 172.21.15.107:0/1770124043 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d80065c0 con 0x7ff104102930 2024-07-26T18:18:16.040 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:16.038+0000 7ff0e9600640 1 -- 172.21.15.107:0/4208142334 --> [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7ff0d4006640 con 0x565421993390 2024-07-26T18:18:16.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:16.039+0000 7ff101600640 1 -- 172.21.15.107:0/1770124043 <== mon.2 v2:172.21.15.112:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7ff0d4006640 con 0x7ff104102930 2024-07-26T18:18:16.041 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:16.039+0000 7ff0eaa00640 1 -- 172.21.15.107:0/4208142334 <== mon.2 v2:172.21.15.112:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7ff0d80065c0 con 0x565421993390 2024-07-26T18:18:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:16 smithi107 ceph-mon[32659]: pgmap v10770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19632.196562 secs 2024-07-26T18:18:17.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:16 smithi132 ceph-mon[37580]: pgmap v10770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19632.196562 secs 2024-07-26T18:18:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:16 smithi112 ceph-mon[37501]: pgmap v10770: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:17.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19632.196562 secs 2024-07-26T18:18:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:18 smithi107 ceph-mon[32659]: pgmap v10771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:19.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:18 smithi132 ceph-mon[37580]: pgmap v10771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:18 smithi112 ceph-mon[37501]: pgmap v10771: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:21.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:20 smithi107 ceph-mon[32659]: pgmap v10772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:20 smithi132 ceph-mon[37580]: pgmap v10772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:20 smithi112 ceph-mon[37501]: pgmap v10772: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19637.196684 secs 2024-07-26T18:18:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19637.196684 secs 2024-07-26T18:18:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19637.196684 secs 2024-07-26T18:18:22.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:22.860+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:18:22.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:22.860+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:18:22.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:22.860+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:18:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:23 smithi107 ceph-mon[32659]: pgmap v10773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:18:23.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:23 smithi132 ceph-mon[37580]: pgmap v10773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:18:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:23 smithi112 ceph-mon[37501]: pgmap v10773: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T18:18:25.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:25 smithi107 ceph-mon[32659]: pgmap v10774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:18:25.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:25 smithi132 ceph-mon[37580]: pgmap v10774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:18:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:25 smithi112 ceph-mon[37501]: pgmap v10774: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T18:18:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:18:26.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:18:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:18:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:27 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19642.196811 secs 2024-07-26T18:18:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:27 smithi107 ceph-mon[32659]: pgmap v10775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:18:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:27 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:18:27.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:27 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:18:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:27 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19642.196811 secs 2024-07-26T18:18:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:27 smithi132 ceph-mon[37580]: pgmap v10775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:18:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:27 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:18:27.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:27 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:18:27.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:27 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19642.196811 secs 2024-07-26T18:18:27.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:27 smithi112 ceph-mon[37501]: pgmap v10775: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:27.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:18:27.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:27 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:18:27.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:27 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:18:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:28 smithi132 ceph-mon[37580]: pgmap v10776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:18:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:28 smithi132 ceph-mon[37580]: pgmap v10777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:28 smithi112 ceph-mon[37501]: pgmap v10776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:18:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:28 smithi112 ceph-mon[37501]: pgmap v10777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:28 smithi107 ceph-mon[32659]: pgmap v10776: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:18:28.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:28 smithi107 ceph-mon[32659]: pgmap v10777: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:30.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:30 smithi132 ceph-mon[37580]: pgmap v10778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:18:30.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:30 smithi112 ceph-mon[37501]: pgmap v10778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:18:30.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:30 smithi107 ceph-mon[32659]: pgmap v10778: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:18:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:31 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19647.196942 secs 2024-07-26T18:18:31.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:31 smithi112 ceph-mon[37501]: pgmap v10779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:18:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:31 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19647.196942 secs 2024-07-26T18:18:31.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:31 smithi107 ceph-mon[32659]: pgmap v10779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:18:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:31 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19647.196942 secs 2024-07-26T18:18:31.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:31 smithi132 ceph-mon[37580]: pgmap v10779: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:18:34.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:33 smithi107 ceph-mon[32659]: pgmap v10780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:18:34.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:33 smithi132 ceph-mon[37580]: pgmap v10780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:18:34.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:33 smithi112 ceph-mon[37501]: pgmap v10780: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T18:18:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:35 smithi107 ceph-mon[32659]: pgmap v10781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:18:36.059 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:35 smithi132 ceph-mon[37580]: pgmap v10781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:18:36.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:35 smithi112 ceph-mon[37501]: pgmap v10781: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:18:37.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:36 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19652.197040 secs 2024-07-26T18:18:37.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:36 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19652.197040 secs 2024-07-26T18:18:37.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:36 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19652.197040 secs 2024-07-26T18:18:37.867 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:37.865+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:18:37.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:37.865+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:18:37.868 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:37.865+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:18:38.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:37 smithi132 ceph-mon[37580]: pgmap v10782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:18:38.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:37 smithi107 ceph-mon[32659]: pgmap v10782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:18:38.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:37 smithi112 ceph-mon[37501]: pgmap v10782: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:18:40.060 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:39 smithi132 ceph-mon[37580]: pgmap v10783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:40.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:39 smithi112 ceph-mon[37501]: pgmap v10783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:40.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:39 smithi107 ceph-mon[32659]: pgmap v10783: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:41 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19657.197157 secs 2024-07-26T18:18:42.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:41 smithi112 ceph-mon[37501]: pgmap v10784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:41 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19657.197157 secs 2024-07-26T18:18:42.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:41 smithi107 ceph-mon[32659]: pgmap v10784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:41 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19657.197157 secs 2024-07-26T18:18:42.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:41 smithi132 ceph-mon[37580]: pgmap v10784: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:44.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:43 smithi112 ceph-mon[37501]: pgmap v10785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:44.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:43 smithi107 ceph-mon[32659]: pgmap v10785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:44.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:43 smithi132 ceph-mon[37580]: pgmap v10785: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:46.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:45 smithi112 ceph-mon[37501]: pgmap v10786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:46.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:45 smithi107 ceph-mon[32659]: pgmap v10786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:46.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:45 smithi132 ceph-mon[37580]: pgmap v10786: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19662.197273 secs 2024-07-26T18:18:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19662.197273 secs 2024-07-26T18:18:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19662.197273 secs 2024-07-26T18:18:48.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:47 smithi112 ceph-mon[37501]: pgmap v10787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:18:48.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:47 smithi107 ceph-mon[32659]: pgmap v10787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:18:48.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:47 smithi132 ceph-mon[37580]: pgmap v10787: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:18:50.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:49 smithi112 ceph-mon[37501]: pgmap v10788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:50.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:49 smithi107 ceph-mon[32659]: pgmap v10788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:50.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:49 smithi132 ceph-mon[37580]: pgmap v10788: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:52.112 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19667.197347 secs 2024-07-26T18:18:52.113 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:51 smithi132 ceph-mon[37580]: pgmap v10789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19667.197347 secs 2024-07-26T18:18:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:51 smithi112 ceph-mon[37501]: pgmap v10789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19667.197347 secs 2024-07-26T18:18:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:51 smithi107 ceph-mon[32659]: pgmap v10789: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:18:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:52.882+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:18:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:52.882+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:18:52.883 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:18:52.882+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:18:54.203 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:53 smithi112 ceph-mon[37501]: pgmap v10790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:54.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:53 smithi107 ceph-mon[32659]: pgmap v10790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:54.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:53 smithi132 ceph-mon[37580]: pgmap v10790: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:18:56.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:55 smithi112 ceph-mon[37501]: pgmap v10791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:56.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:55 smithi107 ceph-mon[32659]: pgmap v10791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:56.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:55 smithi132 ceph-mon[37580]: pgmap v10791: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:18:57.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19672.197467 secs 2024-07-26T18:18:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19672.197467 secs 2024-07-26T18:18:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19672.197467 secs 2024-07-26T18:18:58.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:57 smithi112 ceph-mon[37501]: pgmap v10792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:18:58.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:57 smithi107 ceph-mon[32659]: pgmap v10792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:18:58.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:57 smithi132 ceph-mon[37580]: pgmap v10792: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:00.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:18:59 smithi112 ceph-mon[37501]: pgmap v10793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:00.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:18:59 smithi107 ceph-mon[32659]: pgmap v10793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:00.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:18:59 smithi132 ceph-mon[37580]: pgmap v10793: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:02.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19677.197594 secs 2024-07-26T18:19:02.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:01 smithi112 ceph-mon[37501]: pgmap v10794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19677.197594 secs 2024-07-26T18:19:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:01 smithi107 ceph-mon[32659]: pgmap v10794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19677.197594 secs 2024-07-26T18:19:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:01 smithi132 ceph-mon[37580]: pgmap v10794: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:04.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:03 smithi112 ceph-mon[37501]: pgmap v10795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:04.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:03 smithi107 ceph-mon[32659]: pgmap v10795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:04.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:03 smithi132 ceph-mon[37580]: pgmap v10795: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:06.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:05 smithi112 ceph-mon[37501]: pgmap v10796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:19:06.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:05 smithi107 ceph-mon[32659]: pgmap v10796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:19:06.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:05 smithi132 ceph-mon[37580]: pgmap v10796: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:19:07.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19682.197702 secs 2024-07-26T18:19:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19682.197702 secs 2024-07-26T18:19:07.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19682.197702 secs 2024-07-26T18:19:07.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:07.895+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:19:07.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:07.895+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:19:07.897 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:07.895+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:19:08.194 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:07 smithi107 ceph-mon[32659]: pgmap v10797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:19:08.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:07 smithi112 ceph-mon[37501]: pgmap v10797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:19:08.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:07 smithi132 ceph-mon[37580]: pgmap v10797: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:19:10.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:09 smithi112 ceph-mon[37501]: pgmap v10798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:10.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:09 smithi107 ceph-mon[32659]: pgmap v10798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:10.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:09 smithi132 ceph-mon[37580]: pgmap v10798: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:12.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:11 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19687.197818 secs 2024-07-26T18:19:12.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:11 smithi112 ceph-mon[37501]: pgmap v10799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:11 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19687.197818 secs 2024-07-26T18:19:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:11 smithi107 ceph-mon[32659]: pgmap v10799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:11 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19687.197818 secs 2024-07-26T18:19:12.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:11 smithi132 ceph-mon[37580]: pgmap v10799: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:14.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:13 smithi112 ceph-mon[37501]: pgmap v10800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:14.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:13 smithi107 ceph-mon[32659]: pgmap v10800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:14.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:13 smithi132 ceph-mon[37580]: pgmap v10800: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:16.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:15 smithi112 ceph-mon[37501]: pgmap v10801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:19:16.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:15 smithi107 ceph-mon[32659]: pgmap v10801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:19:16.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:15 smithi132 ceph-mon[37580]: pgmap v10801: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:19:17.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:16 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19692.197932 secs 2024-07-26T18:19:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:16 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19692.197932 secs 2024-07-26T18:19:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:16 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19692.197932 secs 2024-07-26T18:19:18.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:17 smithi107 ceph-mon[32659]: pgmap v10802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:19:18.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:17 smithi132 ceph-mon[37580]: pgmap v10802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:19:18.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:17 smithi112 ceph-mon[37501]: pgmap v10802: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:19:20.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:19 smithi107 ceph-mon[32659]: pgmap v10803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:19:20.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:19 smithi132 ceph-mon[37580]: pgmap v10803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:19:20.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:19 smithi112 ceph-mon[37501]: pgmap v10803: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:19:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:21 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19697.198047 secs 2024-07-26T18:19:22.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:21 smithi107 ceph-mon[32659]: pgmap v10804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:19:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:21 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19697.198047 secs 2024-07-26T18:19:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:21 smithi132 ceph-mon[37580]: pgmap v10804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:19:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:21 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19697.198047 secs 2024-07-26T18:19:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:21 smithi112 ceph-mon[37501]: pgmap v10804: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T18:19:22.912 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:22.910+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:19:22.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:22.910+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:19:22.913 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:22.910+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:19:24.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:23 smithi107 ceph-mon[32659]: pgmap v10805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:19:24.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:23 smithi132 ceph-mon[37580]: pgmap v10805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:19:24.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:23 smithi112 ceph-mon[37501]: pgmap v10805: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:19:26.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:25 smithi107 ceph-mon[32659]: pgmap v10806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:19:26.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:25 smithi132 ceph-mon[37580]: pgmap v10806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:19:26.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:25 smithi112 ceph-mon[37501]: pgmap v10806: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T18:19:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:26 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19702.198187 secs 2024-07-26T18:19:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:26 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:19:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:26 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19702.198187 secs 2024-07-26T18:19:27.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:26 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:19:27.351 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:26 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19702.198187 secs 2024-07-26T18:19:27.352 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:26 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T18:19:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:28 smithi107 ceph-mon[32659]: pgmap v10807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:19:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:19:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:28 smithi107 ceph-mon[32659]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:19:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:28 smithi107 ceph-mon[32659]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:19:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:28 smithi132 ceph-mon[37580]: pgmap v10807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:19:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:19:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:28 smithi132 ceph-mon[37580]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:19:28.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:28 smithi132 ceph-mon[37580]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:19:28.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:28 smithi112 ceph-mon[37501]: pgmap v10807: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T18:19:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T18:19:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:28 smithi112 ceph-mon[37501]: from='mgr.14211 172.21.15.107:0/2106856309' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T18:19:28.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:28 smithi112 ceph-mon[37501]: from='mgr.14211 ' entity='mgr.x' 2024-07-26T18:19:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:29 smithi107 ceph-mon[32659]: pgmap v10808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:19:29.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:29 smithi107 ceph-mon[32659]: pgmap v10809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:19:29.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:29 smithi132 ceph-mon[37580]: pgmap v10808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:19:29.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:29 smithi132 ceph-mon[37580]: pgmap v10809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:19:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:29 smithi112 ceph-mon[37501]: pgmap v10808: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T18:19:29.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:29 smithi112 ceph-mon[37501]: pgmap v10809: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:19:31.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:31 smithi107 ceph-mon[32659]: pgmap v10810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:19:31.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:31 smithi132 ceph-mon[37580]: pgmap v10810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:19:31.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:31 smithi112 ceph-mon[37501]: pgmap v10810: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T18:19:32.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:32 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19707.198298 secs 2024-07-26T18:19:32.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:32 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19707.198298 secs 2024-07-26T18:19:32.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:32 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19707.198298 secs 2024-07-26T18:19:33.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:33 smithi107 ceph-mon[32659]: pgmap v10811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:33.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:33 smithi132 ceph-mon[37580]: pgmap v10811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:33.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:33 smithi112 ceph-mon[37501]: pgmap v10811: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:35.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:35 smithi107 ceph-mon[32659]: pgmap v10812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:19:35.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:35 smithi132 ceph-mon[37580]: pgmap v10812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:19:35.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:35 smithi112 ceph-mon[37501]: pgmap v10812: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T18:19:37.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:37 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19712.198433 secs 2024-07-26T18:19:37.311 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:37 smithi132 ceph-mon[37580]: pgmap v10813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:19:37.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:37 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19712.198433 secs 2024-07-26T18:19:37.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:37 smithi112 ceph-mon[37501]: pgmap v10813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:19:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:37 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19712.198433 secs 2024-07-26T18:19:37.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:37 smithi107 ceph-mon[32659]: pgmap v10813: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T18:19:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:37.926+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:19:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:37.926+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:19:37.928 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:37.926+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:19:39.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:39 smithi132 ceph-mon[37580]: pgmap v10814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:19:39.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:39 smithi112 ceph-mon[37501]: pgmap v10814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:19:39.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:39 smithi107 ceph-mon[32659]: pgmap v10814: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T18:19:41.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:41 smithi132 ceph-mon[37580]: pgmap v10815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:41.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:41 smithi112 ceph-mon[37501]: pgmap v10815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:41.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:41 smithi107 ceph-mon[32659]: pgmap v10815: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:42.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:42 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19717.198551 secs 2024-07-26T18:19:42.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:42 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19717.198551 secs 2024-07-26T18:19:42.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:42 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19717.198551 secs 2024-07-26T18:19:43.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:43 smithi112 ceph-mon[37501]: pgmap v10816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:43.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:43 smithi107 ceph-mon[32659]: pgmap v10816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:43.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:43 smithi132 ceph-mon[37580]: pgmap v10816: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:44.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:44 smithi112 ceph-mon[37501]: pgmap v10817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:44.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:44 smithi107 ceph-mon[32659]: pgmap v10817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:44.560 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:44 smithi132 ceph-mon[37580]: pgmap v10817: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:47.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:46 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19722.198672 secs 2024-07-26T18:19:47.201 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:46 smithi112 ceph-mon[37501]: pgmap v10818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:46 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19722.198672 secs 2024-07-26T18:19:47.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:46 smithi107 ceph-mon[32659]: pgmap v10818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:47.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:46 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19722.198672 secs 2024-07-26T18:19:47.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:46 smithi132 ceph-mon[37580]: pgmap v10818: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:49.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:48 smithi112 ceph-mon[37501]: pgmap v10819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:49.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:48 smithi107 ceph-mon[32659]: pgmap v10819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:49.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:48 smithi132 ceph-mon[37580]: pgmap v10819: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:51.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:50 smithi112 ceph-mon[37501]: pgmap v10820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:51.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:50 smithi107 ceph-mon[32659]: pgmap v10820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:51.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:50 smithi132 ceph-mon[37580]: pgmap v10820: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:52.195 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:51 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19727.198787 secs 2024-07-26T18:19:52.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:51 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19727.198787 secs 2024-07-26T18:19:52.211 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:51 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19727.198787 secs 2024-07-26T18:19:52.931 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:52.930+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:19:52.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:52.930+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:19:52.932 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:19:52.930+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:19:53.200 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:52 smithi112 ceph-mon[37501]: pgmap v10821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:53.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:52 smithi107 ceph-mon[32659]: pgmap v10821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:53.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:52 smithi132 ceph-mon[37580]: pgmap v10821: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:19:55.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:54 smithi107 ceph-mon[32659]: pgmap v10822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:55.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:54 smithi132 ceph-mon[37580]: pgmap v10822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:55.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:54 smithi112 ceph-mon[37501]: pgmap v10822: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:57.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:56 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19732.198919 secs 2024-07-26T18:19:57.253 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:56 smithi112 ceph-mon[37501]: pgmap v10823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:56 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19732.198919 secs 2024-07-26T18:19:57.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:56 smithi107 ceph-mon[32659]: pgmap v10823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:56 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19732.198919 secs 2024-07-26T18:19:57.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:56 smithi132 ceph-mon[37580]: pgmap v10823: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:19:59.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:19:58 smithi107 ceph-mon[32659]: pgmap v10824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:59.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:19:58 smithi132 ceph-mon[37580]: pgmap v10824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:19:59.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:19:58 smithi112 ceph-mon[37501]: pgmap v10824: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:01.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:00 smithi107 ceph-mon[32659]: pgmap v10825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:01.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:00 smithi107 ceph-mon[32659]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T18:20:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:00 smithi132 ceph-mon[37580]: pgmap v10825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:01.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:00 smithi132 ceph-mon[37580]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T18:20:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:00 smithi112 ceph-mon[37501]: pgmap v10825: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:01.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:00 smithi112 ceph-mon[37501]: overall HEALTH_WARN 4 failed cephadm daemon(s); 1 MDSs report slow requests 2024-07-26T18:20:02.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:01 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19737.199019 secs 2024-07-26T18:20:02.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:01 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19737.199019 secs 2024-07-26T18:20:02.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:01 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19737.199019 secs 2024-07-26T18:20:03.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:02 smithi107 ceph-mon[32659]: pgmap v10826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:03.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:02 smithi132 ceph-mon[37580]: pgmap v10826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:03.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:02 smithi112 ceph-mon[37501]: pgmap v10826: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:05.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:04 smithi107 ceph-mon[32659]: pgmap v10827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:05.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:04 smithi132 ceph-mon[37580]: pgmap v10827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:05.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:04 smithi112 ceph-mon[37501]: pgmap v10827: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:06 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19742.199139 secs 2024-07-26T18:20:07.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:06 smithi107 ceph-mon[32659]: pgmap v10828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:20:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:06 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19742.199139 secs 2024-07-26T18:20:07.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:06 smithi132 ceph-mon[37580]: pgmap v10828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:20:07.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:06 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19742.199139 secs 2024-07-26T18:20:07.451 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:06 smithi112 ceph-mon[37501]: pgmap v10828: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:20:07.940 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:20:07.939+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:20:07.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:20:07.939+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:20:07.941 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:20:07.939+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:20:09.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:09 smithi107 ceph-mon[32659]: pgmap v10829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:09.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:09 smithi132 ceph-mon[37580]: pgmap v10829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:09.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:09 smithi112 ceph-mon[37501]: pgmap v10829: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:11.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:11 smithi107 ceph-mon[32659]: pgmap v10830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:11.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:11 smithi132 ceph-mon[37580]: pgmap v10830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:11.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:11 smithi112 ceph-mon[37501]: pgmap v10830: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:12.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:12 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19747.199255 secs 2024-07-26T18:20:12.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:12 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19747.199255 secs 2024-07-26T18:20:12.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:12 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19747.199255 secs 2024-07-26T18:20:13.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:13 smithi107 ceph-mon[32659]: pgmap v10831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:13.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:13 smithi132 ceph-mon[37580]: pgmap v10831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:13.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:13 smithi112 ceph-mon[37501]: pgmap v10831: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:15.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:15 smithi107 ceph-mon[32659]: pgmap v10832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:15.309 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:15 smithi132 ceph-mon[37580]: pgmap v10832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:15.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:15 smithi112 ceph-mon[37501]: pgmap v10832: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:17 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19752.199337 secs 2024-07-26T18:20:17.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:17 smithi107 ceph-mon[32659]: pgmap v10833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:20:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:17 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19752.199337 secs 2024-07-26T18:20:17.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:17 smithi132 ceph-mon[37580]: pgmap v10833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:20:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:17 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19752.199337 secs 2024-07-26T18:20:17.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:17 smithi112 ceph-mon[37501]: pgmap v10833: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T18:20:19.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:19 smithi107 ceph-mon[32659]: pgmap v10834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:19.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:19 smithi132 ceph-mon[37580]: pgmap v10834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:19.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:19 smithi112 ceph-mon[37501]: pgmap v10834: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:21.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:21 smithi132 ceph-mon[37580]: pgmap v10835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:21.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:21 smithi112 ceph-mon[37501]: pgmap v10835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:21.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:21 smithi107 ceph-mon[32659]: pgmap v10835: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:22.310 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:22 smithi132 ceph-mon[37580]: 1 slow requests, 0 included below; oldest blocked for > 19757.199458 secs 2024-07-26T18:20:22.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:22 smithi112 ceph-mon[37501]: 1 slow requests, 0 included below; oldest blocked for > 19757.199458 secs 2024-07-26T18:20:22.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:22 smithi107 ceph-mon[32659]: 1 slow requests, 0 included below; oldest blocked for > 19757.199458 secs 2024-07-26T18:20:22.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:20:22.954+0000 7ff108e00640 1 -- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 msgr2=0x565421d76a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.112:6838/3779803434 2024-07-26T18:20:22.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:20:22.954+0000 7ff0eaa00640 1 client.25934 ms_handle_refused on v2:172.21.15.112:6838/3779803434 2024-07-26T18:20:22.956 INFO:teuthology.orchestra.run.smithi107.stderr:2024-07-26T18:20:22.954+0000 7ff108e00640 1 --2- 172.21.15.107:0/4208142334 >> [v2:172.21.15.112:6838/3779803434,v1:172.21.15.112:6839/3779803434] conn(0x5654219729f0 0x565421d76a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T18:20:23.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:23 smithi107 ceph-mon[32659]: pgmap v10836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:23.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:23 smithi112 ceph-mon[37501]: pgmap v10836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:23.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:23 smithi132 ceph-mon[37580]: pgmap v10836: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T18:20:25.450 INFO:journalctl@ceph.mon.b.smithi112.stdout:Jul 26 18:20:25 smithi112 ceph-mon[37501]: pgmap v10837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:25.542 INFO:journalctl@ceph.mon.a.smithi107.stdout:Jul 26 18:20:25 smithi107 ceph-mon[32659]: pgmap v10837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:25.559 INFO:journalctl@ceph.mon.c.smithi132.stdout:Jul 26 18:20:25 smithi132 ceph-mon[37580]: pgmap v10837: 129 pgs: 129 active+clean; 38 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T18:20:25.790 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-07-26T18:20:25.790 DEBUG:teuthology.task.console_log:Killing console logger for smithi107 2024-07-26T18:20:25.791 DEBUG:teuthology.task.console_log:Killing console logger for smithi112 2024-07-26T18:20:25.792 DEBUG:teuthology.task.console_log:Killing console logger for smithi132 2024-07-26T18:20:25.792 DEBUG:teuthology.exit:Finished running handlers